| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-11 | |||
| 15:22:49 | bauzas | but... the pubkey is exactly good | |
| 15:24:17 | bauzas | wtf, /me is wrapping his head | |
| 15:27:56 | sean-k-mooney | my guess is maybe there are som spaces or other whitespace | |
| 15:28:10 | sean-k-mooney | i would try removeing the ^ and $ | |
| 15:29:04 | bauzas | I can't remove those | |
| 15:30:17 | bauzas | sean-k-mooney: https://github.com/openstack/nova/blob/master/nova/tests/functional/api_samples_test_base.py#L248-L251 | |
| 15:30:47 | bauzas | before adding the ^ and $ for regexp, I verified the strings | |
| 15:32:38 | bauzas | https://paste.opendev.org/show/bixG1pb6FdkuMcZNYBqK/ shows the strings matching before we add the trailing chars | |
| 15:46:00 | bauzas | oh, I think I spotted it | |
| 15:54:22 | gibi | -/16 | |
| 15:54:48 | ygk_12345 | Hi all. we have a wallaby setup. the compute compute services are not starting saying "too old compute version 30". But we have deleted that version from the nova.services db table. But still it is picking up version 30 which we cant find. How to resolve this > | |
| 15:55:18 | ygk_12345 | bauzas: any idea about this issue ? | |
| 15:56:09 | ygk_12345 | i tried pdb inside nova/cmd/compute.py file and could see that current_version is getting value 30 | |
| 16:12:12 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133 | |
| 16:12:28 | bauzas | still a WIP due to UTs missing ^ | |
| 16:17:45 | sean-k-mooney | ygk_12345: in addtion to deleting the old compute services you will have to restart the conductors and posibly the other contoller services | |
| 16:17:57 | sean-k-mooney | ygk_12345: they cache the min compute version on start up of the service | |
| 16:18:07 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133 | |
| 16:18:14 | ygk_12345 | sean-k-mooney: is this in addition to the db entries deletion ? | |
| 16:18:42 | sean-k-mooney | yes so you will need to delete the old compute service records assuming those host nolonger exist | |
| 16:18:55 | sean-k-mooney | and if you have already tried to start the conductor ectra then you will have to restart it | |
| 16:19:03 | sean-k-mooney | ygk_12345: you should only delete the old entries | |
| 16:19:08 | sean-k-mooney | if those host are gone | |
| 16:19:15 | sean-k-mooney | and will not be coming back | |
| 16:19:38 | sean-k-mooney | ygk_12345: there is a workaround for this if you are fast forward upgrading | |
| 16:20:02 | sean-k-mooney | but if you use that you need to be aware that we dont support configuration where the min compute service version is not met | |
| 16:20:39 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_compute_service_check_for_ffu | |
| 16:20:42 | ygk_12345 | sean-k-mooney: when restarting the nova-compute service, do they cache any db entries as well ? | |
| 16:21:31 | sean-k-mooney | restarting the nova-comptue service would clear any caching it had since it only caches info in memory | |
| 16:21:56 | sean-k-mooney | we dont cache stuff on disk so there is nothing that would need to be cleaned up manually | |
| 16:23:06 | ygk_12345 | sean-k-mooney: but from where is it picking up the 30 version , even though its not there in the db ? maybe from the memory ? | |
| 16:24:04 | sean-k-mooney | either the binary you are starting is not at the min version or its coming from the db via an rpc to the conductor to get the min service version in the cell | |
| 16:25:00 | ygk_12345 | sean-k-mooney: but the nova.services table doesn't have any trace of version 30 | |
| 16:25:23 | ygk_12345 | sean-k-mooney: also this workaround , is it in control plane nova.conf or in compute nova.conf ? | |
| 16:25:26 | sean-k-mooney | ygk_12345: have you recently done an upgrade to wallaby or are you in the process of an upgrade? | |
| 16:25:38 | sean-k-mooney | ygk_12345: the workaround need to be set on all hosts | |
| 16:25:49 | ygk_12345 | sean-k-mooney: we have upgraded only the control plane | |
| 16:26:00 | sean-k-mooney | from what release | |
| 16:26:07 | ygk_12345 | sean-k-mooney: and we are upgrading computes step by step | |
| 16:26:16 | ygk_12345 | sean-k-mooney: OSA 23.2.0 | |
| 16:26:43 | sean-k-mooney | what openstack release does that map too | |
| 16:26:51 | ygk_12345 | wallaby | |
| 16:27:01 | sean-k-mooney | so 23 is wallayby | |
| 16:27:07 | sean-k-mooney | and you are coming form victoria? | |
| 16:27:07 | ygk_12345 | yes | |
| 16:27:29 | ygk_12345 | yes ussuri->victoria->Wallaby | |
| 16:27:51 | sean-k-mooney | ok so you upgraded the contolers to victoria then upgreaed all the computes | |
| 16:27:58 | sean-k-mooney | then upgraded contoler to wallayby | |
| 16:28:02 | sean-k-mooney | then upgraded the computes | |
| 16:28:39 | sean-k-mooney | you cant upgrade the controlers directly form ussuri to wallayby in one go | |
| 16:28:52 | ygk_12345 | no no. upgraded first the control plane from stein->train->ussuri>vic>wallaby. Then upgraded the computes | |
| 16:29:08 | sean-k-mooney | ok so you are doing a fast forward upgrade | |
| 16:29:17 | sean-k-mooney | or skiplevel depneind on the branding | |
| 16:29:26 | sean-k-mooney | that is not supported by nova directly | |
| 16:29:37 | sean-k-mooney | so you will need to disable our validation with the workaround | |
| 16:29:47 | sean-k-mooney | then upgade allt eh compute to a supported version | |
| 16:29:52 | sean-k-mooney | then renable the check | |
| 16:30:16 | ygk_12345 | so the version 30 is being picked up by cache somewhere ? | |
| 16:30:18 | sean-k-mooney | nova and most service only suport a n to n+1 version delta | |
| 16:30:44 | sean-k-mooney | unlikely | |
| 16:31:02 | sean-k-mooney | its more likely that you are trying to start a stein compute with a wallaby contoler | |
| 16:31:23 | ygk_12345 | yes | |
| 16:31:30 | sean-k-mooney | right that is not supported | |
| 16:31:32 | ygk_12345 | but stein is not 30 version | |
| 16:31:52 | sean-k-mooney | victoria should be version 30 | |
| 16:32:18 | ygk_12345 | i deleted older versions containers from nova.services tables completely | |
| 16:32:31 | sean-k-mooney | that is not a good thing | |
| 16:32:41 | sean-k-mooney | if you have intnaces on the cloud that would break placment | |
| 16:33:04 | ygk_12345 | we have successfully upgraded in other platforms | |
| 16:35:57 | ygk_12345 | sean-k-mooney: i wil try restarting all the nova services | |
| 16:36:25 | ygk_12345 | sean-k-mooney: thanks bro for your time. appreciate it..... | |
| 16:36:26 | sean-k-mooney | the compute service version in stien was 37 by the way https://github.com/openstack/nova/blob/stable/stein/nova/objects/service.py#L34 | |
| 16:36:33 | ygk_12345 | yes | |
| 16:36:36 | ygk_12345 | not 30 | |
| 16:36:40 | sean-k-mooney | 30 was queens | |
| 16:36:44 | ygk_12345 | yes | |
| 16:37:14 | sean-k-mooney | and the miniutm version supproted by a wallaby controler is 52/victoria | |
| 16:38:15 | ygk_12345 | we are totally putting a new os image with wallaby nova-compute version , retaining the vms from the older compute whil rebooting | |
| 16:38:35 | ygk_12345 | the /va/lib/nova/instances is a separate partition | |
| 16:39:22 | sean-k-mooney | so the conductor and other contoller service should not be able to start without bring the computes to victoria first if you are not using the workaround currently | |
| 16:40:14 | sean-k-mooney | i dont know where the queens service is coming form if you have delete the serivcies in the db | |
| 16:40:26 | ygk_12345 | thats what strange | |
| 16:40:31 | sean-k-mooney | but the only place in code it could come form on teh compute node is the nova package | |
| 16:40:56 | sean-k-mooney | implying you have queens code. other wise it has to be coming form the db | |
| 16:42:03 | ygk_12345 | i think its db cache in the memory . i will try a control plane nova services restart and check | |
| 16:43:11 | sean-k-mooney | ygk_12345: i would expect this code to prevent thet conductor form starting https://github.com/openstack/nova/blob/b320f16b851fd1e5238c0b49c780f6a9c6851e48/nova/utils.py#L1053-L1100= | |
| 16:44:28 | ygk_12345 | sean-k-mooney: yes exactly. that part of the code is giving 'current_service_version' to 30 during the pdb | |
| 16:44:48 | ygk_12345 | i cant understand from where it is picking it up | |
| 16:44:59 | sean-k-mooney | have you checked both the api and cell db | |
| 16:45:17 | ygk_12345 | which tables in api and cell ? | |
| 16:46:17 | sean-k-mooney | the service table in the cell db | |
| 16:46:24 | ygk_12345 | it is empty | |
| 16:47:06 | sean-k-mooney | you dont have any entries in cell0 or cell1 | |
| 16:47:35 | ygk_12345 | yes there are.. but ony nova.services table has entries | |
| 16:48:00 | sean-k-mooney | i assume nova.service is cell0? | |
| 16:48:10 | ygk_12345 | no | |
| 16:48:15 | sean-k-mooney | ok its cell1 | |
| 16:48:25 | sean-k-mooney | the nameing depends on the deployemnt too | |
| 16:48:28 | ygk_12345 | just the plain nova db . inside it 'services' table | |
| 16:48:46 | sean-k-mooney | right so that db name "nova" depends on your deployment too | |