Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-07-11
15:56:09 ygk_12345 i tried pdb inside nova/cmd/compute.py file and could see that current_version is getting value 30
16:12:12 opendevreview Sylvain Bauza proposed openstack/nova master: WIP: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133
16:12:28 bauzas still a WIP due to UTs missing ^
16:17:45 sean-k-mooney ygk_12345: in addtion to deleting the old compute services you will have to restart the conductors and posibly the other contoller services
16:17:57 sean-k-mooney ygk_12345: they cache the min compute version on start up of the service
16:18:07 opendevreview Sylvain Bauza proposed openstack/nova master: WIP: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133
16:18:14 ygk_12345 sean-k-mooney: is this in addition to the db entries deletion ?
16:18:42 sean-k-mooney yes so you will need to delete the old compute service records assuming those host nolonger exist
16:18:55 sean-k-mooney and if you have already tried to start the conductor ectra then you will have to restart it
16:19:03 sean-k-mooney ygk_12345: you should only delete the old entries
16:19:08 sean-k-mooney if those host are gone
16:19:15 sean-k-mooney and will not be coming back
16:19:38 sean-k-mooney ygk_12345: there is a workaround for this if you are fast forward upgrading
16:20:02 sean-k-mooney but if you use that you need to be aware that we dont support configuration where the min compute service version is not met
16:20:39 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_compute_service_check_for_ffu
16:20:42 ygk_12345 sean-k-mooney: when restarting the nova-compute service, do they cache any db entries as well ?
16:21:31 sean-k-mooney restarting the nova-comptue service would clear any caching it had since it only caches info in memory
16:21:56 sean-k-mooney we dont cache stuff on disk so there is nothing that would need to be cleaned up manually
16:23:06 ygk_12345 sean-k-mooney: but from where is it picking up the 30 version , even though its not there in the db ? maybe from the memory ?
16:24:04 sean-k-mooney either the binary you are starting is not at the min version or its coming from the db via an rpc to the conductor to get the min service version in the cell
16:25:00 ygk_12345 sean-k-mooney: but the nova.services table doesn't have any trace of version 30
16:25:23 ygk_12345 sean-k-mooney: also this workaround , is it in control plane nova.conf or in compute nova.conf ?
16:25:26 sean-k-mooney ygk_12345: have you recently done an upgrade to wallaby or are you in the process of an upgrade?
16:25:38 sean-k-mooney ygk_12345: the workaround need to be set on all hosts
16:25:49 ygk_12345 sean-k-mooney: we have upgraded only the control plane
16:26:00 sean-k-mooney from what release
16:26:07 ygk_12345 sean-k-mooney: and we are upgrading computes step by step
16:26:16 ygk_12345 sean-k-mooney: OSA 23.2.0
16:26:43 sean-k-mooney what openstack release does that map too
16:26:51 ygk_12345 wallaby
16:27:01 sean-k-mooney so 23 is wallayby
16:27:07 ygk_12345 yes
16:27:07 sean-k-mooney and you are coming form victoria?
16:27:29 ygk_12345 yes ussuri->victoria->Wallaby
16:27:51 sean-k-mooney ok so you upgraded the contolers to victoria then upgreaed all the computes
16:27:58 sean-k-mooney then upgraded contoler to wallayby
16:28:02 sean-k-mooney then upgraded the computes
16:28:39 sean-k-mooney you cant upgrade the controlers directly form ussuri to wallayby in one go
16:28:52 ygk_12345 no no. upgraded first the control plane from stein->train->ussuri>vic>wallaby. Then upgraded the computes
16:29:08 sean-k-mooney ok so you are doing a fast forward upgrade
16:29:17 sean-k-mooney or skiplevel depneind on the branding
16:29:26 sean-k-mooney that is not supported by nova directly
16:29:37 sean-k-mooney so you will need to disable our validation with the workaround
16:29:47 sean-k-mooney then upgade allt eh compute to a supported version
16:29:52 sean-k-mooney then renable the check
16:30:16 ygk_12345 so the version 30 is being picked up by cache somewhere ?
16:30:18 sean-k-mooney nova and most service only suport a n to n+1 version delta
16:30:44 sean-k-mooney unlikely
16:31:02 sean-k-mooney its more likely that you are trying to start a stein compute with a wallaby contoler
16:31:23 ygk_12345 yes
16:31:30 sean-k-mooney right that is not supported
16:31:32 ygk_12345 but stein is not 30 version
16:31:52 sean-k-mooney victoria should be version 30
16:32:18 ygk_12345 i deleted older versions containers from nova.services tables completely
16:32:31 sean-k-mooney that is not a good thing
16:32:41 sean-k-mooney if you have intnaces on the cloud that would break placment
16:33:04 ygk_12345 we have successfully upgraded in other platforms
16:35:57 ygk_12345 sean-k-mooney: i wil try restarting all the nova services
16:36:25 ygk_12345 sean-k-mooney: thanks bro for your time. appreciate it.....
16:36:26 sean-k-mooney the compute service version in stien was 37 by the way https://github.com/openstack/nova/blob/stable/stein/nova/objects/service.py#L34
16:36:33 ygk_12345 yes
16:36:36 ygk_12345 not 30
16:36:40 sean-k-mooney 30 was queens
16:36:44 ygk_12345 yes
16:37:14 sean-k-mooney and the miniutm version supproted by a wallaby controler is 52/victoria
16:38:15 ygk_12345 we are totally putting a new os image with wallaby nova-compute version , retaining the vms from the older compute whil rebooting
16:38:35 ygk_12345 the /va/lib/nova/instances is a separate partition
16:39:22 sean-k-mooney so the conductor and other contoller service should not be able to start without bring the computes to victoria first if you are not using the workaround currently
16:40:14 sean-k-mooney i dont know where the queens service is coming form if you have delete the serivcies in the db
16:40:26 ygk_12345 thats what strange
16:40:31 sean-k-mooney but the only place in code it could come form on teh compute node is the nova package
16:40:56 sean-k-mooney implying you have queens code. other wise it has to be coming form the db
16:42:03 ygk_12345 i think its db cache in the memory . i will try a control plane nova services restart and check
16:43:11 sean-k-mooney ygk_12345: i would expect this code to prevent thet conductor form starting https://github.com/openstack/nova/blob/b320f16b851fd1e5238c0b49c780f6a9c6851e48/nova/utils.py#L1053-L1100=
16:44:28 ygk_12345 sean-k-mooney: yes exactly. that part of the code is giving 'current_service_version' to 30 during the pdb
16:44:48 ygk_12345 i cant understand from where it is picking it up
16:44:59 sean-k-mooney have you checked both the api and cell db
16:45:17 ygk_12345 which tables in api and cell ?
16:46:17 sean-k-mooney the service table in the cell db
16:46:24 ygk_12345 it is empty
16:47:06 sean-k-mooney you dont have any entries in cell0 or cell1
16:47:35 ygk_12345 yes there are.. but ony nova.services table has entries
16:48:00 sean-k-mooney i assume nova.service is cell0?
16:48:10 ygk_12345 no
16:48:15 sean-k-mooney ok its cell1
16:48:25 sean-k-mooney the nameing depends on the deployemnt too
16:48:28 ygk_12345 just the plain nova db . inside it 'services' table
16:48:46 sean-k-mooney right so that db name "nova" depends on your deployment too
16:48:51 ygk_12345 i see nova.cell0 and placement dbs
16:49:09 ygk_12345 and also nova and nova_api
16:49:13 sean-k-mooney ok then its proably teh cell 1 database
16:49:32 ygk_12345 is there a services table inside it as well ?
16:49:47 sean-k-mooney ya so there should be a service table in every cell database
16:49:55 sean-k-mooney but you should not delete the services
16:50:02 ygk_12345 ok
16:50:04 sean-k-mooney if you do it will break your deployment
16:50:10 ygk_12345 yes got it
16:50:22 sean-k-mooney if you delete the services then teh compute agent wil create a new service entry when it starts
16:50:30 ygk_12345 so i will just retry with nova control plane restart
16:50:34 sean-k-mooney that will resullt in a different serivce uuid and break placment

Earlier   Later