Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-07-11
12:09:40 opendevreview Stephen Finucane proposed openstack/placement master: Fix typo in schema https://review.opendev.org/c/openstack/placement/+/849348
12:12:33 stephenfin gibi: bauzas: sean-k-mooney: I've already sort of made my mind up on this but can you take a look at https://review.opendev.org/c/openstack/placement/+/849348 (context is a change from ratailor at https://review.opendev.org/c/openstack/placement/+/848634)
12:15:17 sean-k-mooney[m] ya we would need a new microverion i belive
12:15:29 sean-k-mooney[m] unless we were failing on a 500
12:15:46 sean-k-mooney[m] because we checked in the code somewhere else
12:17:10 bauzas stephenfin: I can try to take a look once I'm done with my own change
12:18:16 stephenfin sean-k-mooney[m] If I revert the code change, the new test I added doesn't fail. That would suggest we were happily accepting an empty object at the code level (presumably because we do falsey checks or similar)
12:18:38 sean-k-mooney[m] ack
12:18:58 gibi "This field may be sent when writing allocations back to the server but will be ignored; this preserves symmetry between read and write representations."
12:19:09 gibi so this value is never used by placement
12:19:22 gibi I think we don't need a microversion in this case
12:19:54 sean-k-mooney[m] oh its the provider tree mappings
12:20:12 sean-k-mooney[m] to lookup the provdier tree form the allocation
12:20:34 gibi (context, the mappings are generated by placement during GET allocation_candidates for the client to know which group are fulfiulled from which RP, so writing this back to placement only make sense for keeping the a_c result directly usable for POST allocations
12:20:38 gibi )
12:20:55 sean-k-mooney[m] ack
12:21:26 sean-k-mooney[m] so we likely can fix it but we should have the release note to call it out
12:21:42 sean-k-mooney[m] which stephenfin has in there patch
12:26:06 gibi yepp, so I'm +2
15:22:37 bauzas https://paste.opendev.org/show/bR65i5QPAN7ieYOEJHF4/ fails because the pubkey doesn't re.match()
15:22:49 bauzas but... the pubkey is exactly good
15:24:17 bauzas wtf, /me is wrapping his head
15:27:56 sean-k-mooney my guess is maybe there are som spaces or other whitespace
15:28:10 sean-k-mooney i would try removeing the ^ and $
15:29:04 bauzas I can't remove those
15:30:17 bauzas sean-k-mooney: https://github.com/openstack/nova/blob/master/nova/tests/functional/api_samples_test_base.py#L248-L251
15:30:47 bauzas before adding the ^ and $ for regexp, I verified the strings
15:32:38 bauzas https://paste.opendev.org/show/bixG1pb6FdkuMcZNYBqK/ shows the strings matching before we add the trailing chars
15:46:00 bauzas oh, I think I spotted it
15:54:22 gibi -/16
15:54:48 ygk_12345 Hi all. we have a wallaby setup. the compute compute services are not starting saying "too old compute version 30". But we have deleted that version from the nova.services db table. But still it is picking up version 30 which we cant find. How to resolve this >
15:55:18 ygk_12345 bauzas: any idea about this issue ?
15:56:09 ygk_12345 i tried pdb inside nova/cmd/compute.py file and could see that current_version is getting value 30
16:12:12 opendevreview Sylvain Bauza proposed openstack/nova master: WIP: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133
16:12:28 bauzas still a WIP due to UTs missing ^
16:17:45 sean-k-mooney ygk_12345: in addtion to deleting the old compute services you will have to restart the conductors and posibly the other contoller services
16:17:57 sean-k-mooney ygk_12345: they cache the min compute version on start up of the service
16:18:07 opendevreview Sylvain Bauza proposed openstack/nova master: WIP: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133
16:18:14 ygk_12345 sean-k-mooney: is this in addition to the db entries deletion ?
16:18:42 sean-k-mooney yes so you will need to delete the old compute service records assuming those host nolonger exist
16:18:55 sean-k-mooney and if you have already tried to start the conductor ectra then you will have to restart it
16:19:03 sean-k-mooney ygk_12345: you should only delete the old entries
16:19:08 sean-k-mooney if those host are gone
16:19:15 sean-k-mooney and will not be coming back
16:19:38 sean-k-mooney ygk_12345: there is a workaround for this if you are fast forward upgrading
16:20:02 sean-k-mooney but if you use that you need to be aware that we dont support configuration where the min compute service version is not met
16:20:39 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.disable_compute_service_check_for_ffu
16:20:42 ygk_12345 sean-k-mooney: when restarting the nova-compute service, do they cache any db entries as well ?
16:21:31 sean-k-mooney restarting the nova-comptue service would clear any caching it had since it only caches info in memory
16:21:56 sean-k-mooney we dont cache stuff on disk so there is nothing that would need to be cleaned up manually
16:23:06 ygk_12345 sean-k-mooney: but from where is it picking up the 30 version , even though its not there in the db ? maybe from the memory ?
16:24:04 sean-k-mooney either the binary you are starting is not at the min version or its coming from the db via an rpc to the conductor to get the min service version in the cell
16:25:00 ygk_12345 sean-k-mooney: but the nova.services table doesn't have any trace of version 30
16:25:23 ygk_12345 sean-k-mooney: also this workaround , is it in control plane nova.conf or in compute nova.conf ?
16:25:26 sean-k-mooney ygk_12345: have you recently done an upgrade to wallaby or are you in the process of an upgrade?
16:25:38 sean-k-mooney ygk_12345: the workaround need to be set on all hosts
16:25:49 ygk_12345 sean-k-mooney: we have upgraded only the control plane
16:26:00 sean-k-mooney from what release
16:26:07 ygk_12345 sean-k-mooney: and we are upgrading computes step by step
16:26:16 ygk_12345 sean-k-mooney: OSA 23.2.0
16:26:43 sean-k-mooney what openstack release does that map too
16:26:51 ygk_12345 wallaby
16:27:01 sean-k-mooney so 23 is wallayby
16:27:07 ygk_12345 yes
16:27:07 sean-k-mooney and you are coming form victoria?
16:27:29 ygk_12345 yes ussuri->victoria->Wallaby
16:27:51 sean-k-mooney ok so you upgraded the contolers to victoria then upgreaed all the computes
16:27:58 sean-k-mooney then upgraded contoler to wallayby
16:28:02 sean-k-mooney then upgraded the computes
16:28:39 sean-k-mooney you cant upgrade the controlers directly form ussuri to wallayby in one go
16:28:52 ygk_12345 no no. upgraded first the control plane from stein->train->ussuri>vic>wallaby. Then upgraded the computes
16:29:08 sean-k-mooney ok so you are doing a fast forward upgrade
16:29:17 sean-k-mooney or skiplevel depneind on the branding
16:29:26 sean-k-mooney that is not supported by nova directly
16:29:37 sean-k-mooney so you will need to disable our validation with the workaround
16:29:47 sean-k-mooney then upgade allt eh compute to a supported version
16:29:52 sean-k-mooney then renable the check
16:30:16 ygk_12345 so the version 30 is being picked up by cache somewhere ?
16:30:18 sean-k-mooney nova and most service only suport a n to n+1 version delta
16:30:44 sean-k-mooney unlikely
16:31:02 sean-k-mooney its more likely that you are trying to start a stein compute with a wallaby contoler
16:31:23 ygk_12345 yes
16:31:30 sean-k-mooney right that is not supported
16:31:32 ygk_12345 but stein is not 30 version
16:31:52 sean-k-mooney victoria should be version 30
16:32:18 ygk_12345 i deleted older versions containers from nova.services tables completely
16:32:31 sean-k-mooney that is not a good thing
16:32:41 sean-k-mooney if you have intnaces on the cloud that would break placment
16:33:04 ygk_12345 we have successfully upgraded in other platforms
16:35:57 ygk_12345 sean-k-mooney: i wil try restarting all the nova services
16:36:25 ygk_12345 sean-k-mooney: thanks bro for your time. appreciate it.....
16:36:26 sean-k-mooney the compute service version in stien was 37 by the way https://github.com/openstack/nova/blob/stable/stein/nova/objects/service.py#L34
16:36:33 ygk_12345 yes
16:36:36 ygk_12345 not 30
16:36:40 sean-k-mooney 30 was queens
16:36:44 ygk_12345 yes
16:37:14 sean-k-mooney and the miniutm version supproted by a wallaby controler is 52/victoria
16:38:15 ygk_12345 we are totally putting a new os image with wallaby nova-compute version , retaining the vms from the older compute whil rebooting
16:38:35 ygk_12345 the /va/lib/nova/instances is a separate partition
16:39:22 sean-k-mooney so the conductor and other contoller service should not be able to start without bring the computes to victoria first if you are not using the workaround currently

Earlier   Later