| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-11 | |||
| 16:26:16 | ygk_12345 | sean-k-mooney: OSA 23.2.0 | |
| 16:26:43 | sean-k-mooney | what openstack release does that map too | |
| 16:26:51 | ygk_12345 | wallaby | |
| 16:27:01 | sean-k-mooney | so 23 is wallayby | |
| 16:27:07 | sean-k-mooney | and you are coming form victoria? | |
| 16:27:07 | ygk_12345 | yes | |
| 16:27:29 | ygk_12345 | yes ussuri->victoria->Wallaby | |
| 16:27:51 | sean-k-mooney | ok so you upgraded the contolers to victoria then upgreaed all the computes | |
| 16:27:58 | sean-k-mooney | then upgraded contoler to wallayby | |
| 16:28:02 | sean-k-mooney | then upgraded the computes | |
| 16:28:39 | sean-k-mooney | you cant upgrade the controlers directly form ussuri to wallayby in one go | |
| 16:28:52 | ygk_12345 | no no. upgraded first the control plane from stein->train->ussuri>vic>wallaby. Then upgraded the computes | |
| 16:29:08 | sean-k-mooney | ok so you are doing a fast forward upgrade | |
| 16:29:17 | sean-k-mooney | or skiplevel depneind on the branding | |
| 16:29:26 | sean-k-mooney | that is not supported by nova directly | |
| 16:29:37 | sean-k-mooney | so you will need to disable our validation with the workaround | |
| 16:29:47 | sean-k-mooney | then upgade allt eh compute to a supported version | |
| 16:29:52 | sean-k-mooney | then renable the check | |
| 16:30:16 | ygk_12345 | so the version 30 is being picked up by cache somewhere ? | |
| 16:30:18 | sean-k-mooney | nova and most service only suport a n to n+1 version delta | |
| 16:30:44 | sean-k-mooney | unlikely | |
| 16:31:02 | sean-k-mooney | its more likely that you are trying to start a stein compute with a wallaby contoler | |
| 16:31:23 | ygk_12345 | yes | |
| 16:31:30 | sean-k-mooney | right that is not supported | |
| 16:31:32 | ygk_12345 | but stein is not 30 version | |
| 16:31:52 | sean-k-mooney | victoria should be version 30 | |
| 16:32:18 | ygk_12345 | i deleted older versions containers from nova.services tables completely | |
| 16:32:31 | sean-k-mooney | that is not a good thing | |
| 16:32:41 | sean-k-mooney | if you have intnaces on the cloud that would break placment | |
| 16:33:04 | ygk_12345 | we have successfully upgraded in other platforms | |
| 16:35:57 | ygk_12345 | sean-k-mooney: i wil try restarting all the nova services | |
| 16:36:25 | ygk_12345 | sean-k-mooney: thanks bro for your time. appreciate it..... | |
| 16:36:26 | sean-k-mooney | the compute service version in stien was 37 by the way https://github.com/openstack/nova/blob/stable/stein/nova/objects/service.py#L34 | |
| 16:36:33 | ygk_12345 | yes | |
| 16:36:36 | ygk_12345 | not 30 | |
| 16:36:40 | sean-k-mooney | 30 was queens | |
| 16:36:44 | ygk_12345 | yes | |
| 16:37:14 | sean-k-mooney | and the miniutm version supproted by a wallaby controler is 52/victoria | |
| 16:38:15 | ygk_12345 | we are totally putting a new os image with wallaby nova-compute version , retaining the vms from the older compute whil rebooting | |
| 16:38:35 | ygk_12345 | the /va/lib/nova/instances is a separate partition | |
| 16:39:22 | sean-k-mooney | so the conductor and other contoller service should not be able to start without bring the computes to victoria first if you are not using the workaround currently | |
| 16:40:14 | sean-k-mooney | i dont know where the queens service is coming form if you have delete the serivcies in the db | |
| 16:40:26 | ygk_12345 | thats what strange | |
| 16:40:31 | sean-k-mooney | but the only place in code it could come form on teh compute node is the nova package | |
| 16:40:56 | sean-k-mooney | implying you have queens code. other wise it has to be coming form the db | |
| 16:42:03 | ygk_12345 | i think its db cache in the memory . i will try a control plane nova services restart and check | |
| 16:43:11 | sean-k-mooney | ygk_12345: i would expect this code to prevent thet conductor form starting https://github.com/openstack/nova/blob/b320f16b851fd1e5238c0b49c780f6a9c6851e48/nova/utils.py#L1053-L1100= | |
| 16:44:28 | ygk_12345 | sean-k-mooney: yes exactly. that part of the code is giving 'current_service_version' to 30 during the pdb | |
| 16:44:48 | ygk_12345 | i cant understand from where it is picking it up | |
| 16:44:59 | sean-k-mooney | have you checked both the api and cell db | |
| 16:45:17 | ygk_12345 | which tables in api and cell ? | |
| 16:46:17 | sean-k-mooney | the service table in the cell db | |
| 16:46:24 | ygk_12345 | it is empty | |
| 16:47:06 | sean-k-mooney | you dont have any entries in cell0 or cell1 | |
| 16:47:35 | ygk_12345 | yes there are.. but ony nova.services table has entries | |
| 16:48:00 | sean-k-mooney | i assume nova.service is cell0? | |
| 16:48:10 | ygk_12345 | no | |
| 16:48:15 | sean-k-mooney | ok its cell1 | |
| 16:48:25 | sean-k-mooney | the nameing depends on the deployemnt too | |
| 16:48:28 | ygk_12345 | just the plain nova db . inside it 'services' table | |
| 16:48:46 | sean-k-mooney | right so that db name "nova" depends on your deployment too | |
| 16:48:51 | ygk_12345 | i see nova.cell0 and placement dbs | |
| 16:49:09 | ygk_12345 | and also nova and nova_api | |
| 16:49:13 | sean-k-mooney | ok then its proably teh cell 1 database | |
| 16:49:32 | ygk_12345 | is there a services table inside it as well ? | |
| 16:49:47 | sean-k-mooney | ya so there should be a service table in every cell database | |
| 16:49:55 | sean-k-mooney | but you should not delete the services | |
| 16:50:02 | ygk_12345 | ok | |
| 16:50:04 | sean-k-mooney | if you do it will break your deployment | |
| 16:50:10 | ygk_12345 | yes got it | |
| 16:50:22 | sean-k-mooney | if you delete the services then teh compute agent wil create a new service entry when it starts | |
| 16:50:30 | ygk_12345 | so i will just retry with nova control plane restart | |
| 16:50:34 | sean-k-mooney | that will resullt in a different serivce uuid and break placment | |
| 16:50:49 | sean-k-mooney | since the hostname uuid pair will fail the unique constrait | |
| 16:50:56 | ygk_12345 | ok | |
| 16:51:20 | sean-k-mooney | try restartin the contolplane contianer but i expect them to fail | |
| 16:51:28 | sean-k-mooney | without the workaround option set | |
| 16:51:41 | sean-k-mooney | since your compute service versions shoudl be <56 | |
| 16:51:44 | ygk_12345 | its working for our other setups perfectly | |
| 16:51:46 | sean-k-mooney | since you have not started them | |
| 16:52:40 | ygk_12345 | anyway let me try and will bother you later | |
| 16:52:44 | sean-k-mooney | ygk_12345: what you are currently trying is not something we expect to wrok upstream so if you are able to do it else wehre you must have the workaround enabled or a downstream patch | |
| 16:52:49 | sean-k-mooney | ok | |
| 16:53:03 | ygk_12345 | sean-k-mooney: thanks | |
| 19:53:19 | mnaser | does nova like to take cpu flag related decisions or is that not a direction nova wants to take anymore | |
| 19:53:54 | mnaser | i've got 100% identical cpus in terms of make/model that fail to live migrate, and it seems like this is because some of them have `tsx-ctrl` and `taa-no` and others dont. | |
| 19:54:24 | mnaser | those two are apparently found in cpu MSR and not in cpuid so they're not visible (see here https://www.qemu.org/docs/master/system/qemu-cpu-models.html?highlight=taa#important-cpu-features-for-intel-x86-hosts ) | |
| 19:55:04 | mnaser | should we warn on it? should we just disable it? it seems like "Same cpu model" isn't really even valid anymore lol | |
| 22:32:09 | TheJulia | It hasn't really felt valid for a long time, to me... but I had a 6 week order lag and ran into something super similar ~12-13 years ago. | |
| #openstack-nova - 2022-07-12 | |||
| 00:31:20 | sean-k-mooney[m] | mnaser: that is something that libvirt not nova has always been in charge of | |
| 00:31:52 | sean-k-mooney[m] | intel can change the cpu flags via microcode and when it comes to tsx they have on several ocations | |
| 00:32:44 | sean-k-mooney[m] | nova for the most part triese to leave all cpu compatiablity checkign to the hyperviors the virt driver is managing | |
| 00:33:00 | sean-k-mooney[m] | the excption to this is the abstction we have via traits | |
| 00:35:17 | sean-k-mooney[m] | nova sepcficaly the libvirt driver uses the libvirt api to introspect the cpu to report traits | |
| 00:35:53 | sean-k-mooney[m] | we do not use cpuid or msrs to detect this as its libvirts jobs to unify the feature flags in a vendor indepentent way | |
| 00:36:18 | sean-k-mooney[m] | and its also libvirts job to determin cpu comparitblity for the most part in live migration. | |
| 00:36:54 | sean-k-mooney[m] | nova has never used the current cpu model or cpu flags to make schduling decision for migrations | |
| 00:37:34 | sean-k-mooney[m] | if you configure required traits in the flavor or image we can take those into account but the current cpu flags of a vm are not an input into the schduleing desicion. | |
| 00:38:09 | sean-k-mooney[m] | and they never have been that has alwasy been delegated to the operator to enforce using host aggreates | |
| 00:39:04 | sean-k-mooney[m] | so nova has not changed in this regard since i started working on openstack for the most part. | |