| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-07-11 | |||
| 16:35:57 | ygk_12345 | sean-k-mooney: i wil try restarting all the nova services | |
| 16:36:25 | ygk_12345 | sean-k-mooney: thanks bro for your time. appreciate it..... | |
| 16:36:26 | sean-k-mooney | the compute service version in stien was 37 by the way https://github.com/openstack/nova/blob/stable/stein/nova/objects/service.py#L34 | |
| 16:36:33 | ygk_12345 | yes | |
| 16:36:36 | ygk_12345 | not 30 | |
| 16:36:40 | sean-k-mooney | 30 was queens | |
| 16:36:44 | ygk_12345 | yes | |
| 16:37:14 | sean-k-mooney | and the miniutm version supproted by a wallaby controler is 52/victoria | |
| 16:38:15 | ygk_12345 | we are totally putting a new os image with wallaby nova-compute version , retaining the vms from the older compute whil rebooting | |
| 16:38:35 | ygk_12345 | the /va/lib/nova/instances is a separate partition | |
| 16:39:22 | sean-k-mooney | so the conductor and other contoller service should not be able to start without bring the computes to victoria first if you are not using the workaround currently | |
| 16:40:14 | sean-k-mooney | i dont know where the queens service is coming form if you have delete the serivcies in the db | |
| 16:40:26 | ygk_12345 | thats what strange | |
| 16:40:31 | sean-k-mooney | but the only place in code it could come form on teh compute node is the nova package | |
| 16:40:56 | sean-k-mooney | implying you have queens code. other wise it has to be coming form the db | |
| 16:42:03 | ygk_12345 | i think its db cache in the memory . i will try a control plane nova services restart and check | |
| 16:43:11 | sean-k-mooney | ygk_12345: i would expect this code to prevent thet conductor form starting https://github.com/openstack/nova/blob/b320f16b851fd1e5238c0b49c780f6a9c6851e48/nova/utils.py#L1053-L1100= | |
| 16:44:28 | ygk_12345 | sean-k-mooney: yes exactly. that part of the code is giving 'current_service_version' to 30 during the pdb | |
| 16:44:48 | ygk_12345 | i cant understand from where it is picking it up | |
| 16:44:59 | sean-k-mooney | have you checked both the api and cell db | |
| 16:45:17 | ygk_12345 | which tables in api and cell ? | |
| 16:46:17 | sean-k-mooney | the service table in the cell db | |
| 16:46:24 | ygk_12345 | it is empty | |
| 16:47:06 | sean-k-mooney | you dont have any entries in cell0 or cell1 | |
| 16:47:35 | ygk_12345 | yes there are.. but ony nova.services table has entries | |
| 16:48:00 | sean-k-mooney | i assume nova.service is cell0? | |
| 16:48:10 | ygk_12345 | no | |
| 16:48:15 | sean-k-mooney | ok its cell1 | |
| 16:48:25 | sean-k-mooney | the nameing depends on the deployemnt too | |
| 16:48:28 | ygk_12345 | just the plain nova db . inside it 'services' table | |
| 16:48:46 | sean-k-mooney | right so that db name "nova" depends on your deployment too | |
| 16:48:51 | ygk_12345 | i see nova.cell0 and placement dbs | |
| 16:49:09 | ygk_12345 | and also nova and nova_api | |
| 16:49:13 | sean-k-mooney | ok then its proably teh cell 1 database | |
| 16:49:32 | ygk_12345 | is there a services table inside it as well ? | |
| 16:49:47 | sean-k-mooney | ya so there should be a service table in every cell database | |
| 16:49:55 | sean-k-mooney | but you should not delete the services | |
| 16:50:02 | ygk_12345 | ok | |
| 16:50:04 | sean-k-mooney | if you do it will break your deployment | |
| 16:50:10 | ygk_12345 | yes got it | |
| 16:50:22 | sean-k-mooney | if you delete the services then teh compute agent wil create a new service entry when it starts | |
| 16:50:30 | ygk_12345 | so i will just retry with nova control plane restart | |
| 16:50:34 | sean-k-mooney | that will resullt in a different serivce uuid and break placment | |
| 16:50:49 | sean-k-mooney | since the hostname uuid pair will fail the unique constrait | |
| 16:50:56 | ygk_12345 | ok | |
| 16:51:20 | sean-k-mooney | try restartin the contolplane contianer but i expect them to fail | |
| 16:51:28 | sean-k-mooney | without the workaround option set | |
| 16:51:41 | sean-k-mooney | since your compute service versions shoudl be <56 | |
| 16:51:44 | ygk_12345 | its working for our other setups perfectly | |
| 16:51:46 | sean-k-mooney | since you have not started them | |
| 16:52:40 | ygk_12345 | anyway let me try and will bother you later | |
| 16:52:44 | sean-k-mooney | ygk_12345: what you are currently trying is not something we expect to wrok upstream so if you are able to do it else wehre you must have the workaround enabled or a downstream patch | |
| 16:52:49 | sean-k-mooney | ok | |
| 16:53:03 | ygk_12345 | sean-k-mooney: thanks | |
| 19:53:19 | mnaser | does nova like to take cpu flag related decisions or is that not a direction nova wants to take anymore | |
| 19:53:54 | mnaser | i've got 100% identical cpus in terms of make/model that fail to live migrate, and it seems like this is because some of them have `tsx-ctrl` and `taa-no` and others dont. | |
| 19:54:24 | mnaser | those two are apparently found in cpu MSR and not in cpuid so they're not visible (see here https://www.qemu.org/docs/master/system/qemu-cpu-models.html?highlight=taa#important-cpu-features-for-intel-x86-hosts ) | |
| 19:55:04 | mnaser | should we warn on it? should we just disable it? it seems like "Same cpu model" isn't really even valid anymore lol | |
| 22:32:09 | TheJulia | It hasn't really felt valid for a long time, to me... but I had a 6 week order lag and ran into something super similar ~12-13 years ago. | |
| #openstack-nova - 2022-07-12 | |||
| 00:31:20 | sean-k-mooney[m] | mnaser: that is something that libvirt not nova has always been in charge of | |
| 00:31:52 | sean-k-mooney[m] | intel can change the cpu flags via microcode and when it comes to tsx they have on several ocations | |
| 00:32:44 | sean-k-mooney[m] | nova for the most part triese to leave all cpu compatiablity checkign to the hyperviors the virt driver is managing | |
| 00:33:00 | sean-k-mooney[m] | the excption to this is the abstction we have via traits | |
| 00:35:17 | sean-k-mooney[m] | nova sepcficaly the libvirt driver uses the libvirt api to introspect the cpu to report traits | |
| 00:35:53 | sean-k-mooney[m] | we do not use cpuid or msrs to detect this as its libvirts jobs to unify the feature flags in a vendor indepentent way | |
| 00:36:18 | sean-k-mooney[m] | and its also libvirts job to determin cpu comparitblity for the most part in live migration. | |
| 00:36:54 | sean-k-mooney[m] | nova has never used the current cpu model or cpu flags to make schduling decision for migrations | |
| 00:37:34 | sean-k-mooney[m] | if you configure required traits in the flavor or image we can take those into account but the current cpu flags of a vm are not an input into the schduleing desicion. | |
| 00:38:09 | sean-k-mooney[m] | and they never have been that has alwasy been delegated to the operator to enforce using host aggreates | |
| 00:39:04 | sean-k-mooney[m] | so nova has not changed in this regard since i started working on openstack for the most part. | |
| 07:33:04 | opendevreview | Sylvain Bauza proposed openstack/nova master: api: Drop generating a keypair and add special chars to naming https://review.opendev.org/c/openstack/nova/+/849133 | |
| 07:33:40 | bauzas | gibi: sean-k-mooney: I'm done with the keypair generation removal | |
| 07:34:13 | bauzas | given Uggla's patches seem good for unshelve, I'll rebase my branch up on Uggla's unshelve API change once he rebases | |
| 07:35:00 | bauzas | gibi: sean-k-mooney: I'll actually be off from tonight to next week (I'll also take Friday) | |
| 07:38:40 | gibi | bauzas: thanks for moving you stuff top of Uggla's that is nice selflessness | |
| 07:39:12 | gibi | bauzas: have a nice PTO, do you have someting on your PTL table we should keep in mind while you are away? | |
| 07:39:37 | bauzas | gibi: nothing in my mind, I'll abandon the yoga open specs next week then | |
| 07:40:04 | bauzas | we're on yoga-2 on Thursday | |
| 07:40:13 | bauzas | July-14 | |
| 07:40:23 | bauzas | that will mean we won't accept new specs | |
| 07:40:33 | bauzas | but actually, we don't have a lot of them for zed | |
| 07:42:10 | gibi | OK, so I guess you will do the official freeze mail when you are back on Monday. That is KO | |
| 07:42:13 | gibi | OK | |
| 07:42:29 | bauzas | yup | |
| 07:42:45 | bauzas | unless you wanna use the axe | |
| 07:43:16 | gibi | nope | |
| 07:43:22 | gibi | the axe is yours :) | |
| 07:43:43 | gibi | and as you said we don't have much open | |
| 07:43:51 | gibi | so there is no need for the axe | |
| 07:45:47 | bauzas | gibi: yeah, in my email, I'll clarify the situation https://review.opendev.org/q/project:openstack/nova-specs+status:open+file:%255Especs/zed/.* | |
| 07:46:08 | bauzas | about ironic's discussion, this won't need to be hold by the deadline | |
| 07:46:37 | bauzas | so only artom's spec is impacted... unless he's able to revive it before the deadline | |
| 07:47:25 | bauzas | I don't know if sean-k-mooney had wheels for https://review.opendev.org/c/openstack/nova-specs/+/821419 | |
| 07:49:51 | bauzas | gibi: about the API changes, those are stacking | |
| 07:49:57 | bauzas | with the same API microversion | |
| 07:50:12 | bauzas | I'm not really worried yet | |
| 07:50:40 | bauzas | but maybe next week, I'd propose some etherpad for trying to organize series between them | |
| 07:50:59 | bauzas | like, Uggla would take the 2.91 as he's close to be merged | |
| 07:51:17 | bauzas | mine would take 2.92 as this is a quite self-contained change | |
| 07:51:35 | bauzas | and we would debate on other patches for 2.93 and others | |