| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-21 | |||
| 09:19:01 | sean-k-mooney[m] | to use the new sha last week | |
| 09:19:33 | sean-k-mooney[m] | most of the patches i wanted to wait for have alredy landed last week | |
| 09:19:50 | sean-k-mooney[m] | we just need to repin the patches to point to the currnent tip of the stable branches | |
| 09:21:02 | sean-k-mooney[m] | elodilles give me 10 mins to grab a coffee and ill go do that when i get back and +1 them | |
| 09:21:31 | elodilles | sean-k-mooney[m]: i've updated the patches with the latest hashes :) | |
| 09:21:50 | elodilles | sean-k-mooney[m]: no need to hurry as i also will go to have lunch ;) | |
| 09:22:38 | elodilles | so enjoy your coffee, and review the patches some time later today o:) | |
| 09:22:48 | sean-k-mooney[m] | oh ok thanks i stilll ment to do this last week so before i start anything else ill review them | |
| 09:23:28 | bauzas | elodilles: ack, will look | |
| 09:23:44 | bauzas | gibi: reopened https://bugs.launchpad.net/nova/+bug/1890244 fwiw | |
| 09:23:45 | elodilles | sean-k-mooney[m] bauzas: ++, thanks in advance! \o/ | |
| 09:26:29 | bauzas | gibi: I'm curious how you didn't went able to reproduce the bug in https://bugs.launchpad.net/nova/+bug/1890244/comments/1 | |
| 09:26:31 | gibi | bauzas: so probably CONF.workarounds.disable_group_policy_check_upcall was the trick why I was not able to reproduce it | |
| 09:26:42 | bauzas | gibi: hah | |
| 09:27:06 | bauzas | gibi: but IMHO we default to False | |
| 09:27:22 | gibi | then I dont know | |
| 09:28:11 | gibi | bauzas: did you reproduced it? | |
| 09:28:14 | bauzas | I need to respin my devstack env | |
| 09:28:42 | bauzas | gibi: no, I only triaged it and we have a downstream BZ that was related | |
| 09:28:49 | bauzas | https://bugzilla.redhat.com/show_bug.cgi?id=2099279 | |
| 09:29:01 | bauzas | the code is pretty simple to look at | |
| 09:30:35 | bauzas | oh wait, you could be right, shit. | |
| 09:31:00 | bauzas | when we populate the ReqSpec, we rehydrate the fields and we can set the group value to None if the group was deleteed | |
| 09:32:04 | bauzas | https://github.com/openstack/nova/commit/94fd36f0582c5dbcf2b9886da7c7bf986d3ad5d1 is super old | |
| 09:35:35 | gibi | I still cannot reproduce | |
| 09:35:35 | gibi | https://paste.opendev.org/show/bXPvHPVJ7FfPgjfhTsSv/ | |
| 09:37:46 | bauzas | gibi: then I don't understand | |
| 09:37:54 | bauzas | gibi: I verified master | |
| 09:38:07 | gibi | the above repro trial I run on master too | |
| 09:38:16 | bauzas | gibi: when we evacuate, we regenerate the old RequestSpec record | |
| 09:38:20 | gibi | I don't have a multinode train env | |
| 09:38:30 | bauzas | and then, we set the hints | |
| 09:39:57 | bauzas | gibi: can you look at your reqspec record and see whether you have the hints dict ? | |
| 09:40:06 | gibi | sure | |
| 09:41:07 | gibi | https://paste.opendev.org/show/baqepgHm3PhuQ9VhvfWi/ | |
| 09:41:29 | gibi | there is the group hint in the dfb | |
| 09:41:30 | gibi | db | |
| 09:41:52 | bauzas | gibi: ok, and you can confirm that df92c028-728a-4321-9df6-6424ca56969e no longer exists in the instance_groups table ? | |
| 09:42:19 | bauzas | ideally we should introspect into https://github.com/openstack/nova/blob/ebe08834f311e8e22bfd9685d7e6e91dab967382/nova/compute/manager.py#L3657 | |
| 09:42:44 | gibi | yes https://paste.opendev.org/show/b9NTbUJdRy4zI9qVlFsL/ | |
| 09:43:22 | bauzas | gibi: could you see what we get as hints from https://github.com/openstack/nova/blob/ebe08834f311e8e22bfd9685d7e6e91dab967382/nova/compute/manager.py#L3656 ? | |
| 09:43:48 | bauzas | b/c I don't see where we could be smart | |
| 09:43:56 | bauzas | we're just getting the hints there | |
| 09:44:01 | gibi | I can add a LOG and repro give me 5 | |
| 09:44:05 | bauzas | and see whether this is a group | |
| 09:44:26 | bauzas | if this a group, we directly lookup the instance_groups table with this UUID | |
| 09:44:37 | bauzas | and that's the upcall | |
| 09:45:57 | sean-k-mooney | so honestly im surpiesed we allow you to delete an instance group if there are instances in it | |
| 09:46:12 | sean-k-mooney | to me that is the really issue here | |
| 09:46:41 | sean-k-mooney | is there a reason we allow that | |
| 09:46:57 | sean-k-mooney | unlike the flavor for example we are not embdeding a copy fo the instance group | |
| 09:47:16 | sean-k-mooney | so we really should not allow ti to be remvoed if its in use | |
| 09:47:23 | sean-k-mooney | do we know why we do today? | |
| 09:48:44 | bauzas | sean-k-mooney: because Instance Groups are a terrible concept ?N | |
| 09:49:03 | sean-k-mooney | not nessiarly they may be implemented badly | |
| 09:49:23 | sean-k-mooney | the concept is not a bad one but the way we implemeted them is not good | |
| 09:52:56 | gibi | bauzas: could this be the reason we bail out https://github.com/openstack/nova/blob/ebe08834f311e8e22bfd9685d7e6e91dab967382/nova/compute/manager.py#L1742 | |
| 09:53:11 | gibi | (still instrumenting the code with logs then I will rerun the repro attempt) | |
| 09:55:00 | gibi | btw I agree with sean that group is a better concept than same_host / different_host hints, as the hints were not reflexive | |
| 09:55:24 | gibi | strike reflexive | |
| 09:55:28 | gibi | symmetric | |
| 09:55:34 | gibi | they wasnt symmetirc | |
| 09:56:01 | bauzas | let's discuss this tomorrow at the bug call | |
| 09:56:04 | bauzas | shit, d/s | |
| 10:11:03 | gibi | I have /etc/nova/nova-cpu.conf:disable_group_policy_check_upcall = True in my env, restaring repro .... | |
| 10:12:25 | sean-k-mooney | isnt that our default | |
| 10:14:45 | gibi | nope | |
| 10:14:49 | gibi | the default is false | |
| 10:14:57 | gibi | I had an explicit True set for some reason | |
| 10:17:21 | sean-k-mooney | devstack set it to true by defualt i think | |
| 10:17:37 | gibi | yepp devastack by default does not even support the upcall | |
| 10:17:54 | gibi | so If I enable the upcall nova blows as no DB access | |
| 10:19:00 | sean-k-mooney | wait what | |
| 10:19:08 | sean-k-mooney | it shoudl be doing that via rpc to conductor | |
| 10:19:20 | sean-k-mooney | is it trying to use the old local conductor path | |
| 10:19:23 | sean-k-mooney | and directly access the db | |
| 10:20:37 | gibi | https://paste.opendev.org/show/bxWFPkh1OoBfWlM15mS0/ | |
| 10:20:59 | gibi | it calls up to the cell conductor | |
| 10:21:06 | gibi | and cell conductor fails to read the api db | |
| 10:21:13 | gibi | as cell conductors has no access to the api db | |
| 10:21:14 | sean-k-mooney | oh | |
| 10:21:20 | sean-k-mooney | right | |
| 10:21:27 | gibi | the upcall is between the cell and the super conductor | |
| 10:21:27 | sean-k-mooney | so this will only work in singel conductor mode | |
| 10:21:32 | sean-k-mooney | which devstack support | |
| 10:21:35 | sean-k-mooney | but its not the default | |
| 10:21:59 | gibi | I can simply promote my cell conductor to a super one by adding the api db settings to it | |
| 10:22:10 | sean-k-mooney | yep that would work | |
| 10:22:21 | gibi | but this really feels like something that upstream we dont test hence dont really support | |
| 10:22:37 | gibi | anyhow I do the propomotion and retry | |
| 10:22:38 | sean-k-mooney | well we do have supprot to test this in devstack | |
| 10:22:46 | gibi | do we have jobs testing it? | |
| 10:22:51 | sean-k-mooney | i dobt it | |
| 10:22:56 | gibi | we have code, we dont have test :D | |
| 10:23:09 | sean-k-mooney | ill check with codesearch | |
| 10:23:13 | sean-k-mooney | and see if anything enables it | |
| 10:23:30 | gibi | anyhow I go and do the promotion now and retry. I still believe that nova handles the missing group gracefully | |
| 10:24:20 | sean-k-mooney | so kolla-ansible disable the upcall out of the box i didnt get any hits form tripleo or pupet so i assuem they have it enabled | |
| 10:24:32 | sean-k-mooney | ooo actully would work since it does not supprot super conductors | |
| 10:25:10 | sean-k-mooney | well if you do multi cell its usign the cell1 conductor as the super conductor | |