| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-26 | |||
| 16:53:33 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-26-16.00.log.html | |
| 16:53:33 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-26-16.00.txt | |
| 16:53:33 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-26-16.00.html | |
| 16:53:33 | opendevmeet | Meeting ended Tue Oct 26 16:53:33 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:53:33 | bauzas | #endmeeting | |
| 16:53:41 | gibi | bauzas: thanks! | |
| 16:53:55 | bauzas | yeah me too | |
| 16:54:44 | sean-k-mooney | fyi like bauzas i will be taking some pto around the weekend | |
| 16:55:07 | bauzas | I'll be Parisian for the weekend | |
| 16:55:22 | bauzas | although I'm already grumbling for nothing. | |
| 16:55:32 | sean-k-mooney | i need to book it still but firday and monday for sure and maybe ill exened it from thursday-turuseday inclusive | |
| 16:55:45 | bauzas | sean-k-mooney: ack, noticed. | |
| 16:56:08 | sean-k-mooney | thanks for doing the PTG summary by the way | |
| 16:56:21 | sean-k-mooney | i have not read it yet but ill try and skim over it later | |
| 16:57:29 | elodilles | sean-k-mooney: ping we if you updated the libvirt/os-vif related patches that you mentioned for the ussuri release, and I'll try to review them ASAP | |
| 16:58:31 | sean-k-mooney | elodilles: i need to redo it for the newer reelase first | |
| 16:59:17 | sean-k-mooney | elodilles: https://review.opendev.org/c/openstack/nova/+/790447 is missing one of the 3 patches that were ment to be squashed together | |
| 16:59:45 | elodilles | sean-k-mooney: ack | |
| 16:59:55 | sean-k-mooney | so i need to fix that one first and then propose it we have had some asks to backport this downstream recently too so i need to fix these up | |
| 17:00:11 | sean-k-mooney | if it end up missing the final release its proably ok but would be nice to include | |
| 17:00:20 | elodilles | hmm, so it needs to be merged in wallaby then victoria then ussuri :S | |
| 17:00:37 | sean-k-mooney | yep which is why im not sure it will make it in 2 weeks | |
| 17:00:53 | sean-k-mooney | so this is really just a nice to have not something to hold the release for | |
| 17:01:37 | elodilles | OK, no problem, let's try then. just ping me and I'll review them :) | |
| 17:01:47 | sean-k-mooney | ack thanks | |
| 17:38:54 | gmann | gibi: 1 comment on this otherwise lgtm https://review.opendev.org/c/openstack/tempest/+/809168 | |
| 17:40:21 | gmann | gibi: sorry for not mentioning about that method to use in my previous comment | |
| 18:33:32 | sean-k-mooney | gmann: i know this is a little out of your normal review spcace but feel like taking a look at https://review.opendev.org/c/openstack/nova/+/811947 | |
| 18:34:16 | sean-k-mooney | gmann: ill ping some other folks tomorow if you dont have time to look but it would be nice to get the last +2 on that | |
| 18:34:25 | sean-k-mooney | anyway going to call it a night o/ | |
| 19:24:41 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | |
| 20:22:56 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | |
| 20:31:30 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with free resources first https://review.opendev.org/c/openstack/nova/+/805649 | |
| #openstack-nova - 2021-10-27 | |||
| 08:56:40 | gibi | good day nova | |
| 09:17:47 | bauzas | hola folks | |
| 09:20:41 | gibi | o/ | |
| 09:20:54 | gibi | -another day in downstream land- | |
| 09:34:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419 | |
| 10:58:44 | gibi | stephenfin: do you recall why you needed to configure both db in a single Database fixture in https://review.opendev.org/c/openstack/nova/+/799526/5/nova/tests/fixtures/nova.py#612 ? | |
| 10:59:29 | gibi | I think our tests are using a separate fixture instance for each db (api, main) | |
| 11:00:18 | stephenfin | gibi: I'm not 100% sure but my guess is that I don't, and that was simply for expediency/laziness :) If SESSION_CONFIGURED became a mapping of DB type to "is configured" bool, we probably wouldn't need that | |
| 11:00:28 | stephenfin | *we don't | |
| 11:01:00 | gibi | I see so we had a single global but with two dbs to configure | |
| 11:01:07 | stephenfin | Yeah, I think so | |
| 11:02:48 | gibi | OK, if that is the only reason then I think I have a way to remove that global flag (based on melwitt's idea) with patch_factory from oslo_db | |
| 11:03:15 | gibi | it is no pretty confusing that we have to Database fixture intantiated one for main and one for api but the first one configures both db | |
| 11:03:38 | gibi | s/no/now | |
| 11:03:47 | gibi | /to/two | |
| 11:03:50 | gibi | /o\ | |
| 11:04:20 | stephenfin | yeah, tbc it could be more complicated than that but I really doubt it | |
| 11:08:07 | gibi | yeah, lets see if my idea works | |
| 11:12:21 | sean-k-mooney | stephenfin: since your about here an easy one for you https://review.opendev.org/c/openstack/nova/+/811947 think we can get that over the line | |
| 11:13:43 | stephenfin | sure, will look now | |
| 11:14:05 | sean-k-mooney | thanks :) | |
| 11:19:44 | sean-k-mooney | stephenfin: based on the ptg discussion woudl you mind removing your -2 on https://review.opendev.org/c/openstack/nova/+/804292 im going to rebase that and the autopep8 one shortly | |
| 13:05:29 | frickler | kashyap: I didn't make progress with reproduction without nova yet, so I created https://gitlab.com/qemu-project/qemu/-/issues/693 for now. let me know if you need additional data there | |
| 13:08:38 | kashyap | frickler: Thanks for the report. A quick one is: were you using nested setup, or was this DevStack instance on a baremetal host (<shudder>)? | |
| 13:09:07 | kashyap | A thumb-rule is to always explicitly state so if you're using a nested setup | |
| 13:09:47 | kashyap | frickler: Can you edit the report to state that "deploy DevStack in a VM?" So that an unsuspecting dev won't run it on their baremetal laptop and wreak havoc... | |
| 13:10:21 | kashyap | I'll add a quick comment there, actualy | |
| 13:17:19 | kashyap | Done | |
| 13:17:52 | kashyap | frickler: I'll check about it w/ a TCG dev | |
| 13:25:41 | frickler | kashyap: yes, nested is correct, I added that to the description. though I could also duplicate on a baremetal host if you assume that it would behave differently | |
| 13:37:59 | kashyap | frickler: No, no need for baremetal. VMs are best. Can you also post the QEMU command-line of the DevStack VM itself? (The level-1 VM) | |
| 13:39:42 | frickler | kashyap: no, I have no admin access to the cloud it is running on. I'm assuming it will essentially look the same, though, just with accel=kvm | |
| 13:43:02 | kashyap | Hmm, not sure if it'll be that same w/ accel=kvm. The details would change quite a bit. The "host" (guest hypervisor) setup can determine the guest behaviour here a lot. | |
| 13:43:19 | kashyap | That's one of the questions I'd expect from a TCG dev | |
| 13:45:38 | frickler | kashyap: o.k., I'll try running cirros locally without devstack in between, that would give the simplest setup in the end | |
| 13:47:06 | kashyap | frickler: Sure; yeah, that'd be the best. The shorter the route to the reproducer, the more likely we can get to the root cause | |
| 13:47:13 | kashyap | frickler: Thanks for all the testing! It's a pain, I Know | |
| 13:47:24 | kashyap | s/K/k/ | |
| 14:14:01 | frickler | kashyap: that went easier than I expected, updated the issue | |
| 14:19:49 | bauzas | gibi: sean-k-mooney: I think we said https://bugs.launchpad.net/nova/+bug/1947753 is valid during our PTG, right? | |
| 14:21:22 | gibi | bauzas: I don't remember discussing this | |
| 14:21:35 | bauzas | gibi: sorry you're maybe right | |
| 14:21:42 | gibi | what I see is that in the bug they evacuate instances without restarting the compute node | |
| 14:21:57 | bauzas | I originally thought this was about evacuate/evacuateback/evacuate | |
| 14:22:33 | bauzas | adding a comment | |
| 14:23:00 | gibi | so far we said that you can only evacuate if you make sure that the compute is dead | |
| 14:23:52 | gibi | in the bug case the compute was halted / stuck, the heartbeat was missing so the service was considered down, nova allowed evacuation, then the compute recovered without the nova-compute service restarted | |
| 14:24:10 | gibi | nova-compute only cleans up evacuated instance during init_host but does not do it periodically | |
| 14:24:45 | gibi | so in this case the evacuated instance was not cleaned up on the source leading to duplicated instances causing corruption | |
| 14:25:55 | gibi | option a) change nova-compute to clean up evacuated instance in a periodic | |
| 14:26:37 | gibi | option b) change the evac API to only allow evacuation if the compute is forced down (meaning the admin mades sure the host is fenced) | |
| 14:27:14 | gibi | option c) declare the current bug as user error as the nova-compute was not restarted as part of the compute node recovery | |
| 14:33:29 | bauzas | gibi: I wrote a large comment on the bvug | |
| 14:33:51 | bauzas | I think CERN is triggering evacuations before verifying the host status | |
| 14:34:32 | sean-k-mooney | bauzas: i dont think we talked baout this either | |
| 14:34:33 | bauzas | as I said, I feel healthchecks can help them getting a better decision-making about whether they need to evacuate or not | |
| 14:34:49 | sean-k-mooney | bauzas: we talk about a related issue with allcoations tha talso impact evacuate | |
| 14:35:03 | bauzas | sean-k-mooney: yeah, my confusion, I originally thought it was about the back-and-forth about evacuate we discussed for pain points | |
| 14:35:05 | sean-k-mooney | e.g. if for any reason we oversubscie the allcoations then we cant evacuate | |
| 14:36:15 | sean-k-mooney | i have not read it fully but it sound like they are not properly fencing the node an ensuring the vm is not running | |
| 14:36:26 | sean-k-mooney | before evacuating if they are having apllciation data currption | |
| 14:41:46 | bauzas | that's literrally what I wrote. | |
| 14:41:58 | bauzas | anyway, moving to a new bug. | |
| 14:42:43 | sean-k-mooney | ack as i said have not finsihed reading the bug description or comments so glad we agree :) | |
| 14:51:50 | kashyap | frickler: Ah-ha, noted. Good news: there's already some response from two QEMU devs, with a patch in a newer version :) | |
| 14:55:27 | frickler | kashyap: yeah I just responded, but I didn't see the patch reference. going from 32M to 1G really sounds a bit excessive, would be good to be able to tune that | |
| 14:55:29 | kashyap | (Well, I don't quite think it's "good news" ...) | |