| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-07 | |||
| 17:28:02 | gibi | bauzas: thanks for the work, I think we made good progress today. I was not able to do that without you. | |
| 17:31:53 | opendevreview | Maksim Malchuk proposed openstack/nova stable/xena: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/829804 | |
| 17:37:04 | opendevreview | Maksim Malchuk proposed openstack/nova stable/wallaby: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/861832 | |
| 20:49:18 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if base image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 21:06:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Add logging for leaking out the non-poisoned libvirt testcase https://review.opendev.org/c/openstack/nova/+/872975 | |
| #openstack-nova - 2023-02-08 | |||
| 01:14:50 | opendevreview | Merged openstack/nova master: Abort startup if nodename conflict is detected https://review.opendev.org/c/openstack/nova/+/872432 | |
| 01:14:58 | opendevreview | Merged openstack/nova master: Stable compute uuid functional tests https://review.opendev.org/c/openstack/nova/+/872441 | |
| 05:58:11 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if base image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 07:19:20 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 07:30:17 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 07:39:33 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 07:52:11 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 08:12:39 | opendevreview | Pierre Libeau proposed openstack/nova master: Add mechanism to manage snapshot during nc init https://review.opendev.org/c/openstack/nova/+/873062 | |
| 08:15:49 | plibeau4 | hello guys, I have push https://review.opendev.org/c/openstack/nova/+/873062 to have your feedback before to start written some tests. I have explain the issue also in this bug: https://bugs.launchpad.net/nova/+bug/2006559 | |
| 08:31:46 | sean-k-mooney | gibi: bauzas when ye are around take a look at https://review.opendev.org/c/openstack/nova/+/873061 and specificaly the failing tests here https://2f622fb6915ea6772a94-26db3adf591a82ec37c96a7d3180086f.ssl.cf2.rackcdn.com/873061/4/check/openstack-tox-py310/be4df4f/testr_results.html those are the unit tests that are leaking | |
| 08:35:54 | bauzas | sean-k-mooney: okay but fwiw the leaked tests for https://launchpad.net/bugs/1946339 | |
| 08:35:55 | sean-k-mooney | ha ok i also know why | |
| 08:35:59 | bauzas | are functional tests | |
| 08:36:19 | sean-k-mooney | https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/virt/libvirt/host.py#L497 | |
| 08:36:44 | sean-k-mooney | the test is calling h.initialize() | |
| 08:37:13 | sean-k-mooney | which calls init_event which does utils.spawn(self._dispatch_thread) | |
| 08:38:40 | sean-k-mooney | bauzas: here are the functionla failures https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e4/873061/4/check/nova-tox-functional-py310/3e45bc4/testr_results.html | |
| 08:38:45 | sean-k-mooney | we have many more of them | |
| 08:40:15 | bauzas | sean-k-mooney: we agreed yesterday on merging it after the RC1, right? | |
| 08:40:32 | bauzas | for Bobcat I mean | |
| 08:41:40 | sean-k-mooney | maybe i created it because i tought it would find all the tests that were broken | |
| 08:43:03 | sean-k-mooney | i was more concenred with teh test fallout then any impact it woudl nova on nova in production | |
| 08:43:41 | sean-k-mooney | the code change is pretty low risk | |
| 08:44:11 | sean-k-mooney | fixign all the broken test could be a lot of work. | |
| 08:45:03 | sean-k-mooney | bauzas: the libvirt event dispatch thread is curently while True | |
| 08:45:05 | sean-k-mooney | https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/virt/libvirt/host.py#L209-L218 | |
| 08:45:33 | sean-k-mooney | to fix that we need a way to stop that thread for the functionl tests at least | |
| 08:46:32 | sean-k-mooney | well or we need to make sure its mocked out properly | |
| 08:48:09 | sean-k-mooney | https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/virt/libvirt/host.py#L607 is always creating 2 greenthreads that never exit if its called today | |
| 08:49:36 | sean-k-mooney | so the libvirt fixture need to be enhanced to stub that out | |
| 08:49:38 | bauzas | sean-k-mooney: the problem is that given we don't know which test is causing trouble, we won't be able to make sure your change can fix it | |
| 08:49:54 | sean-k-mooney | my patch show which tests are causing the issue | |
| 08:50:39 | bauzas | I see | |
| 08:50:45 | sean-k-mooney | bauzas: i added a fixture that detects when tests leak green threads https://review.opendev.org/c/openstack/nova/+/873061/4/nova/tests/fixtures/nova.py#1138 | |
| 08:51:00 | sean-k-mooney | and prints there name although that part currenlty has a race | |
| 08:51:25 | sean-k-mooney | fortunetly when we hit the races it still raise an exception and fails the test | |
| 08:52:36 | bauzas | ok, so then we could use your change for telling which functests leak out | |
| 08:52:54 | sean-k-mooney | yep | |
| 08:53:00 | bauzas | but for the moment, we should only merge your change at the beginning of Bobcat | |
| 08:53:14 | sean-k-mooney | i suspect that much of the cases it in the common code | |
| 08:53:21 | bauzas | just because I want to have time to make sure that if we find some issues, it shouldn't be a time problem | |
| 08:53:38 | bauzas | (I just want to be careful here) | |
| 08:54:35 | sean-k-mooney | so as i noted above. the host.py initalize currently spanw 2 greenthreads and that is call by the libvirt driver | |
| 08:55:06 | sean-k-mooney | i dont think that is stubbed by the the libvirt fixture | |
| 08:55:32 | sean-k-mooney | so today i think all libvirt functional tests are leaking at least 2 greenthreds. | |
| 08:55:55 | sean-k-mooney | the libvit event dispatcher thread and connection event thread | |
| 08:57:04 | bauzas | lemme verify | |
| 08:57:44 | sean-k-mooney | we dont currently save the returned GT and they have a while true so currenlty there is no way to stop those but it would no be hard to add one | |
| 08:58:02 | bauzas | at least we know the leaked thread is calling self.live_migration_abort() | |
| 08:58:23 | bauzas | but I think the leaked thread comes from a RPC call | |
| 08:58:38 | bauzas | not from a libvirt thread | |
| 08:58:45 | gibi | I have some concerns of the pooling in general and I left comments there | |
| 08:58:57 | sean-k-mooney | ack | |
| 08:59:04 | bauzas | honestly, my concern is more about the time here | |
| 08:59:13 | gibi | bauzas, sean-k-mooney: I think the current pooling won't catch the RPC threads as that is created in oslo.messaging | |
| 08:59:37 | bauzas | I see three cores at least looking at this bug while we only have 7 days for merging features | |
| 08:59:39 | sean-k-mooney | gibi: yes it likely wont but those are not created directly by nova | |
| 09:00:05 | sean-k-mooney | gibi: with that said i might be able to make it do that | |
| 09:00:12 | gibi | I will keep rechecking bauzas' | |
| 09:00:22 | gibi | I will keep rechecking bauzas's patch to catch a failure to see where it is coming from | |
| 09:00:29 | bauzas | so, while I think it's important to have a better way to have green threads pooling, I'm just saying that we maybe should try to just find the issue and at least do other stuff | |
| 09:00:44 | bauzas | sean-k-mooney: see,that's a problem then | |
| 09:00:55 | sean-k-mooney | bauzas: no its not | |
| 09:01:12 | bauzas | sean-k-mooney: as I said, I'm pretty sure that the threads that are leaked and create this libvirt exceptioin are RPC calls | |
| 09:01:40 | bauzas | when I say a problem, I mean I'm not sure this change would help then | |
| 09:01:46 | sean-k-mooney | right but ye did not have a repoducer so i tried to create one and found a bunch of issue | |
| 09:01:51 | bauzas | https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html | |
| 09:02:09 | sean-k-mooney | it may not fix the current issue but all the other tests if found may be flaky | |
| 09:02:17 | gibi | sean-k-mooney, bauzas: cool, we have two set of issues to solve then :) | |
| 09:02:19 | bauzas | agreed | |
| 09:02:26 | bauzas | and agreed with gibi | |
| 09:02:29 | sean-k-mooney | and if im wirte we are also constantly builiding up greenthreas as the test run | |
| 09:02:44 | sean-k-mooney | yes | |
| 09:02:44 | bauzas | sean-k-mooneyI'm not saying "NO" to your change and thanks for having worked on it | |
| 09:02:57 | sean-k-mooney | two sets of issues | |
| 09:03:12 | bauzas | sean-k-mooney:I'm just saying that I'm afraid this won't help the functest CI failure we have atm | |
| 09:04:02 | bauzas | anyway, I said loudly yesterday that I'll stop looking at the CI failures today and I'll rather review some changes | |
| 09:04:15 | bauzas | os-vif and os-traits first, and then nova features | |
| 09:04:34 | sean-k-mooney | there arnt any we need for os-vif in this release | |
| 09:04:35 | bauzas | and I'll continue to look at https://review.opendev.org/c/openstack/nova/+/872975 and recheck until we get a -1 | |
| 09:04:42 | sean-k-mooney | not sure about os-traits | |
| 09:04:54 | bauzas | that's what I'll be doing | |
| 09:05:07 | bauzas | people are free to do anything | |
| 09:05:14 | bauzas | they prefer | |
| 09:05:35 | bauzas | I also need to look at the releases we have for os-vif, os-traits and os-rc | |
| 09:06:03 | sean-k-mooney | i approved the os-vif one yesterday | |
| 09:06:09 | sean-k-mooney | i did not look at the others | |
| 09:09:28 | sean-k-mooney | gibi: so we do stub out the events thread https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/tests/fixtures/libvirt.py#L919-L941 | |
| 09:09:35 | sean-k-mooney | but not the other one | |
| 09:10:31 | sean-k-mooney | https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/virt/libvirt/host.py#L616-L619 | |
| 09:11:06 | sean-k-mooney | i wonder if we can just more utils.spawn(self._conn_event_thread) into self._init_events() | |
| 09:12:23 | gibi | sean-k-mooney: I won't mix the events part with the connection thread. the events part uses a native thread | |
| 09:12:50 | bauzas | elodilles: 2023-02-07 14:02:46.446989 | compute1 | neutron-openvswitch-agent: no process found on https://review.opendev.org/c/openstack/nova/+/871702 | |
| 09:13:04 | bauzas | elodilles: that's the second recheck having the same problem | |
| 09:13:05 | gibi | but sure we can wrap the connection thread spawning and mock it | |