| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-07 | |||
| 17:18:18 | gibi | and that path is fixed | |
| 17:18:38 | gibi | sean-k-mooney: yes, we try to log that now for this poison / live_migration_abort() codepath | |
| 17:18:44 | bauzas | sean-k-mooney: yes, I'm trying to see what's firing the greenthread | |
| 17:18:48 | sean-k-mooney | so longterm i still wonder if we should make nova use a green pool | |
| 17:19:00 | sean-k-mooney | and then in the tests we can make each test use there own greenpool | |
| 17:19:18 | sean-k-mooney | and call wait on that in the test cleanup | |
| 17:19:53 | sean-k-mooney | i think that would be relitvly simple to do | |
| 17:20:05 | sean-k-mooney | im just not sure we want to do it 2 weeks before FF | |
| 17:20:12 | bauzas | no | |
| 17:20:17 | bauzas | please :) | |
| 17:20:30 | gibi | sean-k-mooney: we would still need a reproduce for the current failure to see that if the pooling fixes it :) | |
| 17:20:43 | bauzas | gibi: I missed your top comment | |
| 17:20:49 | sean-k-mooney | gibi: yes we would :) | |
| 17:20:50 | bauzas | I'll amend .zuul.yaml | |
| 17:21:26 | sean-k-mooney | gibi: but it would allow use to piosion direct calls to spawn/spaw_n potentially and ensure we cant leek eventlets between cases | |
| 17:21:27 | gibi | so let's get a reproducer first by figuring out the leak tests (we know that there is more than one as simply intersecting testcase lists from failed test workers did not result in a single test case but an empty list) | |
| 17:21:52 | gibi | sean-k-mooney: I'm not against fixing this via pooling :) | |
| 17:21:56 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM: Add logging for leaking out the non-poisoned libvirt testcase https://review.opendev.org/c/openstack/nova/+/872975 | |
| 17:22:59 | bauzas | gibi: sean-k-mooney: I'm not against fixing our concurrency mechanism for func tests, I'm just against doing it *now* :) | |
| 17:23:36 | gibi | I will disappeare soon. I think we can continue this tomorrow. I will look at the patch and call rechecks time to time during my evening | |
| 17:25:33 | bauzas | gibi: if only I was able to reproduce it locally, I could just call tox with -- --until-failure | |
| 17:27:03 | gibi | bauzas: yeah | |
| 17:27:21 | gibi | that is the key. If we have it locally I can add as much runtime to it as I want | |
| 17:27:32 | bauzas | anyway, have a good evening | |
| 17:27:39 | bauzas | and thanks for the help | |
| 17:27:47 | bauzas | I think I'll shortly stop too | |
| 17:28:02 | gibi | bauzas: thanks for the work, I think we made good progress today. I was not able to do that without you. | |
| 17:31:53 | opendevreview | Maksim Malchuk proposed openstack/nova stable/xena: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/829804 | |
| 17:37:04 | opendevreview | Maksim Malchuk proposed openstack/nova stable/wallaby: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/861832 | |
| 20:49:18 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if base image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 21:06:11 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Add logging for leaking out the non-poisoned libvirt testcase https://review.opendev.org/c/openstack/nova/+/872975 | |
| #openstack-nova - 2023-02-08 | |||
| 01:14:50 | opendevreview | Merged openstack/nova master: Abort startup if nodename conflict is detected https://review.opendev.org/c/openstack/nova/+/872432 | |
| 01:14:58 | opendevreview | Merged openstack/nova master: Stable compute uuid functional tests https://review.opendev.org/c/openstack/nova/+/872441 | |
| 05:58:11 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if base image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 07:19:20 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 07:30:17 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 07:39:33 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 07:52:11 | opendevreview | sean mooney proposed openstack/nova master: introduce global greenpool https://review.opendev.org/c/openstack/nova/+/873061 | |
| 08:12:39 | opendevreview | Pierre Libeau proposed openstack/nova master: Add mechanism to manage snapshot during nc init https://review.opendev.org/c/openstack/nova/+/873062 | |
| 08:15:49 | plibeau4 | hello guys, I have push https://review.opendev.org/c/openstack/nova/+/873062 to have your feedback before to start written some tests. I have explain the issue also in this bug: https://bugs.launchpad.net/nova/+bug/2006559 | |
| 08:31:46 | sean-k-mooney | gibi: bauzas when ye are around take a look at https://review.opendev.org/c/openstack/nova/+/873061 and specificaly the failing tests here https://2f622fb6915ea6772a94-26db3adf591a82ec37c96a7d3180086f.ssl.cf2.rackcdn.com/873061/4/check/openstack-tox-py310/be4df4f/testr_results.html those are the unit tests that are leaking | |
| 08:35:54 | bauzas | sean-k-mooney: okay but fwiw the leaked tests for https://launchpad.net/bugs/1946339 | |
| 08:35:55 | sean-k-mooney | ha ok i also know why | |
| 08:35:59 | bauzas | are functional tests | |
| 08:36:19 | sean-k-mooney | https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/virt/libvirt/host.py#L497 | |
| 08:36:44 | sean-k-mooney | the test is calling h.initialize() | |
| 08:37:13 | sean-k-mooney | which calls init_event which does utils.spawn(self._dispatch_thread) | |
| 08:38:40 | sean-k-mooney | bauzas: here are the functionla failures https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e4/873061/4/check/nova-tox-functional-py310/3e45bc4/testr_results.html | |
| 08:38:45 | sean-k-mooney | we have many more of them | |
| 08:40:15 | bauzas | sean-k-mooney: we agreed yesterday on merging it after the RC1, right? | |
| 08:40:32 | bauzas | for Bobcat I mean | |
| 08:41:40 | sean-k-mooney | maybe i created it because i tought it would find all the tests that were broken | |
| 08:43:03 | sean-k-mooney | i was more concenred with teh test fallout then any impact it woudl nova on nova in production | |
| 08:43:41 | sean-k-mooney | the code change is pretty low risk | |
| 08:44:11 | sean-k-mooney | fixign all the broken test could be a lot of work. | |
| 08:45:03 | sean-k-mooney | bauzas: the libvirt event dispatch thread is curently while True | |
| 08:45:05 | sean-k-mooney | https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/virt/libvirt/host.py#L209-L218 | |
| 08:45:33 | sean-k-mooney | to fix that we need a way to stop that thread for the functionl tests at least | |
| 08:46:32 | sean-k-mooney | well or we need to make sure its mocked out properly | |
| 08:48:09 | sean-k-mooney | https://github.com/openstack/nova/blob/ea0526d959f7246c7d741ea24c207b52417d224a/nova/virt/libvirt/host.py#L607 is always creating 2 greenthreads that never exit if its called today | |
| 08:49:36 | sean-k-mooney | so the libvirt fixture need to be enhanced to stub that out | |
| 08:49:38 | bauzas | sean-k-mooney: the problem is that given we don't know which test is causing trouble, we won't be able to make sure your change can fix it | |
| 08:49:54 | sean-k-mooney | my patch show which tests are causing the issue | |
| 08:50:39 | bauzas | I see | |
| 08:50:45 | sean-k-mooney | bauzas: i added a fixture that detects when tests leak green threads https://review.opendev.org/c/openstack/nova/+/873061/4/nova/tests/fixtures/nova.py#1138 | |
| 08:51:00 | sean-k-mooney | and prints there name although that part currenlty has a race | |
| 08:51:25 | sean-k-mooney | fortunetly when we hit the races it still raise an exception and fails the test | |
| 08:52:36 | bauzas | ok, so then we could use your change for telling which functests leak out | |
| 08:52:54 | sean-k-mooney | yep | |
| 08:53:00 | bauzas | but for the moment, we should only merge your change at the beginning of Bobcat | |
| 08:53:14 | sean-k-mooney | i suspect that much of the cases it in the common code | |
| 08:53:21 | bauzas | just because I want to have time to make sure that if we find some issues, it shouldn't be a time problem | |
| 08:53:38 | bauzas | (I just want to be careful here) | |
| 08:54:35 | sean-k-mooney | so as i noted above. the host.py initalize currently spanw 2 greenthreads and that is call by the libvirt driver | |
| 08:55:06 | sean-k-mooney | i dont think that is stubbed by the the libvirt fixture | |
| 08:55:32 | sean-k-mooney | so today i think all libvirt functional tests are leaking at least 2 greenthreds. | |
| 08:55:55 | sean-k-mooney | the libvit event dispatcher thread and connection event thread | |
| 08:57:04 | bauzas | lemme verify | |
| 08:57:44 | sean-k-mooney | we dont currently save the returned GT and they have a while true so currenlty there is no way to stop those but it would no be hard to add one | |
| 08:58:02 | bauzas | at least we know the leaked thread is calling self.live_migration_abort() | |
| 08:58:23 | bauzas | but I think the leaked thread comes from a RPC call | |
| 08:58:38 | bauzas | not from a libvirt thread | |
| 08:58:45 | gibi | I have some concerns of the pooling in general and I left comments there | |
| 08:58:57 | sean-k-mooney | ack | |
| 08:59:04 | bauzas | honestly, my concern is more about the time here | |
| 08:59:13 | gibi | bauzas, sean-k-mooney: I think the current pooling won't catch the RPC threads as that is created in oslo.messaging | |
| 08:59:37 | bauzas | I see three cores at least looking at this bug while we only have 7 days for merging features | |
| 08:59:39 | sean-k-mooney | gibi: yes it likely wont but those are not created directly by nova | |
| 09:00:05 | sean-k-mooney | gibi: with that said i might be able to make it do that | |
| 09:00:12 | gibi | I will keep rechecking bauzas' | |
| 09:00:22 | gibi | I will keep rechecking bauzas's patch to catch a failure to see where it is coming from | |
| 09:00:29 | bauzas | so, while I think it's important to have a better way to have green threads pooling, I'm just saying that we maybe should try to just find the issue and at least do other stuff | |
| 09:00:44 | bauzas | sean-k-mooney: see,that's a problem then | |
| 09:00:55 | sean-k-mooney | bauzas: no its not | |
| 09:01:12 | bauzas | sean-k-mooney: as I said, I'm pretty sure that the threads that are leaked and create this libvirt exceptioin are RPC calls | |
| 09:01:40 | bauzas | when I say a problem, I mean I'm not sure this change would help then | |
| 09:01:46 | sean-k-mooney | right but ye did not have a repoducer so i tried to create one and found a bunch of issue | |
| 09:01:51 | bauzas | https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html | |
| 09:02:09 | sean-k-mooney | it may not fix the current issue but all the other tests if found may be flaky | |
| 09:02:17 | gibi | sean-k-mooney, bauzas: cool, we have two set of issues to solve then :) | |