| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-07 | |||
| 13:31:32 | bauzas | Ran: 4144 tests in 4758.5975 sec. | |
| 13:31:32 | bauzas | - Passed: 4144 | |
| 13:31:32 | bauzas | - Skipped: 0 | |
| 13:31:32 | bauzas | - Expected Fail: 0 | |
| 13:31:32 | bauzas | - Unexpected Success: 0 | |
| 13:31:34 | bauzas | - Failed: 0 | |
| 13:31:36 | bauzas | Sum of execute time for each test: 32443.8935 sec. | |
| 13:31:38 | bauzas | :) | |
| 13:31:51 | sean-k-mooney | what kind of potato is that running on | |
| 13:32:20 | sean-k-mooney | or were you just running those in a loop | |
| 13:32:46 | bauzas | sean-k-mooney: see what we discussed before you arrived | |
| 13:33:08 | bauzas | gibi: yah, maybe | |
| 13:33:17 | dvo-plv | Okay, If Libvirt version will be lower that 6.3, when I will present patch in the blueprint, I will create separate define with Libvirt version | |
| 13:33:23 | bauzas | gibi: I'm now looking at the code and trying to understand what we use | |
| 13:33:48 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Moving privsep profiles to nova/__init__.py https://review.opendev.org/c/openstack/nova/+/872010 | |
| 13:33:51 | kashyap | sean-k-mooney: Hi, reading back. (Was buried elsewhere in an urgent thing) | |
| 13:34:13 | sean-k-mooney | kashyap: its fine it was just on our next libvirt/qemu version | |
| 13:34:14 | kashyap | sean-k-mooney: Yeah, bumping the min versions in 'A' is totally fine. | |
| 13:34:25 | sean-k-mooney | kashyap: actully it used to be its not anymore | |
| 13:34:40 | sean-k-mooney | kashyap: form a pure nova point of view it would be | |
| 13:34:53 | sean-k-mooney | kashyap: but we have PTI/governance requirements | |
| 13:34:59 | kashyap | Right, I'm talking from a Nova PoV | |
| 13:35:05 | sean-k-mooney | that reqire use to supprot 20.04 for A | |
| 13:35:22 | sean-k-mooney | right so because of the other requirement we cant bump it in nova until B | |
| 13:35:40 | sean-k-mooney | kashyap: https://github.com/openstack/governance/blob/master/reference/runtimes/2023.1.rst#additional-testing-for-smooth-upgrade | |
| 13:36:23 | kashyap | What is "support 20.04 for A", I don't get | |
| 13:36:39 | dvo-plv | Thank you for your time and conversation. Have a nice day | |
| 13:36:41 | kashyap | Ah, it is Ubunutu 20.04 | |
| 13:37:09 | sean-k-mooney | yes basicaly every time we cange a base OS in the testign requirement we need to have one release wehre we test the old and new version | |
| 13:37:25 | sean-k-mooney | kashyap: we chavned form 20.04 to 22.04 in this release | |
| 13:37:54 | sean-k-mooney | so the same would happen for debiany 11->12 or centos 9->10 in the future | |
| 13:38:42 | sean-k-mooney | its to ensure you can upgrade openstack without nessiarly needing to upgrade the OS it also mimic how our greneade jobs work | |
| 13:39:36 | sean-k-mooney | its related to https://github.com/openstack/governance/blob/master/resolutions/20220210-release-cadence-adjustment.rst the skip level upgrade release and the new lifecycle for integrated release projects | |
| 13:40:01 | sean-k-mooney | dvo-plv: o/ | |
| 13:40:05 | kashyap | sean-k-mooney: Yeah, the upgradeability makes sense | |
| 13:40:31 | sean-k-mooney | kashyap: bauzas any objection to doing the bump in a few weeks after RC 1 is out | |
| 13:40:41 | sean-k-mooney | better to try and do that early rather then late | |
| 13:40:55 | sean-k-mooney | or at least identify what our next versions should be declared as | |
| 13:41:02 | bauzas | when RC1 is out, then the master branch will be the Bobcat release, so ok | |
| 13:41:14 | kashyap | sean-k-mooney: Definitely agree on doing it earlier | |
| 13:42:20 | opendevreview | Merged openstack/nova master: Move comment about _destroy_evacuated_instances() https://review.opendev.org/c/openstack/nova/+/872348 | |
| 13:42:28 | opendevreview | Merged openstack/nova stable/wallaby: [stable-only][cve] Check VMDK create-type against an allowed list https://review.opendev.org/c/openstack/nova/+/871557 | |
| 13:42:35 | bauzas | \o/ | |
| 13:45:10 | bauzas | gibi: interestingly, if I restrict the logsearch call to ImportError: This test imports the 'libvirt' module, which it should not in the test environment. Please add appropriate mocking to this test." which is the latest exception I only get 19/143 failures that match (from the last 20 days) | |
| 13:45:26 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if base image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 13:45:41 | bauzas | by comparing https://7ffaea22ff93fca2f0ea-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf5.rackcdn.com/869900/7/gate/nova-tox-functional-py38/3b10d8a/testr_results.html to https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d00/868237/9/check/nova-tox-functional-py38/d00d1ff/testr_results.html that's why I think we have this | |
| 13:46:44 | gibi | I dont see the difference both has the import error line | |
| 13:57:31 | bauzas | gibi: I mean, this is just a canary line for not getting the false positives | |
| 13:58:03 | gibi | do you have a false positive where this line is missing? | |
| 14:00:35 | opendevreview | David Hill proposed openstack/nova master: Increase user_data from 64k to 128k https://review.opendev.org/c/openstack/nova/+/872931 | |
| 14:04:31 | bauzas | gibi: one of the false positives is https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aea/850501/15/check/nova-tox-functional-py38/aea02af/testr_results.html | |
| 14:05:22 | bauzas | gibi: you can find the DB issue in job_output.txt but the tests aren't failing | |
| 14:05:25 | bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aea/850501/15/check/nova-tox-functional-py38/aea02af/job-output.txt | |
| 14:06:05 | bauzas | and you won't see the canary line | |
| 14:06:08 | gibi | I see. So the difference between the false positive and a real positive is that the real one hits the libvirt import check and fails the actual check while the false one did not | |
| 14:06:17 | bauzas | yup | |
| 14:06:21 | gibi | s/actual check/actual test/ | |
| 14:06:23 | bauzas | so now I'm trying to find the pattern | |
| 14:06:32 | gibi | I see | |
| 14:06:34 | bauzas | I stestr loaded all the subunites | |
| 14:06:35 | gibi | good progress | |
| 14:06:48 | bauzas | now, I'm grepping this large txtfile I generated | |
| 14:13:57 | bauzas | gibi: see, the fact that we get the same exceptions with or without failing makes me think that the canary isn't maybe just a canary but rather the root cause of the failure | |
| 14:14:26 | bauzas | or rather some condition to a failure | |
| 14:15:00 | bauzas | if we can understand why in some cases we say meh and why not, then we could fix the problem | |
| 14:21:34 | bauzas | gibi: https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html is a good candidate to use as it got | |
| 14:21:48 | bauzas | *both* false positive and true positives | |
| 14:22:00 | bauzas | test_resize_revert_across_azs is a true positive | |
| 14:22:13 | bauzas | while other api failures are false ones | |
| 14:27:29 | gibi | are the hits in the same worker? that would be the best to find a worker with multipl hits as that would mean that worker had more than one leak | |
| 14:28:45 | bauzas | 2023-01-24 18:37:50.710671 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10619, in _live_migration 2023-01-24 18:37:50.710677 | ubuntu-jammy | self.live_migration_abort(instance) | |
| 14:28:56 | bauzas | looks to me all our pain comes from this ^ | |
| 14:31:18 | bauzas | and then I'm confused | |
| 14:31:48 | bauzas | why so the f... are we calling live_migration_abort() is some functional test that just creates an instance ? | |
| 14:31:52 | bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d00/868237/9/check/nova-tox-functional-py38/d00d1ff/testr_results.html | |
| 14:32:02 | gibi | bauzas: the leaked thread calls it :) | |
| 14:32:18 | bauzas | hah | |
| 14:32:33 | gibi | bauzas: I think the global state the leak acts on to infulence the later tests is sys.meta_path from ImportModulePoisonFixture | |
| 14:33:23 | gibi | so my current theory: we leak a thread/eventlet that eventually calls live_migration_abort while a later test runs. As the later test sets the global posion on libvirt import the later test gets the failure | |
| 14:33:56 | bauzas | yeah, the global libvirt object is set to None by something else | |
| 14:33:56 | gibi | if there is no ImportModulePoisonFixture set in the later test then we only see the stack trace | |
| 14:34:14 | bauzas | so the threads get this NoneError and fails whivh tramples the whole terst | |
| 14:34:39 | bauzas | something we merged tampered the libvirt import | |
| 14:34:49 | bauzas | and we need to find it | |
| 14:35:19 | gibi | I think we intentionally posion libvirt import | |
| 14:35:47 | bauzas | by posion, you mean poison, right? | |
| 14:36:19 | bauzas | but yeah got it | |
| 14:36:26 | gibi | yeah, sorry | |
| 14:36:34 | bauzas | afaicr, our libvirt functional tests do poison indeed the import | |
| 14:37:46 | bauzas | the problem is that it seems that some test that calls live_mig_abort() doesn't use the libvirt poisoned instance, hence the issue | |
| 14:38:01 | bauzas | but which one and how to find it ? | |
| 14:39:15 | bauzas | 2023-02-06 16:15:47,325 ERROR [root] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 207, in _send\n reply, failure = reply_q.get(timeout=timeout)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet | |
| 14:39:15 | bauzas | /queue.py", line 322, in get\n return waiter.wait()\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait\n return get_hub().switch()\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n', '_queue.Empty\n', ' | |
| 14:39:15 | bauzas | \nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 203, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 9300, in _post_live_migration\n self._update_scheduler_instance_in | |
| 14:39:15 | bauzas | fo(ctxt, instance)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2219, in _update_scheduler_instance_info\n self.query_client.update_instance_info(context, self.host,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 69, in update_instance_info\n self.scheduler_rpcapi.update_instance_info(context, host_name,\n', ' File "/home/zuul/src/opende | |
| 14:39:15 | bauzas | v.org/openstack/nova/nova/scheduler/rpcapi.py", line 174, in update_instance_info\n cctxt.cast(ctxt, \'update_instance_info\', host_name=host_name,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/fixtures/_fixtures/monkeypatch.py", line 86, in avoid_get\n return captured_method(*args, **kwargs)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38 | |
| 14:39:17 | bauzas | /lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 190, in call\n result = self.transport._send(\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_mess | |
| 14:39:19 | bauzas | aging/_drivers/impl_fake.py", line 222, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_fake.py", line 213, in _send\n raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: No reply on topic scheduler\n'] 2023-02-06 16:15:47,361 WAR | |
| 14:39:21 | bauzas | NING [nova.virt.libvirt.driver] Error monitoring migration: (sqlite3.OperationalError) no such table: compute_nodes | |
| 14:39:53 | gibi | maybe the test has the poison but it is removed after the test finished, but the thread that will do the abort call is leaked to another test that might or might not have (or need) the poision | |