| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-07 | |||
| 13:27:31 | sean-k-mooney | so we could have done that bump some time ago | |
| 13:27:46 | bauzas | sean-k-mooney: we can if you want | |
| 13:27:50 | sean-k-mooney | although we now have new upgrade requirement to test the previous LTS | |
| 13:27:54 | sean-k-mooney | bauzas: i just realsied we cant | |
| 13:28:03 | sean-k-mooney | we need to support focal for A for upgrade reasons | |
| 13:28:16 | sean-k-mooney | bauzas: so we should do this in early B | |
| 13:29:00 | sean-k-mooney | we need to not have 20.04 in our greade job to do this bump | |
| 13:29:12 | bauzas | hmmm ok | |
| 13:29:29 | sean-k-mooney | and the dedicated focal job to go away | |
| 13:29:43 | sean-k-mooney | for B we will be useing 22.04 | |
| 13:30:47 | sean-k-mooney | dvo-plv: so what that means for you is if your patch is after we have done the bump you will not need to do the version check | |
| 13:31:01 | gibi | bauzas: I'm not surpirsed, it seems both of us are missing some hidden ingredients to reproduce the same thing that happens on the gate | |
| 13:31:06 | sean-k-mooney | if its before we do the bump you will ahve to do the version check when reportin the trait | |
| 13:31:32 | bauzas | Ran: 4144 tests in 4758.5975 sec. | |
| 13:31:32 | bauzas | - Passed: 4144 | |
| 13:31:32 | bauzas | - Skipped: 0 | |
| 13:31:32 | bauzas | - Expected Fail: 0 | |
| 13:31:32 | bauzas | - Unexpected Success: 0 | |
| 13:31:34 | bauzas | - Failed: 0 | |
| 13:31:36 | bauzas | Sum of execute time for each test: 32443.8935 sec. | |
| 13:31:38 | bauzas | :) | |
| 13:31:51 | sean-k-mooney | what kind of potato is that running on | |
| 13:32:20 | sean-k-mooney | or were you just running those in a loop | |
| 13:32:46 | bauzas | sean-k-mooney: see what we discussed before you arrived | |
| 13:33:08 | bauzas | gibi: yah, maybe | |
| 13:33:17 | dvo-plv | Okay, If Libvirt version will be lower that 6.3, when I will present patch in the blueprint, I will create separate define with Libvirt version | |
| 13:33:23 | bauzas | gibi: I'm now looking at the code and trying to understand what we use | |
| 13:33:48 | opendevreview | Jorge San Emeterio proposed openstack/nova master: Moving privsep profiles to nova/__init__.py https://review.opendev.org/c/openstack/nova/+/872010 | |
| 13:33:51 | kashyap | sean-k-mooney: Hi, reading back. (Was buried elsewhere in an urgent thing) | |
| 13:34:13 | sean-k-mooney | kashyap: its fine it was just on our next libvirt/qemu version | |
| 13:34:14 | kashyap | sean-k-mooney: Yeah, bumping the min versions in 'A' is totally fine. | |
| 13:34:25 | sean-k-mooney | kashyap: actully it used to be its not anymore | |
| 13:34:40 | sean-k-mooney | kashyap: form a pure nova point of view it would be | |
| 13:34:53 | sean-k-mooney | kashyap: but we have PTI/governance requirements | |
| 13:34:59 | kashyap | Right, I'm talking from a Nova PoV | |
| 13:35:05 | sean-k-mooney | that reqire use to supprot 20.04 for A | |
| 13:35:22 | sean-k-mooney | right so because of the other requirement we cant bump it in nova until B | |
| 13:35:40 | sean-k-mooney | kashyap: https://github.com/openstack/governance/blob/master/reference/runtimes/2023.1.rst#additional-testing-for-smooth-upgrade | |
| 13:36:23 | kashyap | What is "support 20.04 for A", I don't get | |
| 13:36:39 | dvo-plv | Thank you for your time and conversation. Have a nice day | |
| 13:36:41 | kashyap | Ah, it is Ubunutu 20.04 | |
| 13:37:09 | sean-k-mooney | yes basicaly every time we cange a base OS in the testign requirement we need to have one release wehre we test the old and new version | |
| 13:37:25 | sean-k-mooney | kashyap: we chavned form 20.04 to 22.04 in this release | |
| 13:37:54 | sean-k-mooney | so the same would happen for debiany 11->12 or centos 9->10 in the future | |
| 13:38:42 | sean-k-mooney | its to ensure you can upgrade openstack without nessiarly needing to upgrade the OS it also mimic how our greneade jobs work | |
| 13:39:36 | sean-k-mooney | its related to https://github.com/openstack/governance/blob/master/resolutions/20220210-release-cadence-adjustment.rst the skip level upgrade release and the new lifecycle for integrated release projects | |
| 13:40:01 | sean-k-mooney | dvo-plv: o/ | |
| 13:40:05 | kashyap | sean-k-mooney: Yeah, the upgradeability makes sense | |
| 13:40:31 | sean-k-mooney | kashyap: bauzas any objection to doing the bump in a few weeks after RC 1 is out | |
| 13:40:41 | sean-k-mooney | better to try and do that early rather then late | |
| 13:40:55 | sean-k-mooney | or at least identify what our next versions should be declared as | |
| 13:41:02 | bauzas | when RC1 is out, then the master branch will be the Bobcat release, so ok | |
| 13:41:14 | kashyap | sean-k-mooney: Definitely agree on doing it earlier | |
| 13:42:20 | opendevreview | Merged openstack/nova master: Move comment about _destroy_evacuated_instances() https://review.opendev.org/c/openstack/nova/+/872348 | |
| 13:42:28 | opendevreview | Merged openstack/nova stable/wallaby: [stable-only][cve] Check VMDK create-type against an allowed list https://review.opendev.org/c/openstack/nova/+/871557 | |
| 13:42:35 | bauzas | \o/ | |
| 13:45:10 | bauzas | gibi: interestingly, if I restrict the logsearch call to ImportError: This test imports the 'libvirt' module, which it should not in the test environment. Please add appropriate mocking to this test." which is the latest exception I only get 19/143 failures that match (from the last 20 days) | |
| 13:45:26 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if base image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 13:45:41 | bauzas | by comparing https://7ffaea22ff93fca2f0ea-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf5.rackcdn.com/869900/7/gate/nova-tox-functional-py38/3b10d8a/testr_results.html to https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d00/868237/9/check/nova-tox-functional-py38/d00d1ff/testr_results.html that's why I think we have this | |
| 13:46:44 | gibi | I dont see the difference both has the import error line | |
| 13:57:31 | bauzas | gibi: I mean, this is just a canary line for not getting the false positives | |
| 13:58:03 | gibi | do you have a false positive where this line is missing? | |
| 14:00:35 | opendevreview | David Hill proposed openstack/nova master: Increase user_data from 64k to 128k https://review.opendev.org/c/openstack/nova/+/872931 | |
| 14:04:31 | bauzas | gibi: one of the false positives is https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aea/850501/15/check/nova-tox-functional-py38/aea02af/testr_results.html | |
| 14:05:22 | bauzas | gibi: you can find the DB issue in job_output.txt but the tests aren't failing | |
| 14:05:25 | bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aea/850501/15/check/nova-tox-functional-py38/aea02af/job-output.txt | |
| 14:06:05 | bauzas | and you won't see the canary line | |
| 14:06:08 | gibi | I see. So the difference between the false positive and a real positive is that the real one hits the libvirt import check and fails the actual check while the false one did not | |
| 14:06:17 | bauzas | yup | |
| 14:06:21 | gibi | s/actual check/actual test/ | |
| 14:06:23 | bauzas | so now I'm trying to find the pattern | |
| 14:06:32 | gibi | I see | |
| 14:06:34 | bauzas | I stestr loaded all the subunites | |
| 14:06:35 | gibi | good progress | |
| 14:06:48 | bauzas | now, I'm grepping this large txtfile I generated | |
| 14:13:57 | bauzas | gibi: see, the fact that we get the same exceptions with or without failing makes me think that the canary isn't maybe just a canary but rather the root cause of the failure | |
| 14:14:26 | bauzas | or rather some condition to a failure | |
| 14:15:00 | bauzas | if we can understand why in some cases we say meh and why not, then we could fix the problem | |
| 14:21:34 | bauzas | gibi: https://4dca9d38a541907e85e1-0253beca39d73a6e7192d5b32ed5edc2.ssl.cf2.rackcdn.com/860282/2/check/nova-tox-functional-py310/466e0d7/testr_results.html is a good candidate to use as it got | |
| 14:21:48 | bauzas | *both* false positive and true positives | |
| 14:22:00 | bauzas | test_resize_revert_across_azs is a true positive | |
| 14:22:13 | bauzas | while other api failures are false ones | |
| 14:27:29 | gibi | are the hits in the same worker? that would be the best to find a worker with multipl hits as that would mean that worker had more than one leak | |
| 14:28:45 | bauzas | 2023-01-24 18:37:50.710671 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10619, in _live_migration 2023-01-24 18:37:50.710677 | ubuntu-jammy | self.live_migration_abort(instance) | |
| 14:28:56 | bauzas | looks to me all our pain comes from this ^ | |
| 14:31:18 | bauzas | and then I'm confused | |
| 14:31:48 | bauzas | why so the f... are we calling live_migration_abort() is some functional test that just creates an instance ? | |
| 14:31:52 | bauzas | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d00/868237/9/check/nova-tox-functional-py38/d00d1ff/testr_results.html | |
| 14:32:02 | gibi | bauzas: the leaked thread calls it :) | |
| 14:32:18 | bauzas | hah | |
| 14:32:33 | gibi | bauzas: I think the global state the leak acts on to infulence the later tests is sys.meta_path from ImportModulePoisonFixture | |
| 14:33:23 | gibi | so my current theory: we leak a thread/eventlet that eventually calls live_migration_abort while a later test runs. As the later test sets the global posion on libvirt import the later test gets the failure | |
| 14:33:56 | bauzas | yeah, the global libvirt object is set to None by something else | |
| 14:33:56 | gibi | if there is no ImportModulePoisonFixture set in the later test then we only see the stack trace | |
| 14:34:14 | bauzas | so the threads get this NoneError and fails whivh tramples the whole terst | |
| 14:34:39 | bauzas | something we merged tampered the libvirt import | |
| 14:34:49 | bauzas | and we need to find it | |
| 14:35:19 | gibi | I think we intentionally posion libvirt import | |
| 14:35:47 | bauzas | by posion, you mean poison, right? | |
| 14:36:19 | bauzas | but yeah got it | |