| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-04 | |||
| 09:51:38 | sean-k-mooney | and then 2 for ironci start up time | |
| 09:52:11 | sean-k-mooney | if you have time to look at any of those that woudl be great if not i will try and follow up with other to land them in time | |
| 09:52:12 | bauzas | okay, I'm pretty busy before my two-week PTO which will be on next Tuesday but I'll definitely try | |
| 09:52:29 | bauzas | thanks for the effortd | |
| 09:54:41 | sean-k-mooney | as an fyi we are also seting up devstack based nightly ci to test as much of this as possible with real hardware internally (we cant currentl report the logs externally due to internal info in the job) but on the cybrog side we have been developing a kernel module to emulsate gpus, nvme devices and eventually nics | |
| 09:55:07 | sean-k-mooney | so we will have a mix of 1st party ci and internal coverage in the next few months for this | |
| 09:56:38 | sean-k-mooney | joan will be on pto startign tomorrow for the next 3 weeks so we likey will continue the 1st party testign efforts into the start of next cycle | |
| 09:56:40 | sean-k-mooney | https://review.opendev.org/q/topic:%22cyborg-vgpu-support%22 | |
| 09:56:51 | sean-k-mooney | that is the timesliced cybrog vgpu nova patch series | |
| 09:57:09 | sean-k-mooney | again just general context for everyone | |
| 10:22:16 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | |
| 11:05:44 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: check tempest for instability https://review.opendev.org/c/openstack/nova/+/999378 | |
| 11:46:39 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 11:46:39 | opendevreview | Stephen Finucane proposed openstack/nova master: Replace use of datetime.timezone.UTC https://review.opendev.org/c/openstack/nova/+/989515 | |
| 11:46:40 | opendevreview | Stephen Finucane proposed openstack/nova master: trivial: Fix outstanding flake8 issues https://review.opendev.org/c/openstack/nova/+/999714 | |
| 13:07:54 | tkajinam | clif, hm the failure is not reproduced in my local env (and in CI, either) | |
| 13:08:25 | tkajinam | clif, do you mind reporting a bug with a few examples of the failing tests ? | |
| 13:56:00 | sean-k-mooney | tkajinam: are you runing on an mad cpu | |
| 13:56:04 | sean-k-mooney | *amd | |
| 14:00:37 | sean-k-mooney | im running the test now locally to confirm but i also didnt see the failures on my intel cpu | |
| 14:01:37 | sean-k-mooney | and if i cat the file | |
| 14:01:39 | sean-k-mooney | [15:01:17]> cat /sys/module/kvm_amd/parameters/sev | |
| 14:01:41 | sean-k-mooney | [bat error]: '/sys/module/kvm_amd/parameters/sev': No such file or directory (os error 2) | |
| 14:02:20 | sean-k-mooney | i get an error so it would fail on this laptop if the mock didnt work | |
| 14:02:22 | sean-k-mooney | Ran: 17007 tests in 214.5597 sec. | |
| 14:02:24 | sean-k-mooney | - Passed: 16940 | |
| 14:02:26 | sean-k-mooney | - Skipped: 65 | |
| 14:02:28 | sean-k-mooney | - Expected Fail: 2 | |
| 14:02:30 | sean-k-mooney | - Unexpected Success: 0 | |
| 14:02:32 | sean-k-mooney | - Failed: 0 | |
| 14:02:34 | sean-k-mooney | Sum of execute time for each test: 3262.8758 sec. | |
| 14:02:43 | sean-k-mooney | ill try recreating the env just to make sure but on py 3.14 it seams to be working | |
| 14:03:29 | sean-k-mooney | oh | |
| 14:03:45 | sean-k-mooney | maybe this only fails on amd cpus | |
| 14:04:06 | sean-k-mooney | because we may be skiping earlier on intel before we try and read the path | |
| 14:10:03 | dansmith | succeeds on macos arm64, FWIW, so it'd have to be AMD_specific | |
| 14:10:04 | frickler | I have an VM on amd where I can reproduce this, I can give you access if that helps | |
| 14:16:10 | frickler | does this imply that all of our zuul nodes are on intel? | |
| 14:17:10 | frickler | oh, or does it need kvm enabled? maybe we can try running a tox job on a big kvm node | |
| 14:19:23 | sean-k-mooney | frickler: thre are a couple of diffent checks for sev supprot so i would not be surprised if we side step the fialign check on the amd vms as well | |
| 14:19:47 | sean-k-mooney | from pi | |
| 14:19:49 | sean-k-mooney | I found no unpatched access to /sys/module/kvm_amd/parameters/sev in the current checkout. | |
| 14:19:51 | sean-k-mooney | - nova/tests/fixtures/libvirt.py:2727 forces the SEV parameter to appear absent in LibvirtFixture. | |
| 14:19:53 | sean-k-mooney | - SEV-specific tests explicitly patch os.path.exists and open. | |
| 14:19:55 | sean-k-mooney | - SysFsPoisonFixture raises if a unit test actually opens an unmocked /sys path. | |
| 14:19:57 | sean-k-mooney | Validation: | |
| 14:19:59 | sean-k-mooney | - test_driver: 1,186 tests passed | |
| 14:20:01 | sean-k-mooney | - test_host: 127 tests passed | |
| 14:20:03 | sean-k-mooney | One unrelated potential typo exists: the fixture patches sev-es, while the production path is sev_es. | |
| 14:20:23 | sean-k-mooney | tkajinam: ^ | |
| 14:23:01 | sean-k-mooney | im checkign what that actully is now but also in an irc meeting | |
| 14:24:36 | sean-k-mooney | clif: just to confirm were the fialing teest sev specific or unrelated | |
| 14:24:59 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 14:25:43 | frickler | sean-k-mooney: one test was nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_unplug_vifs_with_destroy_vifs_true , doesn't look related to me | |
| 14:25:45 | sean-k-mooney | we may have mocked it in all the sev paths an missed it in the genric tests paths | |
| 14:25:54 | sean-k-mooney | ack | |
| 14:29:40 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 14:31:39 | clif | sean-k-mooney: every test failure I still have in traceback is failing due to the SEV exception | |
| 14:31:54 | clif | I can re-run and save all output | |
| 14:36:51 | clif | https://gist.github.com/ClifHouck/b7186501897e8276afcc9a7827e21789 here's a full traceback of one error | |
| 14:38:54 | clif | yep, I get 217 failures and they're all due to this SEV exception | |
| 14:41:42 | frickler | ok, my test triggers the same failures. maybe add that permanently at least as experimental or periodic job? | |
| 14:42:31 | frickler | https://zuul.opendev.org/t/openstack/build/1af158b6480d4783b0211c4f4d890fed | |
| 14:45:23 | tkajinam | ohhk let me check that | |
| 14:46:54 | sean-k-mooney | frickler: ok so its related to kvm? | |
| 14:48:01 | frickler | either that or it is the mad cpu and we only have that in vexxhost, not 100% sure yet | |
| 14:48:53 | clif | I assume this check is only triggered if an amd cpu is detected | |
| 14:49:43 | sean-k-mooney | clif: yes the exact check is complicated as we are also checking if libvirt has supprot too | |
| 14:50:11 | tkajinam | I suspect that there are something patched properly in the tests which eventually let the driver code to read that path | |
| 14:50:18 | tkajinam | I've not yet identified that yet though | |
| 14:52:29 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 7913, in _get_mem_encryption_config | |
| 14:52:31 | sean-k-mooney | if not self._host.supports_mem_encryption: | |
| 14:52:42 | sean-k-mooney | so that the trigger right | |
| 14:52:53 | sean-k-mooney | _get_guest_config | |
| 14:53:03 | sean-k-mooney | is calling _get_mem_encryption_config | |
| 14:53:16 | sean-k-mooney | whic eventully calls the sev check | |
| 14:53:24 | sean-k-mooney | but that happing alwasy now | |
| 14:54:04 | sean-k-mooney | so a way to fix this is in the libvirt fixutre alwasy mock _kernel_supports_amd_sev | |
| 14:54:14 | sean-k-mooney | and save it in self | |
| 14:54:26 | sean-k-mooney | then you can set the return value as needed in the sev tests | |
| 14:55:30 | sean-k-mooney | most unit test shoudl not get as far as calling _kernel_supports_amd_sev but if they do just returning False by defualt shoudl be fine | |
| 14:55:56 | sean-k-mooney | and you can either disbale the mockign with a class var or save the refence to the mock so you can modify it for the test that do | |
| 14:57:30 | sean-k-mooney | https://github.com/openstack/nova/blob/39dc667a1f9a7c12814f5892b900462560b1a229/nova/virt/libvirt/host.py#L2097 | |
| 14:57:35 | sean-k-mooney | i think that is the new call | |
| 14:57:43 | sean-k-mooney | added in https://github.com/openstack/nova/commit/7dd0d6106bdcf9babb81d7b55fd775b543a6e9ad | |
| 15:02:12 | sean-k-mooney | tkajinam: https://github.com/openstack/nova/commit/7dd0d6106bdcf9babb81d7b55fd775b543a6e9ad#diff-319aeacd4767563ecfa85b0690ccb4dfbdc9ce39173d08058f14cdce764c5040L2728-L2741 | |
| 15:02:32 | sean-k-mooney | so actully it this fixutre change i think | |
| 15:03:30 | sean-k-mooney | the libvirt fixure used ot alwasy mock the calls to read those files | |
| 15:04:47 | sean-k-mooney | self.useFixture(fixtures.MonkeyPatch('os.path.exists', fake_exists)) was overlly broad | |
| 15:05:01 | sean-k-mooney | but you can do a much more targeted mock in the same place | |
| 15:06:32 | sean-k-mooney | self.kernel_sev = self.useFixture(fixtures.MockPatch('nova.virt.libvirt.host.Host._kernel_supports_amd_sev', return_value=False).mock | |
| 15:30:44 | tkajinam | sean-k-mooney, I avoided that fixture in the base class so that I don't have to un-patch it later in test cases where these should be detected properly, but I may have to recheck that approach. | |
| 15:31:28 | tkajinam | but I have to revisit that approach, I agree | |
| 15:31:52 | sean-k-mooney | you wont need to unpatch it just modify the mock that is saved in self | |
| 15:32:13 | sean-k-mooney | but ya i get the intent of remvoign the broad mock | |
| 15:40:05 | tkajinam | yup | |
| 15:41:07 | tkajinam | I'm too tired to dig into it now but will take a further look tomorrow. | |
| 15:41:52 | tkajinam | I guess open is not triggered in intel machine with kvm due to absence of that path (and os.path.exists is not poison-pilled | |
| 16:10:10 | opendevreview | Joan Gilabert proposed openstack/nova master: Add libvirt driver MDEV ARQ support for Cyborg https://review.opendev.org/c/openstack/nova/+/995840 | |
| 16:10:10 | opendevreview | Joan Gilabert proposed openstack/nova master: Report OWNER_NOVA trait on resource providers https://review.opendev.org/c/openstack/nova/+/994299 | |