| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-04 | |||
| 08:36:44 | gibi | bauzas: could you look back at you gpu doc patch we have some question there https://review.opendev.org/c/openstack/nova/+/999607 | |
| 08:37:52 | bauzas | gibi: you're making a great point, this is not only for vfio-mdev but also for vfio-pci variant driver migrations | |
| 08:38:33 | bauzas | I'll amend the doc patch to mention vfio-pci too | |
| 09:19:31 | opendevreview | Sylvain Bauza proposed openstack/nova master: doc: Document postcopy incompatibility with vGPU live migration https://review.opendev.org/c/openstack/nova/+/999607 | |
| 09:45:16 | sean-k-mooney | bauzas: you or cursor added "Co-authored-by: Cursor <cursoragent@cursor.com>" that wrong, but otherwise the commit looks fine to me | |
| 09:45:27 | bauzas | doh | |
| 09:45:35 | bauzas | I'll change that | |
| 09:46:28 | opendevreview | Sylvain Bauza proposed openstack/nova master: doc: Document postcopy incompatibility with vGPU live migration https://review.opendev.org/c/openstack/nova/+/999607 | |
| 09:46:46 | sean-k-mooney | thanks | |
| 09:48:35 | sean-k-mooney | bauzas: speaking of gpus i have some cyborg related gpu patches that i would like to progress on the nova side. | |
| 09:49:09 | sean-k-mooney | if i can find them again :) | |
| 09:49:36 | sean-k-mooney | ah yes https://review.opendev.org/q/topic:%22bp/support-vfio-variant-driver-managed-mode-via-cyborg%22 | |
| 09:49:52 | bauzas | no promises I can review them but I'll try | |
| 09:49:55 | sean-k-mooney | im going to be rebasing those later today | |
| 09:51:03 | sean-k-mooney | i basically have 6 patches in nova that i want to try an land this cycle, those 2 for vfio-varitnt driver suport via cybrog | |
| 09:51:25 | sean-k-mooney | 2 patches that jgilaber is creating for time sliced gpus (vfio-mdev) | |
| 09:51:38 | sean-k-mooney | and then 2 for ironci start up time | |
| 09:52:11 | sean-k-mooney | if you have time to look at any of those that woudl be great if not i will try and follow up with other to land them in time | |
| 09:52:12 | bauzas | okay, I'm pretty busy before my two-week PTO which will be on next Tuesday but I'll definitely try | |
| 09:52:29 | bauzas | thanks for the effortd | |
| 09:54:41 | sean-k-mooney | as an fyi we are also seting up devstack based nightly ci to test as much of this as possible with real hardware internally (we cant currentl report the logs externally due to internal info in the job) but on the cybrog side we have been developing a kernel module to emulsate gpus, nvme devices and eventually nics | |
| 09:55:07 | sean-k-mooney | so we will have a mix of 1st party ci and internal coverage in the next few months for this | |
| 09:56:38 | sean-k-mooney | joan will be on pto startign tomorrow for the next 3 weeks so we likey will continue the 1st party testign efforts into the start of next cycle | |
| 09:56:40 | sean-k-mooney | https://review.opendev.org/q/topic:%22cyborg-vgpu-support%22 | |
| 09:56:51 | sean-k-mooney | that is the timesliced cybrog vgpu nova patch series | |
| 09:57:09 | sean-k-mooney | again just general context for everyone | |
| 10:22:16 | opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | |
| 11:05:44 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: check tempest for instability https://review.opendev.org/c/openstack/nova/+/999378 | |
| 11:46:39 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 11:46:39 | opendevreview | Stephen Finucane proposed openstack/nova master: Replace use of datetime.timezone.UTC https://review.opendev.org/c/openstack/nova/+/989515 | |
| 11:46:40 | opendevreview | Stephen Finucane proposed openstack/nova master: trivial: Fix outstanding flake8 issues https://review.opendev.org/c/openstack/nova/+/999714 | |
| 13:07:54 | tkajinam | clif, hm the failure is not reproduced in my local env (and in CI, either) | |
| 13:08:25 | tkajinam | clif, do you mind reporting a bug with a few examples of the failing tests ? | |
| 13:56:00 | sean-k-mooney | tkajinam: are you runing on an mad cpu | |
| 13:56:04 | sean-k-mooney | *amd | |
| 14:00:37 | sean-k-mooney | im running the test now locally to confirm but i also didnt see the failures on my intel cpu | |
| 14:01:37 | sean-k-mooney | and if i cat the file | |
| 14:01:39 | sean-k-mooney | [15:01:17]> cat /sys/module/kvm_amd/parameters/sev | |
| 14:01:41 | sean-k-mooney | [bat error]: '/sys/module/kvm_amd/parameters/sev': No such file or directory (os error 2) | |
| 14:02:20 | sean-k-mooney | i get an error so it would fail on this laptop if the mock didnt work | |
| 14:02:22 | sean-k-mooney | Ran: 17007 tests in 214.5597 sec. | |
| 14:02:24 | sean-k-mooney | - Passed: 16940 | |
| 14:02:26 | sean-k-mooney | - Skipped: 65 | |
| 14:02:28 | sean-k-mooney | - Expected Fail: 2 | |
| 14:02:30 | sean-k-mooney | - Unexpected Success: 0 | |
| 14:02:32 | sean-k-mooney | - Failed: 0 | |
| 14:02:34 | sean-k-mooney | Sum of execute time for each test: 3262.8758 sec. | |
| 14:02:43 | sean-k-mooney | ill try recreating the env just to make sure but on py 3.14 it seams to be working | |
| 14:03:29 | sean-k-mooney | oh | |
| 14:03:45 | sean-k-mooney | maybe this only fails on amd cpus | |
| 14:04:06 | sean-k-mooney | because we may be skiping earlier on intel before we try and read the path | |
| 14:10:03 | dansmith | succeeds on macos arm64, FWIW, so it'd have to be AMD_specific | |
| 14:10:04 | frickler | I have an VM on amd where I can reproduce this, I can give you access if that helps | |
| 14:16:10 | frickler | does this imply that all of our zuul nodes are on intel? | |
| 14:17:10 | frickler | oh, or does it need kvm enabled? maybe we can try running a tox job on a big kvm node | |
| 14:19:23 | sean-k-mooney | frickler: thre are a couple of diffent checks for sev supprot so i would not be surprised if we side step the fialign check on the amd vms as well | |
| 14:19:47 | sean-k-mooney | from pi | |
| 14:19:49 | sean-k-mooney | I found no unpatched access to /sys/module/kvm_amd/parameters/sev in the current checkout. | |
| 14:19:51 | sean-k-mooney | - nova/tests/fixtures/libvirt.py:2727 forces the SEV parameter to appear absent in LibvirtFixture. | |
| 14:19:53 | sean-k-mooney | - SEV-specific tests explicitly patch os.path.exists and open. | |
| 14:19:55 | sean-k-mooney | - SysFsPoisonFixture raises if a unit test actually opens an unmocked /sys path. | |
| 14:19:57 | sean-k-mooney | Validation: | |
| 14:19:59 | sean-k-mooney | - test_driver: 1,186 tests passed | |
| 14:20:01 | sean-k-mooney | - test_host: 127 tests passed | |
| 14:20:03 | sean-k-mooney | One unrelated potential typo exists: the fixture patches sev-es, while the production path is sev_es. | |
| 14:20:23 | sean-k-mooney | tkajinam: ^ | |
| 14:23:01 | sean-k-mooney | im checkign what that actully is now but also in an irc meeting | |
| 14:24:36 | sean-k-mooney | clif: just to confirm were the fialing teest sev specific or unrelated | |
| 14:24:59 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 14:25:43 | frickler | sean-k-mooney: one test was nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_unplug_vifs_with_destroy_vifs_true , doesn't look related to me | |
| 14:25:45 | sean-k-mooney | we may have mocked it in all the sev paths an missed it in the genric tests paths | |
| 14:25:54 | sean-k-mooney | ack | |
| 14:29:40 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test unit tests on kvm https://review.opendev.org/c/openstack/nova/+/999729 | |
| 14:31:39 | clif | sean-k-mooney: every test failure I still have in traceback is failing due to the SEV exception | |
| 14:31:54 | clif | I can re-run and save all output | |
| 14:36:51 | clif | https://gist.github.com/ClifHouck/b7186501897e8276afcc9a7827e21789 here's a full traceback of one error | |
| 14:38:54 | clif | yep, I get 217 failures and they're all due to this SEV exception | |
| 14:41:42 | frickler | ok, my test triggers the same failures. maybe add that permanently at least as experimental or periodic job? | |
| 14:42:31 | frickler | https://zuul.opendev.org/t/openstack/build/1af158b6480d4783b0211c4f4d890fed | |
| 14:45:23 | tkajinam | ohhk let me check that | |
| 14:46:54 | sean-k-mooney | frickler: ok so its related to kvm? | |
| 14:48:01 | frickler | either that or it is the mad cpu and we only have that in vexxhost, not 100% sure yet | |
| 14:48:53 | clif | I assume this check is only triggered if an amd cpu is detected | |
| 14:49:43 | sean-k-mooney | clif: yes the exact check is complicated as we are also checking if libvirt has supprot too | |
| 14:50:11 | tkajinam | I suspect that there are something patched properly in the tests which eventually let the driver code to read that path | |
| 14:50:18 | tkajinam | I've not yet identified that yet though | |
| 14:52:29 | sean-k-mooney | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 7913, in _get_mem_encryption_config | |
| 14:52:31 | sean-k-mooney | if not self._host.supports_mem_encryption: | |
| 14:52:42 | sean-k-mooney | so that the trigger right | |
| 14:52:53 | sean-k-mooney | _get_guest_config | |
| 14:53:03 | sean-k-mooney | is calling _get_mem_encryption_config | |
| 14:53:16 | sean-k-mooney | whic eventully calls the sev check | |
| 14:53:24 | sean-k-mooney | but that happing alwasy now | |
| 14:54:04 | sean-k-mooney | so a way to fix this is in the libvirt fixutre alwasy mock _kernel_supports_amd_sev | |
| 14:54:14 | sean-k-mooney | and save it in self | |
| 14:54:26 | sean-k-mooney | then you can set the return value as needed in the sev tests | |
| 14:55:30 | sean-k-mooney | most unit test shoudl not get as far as calling _kernel_supports_amd_sev but if they do just returning False by defualt shoudl be fine | |
| 14:55:56 | sean-k-mooney | and you can either disbale the mockign with a class var or save the refence to the mock so you can modify it for the test that do | |
| 14:57:30 | sean-k-mooney | https://github.com/openstack/nova/blob/39dc667a1f9a7c12814f5892b900462560b1a229/nova/virt/libvirt/host.py#L2097 | |
| 14:57:35 | sean-k-mooney | i think that is the new call | |