| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2026-07-16 | |||
| 16:37:28 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add MdevBusManager for mdev bus discovery https://review.opendev.org/c/openstack/cyborg/+/997048 | |
| 16:37:28 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add [mdev] configuration section https://review.opendev.org/c/openstack/cyborg/+/997049 | |
| 16:37:29 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add generic mdev driver https://review.opendev.org/c/openstack/cyborg/+/997050 | |
| 16:37:29 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add duplicate device warning in conductor https://review.opendev.org/c/openstack/cyborg/+/997051 | |
| 16:37:30 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Refactor GPU driver to use MdevBusManager https://review.opendev.org/c/openstack/cyborg/+/997052 | |
| 16:37:30 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add generic mdev driver documentation https://review.opendev.org/c/openstack/cyborg/+/997053 | |
| 16:37:31 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add mdev support to pci-sim kernel module https://review.opendev.org/c/openstack/cyborg/+/997054 | |
| 16:37:32 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add mdev support to pci-sim DevStack plugin https://review.opendev.org/c/openstack/cyborg/+/997055 | |
| 16:37:32 | opendevreview | Joan Gilabert proposed openstack/cyborg master: [WIP] Configure mdev driver lifecycle tests https://review.opendev.org/c/openstack/cyborg/+/997364 | |
| 17:45:16 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Refactor GPU driver to use MdevBusManager https://review.opendev.org/c/openstack/cyborg/+/997052 | |
| 17:45:16 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add generic mdev driver documentation https://review.opendev.org/c/openstack/cyborg/+/997053 | |
| 17:45:17 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add mdev support to pci-sim kernel module https://review.opendev.org/c/openstack/cyborg/+/997054 | |
| 17:45:17 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Add mdev support to pci-sim DevStack plugin https://review.opendev.org/c/openstack/cyborg/+/997055 | |
| 17:45:18 | opendevreview | Joan Gilabert proposed openstack/cyborg master: [WIP] Configure mdev driver lifecycle tests https://review.opendev.org/c/openstack/cyborg/+/997364 | |
| #openstack-cyborg - 2026-07-17 | |||
| 06:44:53 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Add Cyborg MDEV driver lifecycle scenario tests https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997352 | |
| 06:44:53 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Prefer lspci over sysfs for PCI device detection https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997684 | |
| 06:45:57 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Configure pci lifecycle tempest settings via devstack plugin https://review.opendev.org/c/openstack/cyborg/+/997365 | |
| 10:03:31 | jgilaber | sean-k-mooney Hi! I've pushed a patch to change how we check for pci devices in guest vms https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997684 to prefer using lspci | |
| 10:03:57 | jgilaber | I've tested the change in https://review.opendev.org/c/openstack/cyborg/+/997364 and https://review.opendev.org/c/openstack/cyborg/+/997365 multiple times and seems to work well | |
| 10:04:01 | sean-k-mooney | cool | |
| 10:04:11 | sean-k-mooney | were you able ot replciate the failure locally? | |
| 10:04:13 | jgilaber | I've run the tests at lest 8 times between yesterday and today | |
| 10:04:28 | jgilaber | no I have not tested that | |
| 10:04:39 | sean-k-mooney | well im more askign did you see it fail | |
| 10:04:55 | jgilaber | only on ci | |
| 10:05:01 | sean-k-mooney | i have not seen the older verion fail locally either | |
| 10:05:13 | sean-k-mooney | ack ok well we can recheck it a few times | |
| 10:05:40 | jgilaber | yes, we can recheck it a few times more today and check back towards eod or next week | |
| 10:05:58 | jgilaber | I will try it locally as well | |
| 10:06:26 | sean-k-mooney | cool the change looks good | |
| 10:06:51 | sean-k-mooney | out of inteest did we looks at how we do this in whitebox | |
| 10:07:54 | sean-k-mooney | https://opendev.org/openstack/whitebox-tempest-plugin/src/commit/a464d48e5e517ac8e4d04ca72b25042067abcb31/whitebox_tempest_plugin/api/compute/test_sriov.py#L862 | |
| 10:08:08 | sean-k-mooney | so ya def _check_device_in_guest(self, linux_client, vendor_id, product_id): | |
| 10:08:24 | sean-k-mooney | is doing cmd = "lspci -nn | grep {0}:{1} | wc -l".format( | |
| 10:08:26 | sean-k-mooney | vendor_id, product_id) | |
| 10:08:55 | sean-k-mooney | we may wantt to condier just porting that function as this has been sabel but its usign lspci as well | |
| 10:09:05 | sean-k-mooney | so thats good at least | |
| 10:09:07 | jgilaber | very similar | |
| 10:09:39 | sean-k-mooney | so we dont technially need the fall back | |
| 10:09:45 | sean-k-mooney | well | |
| 10:09:55 | jgilaber | I've left the current approach as fallback but we could remove it to simplify maybe | |
| 10:09:58 | sean-k-mooney | lspci is expected to be installled in the ci image | |
| 10:10:28 | sean-k-mooney | for now i think we can keep it but we might simplfy this in a followup | |
| 10:10:58 | sean-k-mooney | i kind of like that _check_device_in_guest is doing the filstering on the vm rather then copying back all the data | |
| 10:11:05 | jgilaber | sounds good to me | |
| 11:38:08 | jgilaber | I've been running the version of the tests in master which failed yesterday but could not reproduce the problem either | |
| 11:38:34 | jgilaber | I've run them 6 or 7 time nows, it's quite weird that it seemed so consistent yesterday | |
| 11:51:47 | chandankumar | https://zuul.opendev.org/t/openstack/builds?change=997365&result=FAILURE&skip=0 , one time py3 and ipv6 failed job ran on rax-ord-main . | |
| 11:52:05 | chandankumar | yesterday | |
| 11:53:54 | chandankumar | https://zuul.opendev.org/t/openstack/builds?job_name=cyborg-tempest-ipv6&change=997365&skip=0 - rest of the passing ipv6 ran on different provider. | |
| 11:54:13 | chandankumar | I am not sure it is linked with provider also. | |
| 12:00:09 | opendevreview | Takashi Kajinami proposed openstack/cyborg master: Add cors middleware defaults https://review.opendev.org/c/openstack/cyborg/+/997712 | |
| 12:00:25 | jgilaber | today it's been running in different providers, will recheck a few more times to see | |
| 13:11:02 | opendevreview | Takashi Kajinami proposed openstack/cyborg master: Add cors middleware defaults https://review.opendev.org/c/openstack/cyborg/+/997712 | |
| 13:31:16 | jgilaber | chandankumar, I think you might be onto something, the ipv6 failed now and it ran on rax-ord-main https://zuul.opendev.org/t/openstack/build/87e90ed8ebbc4b7cad73c3d69d76c6a0 | |
| 13:43:35 | sean-k-mooney | jgilaber: it could be timeing related but i think the lspci approch is going to more robust anyway | |
| 13:43:52 | sean-k-mooney | so ya if its still stable ill approve it in an hour or two | |
| 13:44:34 | sean-k-mooney | i was surpsied when you menteiond it sicne i had never seen it fail personally before | |
| 13:45:16 | sean-k-mooney | but again the ci can show issue we dont see on our local system so im not that surpised either | |
| 13:45:30 | jgilaber | yes, it did seem stable before that point | |
| 13:45:52 | jgilaber | it's super stable in my dev env as well, could not get it to fail there | |
| 13:47:10 | chandankumar | Do we want to add some wait condition https://github.com/openstack/cyborg-tempest-plugin/blob/master/cyborg_tempest_plugin/tests/scenario/test_pci_driver_lifecycle.py#L194 after reboot to make sure server is ip and call _assert_server_pci_ready ? | |
| 13:47:48 | chandankumar | from multiple runs, it is failing on rax-ord-main provider only | |
| 14:10:56 | sean-k-mooney | we shodul rewrite that test i think | |
| 14:11:09 | sean-k-mooney | well | |
| 14:11:33 | sean-k-mooney | we can keep it but yes we need a retry loop | |
| 14:11:53 | sean-k-mooney | i was thinkin gif we wanted to delete that test and just rely on teh nova api reboot tests | |
| 14:12:59 | sean-k-mooney | but for the gues one we need to wait for the vm to be back up before calling _assert_server_pci_ready | |
| 14:13:53 | sean-k-mooney | if we really want this to be more roboust https://github.com/openstack/cyborg-tempest-plugin/blob/master/cyborg_tempest_plugin/tests/scenario/test_pci_driver_lifecycle.py#L161 shoudl be SSHABLE instead of active as well | |
| 14:15:24 | sean-k-mooney | since we are going to try adn ssh into it waiting for SSHABLE which is a tempest concept is more accurate as ACTIVE just means nova thinks its running | |
| 14:15:39 | sean-k-mooney | it does nto mean the guest os has fully booted | |
| 14:52:43 | jgilaber | sean-k-mooney, sorry I had to step out for a bit | |
| 14:53:02 | sean-k-mooney | its fine | |
| 14:53:05 | jgilaber | I can change the state there for 'SSHABLE' but I don't think that is called in the test in question | |
| 14:53:22 | sean-k-mooney | lets revisit that | |
| 14:53:35 | sean-k-mooney | we can look at doign that speratly | |
| 14:53:58 | sean-k-mooney | beasue we likely shoudl do that consitently not just in those tests | |
| 14:54:07 | sean-k-mooney | also when waiting for the server to create | |
| 14:54:14 | sean-k-mooney | not just reboot ectra | |
| 14:54:46 | sean-k-mooney | jgilaber: are you happy enough to proceed with the currnt improment | |
| 14:55:19 | sean-k-mooney | if so i can appove the test change and we can then look at appoving the job change patch | |
| 14:55:41 | sean-k-mooney | ill be around for anouther hour or two then ill be finishing up | |
| 14:58:52 | jgilaber | we could go ahead, I think it is slightly better, although it does not completely solve the problem | |
| 14:59:59 | jgilaber | i.e if we enable the tests we might get gate failures which will be annoying | |
| 15:00:17 | sean-k-mooney | yes but i wont be here next week | |
| 15:00:35 | sean-k-mooney | so we wont be merging anything anyway and ye can tell me how it goes when i get back :P | |
| 15:00:45 | sean-k-mooney | so that sounds like a you problem to me :) | |
| 15:01:45 | jgilaber | yep, that's about right | |
| 15:01:51 | sean-k-mooney | on a seious not however i think i woudl prefer having the test run for a week even if they fail | |
| 15:02:04 | sean-k-mooney | just to actullly flesh out any opther issues with them | |
| 15:02:14 | jgilaber | we can go ahead since it won't be blocking anything and we can collect more data from other patches | |
| 15:47:41 | opendevreview | Merged openstack/cyborg-tempest-plugin master: Prefer lspci over sysfs for PCI device detection https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997684 | |
| 15:48:53 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Add Cyborg MDEV driver lifecycle scenario tests https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997352 | |
| 15:48:54 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Verify guest reboot started in PCI lifecycle test https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/997751 | |
| 15:49:36 | jgilaber | I'm not convinced this will fix the problem but I'll try during the week | |
| 15:50:44 | jgilaber | sean-k-mooney, do you want to merge https://review.opendev.org/c/openstack/cyborg/+/997365 before you leave? | |
| 15:59:58 | sean-k-mooney | i will yes on a call | |
| 16:47:55 | sean-k-mooney | ok its reappoved | |
| 16:48:17 | sean-k-mooney | assuming it passes in gate that shoudl merged | |
| 16:48:47 | sean-k-mooney | im goign to finish up in the next 15 minutes or so | |