| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2026-05-05 | |||
| 14:18:32 | jgilaber | #link https://bugs.launchpad.net/openstack-cyborg/+bug/1970818 | |
| 14:20:43 | jgilaber | this looks like a valid bug to me, although we should verify if the problem still exists, since it was reported for a quite older version | |
| 14:21:06 | chandankumar | yes, I am working on verify that | |
| 14:22:22 | jgilaber | ack, thanks | |
| 14:22:54 | jgilaber | ok to mark it as incomplete and revisit once we have verified it? | |
| 14:24:22 | chandankumar | I just added the comment there, | |
| 14:25:18 | chandankumar | The traceback itself is too long, Not useful for end users | |
| 14:25:55 | jgilaber | indeed, this is valid then, thanks chandankumar! | |
| 14:26:00 | chandankumar | I think we can mark them it as a valid bug, i think it needs to be imprived | |
| 14:26:29 | jgilaber | +1, any thougs about the importance? medium/high? | |
| 14:26:57 | chandankumar | I would set medium as user can see the error message. UX needs to be improved here | |
| 14:27:34 | jgilaber | sounds good, done | |
| 14:27:51 | jgilaber | let's move to the second one | |
| 14:27:56 | jgilaber | #link https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 | |
| 14:28:17 | chandankumar | Let me work on the reproducer for this bug and update it | |
| 14:29:07 | jgilaber | ack, let's revisit this in the future, this looks more complex | |
| 14:29:52 | jgilaber | let's look at the last one | |
| 14:29:57 | jgilaber | #link https://bugs.launchpad.net/openstack-cyborg/+bug/2011754 | |
| 14:31:03 | chandankumar | For this one, we need a gpu device. | |
| 14:31:35 | jgilaber | I'm not sure of the current status of this functionality for PCI passthrough but if it's still valid it will likely need changes in the nova-cyborg interface | |
| 14:31:53 | jgilaber | so it will be more of a feature than a bug | |
| 14:32:29 | chandankumar | yes, agree here. | |
| 14:33:09 | chandankumar | I will try with pci interface and see what happens. | |
| 14:34:18 | jgilaber | ack, we need more investigation let's table the discussion for now. I'll add a comment with what was discussed | |
| 14:34:44 | jgilaber | while I do that, any topic for the open discussion? | |
| 14:36:35 | chandankumar | not from my side. | |
| 14:36:41 | jgilaber | looks like we can wrap early today | |
| 14:36:51 | jgilaber | #topic Volunteers to chair next meeting | |
| 14:36:53 | jgilaber | any volunteer? | |
| 14:37:07 | chandankumar | I can take care of next meeting | |
| 14:37:15 | jgilaber | ack ,thanks chandankumar | |
| 14:37:35 | jgilaber | that is all for today, thanks for joining folks! | |
| 14:37:41 | opendevmeet | Log: https://meetings.opendev.org/meetings/cyborg/2026/cyborg.2026-05-05-14.01.log.html | |
| 14:37:41 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/cyborg/2026/cyborg.2026-05-05-14.01.txt | |
| 14:37:41 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/cyborg/2026/cyborg.2026-05-05-14.01.html | |
| 14:37:41 | opendevmeet | Meeting ended Tue May 5 14:37:41 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 14:37:41 | jgilaber | #endmeeting | |
| 14:40:59 | chandankumar | thank you jgilaber ! | |
| #openstack-cyborg - 2026-05-06 | |||
| 08:50:32 | chandankumar | jgilaber_: Hello, I have tried to reproduced this bug https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 But from bug description, I am not user is removing pci device from the config or from the host directly like unplugging the device. | |
| 08:51:17 | chandankumar | In my reproducer, I removed the pci device from config and then similar traceback comes from cyborg-agent log. | |
| 08:51:31 | chandankumar | Feel free to take a look at the bug and triage accordingly. Thank you! | |
| 08:54:25 | jgilaber | I understood it originally as unplugging the device | |
| 08:56:22 | jgilaber | thanks for the reproducer! | |
| 08:57:12 | jgilaber | I'm not sure what would be the best way to handle such cases | |
| 08:57:36 | chandankumar | one more thing, we delete the device profile, you can see the error message, it is also not clear for end user | |
| 08:57:47 | jgilaber | ideally we would probably unbind the arq and send a new bind request to nova if an equivalent free device exists | |
| 08:58:28 | jgilaber | the agent automatically deletes the device profile? | |
| 09:00:48 | chandankumar | let me try with unbind and see what happens | |
| 10:19:22 | chandankumar | jgilaber: I ubind the device https://paste.openstack.org/raw/bTWU8NVSsBOdJjRNbym3/ but in cyborg, hotplug of a new pci device in running instance is not supported | |
| 10:19:40 | chandankumar | we can create the bind but it does not will have the device inside the vm | |
| 10:20:41 | chandankumar | or I misunderstood send a new bind request to nova if an equivalent free device exist this part | |
| 10:25:45 | jgilaber | hmm I guess that with the current integration any change requires a resize | |
| 10:26:05 | sean-k-mooney | correct it does | |
| 10:26:09 | jgilaber | to be clear my previous comment was me speculating how we could address the bug, I did not expect that was currently supported | |
| 10:26:25 | sean-k-mooney | device profilces shoudl not be modifed if they are refence by a falvor | |
| 10:26:42 | sean-k-mooney | and the only way to chagne the allocation ot a vm is via reisze | |
| 10:27:26 | sean-k-mooney | i am still catching up on email form beign away for a few days so we can chat more about this tomorrow | |
| 10:27:56 | chandankumar | sure | |
| 10:28:04 | jgilaber | sure no problem sean-k-mooney. For context, we're talking about an old bug we dicussed briefly in yesterday's meeting | |
| 10:28:13 | jgilaber | https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 | |
| 10:28:52 | sean-k-mooney | python 3.6 :) | |
| 10:29:00 | sean-k-mooney | ya that been a while | |
| 10:29:56 | sean-k-mooney | this looks simialr to a very old nova bug | |
| 10:30:42 | sean-k-mooney | i think the pci driver is likely missing the protection we ahve on the nova side but ya we can revisit this once im back up to speed | |
| 14:04:19 | sean-k-mooney | https://github.com/openstack/nova/commit/26c41eccade6412f61f9a8721d853b545061adcc https://github.com/openstack/nova/commit/284ea72e96604bdf16d1c5c4db47247334841b2f https://github.com/openstack/nova/commit/0208be629c3853863bcd49b8bdbe2b9889b85012 https://github.com/openstack/nova/commit/f37cdf0c4182103ad81dbf39188ff39955da3850 | |
| 14:04:38 | sean-k-mooney | those are the nova patches related to the isseu reproted in https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 | |
| 14:05:53 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1633120 https://bugs.launchpad.net/nova/+bug/1969496 and https://bugs.launchpad.net/nova/+bug/2115905 are the releated nova bugs we had | |
| 14:07:31 | sean-k-mooney | the tl;dr is if a device is refence by a ARQ and that device is not in the whitelist or viaabel on teh host anymore we cannot remove the device form the db or placmeent until that ARQ is deleted and we shoudl not do that automaticlly | |
| 14:07:46 | sean-k-mooney | the admin need to move or delete the vm or readd the device | |
| 14:08:34 | sean-k-mooney | we shoudl complain very very loadly in teh logs when the compute agent start up in a miscondifured state but we dont geenrally want to make that an agent startup failure as that a potical dos vector if we do | |
| 14:23:08 | jgilaber | I'm not sure from the bug report if it actually crashes the agent or it just logs the traceback in an ugly way | |
| 14:23:23 | jgilaber | Chandan reproduced the error, we can ask him later/tomorrow | |
| 14:30:10 | sean-k-mooney | in either case thsi shoudl be handeled gracefully and we shoudl not attepmt to delete the resouce provider | |
| 14:30:29 | sean-k-mooney | trying to delete it if it has allcoation agaisnt it is a bug in our internal logic | |
| 14:30:53 | sean-k-mooney | the protections in placment are really the last line of defence | |
| #openstack-cyborg - 2026-05-07 | |||
| 10:32:22 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 10:40:37 | tafkamax | Hi I just deployed cyborg and are planning to use it to pass PGPU-s to vm-s for now. | |
| 10:41:06 | tafkamax | I just talked in kolla chat aswell, as it is active and there are is some knowledge there too. | |
| 10:42:18 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 10:42:23 | tafkamax | The GPU shows up in `openstack accelerator device show UUID`... (full message at <https://matrix.org/oftc/media/v1/media/download/AW1EbeU-tbWQQQrjpuO6aBPema1ZFNxz1H-Oe8zT3IzpZWc2PNyPKxAaRLOwQuK9lv5coKnS2iHnahg2PqYfBgFCeeSiEWiwAG1hdHJpeC5vcmcvbmZOem9Rc2Zqak9zb2JialhsU0RSWmNB>) | |
| 10:42:35 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Move job definitions to cyborg repo https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/987621 | |
| 10:42:49 | tafkamax | the traits are pretty empty for it though:... (full message at <https://matrix.org/oftc/media/v1/media/download/AUewwyd-zCY46PWfpF0JsBLctL5NmRnri3j9bK6462IJYTg7QZOmF_RjeTDUs9FvS2nB6diGWvBU9MVBF2J4ynFCeeSiF-zAAG1hdHJpeC5vcmcvZ0xUUWRHYUNOdHhzVlp0RENUU2tsY2xT>) | |
| 10:43:44 | tafkamax | I also tried this accelerator profile... (full message at <https://matrix.org/oftc/media/v1/media/download/ARQew_Tex2aH3YGmBZDRiWwCFloG5Cdc0UaMQ5z9BfAuBKNfoNRQyiIb9zs5J0HZola9q9RhLGjJY1Y3Jb_PGcVCeeSiJUsQAG1hdHJpeC5vcmcvUVdVeEtqaUNXeXBZT2xOd3dnSmZ4c0hv>) | |
| 10:43:54 | tafkamax | Any thoughts? | |
| 11:01:57 | jgilaber | tafkamax, hi! the traits look correct to me. Was the flavor created correctly? | |
| 11:03:18 | tafkamax | openstack flavor show 6943fd8e-08ea-4541-bc11-a167f270e98f... (full message at <https://matrix.org/oftc/media/v1/media/download/AXUWetSMAsXVtnlrnt-ox0XHF4gtEKxjUQ1zhRXjy7dCvygSAi7WqnZJr4jUfNLe3gDhYf-3t8FlaROgiijBg15CeeSjQ8dQAG1hdHJpeC5vcmcvZnZ3ZHdiZ0tkbUhUUnluTUxrRWtJUkpt>) | |
| 11:05:55 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 11:05:58 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Move job definitions to cyborg repo https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/987621 | |
| 11:11:54 | jgilaber | is there any more detail in the nova logs? it looks like placement might be reporting that there are no available gpus | |
| 11:12:31 | jgilaber | could maybe nova be configured to passthrough the gpu as well? | |
| 11:13:44 | tafkamax | We havent explicilty configured nova for that. | |
| 11:13:59 | jgilaber | ack, thanks, can you also check what 'openstack accelerator arq list' reports? | |
| 11:14:13 | tafkamax | kolla-ansible 2025.1 deployment type | |
| 11:15:29 | tafkamax | hmm empty | |
| 11:15:58 | tafkamax | oh I need to do that beforehand? | |
| 11:17:47 | chandankumar | Do we need to create the device profile with resource class like this https://paste.openstack.org/raw/bJEmpgye27s5h4tEMlTf/ | |
| 11:18:03 | chandankumar | for pci and fake fpga device, we pass resource class | |
| 11:18:27 | tafkamax | Like PGPU? | |
| 11:20:40 | chandankumar | sorry it should be GPU | |
| 11:21:48 | tafkamax | ok, thanks, so that comes from openstack accelerator device show command | |