| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2026-05-05 | |||
| 14:36:41 | jgilaber | looks like we can wrap early today | |
| 14:36:51 | jgilaber | #topic Volunteers to chair next meeting | |
| 14:36:53 | jgilaber | any volunteer? | |
| 14:37:07 | chandankumar | I can take care of next meeting | |
| 14:37:15 | jgilaber | ack ,thanks chandankumar | |
| 14:37:35 | jgilaber | that is all for today, thanks for joining folks! | |
| 14:37:41 | jgilaber | #endmeeting | |
| 14:37:41 | opendevmeet | Meeting ended Tue May 5 14:37:41 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 14:37:41 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/cyborg/2026/cyborg.2026-05-05-14.01.html | |
| 14:37:41 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/cyborg/2026/cyborg.2026-05-05-14.01.txt | |
| 14:37:41 | opendevmeet | Log: https://meetings.opendev.org/meetings/cyborg/2026/cyborg.2026-05-05-14.01.log.html | |
| 14:40:59 | chandankumar | thank you jgilaber ! | |
| #openstack-cyborg - 2026-05-06 | |||
| 08:50:32 | chandankumar | jgilaber_: Hello, I have tried to reproduced this bug https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 But from bug description, I am not user is removing pci device from the config or from the host directly like unplugging the device. | |
| 08:51:17 | chandankumar | In my reproducer, I removed the pci device from config and then similar traceback comes from cyborg-agent log. | |
| 08:51:31 | chandankumar | Feel free to take a look at the bug and triage accordingly. Thank you! | |
| 08:54:25 | jgilaber | I understood it originally as unplugging the device | |
| 08:56:22 | jgilaber | thanks for the reproducer! | |
| 08:57:12 | jgilaber | I'm not sure what would be the best way to handle such cases | |
| 08:57:36 | chandankumar | one more thing, we delete the device profile, you can see the error message, it is also not clear for end user | |
| 08:57:47 | jgilaber | ideally we would probably unbind the arq and send a new bind request to nova if an equivalent free device exists | |
| 08:58:28 | jgilaber | the agent automatically deletes the device profile? | |
| 09:00:48 | chandankumar | let me try with unbind and see what happens | |
| 10:19:22 | chandankumar | jgilaber: I ubind the device https://paste.openstack.org/raw/bTWU8NVSsBOdJjRNbym3/ but in cyborg, hotplug of a new pci device in running instance is not supported | |
| 10:19:40 | chandankumar | we can create the bind but it does not will have the device inside the vm | |
| 10:20:41 | chandankumar | or I misunderstood send a new bind request to nova if an equivalent free device exist this part | |
| 10:25:45 | jgilaber | hmm I guess that with the current integration any change requires a resize | |
| 10:26:05 | sean-k-mooney | correct it does | |
| 10:26:09 | jgilaber | to be clear my previous comment was me speculating how we could address the bug, I did not expect that was currently supported | |
| 10:26:25 | sean-k-mooney | device profilces shoudl not be modifed if they are refence by a falvor | |
| 10:26:42 | sean-k-mooney | and the only way to chagne the allocation ot a vm is via reisze | |
| 10:27:26 | sean-k-mooney | i am still catching up on email form beign away for a few days so we can chat more about this tomorrow | |
| 10:27:56 | chandankumar | sure | |
| 10:28:04 | jgilaber | sure no problem sean-k-mooney. For context, we're talking about an old bug we dicussed briefly in yesterday's meeting | |
| 10:28:13 | jgilaber | https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 | |
| 10:28:52 | sean-k-mooney | python 3.6 :) | |
| 10:29:00 | sean-k-mooney | ya that been a while | |
| 10:29:56 | sean-k-mooney | this looks simialr to a very old nova bug | |
| 10:30:42 | sean-k-mooney | i think the pci driver is likely missing the protection we ahve on the nova side but ya we can revisit this once im back up to speed | |
| 14:04:19 | sean-k-mooney | https://github.com/openstack/nova/commit/26c41eccade6412f61f9a8721d853b545061adcc https://github.com/openstack/nova/commit/284ea72e96604bdf16d1c5c4db47247334841b2f https://github.com/openstack/nova/commit/0208be629c3853863bcd49b8bdbe2b9889b85012 https://github.com/openstack/nova/commit/f37cdf0c4182103ad81dbf39188ff39955da3850 | |
| 14:04:38 | sean-k-mooney | those are the nova patches related to the isseu reproted in https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 | |
| 14:05:53 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1633120 https://bugs.launchpad.net/nova/+bug/1969496 and https://bugs.launchpad.net/nova/+bug/2115905 are the releated nova bugs we had | |
| 14:07:31 | sean-k-mooney | the tl;dr is if a device is refence by a ARQ and that device is not in the whitelist or viaabel on teh host anymore we cannot remove the device form the db or placmeent until that ARQ is deleted and we shoudl not do that automaticlly | |
| 14:07:46 | sean-k-mooney | the admin need to move or delete the vm or readd the device | |
| 14:08:34 | sean-k-mooney | we shoudl complain very very loadly in teh logs when the compute agent start up in a miscondifured state but we dont geenrally want to make that an agent startup failure as that a potical dos vector if we do | |
| 14:23:08 | jgilaber | I'm not sure from the bug report if it actually crashes the agent or it just logs the traceback in an ugly way | |
| 14:23:23 | jgilaber | Chandan reproduced the error, we can ask him later/tomorrow | |
| 14:30:10 | sean-k-mooney | in either case thsi shoudl be handeled gracefully and we shoudl not attepmt to delete the resouce provider | |
| 14:30:29 | sean-k-mooney | trying to delete it if it has allcoation agaisnt it is a bug in our internal logic | |
| 14:30:53 | sean-k-mooney | the protections in placment are really the last line of defence | |
| #openstack-cyborg - 2026-05-07 | |||
| 10:32:22 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 10:40:37 | tafkamax | Hi I just deployed cyborg and are planning to use it to pass PGPU-s to vm-s for now. | |
| 10:41:06 | tafkamax | I just talked in kolla chat aswell, as it is active and there are is some knowledge there too. | |
| 10:42:18 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 10:42:23 | tafkamax | The GPU shows up in `openstack accelerator device show UUID`... (full message at <https://matrix.org/oftc/media/v1/media/download/AW1EbeU-tbWQQQrjpuO6aBPema1ZFNxz1H-Oe8zT3IzpZWc2PNyPKxAaRLOwQuK9lv5coKnS2iHnahg2PqYfBgFCeeSiEWiwAG1hdHJpeC5vcmcvbmZOem9Rc2Zqak9zb2JialhsU0RSWmNB>) | |
| 10:42:35 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Move job definitions to cyborg repo https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/987621 | |
| 10:42:49 | tafkamax | the traits are pretty empty for it though:... (full message at <https://matrix.org/oftc/media/v1/media/download/AUewwyd-zCY46PWfpF0JsBLctL5NmRnri3j9bK6462IJYTg7QZOmF_RjeTDUs9FvS2nB6diGWvBU9MVBF2J4ynFCeeSiF-zAAG1hdHJpeC5vcmcvZ0xUUWRHYUNOdHhzVlp0RENUU2tsY2xT>) | |
| 10:43:44 | tafkamax | I also tried this accelerator profile... (full message at <https://matrix.org/oftc/media/v1/media/download/ARQew_Tex2aH3YGmBZDRiWwCFloG5Cdc0UaMQ5z9BfAuBKNfoNRQyiIb9zs5J0HZola9q9RhLGjJY1Y3Jb_PGcVCeeSiJUsQAG1hdHJpeC5vcmcvUVdVeEtqaUNXeXBZT2xOd3dnSmZ4c0hv>) | |
| 10:43:54 | tafkamax | Any thoughts? | |
| 11:01:57 | jgilaber | tafkamax, hi! the traits look correct to me. Was the flavor created correctly? | |
| 11:03:18 | tafkamax | openstack flavor show 6943fd8e-08ea-4541-bc11-a167f270e98f... (full message at <https://matrix.org/oftc/media/v1/media/download/AXUWetSMAsXVtnlrnt-ox0XHF4gtEKxjUQ1zhRXjy7dCvygSAi7WqnZJr4jUfNLe3gDhYf-3t8FlaROgiijBg15CeeSjQ8dQAG1hdHJpeC5vcmcvZnZ3ZHdiZ0tkbUhUUnluTUxrRWtJUkpt>) | |
| 11:05:55 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 11:05:58 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Move job definitions to cyborg repo https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/987621 | |
| 11:11:54 | jgilaber | is there any more detail in the nova logs? it looks like placement might be reporting that there are no available gpus | |
| 11:12:31 | jgilaber | could maybe nova be configured to passthrough the gpu as well? | |
| 11:13:44 | tafkamax | We havent explicilty configured nova for that. | |
| 11:13:59 | jgilaber | ack, thanks, can you also check what 'openstack accelerator arq list' reports? | |
| 11:14:13 | tafkamax | kolla-ansible 2025.1 deployment type | |
| 11:15:29 | tafkamax | hmm empty | |
| 11:15:58 | tafkamax | oh I need to do that beforehand? | |
| 11:17:47 | chandankumar | Do we need to create the device profile with resource class like this https://paste.openstack.org/raw/bJEmpgye27s5h4tEMlTf/ | |
| 11:18:03 | chandankumar | for pci and fake fpga device, we pass resource class | |
| 11:18:27 | tafkamax | Like PGPU? | |
| 11:20:40 | chandankumar | sorry it should be GPU | |
| 11:21:48 | tafkamax | ok, thanks, so that comes from openstack accelerator device show command | |
| 11:29:36 | chandankumar | https://github.com/openstack/cyborg/blob/f111946df6713aa64efa29dd025d47839241c529/cyborg/common/constants.py#L115 | |
| 11:29:45 | chandankumar | "PGPU": orc.PGPU, | |
| 11:29:45 | chandankumar | "VGPU": orc.VGPU, | |
| 11:29:52 | chandankumar | pgpu - for physcial gou | |
| 11:30:13 | chandankumar | pgpu - for physcial gpu and vgpu - for virtual gpu | |
| 11:30:57 | chandankumar | Cyborg also have GPU. | |
| 11:41:45 | tafkamax | I have this in log: 2026-05-07 14:39:05.152 7 WARNING cyborg.accelerator.drivers.gpu.nvidia.sysinfo [-] Unable to load vGPU_type from [gpu_devices] Ensure "enabled_vgpu_types" is set if the gpuis virtualized. but I dont have vgpu enabled. Is this just informational? | |
| 11:45:24 | tafkamax | from nova scheduler then: 2026-05-07 14:40:05.475 1087 ERROR nova.scheduler.client.report [req-3a504845-350b-4532-a33d-f0334e44db4c req-f8f77ce8-89c9-42c9-90e5-a9e7d6eaaf33 204c13dfae0b4214ae00b15a95a5d180 87aa79e7272a4f9b9e66e4582fb28c93 - - default default] Failed to retrieve allocation candidates from placement API for filters: | |
| 11:45:24 | tafkamax | RequestGroup(aggregates=[],forbidden_aggregates=set([]),forbidden_traits=set([]),in_tree=8b9f6c21-0c4a-458a-8541-d481921c6d08,provider_uuids=[],requester_id=None,required_traits=set([]),resources={MEMORY_MB=16384,VCPU=8},use_same_provider=False), | |
| 11:45:24 | tafkamax | RequestGroup(aggregates=[],forbidden_aggregates=set([]),forbidden_traits=set([]),in_tree=None,provider_uuids=[],requester_id='device_profile_0',required_traits=set(['CUSTOM_NVIDIA_26B9']),resources={},use_same_provider=True) | |
| 11:48:56 | chandankumar | I am not sure about the first warning. It might be informative message. | |
| 11:48:57 | tafkamax | Do I need to enable more filters or something? | |
| 11:49:11 | chandankumar | From last error, it does not have any resources. | |
| 11:50:07 | chandankumar | Not sure it is linked with fAILED TO retrive allocation error | |
| 11:50:26 | chandankumar | I will wait for other people to take a look | |
| 11:50:49 | chandankumar | May be resource type would be PGPU or VGPU | |
| 11:51:15 | chandankumar | can you try with PGPU? | |
| 11:51:35 | tafkamax | Yeah, I will try. Also, some commands give this output in CLI: | |
| 11:51:35 | tafkamax | openstack accelerator device enable 4dc9f844-75d4-4e6c-a252-b9986816cc5b | |
| 11:51:35 | tafkamax | 'Proxy' object has no attribute 'enable_device' | |
| 11:51:57 | tafkamax | Is this expected? | |
| 11:57:03 | tafkamax | Do I need to the arq bind thing aswell? | |
| 11:58:31 | chandank` | tafkamax: yes, 'Proxy' object has no attribute 'enable_device' similar error is coming in my env with both enable and disable | |
| 11:59:02 | chandank` | you can open a bug for this on https://bugs.launchpad.net/openstack-cyborg | |
| 12:01:14 | tafkamax | Aww damn | |
| 12:03:08 | tafkamax | openstack accelerator device attribute list | |