| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2026-05-06 | |||
| 14:04:19 | sean-k-mooney | https://github.com/openstack/nova/commit/26c41eccade6412f61f9a8721d853b545061adcc https://github.com/openstack/nova/commit/284ea72e96604bdf16d1c5c4db47247334841b2f https://github.com/openstack/nova/commit/0208be629c3853863bcd49b8bdbe2b9889b85012 https://github.com/openstack/nova/commit/f37cdf0c4182103ad81dbf39188ff39955da3850 | |
| 14:04:38 | sean-k-mooney | those are the nova patches related to the isseu reproted in https://bugs.launchpad.net/openstack-cyborg/+bug/2017513 | |
| 14:05:53 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1633120 https://bugs.launchpad.net/nova/+bug/1969496 and https://bugs.launchpad.net/nova/+bug/2115905 are the releated nova bugs we had | |
| 14:07:31 | sean-k-mooney | the tl;dr is if a device is refence by a ARQ and that device is not in the whitelist or viaabel on teh host anymore we cannot remove the device form the db or placmeent until that ARQ is deleted and we shoudl not do that automaticlly | |
| 14:07:46 | sean-k-mooney | the admin need to move or delete the vm or readd the device | |
| 14:08:34 | sean-k-mooney | we shoudl complain very very loadly in teh logs when the compute agent start up in a miscondifured state but we dont geenrally want to make that an agent startup failure as that a potical dos vector if we do | |
| 14:23:08 | jgilaber | I'm not sure from the bug report if it actually crashes the agent or it just logs the traceback in an ugly way | |
| 14:23:23 | jgilaber | Chandan reproduced the error, we can ask him later/tomorrow | |
| 14:30:10 | sean-k-mooney | in either case thsi shoudl be handeled gracefully and we shoudl not attepmt to delete the resouce provider | |
| 14:30:29 | sean-k-mooney | trying to delete it if it has allcoation agaisnt it is a bug in our internal logic | |
| 14:30:53 | sean-k-mooney | the protections in placment are really the last line of defence | |
| #openstack-cyborg - 2026-05-07 | |||
| 10:32:22 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 10:40:37 | tafkamax | Hi I just deployed cyborg and are planning to use it to pass PGPU-s to vm-s for now. | |
| 10:41:06 | tafkamax | I just talked in kolla chat aswell, as it is active and there are is some knowledge there too. | |
| 10:42:18 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 10:42:23 | tafkamax | The GPU shows up in `openstack accelerator device show UUID`... (full message at <https://matrix.org/oftc/media/v1/media/download/AW1EbeU-tbWQQQrjpuO6aBPema1ZFNxz1H-Oe8zT3IzpZWc2PNyPKxAaRLOwQuK9lv5coKnS2iHnahg2PqYfBgFCeeSiEWiwAG1hdHJpeC5vcmcvbmZOem9Rc2Zqak9zb2JialhsU0RSWmNB>) | |
| 10:42:35 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Move job definitions to cyborg repo https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/987621 | |
| 10:42:49 | tafkamax | the traits are pretty empty for it though:... (full message at <https://matrix.org/oftc/media/v1/media/download/AUewwyd-zCY46PWfpF0JsBLctL5NmRnri3j9bK6462IJYTg7QZOmF_RjeTDUs9FvS2nB6diGWvBU9MVBF2J4ynFCeeSiF-zAAG1hdHJpeC5vcmcvZ0xUUWRHYUNOdHhzVlp0RENUU2tsY2xT>) | |
| 10:43:44 | tafkamax | I also tried this accelerator profile... (full message at <https://matrix.org/oftc/media/v1/media/download/ARQew_Tex2aH3YGmBZDRiWwCFloG5Cdc0UaMQ5z9BfAuBKNfoNRQyiIb9zs5J0HZola9q9RhLGjJY1Y3Jb_PGcVCeeSiJUsQAG1hdHJpeC5vcmcvUVdVeEtqaUNXeXBZT2xOd3dnSmZ4c0hv>) | |
| 10:43:54 | tafkamax | Any thoughts? | |
| 11:01:57 | jgilaber | tafkamax, hi! the traits look correct to me. Was the flavor created correctly? | |
| 11:03:18 | tafkamax | openstack flavor show 6943fd8e-08ea-4541-bc11-a167f270e98f... (full message at <https://matrix.org/oftc/media/v1/media/download/AXUWetSMAsXVtnlrnt-ox0XHF4gtEKxjUQ1zhRXjy7dCvygSAi7WqnZJr4jUfNLe3gDhYf-3t8FlaROgiijBg15CeeSjQ8dQAG1hdHJpeC5vcmcvZnZ3ZHdiZ0tkbUhUUnluTUxrRWtJUkpt>) | |
| 11:05:55 | opendevreview | Joan Gilabert proposed openstack/cyborg master: Move cyborg-tempest job definitions to cyborg https://review.opendev.org/c/openstack/cyborg/+/987620 | |
| 11:05:58 | opendevreview | Joan Gilabert proposed openstack/cyborg-tempest-plugin master: Move job definitions to cyborg repo https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/987621 | |
| 11:11:54 | jgilaber | is there any more detail in the nova logs? it looks like placement might be reporting that there are no available gpus | |
| 11:12:31 | jgilaber | could maybe nova be configured to passthrough the gpu as well? | |
| 11:13:44 | tafkamax | We havent explicilty configured nova for that. | |
| 11:13:59 | jgilaber | ack, thanks, can you also check what 'openstack accelerator arq list' reports? | |
| 11:14:13 | tafkamax | kolla-ansible 2025.1 deployment type | |
| 11:15:29 | tafkamax | hmm empty | |
| 11:15:58 | tafkamax | oh I need to do that beforehand? | |
| 11:17:47 | chandankumar | Do we need to create the device profile with resource class like this https://paste.openstack.org/raw/bJEmpgye27s5h4tEMlTf/ | |
| 11:18:03 | chandankumar | for pci and fake fpga device, we pass resource class | |
| 11:18:27 | tafkamax | Like PGPU? | |
| 11:20:40 | chandankumar | sorry it should be GPU | |
| 11:21:48 | tafkamax | ok, thanks, so that comes from openstack accelerator device show command | |
| 11:29:36 | chandankumar | https://github.com/openstack/cyborg/blob/f111946df6713aa64efa29dd025d47839241c529/cyborg/common/constants.py#L115 | |
| 11:29:45 | chandankumar | "PGPU": orc.PGPU, | |
| 11:29:45 | chandankumar | "VGPU": orc.VGPU, | |
| 11:29:52 | chandankumar | pgpu - for physcial gou | |
| 11:30:13 | chandankumar | pgpu - for physcial gpu and vgpu - for virtual gpu | |
| 11:30:57 | chandankumar | Cyborg also have GPU. | |
| 11:41:45 | tafkamax | I have this in log: 2026-05-07 14:39:05.152 7 WARNING cyborg.accelerator.drivers.gpu.nvidia.sysinfo [-] Unable to load vGPU_type from [gpu_devices] Ensure "enabled_vgpu_types" is set if the gpuis virtualized. but I dont have vgpu enabled. Is this just informational? | |
| 11:45:24 | tafkamax | from nova scheduler then: 2026-05-07 14:40:05.475 1087 ERROR nova.scheduler.client.report [req-3a504845-350b-4532-a33d-f0334e44db4c req-f8f77ce8-89c9-42c9-90e5-a9e7d6eaaf33 204c13dfae0b4214ae00b15a95a5d180 87aa79e7272a4f9b9e66e4582fb28c93 - - default default] Failed to retrieve allocation candidates from placement API for filters: | |
| 11:45:24 | tafkamax | RequestGroup(aggregates=[],forbidden_aggregates=set([]),forbidden_traits=set([]),in_tree=8b9f6c21-0c4a-458a-8541-d481921c6d08,provider_uuids=[],requester_id=None,required_traits=set([]),resources={MEMORY_MB=16384,VCPU=8},use_same_provider=False), | |
| 11:45:24 | tafkamax | RequestGroup(aggregates=[],forbidden_aggregates=set([]),forbidden_traits=set([]),in_tree=None,provider_uuids=[],requester_id='device_profile_0',required_traits=set(['CUSTOM_NVIDIA_26B9']),resources={},use_same_provider=True) | |
| 11:48:56 | chandankumar | I am not sure about the first warning. It might be informative message. | |
| 11:48:57 | tafkamax | Do I need to enable more filters or something? | |
| 11:49:11 | chandankumar | From last error, it does not have any resources. | |
| 11:50:07 | chandankumar | Not sure it is linked with fAILED TO retrive allocation error | |
| 11:50:26 | chandankumar | I will wait for other people to take a look | |
| 11:50:49 | chandankumar | May be resource type would be PGPU or VGPU | |
| 11:51:15 | chandankumar | can you try with PGPU? | |
| 11:51:35 | tafkamax | Yeah, I will try. Also, some commands give this output in CLI: | |
| 11:51:35 | tafkamax | openstack accelerator device enable 4dc9f844-75d4-4e6c-a252-b9986816cc5b | |
| 11:51:35 | tafkamax | 'Proxy' object has no attribute 'enable_device' | |
| 11:51:57 | tafkamax | Is this expected? | |
| 11:57:03 | tafkamax | Do I need to the arq bind thing aswell? | |
| 11:58:31 | chandank` | tafkamax: yes, 'Proxy' object has no attribute 'enable_device' similar error is coming in my env with both enable and disable | |
| 11:59:02 | chandank` | you can open a bug for this on https://bugs.launchpad.net/openstack-cyborg | |
| 12:01:14 | tafkamax | Aww damn | |
| 12:03:08 | tafkamax | openstack accelerator device attribute list | |
| 12:03:08 | tafkamax | 'Proxy' object has no attribute 'attributes' | |
| 12:03:11 | tafkamax | this also return the same | |
| 12:03:43 | tafkamax | https://bugs.launchpad.net/openstack-cyborg/+bug/2151792 | |
| 12:04:44 | chandank` | https://paste.openstack.org/raw/bRpUV3OrPTrGUImCllzA/ - opentack accelerator device attribute list output from master devstack vm | |
| 12:05:30 | tafkamax | Could it be openstacksdk or some client lib version? | |
| 12:06:28 | tafkamax | I am using from 2025.1 UC:... (full message at <https://matrix.org/oftc/media/v1/media/download/AZYa4b2-0u7aEhweDKgZL_Hoq4YysFkqdAt284_KUhwWGucPCZc6Mi3CuUP0OQA3sVMa9LPSYo5LiskzetFd6NFCeeSm4TQgAG1hdHJpeC5vcmcvRmVaa0JwUENuSWlhUFRjdm5OUm5Bc3h2>) | |
| 12:10:19 | tafkamax | Accelerator client initialized using OpenStackSDK: <openstack.accelerator.v2._proxy.Proxy object at 0x7f1aad7e4dd0> | |
| 12:10:19 | tafkamax | 'Proxy' object has no attribute 'attributes' | |
| 12:10:20 | tafkamax | using -vvv | |
| 12:12:00 | chandank` | openstacksdk 4.11.0 is used in master. | |
| 12:12:32 | chandank` | Can you also add openstacksdk and python-cyborgclient version in the bug? If anyone fixes it, we can backport it back | |
| 12:14:22 | tafkamax | edited | |
| 12:14:47 | tafkamax | 2025.2 UC worked for attribute list | |
| 12:15:18 | chandank` | we are missing some backport then | |
| 12:16:07 | tafkamax | but the enable command seems to be not working indeed | |
| 12:20:44 | tafkamax | https://review.opendev.org/c/openstack/openstacksdk/+/883238?usp=search | |
| 12:20:49 | tafkamax | does this need to be backported? | |
| 12:21:01 | tafkamax | its april 4 2025, so after 2025.1 ? | |
| 12:24:37 | jgilaber | yes that commit is missing in 2025.1 https://github.com/openstack/openstacksdk/tree/stable/2025.1/openstack/accelerator | |
| 12:25:54 | chandank` | We have a upgrade job, let me push one patch to get the error in CI | |
| 12:27:00 | tafkamax | But regarding the enable/disable that just seems to be missing? | |
| 12:27:21 | tafkamax | E.g. this needs to be looked in the cyborg API spec and implemented i presume | |
| 12:30:09 | tafkamax | docs.openstack.org/api-ref/accelerator/#enable-a-device ? | |
| 12:31:02 | jgilaber | from a quick glance looks like the controller for that API exists https://github.com/openstack/cyborg/blob/79384661ce73984d2eef05dbee800507d36e997c/cyborg/api/controllers/v2/devices.py#L173 | |
| 12:31:05 | chandank` | yes, it is implemented in cyborg side | |
| 12:31:16 | chandank` | something is misisng or broken on cyborgclient side | |
| 12:33:33 | tafkamax | oh okay | |
| 12:34:09 | tafkamax | So: https://github.com/openstack/python-cyborgclient/blob/master/cyborgclient/osc/v2/device.py#L141 | |
| 12:40:34 | sean-k-mooney | ya its a know gap | |
| 12:40:45 | sean-k-mooney | there are a few issues witht eh current cli | |
| 12:40:55 | sean-k-mooney | and the way its usign the sdk | |
| 12:41:27 | sean-k-mooney | those are thign we plan to fix over this cycle now that we are trying to more activlly maintian cyborg again | |
| 12:42:26 | sean-k-mooney | tafkamax: thanks for filling the bug it will help to have a backlog orf the sepcific brakages | |
| 12:43:38 | sean-k-mooney | chandank`: P in PGPU stands for phsycial becasue its only used for the physical function but this is somethign we will likely evolved in newer drivers | |
| 12:43:56 | sean-k-mooney | the type in the device list and the resouce class are not expected to be the same | |
| 12:47:31 | tafkamax | aha okay, so I should always look for attribute list, when creating a profile | |
| 12:51:14 | tafkamax | Okay. So I need to enable the device actually for it to be able to "bind" to an VM? | |
| 12:52:05 | tafkamax | Hmm I will try to use an API call for enable then. | |