| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-17 | |||
| 18:13:12 | _colby | Hey Guys. We upgraded to oenstack yoga recently and now all our vGPU stuff does not work any more. We had to patch Wallaby, but Xena worked without issues. Now on Yoga we can spin up a single vGPU on a host (has 2 x A40 gpu (SR-IOV with Virtual Functions). But no more than 1 vgpu instance. We are specifying enabled_vgpu_types and device_Addresses for each type. Are we doing this incorrectly now with yoga? | |
| 18:14:45 | bauzas | _colby: I need to leave litterally now, but what's the stack when you ask for more | |
| 18:14:47 | bauzas | ? | |
| 18:15:24 | bauzas | I guess you know all the quirks with SRIOV GPUs | |
| 18:15:24 | _colby | bauzas: it says not valid host | |
| 18:15:47 | bauzas | you may have residues that lead to wrong placement decisions | |
| 18:15:48 | _colby | yes just wondering what changed between xena and yoga. Things worked fine before. | |
| 18:16:05 | bauzas | first thing is to doublecheck your inventories and allocations | |
| 18:16:21 | bauzas | and compare them with the existing mdevs and instances | |
| 18:17:27 | bauzas | particularly if you rebooted the host, you may have mediated devices using the wrong GPUs | |
| 18:17:30 | bauzas | or VFs (rather) | |
| 18:20:10 | _colby | yea we have a script that runs and recreates all the VFs on boot. They are the same. Should we still be specifying the device_addresses for the types? | |
| 18:20:25 | _colby | mdevctl types shows all the correct addresses and types we are using | |
| 18:50:51 | _colby | bauzas: I got some more info. My coworker had tested before. I did some tests of my own and Im seeing a different error. One host wont spin up a vgpu at all. Im seeing: Insufficient compute resources: mdev-capable resource is not available. The VFs are all there and correct, the mdectl types lists all the correct addresses. How do I check the placement inventory to verify it matches correctly? | |
| 18:55:25 | bauzas | _colby: sorry, I'm on and off by now (7:55pm here) | |
| 18:55:51 | bauzas | but you can use openstack resource provider list and then openstack resource provider inventory list for knowing | |
| 18:56:26 | bauzas | _colby: see https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#checking-allocations-and-inventories-for-virtual-gpus | |
| 18:56:52 | bauzas | (instead of seeing the GPUs, you would see the VFs) | |
| 18:57:06 | _colby | yea that looks correct too unfortunatly | |
| 18:57:26 | bauzas | _colby: we also have a bug report for SRIOV GPUs, see https://bugs.launchpad.net/nova/+bug/2041519 | |
| 18:57:34 | bauzas | I have a bugfix series for it | |
| 18:58:48 | bauzas | _colby: with SRIOV VFs, you should make sure that all your mdevs are created | |
| 18:59:00 | bauzas | and then you should verify the sysfs | |
| 18:59:19 | _colby | ok Ill read through this bug report. Thanks | |
| 18:59:20 | bauzas | calculate the number of mdevs, look at which VFs they use | |
| 18:59:40 | bauzas | then, look at the available_instances value for the type | |
| 18:59:45 | bauzas | for each of the VFs | |
| 19:00:12 | bauzas | then, please look at how we create the inventories : | |
| 19:00:49 | bauzas | https://github.com/openstack/nova/blob/a72f7eaac78927892b937d451cbacb24a83c05ac/nova/virt/libvirt/driver.py#L8128-L8171 | |
| 19:01:07 | bauzas | we count the number of mdevs that are already created | |
| 19:01:16 | _colby | ok thanks! Appreciate the help. Will dig some more into this today. | |
| 19:01:36 | bauzas | and then we add it with the value of available_instances for the VF related type | |
| 19:02:10 | bauzas | once you create all the mdevs, you should have VGPU inventories only for the VFs that have mdevs | |
| 19:02:32 | bauzas | and then you should delete the other RPs related the VFs that have none of mdevs | |
| 19:02:50 | bauzas | (unfortunately, due to the bug report, we don't remove automatically them, but my series will do it) | |
| 19:04:03 | bauzas | _colby: I'm in the CET timezone but you can in general ping me until 6pm UTC | |
| 23:48:23 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| #openstack-nova - 2024-01-18 | |||
| 00:09:24 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| 02:03:02 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| 02:10:16 | opendevreview | likui proposed openstack/nova master: Add py3.11 in tox file https://review.opendev.org/c/openstack/nova/+/901587 | |
| 02:10:42 | opendevreview | likui proposed openstack/nova master: Update python classifier in setup.cfg https://review.opendev.org/c/openstack/nova/+/901587 | |
| 02:11:55 | tonyb | Any chance I can get some eyes on: https://review.opendev.org/c/openstack/nova/+/905314 ? | |
| 06:50:39 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/904817 | |
| 08:28:06 | bauzas | tonyb: sent to the gate | |
| 09:21:39 | opendevreview | Merged openstack/nova master: Fix test failures with oslo.limit 2.3.0 https://review.opendev.org/c/openstack/nova/+/905314 | |
| 09:23:53 | tonyb | bauzas: thank you | |
| 09:24:20 | tonyb | melwitt: and thank you too for the quick review | |
| 14:37:35 | opendevreview | Merged openstack/nova master: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/905684 | |
| 15:16:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 15:16:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a functest for verifying multiple VGPU allocations https://review.opendev.org/c/openstack/nova/+/845747 | |
| 16:40:37 | opendevreview | Merged openstack/nova master: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/904817 | |
| 16:53:35 | opendevreview | Doug Szumski proposed openstack/nova master: Use dedicated live migration network during pre-migration https://review.opendev.org/c/openstack/nova/+/906053 | |
| 17:16:43 | _colby | bauzas: thanks for your help yesterday. So from what you said do we need to manually create the mdevs now? I was confused on that part. I see all the VF and available_instances for them all. But mdevctl shows no devices and nova does not seem to create them any more. So is manual creation required now? | |
| 17:17:05 | bauzas | (on a meeting now but I'll reply later) | |
| 17:17:10 | opendevreview | Doug Szumski proposed openstack/nova master: Use dedicated live migration network during pre-migration https://review.opendev.org/c/openstack/nova/+/906053 | |
| 17:30:41 | bauzas | _colby: I'll need to disappear shortly, but please explain me more : are you able to see mediated devices in /sys/bus/mdev ? | |
| 17:32:52 | opendevreview | Doug Szumski proposed openstack/nova master: Use dedicated live migration network during pre-migration https://review.opendev.org/c/openstack/nova/+/906053 | |
| 17:38:47 | _colby | bauzas: yes I see 1 here for all the available VFs: cat /sys/class/mdev_bus/*/mdev_supported_types/nvidia-563/available_instances | |
| 17:39:05 | bauzas | so basically all GPU VFs are available | |
| 17:39:13 | bauzas | do you have instances on the host ? | |
| 17:39:25 | _colby | no this is a fresh host Im testing with | |
| 17:39:31 | bauzas | and do you have placement allocations ? | |
| 17:40:50 | bauzas | _colby: re: the bug I mentioned yesterday, once you create enough mdevs equal to the max number of vGPUs the type can create, then all other VFs will start having 0 available instances but placement will continue to say 'sure, you can use them' | |
| 17:41:13 | _colby | yes the resource provider shows all the VF addresses as seperate items and looking at inventory of one I see the expected output: VGPU, allocation_ratio=1.0,min_unit=1,max_unit=1,reserved=0,step_size=1,total=1,used=0 | |
| 17:41:14 | bauzas | that's why I'm recommending to precreate the mediated devices on the VFs you want and then start nova-compute | |
| 17:41:33 | bauzas | oh, you already started nova-computz | |
| 17:41:35 | bauzas | then, | |
| 17:41:46 | bauzas | 1/ create the mdevs by hand (or by mdevctl) | |
| 17:41:53 | bauzas | 2/ stop nova-compute | |
| 17:42:05 | bauzas | 3/ delete the resource providers matching the unused VFs | |
| 17:42:10 | bauzas | 4/ restart nova-compute | |
| 17:42:27 | bauzas | you should then have placement reflecting the correct resources | |
| 17:42:45 | bauzas | I need to leave now | |
| 17:43:08 | _colby | ok I can do that. Seems like a bit of heavy admin work every time we want to reconfig the vGPU slices. Nova does not want to create them automatically any more? | |
| 17:43:55 | bauzas | that's the bug report I mentioned yesterday | |
| 17:44:05 | _colby | but if that will get it working then we can do that | |
| 17:44:14 | bauzas | nova can continue to create the mediated devices | |
| 17:45:08 | bauzas | but as soon as you create the Nth mdev from a GPU (where N equals the max number of vGPUs you can create from a type), then you'll have inconsistencies between placement and sysfs | |
| 17:45:22 | _colby | I see | |
| 17:45:33 | bauzas | a fix is there, we gonna review it this cycle | |
| 17:45:33 | _colby | but in our case Nova does not seem to be creating any mdevs | |
| 17:45:51 | bauzas | I need to go but shouldn't be | |
| 17:46:07 | _colby | no problem thanks | |
| 17:46:29 | bauzas | I can't remember when we added more logs to _allocate_mdevs() but yoga seems recent enough to have those | |
| 17:48:32 | bauzas | https://github.com/openstack/nova/commit/6feb3350b048606297068841e3feba110bb0b0ab | |
| 17:48:46 | bauzas | nah, antelope | |
| 17:49:41 | _colby | I can manually patch our system to add these logs for testing | |
| 18:19:47 | _colby | bauzas: whenever you get back I made more progress: https://pastebin.com/Xv2bvaf7 | |
| 18:20:24 | _colby | its compaining about no available instances on that device but there is: cat /sys/class/mdev_bus/0000\:17\:02.0/mdev_supported_types/nvidia-563/available_instances | |
| 18:22:22 | _colby | also that device is not listed in the config for that type: device_addresses=0000:17:00.4,0000:17:00.5,0000:17:00.6,0000:17:00.7 | |
| 21:51:51 | opendevreview | Merged openstack/nova master: Allow config to support virtiofs (driver) https://review.opendev.org/c/openstack/nova/+/886522 | |
| 21:51:59 | opendevreview | Merged openstack/nova master: Attach Manila shares via virtiofs (db) https://review.opendev.org/c/openstack/nova/+/831193 | |
| #openstack-nova - 2024-01-19 | |||
| 00:36:34 | opendevreview | Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2024.2 Dalmatian https://review.opendev.org/c/openstack/nova-specs/+/906073 | |
| 05:23:47 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.2: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/906088 | |
| 05:23:48 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.2: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/906089 | |
| 05:44:49 | opendevreview | Amit Uniyal proposed openstack/nova master: Disconnecting volume from the compute host https://review.opendev.org/c/openstack/nova/+/877446 | |
| 05:44:49 | opendevreview | Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648 | |
| 05:44:50 | opendevreview | Amit Uniyal proposed openstack/nova master: Removed explicit call to delete attachment https://review.opendev.org/c/openstack/nova/+/891289 | |
| 08:32:35 | bauzas | _colby: I'm here now but let's discuss this when you're back, I have some questions (saw your pastebin) | |
| 09:19:05 | opendevreview | Amit Uniyal proposed openstack/nova stable/2023.2: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/906089 | |