| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-16 | |||
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 20:19:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 20:19:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 20:38:01 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:44:41 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 20:51:58 | opendevreview | Merged openstack/python-novaclient master: Bump hacking https://review.opendev.org/c/openstack/python-novaclient/+/905707 | |
| 21:06:20 | opendevreview | Merged openstack/placement master: Bump hacking https://review.opendev.org/c/openstack/placement/+/905706 | |
| #openstack-nova - 2024-01-17 | |||
| 02:00:49 | opendevreview | sean mooney proposed openstack/nova master: enable zswap in nova ci jobs. https://review.opendev.org/c/openstack/nova/+/905791 | |
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP Suppport migration with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/905512 | |
| 02:56:31 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: make |
|
| 02:56:32 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 03:19:58 | songwenping | hi team, the latest nova code raise exception for 'nova list' with the error info TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig', what's wrong? | |
| 04:48:56 | opendevreview | Amit Uniyal proposed openstack/nova master: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/905684 | |
| 04:48:57 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes bug 2048184 https://review.opendev.org/c/openstack/nova/+/904817 | |
| 05:20:27 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes bug 2048184 https://review.opendev.org/c/openstack/nova/+/904817 | |
| 06:49:17 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 10:27:34 | bauzas | humpf, upgraded to F39 and now tox and python3.12 are creating me problems :( | |
| 10:27:59 | bauzas | tox==4.12.1 | |
| 10:30:11 | bauzas | good news is that I can still run functional tests if I target by the python version (like -e functional-py39) | |
| 10:42:04 | bauzas | ok, behavioral change with tox, no worriezsq | |
| 11:15:06 | kashyap | Good to know; /me is still to ugprade to F39. Was planning to do this week | |
| 12:53:33 | opendevreview | Fabian Wiesel proposed openstack/nova master: Add more password generation options https://review.opendev.org/c/openstack/nova/+/865669 | |
| 14:47:41 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 14:47:41 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 14:47:42 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 14:47:42 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 14:55:18 | opendevreview | Sylvain Bauza proposed openstack/nova master: Augment the LibvirtLiveMigrateData object https://review.opendev.org/c/openstack/nova/+/904175 | |
| 14:55:18 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 14:55:19 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 14:55:19 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 14:55:20 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 15:17:15 | opendevreview | Hervé Beraud proposed openstack/nova master: bump eventlet to latest version that support python 3.12 https://review.opendev.org/c/openstack/nova/+/905928 | |
| 15:17:15 | opendevreview | Hervé Beraud proposed openstack/nova master: bump eventlet to latest version that support python 3.12 https://review.opendev.org/c/openstack/nova/+/905928 | |
| 17:35:15 | bauzas | argh, I'm hitting py312 issues on my laptop and I can't generate a relnote as we don't support venv target with py versions | |
| 18:09:29 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 18:09:29 | opendevreview | Sylvain Bauza proposed openstack/nova master: Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 18:13:12 | _colby | Hey Guys. We upgraded to oenstack yoga recently and now all our vGPU stuff does not work any more. We had to patch Wallaby, but Xena worked without issues. Now on Yoga we can spin up a single vGPU on a host (has 2 x A40 gpu (SR-IOV with Virtual Functions). But no more than 1 vgpu instance. We are specifying enabled_vgpu_types and device_Addresses for each type. Are we doing this incorrectly now with yoga? | |
| 18:14:45 | bauzas | _colby: I need to leave litterally now, but what's the stack when you ask for more | |
| 18:14:47 | bauzas | ? | |
| 18:15:24 | _colby | bauzas: it says not valid host | |
| 18:15:24 | bauzas | I guess you know all the quirks with SRIOV GPUs | |
| 18:15:47 | bauzas | you may have residues that lead to wrong placement decisions | |
| 18:15:48 | _colby | yes just wondering what changed between xena and yoga. Things worked fine before. | |
| 18:16:05 | bauzas | first thing is to doublecheck your inventories and allocations | |
| 18:16:21 | bauzas | and compare them with the existing mdevs and instances | |
| 18:17:27 | bauzas | particularly if you rebooted the host, you may have mediated devices using the wrong GPUs | |
| 18:17:30 | bauzas | or VFs (rather) | |
| 18:20:10 | _colby | yea we have a script that runs and recreates all the VFs on boot. They are the same. Should we still be specifying the device_addresses for the types? | |
| 18:20:25 | _colby | mdevctl types shows all the correct addresses and types we are using | |
| 18:50:51 | _colby | bauzas: I got some more info. My coworker had tested before. I did some tests of my own and Im seeing a different error. One host wont spin up a vgpu at all. Im seeing: Insufficient compute resources: mdev-capable resource is not available. The VFs are all there and correct, the mdectl types lists all the correct addresses. How do I check the placement inventory to verify it matches correctly? | |
| 18:55:25 | bauzas | _colby: sorry, I'm on and off by now (7:55pm here) | |
| 18:55:51 | bauzas | but you can use openstack resource provider list and then openstack resource provider inventory list for knowing | |
| 18:56:26 | bauzas | _colby: see https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#checking-allocations-and-inventories-for-virtual-gpus | |
| 18:56:52 | bauzas | (instead of seeing the GPUs, you would see the VFs) | |
| 18:57:06 | _colby | yea that looks correct too unfortunatly | |
| 18:57:26 | bauzas | _colby: we also have a bug report for SRIOV GPUs, see https://bugs.launchpad.net/nova/+bug/2041519 | |
| 18:57:34 | bauzas | I have a bugfix series for it | |
| 18:58:48 | bauzas | _colby: with SRIOV VFs, you should make sure that all your mdevs are created | |
| 18:59:00 | bauzas | and then you should verify the sysfs | |
| 18:59:19 | _colby | ok Ill read through this bug report. Thanks | |
| 18:59:20 | bauzas | calculate the number of mdevs, look at which VFs they use | |
| 18:59:40 | bauzas | then, look at the available_instances value for the type | |
| 18:59:45 | bauzas | for each of the VFs | |
| 19:00:12 | bauzas | then, please look at how we create the inventories : | |
| 19:00:49 | bauzas | https://github.com/openstack/nova/blob/a72f7eaac78927892b937d451cbacb24a83c05ac/nova/virt/libvirt/driver.py#L8128-L8171 | |
| 19:01:07 | bauzas | we count the number of mdevs that are already created | |
| 19:01:16 | _colby | ok thanks! Appreciate the help. Will dig some more into this today. | |
| 19:01:36 | bauzas | and then we add it with the value of available_instances for the VF related type | |
| 19:02:10 | bauzas | once you create all the mdevs, you should have VGPU inventories only for the VFs that have mdevs | |
| 19:02:32 | bauzas | and then you should delete the other RPs related the VFs that have none of mdevs | |
| 19:02:50 | bauzas | (unfortunately, due to the bug report, we don't remove automatically them, but my series will do it) | |
| 19:04:03 | bauzas | _colby: I'm in the CET timezone but you can in general ping me until 6pm UTC | |
| 23:48:23 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| #openstack-nova - 2024-01-18 | |||
| 00:09:24 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| 02:03:02 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| 02:10:16 | opendevreview | likui proposed openstack/nova master: Add py3.11 in tox file https://review.opendev.org/c/openstack/nova/+/901587 | |
| 02:10:42 | opendevreview | likui proposed openstack/nova master: Update python classifier in setup.cfg https://review.opendev.org/c/openstack/nova/+/901587 | |
| 02:11:55 | tonyb | Any chance I can get some eyes on: https://review.opendev.org/c/openstack/nova/+/905314 ? | |
| 06:50:39 | opendevreview | Amit Uniyal proposed openstack/nova master: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/904817 | |
| 08:28:06 | bauzas | tonyb: sent to the gate | |
| 09:21:39 | opendevreview | Merged openstack/nova master: Fix test failures with oslo.limit 2.3.0 https://review.opendev.org/c/openstack/nova/+/905314 | |
| 09:23:53 | tonyb | bauzas: thank you | |
| 09:24:20 | tonyb | melwitt: and thank you too for the quick review | |
| 14:37:35 | opendevreview | Merged openstack/nova master: Updates glance fixture for create image https://review.opendev.org/c/openstack/nova/+/905684 | |
| 15:16:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a functest for verifying multiple VGPU allocations https://review.opendev.org/c/openstack/nova/+/845747 | |
| 15:16:37 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 16:40:37 | opendevreview | Merged openstack/nova master: Fixes: bfv vm reboot ends up in an error state. https://review.opendev.org/c/openstack/nova/+/904817 | |
| 16:53:35 | opendevreview | Doug Szumski proposed openstack/nova master: Use dedicated live migration network during pre-migration https://review.opendev.org/c/openstack/nova/+/906053 | |
| 17:16:43 | _colby | bauzas: thanks for your help yesterday. So from what you said do we need to manually create the mdevs now? I was confused on that part. I see all the VF and available_instances for them all. But mdevctl shows no devices and nova does not seem to create them any more. So is manual creation required now? | |
| 17:17:05 | bauzas | (on a meeting now but I'll reply later) | |
| 17:17:10 | opendevreview | Doug Szumski proposed openstack/nova master: Use dedicated live migration network during pre-migration https://review.opendev.org/c/openstack/nova/+/906053 | |
| 17:30:41 | bauzas | _colby: I'll need to disappear shortly, but please explain me more : are you able to see mediated devices in /sys/bus/mdev ? | |
| 17:32:52 | opendevreview | Doug Szumski proposed openstack/nova master: Use dedicated live migration network during pre-migration https://review.opendev.org/c/openstack/nova/+/906053 | |
| 17:38:47 | _colby | bauzas: yes I see 1 here for all the available VFs: cat /sys/class/mdev_bus/*/mdev_supported_types/nvidia-563/available_instances | |
| 17:39:05 | bauzas | so basically all GPU VFs are available | |
| 17:39:13 | bauzas | do you have instances on the host ? | |
| 17:39:25 | _colby | no this is a fresh host Im testing with | |
| 17:39:31 | bauzas | and do you have placement allocations ? | |
| 17:40:50 | bauzas | _colby: re: the bug I mentioned yesterday, once you create enough mdevs equal to the max number of vGPUs the type can create, then all other VFs will start having 0 available instances but placement will continue to say 'sure, you can use them' | |
| 17:41:13 | _colby | yes the resource provider shows all the VF addresses as seperate items and looking at inventory of one I see the expected output: VGPU, allocation_ratio=1.0,min_unit=1,max_unit=1,reserved=0,step_size=1,total=1,used=0 | |