| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-23 | |||
| 10:03:47 | opendevreview | Balazs Gibizer proposed openstack/nova stable/ussuri: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/842956 | |
| 10:35:22 | opendevreview | Merged openstack/nova-specs master: Re-propose spec for ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/835877 | |
| 10:37:13 | opendevreview | Merged openstack/nova-specs master: Re-propose spec for ephemeral encryption for libvirt https://review.opendev.org/c/openstack/nova-specs/+/836075 | |
| 11:23:47 | opendevreview | Balazs Gibizer proposed openstack/nova stable/train: Reproducer unit test for bug 1934094 https://review.opendev.org/c/openstack/nova/+/842962 | |
| 11:23:48 | opendevreview | Balazs Gibizer proposed openstack/nova stable/train: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/842963 | |
| 11:43:39 | songwenping | bauzas: morning, have you used Tesla A100 card, how to use it binding to vm, is there any docs? | |
| 11:50:04 | sean-k-mooney | songwenping: i dont think we have them installed in our lab yet. we were goingt ot be adding some | |
| 11:50:24 | sean-k-mooney | songwenping: i think its more or less the same as the previous generation | |
| 11:51:15 | songwenping | sean-k-mooney: passthrough is same as the previous generation | |
| 11:51:30 | songwenping | but vgpu usage is different. | |
| 11:51:31 | sean-k-mooney | vGPU can be too | |
| 11:51:40 | sean-k-mooney | it depense on if you enabel mig or not | |
| 11:51:57 | sean-k-mooney | form an openstack point of view even with mig enabled the nova config is baically the same | |
| 11:52:16 | sean-k-mooney | we still use mdevs for vGPU in mig mode | |
| 11:52:28 | songwenping | i have enabled mig, but i cannot find the vgpu types. | |
| 11:52:28 | sean-k-mooney | but the mdev is moved to the mig vf | |
| 11:52:54 | sean-k-mooney | songwenping: are you checking the PF or the VFs | |
| 11:53:01 | songwenping | i exec the sirov-manage, but it has no effect. | |
| 11:53:37 | sean-k-mooney | i have not used it but nvidia provide https://github.com/NVIDIA/mig-parted | |
| 11:53:44 | sean-k-mooney | to configre the a100 | |
| 11:55:24 | sean-k-mooney | this would appear to be the set of supported profiles https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html#a100-profiles | |
| 11:56:02 | sean-k-mooney | "sudo nvidia-smi mig -lgip" shoudl list the profiles for you | |
| 11:56:57 | songwenping | we also need create mdev then attach to the guest? | |
| 11:57:27 | sean-k-mooney | the mdevs i think will still be created by nova | |
| 11:57:40 | sean-k-mooney | but you need to lookup the supproted mdev type in /sys | |
| 11:58:23 | sean-k-mooney | so with mig mode there are 2 steps | |
| 11:58:29 | songwenping | yeah, to generate uuid under devices dir by nova | |
| 11:58:31 | sean-k-mooney | partition the gpu wiht mig using the mig profiles | |
| 11:58:57 | sean-k-mooney | then lookup the mig vf pci adress and mdev type and update nova.conf with that | |
| 11:59:45 | songwenping | ok, i'll try to check. | |
| 11:59:47 | sean-k-mooney | then nova will determin the numer of that mdev type that can be created and create an inventory of thaty type inplacement. when you schedule a vm we shoudl create the mdev automaticlly | |
| 12:00:54 | songwenping | right, thanks seak-k-mooney. | |
| 12:01:17 | songwenping | sorry, sean-k-mooney. | |
| 12:02:13 | sean-k-mooney | hehe given i have missspelled my own name in code comment no need to appologies for a typo :) | |
| 12:02:40 | sean-k-mooney | i was just lookign for our downstream bz for a100 but i cant find it readialy | |
| 12:03:16 | sean-k-mooney | i think bauzas wrote up how to do this at some point. they had a load of an 100 to test breifly but we dont have one dedicated for our use currently | |
| 12:05:40 | sean-k-mooney | ah i found it but its private | |
| 12:08:11 | songwenping | look forward to seeing soon. | |
| 12:09:01 | sean-k-mooney | it looks like they were using /usr/lib/nvidia/sriov-manage -e 0000:04:00.0 | |
| 12:09:05 | sean-k-mooney | to allocate teh VFs | |
| 12:09:22 | songwenping | i also use this command | |
| 12:09:22 | sean-k-mooney | and then if you used ll /sys/class/mdev_bus/* | |
| 12:09:25 | sean-k-mooney | you woudl see the vfs | |
| 12:09:30 | sean-k-mooney | are mdev capable | |
| 12:10:10 | sean-k-mooney | they saw the same behvior with both t4(with updated firmware) and a100 | |
| 12:10:13 | songwenping | but get nothing, maybe my env has something problem | |
| 12:10:31 | sean-k-mooney | then they create the mig instance "nvidia-smi mig -cgi 2g.10gb,2g.10gb,2g.10gb -C" | |
| 12:10:48 | sean-k-mooney | and the vf had inventories | |
| 12:10:50 | sean-k-mooney | cat /sys/class/mdev_bus/0000\:04\:01.0/mdev_supported_types/nvidia-475/available_instances | |
| 12:11:07 | sean-k-mooney | so they added | |
| 12:11:09 | sean-k-mooney | [devices] | |
| 12:11:11 | sean-k-mooney | enabled_vgpu_types = nvidia-475,nvidia-1 | |
| 12:11:13 | sean-k-mooney | [vgpu_nvidia-475] | |
| 12:11:15 | sean-k-mooney | device_addresses = 0000:04:00.4,0000:04:00.5,0000:04:00.6 | |
| 12:11:17 | sean-k-mooney | [vgpu_nvidia-1] | |
| 12:11:19 | sean-k-mooney | device_addresses = 0000:00:00.1 | |
| 12:11:21 | sean-k-mooney | to the nova.conf | |
| 12:12:26 | sean-k-mooney | songwenping: the steps bauzas did were | |
| 12:12:45 | sean-k-mooney | nvidia-smi -i 0 -mig 1 | |
| 12:12:47 | songwenping | thanks sean-k-mooney | |
| 12:12:49 | sean-k-mooney | to enable mig | |
| 12:12:54 | sean-k-mooney | which they checked with | |
| 12:13:02 | sean-k-mooney | nvidia-smi -i 0 --query-gpu=pci.bus_id,mig.mode.current --format=csv | |
| 12:13:23 | sean-k-mooney | then "/usr/lib/nvidia/sriov-manage -e 00:04:0000.0" to enable the vfs | |
| 12:13:45 | sean-k-mooney | and then you can use " ll /sys/class/mdev_bus/" | |
| 12:13:54 | sean-k-mooney | to verify that hte device are capable of hosting mdevs | |
| 12:14:23 | sean-k-mooney | then they configugred vgpu instance per mig instance | |
| 12:14:25 | sean-k-mooney | nvidia-smi mig -cgi 2g.10gb,2g.10gb,2g.10gb -C | |
| 12:14:54 | sean-k-mooney | and after that they had 1 mdev inventory per vf | |
| 12:15:02 | sean-k-mooney | which can be checked with "cat /sys/class/mdev_bus/0000\:04\:01.0/mdev_supported_types/nvidia-475/available_instances" | |
| 12:15:19 | sean-k-mooney | songwenping: hopefully that helps | |
| 12:15:44 | songwenping | thanks, this is useful. | |
| 12:16:41 | sean-k-mooney | our downstream docs bz to write this up is still pending | |
| 12:34:58 | sean-k-mooney | dansmith: those keystone select number really are an order of magnitude higher then everything else https://paste.opendev.org/show/bsG9exlvGqaNZrbhEnJp/ | |
| 12:59:49 | gibi | melwitt: can I help somehow to convince you to +2 https://review.opendev.org/c/openstack/nova-specs/+/791047 ? :) | |
| 13:03:04 | sean-k-mooney | gibi: she may have been leaving that to stephenfin to +2w but if melwitt is happy withthe proposal i think stephenfin will be happy that you have incoperated there feedback with regards to the dynmaic alloctaiton fo VF/PFs | |
| 13:05:24 | gibi | ahh OK. | |
| 13:05:42 | gibi | stephenfin: if you have time, I think https://review.opendev.org/c/openstack/nova-specs/+/791047 is ready for a +2+A :) | |
| 13:08:37 | gibi | I'm not sure I understand what the sitting down part means ^^ | |
| 13:09:06 | sean-k-mooney | preparing my self for the shock of us agreeing to actully track pci device in placement after like what 5 years | |
| 13:09:18 | gibi | ahh :d | |
| 13:09:19 | gibi | :D | |
| 13:09:34 | sean-k-mooney | placement was still in nova wehn we started this converastion and nested resouce providers did not exist | |
| 13:10:07 | gibi | yeah it is a long journey | |
| 13:10:14 | gibi | and it is not over yet :) | |
| 13:32:54 | opendevreview | ribaudr proposed openstack/nova-specs master: libvirt: Allow Manila shares to be directly attached to instances https://review.opendev.org/c/openstack/nova-specs/+/833669 | |
| 13:49:53 | dansmith | sean-k-mooney: I know.. something has to be wrong right? | |
| 13:50:44 | sean-k-mooney | presumable. i would expect keysone to be higher then the rest but not 10x | |
| 13:51:01 | sean-k-mooney | and that 10x neutron which is also higher then the rest | |
| 13:51:38 | dansmith | right | |
| 13:52:11 | dansmith | neutron goes over keystone quickly during tempest tho | |
| 14:11:03 | sean-k-mooney | dansmith: while your here we use 2 allocation for live migration correct? one is held by the instnace an the other by the migration uuid? | |
| 14:11:19 | dansmith | yeah | |
| 14:11:39 | sean-k-mooney | do you remember when that change was made it was 5 or more cycles ago right | |
| 14:11:43 | sean-k-mooney | as in its not recent | |
| 14:11:50 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1975490 just in the context of that | |
| 14:11:58 | dansmith | oh jeez, no idea, but I'm sure it was 5+ ago | |
| 14:12:32 | sean-k-mooney | right it was well before train with the new vcpu code | |
| 14:13:23 | sean-k-mooney | they did not say what version of openstack they were using but im guessign it was quite an old version if they are seeing placment check the source allocations like they describe | |
| 14:14:58 | sean-k-mooney | i dont think any supported branch including the ones in EM still use a signel allocation for both hosts | |