| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-23 | |||
| 12:02:13 | sean-k-mooney | hehe given i have missspelled my own name in code comment no need to appologies for a typo :) | |
| 12:02:40 | sean-k-mooney | i was just lookign for our downstream bz for a100 but i cant find it readialy | |
| 12:03:16 | sean-k-mooney | i think bauzas wrote up how to do this at some point. they had a load of an 100 to test breifly but we dont have one dedicated for our use currently | |
| 12:05:40 | sean-k-mooney | ah i found it but its private | |
| 12:08:11 | songwenping | look forward to seeing soon. | |
| 12:09:01 | sean-k-mooney | it looks like they were using /usr/lib/nvidia/sriov-manage -e 0000:04:00.0 | |
| 12:09:05 | sean-k-mooney | to allocate teh VFs | |
| 12:09:22 | sean-k-mooney | and then if you used ll /sys/class/mdev_bus/* | |
| 12:09:22 | songwenping | i also use this command | |
| 12:09:25 | sean-k-mooney | you woudl see the vfs | |
| 12:09:30 | sean-k-mooney | are mdev capable | |
| 12:10:10 | sean-k-mooney | they saw the same behvior with both t4(with updated firmware) and a100 | |
| 12:10:13 | songwenping | but get nothing, maybe my env has something problem | |
| 12:10:31 | sean-k-mooney | then they create the mig instance "nvidia-smi mig -cgi 2g.10gb,2g.10gb,2g.10gb -C" | |
| 12:10:48 | sean-k-mooney | and the vf had inventories | |
| 12:10:50 | sean-k-mooney | cat /sys/class/mdev_bus/0000\:04\:01.0/mdev_supported_types/nvidia-475/available_instances | |
| 12:11:07 | sean-k-mooney | so they added | |
| 12:11:09 | sean-k-mooney | [devices] | |
| 12:11:11 | sean-k-mooney | enabled_vgpu_types = nvidia-475,nvidia-1 | |
| 12:11:13 | sean-k-mooney | [vgpu_nvidia-475] | |
| 12:11:15 | sean-k-mooney | device_addresses = 0000:04:00.4,0000:04:00.5,0000:04:00.6 | |
| 12:11:17 | sean-k-mooney | [vgpu_nvidia-1] | |
| 12:11:19 | sean-k-mooney | device_addresses = 0000:00:00.1 | |
| 12:11:21 | sean-k-mooney | to the nova.conf | |
| 12:12:26 | sean-k-mooney | songwenping: the steps bauzas did were | |
| 12:12:45 | sean-k-mooney | nvidia-smi -i 0 -mig 1 | |
| 12:12:47 | songwenping | thanks sean-k-mooney | |
| 12:12:49 | sean-k-mooney | to enable mig | |
| 12:12:54 | sean-k-mooney | which they checked with | |
| 12:13:02 | sean-k-mooney | nvidia-smi -i 0 --query-gpu=pci.bus_id,mig.mode.current --format=csv | |
| 12:13:23 | sean-k-mooney | then "/usr/lib/nvidia/sriov-manage -e 00:04:0000.0" to enable the vfs | |
| 12:13:45 | sean-k-mooney | and then you can use " ll /sys/class/mdev_bus/" | |
| 12:13:54 | sean-k-mooney | to verify that hte device are capable of hosting mdevs | |
| 12:14:23 | sean-k-mooney | then they configugred vgpu instance per mig instance | |
| 12:14:25 | sean-k-mooney | nvidia-smi mig -cgi 2g.10gb,2g.10gb,2g.10gb -C | |
| 12:14:54 | sean-k-mooney | and after that they had 1 mdev inventory per vf | |
| 12:15:02 | sean-k-mooney | which can be checked with "cat /sys/class/mdev_bus/0000\:04\:01.0/mdev_supported_types/nvidia-475/available_instances" | |
| 12:15:19 | sean-k-mooney | songwenping: hopefully that helps | |
| 12:15:44 | songwenping | thanks, this is useful. | |
| 12:16:41 | sean-k-mooney | our downstream docs bz to write this up is still pending | |
| 12:34:58 | sean-k-mooney | dansmith: those keystone select number really are an order of magnitude higher then everything else https://paste.opendev.org/show/bsG9exlvGqaNZrbhEnJp/ | |
| 12:59:49 | gibi | melwitt: can I help somehow to convince you to +2 https://review.opendev.org/c/openstack/nova-specs/+/791047 ? :) | |
| 13:03:04 | sean-k-mooney | gibi: she may have been leaving that to stephenfin to +2w but if melwitt is happy withthe proposal i think stephenfin will be happy that you have incoperated there feedback with regards to the dynmaic alloctaiton fo VF/PFs | |
| 13:05:24 | gibi | ahh OK. | |
| 13:05:42 | gibi | stephenfin: if you have time, I think https://review.opendev.org/c/openstack/nova-specs/+/791047 is ready for a +2+A :) | |
| 13:08:37 | gibi | I'm not sure I understand what the sitting down part means ^^ | |
| 13:09:06 | sean-k-mooney | preparing my self for the shock of us agreeing to actully track pci device in placement after like what 5 years | |
| 13:09:18 | gibi | ahh :d | |
| 13:09:19 | gibi | :D | |
| 13:09:34 | sean-k-mooney | placement was still in nova wehn we started this converastion and nested resouce providers did not exist | |
| 13:10:07 | gibi | yeah it is a long journey | |
| 13:10:14 | gibi | and it is not over yet :) | |
| 13:32:54 | opendevreview | ribaudr proposed openstack/nova-specs master: libvirt: Allow Manila shares to be directly attached to instances https://review.opendev.org/c/openstack/nova-specs/+/833669 | |
| 13:49:53 | dansmith | sean-k-mooney: I know.. something has to be wrong right? | |
| 13:50:44 | sean-k-mooney | presumable. i would expect keysone to be higher then the rest but not 10x | |
| 13:51:01 | sean-k-mooney | and that 10x neutron which is also higher then the rest | |
| 13:51:38 | dansmith | right | |
| 13:52:11 | dansmith | neutron goes over keystone quickly during tempest tho | |
| 14:11:03 | sean-k-mooney | dansmith: while your here we use 2 allocation for live migration correct? one is held by the instnace an the other by the migration uuid? | |
| 14:11:19 | dansmith | yeah | |
| 14:11:39 | sean-k-mooney | do you remember when that change was made it was 5 or more cycles ago right | |
| 14:11:43 | sean-k-mooney | as in its not recent | |
| 14:11:50 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1975490 just in the context of that | |
| 14:11:58 | dansmith | oh jeez, no idea, but I'm sure it was 5+ ago | |
| 14:12:32 | sean-k-mooney | right it was well before train with the new vcpu code | |
| 14:13:23 | sean-k-mooney | they did not say what version of openstack they were using but im guessign it was quite an old version if they are seeing placment check the source allocations like they describe | |
| 14:14:58 | sean-k-mooney | i dont think any supported branch including the ones in EM still use a signel allocation for both hosts | |
| 14:15:15 | dansmith | yeah I'm sure not | |
| 14:15:41 | sean-k-mooney | we will see if they respond i guess | |
| 14:30:32 | opendevreview | ribaudr proposed openstack/nova-specs master: Allow unshelve to a specific host https://review.opendev.org/c/openstack/nova-specs/+/831506 | |
| 15:14:46 | kashyap | melwitt: When you're about, can you review this, please: https://review.opendev.org/c/openstack/nova/+/838926 (It already has a +2 from gibi) | |
| 15:15:12 | kashyap | (And it is successfully tested by a Red Hat contributor downstream) | |
| 15:18:16 | kashyap | bauzas: --^ | |
| 15:42:05 | melwitt | bauzas, elodilles, gmann: I started an etherpad for stable branch ci here. will fill it in with more info https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 15:43:08 | melwitt | kashyap: thanks for noticing :) | |
| 15:44:39 | melwitt | gibi, sean-k-mooney: yeah I thought stephenfin may want to look at the current spec bc he had a lot of feedback in previous PS | |
| 15:44:54 | stephenfin | oh, I'm looking at it at the moment | |
| 15:44:56 | kashyap | melwitt: I recall debugging it with you in the past; thanks for bringing it to its logical conclusion | |
| 15:45:17 | kashyap | Looks good to me to workaround it. (Also, the CI failure seems unrelated) | |
| 15:45:20 | melwitt | stephenfin: great :) | |
| 15:46:17 | melwitt | kashyap: will also review your patch | |
| 15:46:29 | kashyap | melwitt: It's also a workaround :D | |
| 15:46:36 | melwitt | haha nice | |
| 15:46:53 | elodilles | melwitt: thanks \o/ it was on my TODO as well o:) | |
| 15:47:51 | kashyap | melwitt: But it is functionally tested by a Red Hat support eng downstream | |
| 15:48:01 | kashyap | (And it works) | |
| 15:48:06 | gmann | melwitt: ack, thanks. | |
| 15:48:21 | melwitt | kashyap: ack | |
| 15:49:35 | kashyap | Thank you | |
| 15:54:18 | elodilles | melwitt: i proposed quite many bandit capping patches in the past, but did not notice that placement needs it :-o ( https://review.opendev.org/q/topic:cap-bandit-1.6.2 ) | |
| 15:56:37 | melwitt | elodilles: yeah I remembered that being a thing and I think I found your patches for other projects. placement had been working ok for awhile, not actually sure when this cropped up. backports don't get proposed there often | |
| 15:58:10 | bauzas | Uggla: made a review with a French link on https://review.opendev.org/c/openstack/nova-specs/+/831506/ | |
| 15:58:35 | bauzas | either way, +2d | |
| 15:58:52 | bauzas | cores, a spec seems very close to be accepted https://review.opendev.org/c/openstack/nova-specs/+/831506/4.55 | |
| 15:58:58 | bauzas | cores, a spec seems very close to be accepted https://review.opendev.org/c/openstack/nova-specs/+/831506/4..5 even | |
| 16:00:36 | elodilles | melwitt: yeah, and we have periodic-stable-jobs in placement only since xena: https://review.opendev.org/c/openstack/placement/+/775384 | |
| 16:01:44 | stephenfin | gibi: sean-k-mooney: melwitt: My only comment is on the config option. 'device_list' suggests it's a list but it's not - it's a MultiStrOpt (meaning you'd simply specify it multiple times in nova.conf) | |
| 16:02:13 | melwitt | elodilles: ah right | |
| 16:02:14 | gmann | bauzas: FYI, i will be absent in nova meeting tomorrow as we have 'release naming meeting' at the same time - http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028618.html | |
| 16:02:14 | elodilles | melwitt: we have the periodic-weekly for master, but maybe this worth a backport if we want to catch broken gates on older stable branches | |