| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-02 | |||
| 14:37:56 | bauzas | for a mdev | |
| 14:38:10 | bauzas | we already use this information to track | |
| 14:38:19 | bauzas | in nova today | |
| 14:38:33 | sean-k-mooney | well in the nvidia case the max mdev per type applies to the grandparent device | |
| 14:38:36 | bauzas | what we don't track *now* is to know the parent of a mdev capable device | |
| 14:38:59 | bauzas | and we can't rely on libvirt, it gives you the PCI-E root PCI address (I tested :) ) | |
| 14:40:19 | bauzas | sean-k-mooney: that's what libvirt is giving us https://paste.opendev.org/show/b3MHPT8VoIOBWw1tim2o/ | |
| 14:40:46 | bauzas | we know the parent addr tho | |
| 14:40:48 | bauzas | <capability type='phys_function'> <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/> | |
| 14:41:02 | sean-k-mooney | it tell you the part device | |
| 14:41:05 | bauzas | so you were right, we could use this | |
| 14:41:08 | sean-k-mooney | <parent>pci_0000_40_01_1</parent> | |
| 14:41:23 | bauzas | sean-k-mooney: no, you're wrong, that's not the PF address :) | |
| 14:42:01 | bauzas | [stack@lenovo-sr655-01 ~]$ lspci | grep 40:01.1 | |
| 14:42:01 | bauzas | 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge | |
| 14:42:01 | sean-k-mooney | ok can you give me access to that system so i can take a look | |
| 14:42:13 | bauzas | that's the PCI bridge address :) | |
| 14:42:45 | bauzas | but I told you, we have the parent PF address in the device XML | |
| 14:42:50 | sean-k-mooney | i orgianlly said we could get the info from sysfs because i knwo how to look it up there | |
| 14:43:09 | sean-k-mooney | we can get it form libvirt but i dont rememebr the element fo the top of my head | |
| 14:43:30 | sean-k-mooney | if we have it great | |
| 14:43:52 | sean-k-mooney | its <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/> | |
| 14:44:05 | sean-k-mooney | form the capability type='phys_function' YES | |
| 14:46:02 | bauzas | yeah, and thru sysfs, we have access to the physical function in /sys/class/mdev_bus/0000\:41\:00.4/physfn/ | |
| 14:46:06 | dansmith | okay done with my other meeting | |
| 14:46:18 | bauzas | but that's not a trivial change | |
| 14:46:29 | bauzas | we need to implement some libvirt config change | |
| 14:46:31 | sean-k-mooney | we can and should use the info form libvirt since its there | |
| 14:46:56 | sean-k-mooney | bauzas: we already have parsing logic for this in the pci tracker code in the libvirt driver | |
| 14:46:58 | bauzas | we already get information of the mdev capable devices thru libvirt | |
| 14:47:13 | bauzas | sean-k-mooney: oh you're right | |
| 14:47:28 | bauzas | dansmith: take time to digest | |
| 14:48:12 | bauzas | sounds less bugfix-y, more feature-y than my patch but I can do it | |
| 14:49:19 | bauzas | sean-k-mooney: fwiw, this is where we get information for a mdev cap device https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8284 | |
| 14:49:22 | sean-k-mooney | bauzas: we should seperate the back protable fix form the final fix | |
| 14:49:48 | sean-k-mooney | bauzas: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1321 is the functilalyt that extracts the pci info form te node dev | |
| 14:50:00 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8263 is where we get the capacity | |
| 14:50:10 | bauzas | so sounds easier than originally thought | |
| 14:50:13 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1351-L1363 | |
| 14:50:21 | sean-k-mooney | that actully where we are parsing it | |
| 14:50:25 | dansmith | so max_mdevs_per_device works only because once you allocate one mdev type from a card, that card is locked into that type for the rest of the allocations right? | |
| 14:50:54 | bauzas | dansmith: you're correct, there is no heterogenous support | |
| 14:50:58 | artom | sean-k-mooney, we never updated the Neutron docs for the socket PCI NUMA affinity policy: https://docs.openstack.org/api-ref/network/v2/index.html#numa-affinity-policy | |
| 14:50:59 | sean-k-mooney | dansmith: in the simple case yes. but we also only support one mdev_type per card in nova | |
| 14:51:18 | bauzas | yup, there is no behavioural change in support | |
| 14:51:20 | sean-k-mooney | artom: we never updated the neutron extention for it | |
| 14:51:33 | sean-k-mooney | artom: its not a docs issue i belive we need an api change | |
| 14:51:40 | dansmith | sean-k-mooney: wait, is that the case even before we allocate? | |
| 14:51:41 | artom | sean-k-mooney, so we can't actually set it on the port at all? | |
| 14:52:06 | sean-k-mooney | dansmith: yes which means we break if tye allocate the wrong type manually | |
| 14:52:09 | bauzas | are we still talking of GPUs or PCI NUMA affinity now ? | |
| 14:52:13 | sean-k-mooney | dansmith: we have had customer do that | |
| 14:52:19 | artom | Anyways, I interrupted a convo with my dumb questions, carry on, I'll go look at the code | |
| 14:52:30 | sean-k-mooney | bauzas: im talkign to artom about something else | |
| 14:52:35 | artom | bauzas, ignore me and PCI policies | |
| 14:52:45 | bauzas | artom: no worries, we're reaching to some agreement soon I guess | |
| 14:53:04 | bauzas | dansmith: assigning a mdev type to a GPU is statically defined in nova | |
| 14:53:40 | sean-k-mooney | artom: https://github.com/openstack/neutron-lib/blob/master/neutron_lib/constants.py#L676-L681 | |
| 14:53:40 | dansmith | okay I'm confused | |
| 14:53:49 | dansmith | I thought ya'll said the opposite earlier | |
| 14:53:52 | bauzas | changing the value once you create mdevs is a breaking change that we don't support | |
| 14:53:57 | sean-k-mooney | artom: so the api will not accpet it and we are missing the osc support too | |
| 14:54:01 | dansmith | right I understand that part | |
| 14:54:18 | dansmith | are we going to gmeet? | |
| 14:54:26 | bauzas | sure I can fire one | |
| 14:54:33 | dansmith | just saying, since artom showed up we | |
| 14:54:37 | dansmith | are all muddy here :) | |
| 14:55:03 | artom | Yep, I barged in like a rhino on a sugar rush, apologies | |
| 14:55:13 | bauzas | https://meet.google.com/hqy-bgjb-muk for people fancy discussing nvidia nightmare | |
| 14:55:24 | sean-k-mooney | dansmith: for what its worth the patch bauzas proposed should work given the hardware that is aviabel to buy today | |
| 14:55:53 | sean-k-mooney | its jsut depening on some asusmintions we have in nova and in how the current hardware works | |
| 16:49:10 | opendevreview | Will Szumski proposed openstack/nova stable/yoga: Ironic nodes with instance reserved in placement https://review.opendev.org/c/openstack/nova/+/867912 | |
| 17:41:27 | jovial | Is the nova-emulation job busted on stable/yoga? Was interested in progressing this patch: https://review.opendev.org/c/openstack/nova/+/898554 | |
| 17:42:31 | bauzas | sean-k-mooney: dansmith: so, if you don't mind, I'm gonna work on the VF filtering case with max_instances config option | |
| 17:42:45 | bauzas | and then we could nitpick on what we want | |
| 17:44:23 | dansmith | yep | |
| 17:44:46 | bauzas | dansmith: fwiw, there is also https://review.opendev.org/c/openstack/nova/+/899406 | |
| 17:45:07 | bauzas | with that patch, pas-ha[m] is forcing operators to explicitely define all the device_addresses | |
| 17:45:35 | bauzas | but I personnally think that adding a max_instances config option is painless for ops | |
| 17:45:41 | bauzas | so we could mix both | |
| 17:46:15 | bauzas | meaning that we would force ops to either explicitelyt specify the device addresses *or* give the number of VFs that nova needs to create | |
| 17:46:22 | bauzas | this sounds very consistent to me | |
| 17:50:21 | dansmith | max_instances seems much more useful to me so people don't have to encode pci device addresses in their config *just* to avoid the broken behavior | |
| 17:51:33 | bauzas | yeah so we will keep the existing behaviour but add another option for automatic discovery | |
| 17:51:45 | bauzas | dansmith: I definitely appreciate your help on that one | |
| 17:52:00 | bauzas | and I really want to give you review credits back on your own series | |
| 17:57:51 | sean-k-mooney | dansmith: i have some concerns with allow config with out pci adresses | |
| 17:58:11 | sean-k-mooney | i.e. if you allow some of your gpus to be used for pci passthogu and other for vgpu | |
| 17:58:36 | dansmith | right, that's a good reason *to* do it, but not a good reason to *require* it, IMHO | |
| 17:58:59 | sean-k-mooney | if you dont use device_address in that case we coudl passthough a device (that was allowed in the pci devspec) but not disalloed in the mdev config section | |
| 17:59:37 | sean-k-mooney | dansmith: i dont know i always felt that its a psudo security isseu to not explcitly list devices we can expose to a guest | |
| 17:59:45 | sean-k-mooney | i get that it sucks to need to be explict | |
| 17:59:54 | sean-k-mooney | but that the trade off | |
| 18:00:40 | dansmith | but in the mdev case you're not just blindly exposing them, | |
| 18:00:42 | sean-k-mooney | to be clear we have this tradeoff in the pci tracker too | |
| 18:00:50 | dansmith | you've already opted into "all devices with this mdev type" right? | |
| 18:01:12 | sean-k-mooney | yep so its the same as just putting vendor_id and product_id in the pci dev_spec | |
| 18:01:22 | sean-k-mooney | aka the pci whitelist | |
| 18:01:28 | dansmith | I don't think that's the same really | |
| 18:01:45 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Add documentation on server groups https://review.opendev.org/c/openstack/nova/+/899979 | |