| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-10 | |||
| 10:50:50 | sean-k-mooney | not nessiarly what that means is that nova needs to tell neutron the vf specific info | |
| 10:51:20 | sean-k-mooney | so we need to pass info to the ml2 driver in the binding_profile | |
| 10:51:52 | sean-k-mooney | neutron can then use that to generate the socket path. alternitivly we could jsut generate the path ourselve in nova | |
| 10:52:21 | dvo-plv | path is not a problem, specific socket name is the root cause | |
| 10:52:44 | sean-k-mooney | the socket name is path of the path :) | |
| 10:52:47 | sean-k-mooney | but yes | |
| 10:54:51 | dvo-plv | So, we can somehow use port binding profile for this purposes? | |
| 10:56:17 | sean-k-mooney | yes that is what its for to pass info form nova to the network backend | |
| 10:56:27 | sean-k-mooney | we normally sotre the pci_adress in it | |
| 10:56:44 | sean-k-mooney | but we can have other fields like the offset ectra | |
| 10:56:59 | sean-k-mooney | or we can just comppute the socket name and pass that | |
| 10:57:07 | sean-k-mooney | which is proably the better option | |
| 10:57:30 | sean-k-mooney | since nova does not have the base path (neutron/ovs does have that info) | |
| 10:57:53 | sean-k-mooney | so we can provide the socket name in the binding profile and neutron can append that to the base socket path | |
| 10:58:03 | sean-k-mooney | and then we will get the full path back which we can store in the xml | |
| 11:00:20 | dvo-plv | okay, i see. but i have a question how we should handle pci. for socket name stdvio10, we need to use pci 0000:65:01.2. So we need vendor specific mechanism for parse socket name | |
| 11:00:41 | sean-k-mooney | its the other way around | |
| 11:00:49 | sean-k-mooney | we are selecting VFs in the pci tracker | |
| 11:01:01 | sean-k-mooney | and you need to compute the socket name from the adress | |
| 11:01:45 | sean-k-mooney | the main issue i think is still the multi device support | |
| 11:02:07 | sean-k-mooney | since the previous algorthim did not support more hten one nic | |
| 11:03:17 | dvo-plv | yes, currently we do not support multinic setup | |
| 11:03:44 | sean-k-mooney | right so i dont think we should support this in nova until that is addressed | |
| 11:04:57 | dvo-plv | Honestly, i forget about that, I will investigate internal if we have some plan how this should be done in the future, because as far as I know we have plan to provide this feature | |
| 11:06:02 | dvo-plv | But if this is a limitation for be in the common code, what about to pass us with separate os-vif plugin, which will handle all this issuses ? | |
| 11:06:57 | sean-k-mooney | a out of tree os-vif plugin will also reqiure your own ml2 driver in neutron | |
| 11:08:17 | dvo-plv | yes, we can implement it too, like Agilio https://github.com/Netronome/agilio-ovs-openstack-plugin/tree/master/vif_plug_agilio_ovs | |
| 13:57:13 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Compile mdev samples for nova-next https://review.opendev.org/c/openstack/nova/+/897708 | |
| 13:58:11 | dansmith | bauzas: this is getting close to working ^ | |
| 13:58:29 | bauzas | dansmith: wow, thanks | |
| 14:00:36 | dansmith | last round only failed because I didn't sudo something | |
| 14:01:48 | sean-k-mooney | dansmith: would you mind in a followup allowing us to enabel other moduels | |
| 14:02:11 | sean-k-mooney | i.e. mdpy or mbochs? | |
| 14:02:13 | dansmith | sean-k-mooney: yeah I was going to do all the samples, I just wanted to get the mechanics working | |
| 14:02:32 | dansmith | or do you mean *any* other module? | |
| 14:03:05 | sean-k-mooney | i was thinking of makign CONFIG_SAMPLE_VFIO_MDEV_MTTY=m | |
| 14:03:11 | sean-k-mooney | a list | |
| 14:03:20 | sean-k-mooney | but mainly jsut the mtty samples | |
| 14:04:08 | sean-k-mooney | it might be nice if the set of flags to enable in the kernel config was just a var you could set in the local.conf | |
| 14:04:24 | sean-k-mooney | although | |
| 14:04:28 | sean-k-mooney | looking at it again | |
| 14:04:30 | dansmith | it's not that easy | |
| 14:04:36 | sean-k-mooney | you alls need to call make per module | |
| 14:04:44 | sean-k-mooney | so ya never mind | |
| 14:04:55 | dansmith | unless you want to build the whole kernel, you need to know configs and target module files, and the deps | |
| 14:05:01 | bauzas | fwiw, this will actually depend on what we can do with live-migration :) | |
| 14:05:10 | dansmith | but yeah, I'll extend this to do all the mdev samples | |
| 14:05:13 | sean-k-mooney | dansmith: ya lets keep it simple | |
| 14:05:13 | bauzas | if that's done per mtty, cool | |
| 14:05:18 | dansmith | bauzas: oh I know, but we could also apply a patch here | |
| 14:05:35 | bauzas | yeah, let's just start this simple with mtty | |
| 14:05:47 | sean-k-mooney | bauzas: so this is useful even without live migration | |
| 14:05:57 | bauzas | yup, definitely | |
| 14:06:09 | sean-k-mooney | it woudl be a majory win just to get any coverage of mdevs | |
| 14:06:59 | sean-k-mooney | so ya we can add follow up patches to do the nova config generation to enabel them | |
| 14:07:10 | sean-k-mooney | but that would be good to do in the devstack plugin to | |
| 14:07:47 | dansmith | yeah, point me at what I need to do config-wise | |
| 14:07:58 | sean-k-mooney | you technnially dont need that since we can just shove config snipit into config via local.conf | |
| 14:08:28 | dansmith | ack, but if it can do something for each sample it finds or something... | |
| 14:08:39 | sean-k-mooney | am [device]enabled_mdev_types ... | |
| 14:08:41 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#devices.enabled_mdev_types | |
| 14:08:48 | sean-k-mooney | there is an example in teh help text | |
| 14:09:02 | sean-k-mooney | ah that raises a diffent point | |
| 14:09:13 | sean-k-mooney | right now nova required the parent of the mdev to be a pci adress | |
| 14:09:21 | sean-k-mooney | to use these modules we need to lift that limiation | |
| 14:09:24 | dansmith | I really know nothing about mdevs.. once I insert the module is there a list of those types somewhere? or will the type just be mtty? | |
| 14:09:57 | sean-k-mooney | when you insert the module you will be able to see mdev type under /sys | |
| 14:12:28 | sean-k-mooney | it will be mtty-1 or mtty-2 | |
| 14:12:42 | sean-k-mooney | for 1 port or 2 port serial device | |
| 14:12:57 | sean-k-mooney | you can show them with tree /sys/devices/virtual/mtty/mtty/ | |
| 14:14:27 | dansmith | ack | |
| 14:14:45 | sean-k-mooney | the modifciation to make nova mdev support work is minor but nova will need to be twaked slightly as the way we create teh placement RP currenlty usees the parent pci adress in teh rp name | |
| 14:15:16 | dansmith | ack, well, maybe worth getting ahead of that | |
| 14:15:29 | dansmith | grr, my jammy machine is doing unattended upgrades.. I effing hate that new default behavior | |
| 14:15:47 | sean-k-mooney | its been enabled by default i think since 20.04 | |
| 14:16:00 | sean-k-mooney | but ya it ocationally breaks devstack | |
| 14:16:15 | sean-k-mooney | when the apt lock is held by it | |
| 14:16:41 | dansmith | it's just not something I want without being asked, on really any machine | |
| 14:16:58 | sean-k-mooney | i think it defaults to security updates only | |
| 14:17:01 | sean-k-mooney | so i get there intent | |
| 14:17:08 | dansmith | it's just not something I want without being asked, on really any machine | |
| 14:17:45 | sean-k-mooney | its actully an install option too but the cloud images default to it. but ya i get that | |
| 14:25:10 | bauzas | I need to get my daugter from school but I'll be back in a sec | |
| 14:46:57 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Update disk.info in finish_migration https://review.opendev.org/c/openstack/nova/+/897842 | |
| 15:00:21 | bauzas | dansmith: if you want me to look at that, lemme know | |
| 15:00:35 | bauzas | I'll also compile the module in my own devstack instance | |
| 15:00:58 | dansmith | bauzas: look at what? fixing the nova pci parent requirement? | |
| 15:03:29 | dansmith | I forgot, I still have one other thing I need to fix, which is getting the version of the module to match the running kernel.. I'm not sure why that's not working, but I think it's because it's outside the debian build system | |
| 15:04:56 | bauzas | dansmith: wdym about the pci parent requirement ? | |
| 15:05:53 | dansmith | bauzas: see above from sean-k-mooney | |
| 15:06:37 | bauzas | ah | |
| 15:06:44 | bauzas | shit indeed | |
| 15:07:00 | bauzas | -ETOOMANYTHINGSTODOATSAME | |
| 15:09:34 | bauzas | dansmith: sean-k-mooney: we need the parent address to know which GPU this is using | |
| 15:10:10 | dansmith | I think sean-k-mooney was implying that these devices don't have a pci parent | |
| 15:12:31 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8029 | |
| 15:12:45 | bauzas | yeah, but I guess those devices have a parent fortunately | |
| 15:13:19 | bauzas | from what I can read, the parent is "virtual'" | |
| 15:13:47 | bauzas | https://docs.kernel.org/driver-api/vfio-mediated-device.html#directories-and-files-under-the-sysfs-for-each-physical-device | |
| 15:14:14 | bauzas | actually, no, its name is "mtty" | |