Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-10-10
11:00:41 sean-k-mooney its the other way around
11:00:49 sean-k-mooney we are selecting VFs in the pci tracker
11:01:01 sean-k-mooney and you need to compute the socket name from the adress
11:01:45 sean-k-mooney the main issue i think is still the multi device support
11:02:07 sean-k-mooney since the previous algorthim did not support more hten one nic
11:03:17 dvo-plv yes, currently we do not support multinic setup
11:03:44 sean-k-mooney right so i dont think we should support this in nova until that is addressed
11:04:57 dvo-plv Honestly, i forget about that, I will investigate internal if we have some plan how this should be done in the future, because as far as I know we have plan to provide this feature
11:06:02 dvo-plv But if this is a limitation for be in the common code, what about to pass us with separate os-vif plugin, which will handle all this issuses ?
11:06:57 sean-k-mooney a out of tree os-vif plugin will also reqiure your own ml2 driver in neutron
11:08:17 dvo-plv yes, we can implement it too, like Agilio https://github.com/Netronome/agilio-ovs-openstack-plugin/tree/master/vif_plug_agilio_ovs
13:57:13 opendevreview Dan Smith proposed openstack/nova master: WIP: Compile mdev samples for nova-next https://review.opendev.org/c/openstack/nova/+/897708
13:58:11 dansmith bauzas: this is getting close to working ^
13:58:29 bauzas dansmith: wow, thanks
14:00:36 dansmith last round only failed because I didn't sudo something
14:01:48 sean-k-mooney dansmith: would you mind in a followup allowing us to enabel other moduels
14:02:11 sean-k-mooney i.e. mdpy or mbochs?
14:02:13 dansmith sean-k-mooney: yeah I was going to do all the samples, I just wanted to get the mechanics working
14:02:32 dansmith or do you mean *any* other module?
14:03:05 sean-k-mooney i was thinking of makign CONFIG_SAMPLE_VFIO_MDEV_MTTY=m
14:03:11 sean-k-mooney a list
14:03:20 sean-k-mooney but mainly jsut the mtty samples
14:04:08 sean-k-mooney it might be nice if the set of flags to enable in the kernel config was just a var you could set in the local.conf
14:04:24 sean-k-mooney although
14:04:28 sean-k-mooney looking at it again
14:04:30 dansmith it's not that easy
14:04:36 sean-k-mooney you alls need to call make per module
14:04:44 sean-k-mooney so ya never mind
14:04:55 dansmith unless you want to build the whole kernel, you need to know configs and target module files, and the deps
14:05:01 bauzas fwiw, this will actually depend on what we can do with live-migration :)
14:05:10 dansmith but yeah, I'll extend this to do all the mdev samples
14:05:13 sean-k-mooney dansmith: ya lets keep it simple
14:05:13 bauzas if that's done per mtty, cool
14:05:18 dansmith bauzas: oh I know, but we could also apply a patch here
14:05:35 bauzas yeah, let's just start this simple with mtty
14:05:47 sean-k-mooney bauzas: so this is useful even without live migration
14:05:57 bauzas yup, definitely
14:06:09 sean-k-mooney it woudl be a majory win just to get any coverage of mdevs
14:06:59 sean-k-mooney so ya we can add follow up patches to do the nova config generation to enabel them
14:07:10 sean-k-mooney but that would be good to do in the devstack plugin to
14:07:47 dansmith yeah, point me at what I need to do config-wise
14:07:58 sean-k-mooney you technnially dont need that since we can just shove config snipit into config via local.conf
14:08:28 dansmith ack, but if it can do something for each sample it finds or something...
14:08:39 sean-k-mooney am [device]enabled_mdev_types ...
14:08:41 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#devices.enabled_mdev_types
14:08:48 sean-k-mooney there is an example in teh help text
14:09:02 sean-k-mooney ah that raises a diffent point
14:09:13 sean-k-mooney right now nova required the parent of the mdev to be a pci adress
14:09:21 sean-k-mooney to use these modules we need to lift that limiation
14:09:24 dansmith I really know nothing about mdevs.. once I insert the module is there a list of those types somewhere? or will the type just be mtty?
14:09:57 sean-k-mooney when you insert the module you will be able to see mdev type under /sys
14:12:28 sean-k-mooney it will be mtty-1 or mtty-2
14:12:42 sean-k-mooney for 1 port or 2 port serial device
14:12:57 sean-k-mooney you can show them with tree /sys/devices/virtual/mtty/mtty/
14:14:27 dansmith ack
14:14:45 sean-k-mooney the modifciation to make nova mdev support work is minor but nova will need to be twaked slightly as the way we create teh placement RP currenlty usees the parent pci adress in teh rp name
14:15:16 dansmith ack, well, maybe worth getting ahead of that
14:15:29 dansmith grr, my jammy machine is doing unattended upgrades.. I effing hate that new default behavior
14:15:47 sean-k-mooney its been enabled by default i think since 20.04
14:16:00 sean-k-mooney but ya it ocationally breaks devstack
14:16:15 sean-k-mooney when the apt lock is held by it
14:16:41 dansmith it's just not something I want without being asked, on really any machine
14:16:58 sean-k-mooney i think it defaults to security updates only
14:17:01 sean-k-mooney so i get there intent
14:17:08 dansmith it's just not something I want without being asked, on really any machine
14:17:45 sean-k-mooney its actully an install option too but the cloud images default to it. but ya i get that
14:25:10 bauzas I need to get my daugter from school but I'll be back in a sec
14:46:57 opendevreview Pavlo Shchelokovskyy proposed openstack/nova master: Update disk.info in finish_migration https://review.opendev.org/c/openstack/nova/+/897842
15:00:21 bauzas dansmith: if you want me to look at that, lemme know
15:00:35 bauzas I'll also compile the module in my own devstack instance
15:00:58 dansmith bauzas: look at what? fixing the nova pci parent requirement?
15:03:29 dansmith I forgot, I still have one other thing I need to fix, which is getting the version of the module to match the running kernel.. I'm not sure why that's not working, but I think it's because it's outside the debian build system
15:04:56 bauzas dansmith: wdym about the pci parent requirement ?
15:05:53 dansmith bauzas: see above from sean-k-mooney
15:06:37 bauzas ah
15:06:44 bauzas shit indeed
15:07:00 bauzas -ETOOMANYTHINGSTODOATSAME
15:09:34 bauzas dansmith: sean-k-mooney: we need the parent address to know which GPU this is using
15:10:10 dansmith I think sean-k-mooney was implying that these devices don't have a pci parent
15:12:31 bauzas https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8029
15:12:45 bauzas yeah, but I guess those devices have a parent fortunately
15:13:19 bauzas from what I can read, the parent is "virtual'"
15:13:47 bauzas https://docs.kernel.org/driver-api/vfio-mediated-device.html#directories-and-files-under-the-sysfs-for-each-physical-device
15:14:14 bauzas actually, no, its name is "mtty"
15:14:19 bauzas https://github.com/torvalds/linux/tree/master/samples/vfio-mdev#using-the-mtty-vfio-mdev-sample-code
15:14:53 bauzas so, somehow we would need to modify nova to say "meh, if the pci address is "mtty", don't change it"
15:29:45 sean-k-mooney dansmith: correct it wont create a pci device
15:30:13 sean-k-mooney bauzas: yep something like that i was thinig of a centinal like "virtual"
15:30:20 sean-k-mooney instead of allowing any string
15:30:41 bauzas should be something easy
15:30:54 bauzas we have a few things to change, but that may work
15:31:14 bauzas provided we indeed have some kind of global attribute
15:31:21 sean-k-mooney i looke at the code a while ago we only realy use this for the placement rp name
15:31:27 bauzas we key dicts on pci address
15:31:43 bauzas but 'virtual' or whatever else (say mtty) works
15:31:48 sean-k-mooney yep so we just need a hashable key which cna be the mdev_type
15:31:55 bauzas no
15:32:06 sean-k-mooney well thats what mtty would be
15:32:09 bauzas https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7976
15:32:19 bauzas we key types per pci device

Earlier   Later