Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-10-10
14:05:01 bauzas fwiw, this will actually depend on what we can do with live-migration :)
14:05:10 dansmith but yeah, I'll extend this to do all the mdev samples
14:05:13 sean-k-mooney dansmith: ya lets keep it simple
14:05:13 bauzas if that's done per mtty, cool
14:05:18 dansmith bauzas: oh I know, but we could also apply a patch here
14:05:35 bauzas yeah, let's just start this simple with mtty
14:05:47 sean-k-mooney bauzas: so this is useful even without live migration
14:05:57 bauzas yup, definitely
14:06:09 sean-k-mooney it woudl be a majory win just to get any coverage of mdevs
14:06:59 sean-k-mooney so ya we can add follow up patches to do the nova config generation to enabel them
14:07:10 sean-k-mooney but that would be good to do in the devstack plugin to
14:07:47 dansmith yeah, point me at what I need to do config-wise
14:07:58 sean-k-mooney you technnially dont need that since we can just shove config snipit into config via local.conf
14:08:28 dansmith ack, but if it can do something for each sample it finds or something...
14:08:39 sean-k-mooney am [device]enabled_mdev_types ...
14:08:41 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#devices.enabled_mdev_types
14:08:48 sean-k-mooney there is an example in teh help text
14:09:02 sean-k-mooney ah that raises a diffent point
14:09:13 sean-k-mooney right now nova required the parent of the mdev to be a pci adress
14:09:21 sean-k-mooney to use these modules we need to lift that limiation
14:09:24 dansmith I really know nothing about mdevs.. once I insert the module is there a list of those types somewhere? or will the type just be mtty?
14:09:57 sean-k-mooney when you insert the module you will be able to see mdev type under /sys
14:12:28 sean-k-mooney it will be mtty-1 or mtty-2
14:12:42 sean-k-mooney for 1 port or 2 port serial device
14:12:57 sean-k-mooney you can show them with tree /sys/devices/virtual/mtty/mtty/
14:14:27 dansmith ack
14:14:45 sean-k-mooney the modifciation to make nova mdev support work is minor but nova will need to be twaked slightly as the way we create teh placement RP currenlty usees the parent pci adress in teh rp name
14:15:16 dansmith ack, well, maybe worth getting ahead of that
14:15:29 dansmith grr, my jammy machine is doing unattended upgrades.. I effing hate that new default behavior
14:15:47 sean-k-mooney its been enabled by default i think since 20.04
14:16:00 sean-k-mooney but ya it ocationally breaks devstack
14:16:15 sean-k-mooney when the apt lock is held by it
14:16:41 dansmith it's just not something I want without being asked, on really any machine
14:16:58 sean-k-mooney i think it defaults to security updates only
14:17:01 sean-k-mooney so i get there intent
14:17:08 dansmith it's just not something I want without being asked, on really any machine
14:17:45 sean-k-mooney its actully an install option too but the cloud images default to it. but ya i get that
14:25:10 bauzas I need to get my daugter from school but I'll be back in a sec
14:46:57 opendevreview Pavlo Shchelokovskyy proposed openstack/nova master: Update disk.info in finish_migration https://review.opendev.org/c/openstack/nova/+/897842
15:00:21 bauzas dansmith: if you want me to look at that, lemme know
15:00:35 bauzas I'll also compile the module in my own devstack instance
15:00:58 dansmith bauzas: look at what? fixing the nova pci parent requirement?
15:03:29 dansmith I forgot, I still have one other thing I need to fix, which is getting the version of the module to match the running kernel.. I'm not sure why that's not working, but I think it's because it's outside the debian build system
15:04:56 bauzas dansmith: wdym about the pci parent requirement ?
15:05:53 dansmith bauzas: see above from sean-k-mooney
15:06:37 bauzas ah
15:06:44 bauzas shit indeed
15:07:00 bauzas -ETOOMANYTHINGSTODOATSAME
15:09:34 bauzas dansmith: sean-k-mooney: we need the parent address to know which GPU this is using
15:10:10 dansmith I think sean-k-mooney was implying that these devices don't have a pci parent
15:12:31 bauzas https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8029
15:12:45 bauzas yeah, but I guess those devices have a parent fortunately
15:13:19 bauzas from what I can read, the parent is "virtual'"
15:13:47 bauzas https://docs.kernel.org/driver-api/vfio-mediated-device.html#directories-and-files-under-the-sysfs-for-each-physical-device
15:14:14 bauzas actually, no, its name is "mtty"
15:14:19 bauzas https://github.com/torvalds/linux/tree/master/samples/vfio-mdev#using-the-mtty-vfio-mdev-sample-code
15:14:53 bauzas so, somehow we would need to modify nova to say "meh, if the pci address is "mtty", don't change it"
15:29:45 sean-k-mooney dansmith: correct it wont create a pci device
15:30:13 sean-k-mooney bauzas: yep something like that i was thinig of a centinal like "virtual"
15:30:20 sean-k-mooney instead of allowing any string
15:30:41 bauzas should be something easy
15:30:54 bauzas we have a few things to change, but that may work
15:31:14 bauzas provided we indeed have some kind of global attribute
15:31:21 sean-k-mooney i looke at the code a while ago we only realy use this for the placement rp name
15:31:27 bauzas we key dicts on pci address
15:31:43 bauzas but 'virtual' or whatever else (say mtty) works
15:31:48 sean-k-mooney yep so we just need a hashable key which cna be the mdev_type
15:31:55 bauzas no
15:32:06 sean-k-mooney well thats what mtty would be
15:32:09 bauzas https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7976
15:32:19 bauzas we key types per pci device
15:32:39 bauzas but this can be "for mtty those are the types it supports'"
15:32:44 sean-k-mooney yes but i dont think you understaood what i was suggesting
15:33:10 bauzas my bad then
15:33:29 sean-k-mooney basiclly when we see virtual for the pci adress on the config i was suggeting we generate a identify such as virtual_<mdev-type>
15:33:46 sean-k-mooney to use everywhere we actully woudl use the pci adress
15:33:55 bauzas ah
15:34:08 bauzas well, technically the virtual device has N possible types
15:34:11 sean-k-mooney but we could just leave it as a stirng and see on this is not a real pci adres just pass it as is
15:34:36 sean-k-mooney ya it does and unlike some hardware
15:34:36 bauzas for config options, we can just leave it to be 'virtual'
15:34:42 sean-k-mooney they are consumable indepently i belive
15:35:03 bauzas we just parse the option here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7971
15:35:14 bauzas but I guess we can accept if it's the global attribute name
15:35:21 bauzas that said, nothing to be documented
15:35:23 sean-k-mooney bauzas: for what tis worht the parent of the device is libvirt is going to be the root device
15:38:22 bauzas sean-k-mooney: well, if I'm relying on sysfs, this is unfortunately 'mtty' if the path is /sys/devices/virtual/mtty/mtty/
15:38:38 bauzas https://docs.kernel.org/driver-api/vfio-mediated-device.html#directories-and-files-under-the-sysfs-for-each-physical-device
15:39:49 sean-k-mooney yes
15:40:01 sean-k-mooney the mtty module only create 1 virtual device
15:40:07 sean-k-mooney so it will by mtty
15:40:25 sean-k-mooney that also why i suggested using virtual as the placeholder :)
15:41:06 sean-k-mooney so we could use virtual/mtty/mtty or simial as the place holder but i didnt really want to fully log the thing to the sys layout
15:41:41 sean-k-mooney when using the netdev sim module for vdpa
15:41:52 sean-k-mooney you can have it create muplie virutal devices
15:42:09 sean-k-mooney i guess we can keep it simple and just start with one
15:43:31 bauzas yeah, I don't disagree with you
15:53:37 bauzas doh, forgot the reminder and the agenda
15:53:46 bauzas reminder: nova meeting in 6 mins
15:53:48 bauzas her.

Earlier   Later