Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-12
15:09:30 bauzas efried: the mdev allocation logic is made there https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6251
15:09:36 bauzas efried: right
15:09:45 bauzas efried: I want to ideally know the PCI device
15:09:54 bauzas but this mapping is under my responsibility
15:10:03 efried right
15:10:07 bauzas so I need to know which RP is allocated
15:10:17 bauzas and then map the PCI ID thanks to the convention
15:10:30 efried Well, you know which *RP* is allocated. But you don't know its name, only its UUID.
15:10:44 bauzas right, that's what we're discussing since 15 mins
15:10:53 bauzas well, 20 even
15:11:12 efried Right, I think we've come full circle. If you had the provider_tree, you would be sorted. Otherwise you need some other way to maintain a mapping.
15:11:24 efried jaypipes: once you've caught up: Would you support adding a provider_tree arg to spawn()?
15:11:58 bauzas I have one thought
15:12:06 bauzas we always push information down to the driver
15:12:18 bauzas can we have the virt driver getting info from the service ?
15:12:23 bauzas I mean a getter
15:12:48 bauzas I don't like changing our interfaces (at least spawn()) for a very specific usecase
15:13:04 bauzas and 99% of deployments won't care of VGPUs
15:13:24 bauzas but I'm not a virt driver specialist
15:14:22 efried you mean introducing a protected static final ComputeDriver.get_provider_tree() method?
15:14:40 bauzas I dunno, thinking out loud
15:14:51 gibi bauzas: for pci we have InstancePciRequest for NUMA we have InstanceNumaTopology for VGPU we have no object to hold this kind of information
15:14:58 efried hell, update_provider_tree could save off its provider tree as an instance var before it returns
15:15:06 bauzas snap, I need to taxi my kids
15:15:22 bauzas efried: that's called a cache
15:15:35 efried so it would be like the virt driver saving a mapping, except it doesn't need to do any special work to construct it - it's already there.
15:15:37 bauzas efried: and we know it can be a problem
15:15:48 bauzas I need to think about this
15:16:01 bauzas anyway, I need to drop for a couple of hours
15:16:09 bauzas \o
15:20:14 jaypipes efried, bauzas, gibi, cdent: I have said for over a year that the virt driver will need to keep a mapping of the RP UUID to whatever name it uses to refer to that RP internally (mdev, PCI address, whatever the heck IBM does, etc)
15:20:29 cdent yup
15:21:03 cdent virt driver's job
15:21:12 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: [WIP] libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein" https://review.openstack.org/632507
15:22:59 gibi jaypipes: while I agree it is a bit more complicate if the RP is created outside of the virt driver like when neutron and cyborg creates the RP but the virt driver consumes the RP
15:23:20 gibi clearly for VGPU your suggestion works
15:24:59 sean-k-mooney gibi: well in the cyboge/neutron case nova does not need to consomne the allocation
15:25:22 sean-k-mooney it just needs to pass it back to cyborg/neutorn so can they do the transaltion then
15:25:44 gibi sean-k-mooney: for sriov ports, the virt driver still responsible to select the PCI VF
15:26:08 sean-k-mooney yes this is true but currently they are not in placement
15:26:08 gibi sean-k-mooney: and the PCI VF should come from the same RP
15:26:18 gibi as the bandwidth
15:26:22 sean-k-mooney the same RP as the bandwith i am assuming
15:26:24 sean-k-mooney ya
15:26:38 gibi so transitively the virt driver needs to know which RP is used for the bandwidth
15:27:10 sean-k-mooney well the bandwith woudl be an inventor on teh PF RP and we soudl similar have an inventory of VFs
15:28:05 sean-k-mooney so as long as the VF resust and bandwith request are in teh same requst group we shoudl be ok the issue is that would require 2 services to manage different inventoies in the same RP
15:28:13 sean-k-mooney which we dont support today
15:28:30 gibi sean-k-mooney: yeah, it is solved when VF inventory is moved to the placement
15:29:17 sean-k-mooney i personally had assumed that the resouce track would evolve to sotre the placement RP mappings
15:29:59 efried jaypipes: So that mapping has to be created in update_provider_tree (and could in fact *be* the provider tree created therein) and retrievable from spawn.
15:30:51 efried If we state it that way ^, then I guess we can leave it up to each virt driver how it wants to create and store the mapping. Shoving the provider tree into an instance var seems like as easy a way as any.
15:33:55 jaypipes efried: it could, yes. though there's nothing in what I said that *requires* it be that way. After all, the PCI device management code (gawd save us all), external Neutron agents (per gibi's series) and other things could easily keep a map of UUID -> internal name in whatever way they wanted. Shit, I believe I proposed at some point that a simple YAML file on disk storing the map of UUID -> internal device/provider name would be just fine
15:34:40 openstackgerrit Ralf Haferkamp proposed openstack/nova master: compute: reset instance events on reload https://review.openstack.org/420026
15:43:45 sean-k-mooney jaypipes: the other option which im not a huge fan of would be just to set the service specific internal name in the RP name field
15:44:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360
15:44:40 sean-k-mooney jaypipes: that does not really work well in the general case and dose not work at all in the inter service case hence why im not a huge fan of that.
15:45:00 kashyap mdbooth: If you get some time, I'd really appreciate some libvirt driver and sensible eyes in general on this: https://review.openstack.org/#/c/632507/
15:45:18 kashyap mdbooth: I know it not fully complete, but the serial / char dev code is tripping me up a bit.
15:46:01 kashyap (Funnily enough, for me, the bug that is referenced in the code that's being deleted ... was a bug I filed 5 years ago)
15:46:11 sean-k-mooney in the sriov case if the RP name was the PF adress neutron and nova would agree on it at least
15:46:59 sean-k-mooney but really that is just "bit stealing" rather and a convention rather then a dedicated mechanium
15:48:15 jaypipes sean-k-mooney: that's basically what gibi's doing in https://review.openstack.org/#/c/623543/23/nova/compute/manager.py@2138
15:48:47 efried sean-k-mooney: We already do that in the vgpu code too. It doesn't help, because the allocation doesn't include the rp name.
15:48:53 sean-k-mooney i mean it will work as long as the service that need to share an RP agree on on how to chose the name
15:49:01 jaypipes the issue is that I see that as a fragile solution that depends on the name structure being codified and not changing ever. as soon as the device naming convention changes, everything would blow up.
15:49:24 cdent ++
15:49:25 sean-k-mooney efried: well you could lookup the rp by its uuid if you need the info form the allocation
15:49:28 efried shrug, be like saying if the mapping algorithm changes everything blows up
15:49:37 efried sean-k-mooney: This is where we started.
15:49:41 efried sean-k-mooney: Where do you look up the rp?
15:49:52 sean-k-mooney in placement
15:49:55 efried in placement? no, because we've said we don't want virt talking directly to placement
15:50:06 efried in the provider tree? Sure, but you have to get the provider tree somehow
15:50:14 jaypipes providers should always be *externally referenced* (i.e. any time two services need to exchange info about a provider) using a UUID. internally, within a service, the service should be responsible for caching an internal name -> UUID map.
15:50:19 efried in a mapping? Sure, but the mapping needs to be saved by the virt driver
15:50:27 sean-k-mooney if the virt dirver create the RP with a specific name in know it could take the uuid for the allcoation and look it up and get the name back
15:50:49 efried by talking to placement? No, see above. By looking in a mapping? Sure, see above.
15:51:04 sean-k-mooney efried: it only need to be saved for efficency.
15:52:19 efried sean-k-mooney: The proposed solution is for update_provider_tree to save a mapping of RP UUID to <something> and then for spawn to retrieve that information. The easiest thing would be for update_provider_tree simply to save the provider_tree as an instance var on the virt driver. Then spawn can just ask that guy for the provider with a given uuid and get the name. Since the name is already generated by convention, it can use it
15:53:04 sean-k-mooney efried: sure that makes sense
15:53:11 efried if we really don't like using the naming convention, upt could save some other arbitrarily-formatted dict as an instance var, to the same end.
15:53:52 sean-k-mooney but making that internal mapping to the uuid stable it the respociblity of the service consuming placment in that case not placement
15:54:16 efried yes, update_provider_tree is that thing in either case.
15:54:49 efried even if spawn got to request the provider record from placement and use the naming convention, it was still upt's responsibility to name the provider appropriately.
15:55:06 sean-k-mooney ya
15:55:17 efried but I prefer the idea of upt caching^wahem, saving something as an instance var.
15:55:53 sean-k-mooney isnt this partly solved by https://review.openstack.org/#/c/612497/ also
15:56:03 sean-k-mooney the provider ymal file
15:56:19 efried sean-k-mooney: It could be, but I kind of hate the idea of making it the operator's responsibility
15:56:53 efried as long as there's reasonable default behavior if that file is missing (or composed poorly) it would be okay.
15:56:54 sean-k-mooney well there is no reason the virtdriver could not generate it on first boot if its not provided
15:57:03 sean-k-mooney and then operator could add to it
15:57:17 efried also ew.
15:57:28 sean-k-mooney that said that might mess with som config managment stuff
15:57:30 efried Now what if I hot-(un)plug devices or otherwise change resources.
15:57:33 efried yeah
15:57:56 efried now I'm forced to hand-edit the file after that first time. Or remember to blow it away so it can be regenerated, but then I lose my customizations.
15:58:03 sean-k-mooney you first update teh yaml file then send a sig hup to the compute agent then you remove the hardware
15:58:35 efried as opposed to just... removing the hardware and everything works.

Earlier   Later