Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-12
15:15:37 bauzas efried: and we know it can be a problem
15:15:48 bauzas I need to think about this
15:16:01 bauzas anyway, I need to drop for a couple of hours
15:16:09 bauzas \o
15:20:14 jaypipes efried, bauzas, gibi, cdent: I have said for over a year that the virt driver will need to keep a mapping of the RP UUID to whatever name it uses to refer to that RP internally (mdev, PCI address, whatever the heck IBM does, etc)
15:20:29 cdent yup
15:21:03 cdent virt driver's job
15:21:12 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: [WIP] libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION for "Stein" https://review.openstack.org/632507
15:22:59 gibi jaypipes: while I agree it is a bit more complicate if the RP is created outside of the virt driver like when neutron and cyborg creates the RP but the virt driver consumes the RP
15:23:20 gibi clearly for VGPU your suggestion works
15:24:59 sean-k-mooney gibi: well in the cyboge/neutron case nova does not need to consomne the allocation
15:25:22 sean-k-mooney it just needs to pass it back to cyborg/neutorn so can they do the transaltion then
15:25:44 gibi sean-k-mooney: for sriov ports, the virt driver still responsible to select the PCI VF
15:26:08 gibi sean-k-mooney: and the PCI VF should come from the same RP
15:26:08 sean-k-mooney yes this is true but currently they are not in placement
15:26:18 gibi as the bandwidth
15:26:22 sean-k-mooney the same RP as the bandwith i am assuming
15:26:24 sean-k-mooney ya
15:26:38 gibi so transitively the virt driver needs to know which RP is used for the bandwidth
15:27:10 sean-k-mooney well the bandwith woudl be an inventor on teh PF RP and we soudl similar have an inventory of VFs
15:28:05 sean-k-mooney so as long as the VF resust and bandwith request are in teh same requst group we shoudl be ok the issue is that would require 2 services to manage different inventoies in the same RP
15:28:13 sean-k-mooney which we dont support today
15:28:30 gibi sean-k-mooney: yeah, it is solved when VF inventory is moved to the placement
15:29:17 sean-k-mooney i personally had assumed that the resouce track would evolve to sotre the placement RP mappings
15:29:59 efried jaypipes: So that mapping has to be created in update_provider_tree (and could in fact *be* the provider tree created therein) and retrievable from spawn.
15:30:51 efried If we state it that way ^, then I guess we can leave it up to each virt driver how it wants to create and store the mapping. Shoving the provider tree into an instance var seems like as easy a way as any.
15:33:55 jaypipes efried: it could, yes. though there's nothing in what I said that *requires* it be that way. After all, the PCI device management code (gawd save us all), external Neutron agents (per gibi's series) and other things could easily keep a map of UUID -> internal name in whatever way they wanted. Shit, I believe I proposed at some point that a simple YAML file on disk storing the map of UUID -> internal device/provider name would be just fine
15:34:40 openstackgerrit Ralf Haferkamp proposed openstack/nova master: compute: reset instance events on reload https://review.openstack.org/420026
15:43:45 sean-k-mooney jaypipes: the other option which im not a huge fan of would be just to set the service specific internal name in the RP name field
15:44:40 sean-k-mooney jaypipes: that does not really work well in the general case and dose not work at all in the inter service case hence why im not a huge fan of that.
15:44:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360
15:45:00 kashyap mdbooth: If you get some time, I'd really appreciate some libvirt driver and sensible eyes in general on this: https://review.openstack.org/#/c/632507/
15:45:18 kashyap mdbooth: I know it not fully complete, but the serial / char dev code is tripping me up a bit.
15:46:01 kashyap (Funnily enough, for me, the bug that is referenced in the code that's being deleted ... was a bug I filed 5 years ago)
15:46:11 sean-k-mooney in the sriov case if the RP name was the PF adress neutron and nova would agree on it at least
15:46:59 sean-k-mooney but really that is just "bit stealing" rather and a convention rather then a dedicated mechanium
15:48:15 jaypipes sean-k-mooney: that's basically what gibi's doing in https://review.openstack.org/#/c/623543/23/nova/compute/manager.py@2138
15:48:47 efried sean-k-mooney: We already do that in the vgpu code too. It doesn't help, because the allocation doesn't include the rp name.
15:48:53 sean-k-mooney i mean it will work as long as the service that need to share an RP agree on on how to chose the name
15:49:01 jaypipes the issue is that I see that as a fragile solution that depends on the name structure being codified and not changing ever. as soon as the device naming convention changes, everything would blow up.
15:49:24 cdent ++
15:49:25 sean-k-mooney efried: well you could lookup the rp by its uuid if you need the info form the allocation
15:49:28 efried shrug, be like saying if the mapping algorithm changes everything blows up
15:49:37 efried sean-k-mooney: This is where we started.
15:49:41 efried sean-k-mooney: Where do you look up the rp?
15:49:52 sean-k-mooney in placement
15:49:55 efried in placement? no, because we've said we don't want virt talking directly to placement
15:50:06 efried in the provider tree? Sure, but you have to get the provider tree somehow
15:50:14 jaypipes providers should always be *externally referenced* (i.e. any time two services need to exchange info about a provider) using a UUID. internally, within a service, the service should be responsible for caching an internal name -> UUID map.
15:50:19 efried in a mapping? Sure, but the mapping needs to be saved by the virt driver
15:50:27 sean-k-mooney if the virt dirver create the RP with a specific name in know it could take the uuid for the allcoation and look it up and get the name back
15:50:49 efried by talking to placement? No, see above. By looking in a mapping? Sure, see above.
15:51:04 sean-k-mooney efried: it only need to be saved for efficency.
15:52:19 efried sean-k-mooney: The proposed solution is for update_provider_tree to save a mapping of RP UUID to <something> and then for spawn to retrieve that information. The easiest thing would be for update_provider_tree simply to save the provider_tree as an instance var on the virt driver. Then spawn can just ask that guy for the provider with a given uuid and get the name. Since the name is already generated by convention, it can use it
15:53:04 sean-k-mooney efried: sure that makes sense
15:53:11 efried if we really don't like using the naming convention, upt could save some other arbitrarily-formatted dict as an instance var, to the same end.
15:53:52 sean-k-mooney but making that internal mapping to the uuid stable it the respociblity of the service consuming placment in that case not placement
15:54:16 efried yes, update_provider_tree is that thing in either case.
15:54:49 efried even if spawn got to request the provider record from placement and use the naming convention, it was still upt's responsibility to name the provider appropriately.
15:55:06 sean-k-mooney ya
15:55:17 efried but I prefer the idea of upt caching^wahem, saving something as an instance var.
15:55:53 sean-k-mooney isnt this partly solved by https://review.openstack.org/#/c/612497/ also
15:56:03 sean-k-mooney the provider ymal file
15:56:19 efried sean-k-mooney: It could be, but I kind of hate the idea of making it the operator's responsibility
15:56:53 efried as long as there's reasonable default behavior if that file is missing (or composed poorly) it would be okay.
15:56:54 sean-k-mooney well there is no reason the virtdriver could not generate it on first boot if its not provided
15:57:03 sean-k-mooney and then operator could add to it
15:57:17 efried also ew.
15:57:28 sean-k-mooney that said that might mess with som config managment stuff
15:57:30 efried Now what if I hot-(un)plug devices or otherwise change resources.
15:57:33 efried yeah
15:57:56 efried now I'm forced to hand-edit the file after that first time. Or remember to blow it away so it can be regenerated, but then I lose my customizations.
15:58:03 sean-k-mooney you first update teh yaml file then send a sig hup to the compute agent then you remove the hardware
15:58:35 efried as opposed to just... removing the hardware and everything works.
15:59:01 sean-k-mooney that is not really a thing today
15:59:14 efried neither is generic numa affinity
15:59:18 sean-k-mooney i mean i "might" work in some cases
15:59:45 efried Seems like all of this is stuff we're striving for.
16:00:16 efried The whole reason update_provider_tree happens on a loop is so that it can, in the future, automatically adjust the provider tree when stuff like this happens.
16:00:27 sean-k-mooney i think https://etherpad.openstack.org/p/nova-ptg-train everyting from lin 7-30 is very related
16:01:35 efried In fact, I think that it might "just work" after Sylvain's vgpu stuff lands to hot-(un)plug a GPU.
16:01:45 efried if you unplug one that has allocations against it, you can expect problems; but otherwise...
16:02:07 efried and even then, I think we've accounted for that by setting reserved=total.
16:02:37 sean-k-mooney well there were edgecases like this https://review.openstack.org/#/c/626381/
16:03:12 sean-k-mooney if the device you remove had allocation against it we cant remove it form placemetn until those allocation are also removed
16:03:48 sean-k-mooney so the auto updating is greate but there are also dragons lurking there
16:05:09 efried right, like I said, that case is accounted for (at least I definitely remember this code in the xen vgpu patches) where, if a card goes away, but has allocations against it, we set reserved=total so no futher allocations can be made against it, until all the existing allocations go away, at which point we can delete the provider.
16:06:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528
16:06:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239
16:06:12 sean-k-mooney ya so we have solution for this edgecase and likely other but its a little more involved then you would assume at first glance
16:06:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529
16:06:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240
16:06:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
16:06:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459
16:06:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Record requester in the InstancePCIRequest https://review.openstack.org/625310
16:06:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421
16:06:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ensure that bandwidth and VF are from the same PF https://review.openstack.org/623543
16:06:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add pf_interface_name tag to passthrough_whitelist https://review.openstack.org/625311
16:06:17 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360
16:07:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360

Earlier   Later