| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-04 | |||
| 13:28:31 | sahid | jaypipes: it' good point, since currently libvirt is using PciDevice, so it will be a good transition then | |
| 13:28:37 | bauzas | then the virt driver would allocate from the pool of physical devices it manages | |
| 13:28:55 | jaypipes | sahid: yup | |
| 13:29:40 | jaypipes | bauzas: no, not really... the device will have already been picked by the scheduler. all the virt driver would need to do is plumb the specific device to the guest. | |
| 13:29:53 | bauzas | jaypipes: sahid: I guess the most important matter is that whatever the technical persistence is, it's not provided outside of the virt driver | |
| 13:30:03 | jaypipes | bauzas: and that's the point we're trying to get to. the RT and scheduler do the claiming/allocating resources stuff and the virt driver does the guest plumbing. | |
| 13:30:23 | bauzas | jaypipes: if we pass VGPU:8 as a RC | |
| 13:30:41 | bauzas | jaypipes: then the allocation would be against the root RP | |
| 13:30:45 | bauzas | for queens I mean | |
| 13:30:57 | jaypipes | bauzas: no, not necessarily. | |
| 13:31:08 | bauzas | I'm all ears :) | |
| 13:31:39 | jaypipes | bauzas: we're aiming to get n-r-p work done in Queens. so, the allocation would be against one or more child providers (in libvirt, those would be pGPUs, in Xen they would be pGPU groups). | |
| 13:32:53 | jaypipes | bauzas: so all the virt driver would be responsible for doing is a) looking up physical device information by resource provider UUID and b) doing the necessary guest plumbing for the device (in other words, in libvirt's case, writing the XML snippet information for the device, etc) | |
| 13:33:01 | bauzas | I agree, I just thought we said we could try to provide GPU resources as a global resource class for the node in Queens | |
| 13:33:41 | bauzas | if nested-RPs is already there, then of course we would modify that to just lookup the child RP | |
| 13:34:13 | dansmith | bauzas: yes that's wht we should do | |
| 13:34:27 | dansmith | bauzas: we can expose gpu resources for the compute node right now | |
| 13:36:26 | sahid | efried: please ping me when you have a moment so we can talk about your work on the generic devices management | |
| 13:36:29 | efried | jaypipes GDM dig accepted | |
| 13:36:40 | jaypipes | efried: lol :) | |
| 13:36:42 | efried | sahid Now's good, unless I need to catch up on the ML first. | |
| 13:37:18 | efried | jaypipes BUT - I've actually been thinking along the lines that, once NRP is in place, there will be no need for such a thing as a GDM. | |
| 13:37:35 | sahid | i did not expected to see you respond so quickly :) | |
| 13:37:48 | maciejjozefczyk | mriedem: Yes other patch should be applied to rollback migration when delete is called, similiar to: https://review.openstack.org/#/c/185958/ | |
| 13:38:16 | maciejjozefczyk | mriedem: I solved an effect of broken migration, not the source | |
| 13:38:29 | jaypipes | efried: oh, there still will be. | |
| 13:38:42 | jaypipes | efried: there's still a need for discovery of hardware on the host. | |
| 13:38:48 | efried | virt driver | |
| 13:38:58 | sahid | efried: do you have some pointers of work in progress? | |
| 13:39:00 | efried | whitelisting? virt driver | |
| 13:39:04 | jaypipes | efried: doing so would probably lead to a lot of dup code. | |
| 13:39:17 | efried | Mapping devices to RPs? virt driver. | |
| 13:39:49 | efried | sahid The only "work in progress" is scribbles on etherpads, which we discussed at the PTG. | |
| 13:40:09 | sahid | efried: yes i was not here, if you can give me the link | |
| 13:40:32 | efried | sahid https://etherpad.openstack.org/p/nova-ptg-queens-generic-device-management | |
| 13:40:49 | efried | sahid It has generally been a drive towards understanding how devices are going to be managed once we go full-bore with placement & resource providers | |
| 13:40:59 | efried | sahid The goal being to get rid of the existing PCI manager code. | |
| 13:41:25 | mriedem | gmann_sleep: doesn't it seem odd that we still have this filtering code when listing instances to be able to filter by metadata and system_metadata? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2328-L2332 | |
| 13:41:32 | mriedem | i thought that was a 400 in the API now | |
| 13:41:56 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L182 | |
| 13:42:14 | efried | jaypipes Potentially duplication among the linuxy hypervisors, I suppose. I would still expect the code to ultimately run under the auspices of the virt driver. | |
| 13:42:15 | sahid | efried: no transition phase as suggested jaypipes by adding a update_from_inventory() method to the PciManager? | |
| 13:43:15 | efried | jaypipes So I could see having some shared class above the ComputeDriver base class that provides linuxy impls for devicey methods. | |
| 13:44:15 | efried | sahid Well, the cores have made their position pretty clear: Investment in the existing PCI manager is going to be very limited. | |
| 13:46:19 | efried | sahid That said, I think it may be possible to get very close to what you want just using NRP with traits and careful modeling, which is happening in Queens. | |
| 13:46:44 | efried | sahid Fancy use cases like (anti)affinity won't work yet. | |
| 13:46:49 | jaypipes | efried: yeah, we're gonna need to have some sort of transitionary plan anyway... | |
| 13:47:33 | efried | jaypipes Oh, like sahid is saying with update_from_inventory() - presumably a RT method that peels dev info out of the virt inventory and flushes it back to the PCI manager? | |
| 13:47:54 | jaypipes | efried: chatting with dansmith and bauzas, I'm cool with trying to get single-inventory VGPU support pushed for Queens if we miss nested resource provider targets. That would means no support for multiple GPU types on a single compute node, but would at least get some rudimentary VGPU support | |
| 13:49:08 | dansmith | jaypipes: multiple gpu types can be handled with aggregates until we have richer support, and at least one customer has told me that's fine, FWIW | |
| 13:49:18 | jaypipes | k | |
| 13:49:18 | bauzas | efried: I just wanted to clarify how we would translate a specific {'VPGU:1"} request into something very virt-specific | |
| 13:49:40 | bauzas | heh VGPUs even | |
| 13:49:41 | sahid | it seems that XenServer provides an abstraction which makes easy what you want to achieve | |
| 13:49:48 | sahid | but that does not look reasonable for libvirt | |
| 13:50:05 | efried | bauzas So the only way we're going to translate an allocation to a specific device is if the RP is modeled as representing that specific device. | |
| 13:50:31 | sahid | XenServer is doing the managmenet ofr the devices, libvirt is using the PciManager | |
| 13:50:56 | bauzas | efried: sahid: dansmith: I just feel we need code so we could chat on the details | |
| 13:51:09 | bauzas | reporting the inventory for those resource classes is easy | |
| 13:51:21 | efried | The virt driver will ultimately be responsible for designing that model, maintaining that mapping, and providing the appropriate RPs and inventory to placement (via get_inventory, or update_provider_tree, or whatever it winds up being) | |
| 13:51:25 | dansmith | bauzas: you're asking how the virt driver knows a thing has been requested? | |
| 13:51:44 | bauzas | dansmith: yup, correct | |
| 13:51:57 | dansmith | bauzas: in the simple case it can just look at extra_specs on instance.flavor | |
| 13:52:00 | bauzas | dansmith: jaypipes told me RT looks up the allocation | |
| 13:52:11 | bauzas | dansmith: yeah that was my initial approach | |
| 13:52:17 | dansmith | bauzas: we have a routine that can parse the flavor and give you a merged resource view | |
| 13:52:27 | bauzas | that would be an easy thing then | |
| 13:52:33 | bauzas | okay, I need coding then | |
| 13:52:34 | dansmith | or look at the allocation, yeah, but plumbing that from rt to virt will take a little work | |
| 13:52:49 | efried | The virt driver will be able to see the allocation, no? | |
| 13:52:51 | bauzas | yeah, for a POC, introspecting the flavor seems the quickiest path | |
| 13:52:55 | dansmith | you will need the allocation once we have n-r-p or traits though | |
| 13:53:02 | mriedem | these -1s from zuul messing up my dashboard is messing up my life | |
| 13:53:02 | bauzas | efried: it requires a new interface which we don't have yet | |
| 13:53:10 | dansmith | efried: it could fetch it itself, yeah, but better if it didn't I think | |
| 13:53:30 | bauzas | dansmith: yeah, I'm not a fan of the virt driver calling placement | |
| 13:53:36 | dansmith | bauzas: yeah, a good first step would be figuring out the best way to tell the virt driver about the allocation | |
| 13:53:45 | dansmith | bauzas: yep | |
| 13:53:52 | efried | Talking like something in the Instance object? | |
| 13:53:55 | bauzas | anyway, /me coding then | |
| 13:53:59 | dansmith | efried: no | |
| 13:54:08 | efried | param to spawn? | |
| 13:54:09 | dansmith | efried: maybe just a param in spawn | |
| 13:54:11 | dansmith | yeah | |
| 13:54:13 | efried | ight | |
| 13:54:30 | efried | btw, virt will at some point be calling placement. | |
| 13:54:33 | efried | Not necessarily in spawn | |
| 13:54:45 | bauzas | dansmith: jaypipes was thinking of a specific interface for nested RPs | |
| 13:54:57 | bauzas | dansmith: something like update_my_provider_tree() | |
| 13:54:59 | dansmith | efried: you're saying that because of virt drivers reporting resource? | |
| 13:55:01 | efried | but in init_host, and/or get_inventory, and/or update_provider_tree, whatever - to set up the RPs and whatnot. | |
| 13:55:27 | efried | Yeah, for example, to know whether a RP has been created yet. | |
| 13:55:31 | dansmith | efried: that should be abstracted by the compute manager, not virt calling placement directly | |
| 13:55:32 | dansmith | IMHO | |
| 13:55:33 | dansmith | bauzas: yep makes sense | |
| 13:55:41 | efried | dansmith Yeah, I suppose it could be. | |
| 13:55:54 | bauzas | I second dansmith on not having placement calls from the driver | |
| 13:56:02 | efried | This means virt is always responsible for producing RP UUIDs. | |
| 13:56:07 | dansmith | efried: I think we should shoot for that goal, and if there's some compelling reason to break that rule, then we can discuss it | |
| 13:56:14 | efried | dansmith Dig. | |