| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-04 | |||
| 12:38:38 | bauzas | jaypipes: around ? | |
| 12:39:10 | bauzas | jaypipes: I discussed with sahid about the vGPU spec | |
| 12:39:28 | bauzas | jaypipes: and we discussed about a specific question | |
| 12:39:45 | jaypipes | bauzas: yep, go ahead | |
| 12:39:56 | bauzas | jaypipes: say the virt driver is providing an inventory for VGPU:8 | |
| 12:40:16 | bauzas | jaypipes: then the scheduler would allocate a VGPU:1 | |
| 12:40:45 | bauzas | jaypipes: if so, the compute service would ask to get a vGPU to the virt driver | |
| 12:41:17 | bauzas | jaypipes: so, it would be the virt driver that would look at which MDEV to use ? | |
| 12:42:01 | jaypipes | bauzas: well, since QEMU/KVM is the only one that understands mdev currently, yes. | |
| 12:42:20 | jaypipes | bauzas: but placement has no concerns about that. | |
| 12:42:56 | bauzas | okay, say I have multiple children RPs (because for example of multple GPUs) | |
| 12:43:07 | jaypipes | bauzas: placement deals with UUIDs, of course. so it's up to the virt driver (or the generic device manager in the future once efried hurries up and finishes that work please) to map UUIDs of resource providers to internal names/identifiers of actual devices. | |
| 12:43:22 | jaypipes | bauzas: or pGPU *groups* in the case of Xen, but yes. | |
| 12:44:13 | bauzas | jaypipes: okay, so it would be the virt driver who would mapping in between RP UUIDs and specific virt identifiers ? | |
| 12:44:56 | bauzas | say I have two children UUIDs, we would be passing the one that was allocated to the virt driver, so it would use one specific mdev or PCI device ? | |
| 12:45:03 | jaypipes | bauzas: yep, and that's the update_provider_tree() virt driver method that we've been discussing at the PTG and since. | |
| 12:45:22 | bauzas | ok | |
| 12:45:37 | jaypipes | bauzas: again, the scheduler doesn't know or care what the PCI device address or mdev identifier is. | |
| 12:45:52 | jaypipes | bauzas: it will just be passing a UUID in the allocation request. | |
| 12:45:58 | bauzas | jaypipes: yup I know about that, hence my discussion with sahid | |
| 12:46:11 | jaypipes | bauzas: and the virt driver (or generic device manager in future) is responsible for looking up that UUID to a known device. | |
| 12:46:44 | jaypipes | bauzas: I just wish efried would get off his ass and finish the generic device manager already. sheeesh. ;) | |
| 12:47:10 | bauzas | jaypipes: okay I'm cool with that, that's just different from the existing, where we have some specific tracker that passes the specific virt device to the virt driver | |
| 12:47:20 | jaypipes | bauzas: yes, understood. | |
| 12:47:32 | bauzas | roger. | |
| 12:48:04 | jaypipes | bauzas: same | |
| 12:48:09 | bauzas | jaypipes: looks like something was misundertood in the ML thread, but I'll leave sahid explain it | |
| 12:48:52 | jaypipes | bauzas: again, it's not that we're not responding on the ML. it's that we have priority items we're trying to complete before sahid's patch series. | |
| 12:49:08 | bauzas | from what I understand, for Queens, we would just map in the driver between the virt device (here, mdevs) and the RP UUID, that's it | |
| 12:54:24 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/ocata: Updated from global requirements https://review.openstack.org/490256 | |
| 12:58:14 | bauzas | jaypipes: follow-up thoughts | |
| 12:58:57 | bauzas | jaypipes: since we create allocations by the scheduler, we could imagine an allocation for VGPU:1 | |
| 12:59:03 | gmann | Nova API meeting in 5 min on #openstack-meeting-4 channel | |
| 12:59:29 | bauzas | jaypipes: then, how the virt driver is knowing that we asked for that specific resource and should therefore use an existing mdev ? | |
| 12:59:54 | bauzas | because we're passing the flavor and we introspect it ? | |
| 13:01:08 | gmann | takashin: can you check my comment in - https://review.openstack.org/#/c/459483/33 | |
| 13:01:32 | gmann | takashin: i feel we should return 'type' always | |
| 13:01:53 | takashin | gmann: Thank you for your review. | |
| 13:02:08 | takashin | gmann: I will check your comment. | |
| 13:02:30 | gmann | takashin: thanks. i am reviewing tempest patch also, hope we get all these in soon. | |
| 13:03:15 | gmann | takashin: got that while checking the schema where you added 'type' as required param | |
| 13:09:54 | jaypipes | bauzas: sorry, reading back... electricians at my house | |
| 13:11:53 | mriedem | still looking for a final +2 on this pike regression fix https://review.openstack.org/#/c/507938/ | |
| 13:12:05 | mriedem | bauzas: ^ is in your wheelhouse | |
| 13:12:52 | bauzas | mriedem: oh coolness | |
| 13:14:24 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase https://review.openstack.org/499539 | |
| 13:14:25 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: factor out compute service start in ServerMovingTest https://review.openstack.org/503037 | |
| 13:14:25 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test resource allocation during soft delete https://review.openstack.org/495159 | |
| 13:15:26 | bauzas | mriedem: if you remember, I also have https://review.openstack.org/#/c/481116/ that is related | |
| 13:15:36 | bauzas | I need to rebase it | |
| 13:17:09 | bauzas | jaypipes: np, my 2nd question is more about how you see the virt driver consuming that specific allocation | |
| 13:17:25 | bauzas | jaypipes: that will be set by the scheduler | |
| 13:18:02 | bauzas | jaypipes: for the moment, we can just do things in the virt driver by introspecting the flavor and see that if we have VGPU:1 in the flavor, we need to hook up a mdev device | |
| 13:18:22 | bauzas | jaypipes: but that's not a super clear interface | |
| 13:18:40 | bauzas | note that the problem is identical for Xen | |
| 13:18:50 | bauzas | with the slight detail it doesn't use mdevs, of course | |
| 13:19:27 | sahid | bauzas, jaypipes: without talking about mdev, how that is going to work for the physical devices, which is on what you are working i think | |
| 13:19:52 | mriedem | maciejjozefczyk: question about https://review.openstack.org/#/c/494973/ - it was listed as Partial-Bug fix in the commit message but why? is there more to fix in that bug? | |
| 13:19:59 | jaypipes | bauzas: k, so the way we've been talking about that is that on startup, the generic device manager (or virt driver) would go through the host devices it discovers and populate a ProviderTree object supplied to it by the RT. The ProviderTree allows looking up providers by UUID or by name. So, when creating nodes in the ProviderTree, the device manager / virt driver would create the resource provider with a UUID and a unique name. It would then | |
| 13:19:59 | jaypipes | look up resource providers by UUID or by name later on | |
| 13:20:54 | bauzas | jaypipes: that I understood | |
| 13:21:10 | bauzas | jaypipes: how we consume that ? by passing the allocation down to the compute ? | |
| 13:22:35 | jaypipes | bauzas: yes, the allocation request contains the resource provider UUID(s) of the providers providing resources for an instance. If the scheduler claimed one VGPU resource against a particular device represented by UUID1, the allocation request will contain UUID1: {resources: {VGPU: 1}}} | |
| 13:23:19 | jaypipes | bauzas: and it will be up to the virt driver or generic device manager to look up which device corresponds to which UUID. | |
| 13:23:25 | gmann | sdague: mriedem need your eyes and feedback on policy removal list in this spec - https://review.openstack.org/#/c/508101/ | |
| 13:23:31 | bauzas | jaypipes: so that requires https://review.openstack.org/#/c/486215 | |
| 13:24:17 | bauzas | jaypipes: if we say we're just going to do a quick POC for Queens with vGPUs, we won't have that yet, so permission to just introspect the flavor in the virt driver ? | |
| 13:24:19 | jaypipes | bauzas: not really, no. The compute host looks up allocations for an instance at the moment by doing a GET /allocations/{consumer_uuid} in the RT | |
| 13:24:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: use already loaded BDM in instance. |
|
| 13:24:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: use already loaded BDM in instance. |
|
| 13:24:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: use already loaded BDM in instance.create https://review.openstack.org/483969 | |
| 13:24:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: doc: fix flavor notes https://review.openstack.org/509438 | |
| 13:24:29 | jaypipes | bauzas: the alternate hosts thing is for retry ability. | |
| 13:24:32 | bauzas | jaypipes: oh that's right | |
| 13:24:38 | bauzas | we don't need to pass them over RPC | |
| 13:24:59 | bauzas | because compute is already looking up them | |
| 13:25:22 | bauzas | okay, so the RT knows the allocation, it can then passes the allocation to the virt driver, right? | |
| 13:25:31 | bauzas | jaypipes: ^ | |
| 13:25:34 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Account for compute.metrics.update in legacy notification whitelist https://review.openstack.org/509439 | |
| 13:25:59 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Account for compute.metrics.update in legacy notification whitelist https://review.openstack.org/509440 | |
| 13:26:02 | jaypipes | bauzas: yes. | |
| 13:26:09 | bauzas | gotcha | |
| 13:26:11 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Account for compute.metrics.update in legacy notification whitelist https://review.openstack.org/509441 | |
| 13:26:46 | bauzas | jaypipes: sahid: I think it's enough for us to be able to assign vGPUs for Queens | |
| 13:27:09 | mriedem | sdague: how do you feel about this backport https://review.openstack.org/#/c/505546/ ? | |
| 13:27:12 | jaypipes | bauzas, sahid: if the virt driver (or generic device manager in the future) wants to store that mapping of mdev identifier to resource provider UUID in a DB table (pci_devices?), cool. If it wants to store it in etcd, cool. An inventory.yaml file on the host? also cool, doesn't matter to me :) | |
| 13:27:24 | bauzas | sahid: the virt driver reports how many vGPUs it can assign by filling in the get_inventory() method | |
| 13:27:40 | mriedem | sdague: i think it's ok, it's adding the ability to specify certs when talking to keystone for the os-quota-sets and flavor-access APIs | |
| 13:28:17 | bauzas | sahid: then at the creation time, the RT would pass the allocated claim (I mean the allocation record) to the virt driver so it would get the RP UUID and the amount to consume | |
| 13:28:31 | sahid | jaypipes: it' good point, since currently libvirt is using PciDevice, so it will be a good transition then | |
| 13:28:37 | bauzas | then the virt driver would allocate from the pool of physical devices it manages | |
| 13:28:55 | jaypipes | sahid: yup | |
| 13:29:40 | jaypipes | bauzas: no, not really... the device will have already been picked by the scheduler. all the virt driver would need to do is plumb the specific device to the guest. | |
| 13:29:53 | bauzas | jaypipes: sahid: I guess the most important matter is that whatever the technical persistence is, it's not provided outside of the virt driver | |
| 13:30:03 | jaypipes | bauzas: and that's the point we're trying to get to. the RT and scheduler do the claiming/allocating resources stuff and the virt driver does the guest plumbing. | |
| 13:30:23 | bauzas | jaypipes: if we pass VGPU:8 as a RC | |
| 13:30:41 | bauzas | jaypipes: then the allocation would be against the root RP | |
| 13:30:45 | bauzas | for queens I mean | |
| 13:30:57 | jaypipes | bauzas: no, not necessarily. | |
| 13:31:08 | bauzas | I'm all ears :) | |