| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-31 | |||
| 18:43:38 | dansmith | clear as mud? :) | |
| 18:43:39 | dansmith | jaypipes may have a better idea of how that will actually shake out long term, | |
| 18:43:48 | dansmith | but regardless, placement will only ever be counting things, not choosing things | |
| 18:44:07 | efried | dansmith Yeah, this is a shared jaypipes dream. | |
| 18:44:19 | efried | Lest you should think it was just efried's pipe dream | |
| 18:45:49 | efried | Okay, so there's gonna be a generic device passthrough management module of some sort that sits in nova and coordinates among placement, scheduler, and virt driver to do whitelisting, claims, allocations | |
| 18:46:04 | dansmith | well, there is that today :) | |
| 18:46:24 | efried | Just for PCI, right? | |
| 18:46:29 | dansmith | (sans the telling-placement-about-it) thing | |
| 18:46:55 | dansmith | ...just for pci.... aren't we talking about pci here? | |
| 18:47:23 | efried | Well, that's one of Jay's bugbears: it should work the same for any device, not just PCI. | |
| 18:47:38 | dansmith | um | |
| 18:47:43 | dansmith | I'm confused then | |
| 18:47:51 | efried | Which intersects with one of my bugbears, which is: not every device has a PCI address. | |
| 18:48:13 | dansmith | GPU devices need not be managed as raw PCI devices for this I think | |
| 18:48:35 | dansmith | maybe that's what you mean | |
| 18:48:55 | dansmith | so I think eventually you should be able to say I want VGPU=1, traits=nvidia,gen5 | |
| 18:49:06 | dansmith | and placement will pick you a host that has one of those available | |
| 18:49:36 | dansmith | when you get to the host, that compute node will tell the virt driver (which I think will be generalizing gpus, at least in the case of libvirt), to give it a gpu, which it'll assign to your guest | |
| 18:49:41 | artom | efried, is not every device having a PCI address a problem? My understanding is that the compute reports its resources (devices) to placement | |
| 18:49:50 | artom | How it figures out what resources it has is up to it | |
| 18:50:03 | dansmith | artom: it doesn't report which resources it has, it reports how many of each it has | |
| 18:50:32 | artom | dansmith, ah, right, thanks | |
| 18:50:45 | efried | Right right. So there's the gap. Right now, artom, not having a PCI address *is* a pretty serious problem. | |
| 18:50:51 | openstackgerrit | Lucian Petrut proposed openstack/nova master: HyperV: Perform proper cleanup after failed instance spawns https://review.openstack.org/499690 | |
| 18:50:56 | artom | So for PCI devices it can use addresses, for other stuff, or PCI devices that don't have addresses for whatever reason, it can use a different mechanism | |
| 18:51:15 | efried | Because the existing PCI device management subsystem expects every device it gets from the virt driver to have a PCI address in domain:bus:slot.func format. | |
| 18:51:22 | artom | efried, ah, it's a problem because we can't manage them with the current way of doing things? | |
| 18:51:37 | dansmith | for devices we can generalize (like a GPU or NIC) we can talk about what we want in terms of quantity and traits | |
| 18:51:56 | dansmith | we can kindof ask for that in the flavor today, | |
| 18:52:14 | dansmith | but we're not exposing things like GPUs from things like libvirt yet, | |
| 18:52:18 | dansmith | which needs to happen | |
| 18:53:18 | efried | "ask for that in the flavor today" - are you talking about pci_passthrough:alias or something else? | |
| 18:53:23 | dansmith | no, | |
| 18:53:34 | dansmith | in the flavor, you can do things like resources:VGPU=1 | |
| 18:53:47 | dansmith | and placement will not consider hosts that don't expose a VGPU resource type that has at least one available | |
| 18:53:54 | dansmith | nothing does right now, but you can ask for it | |
| 18:55:00 | efried | heh, okay. But we don't have any accomodation for resources:PCI_DEVICE=1,traits=whatever ? | |
| 18:55:19 | dansmith | well, that doesn't make sense | |
| 18:55:26 | dansmith | you wouldn't ask for one pci device, any pci device | |
| 18:55:42 | dansmith | like "give me a nic, or a sata controller, or a serial UART, I'll take anything!" | |
| 18:55:58 | efried | No, I agree that's too broad. | |
| 18:56:09 | artom | Maybe they're working on the pci kernel subsystem ;) | |
| 18:56:24 | dansmith | but you might ask for a NIC or a NIC_PF or a NIC_VF I thnk | |
| 18:56:41 | efried | And I *think* I see that we can accomodate broad groupings | |
| 18:56:58 | efried | But I'm concerned that we can't accomodate narrow groupings. | |
| 18:57:11 | efried | In particular, what if I want a specific device? | |
| 18:57:13 | dansmith | jay might argue that we should ask for a NIC=1, traits=definiteily_a_pf | |
| 18:57:30 | artom | efried, define 'specific device'? | |
| 18:57:44 | dansmith | efried: like "give me device 00:01:5f on host $foo" ? | |
| 18:57:44 | efried | "device with UUID X" | |
| 18:57:49 | artom | Because the 'non-cloud way of doing things' argument will come up really fast with that one ;) | |
| 18:57:51 | dansmith | because that's not a thing we want to do | |
| 18:57:53 | dansmith | yeah | |
| 18:57:56 | efried | heh | |
| 18:57:56 | dansmith | reeeeeal fast :) | |
| 18:58:13 | efried | Well, okay, let me put it a different way. | |
| 18:58:41 | efried | I want my aliasing to be able to create a group of specific devices. | |
| 18:58:56 | openstackgerrit | Merged openstack/nova master: Fix _delete_inventory log message in report client https://review.openstack.org/498833 | |
| 18:58:58 | dansmith | you mean like we can do with pci today | |
| 18:59:04 | efried | EXCEPT | |
| 18:59:07 | dansmith | where you choose a vendor and model, and we gather up all those things | |
| 18:59:11 | efried | today we can only do it by product/vendor ID. | |
| 18:59:23 | efried | And only a single product/vendor ID pair. | |
| 18:59:31 | dansmith | right, because that's the industry standard | |
| 18:59:47 | efried | nono, this isn't a weirdo platform thing anymore. | |
| 18:59:57 | efried | I have two NICs, exact same vendor/product IDs | |
| 19:00:02 | efried | But they're attached to different networks. | |
| 19:00:08 | efried | I don't want them in the same alias group | |
| 19:00:15 | dansmith | right, so you ask for a nic attached to network foo, | |
| 19:00:22 | dansmith | not "give me the fifth nic from the left" | |
| 19:00:37 | dansmith | NIC=1, traits=network-public | |
| 19:00:38 | dansmith | or something | |
| 19:00:51 | efried | Right, got it. | |
| 19:01:10 | artom | So that would put potentially mutable deployment details into the nova.conf on the compute? | |
| 19:01:26 | artom | That seems... unwise | |
| 19:01:31 | dansmith | artom: well, that's kinda part of the problem with the pci stuff today, | |
| 19:01:40 | dansmith | but the nic attachment thing is managed dynamically today I think | |
| 19:01:52 | dansmith | like we have some way of correlating actual nics to what physical network they're attached to | |
| 19:02:01 | efried | For SR-IOV we do | |
| 19:02:05 | dansmith | right | |
| 19:02:21 | efried | pci_passthrough_devices lists a physical_network on a PF | |
| 19:02:28 | dansmith | yeah | |
| 19:02:38 | efried | But that's a single, specific attr | |
| 19:03:10 | dansmith | artom: so yeah if someone goes and moves the yellow cable to the blue port in the datacenter, then bad things happen, but that's kinda the risk you run with giving people hardware passthrough It hink | |
| 19:03:29 | efried | So with generic device management, you could have arbitrary traits, one of which is physical_network | |
| 19:03:44 | artom | Yeah, and now that I think about it more, there's no way to avoid it really | |
| 19:03:51 | artom | No sane way, at any rate | |
| 19:04:06 | dansmith | efried: yeah like maybe you'd want has-tls-offload on your nic as well as physnet-private37 | |
| 19:04:11 | artom | Unless you want to start having admin REST APIs where you associate device addresses with what network they're connected to | |
| 19:04:41 | efried | So qualitative attributes are handle via traits; and in the example of physical_network, I sorta have to have the key and value both stuffed into the value? | |
| 19:05:02 | efried | Like, CUSTOM_NETWORK_MYNET would be a trait? | |
| 19:05:03 | dansmith | efried: well, you have to arrange things as traits, not k=v | |
| 19:05:06 | dansmith | yes | |
| 19:05:18 | efried | And I'd have to have a separate one for every network. | |
| 19:05:29 | efried | Okay; so how are qualitative attrs handled? | |
| 19:05:42 | efried | Like, "give me a VGPU >= 10GHz" | |
| 19:05:52 | dansmith | we don't do that | |
| 19:05:55 | artom | Actually, in that situation, wouldn't the physical network become a resource, not a trait? | |
| 19:06:01 | efried | sorry ^ quantitative | |