| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 13:44:11 | bauzas | changed* | |
| 13:44:14 | efried | alex_xu_: Oh, but we do :) | |
| 13:44:53 | bauzas | or, we magically provide a 2nd level of the tree with nested RPs, meaning that a pGPU will have multiple children, each one being something like pGPU_thistype | |
| 13:45:30 | efried | bauzas: That only helps if you want to lock it down (predefine and preallocate) | |
| 13:45:36 | bauzas | the second solution makes the thing less impactful for operators, but it adds a second level to the tree just for VGPUs | |
| 13:45:47 | bauzas | efried: not really | |
| 13:46:03 | bauzas | efried: because at start, each inventory will provide all the possible vGPUs | |
| 13:46:24 | efried | bauzas: But you'd still be counting on an allocation from *here* affecting the inventory over *there* | |
| 13:46:30 | efried | which we can't do. | |
| 13:46:40 | bauzas | efried: in theory the allocation would be against the pGPU_type | |
| 13:46:46 | bauzas | not the pGPU itself | |
| 13:47:02 | efried | Yeah, I get that, but you would have had to pre-designate how many of each type you start with. | |
| 13:47:21 | bauzas | efried: that's the config option | |
| 13:47:23 | efried | You can't show "full" inventory for all types. | |
| 13:47:35 | efried | Okay, right, so swhat I'm saying, it's locked down ahead of time. | |
| 13:47:40 | efried | via the config option | |
| 13:47:47 | bauzas | for the moment, the config option is a ListOpt supporting types | |
| 13:48:15 | bauzas | but honestly, I think we should do something less crazy and just have an option that would do like pci passthrough | |
| 13:48:28 | bauzas | ie. "for that PCI device, here is the type" | |
| 13:48:51 | bauzas | jaypipes: thoughts on that ? ^ | |
| 13:48:58 | efried | bauzas: If you want ultimate flexibility, I think you need to provide inventory of some RC that lets you represent "units of VGPU-ness", where different types consume a different number of that resource. And then your request would have to be well-behaved and ask for resources=VGPU:1,VGPU_UNITY_THINGY:4 | |
| 13:49:26 | efried | ...where the request for VGPU_UNITY_THINGY has to be correct for the type you're requesting. | |
| 13:49:53 | openstackgerrit | Merged openstack/os-traits master: Updated from global requirements https://review.openstack.org/551599 | |
| 13:50:04 | bauzas | efried: that's where I think we're over engineering | |
| 13:50:22 | efried | That would be clearer as | |
| 13:50:22 | efried | resources=VGPU_TYPE_X:1,VGPU_UNITY_THINGY:4 | |
| 13:50:22 | efried | or | |
| 13:50:22 | efried | resources=VGPU:1,VGPU_UNITY_THINGY:4&required=VGPU_TYPE_X | |
| 13:50:25 | bauzas | efried: operators want flavors like VGPU=2&trait=MY_TYPE | |
| 13:50:58 | bauzas | yeah, honestly, I feel for the short term a config option that would do a whitelist seems the most acceptable solution | |
| 13:51:00 | efried | bauzas: Yeah, I get that. This isn't the only place we've seen where it would be useful to provide a translation layer from the flavor to the actual placement request. | |
| 13:51:38 | efried | bauzas: I think perhaps we're oversimplifying/idealizing by thinking that a direct mapping of flavor to placement request is going to allow us to cover everything. | |
| 13:52:06 | bauzas | let's incrementally try to resolve the usecase | |
| 13:52:20 | bauzas | for queens, we supported a single type | |
| 13:52:30 | efried | bauzas: Basically, requiring the admin to understand the nuances and quirks of both the provider tree as modeled by the virt driver, and the syntax and semantics of the placement API query. | |
| 13:52:48 | bauzas | efried: that's where I think a whitelist could be more understandable | |
| 13:53:09 | bauzas | efried: we could have enabled_vgpu_types that would keep the existing types we agree | |
| 13:53:23 | bauzas | and then a second option that would tell for which PCI ID which type | |
| 13:53:42 | efried | bauzas: I'm okay with that idea in general, but I'm going to be watching like a hawk to make sure we retain the separation of platform-specific syntax. | |
| 13:53:51 | bauzas | in that case, the PGPU inventory would be of one type, problem solved. | |
| 13:53:51 | efried | E.g. "NO PCI ID!" | |
| 13:54:27 | bauzas | efried: the PCI ID thingy is just within the driver | |
| 13:54:35 | bauzas | no crazypants about PCI tracking | |
| 13:54:56 | bauzas | litterally ask libvirt to pick that type for that pGPU | |
| 13:55:16 | efried | bauzas: If there's a PCI ID in the file, then we have to say that the file gets parsed by virt alone. Kind of thing. | |
| 13:55:48 | bauzas | efried: yeah, zactly that | |
| 13:56:04 | efried | bauzas: So... you want to do something like this for Rocky? | |
| 13:56:08 | bauzas | I guess | |
| 13:56:17 | bauzas | if nested RPs is a thing :) | |
| 13:56:22 | efried | bauzas: Cause this is edging unequivocally into Generic Device Management territory. | |
| 13:56:34 | alex_xu_ | bauzas: when your request VGPU_thistype, how do you change total=0 for the VGPU_thosetype? | |
| 13:56:48 | bauzas | alex_xu_: that's magically done by sysfs | |
| 13:57:03 | bauzas | alex_xu_: which I'm using for getting the inventory | |
| 13:57:05 | alex_xu_ | bauzas: but how the placement to know that | |
| 13:57:20 | bauzas | alex_xu_: because we provide inventories of VGPU resouces as of queens :) | |
| 13:57:27 | efried | bauzas: Wait, you're talking about doing that at setup time, not at allocation time, right? | |
| 13:57:28 | bauzas | that are populated based on sysfs :) | |
| 13:58:12 | bauzas | efried: alex_xu_ is talking of the case where we would have pGPU types as children | |
| 13:58:14 | alex_xu_ | bauzas: if there are two requests at same time. One for VGPU_thistype, another for VGPU_thosetype. So there will be race case. Only one can successful in the host finally | |
| 13:58:18 | efried | bauzas: Bricks will be shat by several people if you start talking about modifying inventory of Y because X got allocated. | |
| 13:58:32 | bauzas | alex_xu_: yeah I considered the race condition | |
| 13:58:41 | bauzas | alex_xu_: and that's actually a good call for not doing that | |
| 13:58:52 | bauzas | but rather doing a whitelisting on the virt driver direcrtly | |
| 13:59:07 | alex_xu_ | bauzas: so limit only one type in each host? | |
| 14:00:27 | bauzas | alex_xu_: no, one type per physical GPU | |
| 14:00:41 | alex_xu_ | ah, i got it | |
| 14:00:42 | bauzas | one type per host is already here | |
| 14:00:59 | bauzas | except the fact we don't use traits atm | |
| 14:01:29 | bauzas | anyway, will just propose something soon so we could see if that requires some spec | |
| 14:01:43 | bauzas | or at least some spec amendment | |
| 14:01:50 | efried | bauzas: Right, so you're setting up the types & inventories the first time you load up. The first time update_provider_tree runs, it reads that config file and sets up the provider tree with types & inventories according to what the admin put in there. Right? | |
| 14:01:59 | bauzas | ++ | |
| 14:02:12 | efried | Cool cool | |
| 14:02:16 | alex_xu_ | the race condidates only happened one time, if people accept that, it also ok | |
| 14:02:20 | bauzas | efried: https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/add-support-for-vgpu.html describes that partially | |
| 14:02:34 | jaypipes | guh, it would have been nice to have more than a few hours of sleep before spec sprint day... :( | |
| 14:02:41 | efried | bauzas: I can also see a solution that goes halfway in the future... | |
| 14:02:55 | bauzas | jaypipes: take caffeine or drugs, either. | |
| 14:03:32 | bauzas | jaypipes: FWIW, https://review.openstack.org/#/c/541290/7/specs/rocky/approved/numa-aware-vswitches.rst will require me some paracetamol | |
| 14:03:36 | bauzas | stephenfin: ^ ;) | |
| 14:03:50 | cdent | jaypipes: start on an easy one: https://review.openstack.org/#/c/418393/ | |
| 14:03:52 | edleafe | jaypipes: have you tried the blue meth? | |
| 14:04:02 | bauzas | always take the blue pill | |
| 14:04:04 | stephenfin | I hear the blue meth is real good | |
| 14:04:13 | bauzas | never choose the red one | |
| 14:04:35 | efried | bauzas: In a fashion similar to neutron port creation, the user could create the VGPU resource before doing the boot request. The VGPU creation would eventually hit vendor-specific code (maybe virt, maybe some agent - cyborg?) which actually *would* tweak the inventories and available types. Then you would attach that VGPU to your boot request, just like you would a port. | |
| 14:05:03 | alex_xu_ | efried: any legal drugs in Florida? | |
| 14:05:26 | efried | alex_xu_: Legal schmegal. But I'm in Texas. | |
| 14:05:30 | bauzas | efried: creating the mediated devices in advance is already a possibility | |
| 14:05:32 | efried | In Florida, they wash up on the beach. | |
| 14:08:46 | kashyap | alex_xu_: Hey there | |
| 14:09:12 | kashyap | alex_xu_: About your comment here: https://review.openstack.org/#/c/534384/16/nova/tests/unit/virt/libvirt/test_config.py | |
| 14:09:21 | efried | bauzas: But having that creation impact placement inventories would be new. | |
| 14:09:34 | bauzas | efried: no, it's already the case | |
| 14:09:42 | kashyap | alex_xu_: The test you pointed out is slightly different in that, it is testing: 'obj.extra_flags' | |
| 14:09:43 | efried | oh? | |
| 14:09:57 | bauzas | efried: if you create a mediated device, libvirt will just add that to the total | |
| 14:10:11 | bauzas | I should write a blogpost on this... | |
| 14:10:34 | bauzas | because creating a mediated device doesn't mean you *use* it for a guesrt | |
| 14:10:53 | bauzas | so, placement is getting an inventory of 'available+created' as a total already | |