| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-27 | |||
| 13:31:31 | stephenfin | mikal: Just the man I'm looking for. Fancy jotting down your thoughts on https://review.openstack.org/554195 | |
| 13:31:38 | sahid | the agent can return this information to nova, no? | |
| 13:32:19 | stephenfin | (random aside: the OVS (-DPDK) documentation is hands-down awful. It sucks that we have to resort to random blogs for this critical information) | |
| 13:32:38 | stephenfin | sahid: If it's what we want then I don't see why not. Lemme check | |
| 13:33:42 | stephenfin | ooh, so 'pmd-rxq-show' does seem to be exactly what I wanted | |
| 13:35:06 | stephenfin | ...and we'd just assume that OVS was configured correctly so that there are PMD threads on all NUMA nodes, which is Red Hat's guidance at least | |
| 13:35:18 | stephenfin | sean-k-mooney[m]: If you're about, any thoughts on ^ | |
| 13:36:17 | efried | alex_xu_: Ah, then that's a mistake in modeling. | |
| 13:37:20 | efried | alex_xu_: The RP tree should not be set up such that the compute host provides inventory of FPGA. | |
| 13:37:45 | alex_xu_ | efried: yes, I realized that also, that isn't the FPGA case now | |
| 13:37:49 | bauzas | alex_xu_: efried: which specific spec are you discussing ? | |
| 13:37:59 | stephenfin | efried: See what you made me do? https://review.openstack.org/556850 | |
| 13:38:19 | alex_xu_ | bauzas: here https://review.openstack.org/#/c/554305/ | |
| 13:38:46 | alex_xu_ | efried: basically, it only can happened in the compute node RP, | |
| 13:39:04 | bauzas | alex_xu_: ack, thanks | |
| 13:39:15 | efried | alex_xu_: Well, really any RP in the tree which provides inventory in multiple RCs that are "unrelated" to each other. | |
| 13:39:25 | efried | alex_xu_: This is actually what I wrote a bug about last year. | |
| 13:39:54 | alex_xu_ | efried: ha, currently we have VGPU, but will change soon :) | |
| 13:40:00 | kashyap | "the operator is going to configure OVS and actually DPDK based on where the device is located" --> Hope the Operator has the necessary PhDs to configure DPDK et al | |
| 13:40:33 | alex_xu_ | efried: but I agree with that | |
| 13:40:47 | alex_xu_ | I feel it will be rare case in the future | |
| 13:41:00 | bauzas | alex_xu_: efried: FWIW, I discussed this morning with jianghuaw_ about some possible quirks with VGPUs | |
| 13:41:10 | bauzas | and I'd love your thoughts | |
| 13:41:16 | bauzas | because it would need to spec up | |
| 13:41:36 | bauzas | the fact is that physical GPUs support multiple types | |
| 13:41:51 | bauzas | depending on the type, you can have different number of vGPUs | |
| 13:41:58 | bauzas | so, guess what? | |
| 13:42:12 | efried | alex_xu_: I can't find the bug right offhand, but I think the scenario I described was: compute RP has HDD disk, sharing provider (or in fact another RP in the tree) has SSD disk. I ask for disk & VCPU, I ask for the HDD trait. I will get back candidates which include the SSD, because the compute RP is satisfying the HDD trait, even though it's not providing the disk resource. | |
| 13:42:18 | alex_xu_ | we need to update inventory based the user request? | |
| 13:42:26 | bauzas | alex_xu_: efried: in theory, a nested RP that would be a pGPU could have different inventories for the same resource class | |
| 13:43:12 | bauzas | alex_xu_: efried: but the trick is, once a vGPU is created, then all the other types but the one picked for that vGPU will have total=0 for their inventories | |
| 13:43:30 | efried | bauzas: I think we've discussed similar issues in the past. Trying to recall what we came up with as a solution... | |
| 13:43:35 | bauzas | alex_xu_: efried: so there are 2 ways to consider that | |
| 13:43:53 | bauzas | either we ask the operator to define which type he wants to support per-GPU | |
| 13:44:06 | alex_xu_ | efried: ah, I see, but I guess that bug is killed by the reality that we don't support two different storage on the host | |
| 13:44:08 | bauzas | and in that case, the enabled_vgpu_types option needs to be change | |
| 13:44:11 | bauzas | changed* | |
| 13:44:14 | efried | alex_xu_: Oh, but we do :) | |
| 13:44:53 | bauzas | or, we magically provide a 2nd level of the tree with nested RPs, meaning that a pGPU will have multiple children, each one being something like pGPU_thistype | |
| 13:45:30 | efried | bauzas: That only helps if you want to lock it down (predefine and preallocate) | |
| 13:45:36 | bauzas | the second solution makes the thing less impactful for operators, but it adds a second level to the tree just for VGPUs | |
| 13:45:47 | bauzas | efried: not really | |
| 13:46:03 | bauzas | efried: because at start, each inventory will provide all the possible vGPUs | |
| 13:46:24 | efried | bauzas: But you'd still be counting on an allocation from *here* affecting the inventory over *there* | |
| 13:46:30 | efried | which we can't do. | |
| 13:46:40 | bauzas | efried: in theory the allocation would be against the pGPU_type | |
| 13:46:46 | bauzas | not the pGPU itself | |
| 13:47:02 | efried | Yeah, I get that, but you would have had to pre-designate how many of each type you start with. | |
| 13:47:21 | bauzas | efried: that's the config option | |
| 13:47:23 | efried | You can't show "full" inventory for all types. | |
| 13:47:35 | efried | Okay, right, so swhat I'm saying, it's locked down ahead of time. | |
| 13:47:40 | efried | via the config option | |
| 13:47:47 | bauzas | for the moment, the config option is a ListOpt supporting types | |
| 13:48:15 | bauzas | but honestly, I think we should do something less crazy and just have an option that would do like pci passthrough | |
| 13:48:28 | bauzas | ie. "for that PCI device, here is the type" | |
| 13:48:51 | bauzas | jaypipes: thoughts on that ? ^ | |
| 13:48:58 | efried | bauzas: If you want ultimate flexibility, I think you need to provide inventory of some RC that lets you represent "units of VGPU-ness", where different types consume a different number of that resource. And then your request would have to be well-behaved and ask for resources=VGPU:1,VGPU_UNITY_THINGY:4 | |
| 13:49:26 | efried | ...where the request for VGPU_UNITY_THINGY has to be correct for the type you're requesting. | |
| 13:49:53 | openstackgerrit | Merged openstack/os-traits master: Updated from global requirements https://review.openstack.org/551599 | |
| 13:50:04 | bauzas | efried: that's where I think we're over engineering | |
| 13:50:22 | efried | resources=VGPU:1,VGPU_UNITY_THINGY:4&required=VGPU_TYPE_X | |
| 13:50:22 | efried | or | |
| 13:50:22 | efried | resources=VGPU_TYPE_X:1,VGPU_UNITY_THINGY:4 | |
| 13:50:22 | efried | That would be clearer as | |
| 13:50:25 | bauzas | efried: operators want flavors like VGPU=2&trait=MY_TYPE | |
| 13:50:58 | bauzas | yeah, honestly, I feel for the short term a config option that would do a whitelist seems the most acceptable solution | |
| 13:51:00 | efried | bauzas: Yeah, I get that. This isn't the only place we've seen where it would be useful to provide a translation layer from the flavor to the actual placement request. | |
| 13:51:38 | efried | bauzas: I think perhaps we're oversimplifying/idealizing by thinking that a direct mapping of flavor to placement request is going to allow us to cover everything. | |
| 13:52:06 | bauzas | let's incrementally try to resolve the usecase | |
| 13:52:20 | bauzas | for queens, we supported a single type | |
| 13:52:30 | efried | bauzas: Basically, requiring the admin to understand the nuances and quirks of both the provider tree as modeled by the virt driver, and the syntax and semantics of the placement API query. | |
| 13:52:48 | bauzas | efried: that's where I think a whitelist could be more understandable | |
| 13:53:09 | bauzas | efried: we could have enabled_vgpu_types that would keep the existing types we agree | |
| 13:53:23 | bauzas | and then a second option that would tell for which PCI ID which type | |
| 13:53:42 | efried | bauzas: I'm okay with that idea in general, but I'm going to be watching like a hawk to make sure we retain the separation of platform-specific syntax. | |
| 13:53:51 | efried | E.g. "NO PCI ID!" | |
| 13:53:51 | bauzas | in that case, the PGPU inventory would be of one type, problem solved. | |
| 13:54:27 | bauzas | efried: the PCI ID thingy is just within the driver | |
| 13:54:35 | bauzas | no crazypants about PCI tracking | |
| 13:54:56 | bauzas | litterally ask libvirt to pick that type for that pGPU | |
| 13:55:16 | efried | bauzas: If there's a PCI ID in the file, then we have to say that the file gets parsed by virt alone. Kind of thing. | |
| 13:55:48 | bauzas | efried: yeah, zactly that | |
| 13:56:04 | efried | bauzas: So... you want to do something like this for Rocky? | |
| 13:56:08 | bauzas | I guess | |
| 13:56:17 | bauzas | if nested RPs is a thing :) | |
| 13:56:22 | efried | bauzas: Cause this is edging unequivocally into Generic Device Management territory. | |
| 13:56:34 | alex_xu_ | bauzas: when your request VGPU_thistype, how do you change total=0 for the VGPU_thosetype? | |
| 13:56:48 | bauzas | alex_xu_: that's magically done by sysfs | |
| 13:57:03 | bauzas | alex_xu_: which I'm using for getting the inventory | |
| 13:57:05 | alex_xu_ | bauzas: but how the placement to know that | |
| 13:57:20 | bauzas | alex_xu_: because we provide inventories of VGPU resouces as of queens :) | |
| 13:57:27 | efried | bauzas: Wait, you're talking about doing that at setup time, not at allocation time, right? | |
| 13:57:28 | bauzas | that are populated based on sysfs :) | |
| 13:58:12 | bauzas | efried: alex_xu_ is talking of the case where we would have pGPU types as children | |
| 13:58:14 | alex_xu_ | bauzas: if there are two requests at same time. One for VGPU_thistype, another for VGPU_thosetype. So there will be race case. Only one can successful in the host finally | |
| 13:58:18 | efried | bauzas: Bricks will be shat by several people if you start talking about modifying inventory of Y because X got allocated. | |
| 13:58:32 | bauzas | alex_xu_: yeah I considered the race condition | |
| 13:58:41 | bauzas | alex_xu_: and that's actually a good call for not doing that | |
| 13:58:52 | bauzas | but rather doing a whitelisting on the virt driver direcrtly | |
| 13:59:07 | alex_xu_ | bauzas: so limit only one type in each host? | |