| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-15 | |||
| 20:12:17 | artom | Their domain capabilities docs aren't exchaustive :/ https://libvirt.org/html/libvirt-libvirt-domain.html#virConnectGetDomainCapabilities | |
| 20:14:27 | cfriesen | mriedem: I think HPET is supported on x86 since like 2005. | |
| 20:14:45 | artom | So, on a physical machine with an HPET, virsh capabilities doesn't include any mention of it | |
| 20:15:18 | cfriesen | on qemu at least, my understanding is that HPET is always emulated using other clocksources | |
| 20:15:56 | dansmith | cfriesen: is that's the case, then what is the point of the trait? | |
| 20:16:23 | dansmith | if there aren't any x86-capable hypervisors that don't support HPET we shouldn't need it at all right? | |
| 20:17:05 | dansmith | presumably if you don't have HPET enabled on your kernel and you're using libvirt for lxc, then just keying off of "is x86" isn't enough | |
| 20:17:12 | mriedem | artom: yeah this was close to the same thing https://www.reddit.com/r/VFIO/comments/5g6rxb/a_fix_for_windows_vm_performancestuttering_when/ | |
| 20:17:21 | cfriesen | dansmith: I'm told that some operators might not want to enable HPET for all guests for security reasons. | |
| 20:17:49 | cfriesen | dansmith: so I was kind of assuming that we'd have a hypervisor specific config option that when turned off would prevent the trait from being advertised | |
| 20:17:53 | dansmith | cfriesen: so what's the key? | |
| 20:17:54 | dansmith | dear god | |
| 20:18:11 | dansmith | I think we're maybe uncovering that this is a super tiny line item on one person's wish list :) | |
| 20:18:25 | mriedem | so we're back to my original question in the spec, which is some external thing can set hpet on the compute node resource provider | |
| 20:19:12 | dansmith | mriedem: I don't remember where we landed with efried on being able to set custom traits on compute-managed providers | |
| 20:19:23 | mriedem | this is a standard trait | |
| 20:19:24 | dansmith | I thought the current thinking was that those would get wiped | |
| 20:19:27 | mriedem | or being proposed as one | |
| 20:19:32 | mriedem | we'll merge them | |
| 20:19:32 | dansmith | not what I meant | |
| 20:19:35 | dansmith | I don't mean custom I mean random | |
| 20:19:47 | mriedem | things that the virt driver reports + custom set external things get merged | |
| 20:19:47 | cfriesen | dansmith: the note at the PTG was literally "some way to request an HPET in the guest for x86 guests" :) | |
| 20:19:48 | efried | It's the issue of "trait ownership" | |
| 20:19:48 | dansmith | meaning set by the operator, unknown to the compute service | |
| 20:20:01 | dansmith | efried: right | |
| 20:20:16 | efried | It sucks, but basically we need to decide whether the HPET trait is owned by compute or not. | |
| 20:20:20 | mriedem | also discussed at some length related to https://review.openstack.org/#/c/538498/ | |
| 20:20:23 | efried | If it is, then the user *can't* set it. | |
| 20:20:32 | efried | if they do, and compute disagrees, compute will wipe it. | |
| 20:20:47 | efried | and vice versa. | |
| 20:21:05 | dansmith | right, so did we end up being able to declare some as compute-manage and some not, on a given provider? | |
| 20:21:12 | efried | unless we figure some override in the (as yet vaporware) provider customization file... | |
| 20:21:12 | mriedem | it sounds like the only way the libvirt driver can know is if it execs "cat /sys/devices/system/clocksource/clocksource0/available_clocksource" on startup | |
| 20:21:38 | mriedem | and greps for hpet | |
| 20:21:39 | efried | dansmith: Otherwise, it's basically just hardcoded, and we would have to document it somewhere. | |
| 20:21:44 | dansmith | mriedem: even still cfriesen is arguing that it's an operator decision, not a property of the host | |
| 20:21:54 | cfriesen | mriedem: that's whether the host kernel has the hpet clocksource available. I don't think that impacts whether it can provide an emulated HPET to the guest. | |
| 20:21:58 | efried | except for custom traits, which we can namespace like we talked about in denver. | |
| 20:22:16 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 20:22:22 | mriedem | namespace custom traits? | |
| 20:22:27 | mriedem | i thought CUSTOM_* was it? | |
| 20:22:43 | mriedem | dansmith: yeah b/c some ops want to disable it... | |
| 20:23:10 | dansmith | yeah, so I definitely thought this was going to be an actual capability, not just a "is this turned on" sort of thing when we last discussed it | |
| 20:23:26 | dansmith | not even is it turned on, but is it in policy for this host | |
| 20:24:03 | efried | mriedem: namespaced to signify the owner. Like if the trait.startswith('CUSTOM_NOVA_') then the op isn't allowed to touch it. Otherwise, fair game. | |
| 20:24:17 | efried | ...until CUSTOM_CYBORG_ is a thing, etc. | |
| 20:24:51 | efried | but at least then we have to have a short hardcoded list of possible owners and their namespaces, rather than a long hardcoded list of individual traits (or trait regexes) that are system-owned. | |
| 20:24:59 | mriedem | are there any cases where we'll have CUSTOM_NOVA_* traits that aren't just standard traits in os-traits? | |
| 20:25:23 | mriedem | CUSTOM_NOVA_* sounds pretty not custom to me | |
| 20:25:25 | efried | mriedem: An example would be a trait signifying the PCI address of a device. | |
| 20:26:13 | mriedem | is that a trait? | |
| 20:26:21 | efried | it's not a *capability* | |
| 20:26:24 | efried | But it is a *trait*. | |
| 20:26:26 | efried | Dammit. | |
| 20:26:31 | mriedem | it's a....thing | |
| 20:27:18 | cfriesen | is it schedulable? | |
| 20:27:18 | mriedem | what's the point of that being a trait? so we can schedule to a specific PCI address? | |
| 20:27:25 | efried | yes | |
| 20:27:30 | efried | pets, baby. | |
| 20:27:36 | mriedem | huh | |
| 20:27:48 | cfriesen | but can't the IOMMU redirect to whatever virtual address is desired? | |
| 20:27:48 | efried | more realistically, trait for vendor/product ID. | |
| 20:28:24 | mriedem | i seem to remember people saying we weren't going to allow scheduling to that level of granularity, like down to host numa node and such | |
| 20:28:40 | efried | I do this weekly now, look up 'trait' in the dictionary to assure myself that 'capability' is nowhere in the definition. Also not in the thesaurus. | |
| 20:28:50 | cfriesen | wouldn't "pci devices with a given vendor/product-id" be a countable resource ? | |
| 20:29:16 | efried | cfriesen: "PCI device" is the countable resource. "Vendor ID" and "Product ID" are traits. (But not capabilities) | |
| 20:29:30 | efried | I mean, arguably they're metadata (key/value). | |
| 20:29:35 | efried | But since we don't have that... | |
| 20:29:51 | cfriesen | I'd suggest that a whole set of capabilities are implied by the vendor/product | |
| 20:29:57 | efried | Completely agree. | |
| 20:30:20 | efried | But that overlaps with a set of capabilities implied by a different vendor/product. | |
| 20:30:38 | efried | In a perfect world, we would have a zillion individual traits for each of those capabilities | |
| 20:30:43 | efried | but | |
| 20:30:43 | efried | and I would schedule based on those capabilities | |
| 20:30:48 | efried | it is not a perfect world | |
| 20:30:57 | efried | where vendors are going to supply us handy dandy tables of those capabilities | |
| 20:31:02 | cfriesen | ick...then you'd need to specify which drivers are in your image, and suport all of them. | |
| 20:31:03 | efried | so we have to make do with what we've got. | |
| 20:31:09 | mriedem | so this is the thing where because we don't have metadata on resource providers, we're going to have super-encoded traits right? | |
| 20:31:11 | efried | yeah, something like that. | |
| 20:31:19 | mriedem | CUSTOM_NOVA_PCI_VENDOR_FOO_PRODUCT_BAR? | |
| 20:31:49 | efried | because then I can schedule based on required=in:{buncha vendor/product IDs that have the capabilities I need}, whereas if it was encoded in the resource class, that would not be possible. | |
| 20:32:15 | efried | mriedem: CUSTOM_NOVA_PCI_VENDORID_FOO and CUSTOM_NOVA_PCI_PRODUCTID_BAR, separate traits, but yeah. | |
| 20:32:59 | cfriesen | efried: is there a logical AND in there? since different vendors can have the same product ID | |
| 20:33:39 | efried | cfriesen: The boolean logic that's possible with `required` is not complete, but should be rich enough to get where you need to be. | |
| 20:34:13 | cfriesen | efried: I think you'd need something like ((A AND B) OR (C AND D)) | |
| 20:34:17 | efried | btw, this idea is far from having any kind of consensus behind it. There are certain people who will need to be cold underground before it becomes reality. | |
| 20:35:25 | efried | cfriesen: True, and I don't think you can quiiite get there actually. What are the odds of ((A or C) AND (B OR D)) winding up meaningfully different in such a way that it actually hits a wrong device? | |
| 20:36:48 | efried | i.e. ((vendorA or vendorC) AND (productB or productD)) when what you really want is ((vendorA and productB) or (vendorC and productD)) => what are the chances that (vendorA+productD) or (vendorB+productC) actually exist? | |
| 20:39:23 | cfriesen | well, each vendor ID is unique, but I think product IDs are only unique within a given vendor | |
| 20:40:31 | cfriesen | but we may be lucky and not have overlap for commonly-used devices. | |
| 20:48:18 | edleafe | efried: don't put too much weight on the name 'trait'. That itself was a compromise betwee those who wanted them named 'capability' and those who wanted 'tag' | |
| 20:48:31 | edleafe | efried: It was always supposed to represent capabilities | |
| 20:51:43 | efried | I'll certainly admit that if they were called 'capabilities' I would not argue as strongly to put stuff like product IDs in them. | |
| 20:51:56 | efried | But they ain't. | |
| 20:52:09 | efried | Juliet had something to say about this. And look how that turned out. | |
| 20:53:58 | edleafe | "A capability by any other name would still generate semantic arguments..." | |
| 20:53:59 | cfriesen | so where are we at for HPET (and vTPM)? I'd suggest that for vTPM it's actually a capability since it depends on hypervisor version and such. For HPET it's two things: 1) asking for it where currently it's turned off unconditionally, and 2) maybe allowing operators to refuse to provide it. | |
| 20:54:50 | mriedem | cfriesen: my current question on the hpet spec is who sets the trait | |