| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-15 | |||
| 20:05:53 | mriedem | good point | |
| 20:08:36 | artom | tsc hpet acpi_pm | |
| 20:08:36 | artom | So, hardware-wise, it's apparently as easy as [artom@jayne ~]$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource | |
| 20:08:46 | artom | But. Do we need anything from libvirt? | |
| 20:09:34 | mriedem | i was hoping something would be in the host capabilities API | |
| 20:09:35 | mriedem | http://logs.openstack.org/39/587439/4/check/tempest-full/fdb1cf9/controller/logs/screen-n-cpu.txt.gz#_Sep_11_14_44_27_658449 | |
| 20:09:37 | mriedem | but i don't see that | |
| 20:10:28 | mriedem | or maybe hpet just doesn't show up there | |
| 20:10:33 | mriedem | ? | |
| 20:10:36 | mriedem | because it's nested virt? | |
| 20:10:55 | dansmith | mriedem: I don't see it on my physical box either | |
| 20:11:07 | mriedem | physical box is kvm or qemu? | |
| 20:11:29 | mriedem | not sure if that matters | |
| 20:11:32 | artom | The SEV guys had to dig into a commit to see how libvirt reports SEV | |
| 20:11:34 | artom | (https://libvirt.org/git/?p=libvirt.git;a=commit;h=6688393c6b222b5d7cba238f21d55134611ede9c) | |
| 20:12:01 | dansmith | mriedem: physical box meaning baremetal | |
| 20:12:17 | artom | Their domain capabilities docs aren't exchaustive :/ https://libvirt.org/html/libvirt-libvirt-domain.html#virConnectGetDomainCapabilities | |
| 20:14:27 | cfriesen | mriedem: I think HPET is supported on x86 since like 2005. | |
| 20:14:45 | artom | So, on a physical machine with an HPET, virsh capabilities doesn't include any mention of it | |
| 20:15:18 | cfriesen | on qemu at least, my understanding is that HPET is always emulated using other clocksources | |
| 20:15:56 | dansmith | cfriesen: is that's the case, then what is the point of the trait? | |
| 20:16:23 | dansmith | if there aren't any x86-capable hypervisors that don't support HPET we shouldn't need it at all right? | |
| 20:17:05 | dansmith | presumably if you don't have HPET enabled on your kernel and you're using libvirt for lxc, then just keying off of "is x86" isn't enough | |
| 20:17:12 | mriedem | artom: yeah this was close to the same thing https://www.reddit.com/r/VFIO/comments/5g6rxb/a_fix_for_windows_vm_performancestuttering_when/ | |
| 20:17:21 | cfriesen | dansmith: I'm told that some operators might not want to enable HPET for all guests for security reasons. | |
| 20:17:49 | cfriesen | dansmith: so I was kind of assuming that we'd have a hypervisor specific config option that when turned off would prevent the trait from being advertised | |
| 20:17:53 | dansmith | cfriesen: so what's the key? | |
| 20:17:54 | dansmith | dear god | |
| 20:18:11 | dansmith | I think we're maybe uncovering that this is a super tiny line item on one person's wish list :) | |
| 20:18:25 | mriedem | so we're back to my original question in the spec, which is some external thing can set hpet on the compute node resource provider | |
| 20:19:12 | dansmith | mriedem: I don't remember where we landed with efried on being able to set custom traits on compute-managed providers | |
| 20:19:23 | mriedem | this is a standard trait | |
| 20:19:24 | dansmith | I thought the current thinking was that those would get wiped | |
| 20:19:27 | mriedem | or being proposed as one | |
| 20:19:32 | mriedem | we'll merge them | |
| 20:19:32 | dansmith | not what I meant | |
| 20:19:35 | dansmith | I don't mean custom I mean random | |
| 20:19:47 | mriedem | things that the virt driver reports + custom set external things get merged | |
| 20:19:47 | cfriesen | dansmith: the note at the PTG was literally "some way to request an HPET in the guest for x86 guests" :) | |
| 20:19:48 | efried | It's the issue of "trait ownership" | |
| 20:19:48 | dansmith | meaning set by the operator, unknown to the compute service | |
| 20:20:01 | dansmith | efried: right | |
| 20:20:16 | efried | It sucks, but basically we need to decide whether the HPET trait is owned by compute or not. | |
| 20:20:20 | mriedem | also discussed at some length related to https://review.openstack.org/#/c/538498/ | |
| 20:20:23 | efried | If it is, then the user *can't* set it. | |
| 20:20:32 | efried | if they do, and compute disagrees, compute will wipe it. | |
| 20:20:47 | efried | and vice versa. | |
| 20:21:05 | dansmith | right, so did we end up being able to declare some as compute-manage and some not, on a given provider? | |
| 20:21:12 | efried | unless we figure some override in the (as yet vaporware) provider customization file... | |
| 20:21:12 | mriedem | it sounds like the only way the libvirt driver can know is if it execs "cat /sys/devices/system/clocksource/clocksource0/available_clocksource" on startup | |
| 20:21:38 | mriedem | and greps for hpet | |
| 20:21:39 | efried | dansmith: Otherwise, it's basically just hardcoded, and we would have to document it somewhere. | |
| 20:21:44 | dansmith | mriedem: even still cfriesen is arguing that it's an operator decision, not a property of the host | |
| 20:21:54 | cfriesen | mriedem: that's whether the host kernel has the hpet clocksource available. I don't think that impacts whether it can provide an emulated HPET to the guest. | |
| 20:21:58 | efried | except for custom traits, which we can namespace like we talked about in denver. | |
| 20:22:16 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 20:22:22 | mriedem | namespace custom traits? | |
| 20:22:27 | mriedem | i thought CUSTOM_* was it? | |
| 20:22:43 | mriedem | dansmith: yeah b/c some ops want to disable it... | |
| 20:23:10 | dansmith | yeah, so I definitely thought this was going to be an actual capability, not just a "is this turned on" sort of thing when we last discussed it | |
| 20:23:26 | dansmith | not even is it turned on, but is it in policy for this host | |
| 20:24:03 | efried | mriedem: namespaced to signify the owner. Like if the trait.startswith('CUSTOM_NOVA_') then the op isn't allowed to touch it. Otherwise, fair game. | |
| 20:24:17 | efried | ...until CUSTOM_CYBORG_ is a thing, etc. | |
| 20:24:51 | efried | but at least then we have to have a short hardcoded list of possible owners and their namespaces, rather than a long hardcoded list of individual traits (or trait regexes) that are system-owned. | |
| 20:24:59 | mriedem | are there any cases where we'll have CUSTOM_NOVA_* traits that aren't just standard traits in os-traits? | |
| 20:25:23 | mriedem | CUSTOM_NOVA_* sounds pretty not custom to me | |
| 20:25:25 | efried | mriedem: An example would be a trait signifying the PCI address of a device. | |
| 20:26:13 | mriedem | is that a trait? | |
| 20:26:21 | efried | it's not a *capability* | |
| 20:26:24 | efried | But it is a *trait*. | |
| 20:26:26 | efried | Dammit. | |
| 20:26:31 | mriedem | it's a....thing | |
| 20:27:18 | cfriesen | is it schedulable? | |
| 20:27:18 | mriedem | what's the point of that being a trait? so we can schedule to a specific PCI address? | |
| 20:27:25 | efried | yes | |
| 20:27:30 | efried | pets, baby. | |
| 20:27:36 | mriedem | huh | |
| 20:27:48 | cfriesen | but can't the IOMMU redirect to whatever virtual address is desired? | |
| 20:27:48 | efried | more realistically, trait for vendor/product ID. | |
| 20:28:24 | mriedem | i seem to remember people saying we weren't going to allow scheduling to that level of granularity, like down to host numa node and such | |
| 20:28:40 | efried | I do this weekly now, look up 'trait' in the dictionary to assure myself that 'capability' is nowhere in the definition. Also not in the thesaurus. | |
| 20:28:50 | cfriesen | wouldn't "pci devices with a given vendor/product-id" be a countable resource ? | |
| 20:29:16 | efried | cfriesen: "PCI device" is the countable resource. "Vendor ID" and "Product ID" are traits. (But not capabilities) | |
| 20:29:30 | efried | I mean, arguably they're metadata (key/value). | |
| 20:29:35 | efried | But since we don't have that... | |
| 20:29:51 | cfriesen | I'd suggest that a whole set of capabilities are implied by the vendor/product | |
| 20:29:57 | efried | Completely agree. | |
| 20:30:20 | efried | But that overlaps with a set of capabilities implied by a different vendor/product. | |
| 20:30:38 | efried | In a perfect world, we would have a zillion individual traits for each of those capabilities | |
| 20:30:43 | efried | but | |
| 20:30:43 | efried | and I would schedule based on those capabilities | |
| 20:30:48 | efried | it is not a perfect world | |
| 20:30:57 | efried | where vendors are going to supply us handy dandy tables of those capabilities | |
| 20:31:02 | cfriesen | ick...then you'd need to specify which drivers are in your image, and suport all of them. | |
| 20:31:03 | efried | so we have to make do with what we've got. | |
| 20:31:09 | mriedem | so this is the thing where because we don't have metadata on resource providers, we're going to have super-encoded traits right? | |
| 20:31:11 | efried | yeah, something like that. | |
| 20:31:19 | mriedem | CUSTOM_NOVA_PCI_VENDOR_FOO_PRODUCT_BAR? | |
| 20:31:49 | efried | because then I can schedule based on required=in:{buncha vendor/product IDs that have the capabilities I need}, whereas if it was encoded in the resource class, that would not be possible. | |
| 20:32:15 | efried | mriedem: CUSTOM_NOVA_PCI_VENDORID_FOO and CUSTOM_NOVA_PCI_PRODUCTID_BAR, separate traits, but yeah. | |