| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-07 | |||
| 14:12:30 | mriedem | https://github.com/openstack/nova/blob/536acbfe0572f10ea84f330f2f29b07ca9279114/nova/scheduler/host_manager.py#L194 | |
| 14:12:55 | kosamara | sean-k-mooney: not in nova, in the nova-powervm fork: https://review.openstack.org/#/c/579359/10/doc/source/specs/rocky/device-passthrough.rst | |
| 14:13:13 | maciejjozefczyk | ok, so I vote for dropping resource_change logic, but first let me see how we use it in our deployment | |
| 14:13:17 | efried | kosamara: That would be cool, please add me as a reviewer. | |
| 14:13:22 | sean-k-mooney | kosamara: if your are doing that can you also make sure to pull the pci feature flags from the pci_devices table in the nova db and add them as traits to the RP | |
| 14:13:52 | maciejjozefczyk | mriedem: anyway my RUN team doesn't scream about this issue anymore | |
| 14:13:55 | mriedem | maciejjozefczyk: since I40c17ed88f50ecbdedc4daf368fff10e90e7be11 i'm not sure this check in the HostState object even matters | |
| 14:14:06 | mriedem | we don't cache HostStates in the scheduler anymore | |
| 14:21:14 | efried | - Specify traits | |
| 14:21:14 | efried | - Specify a resource class | |
| 14:21:14 | efried | - Identify devices to be permitted for passthrough | |
| 14:21:14 | efried | kosamara: One main aspect of the nova-powervm spec that I expect to be carried through to nova proper is a YAML configuration file allowing the operator to: | |
| 14:22:13 | kosamara | sean-k-mooney: I wasn't aware of this info in the pci_devices table. What exactly is it? | |
| 14:22:33 | maciejjozefczyk | mriedem: anyway I need to go, I'll check this resource_updated() logic with my patch once again, I'll leave comment tonight | |
| 14:22:38 | sean-k-mooney | efried: um im not sure about that... given how we did the numa aware vswitch spec i would have assumed we would have used dynamic config insead of a yaml file | |
| 14:22:43 | efried | maciejjozefczyk: Thanks! | |
| 14:23:00 | efried | sean-k-mooney: -1 to dynamic config. | |
| 14:23:19 | kosamara | efried: I may propose something more basic, with the existing passthrough_whitelist conf. | |
| 14:23:50 | efried | sean-k-mooney: I think the only reason we did that instead of yaml is to minimize the effort. | |
| 14:23:55 | sean-k-mooney | kosamara: for network devices we use ethtool ioctls via libvirt to get the nic feature flags such as tcp checksume offload | |
| 14:23:56 | efried | kosamara: Oh dear gods please no. | |
| 14:24:35 | sean-k-mooney | kosamara: adding more stuff to passthough_whitelist is basically an automatic -3 | |
| 14:24:41 | mdbooth | lol | |
| 14:25:12 | efried | sean-k-mooney: We talked about using yaml in Denver. It really makes the most sense for this kind of thing, because trying to manage nested hierarchical data via oslo_config is a major PITA. | |
| 14:25:51 | sean-k-mooney | efried: well there is an argument to be made that today we dont use yaml for any other configs so we should not intoduce it for this feature | |
| 14:25:55 | efried | kosamara: ...Also, automatically generated traits. In my spec we've "namespaced" the generated traits with _POWERVM_ but some of them will potentially overlap on any platform (e.g. vendor & product IDs) | |
| 14:26:10 | sean-k-mooney | efried: that said im not really against it either | |
| 14:26:31 | kosamara | efried: I had it like that in my mind following a previous discussion with sean-k-mooney and gibi. I'll rethink it. | |
| 14:26:37 | efried | sean-k-mooney: IIRC jaypipes was a proponent and even dansmith was in agreement. | |
| 14:27:23 | efried | kosamara: In case the prospect of yaml schema/parsing is intimidating, here's code: https://review.openstack.org/#/c/579289/ | |
| 14:28:15 | sean-k-mooney | efried: well jaypipes didnt want more semantics to the whitelist and having a dedicated config not in nova was cleaner as nova was not using dynamic config at the time. as i said im not against it but its adding another dependicy to nova e.g. yaml parsing | |
| 14:29:01 | sean-k-mooney | efried: are you going to propose that spec to nova propper for stien? | |
| 14:29:58 | sean-k-mooney | efried: if so is the scope just pci devices or generic device passthouhg. we had talked about expanding it to usb/sata devices in denver too but not sure if that is a different spec | |
| 14:30:12 | sean-k-mooney | * should be a different sepc | |
| 14:31:05 | efried | sean-k-mooney: I hadn't yet decided whether to propose a nova spec for Stein or wait until T, but it sounds like kosamara may be interested in doing it for Stein. | |
| 14:31:33 | efried | sean-k-mooney: The way I've written the nova-powervm spec, the schema would be easily extensible to support non-PCI. | |
| 14:31:34 | jaypipes | efried: vendor and product IDs should not be traits. | |
| 14:31:47 | efried | sigh | |
| 14:31:58 | jaypipes | just sayin. | |
| 14:32:07 | mdbooth | lyarwood: Hey, this is interesting | |
| 14:32:40 | sean-k-mooney | efried: i abandonded my nic feature based schduling work in favour of using this in the furutre but not sure ill be working on that now | |
| 14:32:52 | mdbooth | lyarwood: Still investigating your functional failures, came across _terminate_volume_connections in ComputeManager, which does exactly what I proposed | |
| 14:33:03 | sean-k-mooney | jaypipes: im assuming you would prefer a resouce_class to track that | |
| 14:33:12 | jaypipes | sean-k-mooney: no. | |
| 14:33:16 | sean-k-mooney | jaypipes: no? | |
| 14:33:17 | mdbooth | lyarwood: Specifically create a blank attachment, delete the old attachment, update the bdm to point to the blank. | |
| 14:33:29 | jaypipes | sean-k-mooney: I'm just saying traits are capabilities. they aren't key/value metadata items. | |
| 14:33:37 | efried | sean-k-mooney: I think jaypipes wants traits for the *capabilities* associated with a vendor/product | |
| 14:33:38 | sean-k-mooney | ah ok | |
| 14:33:44 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/pike: Fix message for unexpected external event https://review.openstack.org/589503 | |
| 14:33:45 | jaypipes | efried: bingo. | |
| 14:33:56 | sean-k-mooney | yep i agree that they are capablityes not metadata | |
| 14:34:01 | efried | which irl will entail maintaining a matrix of vendor/product to capabilities | |
| 14:34:18 | jaypipes | efried: correct. | |
| 14:34:18 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Fix message for unexpected external event https://review.openstack.org/589505 | |
| 14:34:24 | efried | which means whenever a new device is introduced that we want to support, we need to change code. | |
| 14:34:24 | sean-k-mooney | i was implying that vendor/product id is metadata that should be associated with the resouce_class | |
| 14:34:37 | jaypipes | efried: which... *gasp* the friggin vendors should be responsible for. | |
| 14:34:53 | efried | jaypipes: The vendors should be responsible for proposing nova patches to support their devices? | |
| 14:34:59 | sean-k-mooney | i had asked previously about intodusing resouce class metadata at some point but did not really push the point in the past | |
| 14:35:21 | jaypipes | efried: no. the vendors should be responsible for keeping the matrix of capabilities up to date with their product lines. | |
| 14:35:41 | efried | jaypipes: And that matrix of capabilities should be discoverable by querying the device somehow | |
| 14:35:54 | jaypipes | efried: in the same way they are responsible for ensuring the pciids database is kept up to date with all their vendor, subvendor/reseller and product information. | |
| 14:35:58 | sean-k-mooney | jaypipes: well the vendor id/product id is important for other reasons such as knowing what driver is required for the device | |
| 14:36:19 | efried | If that were the case, and if vendors could agree on names (IDs?) for capabilities across the board, I could get behind it. | |
| 14:36:59 | sean-k-mooney | efried: well intel has been pushing to try and stardise some of them in etsi and dmtf(redfish) | |
| 14:37:00 | efried | But I think we need to have a realistic fallback plan so that, if such a nirvana does not come into being, operators will still be able to ask for a GPU by product ID. | |
| 14:37:24 | sean-k-mooney | from an openstack point of view that what the standard traits in os-traits are for | |
| 14:37:59 | jaypipes | efried: well, if vendors want to enable their technology in OpenStack, they should work with us. Instead, nova needs to jump through a bunch of hacky hoops to work with vendors. ala https://review.openstack.org/#/c/579897/. which is complete bullshit, IMHO. | |
| 14:38:22 | efried | yes, I noticed you getting excited about that yesterday :) | |
| 14:39:16 | sean-k-mooney | jaypipes: ya.. well that is working around a limitation that nvidia put in there driver to prevent you using there gpus in vm unless you baught there datacenter gpus | |
| 14:39:37 | jaypipes | sean-k-mooney: I'm fully aware of that, yes. | |
| 14:40:14 | jaypipes | sean-k-mooney: please see my comment on whether we are OpenStack or WorkaroundForClosedStack | |
| 14:40:42 | jaypipes | sean-k-mooney: I think the answer has already been answered on that, actually. we have been VendorStack for about 5 years now. | |
| 14:40:45 | sean-k-mooney | oh i see it i have it open now. i agree that we should not need to do this but its a usecase they have | |
| 14:41:49 | sean-k-mooney | i think there are usecase beyond the gpu case for wanting to hide the hypervios signture | |
| 14:42:01 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 14:42:01 | jaypipes | name one. | |
| 14:42:11 | sean-k-mooney | linux kernel dev | |
| 14:42:57 | sean-k-mooney | anyway i know that is a reach but for ovs-dpdk i dev i have used trick in the past to allow me to use openstack as a dev env | |
| 14:43:19 | sean-k-mooney | such as setting the nic model to e1000 to i could test the non viutalised nic binding workflow | |
| 14:44:40 | jaypipes | sean-k-mooney: you're performing use case gymnastics here. | |
| 14:45:37 | sean-k-mooney | yes and no. in the early days of dpdk support i had to do thing like set the cpu-mode to host-passthough becasue i did not have the cpu feature reqiure to even compile it otherwise | |
| 14:46:35 | dansmith | jaypipes: it's a common case for kernel development to use virtualization to simulate environments for that dev.. like fake numa nodes, pci devices, etc | |
| 14:46:39 | sean-k-mooney | as i said i was reaching a bit but if i want to do driver dev and i want to do that in a vm on openstack then i might need to hide the hypervisor signiture. that said i also recognise that openstack is not a virtualisation stack | |
| 14:46:49 | dansmith | jaypipes: that's not to say that I think IaaS really needs to worry about such things | |
| 14:47:05 | dansmith | jaypipes: given that developers can use bare hypervisors on their laptops to do that with much better control | |
| 14:47:47 | sean-k-mooney | dansmith: yep that said for ci of that code openstack is a much more compelling option | |
| 14:48:05 | dansmith | yup, CI is another reasonable use case imho | |
| 14:49:04 | sean-k-mooney | the alternitive without hypervior hiding is ironic but that is arguable more complex. the difference is the complexity is on the enduer not the technical debt nova has to maintain | |
| 14:49:38 | dansmith | sean-k-mooney: note that I'm defending the dev case for needing finer grained control, but *not* defending the hypervisor-hiding case | |
| 14:49:51 | dansmith | the latter is mostly total closed-source proprietary BS | |
| 14:50:20 | jaypipes | sean-k-mooney, dansmith: in that case, the use case would be to be able to *set* the vendor ID to a particular value, not hardcode it to "1234567890ab" just for the HyperV emulator just to avoid licensing issues with Nvidia: https://review.openstack.org/#/c/579897/4/nova/virt/libvirt/config.py@2241 | |
| 14:50:50 | lyarwood | s10: hey, thanks for the report, so what's the actual knock of impact of update_available_resource taking longer here? Scheduling accuracy? | |
| 14:51:26 | sean-k-mooney | jaypipes: well i had originally asked shoudl we just remove the hypervier section entirly if hiding was asked for to fully hide the fact we were in a vm but that has performance impacts | |
| 14:52:34 | sean-k-mooney | but yes making this more generic to allow setting a spefic vendor id would be just as vaild | |
| 14:53:06 | s10 | lyarwood: It looks like, when I try to live migrate instances from this host, all operations should wait for this update_available_resource every minute. So I have a small live migration window: 30 seconds every minute, only after update_available_resource ends and before it starts. | |
| 14:54:34 | sean-k-mooney | s10: why? | |
| 14:55:28 | jaypipes | s10: why are you running update_available_resource every minute? | |
| 14:55:45 | s10 | sean-k-mooney: I don't know, that's what I see in logs. No live migration could be performed during update_available_resources. If change libvirt/driver.py to like before first commits, migrations goes without pause. | |