| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-07 | |||
| 14:07:51 | kosamara | sean-k-mooney: yes, which is what efried 's spec does. | |
| 14:08:30 | maciejjozefczyk | mriedem: but for me if 'nova compute-show' shows that compute is UP and timestamp is new - it says, yea, compute is working | |
| 14:08:51 | maciejjozefczyk | mriedem: maybe somebody use it as source for some scripts | |
| 14:09:01 | maciejjozefczyk | mriedem: but ye, service-list should be used anyway for that pourpose | |
| 14:09:19 | mriedem | i want to say i think there is something in the scheduler that cares about the compute node updated_at time and uses it for some refresh threshold | |
| 14:09:43 | maciejjozefczyk | mriedem: no idea | |
| 14:09:57 | mriedem | HostManager._check_for_nodes_rebalance | |
| 14:10:02 | mriedem | if (self.updated and compute.updated_at | |
| 14:10:02 | mriedem | and self.updated > compute.updated_at): | |
| 14:10:02 | mriedem | return | |
| 14:10:56 | maciejjozefczyk | yep, so thats the issue | |
| 14:11:31 | maciejjozefczyk | mriedem: the code is from placement? | |
| 14:11:40 | mriedem | no that's in the nova scheduler HostManager | |
| 14:11:53 | maciejjozefczyk | mriedem: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L929 | |
| 14:12:03 | sean-k-mooney | kosamara: oh i did not know efried had a pci device in placement spec up for stien cool il should review that | |
| 14:12:18 | maciejjozefczyk | mriedem: ah, so it bases on the db, right? | |
| 14:12:21 | kosamara | efried: I would like to contribute. I think I could start with making the libvirt driver report PCI RPs, similar to your spec. I can propose a small spec just for that for Stein. | |
| 14:12:21 | mriedem | right | |
| 14:12:30 | mriedem | https://github.com/openstack/nova/blob/536acbfe0572f10ea84f330f2f29b07ca9279114/nova/scheduler/host_manager.py#L194 | |
| 14:12:55 | kosamara | sean-k-mooney: not in nova, in the nova-powervm fork: https://review.openstack.org/#/c/579359/10/doc/source/specs/rocky/device-passthrough.rst | |
| 14:13:13 | maciejjozefczyk | ok, so I vote for dropping resource_change logic, but first let me see how we use it in our deployment | |
| 14:13:17 | efried | kosamara: That would be cool, please add me as a reviewer. | |
| 14:13:22 | sean-k-mooney | kosamara: if your are doing that can you also make sure to pull the pci feature flags from the pci_devices table in the nova db and add them as traits to the RP | |
| 14:13:52 | maciejjozefczyk | mriedem: anyway my RUN team doesn't scream about this issue anymore | |
| 14:13:55 | mriedem | maciejjozefczyk: since I40c17ed88f50ecbdedc4daf368fff10e90e7be11 i'm not sure this check in the HostState object even matters | |
| 14:14:06 | mriedem | we don't cache HostStates in the scheduler anymore | |
| 14:21:14 | efried | kosamara: One main aspect of the nova-powervm spec that I expect to be carried through to nova proper is a YAML configuration file allowing the operator to: | |
| 14:21:14 | efried | - Identify devices to be permitted for passthrough | |
| 14:21:14 | efried | - Specify a resource class | |
| 14:21:14 | efried | - Specify traits | |
| 14:22:13 | kosamara | sean-k-mooney: I wasn't aware of this info in the pci_devices table. What exactly is it? | |
| 14:22:33 | maciejjozefczyk | mriedem: anyway I need to go, I'll check this resource_updated() logic with my patch once again, I'll leave comment tonight | |
| 14:22:38 | sean-k-mooney | efried: um im not sure about that... given how we did the numa aware vswitch spec i would have assumed we would have used dynamic config insead of a yaml file | |
| 14:22:43 | efried | maciejjozefczyk: Thanks! | |
| 14:23:00 | efried | sean-k-mooney: -1 to dynamic config. | |
| 14:23:19 | kosamara | efried: I may propose something more basic, with the existing passthrough_whitelist conf. | |
| 14:23:50 | efried | sean-k-mooney: I think the only reason we did that instead of yaml is to minimize the effort. | |
| 14:23:55 | sean-k-mooney | kosamara: for network devices we use ethtool ioctls via libvirt to get the nic feature flags such as tcp checksume offload | |
| 14:23:56 | efried | kosamara: Oh dear gods please no. | |
| 14:24:35 | sean-k-mooney | kosamara: adding more stuff to passthough_whitelist is basically an automatic -3 | |
| 14:24:41 | mdbooth | lol | |
| 14:25:12 | efried | sean-k-mooney: We talked about using yaml in Denver. It really makes the most sense for this kind of thing, because trying to manage nested hierarchical data via oslo_config is a major PITA. | |
| 14:25:51 | sean-k-mooney | efried: well there is an argument to be made that today we dont use yaml for any other configs so we should not intoduce it for this feature | |
| 14:25:55 | efried | kosamara: ...Also, automatically generated traits. In my spec we've "namespaced" the generated traits with _POWERVM_ but some of them will potentially overlap on any platform (e.g. vendor & product IDs) | |
| 14:26:10 | sean-k-mooney | efried: that said im not really against it either | |
| 14:26:31 | kosamara | efried: I had it like that in my mind following a previous discussion with sean-k-mooney and gibi. I'll rethink it. | |
| 14:26:37 | efried | sean-k-mooney: IIRC jaypipes was a proponent and even dansmith was in agreement. | |
| 14:27:23 | efried | kosamara: In case the prospect of yaml schema/parsing is intimidating, here's code: https://review.openstack.org/#/c/579289/ | |
| 14:28:15 | sean-k-mooney | efried: well jaypipes didnt want more semantics to the whitelist and having a dedicated config not in nova was cleaner as nova was not using dynamic config at the time. as i said im not against it but its adding another dependicy to nova e.g. yaml parsing | |
| 14:29:01 | sean-k-mooney | efried: are you going to propose that spec to nova propper for stien? | |
| 14:29:58 | sean-k-mooney | efried: if so is the scope just pci devices or generic device passthouhg. we had talked about expanding it to usb/sata devices in denver too but not sure if that is a different spec | |
| 14:30:12 | sean-k-mooney | * should be a different sepc | |
| 14:31:05 | efried | sean-k-mooney: I hadn't yet decided whether to propose a nova spec for Stein or wait until T, but it sounds like kosamara may be interested in doing it for Stein. | |
| 14:31:33 | efried | sean-k-mooney: The way I've written the nova-powervm spec, the schema would be easily extensible to support non-PCI. | |
| 14:31:34 | jaypipes | efried: vendor and product IDs should not be traits. | |
| 14:31:47 | efried | sigh | |
| 14:31:58 | jaypipes | just sayin. | |
| 14:32:07 | mdbooth | lyarwood: Hey, this is interesting | |
| 14:32:40 | sean-k-mooney | efried: i abandonded my nic feature based schduling work in favour of using this in the furutre but not sure ill be working on that now | |
| 14:32:52 | mdbooth | lyarwood: Still investigating your functional failures, came across _terminate_volume_connections in ComputeManager, which does exactly what I proposed | |
| 14:33:03 | sean-k-mooney | jaypipes: im assuming you would prefer a resouce_class to track that | |
| 14:33:12 | jaypipes | sean-k-mooney: no. | |
| 14:33:16 | sean-k-mooney | jaypipes: no? | |
| 14:33:17 | mdbooth | lyarwood: Specifically create a blank attachment, delete the old attachment, update the bdm to point to the blank. | |
| 14:33:29 | jaypipes | sean-k-mooney: I'm just saying traits are capabilities. they aren't key/value metadata items. | |
| 14:33:37 | efried | sean-k-mooney: I think jaypipes wants traits for the *capabilities* associated with a vendor/product | |
| 14:33:38 | sean-k-mooney | ah ok | |
| 14:33:44 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/pike: Fix message for unexpected external event https://review.openstack.org/589503 | |
| 14:33:45 | jaypipes | efried: bingo. | |
| 14:33:56 | sean-k-mooney | yep i agree that they are capablityes not metadata | |
| 14:34:01 | efried | which irl will entail maintaining a matrix of vendor/product to capabilities | |
| 14:34:18 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Fix message for unexpected external event https://review.openstack.org/589505 | |
| 14:34:18 | jaypipes | efried: correct. | |
| 14:34:24 | sean-k-mooney | i was implying that vendor/product id is metadata that should be associated with the resouce_class | |
| 14:34:24 | efried | which means whenever a new device is introduced that we want to support, we need to change code. | |
| 14:34:37 | jaypipes | efried: which... *gasp* the friggin vendors should be responsible for. | |
| 14:34:53 | efried | jaypipes: The vendors should be responsible for proposing nova patches to support their devices? | |
| 14:34:59 | sean-k-mooney | i had asked previously about intodusing resouce class metadata at some point but did not really push the point in the past | |
| 14:35:21 | jaypipes | efried: no. the vendors should be responsible for keeping the matrix of capabilities up to date with their product lines. | |
| 14:35:41 | efried | jaypipes: And that matrix of capabilities should be discoverable by querying the device somehow | |
| 14:35:54 | jaypipes | efried: in the same way they are responsible for ensuring the pciids database is kept up to date with all their vendor, subvendor/reseller and product information. | |
| 14:35:58 | sean-k-mooney | jaypipes: well the vendor id/product id is important for other reasons such as knowing what driver is required for the device | |
| 14:36:19 | efried | If that were the case, and if vendors could agree on names (IDs?) for capabilities across the board, I could get behind it. | |
| 14:36:59 | sean-k-mooney | efried: well intel has been pushing to try and stardise some of them in etsi and dmtf(redfish) | |
| 14:37:00 | efried | But I think we need to have a realistic fallback plan so that, if such a nirvana does not come into being, operators will still be able to ask for a GPU by product ID. | |
| 14:37:24 | sean-k-mooney | from an openstack point of view that what the standard traits in os-traits are for | |
| 14:37:59 | jaypipes | efried: well, if vendors want to enable their technology in OpenStack, they should work with us. Instead, nova needs to jump through a bunch of hacky hoops to work with vendors. ala https://review.openstack.org/#/c/579897/. which is complete bullshit, IMHO. | |
| 14:38:22 | efried | yes, I noticed you getting excited about that yesterday :) | |
| 14:39:16 | sean-k-mooney | jaypipes: ya.. well that is working around a limitation that nvidia put in there driver to prevent you using there gpus in vm unless you baught there datacenter gpus | |
| 14:39:37 | jaypipes | sean-k-mooney: I'm fully aware of that, yes. | |
| 14:40:14 | jaypipes | sean-k-mooney: please see my comment on whether we are OpenStack or WorkaroundForClosedStack | |
| 14:40:42 | jaypipes | sean-k-mooney: I think the answer has already been answered on that, actually. we have been VendorStack for about 5 years now. | |
| 14:40:45 | sean-k-mooney | oh i see it i have it open now. i agree that we should not need to do this but its a usecase they have | |
| 14:41:49 | sean-k-mooney | i think there are usecase beyond the gpu case for wanting to hide the hypervios signture | |
| 14:42:01 | jaypipes | name one. | |
| 14:42:01 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 14:42:11 | sean-k-mooney | linux kernel dev | |
| 14:42:57 | sean-k-mooney | anyway i know that is a reach but for ovs-dpdk i dev i have used trick in the past to allow me to use openstack as a dev env | |
| 14:43:19 | sean-k-mooney | such as setting the nic model to e1000 to i could test the non viutalised nic binding workflow | |
| 14:44:40 | jaypipes | sean-k-mooney: you're performing use case gymnastics here. | |