| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-07 | |||
| 14:23:19 | kosamara | efried: I may propose something more basic, with the existing passthrough_whitelist conf. | |
| 14:23:50 | efried | sean-k-mooney: I think the only reason we did that instead of yaml is to minimize the effort. | |
| 14:23:55 | sean-k-mooney | kosamara: for network devices we use ethtool ioctls via libvirt to get the nic feature flags such as tcp checksume offload | |
| 14:23:56 | efried | kosamara: Oh dear gods please no. | |
| 14:24:35 | sean-k-mooney | kosamara: adding more stuff to passthough_whitelist is basically an automatic -3 | |
| 14:24:41 | mdbooth | lol | |
| 14:25:12 | efried | sean-k-mooney: We talked about using yaml in Denver. It really makes the most sense for this kind of thing, because trying to manage nested hierarchical data via oslo_config is a major PITA. | |
| 14:25:51 | sean-k-mooney | efried: well there is an argument to be made that today we dont use yaml for any other configs so we should not intoduce it for this feature | |
| 14:25:55 | efried | kosamara: ...Also, automatically generated traits. In my spec we've "namespaced" the generated traits with _POWERVM_ but some of them will potentially overlap on any platform (e.g. vendor & product IDs) | |
| 14:26:10 | sean-k-mooney | efried: that said im not really against it either | |
| 14:26:31 | kosamara | efried: I had it like that in my mind following a previous discussion with sean-k-mooney and gibi. I'll rethink it. | |
| 14:26:37 | efried | sean-k-mooney: IIRC jaypipes was a proponent and even dansmith was in agreement. | |
| 14:27:23 | efried | kosamara: In case the prospect of yaml schema/parsing is intimidating, here's code: https://review.openstack.org/#/c/579289/ | |
| 14:28:15 | sean-k-mooney | efried: well jaypipes didnt want more semantics to the whitelist and having a dedicated config not in nova was cleaner as nova was not using dynamic config at the time. as i said im not against it but its adding another dependicy to nova e.g. yaml parsing | |
| 14:29:01 | sean-k-mooney | efried: are you going to propose that spec to nova propper for stien? | |
| 14:29:58 | sean-k-mooney | efried: if so is the scope just pci devices or generic device passthouhg. we had talked about expanding it to usb/sata devices in denver too but not sure if that is a different spec | |
| 14:30:12 | sean-k-mooney | * should be a different sepc | |
| 14:31:05 | efried | sean-k-mooney: I hadn't yet decided whether to propose a nova spec for Stein or wait until T, but it sounds like kosamara may be interested in doing it for Stein. | |
| 14:31:33 | efried | sean-k-mooney: The way I've written the nova-powervm spec, the schema would be easily extensible to support non-PCI. | |
| 14:31:34 | jaypipes | efried: vendor and product IDs should not be traits. | |
| 14:31:47 | efried | sigh | |
| 14:31:58 | jaypipes | just sayin. | |
| 14:32:07 | mdbooth | lyarwood: Hey, this is interesting | |
| 14:32:40 | sean-k-mooney | efried: i abandonded my nic feature based schduling work in favour of using this in the furutre but not sure ill be working on that now | |
| 14:32:52 | mdbooth | lyarwood: Still investigating your functional failures, came across _terminate_volume_connections in ComputeManager, which does exactly what I proposed | |
| 14:33:03 | sean-k-mooney | jaypipes: im assuming you would prefer a resouce_class to track that | |
| 14:33:12 | jaypipes | sean-k-mooney: no. | |
| 14:33:16 | sean-k-mooney | jaypipes: no? | |
| 14:33:17 | mdbooth | lyarwood: Specifically create a blank attachment, delete the old attachment, update the bdm to point to the blank. | |
| 14:33:29 | jaypipes | sean-k-mooney: I'm just saying traits are capabilities. they aren't key/value metadata items. | |
| 14:33:37 | efried | sean-k-mooney: I think jaypipes wants traits for the *capabilities* associated with a vendor/product | |
| 14:33:38 | sean-k-mooney | ah ok | |
| 14:33:44 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/pike: Fix message for unexpected external event https://review.openstack.org/589503 | |
| 14:33:45 | jaypipes | efried: bingo. | |
| 14:33:56 | sean-k-mooney | yep i agree that they are capablityes not metadata | |
| 14:34:01 | efried | which irl will entail maintaining a matrix of vendor/product to capabilities | |
| 14:34:18 | jaypipes | efried: correct. | |
| 14:34:18 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Fix message for unexpected external event https://review.openstack.org/589505 | |
| 14:34:24 | efried | which means whenever a new device is introduced that we want to support, we need to change code. | |
| 14:34:24 | sean-k-mooney | i was implying that vendor/product id is metadata that should be associated with the resouce_class | |
| 14:34:37 | jaypipes | efried: which... *gasp* the friggin vendors should be responsible for. | |
| 14:34:53 | efried | jaypipes: The vendors should be responsible for proposing nova patches to support their devices? | |
| 14:34:59 | sean-k-mooney | i had asked previously about intodusing resouce class metadata at some point but did not really push the point in the past | |
| 14:35:21 | jaypipes | efried: no. the vendors should be responsible for keeping the matrix of capabilities up to date with their product lines. | |
| 14:35:41 | efried | jaypipes: And that matrix of capabilities should be discoverable by querying the device somehow | |
| 14:35:54 | jaypipes | efried: in the same way they are responsible for ensuring the pciids database is kept up to date with all their vendor, subvendor/reseller and product information. | |
| 14:35:58 | sean-k-mooney | jaypipes: well the vendor id/product id is important for other reasons such as knowing what driver is required for the device | |
| 14:36:19 | efried | If that were the case, and if vendors could agree on names (IDs?) for capabilities across the board, I could get behind it. | |
| 14:36:59 | sean-k-mooney | efried: well intel has been pushing to try and stardise some of them in etsi and dmtf(redfish) | |
| 14:37:00 | efried | But I think we need to have a realistic fallback plan so that, if such a nirvana does not come into being, operators will still be able to ask for a GPU by product ID. | |
| 14:37:24 | sean-k-mooney | from an openstack point of view that what the standard traits in os-traits are for | |
| 14:37:59 | jaypipes | efried: well, if vendors want to enable their technology in OpenStack, they should work with us. Instead, nova needs to jump through a bunch of hacky hoops to work with vendors. ala https://review.openstack.org/#/c/579897/. which is complete bullshit, IMHO. | |
| 14:38:22 | efried | yes, I noticed you getting excited about that yesterday :) | |
| 14:39:16 | sean-k-mooney | jaypipes: ya.. well that is working around a limitation that nvidia put in there driver to prevent you using there gpus in vm unless you baught there datacenter gpus | |
| 14:39:37 | jaypipes | sean-k-mooney: I'm fully aware of that, yes. | |
| 14:40:14 | jaypipes | sean-k-mooney: please see my comment on whether we are OpenStack or WorkaroundForClosedStack | |
| 14:40:42 | jaypipes | sean-k-mooney: I think the answer has already been answered on that, actually. we have been VendorStack for about 5 years now. | |
| 14:40:45 | sean-k-mooney | oh i see it i have it open now. i agree that we should not need to do this but its a usecase they have | |
| 14:41:49 | sean-k-mooney | i think there are usecase beyond the gpu case for wanting to hide the hypervios signture | |
| 14:42:01 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667 | |
| 14:42:01 | jaypipes | name one. | |
| 14:42:11 | sean-k-mooney | linux kernel dev | |
| 14:42:57 | sean-k-mooney | anyway i know that is a reach but for ovs-dpdk i dev i have used trick in the past to allow me to use openstack as a dev env | |
| 14:43:19 | sean-k-mooney | such as setting the nic model to e1000 to i could test the non viutalised nic binding workflow | |
| 14:44:40 | jaypipes | sean-k-mooney: you're performing use case gymnastics here. | |
| 14:45:37 | sean-k-mooney | yes and no. in the early days of dpdk support i had to do thing like set the cpu-mode to host-passthough becasue i did not have the cpu feature reqiure to even compile it otherwise | |
| 14:46:35 | dansmith | jaypipes: it's a common case for kernel development to use virtualization to simulate environments for that dev.. like fake numa nodes, pci devices, etc | |
| 14:46:39 | sean-k-mooney | as i said i was reaching a bit but if i want to do driver dev and i want to do that in a vm on openstack then i might need to hide the hypervisor signiture. that said i also recognise that openstack is not a virtualisation stack | |
| 14:46:49 | dansmith | jaypipes: that's not to say that I think IaaS really needs to worry about such things | |
| 14:47:05 | dansmith | jaypipes: given that developers can use bare hypervisors on their laptops to do that with much better control | |
| 14:47:47 | sean-k-mooney | dansmith: yep that said for ci of that code openstack is a much more compelling option | |
| 14:48:05 | dansmith | yup, CI is another reasonable use case imho | |
| 14:49:04 | sean-k-mooney | the alternitive without hypervior hiding is ironic but that is arguable more complex. the difference is the complexity is on the enduer not the technical debt nova has to maintain | |
| 14:49:38 | dansmith | sean-k-mooney: note that I'm defending the dev case for needing finer grained control, but *not* defending the hypervisor-hiding case | |
| 14:49:51 | dansmith | the latter is mostly total closed-source proprietary BS | |
| 14:50:20 | jaypipes | sean-k-mooney, dansmith: in that case, the use case would be to be able to *set* the vendor ID to a particular value, not hardcode it to "1234567890ab" just for the HyperV emulator just to avoid licensing issues with Nvidia: https://review.openstack.org/#/c/579897/4/nova/virt/libvirt/config.py@2241 | |
| 14:50:50 | lyarwood | s10: hey, thanks for the report, so what's the actual knock of impact of update_available_resource taking longer here? Scheduling accuracy? | |
| 14:51:26 | sean-k-mooney | jaypipes: well i had originally asked shoudl we just remove the hypervier section entirly if hiding was asked for to fully hide the fact we were in a vm but that has performance impacts | |
| 14:52:34 | sean-k-mooney | but yes making this more generic to allow setting a spefic vendor id would be just as vaild | |
| 14:53:06 | s10 | lyarwood: It looks like, when I try to live migrate instances from this host, all operations should wait for this update_available_resource every minute. So I have a small live migration window: 30 seconds every minute, only after update_available_resource ends and before it starts. | |
| 14:54:34 | sean-k-mooney | s10: why? | |
| 14:55:28 | jaypipes | s10: why are you running update_available_resource every minute? | |
| 14:55:45 | s10 | sean-k-mooney: I don't know, that's what I see in logs. No live migration could be performed during update_available_resources. If change libvirt/driver.py to like before first commits, migrations goes without pause. | |
| 14:56:02 | cdent | jaypipes: that's the periodic job default | |
| 14:56:15 | sean-k-mooney | s10: well that makes more sense | |
| 14:56:23 | jaypipes | quoi? wtf is it that often? :( | |
| 14:56:35 | sean-k-mooney | we are running the periodic job on a green thread | |
| 14:56:43 | cdent | jaypipes: dunno, but that's what it is | |
| 14:56:50 | jaypipes | sean-k-mooney: there is a semaphore lock around it. | |
| 14:57:26 | sean-k-mooney | so if update_available_resources is taking 30sec on a node with 150 instance then the compute agent would be tied up for 30secs every minute executing it | |
| 14:58:24 | s10 | jaypipes: because update_resources_interval run with default periodic interval with default value update_resources_interval=0, and periodic_task_interval=60 by default. | |
| 14:58:30 | sean-k-mooney | jaypipes: the point im trying to make is the periodic jobs are time sharing with the rest of the compute agent | |
| 14:59:18 | stephenfin | efried: Does POWER expose NUMA to the OS? | |
| 14:59:35 | sean-k-mooney | stephenfin: as in powerpc ? | |
| 14:59:41 | stephenfin | sean-k-mooney: up | |
| 14:59:42 | stephenfin | *yup | |
| 14:59:45 | dansmith | sean-k-mooney: but most of that is waiting for IO so it's not keeping the process busy | |
| 14:59:47 | mdbooth | lyarwood: I have another meeting now, so I've chucked a brain dump in the functional failure review. | |
| 14:59:56 | efried | stephenfin: My understanding is that POWER handles NUMA under the covers, and does it well enough that the deployer doesn't need control. | |
| 15:00:07 | sean-k-mooney | stephenfin: numa is exposed to linux via the bios so it should would the same on powerpc | |