| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-23 | |||
| 14:11:07 | figleaf | bauwser: if you configure all your machines not to use NUMA, and request a NUMA-specific flavor, you will get NoValidHost. The reverse would also be true | |
| 14:11:20 | bauwser | figleaf: because I want to have both compute workloads that can do NUMA affinity and regular workloads that don't care | |
| 14:11:42 | bauwser | figleaf: again, that's a huuuuuge behavioural change | |
| 14:11:50 | figleaf | bauwser: sure, then configure some of your machines for NUMA, and others for not-NUMA | |
| 14:11:58 | bauwser | it's suboptimal | |
| 14:11:58 | figleaf | The balance is up to the operator | |
| 14:12:09 | sean-k-mooney[m] | figleaf: we don't have a compute node option for ignore numa | |
| 14:12:27 | cdent | making the optimal solution is introducing an enormous amount of complexity that is making hard to finish _any_ solution | |
| 14:12:28 | bauwser | sean-k-mooney[m]: I guess figleaf wants it | |
| 14:12:46 | sean-k-mooney[m] | Figleaf but that is not something you can configure normally | |
| 14:12:53 | bauwser | if you opt for non-NUMA, then you get a flat inventoty | |
| 14:13:14 | bauwser | if you opt for NUMA topology, then you're stuck with NUMA-only workloads on that single node | |
| 14:13:34 | bauwser | :/ | |
| 14:14:04 | figleaf | sean-k-mooney[m]: it would be how resources are reported. If the virt driver reports that a NUMA node owns some inventory, then that NUMA node has to satisfy the request. There is no concept in placement for the kind of sharing you're talking about | |
| 14:14:05 | bauwser | cdent: that's the whole purpose of the spec | |
| 14:14:28 | cdent | bauwser: which "that's"? | |
| 14:14:30 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 14:14:34 | bauwser | cdent: to see if we can model the NUMA inventories in any other ways that would help to solve the problem without requiring too much engineering | |
| 14:15:06 | leakypipes | cfriesen, bauwser, sahid, sean-k-mooney[m], finucannot, fried_rice: ^^ ok, new revision pushed. lemme know if that looks a bit better. | |
| 14:15:11 | sean-k-mooney[m] | figleaf: again how would yo do that. Numa is a feature of the hardware its not a software option so your talking about ordering hardware with out multiple memory controllers or configure mirrors dimms in the bios | |
| 14:15:18 | bauwser | leakypipes: you missed a NUMA conversation | |
| 14:15:31 | cdent | bauwser: okay. My statement was more along the lines of "trying to satisfy all this NUMA and other enhanced platform stuff, in general, is breaking the 80/20 rule" | |
| 14:15:55 | leakypipes | bauwser: not really. see the spec. :) | |
| 14:16:00 | bauwser | cdent: the problem is that like sean-k-mooney[m] said, modern architectures are NUMA-based, but 80% of our workloads don't care | |
| 14:16:13 | sean-k-mooney[m] | figleaf: not in placement but that is how nova works today. | |
| 14:16:33 | sean-k-mooney[m] | leakypipes: cool will do | |
| 14:16:36 | bauwser | leakypipes: hah, teaser! | |
| 14:16:37 | openstackgerrit | Merged openstack/nova master: Add the version description for InstanceActionEventList https://review.openstack.org/555711 | |
| 14:16:39 | bauwser | I like that | |
| 14:16:53 | sean-k-mooney[m] | Lleakypipes yes this is related to that spec too | |
| 14:17:14 | figleaf | sean-k-mooney[m]: it is software that reports what hardware exists to placement. If it configure to respect the NUMA layout, it reports the inventory that NUMA nodes "own". If it is configured to sum up the resources, then it reports inventory of the compute node. | |
| 14:17:17 | bauwser | I quote "placement is the most ambitious crossover of the history" | |
| 14:18:02 | sean-k-mooney[m] | cdent: well in this case we are breaking insance that dont care about numa by fitting them to numa nodes | |
| 14:19:13 | bauwser | leakypipes: https://twitter.com/sylvainbauza/status/977187898167611392 | |
| 14:20:34 | giblet | bauwser: lol | |
| 14:21:18 | openstackgerrit | Merged openstack/nova master: remove _cleanup_running_deleted_instances repeat detach volume https://review.openstack.org/554090 | |
| 14:21:34 | openstackgerrit | Merged openstack/nova master: deprecate fping_path config option https://review.openstack.org/526602 | |
| 14:22:09 | mriedem | there is no reason we need to warn for this right? http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n3340 | |
| 14:22:22 | mriedem | the user is trying to do something the api allows them to do, so the operator shouldn't be warned about it | |
| 14:22:46 | mriedem | https://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2717 | |
| 14:22:51 | bauwser | I think it's time for me writing a new rev' for NUMA and stating the concerns in the spexc | |
| 14:25:07 | cdent | my laptop battery and my brain battery are dying | |
| 14:26:13 | bauwser | cdent: take a power bank | |
| 14:26:31 | cdent | bauwser: you got one for brains? hook me up | |
| 14:26:33 | bauwser | cdent: that said, I can't perclue where to plug the bank | |
| 14:27:35 | bauwser | s/perclue/guess | |
| 14:28:03 | bauwser | note for self, stop buying expensive nespresso machines that are just subject to failures | |
| 14:28:23 | bauwser | and start finding other ways for making coffee | |
| 14:29:58 | openstackgerrit | Tyler Blakeslee proposed openstack/nova master: Add __repr__ for NovaException https://review.openstack.org/555812 | |
| 14:31:35 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527 | |
| 14:31:36 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add --enable and --disable options to nova-manage update_cell https://review.openstack.org/555416 | |
| 14:31:36 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Modify nova-manage cell_v2 list_cells to display "disabled" column https://review.openstack.org/555415 | |
| 14:31:37 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled option to create_cell command https://review.openstack.org/555417 | |
| 14:35:20 | sahid | leakypipes: ++ on you spec cpu-resources | |
| 14:38:51 | bauwser | leakypipes: time for a question ? | |
| 14:39:14 | bauwser | leakypipes: about https://review.openstack.org/#/c/555081/3/specs/rocky/approved/cpu-resources.rst@249 and beyond | |
| 14:39:32 | leakypipes | bauwser: sure | |
| 14:39:39 | leakypipes | sahid: +2 on tx/rx | |
| 14:39:47 | leakypipes | sahid: I will let mriedem +W it though | |
| 14:39:47 | bauwser | leakypipes: just noting a few things on the spec | |
| 14:40:04 | bauwser | leakypipes: sahid: the TX/RX was next in my review queue | |
| 14:40:05 | superdan | leakypipes: mriedem said he was leaving it to us in the meeting, so I think you can +W | |
| 14:40:14 | bauwser | I have a fucking low number of reviews this week | |
| 14:40:55 | leakypipes | superdan: got it. done. | |
| 14:41:19 | leakypipes | bauwser: reading specs takes a long time... | |
| 14:41:28 | sahid | superdan thanks for having so active on it | |
| 14:41:42 | superdan | sahid: s/having/being/ | |
| 14:41:50 | leakypipes | :) | |
| 14:41:52 | sahid | :) | |
| 14:44:40 | mriedem | kashyap: i updated https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix for queens and added the ubuntu 18.04 LTS package versions based on https://packages.ubuntu.com/bionic/allpackages | |
| 14:44:46 | figleaf | leakypipes: quick question about the spec. When there are NUMA nodes with V/PCPU inventory, and a non-granular request is received, will there be allocation candidates returned for every possible combination of NUMA providers? | |
| 14:44:48 | mriedem | looks like all of the other distros in that table are old | |
| 14:45:00 | kashyap | mriedem: Ah-ha. I have it open in one tab | |
| 14:45:11 | kashyap | And was reviewing your historical changes there yesterday | |
| 14:45:17 | mriedem | kashyap: so since we didn't bump in queens, | |
| 14:45:18 | kashyap | Will get to it, once I respond to this bug-from-hell downstream | |
| 14:45:35 | mriedem | we already have a 'next min version' in the driver, which is actually kind of old now, libvirt 1.3.1 | |
| 14:45:37 | leakypipes | figleaf: yes, alloc candidates would contain alloc requests against each NUMA node that had space to fit the sum amount of requested VCPU/PCPU | |
| 14:45:48 | mriedem | the question is then what will the next min version be in Solar | |
| 14:45:53 | kashyap | What is Solar? | |
| 14:45:54 | leakypipes | figleaf: I can add a note to clarify that. | |
| 14:45:57 | kashyap | A new Ubuntu version? | |
| 14:46:01 | mriedem | solar is next openstack | |
| 14:46:12 | mriedem | (pending that name is approved by legal) | |
| 14:46:15 | kashyap | Aah, I didn't check the release naming for the next one | |
| 14:46:32 | leakypipes | I guess my suggestion of "Suck It, Trebek" didn't fly. | |
| 14:46:57 | figleaf | leakypipes: that's what I thought. When an a-c is chosen and allocated against, is there anything that constrains the building of the VM to the particular NUMA node that was allocated against? | |
| 14:47:32 | bauwser | figleaf: leakypipes: reviewed | |
| 14:47:33 | openstackgerrit | Merged openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605 | |
| 14:47:33 | leakypipes | figleaf: nothing besides the provider UUID, no. | |
| 14:48:14 | bauwser | figleaf: leakypipes: my main concern is that existing flavors that don't ask for NUMA specifics shouldn't be constrained on NUMA nodes | |
| 14:48:25 | figleaf | leakypipes: and there is no relation between the RP UUID in placement and a particular NUMA node in hardware, right? | |
| 14:48:45 | leakypipes | bauwser: they aren't "constrained". they just end up on a NUMA node is the host system is NUMA.. | |
| 14:48:52 | leakypipes | s/is/if/ | |
| 14:48:59 | bauwser | leakypipes: back up a second | |
| 14:49:12 | bauwser | leakypipes: I'm taking the example I had previously | |
| 14:49:20 | bauwser | say I ask for two VCPUs, period. | |
| 14:49:48 | bauwser | what if my compute only has only vCPU left on one NUMA node, but one other vCPU left on another NUMA node ? | |
| 14:49:59 | bauwser | at the moment, nova will satisfy that | |
| 14:50:18 | leakypipes | bauwser: if the host system is NUMA, then the VCPU inventories will be against child provider representing NUMA nodes. And the allocation candidates will contain allocation requests that consume against one of those NUMA node resource providers. | |