| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-23 | |||
| 14:06:51 | bauwser | sean-k-mooney[m]: NUMA related things should be covered in https://review.openstack.org/#/c/552924/3/specs/rocky/approved/numa-topology-with-rps.rst | |
| 14:07:38 | bauwser | figleaf: I don't disagree with that statement | |
| 14:07:46 | sean-k-mooney[m] | figleaf: almost all server systems are numa system even systems with one socket | |
| 14:08:05 | bauwser | figleaf: I'm just trying to evaluate the operator impact of a NoValidHost if we begin counting NUMA nodes for non-NUMA specific instances | |
| 14:08:25 | figleaf | sean-k-mooney[m]: sure, but the choice on how to consume them (respecting NUMA, or ignoring) is an up-front decision by the operator. | |
| 14:08:38 | sean-k-mooney[m] | bauwser: i think it also needs to be in jays cpu resources spec | |
| 14:09:10 | bauwser | figleaf: so you agree with the fact that a classic flavor just asking for VCPUs shouldn't be accounting NUMA nodes, right? | |
| 14:09:28 | sean-k-mooney[m] | figleaf: no its a choice of the virt driver | |
| 14:09:36 | bauwser | how the inventory is modeled is one thing | |
| 14:09:53 | bauwser | and I'm fine with having VCPU resources on a NUMA node if that helps | |
| 14:09:57 | figleaf | bauwser: of course, but it would only select a host that was configured *not* to use NUMA. IOW, the ram, vcpu, etc, was inventory of the compute | |
| 14:10:15 | bauwser | but my point is, I don't want to get a NoValidHost if that restricts my existing flavor to specific NUMA nodes | |
| 14:10:51 | openstackgerrit | Merged openstack/nova master: Change compute mgr placement check to region_name https://review.openstack.org/554759 | |
| 14:10:54 | bauwser | figleaf: that's super costly then | |
| 14:11:06 | openstackgerrit | Merged openstack/nova master: Remove translate and a TODO https://review.openstack.org/554827 | |
| 14:11:07 | figleaf | bauwser: if you configure all your machines not to use NUMA, and request a NUMA-specific flavor, you will get NoValidHost. The reverse would also be true | |
| 14:11:20 | bauwser | figleaf: because I want to have both compute workloads that can do NUMA affinity and regular workloads that don't care | |
| 14:11:42 | bauwser | figleaf: again, that's a huuuuuge behavioural change | |
| 14:11:50 | figleaf | bauwser: sure, then configure some of your machines for NUMA, and others for not-NUMA | |
| 14:11:58 | figleaf | The balance is up to the operator | |
| 14:11:58 | bauwser | it's suboptimal | |
| 14:12:09 | sean-k-mooney[m] | figleaf: we don't have a compute node option for ignore numa | |
| 14:12:27 | cdent | making the optimal solution is introducing an enormous amount of complexity that is making hard to finish _any_ solution | |
| 14:12:28 | bauwser | sean-k-mooney[m]: I guess figleaf wants it | |
| 14:12:46 | sean-k-mooney[m] | Figleaf but that is not something you can configure normally | |
| 14:12:53 | bauwser | if you opt for non-NUMA, then you get a flat inventoty | |
| 14:13:14 | bauwser | if you opt for NUMA topology, then you're stuck with NUMA-only workloads on that single node | |
| 14:13:34 | bauwser | :/ | |
| 14:14:04 | figleaf | sean-k-mooney[m]: it would be how resources are reported. If the virt driver reports that a NUMA node owns some inventory, then that NUMA node has to satisfy the request. There is no concept in placement for the kind of sharing you're talking about | |
| 14:14:05 | bauwser | cdent: that's the whole purpose of the spec | |
| 14:14:28 | cdent | bauwser: which "that's"? | |
| 14:14:30 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 14:14:34 | bauwser | cdent: to see if we can model the NUMA inventories in any other ways that would help to solve the problem without requiring too much engineering | |
| 14:15:06 | leakypipes | cfriesen, bauwser, sahid, sean-k-mooney[m], finucannot, fried_rice: ^^ ok, new revision pushed. lemme know if that looks a bit better. | |
| 14:15:11 | sean-k-mooney[m] | figleaf: again how would yo do that. Numa is a feature of the hardware its not a software option so your talking about ordering hardware with out multiple memory controllers or configure mirrors dimms in the bios | |
| 14:15:18 | bauwser | leakypipes: you missed a NUMA conversation | |
| 14:15:31 | cdent | bauwser: okay. My statement was more along the lines of "trying to satisfy all this NUMA and other enhanced platform stuff, in general, is breaking the 80/20 rule" | |
| 14:15:55 | leakypipes | bauwser: not really. see the spec. :) | |
| 14:16:00 | bauwser | cdent: the problem is that like sean-k-mooney[m] said, modern architectures are NUMA-based, but 80% of our workloads don't care | |
| 14:16:13 | sean-k-mooney[m] | figleaf: not in placement but that is how nova works today. | |
| 14:16:33 | sean-k-mooney[m] | leakypipes: cool will do | |
| 14:16:36 | bauwser | leakypipes: hah, teaser! | |
| 14:16:37 | openstackgerrit | Merged openstack/nova master: Add the version description for InstanceActionEventList https://review.openstack.org/555711 | |
| 14:16:39 | bauwser | I like that | |
| 14:16:53 | sean-k-mooney[m] | Lleakypipes yes this is related to that spec too | |
| 14:17:14 | figleaf | sean-k-mooney[m]: it is software that reports what hardware exists to placement. If it configure to respect the NUMA layout, it reports the inventory that NUMA nodes "own". If it is configured to sum up the resources, then it reports inventory of the compute node. | |
| 14:17:17 | bauwser | I quote "placement is the most ambitious crossover of the history" | |
| 14:18:02 | sean-k-mooney[m] | cdent: well in this case we are breaking insance that dont care about numa by fitting them to numa nodes | |
| 14:19:13 | bauwser | leakypipes: https://twitter.com/sylvainbauza/status/977187898167611392 | |
| 14:20:34 | giblet | bauwser: lol | |
| 14:21:18 | openstackgerrit | Merged openstack/nova master: remove _cleanup_running_deleted_instances repeat detach volume https://review.openstack.org/554090 | |
| 14:21:34 | openstackgerrit | Merged openstack/nova master: deprecate fping_path config option https://review.openstack.org/526602 | |
| 14:22:09 | mriedem | there is no reason we need to warn for this right? http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n3340 | |
| 14:22:22 | mriedem | the user is trying to do something the api allows them to do, so the operator shouldn't be warned about it | |
| 14:22:46 | mriedem | https://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2717 | |
| 14:22:51 | bauwser | I think it's time for me writing a new rev' for NUMA and stating the concerns in the spexc | |
| 14:25:07 | cdent | my laptop battery and my brain battery are dying | |
| 14:26:13 | bauwser | cdent: take a power bank | |
| 14:26:31 | cdent | bauwser: you got one for brains? hook me up | |
| 14:26:33 | bauwser | cdent: that said, I can't perclue where to plug the bank | |
| 14:27:35 | bauwser | s/perclue/guess | |
| 14:28:03 | bauwser | note for self, stop buying expensive nespresso machines that are just subject to failures | |
| 14:28:23 | bauwser | and start finding other ways for making coffee | |
| 14:29:58 | openstackgerrit | Tyler Blakeslee proposed openstack/nova master: Add __repr__ for NovaException https://review.openstack.org/555812 | |
| 14:31:35 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527 | |
| 14:31:36 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Modify nova-manage cell_v2 list_cells to display "disabled" column https://review.openstack.org/555415 | |
| 14:31:36 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add --enable and --disable options to nova-manage update_cell https://review.openstack.org/555416 | |
| 14:31:37 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled option to create_cell command https://review.openstack.org/555417 | |
| 14:35:20 | sahid | leakypipes: ++ on you spec cpu-resources | |
| 14:38:51 | bauwser | leakypipes: time for a question ? | |
| 14:39:14 | bauwser | leakypipes: about https://review.openstack.org/#/c/555081/3/specs/rocky/approved/cpu-resources.rst@249 and beyond | |
| 14:39:32 | leakypipes | bauwser: sure | |
| 14:39:39 | leakypipes | sahid: +2 on tx/rx | |
| 14:39:47 | bauwser | leakypipes: just noting a few things on the spec | |
| 14:39:47 | leakypipes | sahid: I will let mriedem +W it though | |
| 14:40:04 | bauwser | leakypipes: sahid: the TX/RX was next in my review queue | |
| 14:40:05 | superdan | leakypipes: mriedem said he was leaving it to us in the meeting, so I think you can +W | |
| 14:40:14 | bauwser | I have a fucking low number of reviews this week | |
| 14:40:55 | leakypipes | superdan: got it. done. | |
| 14:41:19 | leakypipes | bauwser: reading specs takes a long time... | |
| 14:41:28 | sahid | superdan thanks for having so active on it | |
| 14:41:42 | superdan | sahid: s/having/being/ | |
| 14:41:50 | leakypipes | :) | |
| 14:41:52 | sahid | :) | |
| 14:44:40 | mriedem | kashyap: i updated https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix for queens and added the ubuntu 18.04 LTS package versions based on https://packages.ubuntu.com/bionic/allpackages | |
| 14:44:46 | figleaf | leakypipes: quick question about the spec. When there are NUMA nodes with V/PCPU inventory, and a non-granular request is received, will there be allocation candidates returned for every possible combination of NUMA providers? | |
| 14:44:48 | mriedem | looks like all of the other distros in that table are old | |
| 14:45:00 | kashyap | mriedem: Ah-ha. I have it open in one tab | |
| 14:45:11 | kashyap | And was reviewing your historical changes there yesterday | |
| 14:45:17 | mriedem | kashyap: so since we didn't bump in queens, | |
| 14:45:18 | kashyap | Will get to it, once I respond to this bug-from-hell downstream | |
| 14:45:35 | mriedem | we already have a 'next min version' in the driver, which is actually kind of old now, libvirt 1.3.1 | |
| 14:45:37 | leakypipes | figleaf: yes, alloc candidates would contain alloc requests against each NUMA node that had space to fit the sum amount of requested VCPU/PCPU | |
| 14:45:48 | mriedem | the question is then what will the next min version be in Solar | |
| 14:45:53 | kashyap | What is Solar? | |
| 14:45:54 | leakypipes | figleaf: I can add a note to clarify that. | |
| 14:45:57 | kashyap | A new Ubuntu version? | |
| 14:46:01 | mriedem | solar is next openstack | |
| 14:46:12 | mriedem | (pending that name is approved by legal) | |
| 14:46:15 | kashyap | Aah, I didn't check the release naming for the next one | |