| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-23 | |||
| 14:00:15 | bauwser | like I wrote, we can have Nova to ask for both resource classes if it sees that | |
| 14:00:23 | fried_rice | So look, there are ways to handle this with and without changing the placement API | |
| 14:00:33 | fried_rice | yeah, that'd be one way. | |
| 14:00:47 | sean-k-mooney[m] | Existing flavours without numa_nodes set dont have numa affinity for cpus | |
| 14:00:53 | fried_rice | The other way is to have nova see that the flavor doesn't care about NUMA affinity and *nova* splits up the resource requests | |
| 14:00:56 | bauwser | I mean, if a flavor tells only about vCPUs, then query Placement only for that, but make allocations for both VCPU and NUMA_SOMETHING_VCPU | |
| 14:01:11 | bauwser | fried_rice: that's what I wrote in the spec | |
| 14:01:16 | bauwser | nova could make the translation | |
| 14:01:38 | bauwser | leakypipes: ^ I hope you follow | |
| 14:01:46 | fried_rice | bauwser: Cool; either way, we're having nova massage the request. But I prefer the path where we're actually representing true inventory numbers. | |
| 14:01:48 | sean-k-mooney[m] | Fried_rice how does nova know how to split it up | |
| 14:02:12 | fried_rice | sean-k-mooney[m]: For VCPU, it's one per request group, plain and simple. | |
| 14:02:13 | bauwser | sean-k-mooney[m]: there are two different things, the placement query and the allocation | |
| 14:02:39 | bauwser | fried_rice: also, note I'd like to not depend on request groups for NUMA | |
| 14:02:48 | fried_rice | bauwser: Whyzat? | |
| 14:03:09 | fried_rice | I don't think you can get around it, especially when you start talking about e.g. NICs. | |
| 14:03:12 | sean-k-mooney[m] | Flavor has 16 vcpus and dose not care about numa. All hosts have 2 10 cpus. How does nova know it needs to split into two groups | |
| 14:03:27 | bauwser | again, that requires operators opiniating about which specific placement they should do per flavor for non-NUMA specific instances | |
| 14:03:36 | fried_rice | sean-k-mooney[m]: it doesn't split into two groups. It splits into 16 | |
| 14:03:51 | bauwser | sean-k-mooney[m]: my vision was that we would ask placement only for VCPUs that are on the root RP | |
| 14:03:55 | fried_rice | Sorry guys, got to run for a bit. | |
| 14:04:07 | bauwser | sean-k-mooney[m]: but once we get a RP, we would make the allocation against a child | |
| 14:04:12 | bauwser | well, erm | |
| 14:04:16 | bauwser | nevermind | |
| 14:04:19 | bauwser | it's stupid | |
| 14:04:34 | bauwser | what we need is to get all the children that match the query | |
| 14:04:34 | sean-k-mooney[m] | So 16 request of 1 cpu | |
| 14:04:45 | bauwser | and only consume one of those | |
| 14:04:59 | bauwser | fuuu, I'm getting headaches | |
| 14:05:59 | sean-k-mooney[m] | leakypipes: is this something we should cover in your spec | |
| 14:06:19 | figleaf | bauwser: "only in root" - that was one of the decisions we made back in Atlanta. A machine would be configured by the operator ahead of time. If a VCPU was assigned to a NUMA node, it could not be considered also to be part of the root compute node | |
| 14:06:51 | bauwser | sean-k-mooney[m]: NUMA related things should be covered in https://review.openstack.org/#/c/552924/3/specs/rocky/approved/numa-topology-with-rps.rst | |
| 14:07:38 | bauwser | figleaf: I don't disagree with that statement | |
| 14:07:46 | sean-k-mooney[m] | figleaf: almost all server systems are numa system even systems with one socket | |
| 14:08:05 | bauwser | figleaf: I'm just trying to evaluate the operator impact of a NoValidHost if we begin counting NUMA nodes for non-NUMA specific instances | |
| 14:08:25 | figleaf | sean-k-mooney[m]: sure, but the choice on how to consume them (respecting NUMA, or ignoring) is an up-front decision by the operator. | |
| 14:08:38 | sean-k-mooney[m] | bauwser: i think it also needs to be in jays cpu resources spec | |
| 14:09:10 | bauwser | figleaf: so you agree with the fact that a classic flavor just asking for VCPUs shouldn't be accounting NUMA nodes, right? | |
| 14:09:28 | sean-k-mooney[m] | figleaf: no its a choice of the virt driver | |
| 14:09:36 | bauwser | how the inventory is modeled is one thing | |
| 14:09:53 | bauwser | and I'm fine with having VCPU resources on a NUMA node if that helps | |
| 14:09:57 | figleaf | bauwser: of course, but it would only select a host that was configured *not* to use NUMA. IOW, the ram, vcpu, etc, was inventory of the compute | |
| 14:10:15 | bauwser | but my point is, I don't want to get a NoValidHost if that restricts my existing flavor to specific NUMA nodes | |
| 14:10:51 | openstackgerrit | Merged openstack/nova master: Change compute mgr placement check to region_name https://review.openstack.org/554759 | |
| 14:10:54 | bauwser | figleaf: that's super costly then | |
| 14:11:06 | openstackgerrit | Merged openstack/nova master: Remove translate and a TODO https://review.openstack.org/554827 | |
| 14:11:07 | figleaf | bauwser: if you configure all your machines not to use NUMA, and request a NUMA-specific flavor, you will get NoValidHost. The reverse would also be true | |
| 14:11:20 | bauwser | figleaf: because I want to have both compute workloads that can do NUMA affinity and regular workloads that don't care | |
| 14:11:42 | bauwser | figleaf: again, that's a huuuuuge behavioural change | |
| 14:11:50 | figleaf | bauwser: sure, then configure some of your machines for NUMA, and others for not-NUMA | |
| 14:11:58 | bauwser | it's suboptimal | |
| 14:11:58 | figleaf | The balance is up to the operator | |
| 14:12:09 | sean-k-mooney[m] | figleaf: we don't have a compute node option for ignore numa | |
| 14:12:27 | cdent | making the optimal solution is introducing an enormous amount of complexity that is making hard to finish _any_ solution | |
| 14:12:28 | bauwser | sean-k-mooney[m]: I guess figleaf wants it | |
| 14:12:46 | sean-k-mooney[m] | Figleaf but that is not something you can configure normally | |
| 14:12:53 | bauwser | if you opt for non-NUMA, then you get a flat inventoty | |
| 14:13:14 | bauwser | if you opt for NUMA topology, then you're stuck with NUMA-only workloads on that single node | |
| 14:13:34 | bauwser | :/ | |
| 14:14:04 | figleaf | sean-k-mooney[m]: it would be how resources are reported. If the virt driver reports that a NUMA node owns some inventory, then that NUMA node has to satisfy the request. There is no concept in placement for the kind of sharing you're talking about | |
| 14:14:05 | bauwser | cdent: that's the whole purpose of the spec | |
| 14:14:28 | cdent | bauwser: which "that's"? | |
| 14:14:30 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 14:14:34 | bauwser | cdent: to see if we can model the NUMA inventories in any other ways that would help to solve the problem without requiring too much engineering | |
| 14:15:06 | leakypipes | cfriesen, bauwser, sahid, sean-k-mooney[m], finucannot, fried_rice: ^^ ok, new revision pushed. lemme know if that looks a bit better. | |
| 14:15:11 | sean-k-mooney[m] | figleaf: again how would yo do that. Numa is a feature of the hardware its not a software option so your talking about ordering hardware with out multiple memory controllers or configure mirrors dimms in the bios | |
| 14:15:18 | bauwser | leakypipes: you missed a NUMA conversation | |
| 14:15:31 | cdent | bauwser: okay. My statement was more along the lines of "trying to satisfy all this NUMA and other enhanced platform stuff, in general, is breaking the 80/20 rule" | |
| 14:15:55 | leakypipes | bauwser: not really. see the spec. :) | |
| 14:16:00 | bauwser | cdent: the problem is that like sean-k-mooney[m] said, modern architectures are NUMA-based, but 80% of our workloads don't care | |
| 14:16:13 | sean-k-mooney[m] | figleaf: not in placement but that is how nova works today. | |
| 14:16:33 | sean-k-mooney[m] | leakypipes: cool will do | |
| 14:16:36 | bauwser | leakypipes: hah, teaser! | |
| 14:16:37 | openstackgerrit | Merged openstack/nova master: Add the version description for InstanceActionEventList https://review.openstack.org/555711 | |
| 14:16:39 | bauwser | I like that | |
| 14:16:53 | sean-k-mooney[m] | Lleakypipes yes this is related to that spec too | |
| 14:17:14 | figleaf | sean-k-mooney[m]: it is software that reports what hardware exists to placement. If it configure to respect the NUMA layout, it reports the inventory that NUMA nodes "own". If it is configured to sum up the resources, then it reports inventory of the compute node. | |
| 14:17:17 | bauwser | I quote "placement is the most ambitious crossover of the history" | |
| 14:18:02 | sean-k-mooney[m] | cdent: well in this case we are breaking insance that dont care about numa by fitting them to numa nodes | |
| 14:19:13 | bauwser | leakypipes: https://twitter.com/sylvainbauza/status/977187898167611392 | |
| 14:20:34 | giblet | bauwser: lol | |
| 14:21:18 | openstackgerrit | Merged openstack/nova master: remove _cleanup_running_deleted_instances repeat detach volume https://review.openstack.org/554090 | |
| 14:21:34 | openstackgerrit | Merged openstack/nova master: deprecate fping_path config option https://review.openstack.org/526602 | |
| 14:22:09 | mriedem | there is no reason we need to warn for this right? http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n3340 | |
| 14:22:22 | mriedem | the user is trying to do something the api allows them to do, so the operator shouldn't be warned about it | |
| 14:22:46 | mriedem | https://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2717 | |
| 14:22:51 | bauwser | I think it's time for me writing a new rev' for NUMA and stating the concerns in the spexc | |
| 14:25:07 | cdent | my laptop battery and my brain battery are dying | |
| 14:26:13 | bauwser | cdent: take a power bank | |
| 14:26:31 | cdent | bauwser: you got one for brains? hook me up | |
| 14:26:33 | bauwser | cdent: that said, I can't perclue where to plug the bank | |
| 14:27:35 | bauwser | s/perclue/guess | |
| 14:28:03 | bauwser | note for self, stop buying expensive nespresso machines that are just subject to failures | |
| 14:28:23 | bauwser | and start finding other ways for making coffee | |
| 14:29:58 | openstackgerrit | Tyler Blakeslee proposed openstack/nova master: Add __repr__ for NovaException https://review.openstack.org/555812 | |
| 14:31:35 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527 | |
| 14:31:36 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add --enable and --disable options to nova-manage update_cell https://review.openstack.org/555416 | |
| 14:31:36 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Modify nova-manage cell_v2 list_cells to display "disabled" column https://review.openstack.org/555415 | |
| 14:31:37 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled option to create_cell command https://review.openstack.org/555417 | |
| 14:35:20 | sahid | leakypipes: ++ on you spec cpu-resources | |