| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 13:36:12 | efried | yes, agree completely. | |
| 13:36:17 | bauzas | provided I'm clear on my spec | |
| 13:36:37 | stephenfin | sahid: So for 4 guest nodes, if it can fit 3 on one host node and 1 on the other, that would be OK | |
| 13:36:42 | bauzas | that, whatever logic I dislike on filter, it's not intended to be in Placement | |
| 13:36:56 | sahid | stephenfin: please don't do that | |
| 13:36:58 | efried | bauzas: Would even work if we wanted to support explicit sharding/anti-affinity - like I said earlier, the filter would be responsible for getting rid of the candidates where placement "incorrectly" clustered separate groups into the same RP. | |
| 13:37:03 | Shilpa | stephenfin: ok will check further, if i will get something, will let you know, thanks | |
| 13:37:04 | bauzas | ++ | |
| 13:37:06 | bauzas | efried: ++ | |
| 13:37:20 | efried | sahid: Please clarify. | |
| 13:37:21 | sahid | you are going to break the whole aim of numa toplogy | |
| 13:37:21 | stephenfin | sahid: But we're not breaking the NUMA affinity with PCI devices or the likes. At least one of those guest nodes would have to be placed on the same host NUMA node as the PCI device | |
| 13:37:39 | efried | sahid: You're saying that if we request 4 (host) nodes, we must get 4 (host) nodes? | |
| 13:37:45 | stephenfin | sahid: This is no different to what we do now | |
| 13:37:46 | sahid | yes | |
| 13:38:08 | mdbooth | The multiattach job seems to be unhappy today. Did I miss any traffic about that? | |
| 13:38:10 | efried | sahid: Yeah, bauzas pointed to the flavor docs earlier which strongly implied (to my understanding) that when I say 4, I mean "at most 4" | |
| 13:38:23 | efried | mdbooth: I was waiting for mriedem to arrive so I could pester him about that. | |
| 13:38:28 | stephenfin | sahid: If I have a guest with a PCI device and hw:numa_nodes=2, then only one guest NUMA node would be placed on the same node as the PCI device | |
| 13:38:33 | stephenfin | *same host nodes as | |
| 13:38:48 | stephenfin | sahid: Come to think of it, this would improve performance | |
| 13:38:51 | stephenfin | *could | |
| 13:38:52 | efried | mriedem: Here's an example: http://logs.openstack.org/49/560349/1/check/nova-multiattach/ce31aed/logs/screen-n-cpu.txt.gz?level=ERROR | |
| 13:38:56 | mdbooth | efried: Ok. I won't double-pester him, then :) | |
| 13:39:20 | sahid | stephenfin: again we should not accept the host | |
| 13:39:32 | sahid | we sould never break the guest numa topology | |
| 13:39:33 | stephenfin | sahid: e.g. if we could fit the two guest NUMA nodes on the same host NUMA node as the PCI device, we'd be guaranteeing affinity for all guest CPUs | |
| 13:39:33 | efried | mriedem: ...which looks kinda like something wrong in the CI env, not in the code. But I'm pretty ignorant in this realm. | |
| 13:39:36 | mriedem | did you guys try doing a logstash query to find out when that started and if it's just that job? | |
| 13:39:39 | stephenfin | as opposed to half of them | |
| 13:39:48 | mriedem | my guess would be it's related to https://review.openstack.org/#/c/554314/ | |
| 13:40:01 | stephenfin | "stephenfin: again we should not accept the host" what do you mean? | |
| 13:40:06 | efried | mriedem: See, this is why we ask you. | |
| 13:40:09 | sahid | as i have indicated it's all about performance and distance between cpu, memory channel and pci devices | |
| 13:40:13 | mdbooth | efried: Hehe | |
| 13:40:23 | efried | mriedem: You came up with that shit off the top of your head; it would have taken me years to figure out. | |
| 13:40:30 | sahid | you are going to break this affinity | |
| 13:40:32 | stephenfin | sahid: I don't think you're understanding what I'm saying | |
| 13:40:32 | mriedem | but i'll be gone some day | |
| 13:40:42 | sahid | stephenfin: re-reading, sorry | |
| 13:41:00 | stephenfin | sahid: Let's table this for discussion on the spec. I think bauzas has his answer for now :) | |
| 13:41:04 | mdbooth | Was about to say that looks like a platform error | |
| 13:41:08 | stephenfin | sahid: good talk! :) | |
| 13:41:23 | mriedem | efried: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Connection%20to%20libvirt%20failed%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%3A%20libvirtError%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 13:41:35 | sahid | stephenfin: sure | |
| 13:41:37 | mriedem | it's definitely due to that change, since it's just that job | |
| 13:42:06 | efried | stephenfin: I don't know if bauzas has his answer. The question was, "if the user asks for 4 nodes, do we gotta get exactly 4 nodes from the host, or is it okay if it's anywhere from 1..4?" | |
| 13:42:21 | efried | stephenfin: Before sahid chimed in, we thought "1..4 is fine" was the answer. | |
| 13:42:32 | efried | stephenfin: Now we don't know anymore. | |
| 13:42:45 | efried | stephenfin: And bauzas definitely needs that answer to do his next spec edit. | |
| 13:42:46 | mriedem | efried: i'm hoping that https://review.openstack.org/#/c/554317/ will handle it | |
| 13:43:19 | stephenfin | efried: Hmm, can we discuss that on the spec so I can write something longer. I'm clearly not doing a good job of explaining here | |
| 13:43:21 | efried | mriedem: But that guy is failing the same way http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/screen-n-cpu.txt.gz?level=ERROR | |
| 13:43:49 | stephenfin | and/or I'm missing what sahid is trying to say | |
| 13:44:01 | stephenfin | efried: bauzas can just choose one and we'll weigh in there | |
| 13:44:10 | efried | fair | |
| 13:47:19 | sahid | i just saying that our current behavior to handle numa topology for guests is right we should not consider ok a guest requesting 2 NUMA nodes fit on one host NUMA node | |
| 13:47:32 | mriedem | efried: i think this is the problem http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/devstacklog.txt.gz#_2018-04-11_18_10_24_388 | |
| 13:48:58 | sahid | that because it's all performance related and user are going to place thier applications based on guest topology. if we provide a fake the performance will be degraded | |
| 13:50:12 | efried | sahid: Is it ever the case that performance is *improved* by forcing resources to come from *separate* NUMA nodes? | |
| 13:50:25 | efried | ohh, this is the parallelism thing. | |
| 13:51:02 | mriedem | nvm | |
| 13:51:03 | mdbooth | mriedem: If you get a sec could you remove the procedural -2 on https://review.openstack.org/#/c/526346/ ? | |
| 13:51:24 | mriedem | done | |
| 13:51:29 | mdbooth | mriedem: Thanks | |
| 13:52:47 | sahid | efried: it's just that, the vCPUs threads and memory will be placed on the same host numa node, so yes the performance will be ever improved | |
| 13:53:08 | sahid | it's the only reason why numa exist | |
| 13:53:29 | efried | sahid: I understand how performance is improved by placing on the *same* node. I'm asking if it's important to be able to explicitly *separate*. | |
| 13:54:47 | efried | sahid: Because in the case we're talking about, --property hw:numa_nodes=N will spread your resources across *at most* N NUMA nodes. It could end up being anywhere from 1..N. (But it will never be >N) | |
| 13:54:51 | sahid | efried: it depends of the use-case and application. you can request only one. the point is to have cpu/memory on same node | |
| 13:54:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make the nova-multiattach job non-voting temporarily https://review.openstack.org/560909 | |
| 13:54:53 | mriedem | efried: stephenfin: ^ | |
| 13:55:46 | efried | mriedem: +2 | |
| 13:56:16 | stephenfin | sahid: Right, but the CPU and memory will be on the same node. Again, we're not suggesting splitting an individual guest NUMA node | |
| 13:56:16 | efried | sahid: So to my understanding, in this case, anything <N is a bonus. | |
| 13:56:35 | efried | sahid: ...unless it's not. Which is what we're trying to ascertain. | |
| 13:56:51 | bauzas | efried: stephenfin: sahid: I think I have my answer tho | |
| 13:57:00 | efried | bauzas: Which is what? | |
| 13:57:34 | bauzas | efried: stephenfin: sahid: in my spec, I'll propose Placement to return all hosts accepting up to N NUMA nodes | |
| 13:57:35 | sahid | but it depends on the application. for example for realtime you don't want them to be separated, same story for zero-drop packets that because of some cpu improvements L2/L3 cache | |
| 13:57:54 | sahid | s/you don"t/you want | |
| 13:58:10 | bauzas | efried: stephenfin: efried: but then, the filter will continue to act as before and only accept hosts that equally match the number of nodes asked by the guest | |
| 13:58:23 | stephenfin | sahid: Say I have a guest with a PCI device. If I boot the instance with two NUMA nodes and place both guest NUMA nodes on the same host NUMA node | |
| 13:58:29 | bauzas | it's just we won't ask the filter to check hosts that don't have *at least* | |
| 13:58:40 | efried | bauzas: ++ but it must not only accept *hosts* - it must filter down *candidates* for that host. | |
| 13:58:54 | stephenfin | sahid: Assuming that host NUMA node is the same one that the PCI device is placed on, what would be wrong with that? | |
| 13:58:56 | bauzas | good correction | |
| 13:59:12 | efried | bauzas: Because you could (and usually will) get multiple candidates for the same host, some of which have N and some of which have <N | |
| 13:59:25 | stephenfin | efried: I do wish someone would write a glossary of placement terminology | |
| 13:59:39 | sahid | stephenfin: i understand what you are saying but it again it depends of the application used | |
| 13:59:39 | efried | bauzas: And sometimes you'll *only* get candidates with <N, in which case your filter will have to exclude that host. | |
| 13:59:42 | bauzas | efried: either way, Placement returns candidates to scheduler, but scheduler passes a list of hosts to the filter, right? | |
| 13:59:55 | efried | stephenfin: Ah, by candidate I mean "allocation candidate". | |
| 14:00:01 | stephenfin | sahid: Say for the non-realtime case | |
| 14:00:05 | sahid | or we need an option to say enforce that rule | |
| 14:00:22 | efried | stephenfin: technically the "allocation request" part of the result of GET /allocation_candidates | |
| 14:00:38 | bauzas | efried: say for example Placement returns 2 child RPs, each of them being a NUMA node of host1, it will only pass host1 to the NUMATopoFilter | |
| 14:01:20 | sahid | stephenfin: for realtime use case you want best effort cpu to run on different numa node than the realtime cpus are | |
| 14:01:22 | efried | stephenfin: There will be X candidates coming back from a GET /a_c request, but the total number of providers ("hosts") represented by those candidates can be <=X | |
| 14:01:46 | bauzas | efried: that's not a problem then, right? | |