| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 13:37:46 | sahid | yes | |
| 13:38:08 | mdbooth | The multiattach job seems to be unhappy today. Did I miss any traffic about that? | |
| 13:38:10 | efried | sahid: Yeah, bauzas pointed to the flavor docs earlier which strongly implied (to my understanding) that when I say 4, I mean "at most 4" | |
| 13:38:23 | efried | mdbooth: I was waiting for mriedem to arrive so I could pester him about that. | |
| 13:38:28 | stephenfin | sahid: If I have a guest with a PCI device and hw:numa_nodes=2, then only one guest NUMA node would be placed on the same node as the PCI device | |
| 13:38:33 | stephenfin | *same host nodes as | |
| 13:38:48 | stephenfin | sahid: Come to think of it, this would improve performance | |
| 13:38:51 | stephenfin | *could | |
| 13:38:52 | efried | mriedem: Here's an example: http://logs.openstack.org/49/560349/1/check/nova-multiattach/ce31aed/logs/screen-n-cpu.txt.gz?level=ERROR | |
| 13:38:56 | mdbooth | efried: Ok. I won't double-pester him, then :) | |
| 13:39:20 | sahid | stephenfin: again we should not accept the host | |
| 13:39:32 | sahid | we sould never break the guest numa topology | |
| 13:39:33 | efried | mriedem: ...which looks kinda like something wrong in the CI env, not in the code. But I'm pretty ignorant in this realm. | |
| 13:39:33 | stephenfin | sahid: e.g. if we could fit the two guest NUMA nodes on the same host NUMA node as the PCI device, we'd be guaranteeing affinity for all guest CPUs | |
| 13:39:36 | mriedem | did you guys try doing a logstash query to find out when that started and if it's just that job? | |
| 13:39:39 | stephenfin | as opposed to half of them | |
| 13:39:48 | mriedem | my guess would be it's related to https://review.openstack.org/#/c/554314/ | |
| 13:40:01 | stephenfin | "stephenfin: again we should not accept the host" what do you mean? | |
| 13:40:06 | efried | mriedem: See, this is why we ask you. | |
| 13:40:09 | sahid | as i have indicated it's all about performance and distance between cpu, memory channel and pci devices | |
| 13:40:13 | mdbooth | efried: Hehe | |
| 13:40:23 | efried | mriedem: You came up with that shit off the top of your head; it would have taken me years to figure out. | |
| 13:40:30 | sahid | you are going to break this affinity | |
| 13:40:32 | mriedem | but i'll be gone some day | |
| 13:40:32 | stephenfin | sahid: I don't think you're understanding what I'm saying | |
| 13:40:42 | sahid | stephenfin: re-reading, sorry | |
| 13:41:00 | stephenfin | sahid: Let's table this for discussion on the spec. I think bauzas has his answer for now :) | |
| 13:41:04 | mdbooth | Was about to say that looks like a platform error | |
| 13:41:08 | stephenfin | sahid: good talk! :) | |
| 13:41:23 | mriedem | efried: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Connection%20to%20libvirt%20failed%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%3A%20libvirtError%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 13:41:35 | sahid | stephenfin: sure | |
| 13:41:37 | mriedem | it's definitely due to that change, since it's just that job | |
| 13:42:06 | efried | stephenfin: I don't know if bauzas has his answer. The question was, "if the user asks for 4 nodes, do we gotta get exactly 4 nodes from the host, or is it okay if it's anywhere from 1..4?" | |
| 13:42:21 | efried | stephenfin: Before sahid chimed in, we thought "1..4 is fine" was the answer. | |
| 13:42:32 | efried | stephenfin: Now we don't know anymore. | |
| 13:42:45 | efried | stephenfin: And bauzas definitely needs that answer to do his next spec edit. | |
| 13:42:46 | mriedem | efried: i'm hoping that https://review.openstack.org/#/c/554317/ will handle it | |
| 13:43:19 | stephenfin | efried: Hmm, can we discuss that on the spec so I can write something longer. I'm clearly not doing a good job of explaining here | |
| 13:43:21 | efried | mriedem: But that guy is failing the same way http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/screen-n-cpu.txt.gz?level=ERROR | |
| 13:43:49 | stephenfin | and/or I'm missing what sahid is trying to say | |
| 13:44:01 | stephenfin | efried: bauzas can just choose one and we'll weigh in there | |
| 13:44:10 | efried | fair | |
| 13:47:19 | sahid | i just saying that our current behavior to handle numa topology for guests is right we should not consider ok a guest requesting 2 NUMA nodes fit on one host NUMA node | |
| 13:47:32 | mriedem | efried: i think this is the problem http://logs.openstack.org/17/554317/3/check/nova-multiattach/fd35a93/logs/devstacklog.txt.gz#_2018-04-11_18_10_24_388 | |
| 13:48:58 | sahid | that because it's all performance related and user are going to place thier applications based on guest topology. if we provide a fake the performance will be degraded | |
| 13:50:12 | efried | sahid: Is it ever the case that performance is *improved* by forcing resources to come from *separate* NUMA nodes? | |
| 13:50:25 | efried | ohh, this is the parallelism thing. | |
| 13:51:02 | mriedem | nvm | |
| 13:51:03 | mdbooth | mriedem: If you get a sec could you remove the procedural -2 on https://review.openstack.org/#/c/526346/ ? | |
| 13:51:24 | mriedem | done | |
| 13:51:29 | mdbooth | mriedem: Thanks | |
| 13:52:47 | sahid | efried: it's just that, the vCPUs threads and memory will be placed on the same host numa node, so yes the performance will be ever improved | |
| 13:53:08 | sahid | it's the only reason why numa exist | |
| 13:53:29 | efried | sahid: I understand how performance is improved by placing on the *same* node. I'm asking if it's important to be able to explicitly *separate*. | |
| 13:54:47 | efried | sahid: Because in the case we're talking about, --property hw:numa_nodes=N will spread your resources across *at most* N NUMA nodes. It could end up being anywhere from 1..N. (But it will never be >N) | |
| 13:54:51 | sahid | efried: it depends of the use-case and application. you can request only one. the point is to have cpu/memory on same node | |
| 13:54:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make the nova-multiattach job non-voting temporarily https://review.openstack.org/560909 | |
| 13:54:53 | mriedem | efried: stephenfin: ^ | |
| 13:55:46 | efried | mriedem: +2 | |
| 13:56:16 | efried | sahid: So to my understanding, in this case, anything <N is a bonus. | |
| 13:56:16 | stephenfin | sahid: Right, but the CPU and memory will be on the same node. Again, we're not suggesting splitting an individual guest NUMA node | |
| 13:56:35 | efried | sahid: ...unless it's not. Which is what we're trying to ascertain. | |
| 13:56:51 | bauzas | efried: stephenfin: sahid: I think I have my answer tho | |
| 13:57:00 | efried | bauzas: Which is what? | |
| 13:57:34 | bauzas | efried: stephenfin: sahid: in my spec, I'll propose Placement to return all hosts accepting up to N NUMA nodes | |
| 13:57:35 | sahid | but it depends on the application. for example for realtime you don't want them to be separated, same story for zero-drop packets that because of some cpu improvements L2/L3 cache | |
| 13:57:54 | sahid | s/you don"t/you want | |
| 13:58:10 | bauzas | efried: stephenfin: efried: but then, the filter will continue to act as before and only accept hosts that equally match the number of nodes asked by the guest | |
| 13:58:23 | stephenfin | sahid: Say I have a guest with a PCI device. If I boot the instance with two NUMA nodes and place both guest NUMA nodes on the same host NUMA node | |
| 13:58:29 | bauzas | it's just we won't ask the filter to check hosts that don't have *at least* | |
| 13:58:40 | efried | bauzas: ++ but it must not only accept *hosts* - it must filter down *candidates* for that host. | |
| 13:58:54 | stephenfin | sahid: Assuming that host NUMA node is the same one that the PCI device is placed on, what would be wrong with that? | |
| 13:58:56 | bauzas | good correction | |
| 13:59:12 | efried | bauzas: Because you could (and usually will) get multiple candidates for the same host, some of which have N and some of which have <N | |
| 13:59:25 | stephenfin | efried: I do wish someone would write a glossary of placement terminology | |
| 13:59:39 | efried | bauzas: And sometimes you'll *only* get candidates with <N, in which case your filter will have to exclude that host. | |
| 13:59:39 | sahid | stephenfin: i understand what you are saying but it again it depends of the application used | |
| 13:59:42 | bauzas | efried: either way, Placement returns candidates to scheduler, but scheduler passes a list of hosts to the filter, right? | |
| 13:59:55 | efried | stephenfin: Ah, by candidate I mean "allocation candidate". | |
| 14:00:01 | stephenfin | sahid: Say for the non-realtime case | |
| 14:00:05 | sahid | or we need an option to say enforce that rule | |
| 14:00:22 | efried | stephenfin: technically the "allocation request" part of the result of GET /allocation_candidates | |
| 14:00:38 | bauzas | efried: say for example Placement returns 2 child RPs, each of them being a NUMA node of host1, it will only pass host1 to the NUMATopoFilter | |
| 14:01:20 | sahid | stephenfin: for realtime use case you want best effort cpu to run on different numa node than the realtime cpus are | |
| 14:01:22 | efried | stephenfin: There will be X candidates coming back from a GET /a_c request, but the total number of providers ("hosts") represented by those candidates can be <=X | |
| 14:01:46 | bauzas | efried: that's not a problem then, right? | |
| 14:02:07 | sahid | so bascially you are going to break realtime use case :) | |
| 14:02:16 | efried | sahid: No | |
| 14:02:32 | bauzas | efried: if host1 is having 2 nodes, and those 2 nodes are valid candidates for my query where resources1:VCPU=1&resources2:VCPU=1, I'll still pass to NUMATopoFilter only host1 | |
| 14:02:34 | efried | sahid: We're figuring out how to get N nodes when you ask for N nodes. | |
| 14:02:52 | sahid | efried: that seems reasonable | |
| 14:03:27 | jaypipes | efried, bauzas: #openstack-placement this conversation or mriedem is going to blow up. | |
| 14:03:49 | efried | jaypipes: We're talking about how nova is going to use placement | |
| 14:04:00 | stephenfin | efried: Oh, because the same host can be presented multiple ways? | |
| 14:04:08 | bauzas | jaypipes: the problem is that I'd love to see some NUMA experts chiming in | |
| 14:04:11 | efried | stephenfin: yes, exactly. | |
| 14:04:18 | stephenfin | efried: gotcha. Cheers | |
| 14:04:28 | bauzas | efried: again, I don't think there is a problem | |
| 14:04:41 | stephenfin | jaypipes: The damage should be limited to the west (?) coast. I'll be fine ;) | |
| 14:04:57 | bauzas | efried: the fact that placement is returning a list of candidates won't trample what the filter already does | |