| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-12 | |||
| 13:30:00 | bauzas | sahid: he's just booting a flavor | |
| 13:30:17 | bauzas | but what he knows is that he'll get a guest having its own topology | |
| 13:30:20 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename recreate to evacuate in driver signatures https://review.openstack.org/560900 | |
| 13:30:20 | stephenfin | Yeah, what bauzas said | |
| 13:30:33 | sahid | yes and if he wants 2 numa where don"t want to fake that | |
| 13:30:39 | openstackgerrit | Eric Fried proposed openstack/os-traits master: normalize_name helper https://review.openstack.org/560107 | |
| 13:30:43 | stephenfin | He wants two _guest_ NUMA nodes | |
| 13:30:50 | sahid | yes and if he wants 2 numa nodes for it's guest we don"t want to fake that | |
| 13:31:00 | stephenfin | The entire thing is fake | |
| 13:31:14 | bauzas | sahid: so some evil guy could guess there are no left hosts having 2 numa nodes | |
| 13:31:20 | stephenfin | There aren't actually two guest NUMA nodes. That's just QEMU/KVM mocking it | |
| 13:32:07 | sahid | stephenfin: it's not... the guest memory and vcpus threads are running on a uniq node | |
| 13:32:35 | stephenfin | sahid: I get that. I'm not suggesting splitting a guest's NUMA node across a host NUMA node | |
| 13:32:56 | stephenfin | What I'm saying is there's not reason we shouldn't fit N guest NUMA nodes on the same host NUMA node | |
| 13:32:57 | efried | but the other way around is okay | |
| 13:33:09 | stephenfin | efried: Yeah, precisely | |
| 13:33:15 | sahid | stephenfin: it's not what we do | |
| 13:33:29 | bauzas | so the big thing is | |
| 13:33:33 | bauzas | now we know that | |
| 13:33:38 | bauzas | should Placement do that ? | |
| 13:33:41 | bauzas | I don't think so | |
| 13:33:45 | stephenfin | sahid: Indeed, and I think we can change that | |
| 13:33:47 | bauzas | leave that logic to the filter | |
| 13:33:53 | stephenfin | bauzas: No, neither do I | |
| 13:34:01 | bauzas | what Placement would return is just a list of potential targets | |
| 13:34:12 | stephenfin | So long as we can still ensure at least one guest NUMA node is affined with a given host NUMA node | |
| 13:34:17 | bauzas | some can run that workload but on a single node, some can do that better | |
| 13:34:19 | stephenfin | For things like PCI devices and the likes | |
| 13:34:26 | bauzas | so the filter will pick the best one | |
| 13:34:28 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: RP list: member_of and resources parameters (v1.3, v1.4) https://review.openstack.org/511183 | |
| 13:34:29 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: RP delete inventories (v1.5) https://review.openstack.org/514642 | |
| 13:34:30 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643 | |
| 13:34:31 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644 | |
| 13:34:32 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | |
| 13:34:33 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | |
| 13:34:33 | stephenfin | And vSwitches :) | |
| 13:34:34 | Shilpa | stephenfin: Hi, noVNC 1.0.0, i have checked logs http://logs.openstack.org/72/550172/2/check/tempest-full/f6945b6/job-output.txt.gz, and observed that 1 test case is failed, and that is tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc[id-c640fdff-8ab4-45a4-a5d8-7e6146cbd0dc] | |
| 13:34:34 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: New dict format of allocations (v1.11, v1.12) https://review.openstack.org/542819 | |
| 13:34:35 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Transactionally update allocations (v1.13) https://review.openstack.org/546674 | |
| 13:34:36 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Add nested resource providers (v1.14) https://review.openstack.org/546675 | |
| 13:34:37 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Limit allocation candidates (v1.15, v1.16) https://review.openstack.org/548043 | |
| 13:34:38 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Allocation candidates parameter: required (v1.17) https://review.openstack.org/548326 | |
| 13:34:44 | bauzas | oh shit, gerrit dropbomb | |
| 13:34:47 | efried | Placement will give you candidates telling you exactly which NUMA nodes the *actual* resources come from. The virt driver must adhere to that. What it turns around and tells the guest is entirely up to your conscience :) | |
| 13:34:56 | sahid | stephenfin: what do you want to change? | |
| 13:35:17 | Shilpa | stephengin: checked there, URL is correct though 'http://217.182.140.77:6080/vnc.html?path=websockify?token=30848df8-99b2-4a0c-a615-51a5a444660d' | |
| 13:35:21 | bauzas | efried: Placement will give hosts that match the possibility to support a guest having a specific topology | |
| 13:35:30 | stephenfin | Shilpa: Indeed. I thought that was because of the change in the path but your change apparently works with both 1.0 and pre-1.0 | |
| 13:35:36 | stephenfin | Shilpa: So I don't know what's going on | |
| 13:35:45 | efried | bauzas: Maybe. | |
| 13:35:46 | bauzas | efried: but the filter could restrict that list | |
| 13:35:50 | efried | bauzas: Exactly. | |
| 13:36:05 | efried | The filter is allowed to be stricter than placement. That's... kind of its job. | |
| 13:36:06 | bauzas | efried: okay, so I feel numbered request groups can still work | |
| 13:36:10 | stephenfin | sahid: If a user requests N nodes, then nova should place those on host NUMA nodes as necessary | |
| 13:36:12 | efried | yes, agree completely. | |
| 13:36:17 | bauzas | provided I'm clear on my spec | |
| 13:36:37 | stephenfin | sahid: So for 4 guest nodes, if it can fit 3 on one host node and 1 on the other, that would be OK | |
| 13:36:42 | bauzas | that, whatever logic I dislike on filter, it's not intended to be in Placement | |
| 13:36:56 | sahid | stephenfin: please don't do that | |
| 13:36:58 | efried | bauzas: Would even work if we wanted to support explicit sharding/anti-affinity - like I said earlier, the filter would be responsible for getting rid of the candidates where placement "incorrectly" clustered separate groups into the same RP. | |
| 13:37:03 | Shilpa | stephenfin: ok will check further, if i will get something, will let you know, thanks | |
| 13:37:04 | bauzas | ++ | |
| 13:37:06 | bauzas | efried: ++ | |
| 13:37:20 | efried | sahid: Please clarify. | |
| 13:37:21 | stephenfin | sahid: But we're not breaking the NUMA affinity with PCI devices or the likes. At least one of those guest nodes would have to be placed on the same host NUMA node as the PCI device | |
| 13:37:21 | sahid | you are going to break the whole aim of numa toplogy | |
| 13:37:39 | efried | sahid: You're saying that if we request 4 (host) nodes, we must get 4 (host) nodes? | |
| 13:37:45 | stephenfin | sahid: This is no different to what we do now | |
| 13:37:46 | sahid | yes | |
| 13:38:08 | mdbooth | The multiattach job seems to be unhappy today. Did I miss any traffic about that? | |
| 13:38:10 | efried | sahid: Yeah, bauzas pointed to the flavor docs earlier which strongly implied (to my understanding) that when I say 4, I mean "at most 4" | |
| 13:38:23 | efried | mdbooth: I was waiting for mriedem to arrive so I could pester him about that. | |
| 13:38:28 | stephenfin | sahid: If I have a guest with a PCI device and hw:numa_nodes=2, then only one guest NUMA node would be placed on the same node as the PCI device | |
| 13:38:33 | stephenfin | *same host nodes as | |
| 13:38:48 | stephenfin | sahid: Come to think of it, this would improve performance | |
| 13:38:51 | stephenfin | *could | |
| 13:38:52 | efried | mriedem: Here's an example: http://logs.openstack.org/49/560349/1/check/nova-multiattach/ce31aed/logs/screen-n-cpu.txt.gz?level=ERROR | |
| 13:38:56 | mdbooth | efried: Ok. I won't double-pester him, then :) | |
| 13:39:20 | sahid | stephenfin: again we should not accept the host | |
| 13:39:32 | sahid | we sould never break the guest numa topology | |
| 13:39:33 | efried | mriedem: ...which looks kinda like something wrong in the CI env, not in the code. But I'm pretty ignorant in this realm. | |
| 13:39:33 | stephenfin | sahid: e.g. if we could fit the two guest NUMA nodes on the same host NUMA node as the PCI device, we'd be guaranteeing affinity for all guest CPUs | |
| 13:39:36 | mriedem | did you guys try doing a logstash query to find out when that started and if it's just that job? | |
| 13:39:39 | stephenfin | as opposed to half of them | |
| 13:39:48 | mriedem | my guess would be it's related to https://review.openstack.org/#/c/554314/ | |
| 13:40:01 | stephenfin | "stephenfin: again we should not accept the host" what do you mean? | |
| 13:40:06 | efried | mriedem: See, this is why we ask you. | |
| 13:40:09 | sahid | as i have indicated it's all about performance and distance between cpu, memory channel and pci devices | |
| 13:40:13 | mdbooth | efried: Hehe | |
| 13:40:23 | efried | mriedem: You came up with that shit off the top of your head; it would have taken me years to figure out. | |
| 13:40:30 | sahid | you are going to break this affinity | |
| 13:40:32 | mriedem | but i'll be gone some day | |
| 13:40:32 | stephenfin | sahid: I don't think you're understanding what I'm saying | |
| 13:40:42 | sahid | stephenfin: re-reading, sorry | |
| 13:41:00 | stephenfin | sahid: Let's table this for discussion on the spec. I think bauzas has his answer for now :) | |
| 13:41:04 | mdbooth | Was about to say that looks like a platform error | |
| 13:41:08 | stephenfin | sahid: good talk! :) | |
| 13:41:23 | mriedem | efried: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Connection%20to%20libvirt%20failed%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%3A%20libvirtError%3A%20Failed%20to%20connect%20socket%20to%20'%2Fvar%2Frun%2Flibvirt%2Flibvirt-sock'%3A%20Permission%20denied%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 13:41:35 | sahid | stephenfin: sure | |