| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-04 | |||
| 14:53:44 | dansmith | having compute nodes behave like one thing or another isn't something I'd like to see us doing long-term | |
| 14:53:49 | bauzas | we could also have an implicit switch | |
| 14:54:13 | bauzas | like, if you ask for mempages, then basically you ask for NUMA things | |
| 14:54:28 | bauzas | (even if, and I hate to say, is unrelated) | |
| 14:54:32 | bauzas | or cpu_pin | |
| 14:54:36 | efried | dansmith: the reason we didn't want to do that is because, if your priority is "land my VM", that becomes difficult/impossible as your cloud reaches saturation. | |
| 14:54:36 | bauzas | or whatever | |
| 14:54:50 | sean-k-mooney | bauzas: those are properties fo teh guest | |
| 14:54:54 | sean-k-mooney | bauzas: not of the host | |
| 14:54:54 | dansmith | efried: that is specifically called out in our project scope as not a problem nova solves | |
| 14:55:06 | stephenfin | ralonsoh: I can do 'GET /floatingips' (or whatever the API is). Each item in the response may contain a 'port_details' field but only if this extension is enabled. Correct? | |
| 14:55:07 | dansmith | efried: i.e. fitting the last VM into available memory | |
| 14:55:10 | bauzas | sean-k-mooney: for cpu pinset, it's for host | |
| 14:55:23 | bauzas | but I agree with large pages | |
| 14:55:46 | ralonsoh | stephenfin, exactly, the key 'port_details' will be there only if the extension is enabled | |
| 14:55:46 | bauzas | dammit, I'm torn | |
| 14:55:49 | sean-k-mooney | ya so we could enable it by default if you have defiend cpu_dedicated_set | |
| 14:56:05 | efried | bauzas: that's backwards though. If you ask the scheduler for large pages, you're asking to land on a host that knows how to do that. You can't use that question to decide to make a host large page-aware. | |
| 14:56:17 | sean-k-mooney | if cpu_dedicated_set is defiend then you are reporting PCPU and the only instance that can consume PCPUs have a numa toplogy | |
| 14:56:51 | sean-k-mooney | so that is once case where yes we can implcitly enable the numa reporting | |
| 14:57:36 | efried | dansmith: So how did we get to a point where we decided it was important to segregate the cloud? Hate to drag you into a second simultaneous discussion stephenfin, but weren't you part of that? | |
| 14:57:37 | stephenfin | ralonsoh: Okay. So if I do 'GET /ports/{port_id}', will that response always contain a 'device_id' field? If not, is this because it's provided by the same extension? | |
| 14:57:40 | sean-k-mooney | having hugepages allocated on the host is not really a good enough reason in my book and we should not assume that all numa hosts use cpu pinning | |
| 14:57:54 | bauzas | sean-k-mooney: right, that's my point | |
| 14:58:09 | bauzas | you could only care about large pages or just standard NUMA sharding | |
| 14:58:20 | dansmith | efried: segregate what? numa and non-numa instances? | |
| 14:58:31 | efried | yes | |
| 14:58:31 | sean-k-mooney | bauzas: well you could be useing the hugepages on the host for a dpdk vswitch for example | |
| 14:58:36 | bauzas | like, "I want 2 vCPUs on the same NUMA node" doesn't absolutely require CPU pinning | |
| 14:58:36 | sean-k-mooney | it might not be for the vms | |
| 14:58:38 | stephenfin | efried: Because we can't make everything have NUMA | |
| 14:58:50 | efried | stephenfin: right, so dansmith wants to know why not | |
| 14:58:50 | ralonsoh | stephenfin, exactly https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_common.py#L231 | |
| 14:58:56 | sean-k-mooney | bauzas: right that just need hw:numa_nodes=1 | |
| 14:59:02 | ralonsoh | stephenfin, the device_id will be there | |
| 14:59:20 | sean-k-mooney | bauzas: i use hugepages on my home system but not pinning | |
| 14:59:36 | stephenfin | efried: 2 sockets w/ a 32 core CPU in each socket (no HyperThreading). Go boot a 33 core instance | |
| 14:59:40 | bauzas | either way, I think we need to move forward | |
| 14:59:45 | sean-k-mooney | bauzas: beacue i want cpu oversubscition but not memory over subscrtion | |
| 15:00:08 | bauzas | I'll write the new revision with a bool flag and mention the alternative of an automatic all-NUMA world in the spec | |
| 15:00:09 | stephenfin | You can't because the instance will no longer split across the NUMA nodes, and we don't let an instance oversubscribe against itself | |
| 15:00:17 | bauzas | people will chime in and we'll see | |
| 15:00:24 | efried | dansmith: --^ | |
| 15:00:39 | dansmith | stephenfin: efried: because what nova currently provides is "no numa means 1 numa" yeah? | |
| 15:00:44 | sean-k-mooney | efried: yes we are aware that is the giant vm case. | |
| 15:00:50 | stephenfin | ralonsoh: It will *always* be there? | |
| 15:00:57 | stephenfin | dansmith: No, no NUMA means no NUMA | |
| 15:00:58 | ralonsoh | stephenfin, yes | |
| 15:00:59 | stephenfin | currently | |
| 15:01:18 | bauzas | dansmith: nova currently provides "I can spread my VM across many NUMA nodes if I don't care' | |
| 15:01:20 | dansmith | stephenfin: how is no numa and 1 numa node different to theguest/ | |
| 15:01:23 | ralonsoh | stephenfin, in the port this key, "device_id", is mandatory | |
| 15:01:33 | sean-k-mooney | dansmith: in the guest it does not | |
| 15:01:35 | dansmith | bauzas: right, we lie and say it's one node when it's not you mean? | |
| 15:01:38 | dansmith | sean-k-mooney: right exactly | |
| 15:01:54 | bauzas | dansmith: while that being broken if you turn into the new modeling | |
| 15:01:59 | sean-k-mooney | dansmith: but if you do hw:numa_nodes=1 we require that he guest virtual numa node be mapped to at most 1 host numa node | |
| 15:02:10 | sean-k-mooney | and that is where it breaks | |
| 15:02:10 | stephenfin | It's not for the guest, but we insist on 1 guest NUMA node being mapped to one host NUMA node | |
| 15:02:11 | bauzas | dansmith: it's a breaking change tbc | |
| 15:02:11 | dansmith | so my point is, for the giant vm case right now, we either require you to ignore or be insulated from the numaness that *does* exist, or configure it in a detailed fashion | |
| 15:02:25 | stephenfin | dansmith: correct | |
| 15:02:41 | bauzas | that's the proposal | |
| 15:02:50 | dansmith | that's all I'm saying.. nobody is opting *into* this behavior, they're choosing it because it's less bad than having to hand configure flavors | |
| 15:03:14 | dansmith | ...for their giant vms | |
| 15:03:19 | stephenfin | don't use anything that would configure NUMA (hw:numa_nodes, hw:cpu_policy=dedicated, hw:mem_page_size=foo) or hand tune for your chosen host topology | |
| 15:03:23 | stephenfin | dansmith: Yup, agreed | |
| 15:03:28 | dansmith | efried: ^ | |
| 15:03:42 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" for Ussuri https://review.opendev.org/693844 | |
| 15:03:47 | bauzas | the ideal would be us being able to serve an instance request *not* asking for NUMA with a NUMA-aware inventory | |
| 15:04:22 | dansmith | right | |
| 15:04:22 | bauzas | because if so, we would just make the switch for *all* hosts | |
| 15:04:28 | dansmith | and | |
| 15:04:38 | dansmith | we'd not be pretending with the topology | |
| 15:04:39 | stephenfin | ralonsoh: Thanks! So I'm going to rework this to check if the port details extension is enabled. If it is, use 'device_id' from that. If it is not, make a second request to '/ports/{port_id}' and extract 'device_id' from that | |
| 15:04:48 | stephenfin | ralonsoh: Does that sound reasonable? | |
| 15:05:00 | efried | Am I understanding that you can't boot a huge VM today either? | |
| 15:05:03 | stephenfin | ralonsoh: And assuming you're not planning to fix this yourself. If you've started, please continue :) | |
| 15:05:05 | bauzas | so, basically, the flag is just an expression of us being unable to serve such query :( | |
| 15:05:17 | sean-k-mooney | efried: you can but only if you use no numa feature | |
| 15:05:17 | bauzas | a technical limitation of our own | |
| 15:05:18 | ralonsoh | stephenfin, no, I didn't start doing it hehehe | |
| 15:05:28 | ralonsoh | stephenfin, but yes, this should be the procedure | |
| 15:05:32 | sean-k-mooney | or you use multiple guset numa nodes | |
| 15:05:34 | stephenfin | efried: You can but only if you don't configure anything with NUMA | |
| 15:05:42 | sean-k-mooney | got to join a call | |
| 15:05:46 | stephenfin | darn, ninja'd by sean-k-mooney | |
| 15:05:49 | efried | Feel like that came all the way back in a circle. | |
| 15:06:42 | stephenfin | efried: the original question was why can't we always report NUMA to placement, yeah? | |
| 15:07:45 | efried | But now we're considering removing that first possibility by forcing all hosts to be NUMA-aware. | |
| 15:07:45 | efried | And the proposal we started the morning with would allow you to do the same. | |
| 15:07:45 | efried | IOW to boot a huge instance today, you can artificially give it a multi-numa topo, or you can say nothing about NUMA. | |
| 15:07:57 | efried | sorry, removing the *second* possibility | |
| 15:08:31 | stephenfin | I don't think it's possible to force all hosts to be NUMA-aware | |
| 15:09:26 | bauzas | that's the crux of the problem | |
| 15:09:51 | efried | But dansmith has been arguing against that. | |
| 15:09:51 | efried | And you could only boot flavors with hw:numa*isms into the former; and you could only boot flavors *without* numa*isms into the latter. | |
| 15:09:51 | efried | stephenfin: I thought we had this all sussed out. I thought we were going to segregate the cloud into NUMA-aware (placement resources split along NUMA RPs) and non-NUMA-aware (what it looks like today, with all proc/mem on the root RP) hosts | |
| 15:09:54 | bauzas | are we able nowadays to express a placement query for a large VM with a NUMA-aware host ? | |
| 15:10:25 | bauzas | given the new proposal you made in the etherpad | |