| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-04 | |||
| 14:39:43 | stephenfin | ralonsoh: ah, so previously we were making two requests to neutron via neutronclient: one to retrieve all floating IPs and one to retrieve all ports | |
| 14:39:43 | bauzas | that looks seamless indeed | |
| 14:39:57 | bauzas | but my only concern is that this conf opt is only for Ussuri | |
| 14:40:00 | sean-k-mooney | e.g. the i want to use openstack to run one giant vm per host to run anohter orcastrator on top usecase we all hate | |
| 14:40:11 | bauzas | and then we would turn into it either way in Victoria | |
| 14:40:17 | sean-k-mooney | bauzas: i wont be | |
| 14:40:20 | dansmith | efried: sean-k-mooney bauzas but they will have to reshape eventually, right? | |
| 14:40:22 | stephenfin | ralonsoh: and then we simply matched 'port_id' for each entry in the former to the corresponding port from the latter. I tried to make that cleverer by using 'port_details', but you're saying that's an extension that I can't rely on | |
| 14:40:29 | sean-k-mooney | i will need to be kept as long as we support non numa instnaces | |
| 14:40:30 | bauzas | dansmith is expressing my concern | |
| 14:40:39 | stephenfin | ralonsoh: I can fix that now if you haven't already, but how can I check if that extension is available? | |
| 14:40:41 | efried | dansmith: I'm not sure we ever need to force them to make a host NUMA-aware if they don't want to. | |
| 14:40:56 | ralonsoh | stephenfin, exactly, this is an extension and this key is not mandatory | |
| 14:41:12 | bauzas | efried: if so, we somehow need to make the path clear that we *will* remove filter things in Victoria either way | |
| 14:41:14 | sean-k-mooney | dansmith: they will have to reshape only if we decied that all instance will have a numa toplogy of 1 gust numa node by defualt | |
| 14:41:18 | efried | iow the segregated-on-NUMA-ness is a permanent characteristic of the data center | |
| 14:41:58 | dansmith | if they can stay off forever, then fine, but I imagine that leaves us with a pretty big variable for a long time | |
| 14:42:05 | bauzas | also, | |
| 14:42:23 | bauzas | what I'm not okay is keeping the old filter processing for NUMA placement for a while | |
| 14:42:39 | dansmith | anyway, my point was that 95% of people will not pay attention and flip that flag until they have to, so reshape-on-flag only buys you N cycles until you turn it on permanently | |
| 14:42:48 | ralonsoh | stephenfin, you can use the neutron-client | |
| 14:42:49 | sean-k-mooney | dansmith: ya so i have been advocating that we should make all instnaces numa instance for a long time. but it breaks the "run one giant vm" usecase that some people care about | |
| 14:43:03 | ralonsoh | stephenfin, you can retrieve the extension list | |
| 14:43:29 | ralonsoh | but in this case, do you need it? just checking if this key is in the FIP dict | |
| 14:43:55 | efried | Hm, dansmith won't "people who care about NUMA" (like "edge") flip it right away so they can take advantage? Otherwise NUMA-ness won't work at all. | |
| 14:43:58 | dansmith | sean-k-mooney: you mean just in how they specify the one big VM, not that we need to be able to provide numa-ignorant vms right? | |
| 14:43:58 | sean-k-mooney | dansmith: if we decided that if you want to do that you have to specify multiple numa nodes in the flavor then we could get rid of the flag | |
| 14:44:06 | bauzas | efried: that's my point | |
| 14:44:25 | sean-k-mooney | dansmith: ya they would jsut add hw:numa_nodes=2 or whatever to the giant flavor | |
| 14:44:29 | efried | Like, I thought we were making this dividing line here such that, if you ask for a NUMA topo, and you don't have any computes with that flag switched on, you just won't land. | |
| 14:44:32 | stephenfin | ralonsoh: I need to figure out what instance the floating IP is associated with so I can return 'instance_id' in the API response | |
| 14:44:34 | bauzas | efried: if we want them to flip the conf opt, we somehow need to stop supporting the existing by the filter | |
| 14:44:58 | efried | bauzas: but we don't *want* them to flip the conf opt... unless they *need* NUMA on that host. | |
| 14:44:59 | stephenfin | ralonsoh: which appears to be stored in the 'device_id' field of the floating IP's port | |
| 14:45:06 | dansmith | efried: if it's required for numa at all then don't those people have to switch at upgrade anyway? | |
| 14:45:20 | efried | dansmith: only if they want NUMA-aware VMs on that host. | |
| 14:45:26 | bauzas | efried: correct | |
| 14:45:36 | bauzas | efried: people who don't care a bit about NUMA wouldn't get reshapes | |
| 14:45:54 | sean-k-mooney | dansmith: as it stand we should not be mixing numa vms and non numa vms on the same host | |
| 14:45:59 | dansmith | efried: right so people that have numa guests right now, they do the upgrade and if they don't flip it, they're unable to boot new instances? | |
| 14:46:07 | bauzas | but people who care should opt-in now so that in Victoria we could remove the filter bits responsible for placing instances | |
| 14:46:10 | sean-k-mooney | that is becasue of how we track and affitize the numa vms memory | |
| 14:46:14 | efried | ah, I see what you're saying. Yes dansmith that's right. | |
| 14:46:22 | dansmith | sean-k-mooney: I understand | |
| 14:46:38 | dansmith | efried: right, so anyone else will not flip that flag until they have to | |
| 14:46:46 | efried | why is that a problem? | |
| 14:47:03 | sean-k-mooney | dansmith: yes we could maybe check if the host has numa instnace currenly an for enable it? | |
| 14:47:05 | dansmith | efried: as I said before, if we're going to support numa-aware and numa-ignorant forever, then maybe it's not | |
| 14:47:23 | dansmith | sean-k-mooney: seems a little scary | |
| 14:47:30 | sean-k-mooney | ya i agree | |
| 14:47:34 | efried | Right, that's what I thought we were going to do. If FooHost doesn't ever care about NUMA instances, it can just leave that flag off forever, and never have a flavor with hw:numa* in it, and go on happily forever. | |
| 14:47:48 | ralonsoh | stephenfin, yes, but I'm reviewing the code and if the extension is not enabled, this information won't be there, in the FIP | |
| 14:47:53 | sean-k-mooney | i dont really like basing the behavior off what the current vms are using | |
| 14:48:03 | efried | gibi, bauzas: To close on group_policy: sean-k-mooney and I talked about it a bit yesterday and decided that, in order to support both NUMA and bandwidth, we need to retain the group_policy=none default and do the anti-affinity in the NTF | |
| 14:48:03 | efried | UNTIL we can design & implement proper granular anti-affinity syntax in placement, which IMO is too ambitious for Ussuri. | |
| 14:48:09 | efried | sean-k-mooney: +1 | |
| 14:48:19 | sean-k-mooney | efried: keep in mind that almost all host are numa hosts | |
| 14:48:32 | dansmith | sean-k-mooney: in reality, ALL of them are | |
| 14:48:38 | efried | meaning almost all hosts are *capable* of NUMA. | |
| 14:48:45 | efried | Not that all hosts are hosting NUMA-aware instances. | |
| 14:48:49 | sean-k-mooney | dansmith: yes | |
| 14:48:59 | dansmith | not capable, they are numa and if not configured, are losing performance to that fact | |
| 14:49:00 | sean-k-mooney | efried: correct | |
| 14:49:12 | efried | right, which some VMs don't care about. | |
| 14:49:13 | stephenfin | ralonsoh: WDYM? As I understood it, we won't have the 'port_details' field but the 'device_id' field will still be there in the ports API response, right? | |
| 14:49:41 | sean-k-mooney | efried: no i think dansmit ment at the host level. not the vm level | |
| 14:49:41 | dansmith | efried: no, they all care about it, just some operators don't care to take on the burden of configuring it because we make it so difficult | |
| 14:49:48 | ralonsoh | stephenfin, no no | |
| 14:50:00 | dansmith | nobody is opting out of numa for any reason other than the cost benefit isn't there for configuring it | |
| 14:50:02 | ralonsoh | stephenfin, if you don't have 'port_details', you won't have this info | |
| 14:50:14 | stephenfin | ohhhh, they're tied together | |
| 14:50:21 | stephenfin | ? | |
| 14:50:23 | ralonsoh | stephenfin, if you really need the host but you don't have 'port_details' | |
| 14:50:33 | efried | dansmith: oh? Not because opting out lets them squeeze more VMs onto their host? | |
| 14:50:35 | ralonsoh | then you have the "port_id" always | |
| 14:50:55 | sean-k-mooney | efried: well it depends | |
| 14:50:57 | ralonsoh | stephenfin, and then, you can call the Neutron API to retrieve the port info (and the host) | |
| 14:51:04 | sean-k-mooney | in general no | |
| 14:51:08 | efried | yes, it depends, that's what I'm saying. | |
| 14:51:20 | sean-k-mooney | but if you use pinning or hugepage that disable oversubsription or cpus or memroy | |
| 14:51:23 | dansmith | efried: only because complexity of what we provide makes that increasingly difficult as density increases | |
| 14:51:36 | sean-k-mooney | if you jsut use numa over subsription is allowed | |
| 14:51:43 | efried | right, and that's not a fitting problem we're going to solve any time soon. | |
| 14:52:20 | bauzas | I feel we need to define a clear upgrade path | |
| 14:52:33 | stephenfin | ralonsoh: I'm confused. You seem to be saying the same thing as me :) Let me try again | |
| 14:52:35 | bauzas | 1/ for NUMA-aware instances | |
| 14:52:42 | bauzas | 2/ for non-NUMA-aware instances | |
| 14:52:46 | efried | 1/ flip the switch | |
| 14:52:46 | efried | 2/ don't | |
| 14:53:01 | efried | If we've already decided we're going to segregate, it really is that simple. | |
| 14:53:08 | sean-k-mooney | so the boolean config option allows us to punt on the final decission on this for a cycle or two. | |
| 14:53:09 | dansmith | is there some reason that the switch can't be flipped for everyone? | |
| 14:53:23 | sean-k-mooney | e.g. default to on | |
| 14:53:27 | sean-k-mooney | and you opt out? | |
| 14:53:44 | dansmith | having compute nodes behave like one thing or another isn't something I'd like to see us doing long-term | |
| 14:53:49 | bauzas | we could also have an implicit switch | |
| 14:54:13 | bauzas | like, if you ask for mempages, then basically you ask for NUMA things | |
| 14:54:28 | bauzas | (even if, and I hate to say, is unrelated) | |
| 14:54:32 | bauzas | or cpu_pin | |
| 14:54:36 | bauzas | or whatever | |