| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 16:49:44 | efried | it won't | |
| 16:49:46 | sean-k-mooney | so it is still usful | |
| 16:49:51 | efried | it will do exactly nothing | |
| 16:49:59 | efried | except inject an extra placement call into every scheduling request. | |
| 16:50:08 | sean-k-mooney | it will because non numa instance will ignore numa nosts | |
| 16:50:17 | sean-k-mooney | and numa instance will ignore non numa nosts | |
| 16:50:18 | efried | there will be | |
| 16:50:18 | efried | no | |
| 16:50:18 | efried | numa | |
| 16:50:18 | efried | hosts. | |
| 16:50:31 | sean-k-mooney | people will turn this on | |
| 16:51:07 | sean-k-mooney | i can almost guarentee the then first release that we productise downstream with this feature will have it enabeld by default regardless of the upstream defaul | |
| 16:51:35 | efried | what, do your downstream releases force only flavors with guest numa topos? | |
| 16:51:51 | artom | sean-k-mooney, I don't see how we can do that... | |
| 16:52:07 | artom | Wouldn't that make all computes essentially not usable for non-NUMA instances? | |
| 16:52:12 | efried | ^ | |
| 16:52:16 | artom | If they're too big to fit on 1 NUMA node? | |
| 16:52:18 | gibi | efried: actually that what happens in my downstream project as we use pinning and huge page | |
| 16:52:21 | sean-k-mooney | no they dont but we do force the numa hosts to be partioned | |
| 16:52:49 | artom | So? | |
| 16:52:52 | efried | artom: even if they're not too big. Because we're adding the HW_NUMA_ROOT trait. | |
| 16:52:56 | sean-k-mooney | so all the hosts that will run numa workloads will have the feature turned on and the non numa host will have it off | |
| 16:52:59 | artom | They'd still all suddenly NUMA-exposing | |
| 16:53:19 | artom | sean-k-mooney, uh, so that's not "on by default as soon as they upgrade" :) | |
| 16:53:33 | artom | efried, oh right | |
| 16:54:06 | sean-k-mooney | it will be enabled in roles that configure the host for dpdk, hugepages or pinning | |
| 16:54:40 | sean-k-mooney | i would argue it should be set to on for all host and provide a way for them to opt out if they need the giant vm case | |
| 16:54:51 | sean-k-mooney | non of our telco customer need that case | |
| 16:55:17 | artom | sean-k-mooney, our telco customers are not 100% of openstack users | |
| 16:55:24 | artom | Would CERN want that, for example? :) | |
| 16:55:43 | sean-k-mooney | i dont think so no | |
| 16:55:56 | sean-k-mooney | i think they understand the perfomance cost and would use numa instances | |
| 16:56:18 | sean-k-mooney | cern are not going to trow away 30% of there compute performance | |
| 16:56:59 | sean-k-mooney | plus they have ironci if they really need to alocate all the resouce of a full host to an instance | |
| 16:57:06 | artom | sean-k-mooney, maybe, maybe not. I don't know how they, or anyone else how isn't a RH telco customer, operate. Which is why I'm weary of making this opt-out, and would feel safer making it opt-in. | |
| 16:57:20 | artom | If our deployment tooling wants to turn it on by default in some cases, that's cool | |
| 16:57:26 | artom | But not a Nova default | |
| 16:58:02 | sean-k-mooney | i think as proposed we get the best of both worlds | |
| 16:58:23 | sean-k-mooney | the fallback mechanisum will mean that it will just work if you dont set anything | |
| 16:58:49 | bauzas | sorry you lost my attention by not highlighting me | |
| 16:58:52 | sean-k-mooney | and we can add a nova status check to warn that you shoudl enable this before upgrading to Victora | |
| 16:59:15 | artom | sean-k-mooney, wait, the current proposed thing is disable_placement_numa_reporting = <bool> (default True for Ussuri) | |
| 16:59:24 | sean-k-mooney | yes | |
| 16:59:29 | artom | Which is what I'm advocating for | |
| 16:59:30 | sean-k-mooney | it will be disabled in ussuri | |
| 16:59:37 | artom | OK, so we agree :) | |
| 16:59:46 | sean-k-mooney | and hopefully enabled by default in Victoria | |
| 16:59:47 | dansmith | and that means what for flavors that currently have numa? | |
| 17:00:04 | sean-k-mooney | dansmith: the inital placement query will be empty | |
| 17:00:12 | sean-k-mooney | then we fall back to the current query | |
| 17:00:15 | artom | dansmith, IIUC same as what we did for PCPUs - try the new Placement query, if it comes back empty, try the legacy one | |
| 17:00:24 | sean-k-mooney | and leave it to the numa toplogy filter to do all the work | |
| 17:00:57 | sean-k-mooney | so they will just work | |
| 17:01:57 | dansmith | ack, yep, I think that's the sane path for U | |
| 17:02:27 | sean-k-mooney | stephenfin: by the way are you removing the fallback for PCPUs in U | |
| 17:02:47 | sean-k-mooney | stephenfin: that was the plan but i dont think you have had time to work on it | |
| 17:02:52 | stephenfin | I could but I was thinking I'd wait another cycle | |
| 17:02:52 | dansmith | if it's on by default, could we do the opposite for non-numa flavors? meaning, query with numa_nodes=1 and if we get no options, then try with =2, etc? | |
| 17:02:58 | dansmith | until we get a range capability with placement | |
| 17:03:23 | stephenfin | To let it bake in more. It's just dead code once the correct config options have been set | |
| 17:03:32 | sean-k-mooney | am maybe but the performace of that might not be great | |
| 17:03:39 | sean-k-mooney | or acccptable. | |
| 17:03:44 | artom | dansmith, that would assign a NUMA topology even if the user didn't request it (explicitly or implicitly), no? | |
| 17:04:00 | sean-k-mooney | artom: ya but that would actully be fine | |
| 17:04:10 | sean-k-mooney | they expressed no prefernce | |
| 17:04:10 | artom | NUMA topologies have a whole bunch of limitations :) | |
| 17:04:15 | sean-k-mooney | so we can decided what that is | |
| 17:04:32 | sean-k-mooney | artom: you removed the main one e.g. live migration | |
| 17:04:40 | artom | True... | |
| 17:04:49 | openstackgerrit | Douglas Mendizábal proposed openstack/nova master: Allow TLS ciphers/protocols to be configurable for console proxies https://review.opendev.org/679502 | |
| 17:04:49 | openstackgerrit | Douglas Mendizábal proposed openstack/nova master: Allow TLS ciphers/protocols to be configurable for console proxies https://review.opendev.org/679502 | |
| 17:05:04 | dansmith | artom: yes, it would, but I definitely think that we should be getting to the place where we don't just pretend numa doesn't exist | |
| 17:05:08 | artom | I'd still feel uncomfortable springing that on people | |
| 17:05:23 | dansmith | artom: so we want people to not have to worry about it in detail, but not just let them totally ignore it, IMHO | |
| 17:06:09 | sean-k-mooney | i think that is something we shoudl consider in V when we consider changing the default | |
| 17:06:26 | dansmith | yup | |
| 17:06:35 | sean-k-mooney | if there is a sensable way to express it to placmenet or a sensible way to make it work form our side we should explore it | |
| 17:06:43 | dansmith | what I don't think we should do, is continue to have two distinct ways of running instances forever | |
| 17:07:10 | sean-k-mooney | i agree with that | |
| 17:07:31 | artom | Same here | |
| 17:07:49 | artom | Though I'm not entirely convinced giving everyone a NUMA topology is the way to do it | |
| 17:07:57 | dansmith | similar to cellsv1, that didn't work out well, and we never closed the feature and bug gap for people using it until cellsv2 where we just make everyone use it.. it's a little more overhead, but the gap is way smaller | |
| 17:08:17 | dansmith | artom: but everyone _has_ a numa topology | |
| 17:08:34 | artom | dansmith, you know what I mean ;) | |
| 17:08:43 | dansmith | if we need to retool the numa stuff in nova (and probably placement) then that's what we need to do | |
| 17:08:58 | artom | I think I'd lean towards the can_split stuff in placement | |
| 17:09:20 | artom | Though I admittedly have no idea how complicated that would be | |
| 17:09:24 | dansmith | yep, it seems like the major barrier here is lack of expressivity with what we ask of placement when we don't care as much | |
| 17:09:42 | artom | Like, if the host is NUMA, fine, expose it | |
| 17:10:06 | artom | But as you said, being able to say "I don't care about NUMA" would be the best way to solve this, I think | |
| 17:10:15 | sean-k-mooney | artom: can split is very ineefincet to implement in sql | |
| 17:10:36 | artom | So, I have beef with "placement has to be SQL", but that's just me ;) | |
| 17:10:55 | sean-k-mooney | artom: if "i dont care about numa" means we are free to invent numa toploigies then sure | |
| 17:11:13 | artom | sean-k-mooney, weeelll... wouldn't that lead to a bunch of packing problems? | |
| 17:11:17 | bauzas | folks, I have to disappear | |
| 17:11:28 | bauzas | leave comments, disagreements, concers on the spec | |
| 17:11:29 | sean-k-mooney | artom: oh it doesnt have to be sql. but it would be very hard to support can split with the current impeneation | |
| 17:11:41 | sean-k-mooney | artom: no | |
| 17:11:46 | artom | I guess not, if you retry with enough combinations of guest NUMA nodes and CPUs per node | |
| 17:12:48 | artom | So if you some how end up with NUMA0 with 1 CPU free, and NUMA1 with 3 CPUs free, and boot an instance with 4 CPUs, you would need to retry until you get to the numa0_cpus:1,numa2_cpus=3 combo | |