| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 16:44:26 | sean-k-mooney | efried: i would hope we remove it when we remove the config option | |
| 16:44:32 | gibi | dansmith: I think it is not OK to loose all the NUMA aware capacity at upgrade | |
| 16:45:04 | sean-k-mooney | we could remove it when we change the default but i would hope it will not last more then 2 releases | |
| 16:45:04 | dansmith | gibi: okay you were describing the problem, not the goal, is that right? | |
| 16:45:37 | efried | Then we kind of have to position this as a "tech preview" or experimental change. | |
| 16:45:38 | gibi | dansmith: my goal is not to loose the capacity at upgrade. I dont like the the quoted part | |
| 16:45:44 | efried | It just seems pretty pointless to me. | |
| 16:45:48 | dansmith | gibi: ++ | |
| 16:46:03 | efried | because we're going to do a bunch of work to enable something that nobody is going to use. | |
| 16:46:24 | sean-k-mooney | well since we are not enableing it by default in ussuri it kind of will be | |
| 16:46:54 | sean-k-mooney | but in vitora i think we should enable numa reporting by default. i would be happy to do that in ussuri but that might be a bit agressive | |
| 16:47:33 | efried | I buy that we can't enable it by default as long as we can't fit non-NUMA-aware workloads onto NUMA-modeled hosts. | |
| 16:48:03 | sean-k-mooney | efried: well actully im not sure i agree with that but thats a sperate dicussion | |
| 16:48:19 | efried | but with this fallback mechanism as designed, we're giving operators NO reason to switch over. | |
| 16:48:55 | sean-k-mooney | efried: well with my downstream hat on i will be pushing to make numa on the default for our next lts | |
| 16:48:56 | efried | which means we might as well not bother with this incremental improvement. We might as well just wait until we've solved the fitting problem. | |
| 16:49:17 | sean-k-mooney | efried: we have said that for 4+ releases | |
| 16:49:38 | sean-k-mooney | this will imporve schdulign time and reduce the chance of racecs for numa instnaces | |
| 16:49:43 | efried | no | |
| 16:49:44 | efried | it won't | |
| 16:49:46 | sean-k-mooney | so it is still usful | |
| 16:49:51 | efried | it will do exactly nothing | |
| 16:49:59 | efried | except inject an extra placement call into every scheduling request. | |
| 16:50:08 | sean-k-mooney | it will because non numa instance will ignore numa nosts | |
| 16:50:17 | sean-k-mooney | and numa instance will ignore non numa nosts | |
| 16:50:18 | efried | hosts. | |
| 16:50:18 | efried | numa | |
| 16:50:18 | efried | no | |
| 16:50:18 | efried | there will be | |
| 16:50:31 | sean-k-mooney | people will turn this on | |
| 16:51:07 | sean-k-mooney | i can almost guarentee the then first release that we productise downstream with this feature will have it enabeld by default regardless of the upstream defaul | |
| 16:51:35 | efried | what, do your downstream releases force only flavors with guest numa topos? | |
| 16:51:51 | artom | sean-k-mooney, I don't see how we can do that... | |
| 16:52:07 | artom | Wouldn't that make all computes essentially not usable for non-NUMA instances? | |
| 16:52:12 | efried | ^ | |
| 16:52:16 | artom | If they're too big to fit on 1 NUMA node? | |
| 16:52:18 | gibi | efried: actually that what happens in my downstream project as we use pinning and huge page | |
| 16:52:21 | sean-k-mooney | no they dont but we do force the numa hosts to be partioned | |
| 16:52:49 | artom | So? | |
| 16:52:52 | efried | artom: even if they're not too big. Because we're adding the HW_NUMA_ROOT trait. | |
| 16:52:56 | sean-k-mooney | so all the hosts that will run numa workloads will have the feature turned on and the non numa host will have it off | |
| 16:52:59 | artom | They'd still all suddenly NUMA-exposing | |
| 16:53:19 | artom | sean-k-mooney, uh, so that's not "on by default as soon as they upgrade" :) | |
| 16:53:33 | artom | efried, oh right | |
| 16:54:06 | sean-k-mooney | it will be enabled in roles that configure the host for dpdk, hugepages or pinning | |
| 16:54:40 | sean-k-mooney | i would argue it should be set to on for all host and provide a way for them to opt out if they need the giant vm case | |
| 16:54:51 | sean-k-mooney | non of our telco customer need that case | |
| 16:55:17 | artom | sean-k-mooney, our telco customers are not 100% of openstack users | |
| 16:55:24 | artom | Would CERN want that, for example? :) | |
| 16:55:43 | sean-k-mooney | i dont think so no | |
| 16:55:56 | sean-k-mooney | i think they understand the perfomance cost and would use numa instances | |
| 16:56:18 | sean-k-mooney | cern are not going to trow away 30% of there compute performance | |
| 16:56:59 | sean-k-mooney | plus they have ironci if they really need to alocate all the resouce of a full host to an instance | |
| 16:57:06 | artom | sean-k-mooney, maybe, maybe not. I don't know how they, or anyone else how isn't a RH telco customer, operate. Which is why I'm weary of making this opt-out, and would feel safer making it opt-in. | |
| 16:57:20 | artom | If our deployment tooling wants to turn it on by default in some cases, that's cool | |
| 16:57:26 | artom | But not a Nova default | |
| 16:58:02 | sean-k-mooney | i think as proposed we get the best of both worlds | |
| 16:58:23 | sean-k-mooney | the fallback mechanisum will mean that it will just work if you dont set anything | |
| 16:58:49 | bauzas | sorry you lost my attention by not highlighting me | |
| 16:58:52 | sean-k-mooney | and we can add a nova status check to warn that you shoudl enable this before upgrading to Victora | |
| 16:59:15 | artom | sean-k-mooney, wait, the current proposed thing is disable_placement_numa_reporting = <bool> (default True for Ussuri) | |
| 16:59:24 | sean-k-mooney | yes | |
| 16:59:29 | artom | Which is what I'm advocating for | |
| 16:59:30 | sean-k-mooney | it will be disabled in ussuri | |
| 16:59:37 | artom | OK, so we agree :) | |
| 16:59:46 | sean-k-mooney | and hopefully enabled by default in Victoria | |
| 16:59:47 | dansmith | and that means what for flavors that currently have numa? | |
| 17:00:04 | sean-k-mooney | dansmith: the inital placement query will be empty | |
| 17:00:12 | sean-k-mooney | then we fall back to the current query | |
| 17:00:15 | artom | dansmith, IIUC same as what we did for PCPUs - try the new Placement query, if it comes back empty, try the legacy one | |
| 17:00:24 | sean-k-mooney | and leave it to the numa toplogy filter to do all the work | |
| 17:00:57 | sean-k-mooney | so they will just work | |
| 17:01:57 | dansmith | ack, yep, I think that's the sane path for U | |
| 17:02:27 | sean-k-mooney | stephenfin: by the way are you removing the fallback for PCPUs in U | |
| 17:02:47 | sean-k-mooney | stephenfin: that was the plan but i dont think you have had time to work on it | |
| 17:02:52 | dansmith | if it's on by default, could we do the opposite for non-numa flavors? meaning, query with numa_nodes=1 and if we get no options, then try with =2, etc? | |
| 17:02:52 | stephenfin | I could but I was thinking I'd wait another cycle | |
| 17:02:58 | dansmith | until we get a range capability with placement | |
| 17:03:23 | stephenfin | To let it bake in more. It's just dead code once the correct config options have been set | |
| 17:03:32 | sean-k-mooney | am maybe but the performace of that might not be great | |
| 17:03:39 | sean-k-mooney | or acccptable. | |
| 17:03:44 | artom | dansmith, that would assign a NUMA topology even if the user didn't request it (explicitly or implicitly), no? | |
| 17:04:00 | sean-k-mooney | artom: ya but that would actully be fine | |
| 17:04:10 | artom | NUMA topologies have a whole bunch of limitations :) | |
| 17:04:10 | sean-k-mooney | they expressed no prefernce | |
| 17:04:15 | sean-k-mooney | so we can decided what that is | |
| 17:04:32 | sean-k-mooney | artom: you removed the main one e.g. live migration | |
| 17:04:40 | artom | True... | |
| 17:04:49 | openstackgerrit | Douglas Mendizábal proposed openstack/nova master: Allow TLS ciphers/protocols to be configurable for console proxies https://review.opendev.org/679502 | |
| 17:04:49 | openstackgerrit | Douglas Mendizábal proposed openstack/nova master: Allow TLS ciphers/protocols to be configurable for console proxies https://review.opendev.org/679502 | |
| 17:05:04 | dansmith | artom: yes, it would, but I definitely think that we should be getting to the place where we don't just pretend numa doesn't exist | |
| 17:05:08 | artom | I'd still feel uncomfortable springing that on people | |
| 17:05:23 | dansmith | artom: so we want people to not have to worry about it in detail, but not just let them totally ignore it, IMHO | |
| 17:06:09 | sean-k-mooney | i think that is something we shoudl consider in V when we consider changing the default | |
| 17:06:26 | dansmith | yup | |
| 17:06:35 | sean-k-mooney | if there is a sensable way to express it to placmenet or a sensible way to make it work form our side we should explore it | |
| 17:06:43 | dansmith | what I don't think we should do, is continue to have two distinct ways of running instances forever | |
| 17:07:10 | sean-k-mooney | i agree with that | |
| 17:07:31 | artom | Same here | |
| 17:07:49 | artom | Though I'm not entirely convinced giving everyone a NUMA topology is the way to do it | |