| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 16:38:43 | sean-k-mooney | yes but without using excption for control flow :P | |
| 16:38:52 | efried | Sorry, let me clarify | |
| 16:39:03 | bauzas | efried: sean-k-mooney: I could have provided a link to the implementation instead of the spec :) | |
| 16:39:04 | efried | I understand the what/how. I don't understand the why. | |
| 16:39:14 | bauzas | efried: because, | |
| 16:39:24 | bauzas | say a rolling upgrade | |
| 16:39:27 | sean-k-mooney | efried: so we dont need to have a global flag in the schduler to turn on the translation | |
| 16:40:04 | bauzas | or just a Ussuri cloud with only one node being transformed | |
| 16:40:10 | bauzas | then we could have some problems | |
| 16:40:12 | sean-k-mooney | the same reason we did it for PCPUs to make upgrdade simpler | |
| 16:40:41 | bauzas | efried: have you seen the Upgrade Impact section already ? | |
| 16:40:47 | sean-k-mooney | bauzas: ya the single node case is basicaly during a rolling upgrade | |
| 16:40:50 | bauzas | I tried to explain the *why* there | |
| 16:41:30 | efried | Because what's going to happen here is: | |
| 16:41:30 | efried | I'll upgrade to Ussuri, where my workaround option is set to not reshape; | |
| 16:41:30 | efried | and all my flavors will continue to work, only a teeny bit slower because I'll get empty allocation candidates on the first query 100% of the time. | |
| 16:41:30 | efried | And I won't notice, so I'll never flip the workaround off. | |
| 16:42:33 | efried | so, | |
| 16:42:33 | efried | Am I misunderstanding what release/combination will have this fallback mechanism in play? | |
| 16:43:18 | gibi | efried: I understand the need to push the operators to switch. But pushing them to do the switch right at the upgrade feels too much to me. It would like an ultimate. "When you upgrade to Ussuri you will loose all the NUMA aware capacity of your cloud, but you can get them back iff you do the reshape one every NUMA aware compute, but you should not do that all at once as that will overload placement." | |
| 16:43:47 | gibi | s/one/on/ | |
| 16:44:06 | dansmith | gibi: you think that's okay? | |
| 16:44:08 | efried | Well, I don't buy that it would overload placement. The reshape will be one query per compute host. | |
| 16:44:26 | sean-k-mooney | efried: i would hope we remove it when we remove the config option | |
| 16:44:32 | gibi | dansmith: I think it is not OK to loose all the NUMA aware capacity at upgrade | |
| 16:45:04 | dansmith | gibi: okay you were describing the problem, not the goal, is that right? | |
| 16:45:04 | sean-k-mooney | we could remove it when we change the default but i would hope it will not last more then 2 releases | |
| 16:45:37 | efried | Then we kind of have to position this as a "tech preview" or experimental change. | |
| 16:45:38 | gibi | dansmith: my goal is not to loose the capacity at upgrade. I dont like the the quoted part | |
| 16:45:44 | efried | It just seems pretty pointless to me. | |
| 16:45:48 | dansmith | gibi: ++ | |
| 16:46:03 | efried | because we're going to do a bunch of work to enable something that nobody is going to use. | |
| 16:46:24 | sean-k-mooney | well since we are not enableing it by default in ussuri it kind of will be | |
| 16:46:54 | sean-k-mooney | but in vitora i think we should enable numa reporting by default. i would be happy to do that in ussuri but that might be a bit agressive | |
| 16:47:33 | efried | I buy that we can't enable it by default as long as we can't fit non-NUMA-aware workloads onto NUMA-modeled hosts. | |
| 16:48:03 | sean-k-mooney | efried: well actully im not sure i agree with that but thats a sperate dicussion | |
| 16:48:19 | efried | but with this fallback mechanism as designed, we're giving operators NO reason to switch over. | |
| 16:48:55 | sean-k-mooney | efried: well with my downstream hat on i will be pushing to make numa on the default for our next lts | |
| 16:48:56 | efried | which means we might as well not bother with this incremental improvement. We might as well just wait until we've solved the fitting problem. | |
| 16:49:17 | sean-k-mooney | efried: we have said that for 4+ releases | |
| 16:49:38 | sean-k-mooney | this will imporve schdulign time and reduce the chance of racecs for numa instnaces | |
| 16:49:43 | efried | no | |
| 16:49:44 | efried | it won't | |
| 16:49:46 | sean-k-mooney | so it is still usful | |
| 16:49:51 | efried | it will do exactly nothing | |
| 16:49:59 | efried | except inject an extra placement call into every scheduling request. | |
| 16:50:08 | sean-k-mooney | it will because non numa instance will ignore numa nosts | |
| 16:50:17 | sean-k-mooney | and numa instance will ignore non numa nosts | |
| 16:50:18 | efried | there will be | |
| 16:50:18 | efried | no | |
| 16:50:18 | efried | numa | |
| 16:50:18 | efried | hosts. | |
| 16:50:31 | sean-k-mooney | people will turn this on | |
| 16:51:07 | sean-k-mooney | i can almost guarentee the then first release that we productise downstream with this feature will have it enabeld by default regardless of the upstream defaul | |
| 16:51:35 | efried | what, do your downstream releases force only flavors with guest numa topos? | |
| 16:51:51 | artom | sean-k-mooney, I don't see how we can do that... | |
| 16:52:07 | artom | Wouldn't that make all computes essentially not usable for non-NUMA instances? | |
| 16:52:12 | efried | ^ | |
| 16:52:16 | artom | If they're too big to fit on 1 NUMA node? | |
| 16:52:18 | gibi | efried: actually that what happens in my downstream project as we use pinning and huge page | |
| 16:52:21 | sean-k-mooney | no they dont but we do force the numa hosts to be partioned | |
| 16:52:49 | artom | So? | |
| 16:52:52 | efried | artom: even if they're not too big. Because we're adding the HW_NUMA_ROOT trait. | |
| 16:52:56 | sean-k-mooney | so all the hosts that will run numa workloads will have the feature turned on and the non numa host will have it off | |
| 16:52:59 | artom | They'd still all suddenly NUMA-exposing | |
| 16:53:19 | artom | sean-k-mooney, uh, so that's not "on by default as soon as they upgrade" :) | |
| 16:53:33 | artom | efried, oh right | |
| 16:54:06 | sean-k-mooney | it will be enabled in roles that configure the host for dpdk, hugepages or pinning | |
| 16:54:40 | sean-k-mooney | i would argue it should be set to on for all host and provide a way for them to opt out if they need the giant vm case | |
| 16:54:51 | sean-k-mooney | non of our telco customer need that case | |
| 16:55:17 | artom | sean-k-mooney, our telco customers are not 100% of openstack users | |
| 16:55:24 | artom | Would CERN want that, for example? :) | |
| 16:55:43 | sean-k-mooney | i dont think so no | |
| 16:55:56 | sean-k-mooney | i think they understand the perfomance cost and would use numa instances | |
| 16:56:18 | sean-k-mooney | cern are not going to trow away 30% of there compute performance | |
| 16:56:59 | sean-k-mooney | plus they have ironci if they really need to alocate all the resouce of a full host to an instance | |
| 16:57:06 | artom | sean-k-mooney, maybe, maybe not. I don't know how they, or anyone else how isn't a RH telco customer, operate. Which is why I'm weary of making this opt-out, and would feel safer making it opt-in. | |
| 16:57:20 | artom | If our deployment tooling wants to turn it on by default in some cases, that's cool | |
| 16:57:26 | artom | But not a Nova default | |
| 16:58:02 | sean-k-mooney | i think as proposed we get the best of both worlds | |
| 16:58:23 | sean-k-mooney | the fallback mechanisum will mean that it will just work if you dont set anything | |
| 16:58:49 | bauzas | sorry you lost my attention by not highlighting me | |
| 16:58:52 | sean-k-mooney | and we can add a nova status check to warn that you shoudl enable this before upgrading to Victora | |
| 16:59:15 | artom | sean-k-mooney, wait, the current proposed thing is disable_placement_numa_reporting = <bool> (default True for Ussuri) | |
| 16:59:24 | sean-k-mooney | yes | |
| 16:59:29 | artom | Which is what I'm advocating for | |
| 16:59:30 | sean-k-mooney | it will be disabled in ussuri | |
| 16:59:37 | artom | OK, so we agree :) | |
| 16:59:46 | sean-k-mooney | and hopefully enabled by default in Victoria | |
| 16:59:47 | dansmith | and that means what for flavors that currently have numa? | |
| 17:00:04 | sean-k-mooney | dansmith: the inital placement query will be empty | |
| 17:00:12 | sean-k-mooney | then we fall back to the current query | |
| 17:00:15 | artom | dansmith, IIUC same as what we did for PCPUs - try the new Placement query, if it comes back empty, try the legacy one | |
| 17:00:24 | sean-k-mooney | and leave it to the numa toplogy filter to do all the work | |
| 17:00:57 | sean-k-mooney | so they will just work | |
| 17:01:57 | dansmith | ack, yep, I think that's the sane path for U | |
| 17:02:27 | sean-k-mooney | stephenfin: by the way are you removing the fallback for PCPUs in U | |
| 17:02:47 | sean-k-mooney | stephenfin: that was the plan but i dont think you have had time to work on it | |
| 17:02:52 | stephenfin | I could but I was thinking I'd wait another cycle | |
| 17:02:52 | dansmith | if it's on by default, could we do the opposite for non-numa flavors? meaning, query with numa_nodes=1 and if we get no options, then try with =2, etc? | |