| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-11 | |||
| 19:47:34 | sean-k-mooney | it was a gloabl config untill we agreed to do the fallback at the end of train | |
| 19:48:01 | efried | dansmith: isn't that going to end up violating pack/spread weighing and server affinity groups, because it will always favor upgraded/reshaped computes? | |
| 19:48:38 | dansmith | yes? | |
| 19:49:01 | efried | I mean, I see your point, that you really can't upgrade a compute *and* reshape it unless the scheduler is going to do some translating. | |
| 19:49:04 | dansmith | or, do both queries, merge the two results and let the filters/weighers decide? | |
| 19:49:11 | efried | gross. But yeah. | |
| 19:49:33 | dansmith | doesn't seem more gross to me.. two queries, yes, but everything will start with two queries effectively | |
| 19:50:10 | sean-k-mooney | dansmith: do we merge them for PCPUs or is there a reason we dont? | |
| 19:50:17 | dansmith | idk | |
| 19:50:19 | efried | IIRC we don't | |
| 19:50:31 | efried | we do one, then if no results, we do the other | |
| 19:50:47 | sean-k-mooney | right which by the way we still do | |
| 19:51:05 | efried | as for a reason, probably just didn't think about the weighing etc. | |
| 19:51:25 | sean-k-mooney | no i remebere this coming up | |
| 19:51:42 | sean-k-mooney | i just dont recally why we chose to/not to merge them | |
| 19:52:35 | sean-k-mooney | so is the poposal always do both queries and merge them or to the progressive spliting | |
| 19:53:17 | efried | Both | |
| 19:53:22 | efried | because | |
| 19:53:40 | efried | we want upgraded hosts to cut over to NUMA-modeled by default. | |
| 19:53:52 | sean-k-mooney | ok so progressive for non numa and both qureies for numa | |
| 19:54:17 | efried | I think we need two queries for both. | |
| 19:54:41 | sean-k-mooney | i think it will be more like 2 for numa and 5 for non numa | |
| 19:54:46 | efried | yeah | |
| 19:55:03 | sean-k-mooney | 5 asumign we dint max_numa_nodes=5 | |
| 19:55:08 | efried | 4 | |
| 19:55:15 | sean-k-mooney | sorry 4 | |
| 19:55:25 | efried | I asked this earlier: what's the max number of NUMA nodes we know of on any system? Is it 4? | |
| 19:55:28 | sean-k-mooney | could we bail out if the first numa query passed | |
| 19:55:34 | efried | yes | |
| 19:56:35 | sean-k-mooney | on a 32-64 core 2 socket amd eypci host with a numa node per l3 region 16-32 numa node | |
| 19:56:56 | efried | ye gods | |
| 19:57:04 | sean-k-mooney | if you dont expose a numa node per l3 region then 8 is more realisitc | |
| 19:57:15 | efried | Can we tell from the db? | |
| 19:57:32 | efried | I guess we could tell by querying placement. | |
| 19:57:34 | sean-k-mooney | its in teh hohst numa toplogy blob in the cell db | |
| 19:57:53 | sean-k-mooney | so yes technically. | |
| 19:58:08 | efried | but I don't know that we really want to go discovering that every time we schedule. | |
| 19:58:24 | sean-k-mooney | every time defintly not | |
| 19:58:31 | sean-k-mooney | schduler config option? | |
| 19:58:33 | efried | configurable [scheduler]max_implicit_numa_nodes ? | |
| 19:58:36 | efried | yeah | |
| 19:58:51 | efried | because even if it's possible to have a 32-way split, that would almost never be a good idea probably. | |
| 19:58:57 | sean-k-mooney | set it to 0 to diable numa qureis entrily otherwise its the limit | |
| 19:59:17 | sean-k-mooney | efried: it give you almost a 35% performace boost | |
| 19:59:19 | efried | "disable" meaning what? | |
| 19:59:43 | sean-k-mooney | efried: disabel mean i have disable numa reporting in my entire cloud so dont even try | |
| 20:00:13 | efried | Need to grok how that query would be different from the non-upgraded query. | |
| 20:00:24 | sean-k-mooney | it would be the same | |
| 20:00:32 | sean-k-mooney | just want we do today in tran | |
| 20:00:33 | efried | Oh, because it would have the HW_NUMA_ROOT trait. | |
| 20:01:09 | sean-k-mooney | well require=!HW_NUMA_ROOT would be the detla form train i guess | |
| 20:01:22 | sean-k-mooney | but if no host reports numa thats a noop | |
| 20:01:55 | efried | In this new picture, I'm not sure we need/want to forbid that trait. | |
| 20:02:09 | sean-k-mooney | we dont | |
| 20:02:16 | sean-k-mooney | if we are allowed to invent numa toplogies | |
| 20:02:26 | sean-k-mooney | we could specificly to land on un upgraded hosts | |
| 20:02:30 | sean-k-mooney | but it has no other use | |
| 20:02:55 | efried | in this case we don't *want* to target un-upgraded hosts. | |
| 20:03:05 | efried | we want to *allow* landing there, but not *force* it ever. | |
| 20:03:18 | sean-k-mooney | sure | |
| 20:04:08 | efried | hum, but what we don't want is to land across numa nodes on an upgraded host. So yeah, I think the 'fallback' query in both cases should have !HW_NUMA_ROOT. | |
| 20:04:23 | sean-k-mooney | so [scheduler]max_implicit_numa_nodes=0 means dont add HW_NUMA_ROOT or granular groups, anyting above 0 is the amount of numa nodes to try for progressive spliting | |
| 20:04:51 | efried | yeah. But only for NUMA-agnostic flavors. For NUMA-aware flavors, we have an explicit number of nodes we're trying for. | |
| 20:05:03 | sean-k-mooney | yes | |
| 20:05:09 | efried | Do you have it in you to write this up, sean-k-mooney? | |
| 20:05:27 | efried | I made a start on it, but I need to step away for... possibly the rest of the day. | |
| 20:05:32 | sean-k-mooney | ill start a second etherpad. | |
| 20:05:45 | sean-k-mooney | or if you have a start put it in one and i can extend it. | |
| 20:06:01 | sean-k-mooney | efried: do we still need teh per host config option in this model | |
| 20:06:15 | sean-k-mooney | i think no if we do the progressive spliting | |
| 20:06:34 | sean-k-mooney | but we might want to for perfomcne reasin if we jsut want to turn it off | |
| 20:06:58 | efried | yes we do, because sometimes the progressive splitting won't get a result, and they want to force a host to behave like a Train host. | |
| 20:07:15 | sean-k-mooney | ok | |
| 20:07:17 | efried | but that's why the workaround is *off* by default. You have to really need it to turn it on. | |
| 20:07:35 | sean-k-mooney | ok that makes sense | |
| 20:07:59 | sean-k-mooney | dansmith: would you be oke with a [scheduler]/max_implicit_numa_nodes config option | |
| 20:08:13 | sean-k-mooney | to contol the progessive spliting | |
| 20:08:34 | efried | "config-driven API behavior" warning. Not sure I see a better alternative though. | |
| 20:09:26 | sean-k-mooney | efried: well the virt driver can today dowhatever the hell it like in this case anyway so im not sure its an observable thing | |
| 20:09:34 | sean-k-mooney | at least form the api perspctive | |
| 20:09:46 | sean-k-mooney | but i get where your coming form | |
| 20:15:47 | dansmith | that's totally not config-driven api behavior | |
| 20:15:52 | dansmith | and yes, I think that's fine | |
| 20:18:17 | sean-k-mooney | ok ill try to write this up in a comment to the spec and then ill try not to melt bauzas brain when i try to explain this to him tomorrow in our downstream tech call | |
| 20:18:54 | sean-k-mooney | i think 90% of the spec woudl remain the same we just need to update the section that refence the fallback and upgrade impact | |
| 20:22:14 | efried | sean-k-mooney: I left a comment | |
| 20:22:35 | efried | I think I covered the high points, but I'm pretty fried (*e*fried) so I probably missed some things, if you want to fill in. | |
| 20:22:41 | efried | gtg o/ | |
| 20:22:57 | sean-k-mooney | efried_afk: ill review it after coffee | |
| 20:23:00 | efried_afk | thx | |
| 20:23:01 | sean-k-mooney | efried_afk: o/ | |
| 21:53:38 | gmann | johnthetubaguy: what you think of passing service as actual target in service policies? - https://review.opendev.org/#/c/676688/8/nova/api/openstack/compute/services.py | |
| 22:35:04 | artom | We should probably address those errors when running func tests: | |
| 22:35:05 | artom | Exception ignored in: <function _after_fork at 0x7f6382a2ed40> | |
| 22:35:05 | artom | Traceback (most recent call last): | |
| 22:35:05 | artom | File "/usr/lib64/python3.7/threading.py", line 1373, in _after_fork | |
| 22:35:05 | artom | assert len(_active) == 1 | |
| 22:42:35 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Skip all integration jobs for policies only changes. https://review.opendev.org/707268 | |
| 22:43:20 | gmann | efried_afk: stephenfins dansmith gibi melwitt ^^ this will speed up the gate for policy BP changes. | |
| 22:43:33 | gmann | alex_xu: ^^ | |