Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-11
19:52:35 sean-k-mooney so is the poposal always do both queries and merge them or to the progressive spliting
19:53:17 efried Both
19:53:22 efried because
19:53:40 efried we want upgraded hosts to cut over to NUMA-modeled by default.
19:53:52 sean-k-mooney ok so progressive for non numa and both qureies for numa
19:54:17 efried I think we need two queries for both.
19:54:41 sean-k-mooney i think it will be more like 2 for numa and 5 for non numa
19:54:46 efried yeah
19:55:03 sean-k-mooney 5 asumign we dint max_numa_nodes=5
19:55:08 efried 4
19:55:15 sean-k-mooney sorry 4
19:55:25 efried I asked this earlier: what's the max number of NUMA nodes we know of on any system? Is it 4?
19:55:28 sean-k-mooney could we bail out if the first numa query passed
19:55:34 efried yes
19:56:35 sean-k-mooney on a 32-64 core 2 socket amd eypci host with a numa node per l3 region 16-32 numa node
19:56:56 efried ye gods
19:57:04 sean-k-mooney if you dont expose a numa node per l3 region then 8 is more realisitc
19:57:15 efried Can we tell from the db?
19:57:32 efried I guess we could tell by querying placement.
19:57:34 sean-k-mooney its in teh hohst numa toplogy blob in the cell db
19:57:53 sean-k-mooney so yes technically.
19:58:08 efried but I don't know that we really want to go discovering that every time we schedule.
19:58:24 sean-k-mooney every time defintly not
19:58:31 sean-k-mooney schduler config option?
19:58:33 efried configurable [scheduler]max_implicit_numa_nodes ?
19:58:36 efried yeah
19:58:51 efried because even if it's possible to have a 32-way split, that would almost never be a good idea probably.
19:58:57 sean-k-mooney set it to 0 to diable numa qureis entrily otherwise its the limit
19:59:17 sean-k-mooney efried: it give you almost a 35% performace boost
19:59:19 efried "disable" meaning what?
19:59:43 sean-k-mooney efried: disabel mean i have disable numa reporting in my entire cloud so dont even try
20:00:13 efried Need to grok how that query would be different from the non-upgraded query.
20:00:24 sean-k-mooney it would be the same
20:00:32 sean-k-mooney just want we do today in tran
20:00:33 efried Oh, because it would have the HW_NUMA_ROOT trait.
20:01:09 sean-k-mooney well require=!HW_NUMA_ROOT would be the detla form train i guess
20:01:22 sean-k-mooney but if no host reports numa thats a noop
20:01:55 efried In this new picture, I'm not sure we need/want to forbid that trait.
20:02:09 sean-k-mooney we dont
20:02:16 sean-k-mooney if we are allowed to invent numa toplogies
20:02:26 sean-k-mooney we could specificly to land on un upgraded hosts
20:02:30 sean-k-mooney but it has no other use
20:02:55 efried in this case we don't *want* to target un-upgraded hosts.
20:03:05 efried we want to *allow* landing there, but not *force* it ever.
20:03:18 sean-k-mooney sure
20:04:08 efried hum, but what we don't want is to land across numa nodes on an upgraded host. So yeah, I think the 'fallback' query in both cases should have !HW_NUMA_ROOT.
20:04:23 sean-k-mooney so [scheduler]max_implicit_numa_nodes=0 means dont add HW_NUMA_ROOT or granular groups, anyting above 0 is the amount of numa nodes to try for progressive spliting
20:04:51 efried yeah. But only for NUMA-agnostic flavors. For NUMA-aware flavors, we have an explicit number of nodes we're trying for.
20:05:03 sean-k-mooney yes
20:05:09 efried Do you have it in you to write this up, sean-k-mooney?
20:05:27 efried I made a start on it, but I need to step away for... possibly the rest of the day.
20:05:32 sean-k-mooney ill start a second etherpad.
20:05:45 sean-k-mooney or if you have a start put it in one and i can extend it.
20:06:01 sean-k-mooney efried: do we still need teh per host config option in this model
20:06:15 sean-k-mooney i think no if we do the progressive spliting
20:06:34 sean-k-mooney but we might want to for perfomcne reasin if we jsut want to turn it off
20:06:58 efried yes we do, because sometimes the progressive splitting won't get a result, and they want to force a host to behave like a Train host.
20:07:15 sean-k-mooney ok
20:07:17 efried but that's why the workaround is *off* by default. You have to really need it to turn it on.
20:07:35 sean-k-mooney ok that makes sense
20:07:59 sean-k-mooney dansmith: would you be oke with a [scheduler]/max_implicit_numa_nodes config option
20:08:13 sean-k-mooney to contol the progessive spliting
20:08:34 efried "config-driven API behavior" warning. Not sure I see a better alternative though.
20:09:26 sean-k-mooney efried: well the virt driver can today dowhatever the hell it like in this case anyway so im not sure its an observable thing
20:09:34 sean-k-mooney at least form the api perspctive
20:09:46 sean-k-mooney but i get where your coming form
20:15:47 dansmith that's totally not config-driven api behavior
20:15:52 dansmith and yes, I think that's fine
20:18:17 sean-k-mooney ok ill try to write this up in a comment to the spec and then ill try not to melt bauzas brain when i try to explain this to him tomorrow in our downstream tech call
20:18:54 sean-k-mooney i think 90% of the spec woudl remain the same we just need to update the section that refence the fallback and upgrade impact
20:22:14 efried sean-k-mooney: I left a comment
20:22:35 efried I think I covered the high points, but I'm pretty fried (*e*fried) so I probably missed some things, if you want to fill in.
20:22:41 efried gtg o/
20:22:57 sean-k-mooney efried_afk: ill review it after coffee
20:23:00 efried_afk thx
20:23:01 sean-k-mooney efried_afk: o/
21:53:38 gmann johnthetubaguy: what you think of passing service as actual target in service policies? - https://review.opendev.org/#/c/676688/8/nova/api/openstack/compute/services.py
22:35:04 artom We should probably address those errors when running func tests:
22:35:05 artom Exception ignored in: <function _after_fork at 0x7f6382a2ed40>
22:35:05 artom Traceback (most recent call last):
22:35:05 artom File "/usr/lib64/python3.7/threading.py", line 1373, in _after_fork
22:35:05 artom assert len(_active) == 1
22:42:35 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Skip all integration jobs for policies only changes. https://review.opendev.org/707268
22:43:20 gmann efried_afk: stephenfins dansmith gibi melwitt ^^ this will speed up the gate for policy BP changes.
22:43:33 gmann alex_xu: ^^
22:44:26 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Skip to run all integration jobs for policies-only changes. https://review.opendev.org/707268
22:44:53 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Skip to run all integration jobs for policies-only changes. https://review.opendev.org/707268
23:06:01 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Skip to run all integration jobs for policies-only changes. https://review.opendev.org/707268
23:06:16 gmann melwitt: updated ^^
23:06:31 melwitt ack
23:23:10 sean-k-mooney dansmith: efried_afk: i did a thing. https://review.opendev.org/#/c/552924/17/specs/ussuri/approved/numa-topology-with-rps.rst@516
23:23:38 sean-k-mooney dansmith: efried_afk its a trivial poc of the progresive generation of the numa toplogies for a non numa vm
23:24:19 sean-k-mooney just the toplogy object not the queries but i could proably hack that up tomorrow
#openstack-nova - 2020-02-12
09:02:31 gibi efried_afk, sean-k-mooney, dansmith: read up on the numa fallback discussion from yesterday. impressive work. I need to digest the result
09:30:22 openstackgerrit ZhouHeng proposed openstack/nova-specs master: Support set port vnic type to create server https://review.opendev.org/672400
09:38:36 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Fix attached encrypted LUKSv1 volume extension https://review.opendev.org/706900
09:44:21 openstackgerrit Guo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120
10:00:57 bauzas gibi: yeah I did it too
10:01:06 bauzas I'll work on a new revision
10:14:40 bauzas gibi: honestly, I'm looking at https://review.opendev.org/#/c/552924/17/specs/ussuri/approved/numa-topology-with-rps.rst@516 and I don't know what the outcome consensus...

Earlier   Later