Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-11
14:55:52 sean-k-mooney dansmith: it proably was not you but apprently it was form redhat an i know stephn and i relly hate that idea
14:55:58 alex_xu dansmith: haha
14:56:15 stephenfin I really don't want to blow something out of the water, but I feel by doing this we're really just masking a problem with libvirt (or QEMU)
14:56:35 stephenfin and we're also creating yet another special case that we'll have to clean up later
14:57:04 mriedem so with pmem do you have to have an implicit numa topology?
14:57:12 stephenfin yup, see here
14:57:26 stephenfin https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615
14:57:28 mriedem i mean, will libvirt fail to boot the guest w/o numa defined in the xml?
14:57:31 stephenfin yup
14:57:32 mriedem ah ok
14:57:34 alex_xu sean-k-mooney: stephenfin ok, let me try to remove that, maybe it is super fast and simple than our discussion :)
14:57:50 mriedem alex_xu: remove what? qemu code?
14:57:58 stephenfin alex_xu: Remove the implicit NUMA topology? Doesn't look like that's an option. See the above link
14:58:03 mriedem that is likely years before it hits LTS
14:58:33 stephenfin libvirt is telling us we need a NUMA topology, so we must give it a NUMA topology
14:58:46 alex_xu stephenfin: add numa topology explicityly
14:58:54 mriedem ok so if qemu requires implicit numa, then our options are hack it and create a unicorn (bad for nova), change qemu and defer the feature (bad for intel), or do numa affinity as usual in nova,
14:58:55 mriedem right?
14:58:55 bauzas oh gosh
14:59:14 stephenfin mriedem: Right. But
14:59:17 sean-k-mooney mriedem: yes
14:59:21 openstackgerrit Eric Fried proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329
14:59:39 stephenfin the reason they avoided doing it that way was because they wanted to avoid doing more stuff in NUMATopologyFilter
14:59:42 dansmith artom: commented on the top numa functional patch
14:59:46 bauzas ... and all Hatters technically need to bail out for attending some meeting :)
14:59:56 dansmith bauzas: not all of them :)
15:00:12 artom dansmith, ack - I'm still trying to write words that make sense for https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@179
15:00:27 sean-k-mooney dansmith: can i join your team to not do numa stuff. it seam simpler
15:00:27 mriedem stephenfin: would the numa topo filter need to look at pmems or something? or just the hardware module that the filter calls?
15:00:27 alex_xu stephenfin: emm...no, that isn't the reason
15:00:30 bauzas dansmith: heh, so true
15:00:38 artom It's tricky because... just because we found some old computes in the cell, doesn't mean the source and dest are old
15:00:38 stephenfin to be able to do NUMA affinity properly, we'll need to provide information about the VPMEM devices available in each node as part of the NUMATopology/NUMACell objects
15:00:41 dansmith sean-k-mooney: heh
15:00:42 openstackgerrit Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513
15:00:46 artom So I want to word around that
15:00:53 dansmith artom: ack
15:01:30 mriedem artom: the code isn't granular to just the source and selected dest - where you're checking we don't have a dest yet in some cases,
15:01:37 mriedem so i'm not sure why that would matter for the reno
15:01:41 mriedem oh this is the log
15:01:58 artom mriedem, yeah - I don't want to say "WARNING LEGACY BEHAVIOR" unconditionally
15:01:59 stephenfin Yeah, the NUMA fitting code in the hardware module only gets InstanceNUMATopology, NUMATopology, PCIRequest and whatever the host PCI tracking object is
15:02:13 mriedem artom: i think it's fine to say computes aren't new enough
15:02:17 mriedem that's generic enough
15:02:32 artom Yeah, except they *might* be new enough
15:02:37 mriedem that doesn't matter
15:02:44 artom If that specific source and dest are, even if others in the cell aren't
15:02:45 mriedem b/c that's not how the code works
15:02:55 alex_xu stephenfin: ah, you complain the part of explicltly numa but no affinity
15:02:56 mriedem iow i don't see a problem with the paraphrase that dan gave
15:03:13 alex_xu stephenfin: but you are ok with implicity numa?
15:03:21 artom So, if we got passed the service check, it means there are some old computes in the cell
15:03:26 alex_xu to be host, I'm confused what we discussion for now....
15:03:32 mriedem artom: correct
15:03:35 artom But... the actual source and dest might both be new
15:03:44 artom In which case, assuming the workaround is enabled, NUMA-LM will happen
15:03:45 stephenfin alex_xu: That was the reason we didn't do the implicit NUMA affinity, right?
15:03:46 mriedem sure, but we don't necessarily have a dest at the point of that check
15:03:51 mriedem yes
15:04:08 artom So I don't want to say "you enabled the workaround, things will explode"
15:04:14 mriedem artom: i mentioned this here https://review.opendev.org/#/c/640021/48/nova/conductor/tasks/live_migrate.py@196
15:04:20 artom More like "you enabled the workaround, things will explode if the source and dest are old"
15:04:25 artom (explode == eggagerating)
15:04:28 luyao stephenfin, stephenfin : i'm confused about 'implicit' numa
15:04:35 spatel Help!! - I want to evacuate compute nodes because it has hardware issue, but all running vm using local-disk ( NOT shared disk ) can i do evacuate?
15:04:35 alex_xu stephenfin: no, that isn't. The reason do implicit NUMA topology for instance just due to someone asking that usecase in the PTG
15:04:38 mriedem artom: i don't think that's what dan suggested
15:04:49 mriedem "Computes are too old to do the smart thing, workaround is enabled, doing that"
15:04:59 dansmith spatel: see topic
15:05:13 alex_xu stephenfin: the explicity NUMA topology for instance and without affinity is clear plan in the PTG
15:05:15 mriedem you adjust the warning to say that computes are too old to do the supported method or whatever
15:05:23 dansmith spatel: you might notice that we're a day before FF and there are multiple dev-related discussions going on in here
15:05:53 spatel dansmith: sorry about that, you guys continue.
15:05:53 luyao stephenfin,alex_xu: what is implicit NUMA affinity
15:05:58 stephenfin alex_xu: It's sounding like the implications of that plan were not properly understood though. Certainly I didn't grasp them
15:07:01 alex_xu luyao: I guess that is just stephenfin typing wrong word, actually mean implicity NUMA topology
15:07:19 alex_xu stephenfin: ^ is that right? or you pointed to something I may not get
15:07:26 mriedem artom: i left a suggestion on wording
15:07:28 stephenfin luyao: Implicit NUMA affinity is what we do for hugepages and CPU pinning. If you use either of those features, your guest will have a one-node NUMA topology and it will be pinned to a host NUMA node
15:07:45 stephenfin alex_xu: Yeah, an implicit NUMA topology
15:08:19 luyao stephenfin: I thought implicit Alex said is the guest numa not bind to host numa
15:08:24 mriedem stephenfin: oh i have a solution - drop the numa part but only support vpmems on ppc64 https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 :)
15:08:32 mriedem power to the rescue
15:08:47 stephenfin fight the power
15:08:51 luyao stephenfin, alex_xu: so you are discuss different things ?
15:09:15 mriedem does tonyb have openshift working on power yet?
15:09:23 stephenfin luyao: no, the disagreement is on the lack of pinning
15:09:45 artom mriedem, https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@200
15:10:40 stephenfin the fact that for hugepages, the single guest NUMA cell is pinned to a host NUMA cell, but for VPMEM, the single guest NUMA cell floats across all host NUMA cells
15:10:48 stephenfin *for hugepages and CPU pinning
15:11:19 alex_xu stephenfin: yes, so you want the pinning, but is it ok without affinity for now?
15:11:25 stephenfin bauzas: Think you could hit these two too? https://review.opendev.org/#/c/681060/ https://review.opendev.org/#/c/681061/
15:11:43 stephenfin alex_xu: I'll take that as a middle ground, yes
15:12:04 bauzas stephenfin: looking
15:12:15 alex_xu stephenfin: so the only I need is fall into this branch https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1740, right
15:12:24 stephenfin yup
15:12:28 alex_xu cool
15:12:57 mriedem artom: replied
15:13:02 mriedem with anger
15:13:07 luyao stephenfin: Does it make sense to do the numa pinning now without affinity.

Earlier   Later