Posted	Nick	Remark
#openstack-nova - 2019-09-11
14:46:56	alex_xu	mriedem: so that is works a later fix?
14:47:23	sean-k-mooney	alex_xu: we would need to change qemu/libvirt to not do what the current code does
14:47:38	mriedem	alex_xu: i'm not reviewing the series nor am i following the argument here, but i know when we say "we'll hack it in x and fix it for real in x+1" that x+1 rarely happens if ever (x+5)
14:47:40	sean-k-mooney	or require a realy numa toplogy with affity
14:49:57	mriedem	bauzas: you know you already have head count to work on numa in placement support for nova in ussuri?
14:50:50	bauzas	mriedem: if I don't have other things like internal crisises, then yes
14:50:59	bauzas	or escalations if you prefer
14:51:00	sean-k-mooney	at this point not having it block so much of our roadmap since we cant do it the non plamcent way i dont see we have any choice but to dedicate headcount to it
14:51:29	mriedem	bauzas: so that means no
14:51:41	bauzas	sean-k-mooney: yeah and I went out of steam trying to find some other way for vGPU affinity
14:52:16	mriedem	let's just not bullshit ourselves here is all i'm saying
14:52:21	bauzas	mriedem: nah, it just means I'm telling the truth : we can't continue implementing NUMA related features without having NUMA support in placement
14:52:30	mriedem	sure you can!
14:52:36	bauzas	and that's something I'll make it clea to who it could be
14:52:39	mriedem	stephenfin doesn't see why ont
14:52:40	mriedem	*not
14:52:47	stephenfin	jaypipes
14:52:54	stephenfin	that was the main reason why, in the past :)
14:52:56	sean-k-mooney	no not jaypies
14:53:04	mriedem	god rest his soul
14:53:08	stephenfin	placement will solve <X>
14:53:08	stephenfin	RIP
14:53:11	sean-k-mooney	we can implemntate we jsut cant get it merged
14:53:50	sean-k-mooney	anyone here if his house/partner/dogs were safe after the storm
14:53:53	mriedem	so what's the actual decision that's blocking the pmems stuff at this point? whether numa is implied or not?
14:54:27	stephenfin	by specifying VPMEM, you get an implicit NUMA topology but that implicit NUMA topology works differently to every other implicit NUMA topology
14:54:39	mriedem	differently how
14:54:41	dansmith	that's BS
14:54:44	dansmith	don't do that
14:54:54	alex_xu	dansmith: you said yes in PTG :)
14:54:57	sean-k-mooney	it does not give numa affinity fo the cpus and memory
14:55:02	stephenfin	normally, each guest NUMA node is mapped to a unique host NUMA node
14:55:07	stephenfin	with this, they'll float
14:55:10	sean-k-mooney	so its not the same as hw:numa_nodes=1
14:55:15	dansmith	alex_xu: sean-k-mooney said I said that too, but I don't agree that I did
14:55:34	stephenfin	so the guest has a NUMA topology but it's literally there to work around a quirk in libvirt
14:55:38	dansmith	alex_xu: I know that because I'm quite sure I didn't say anything positive about vpmems
14:55:52	sean-k-mooney	dansmith: it proably was not you but apprently it was form redhat an i know stephn and i relly hate that idea
14:55:58	alex_xu	dansmith: haha
14:56:15	stephenfin	I really don't want to blow something out of the water, but I feel by doing this we're really just masking a problem with libvirt (or QEMU)
14:56:35	stephenfin	and we're also creating yet another special case that we'll have to clean up later
14:57:04	mriedem	so with pmem do you have to have an implicit numa topology?
14:57:12	stephenfin	yup, see here
14:57:26	stephenfin	https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615
14:57:28	mriedem	i mean, will libvirt fail to boot the guest w/o numa defined in the xml?
14:57:31	stephenfin	yup
14:57:32	mriedem	ah ok
14:57:34	alex_xu	sean-k-mooney: stephenfin ok, let me try to remove that, maybe it is super fast and simple than our discussion :)
14:57:50	mriedem	alex_xu: remove what? qemu code?
14:57:58	stephenfin	alex_xu: Remove the implicit NUMA topology? Doesn't look like that's an option. See the above link
14:58:03	mriedem	that is likely years before it hits LTS
14:58:33	stephenfin	libvirt is telling us we need a NUMA topology, so we must give it a NUMA topology
14:58:46	alex_xu	stephenfin: add numa topology explicityly
14:58:54	mriedem	ok so if qemu requires implicit numa, then our options are hack it and create a unicorn (bad for nova), change qemu and defer the feature (bad for intel), or do numa affinity as usual in nova,
14:58:55	mriedem	right?
14:58:55	bauzas	oh gosh
14:59:14	stephenfin	mriedem: Right. But
14:59:17	sean-k-mooney	mriedem: yes
14:59:21	openstackgerrit	Eric Fried proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329
14:59:39	stephenfin	the reason they avoided doing it that way was because they wanted to avoid doing more stuff in NUMATopologyFilter
14:59:42	dansmith	artom: commented on the top numa functional patch
14:59:46	bauzas	... and all Hatters technically need to bail out for attending some meeting :)
14:59:56	dansmith	bauzas: not all of them :)
15:00:12	artom	dansmith, ack - I'm still trying to write words that make sense for https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@179
15:00:27	sean-k-mooney	dansmith: can i join your team to not do numa stuff. it seam simpler
15:00:27	mriedem	stephenfin: would the numa topo filter need to look at pmems or something? or just the hardware module that the filter calls?
15:00:27	alex_xu	stephenfin: emm...no, that isn't the reason
15:00:30	bauzas	dansmith: heh, so true
15:00:38	artom	It's tricky because... just because we found some old computes in the cell, doesn't mean the source and dest are old
15:00:38	stephenfin	to be able to do NUMA affinity properly, we'll need to provide information about the VPMEM devices available in each node as part of the NUMATopology/NUMACell objects
15:00:41	dansmith	sean-k-mooney: heh
15:00:42	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513
15:00:46	artom	So I want to word around that
15:00:53	dansmith	artom: ack
15:01:30	mriedem	artom: the code isn't granular to just the source and selected dest - where you're checking we don't have a dest yet in some cases,
15:01:37	mriedem	so i'm not sure why that would matter for the reno
15:01:41	mriedem	oh this is the log
15:01:58	artom	mriedem, yeah - I don't want to say "WARNING LEGACY BEHAVIOR" unconditionally
15:01:59	stephenfin	Yeah, the NUMA fitting code in the hardware module only gets InstanceNUMATopology, NUMATopology, PCIRequest and whatever the host PCI tracking object is
15:02:13	mriedem	artom: i think it's fine to say computes aren't new enough
15:02:17	mriedem	that's generic enough
15:02:32	artom	Yeah, except they *might* be new enough
15:02:37	mriedem	that doesn't matter
15:02:44	artom	If that specific source and dest are, even if others in the cell aren't
15:02:45	mriedem	b/c that's not how the code works
15:02:55	alex_xu	stephenfin: ah, you complain the part of explicltly numa but no affinity
15:02:56	mriedem	iow i don't see a problem with the paraphrase that dan gave
15:03:13	alex_xu	stephenfin: but you are ok with implicity numa?
15:03:21	artom	So, if we got passed the service check, it means there are some old computes in the cell
15:03:26	alex_xu	to be host, I'm confused what we discussion for now....
15:03:32	mriedem	artom: correct
15:03:35	artom	But... the actual source and dest might both be new
15:03:44	artom	In which case, assuming the workaround is enabled, NUMA-LM will happen
15:03:45	stephenfin	alex_xu: That was the reason we didn't do the implicit NUMA affinity, right?
15:03:46	mriedem	sure, but we don't necessarily have a dest at the point of that check
15:03:51	mriedem	yes
15:04:08	artom	So I don't want to say "you enabled the workaround, things will explode"
15:04:14	mriedem	artom: i mentioned this here https://review.opendev.org/#/c/640021/48/nova/conductor/tasks/live_migrate.py@196
15:04:20	artom	More like "you enabled the workaround, things will explode if the source and dest are old"
15:04:25	artom	(explode == eggagerating)

Earlier   Later