| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 14:52:39 | mriedem | stephenfin doesn't see why ont | |
| 14:52:40 | mriedem | *not | |
| 14:52:47 | stephenfin | jaypipes | |
| 14:52:54 | stephenfin | that was the main reason why, in the past :) | |
| 14:52:56 | sean-k-mooney | no not jaypies | |
| 14:53:04 | mriedem | god rest his soul | |
| 14:53:08 | stephenfin | RIP | |
| 14:53:08 | stephenfin | placement will solve <X> | |
| 14:53:11 | sean-k-mooney | we can implemntate we jsut cant get it merged | |
| 14:53:50 | sean-k-mooney | anyone here if his house/partner/dogs were safe after the storm | |
| 14:53:53 | mriedem | so what's the actual decision that's blocking the pmems stuff at this point? whether numa is implied or not? | |
| 14:54:27 | stephenfin | by specifying VPMEM, you get an implicit NUMA topology but that implicit NUMA topology works differently to every other implicit NUMA topology | |
| 14:54:39 | mriedem | differently how | |
| 14:54:41 | dansmith | that's BS | |
| 14:54:44 | dansmith | don't do that | |
| 14:54:54 | alex_xu | dansmith: you said yes in PTG :) | |
| 14:54:57 | sean-k-mooney | it does not give numa affinity fo the cpus and memory | |
| 14:55:02 | stephenfin | normally, each guest NUMA node is mapped to a unique host NUMA node | |
| 14:55:07 | stephenfin | with this, they'll float | |
| 14:55:10 | sean-k-mooney | so its not the same as hw:numa_nodes=1 | |
| 14:55:15 | dansmith | alex_xu: sean-k-mooney said I said that too, but I don't agree that I did | |
| 14:55:34 | stephenfin | so the guest has a NUMA topology but it's literally there to work around a quirk in libvirt | |
| 14:55:38 | dansmith | alex_xu: I know that because I'm quite sure I didn't say anything positive about vpmems | |
| 14:55:52 | sean-k-mooney | dansmith: it proably was not you but apprently it was form redhat an i know stephn and i relly hate that idea | |
| 14:55:58 | alex_xu | dansmith: haha | |
| 14:56:15 | stephenfin | I really don't want to blow something out of the water, but I feel by doing this we're really just masking a problem with libvirt (or QEMU) | |
| 14:56:35 | stephenfin | and we're also creating yet another special case that we'll have to clean up later | |
| 14:57:04 | mriedem | so with pmem do you have to have an implicit numa topology? | |
| 14:57:12 | stephenfin | yup, see here | |
| 14:57:26 | stephenfin | https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 | |
| 14:57:28 | mriedem | i mean, will libvirt fail to boot the guest w/o numa defined in the xml? | |
| 14:57:31 | stephenfin | yup | |
| 14:57:32 | mriedem | ah ok | |
| 14:57:34 | alex_xu | sean-k-mooney: stephenfin ok, let me try to remove that, maybe it is super fast and simple than our discussion :) | |
| 14:57:50 | mriedem | alex_xu: remove what? qemu code? | |
| 14:57:58 | stephenfin | alex_xu: Remove the implicit NUMA topology? Doesn't look like that's an option. See the above link | |
| 14:58:03 | mriedem | that is likely years before it hits LTS | |
| 14:58:33 | stephenfin | libvirt is telling us we need a NUMA topology, so we must give it a NUMA topology | |
| 14:58:46 | alex_xu | stephenfin: add numa topology explicityly | |
| 14:58:54 | mriedem | ok so if qemu requires implicit numa, then our options are hack it and create a unicorn (bad for nova), change qemu and defer the feature (bad for intel), or do numa affinity as usual in nova, | |
| 14:58:55 | bauzas | oh gosh | |
| 14:58:55 | mriedem | right? | |
| 14:59:14 | stephenfin | mriedem: Right. But | |
| 14:59:17 | sean-k-mooney | mriedem: yes | |
| 14:59:21 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | |
| 14:59:39 | stephenfin | the reason they avoided doing it that way was because they wanted to avoid doing more stuff in NUMATopologyFilter | |
| 14:59:42 | dansmith | artom: commented on the top numa functional patch | |
| 14:59:46 | bauzas | ... and all Hatters technically need to bail out for attending some meeting :) | |
| 14:59:56 | dansmith | bauzas: not all of them :) | |
| 15:00:12 | artom | dansmith, ack - I'm still trying to write words that make sense for https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@179 | |
| 15:00:27 | alex_xu | stephenfin: emm...no, that isn't the reason | |
| 15:00:27 | mriedem | stephenfin: would the numa topo filter need to look at pmems or something? or just the hardware module that the filter calls? | |
| 15:00:27 | sean-k-mooney | dansmith: can i join your team to not do numa stuff. it seam simpler | |
| 15:00:30 | bauzas | dansmith: heh, so true | |
| 15:00:38 | stephenfin | to be able to do NUMA affinity properly, we'll need to provide information about the VPMEM devices available in each node as part of the NUMATopology/NUMACell objects | |
| 15:00:38 | artom | It's tricky because... just because we found some old computes in the cell, doesn't mean the source and dest are old | |
| 15:00:41 | dansmith | sean-k-mooney: heh | |
| 15:00:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513 | |
| 15:00:46 | artom | So I want to word around that | |
| 15:00:53 | dansmith | artom: ack | |
| 15:01:30 | mriedem | artom: the code isn't granular to just the source and selected dest - where you're checking we don't have a dest yet in some cases, | |
| 15:01:37 | mriedem | so i'm not sure why that would matter for the reno | |
| 15:01:41 | mriedem | oh this is the log | |
| 15:01:58 | artom | mriedem, yeah - I don't want to say "WARNING LEGACY BEHAVIOR" unconditionally | |
| 15:01:59 | stephenfin | Yeah, the NUMA fitting code in the hardware module only gets InstanceNUMATopology, NUMATopology, PCIRequest and whatever the host PCI tracking object is | |
| 15:02:13 | mriedem | artom: i think it's fine to say computes aren't new enough | |
| 15:02:17 | mriedem | that's generic enough | |
| 15:02:32 | artom | Yeah, except they *might* be new enough | |
| 15:02:37 | mriedem | that doesn't matter | |
| 15:02:44 | artom | If that specific source and dest are, even if others in the cell aren't | |
| 15:02:45 | mriedem | b/c that's not how the code works | |
| 15:02:55 | alex_xu | stephenfin: ah, you complain the part of explicltly numa but no affinity | |
| 15:02:56 | mriedem | iow i don't see a problem with the paraphrase that dan gave | |
| 15:03:13 | alex_xu | stephenfin: but you are ok with implicity numa? | |
| 15:03:21 | artom | So, if we got passed the service check, it means there are some old computes in the cell | |
| 15:03:26 | alex_xu | to be host, I'm confused what we discussion for now.... | |
| 15:03:32 | mriedem | artom: correct | |
| 15:03:35 | artom | But... the actual source and dest might both be new | |
| 15:03:44 | artom | In which case, assuming the workaround is enabled, NUMA-LM will happen | |
| 15:03:45 | stephenfin | alex_xu: That was the reason we didn't do the implicit NUMA affinity, right? | |
| 15:03:46 | mriedem | sure, but we don't necessarily have a dest at the point of that check | |
| 15:03:51 | mriedem | yes | |
| 15:04:08 | artom | So I don't want to say "you enabled the workaround, things will explode" | |
| 15:04:14 | mriedem | artom: i mentioned this here https://review.opendev.org/#/c/640021/48/nova/conductor/tasks/live_migrate.py@196 | |
| 15:04:20 | artom | More like "you enabled the workaround, things will explode if the source and dest are old" | |
| 15:04:25 | artom | (explode == eggagerating) | |
| 15:04:28 | luyao | stephenfin, stephenfin : i'm confused about 'implicit' numa | |
| 15:04:35 | alex_xu | stephenfin: no, that isn't. The reason do implicit NUMA topology for instance just due to someone asking that usecase in the PTG | |
| 15:04:35 | spatel | Help!! - I want to evacuate compute nodes because it has hardware issue, but all running vm using local-disk ( NOT shared disk ) can i do evacuate? | |
| 15:04:38 | mriedem | artom: i don't think that's what dan suggested | |
| 15:04:49 | mriedem | "Computes are too old to do the smart thing, workaround is enabled, doing that" | |
| 15:04:59 | dansmith | spatel: see topic | |
| 15:05:13 | alex_xu | stephenfin: the explicity NUMA topology for instance and without affinity is clear plan in the PTG | |
| 15:05:15 | mriedem | you adjust the warning to say that computes are too old to do the supported method or whatever | |
| 15:05:23 | dansmith | spatel: you might notice that we're a day before FF and there are multiple dev-related discussions going on in here | |
| 15:05:53 | luyao | stephenfin,alex_xu: what is implicit NUMA affinity | |
| 15:05:53 | spatel | dansmith: sorry about that, you guys continue. | |
| 15:05:58 | stephenfin | alex_xu: It's sounding like the implications of that plan were not properly understood though. Certainly I didn't grasp them | |
| 15:07:01 | alex_xu | luyao: I guess that is just stephenfin typing wrong word, actually mean implicity NUMA topology | |
| 15:07:19 | alex_xu | stephenfin: ^ is that right? or you pointed to something I may not get | |