| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 14:42:04 | sean-k-mooney | so that break operators understand of what we men when we say there is an implcit numa toplogy | |
| 14:42:26 | sean-k-mooney | alex_xu: we can crate a numa toplogy as a impmeenation detail | |
| 14:42:53 | sean-k-mooney | but we should never telll people we are because they will assume it works the same way as the other cases we do that | |
| 14:44:40 | sean-k-mooney | alex_xu: just to clarify we use teh terms implcit and explcit numa topoyg in downstream docs and in bug reports and customer expect an implict numa toplogy to be the same as and expclti one. that is the only reason i bring up docs/release notes | |
| 14:45:23 | mriedem | here is an idea, defer vpmems since you're still arguing about designs 1 day before FF and just sort it out in Ussuri :) | |
| 14:45:31 | mriedem | rather than defer and promise to fix things later | |
| 14:45:54 | alex_xu | mriedem: come on... | |
| 14:46:53 | sean-k-mooney | as i said lest just follow the spec and use the code as it is | |
| 14:46:56 | alex_xu | mriedem: so that is works a later fix? | |
| 14:47:23 | sean-k-mooney | alex_xu: we would need to change qemu/libvirt to not do what the current code does | |
| 14:47:38 | mriedem | alex_xu: i'm not reviewing the series nor am i following the argument here, but i know when we say "we'll hack it in x and fix it for real in x+1" that x+1 rarely happens if ever (x+5) | |
| 14:47:40 | sean-k-mooney | or require a realy numa toplogy with affity | |
| 14:49:57 | mriedem | bauzas: you know you already have head count to work on numa in placement support for nova in ussuri? | |
| 14:50:50 | bauzas | mriedem: if I don't have other things like internal crisises, then yes | |
| 14:50:59 | bauzas | or escalations if you prefer | |
| 14:51:00 | sean-k-mooney | at this point not having it block so much of our roadmap since we cant do it the non plamcent way i dont see we have any choice but to dedicate headcount to it | |
| 14:51:29 | mriedem | bauzas: so that means no | |
| 14:51:41 | bauzas | sean-k-mooney: yeah and I went out of steam trying to find some other way for vGPU affinity | |
| 14:52:16 | mriedem | let's just not bullshit ourselves here is all i'm saying | |
| 14:52:21 | bauzas | mriedem: nah, it just means I'm telling the truth : we can't continue implementing NUMA related features without having NUMA support in placement | |
| 14:52:30 | mriedem | sure you can! | |
| 14:52:36 | bauzas | and that's something I'll make it clea to who it could be | |
| 14:52:39 | mriedem | stephenfin doesn't see why ont | |
| 14:52:40 | mriedem | *not | |
| 14:52:47 | stephenfin | jaypipes | |
| 14:52:54 | stephenfin | that was the main reason why, in the past :) | |
| 14:52:56 | sean-k-mooney | no not jaypies | |
| 14:53:04 | mriedem | god rest his soul | |
| 14:53:08 | stephenfin | placement will solve <X> | |
| 14:53:08 | stephenfin | RIP | |
| 14:53:11 | sean-k-mooney | we can implemntate we jsut cant get it merged | |
| 14:53:50 | sean-k-mooney | anyone here if his house/partner/dogs were safe after the storm | |
| 14:53:53 | mriedem | so what's the actual decision that's blocking the pmems stuff at this point? whether numa is implied or not? | |
| 14:54:27 | stephenfin | by specifying VPMEM, you get an implicit NUMA topology but that implicit NUMA topology works differently to every other implicit NUMA topology | |
| 14:54:39 | mriedem | differently how | |
| 14:54:41 | dansmith | that's BS | |
| 14:54:44 | dansmith | don't do that | |
| 14:54:54 | alex_xu | dansmith: you said yes in PTG :) | |
| 14:54:57 | sean-k-mooney | it does not give numa affinity fo the cpus and memory | |
| 14:55:02 | stephenfin | normally, each guest NUMA node is mapped to a unique host NUMA node | |
| 14:55:07 | stephenfin | with this, they'll float | |
| 14:55:10 | sean-k-mooney | so its not the same as hw:numa_nodes=1 | |
| 14:55:15 | dansmith | alex_xu: sean-k-mooney said I said that too, but I don't agree that I did | |
| 14:55:34 | stephenfin | so the guest has a NUMA topology but it's literally there to work around a quirk in libvirt | |
| 14:55:38 | dansmith | alex_xu: I know that because I'm quite sure I didn't say anything positive about vpmems | |
| 14:55:52 | sean-k-mooney | dansmith: it proably was not you but apprently it was form redhat an i know stephn and i relly hate that idea | |
| 14:55:58 | alex_xu | dansmith: haha | |
| 14:56:15 | stephenfin | I really don't want to blow something out of the water, but I feel by doing this we're really just masking a problem with libvirt (or QEMU) | |
| 14:56:35 | stephenfin | and we're also creating yet another special case that we'll have to clean up later | |
| 14:57:04 | mriedem | so with pmem do you have to have an implicit numa topology? | |
| 14:57:12 | stephenfin | yup, see here | |
| 14:57:26 | stephenfin | https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 | |
| 14:57:28 | mriedem | i mean, will libvirt fail to boot the guest w/o numa defined in the xml? | |
| 14:57:31 | stephenfin | yup | |
| 14:57:32 | mriedem | ah ok | |
| 14:57:34 | alex_xu | sean-k-mooney: stephenfin ok, let me try to remove that, maybe it is super fast and simple than our discussion :) | |
| 14:57:50 | mriedem | alex_xu: remove what? qemu code? | |
| 14:57:58 | stephenfin | alex_xu: Remove the implicit NUMA topology? Doesn't look like that's an option. See the above link | |
| 14:58:03 | mriedem | that is likely years before it hits LTS | |
| 14:58:33 | stephenfin | libvirt is telling us we need a NUMA topology, so we must give it a NUMA topology | |
| 14:58:46 | alex_xu | stephenfin: add numa topology explicityly | |
| 14:58:54 | mriedem | ok so if qemu requires implicit numa, then our options are hack it and create a unicorn (bad for nova), change qemu and defer the feature (bad for intel), or do numa affinity as usual in nova, | |
| 14:58:55 | mriedem | right? | |
| 14:58:55 | bauzas | oh gosh | |
| 14:59:14 | stephenfin | mriedem: Right. But | |
| 14:59:17 | sean-k-mooney | mriedem: yes | |
| 14:59:21 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | |
| 14:59:39 | stephenfin | the reason they avoided doing it that way was because they wanted to avoid doing more stuff in NUMATopologyFilter | |
| 14:59:42 | dansmith | artom: commented on the top numa functional patch | |
| 14:59:46 | bauzas | ... and all Hatters technically need to bail out for attending some meeting :) | |
| 14:59:56 | dansmith | bauzas: not all of them :) | |
| 15:00:12 | artom | dansmith, ack - I'm still trying to write words that make sense for https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@179 | |
| 15:00:27 | sean-k-mooney | dansmith: can i join your team to not do numa stuff. it seam simpler | |
| 15:00:27 | mriedem | stephenfin: would the numa topo filter need to look at pmems or something? or just the hardware module that the filter calls? | |
| 15:00:27 | alex_xu | stephenfin: emm...no, that isn't the reason | |
| 15:00:30 | bauzas | dansmith: heh, so true | |
| 15:00:38 | artom | It's tricky because... just because we found some old computes in the cell, doesn't mean the source and dest are old | |
| 15:00:38 | stephenfin | to be able to do NUMA affinity properly, we'll need to provide information about the VPMEM devices available in each node as part of the NUMATopology/NUMACell objects | |
| 15:00:41 | dansmith | sean-k-mooney: heh | |
| 15:00:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513 | |
| 15:00:46 | artom | So I want to word around that | |
| 15:00:53 | dansmith | artom: ack | |
| 15:01:30 | mriedem | artom: the code isn't granular to just the source and selected dest - where you're checking we don't have a dest yet in some cases, | |
| 15:01:37 | mriedem | so i'm not sure why that would matter for the reno | |
| 15:01:41 | mriedem | oh this is the log | |
| 15:01:58 | artom | mriedem, yeah - I don't want to say "WARNING LEGACY BEHAVIOR" unconditionally | |
| 15:01:59 | stephenfin | Yeah, the NUMA fitting code in the hardware module only gets InstanceNUMATopology, NUMATopology, PCIRequest and whatever the host PCI tracking object is | |
| 15:02:13 | mriedem | artom: i think it's fine to say computes aren't new enough | |
| 15:02:17 | mriedem | that's generic enough | |
| 15:02:32 | artom | Yeah, except they *might* be new enough | |
| 15:02:37 | mriedem | that doesn't matter | |
| 15:02:44 | artom | If that specific source and dest are, even if others in the cell aren't | |
| 15:02:45 | mriedem | b/c that's not how the code works | |
| 15:02:55 | alex_xu | stephenfin: ah, you complain the part of explicltly numa but no affinity | |
| 15:02:56 | mriedem | iow i don't see a problem with the paraphrase that dan gave | |
| 15:03:13 | alex_xu | stephenfin: but you are ok with implicity numa? | |
| 15:03:21 | artom | So, if we got passed the service check, it means there are some old computes in the cell | |
| 15:03:26 | alex_xu | to be host, I'm confused what we discussion for now.... | |
| 15:03:32 | mriedem | artom: correct | |
| 15:03:35 | artom | But... the actual source and dest might both be new | |