| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 14:38:03 | sean-k-mooney | its been nonsense for 3 years but that is not going to change. so we should enbrace doing it the plamcnet way. in U to give mriedem a nickel | |
| 14:38:10 | stephenfin | gimme a half hour to check out how tough this would be to do. If it's terrible, I'll hold my tongue | |
| 14:39:00 | alex_xu | stephenfin: yes, it will be very easy to support numa affinity once we have numa in placement | |
| 14:39:09 | sean-k-mooney | its 1 field in the host numa toplogy blob and a tweek to the numa toplogy filter to match on it in the numa_fit_instace_to_host funciton | |
| 14:39:12 | alex_xu | stephenfin: just like vgpu, the first step no numa affinity | |
| 14:39:33 | sean-k-mooney | alex_xu: its very easy to support it without placmeent but not polically viable | |
| 14:39:45 | stephenfin | vGPU didn't imply an implicit guest NUMA topology though, that's the key difference here | |
| 14:40:13 | bauzas | looks like we side-tracked the discussion | |
| 14:40:36 | bauzas | if that's about providing a NUMA support in placement, that's something we'll do in Unicorn | |
| 14:40:54 | bauzas | because we understand the concerns | |
| 14:41:09 | alex_xu | stephenfin: the imply an implicit guest NUMA and NUMA affinity are different thing | |
| 14:41:11 | bauzas | my PMs can pay for a nickel if that's the problem | |
| 14:41:28 | sean-k-mooney | alex_xu: not currently | |
| 14:41:34 | sean-k-mooney | form a nova point of view | |
| 14:41:38 | stephenfin | alex_xu: only for VPMEM though | |
| 14:41:43 | sean-k-mooney | but they woudl be with vpmem | |
| 14:42:04 | sean-k-mooney | so that break operators understand of what we men when we say there is an implcit numa toplogy | |
| 14:42:26 | sean-k-mooney | alex_xu: we can crate a numa toplogy as a impmeenation detail | |
| 14:42:53 | sean-k-mooney | but we should never telll people we are because they will assume it works the same way as the other cases we do that | |
| 14:44:40 | sean-k-mooney | alex_xu: just to clarify we use teh terms implcit and explcit numa topoyg in downstream docs and in bug reports and customer expect an implict numa toplogy to be the same as and expclti one. that is the only reason i bring up docs/release notes | |
| 14:45:23 | mriedem | here is an idea, defer vpmems since you're still arguing about designs 1 day before FF and just sort it out in Ussuri :) | |
| 14:45:31 | mriedem | rather than defer and promise to fix things later | |
| 14:45:54 | alex_xu | mriedem: come on... | |
| 14:46:53 | sean-k-mooney | as i said lest just follow the spec and use the code as it is | |
| 14:46:56 | alex_xu | mriedem: so that is works a later fix? | |
| 14:47:23 | sean-k-mooney | alex_xu: we would need to change qemu/libvirt to not do what the current code does | |
| 14:47:38 | mriedem | alex_xu: i'm not reviewing the series nor am i following the argument here, but i know when we say "we'll hack it in x and fix it for real in x+1" that x+1 rarely happens if ever (x+5) | |
| 14:47:40 | sean-k-mooney | or require a realy numa toplogy with affity | |
| 14:49:57 | mriedem | bauzas: you know you already have head count to work on numa in placement support for nova in ussuri? | |
| 14:50:50 | bauzas | mriedem: if I don't have other things like internal crisises, then yes | |
| 14:50:59 | bauzas | or escalations if you prefer | |
| 14:51:00 | sean-k-mooney | at this point not having it block so much of our roadmap since we cant do it the non plamcent way i dont see we have any choice but to dedicate headcount to it | |
| 14:51:29 | mriedem | bauzas: so that means no | |
| 14:51:41 | bauzas | sean-k-mooney: yeah and I went out of steam trying to find some other way for vGPU affinity | |
| 14:52:16 | mriedem | let's just not bullshit ourselves here is all i'm saying | |
| 14:52:21 | bauzas | mriedem: nah, it just means I'm telling the truth : we can't continue implementing NUMA related features without having NUMA support in placement | |
| 14:52:30 | mriedem | sure you can! | |
| 14:52:36 | bauzas | and that's something I'll make it clea to who it could be | |
| 14:52:39 | mriedem | stephenfin doesn't see why ont | |
| 14:52:40 | mriedem | *not | |
| 14:52:47 | stephenfin | jaypipes | |
| 14:52:54 | stephenfin | that was the main reason why, in the past :) | |
| 14:52:56 | sean-k-mooney | no not jaypies | |
| 14:53:04 | mriedem | god rest his soul | |
| 14:53:08 | stephenfin | placement will solve <X> | |
| 14:53:08 | stephenfin | RIP | |
| 14:53:11 | sean-k-mooney | we can implemntate we jsut cant get it merged | |
| 14:53:50 | sean-k-mooney | anyone here if his house/partner/dogs were safe after the storm | |
| 14:53:53 | mriedem | so what's the actual decision that's blocking the pmems stuff at this point? whether numa is implied or not? | |
| 14:54:27 | stephenfin | by specifying VPMEM, you get an implicit NUMA topology but that implicit NUMA topology works differently to every other implicit NUMA topology | |
| 14:54:39 | mriedem | differently how | |
| 14:54:41 | dansmith | that's BS | |
| 14:54:44 | dansmith | don't do that | |
| 14:54:54 | alex_xu | dansmith: you said yes in PTG :) | |
| 14:54:57 | sean-k-mooney | it does not give numa affinity fo the cpus and memory | |
| 14:55:02 | stephenfin | normally, each guest NUMA node is mapped to a unique host NUMA node | |
| 14:55:07 | stephenfin | with this, they'll float | |
| 14:55:10 | sean-k-mooney | so its not the same as hw:numa_nodes=1 | |
| 14:55:15 | dansmith | alex_xu: sean-k-mooney said I said that too, but I don't agree that I did | |
| 14:55:34 | stephenfin | so the guest has a NUMA topology but it's literally there to work around a quirk in libvirt | |
| 14:55:38 | dansmith | alex_xu: I know that because I'm quite sure I didn't say anything positive about vpmems | |
| 14:55:52 | sean-k-mooney | dansmith: it proably was not you but apprently it was form redhat an i know stephn and i relly hate that idea | |
| 14:55:58 | alex_xu | dansmith: haha | |
| 14:56:15 | stephenfin | I really don't want to blow something out of the water, but I feel by doing this we're really just masking a problem with libvirt (or QEMU) | |
| 14:56:35 | stephenfin | and we're also creating yet another special case that we'll have to clean up later | |
| 14:57:04 | mriedem | so with pmem do you have to have an implicit numa topology? | |
| 14:57:12 | stephenfin | yup, see here | |
| 14:57:26 | stephenfin | https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 | |
| 14:57:28 | mriedem | i mean, will libvirt fail to boot the guest w/o numa defined in the xml? | |
| 14:57:31 | stephenfin | yup | |
| 14:57:32 | mriedem | ah ok | |
| 14:57:34 | alex_xu | sean-k-mooney: stephenfin ok, let me try to remove that, maybe it is super fast and simple than our discussion :) | |
| 14:57:50 | mriedem | alex_xu: remove what? qemu code? | |
| 14:57:58 | stephenfin | alex_xu: Remove the implicit NUMA topology? Doesn't look like that's an option. See the above link | |
| 14:58:03 | mriedem | that is likely years before it hits LTS | |
| 14:58:33 | stephenfin | libvirt is telling us we need a NUMA topology, so we must give it a NUMA topology | |
| 14:58:46 | alex_xu | stephenfin: add numa topology explicityly | |
| 14:58:54 | mriedem | ok so if qemu requires implicit numa, then our options are hack it and create a unicorn (bad for nova), change qemu and defer the feature (bad for intel), or do numa affinity as usual in nova, | |
| 14:58:55 | mriedem | right? | |
| 14:58:55 | bauzas | oh gosh | |
| 14:59:14 | stephenfin | mriedem: Right. But | |
| 14:59:17 | sean-k-mooney | mriedem: yes | |
| 14:59:21 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | |
| 14:59:39 | stephenfin | the reason they avoided doing it that way was because they wanted to avoid doing more stuff in NUMATopologyFilter | |
| 14:59:42 | dansmith | artom: commented on the top numa functional patch | |
| 14:59:46 | bauzas | ... and all Hatters technically need to bail out for attending some meeting :) | |
| 14:59:56 | dansmith | bauzas: not all of them :) | |
| 15:00:12 | artom | dansmith, ack - I'm still trying to write words that make sense for https://review.opendev.org/#/c/640021/50/nova/conductor/tasks/live_migrate.py@179 | |
| 15:00:27 | sean-k-mooney | dansmith: can i join your team to not do numa stuff. it seam simpler | |
| 15:00:27 | mriedem | stephenfin: would the numa topo filter need to look at pmems or something? or just the hardware module that the filter calls? | |
| 15:00:27 | alex_xu | stephenfin: emm...no, that isn't the reason | |
| 15:00:30 | bauzas | dansmith: heh, so true | |
| 15:00:38 | artom | It's tricky because... just because we found some old computes in the cell, doesn't mean the source and dest are old | |
| 15:00:38 | stephenfin | to be able to do NUMA affinity properly, we'll need to provide information about the VPMEM devices available in each node as part of the NUMATopology/NUMACell objects | |
| 15:00:41 | dansmith | sean-k-mooney: heh | |
| 15:00:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Skip querying resource request if no qos port https://review.opendev.org/681513 | |
| 15:00:46 | artom | So I want to word around that | |
| 15:00:53 | dansmith | artom: ack | |
| 15:01:30 | mriedem | artom: the code isn't granular to just the source and selected dest - where you're checking we don't have a dest yet in some cases, | |
| 15:01:37 | mriedem | so i'm not sure why that would matter for the reno | |