| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-11 | |||
| 14:34:40 | sean-k-mooney | righ with the special case code it wont | |
| 14:34:42 | alex_xu | sean-k-mooney: we only implicit to create one in libvirt driver | |
| 14:34:46 | sean-k-mooney | so that is why it works | |
| 14:34:51 | mloza | Is there an option in nova for instance timeout to ERROR state when a compute node fails? | |
| 14:35:11 | sean-k-mooney | but was awsering your question about what does hw:numa_nodes=1 do | |
| 14:35:36 | sean-k-mooney | all other implcit numa toploges act exactly like hw:numa_nodes=1 | |
| 14:35:44 | alex_xu | yes | |
| 14:35:50 | sean-k-mooney | hw:pmem is not the same as hw:numa_nodes=1 | |
| 14:36:11 | stephenfin | okay, I hate the idea of the implicit NUMA affinity when VPMEM is specified working differently from implicit VPMEM for other things (hugepages, CPU pinning), but I agree it doesn't make sense to work that way if we can't provide VPMEM NUMA affinity | |
| 14:36:18 | sean-k-mooney | and we should not imply it is in any docs, comment or release notes | |
| 14:36:35 | stephenfin | I'm going to have a look to see how hard tacking on proper NUMA affinity is | |
| 14:37:02 | sean-k-mooney | stephenfin: its not hard it has been blocked because its not plamcent native. | |
| 14:37:04 | stephenfin | I don't buy the argument that we shouldn't add more things to the NUMATopology object. NUMA in placement isn't ready so we've to use what we got | |
| 14:37:14 | stephenfin | put politely, that's nonsense :) | |
| 14:38:03 | sean-k-mooney | its been nonsense for 3 years but that is not going to change. so we should enbrace doing it the plamcnet way. in U to give mriedem a nickel | |
| 14:38:10 | stephenfin | gimme a half hour to check out how tough this would be to do. If it's terrible, I'll hold my tongue | |
| 14:39:00 | alex_xu | stephenfin: yes, it will be very easy to support numa affinity once we have numa in placement | |
| 14:39:09 | sean-k-mooney | its 1 field in the host numa toplogy blob and a tweek to the numa toplogy filter to match on it in the numa_fit_instace_to_host funciton | |
| 14:39:12 | alex_xu | stephenfin: just like vgpu, the first step no numa affinity | |
| 14:39:33 | sean-k-mooney | alex_xu: its very easy to support it without placmeent but not polically viable | |
| 14:39:45 | stephenfin | vGPU didn't imply an implicit guest NUMA topology though, that's the key difference here | |
| 14:40:13 | bauzas | looks like we side-tracked the discussion | |
| 14:40:36 | bauzas | if that's about providing a NUMA support in placement, that's something we'll do in Unicorn | |
| 14:40:54 | bauzas | because we understand the concerns | |
| 14:41:09 | alex_xu | stephenfin: the imply an implicit guest NUMA and NUMA affinity are different thing | |
| 14:41:11 | bauzas | my PMs can pay for a nickel if that's the problem | |
| 14:41:28 | sean-k-mooney | alex_xu: not currently | |
| 14:41:34 | sean-k-mooney | form a nova point of view | |
| 14:41:38 | stephenfin | alex_xu: only for VPMEM though | |
| 14:41:43 | sean-k-mooney | but they woudl be with vpmem | |
| 14:42:04 | sean-k-mooney | so that break operators understand of what we men when we say there is an implcit numa toplogy | |
| 14:42:26 | sean-k-mooney | alex_xu: we can crate a numa toplogy as a impmeenation detail | |
| 14:42:53 | sean-k-mooney | but we should never telll people we are because they will assume it works the same way as the other cases we do that | |
| 14:44:40 | sean-k-mooney | alex_xu: just to clarify we use teh terms implcit and explcit numa topoyg in downstream docs and in bug reports and customer expect an implict numa toplogy to be the same as and expclti one. that is the only reason i bring up docs/release notes | |
| 14:45:23 | mriedem | here is an idea, defer vpmems since you're still arguing about designs 1 day before FF and just sort it out in Ussuri :) | |
| 14:45:31 | mriedem | rather than defer and promise to fix things later | |
| 14:45:54 | alex_xu | mriedem: come on... | |
| 14:46:53 | sean-k-mooney | as i said lest just follow the spec and use the code as it is | |
| 14:46:56 | alex_xu | mriedem: so that is works a later fix? | |
| 14:47:23 | sean-k-mooney | alex_xu: we would need to change qemu/libvirt to not do what the current code does | |
| 14:47:38 | mriedem | alex_xu: i'm not reviewing the series nor am i following the argument here, but i know when we say "we'll hack it in x and fix it for real in x+1" that x+1 rarely happens if ever (x+5) | |
| 14:47:40 | sean-k-mooney | or require a realy numa toplogy with affity | |
| 14:49:57 | mriedem | bauzas: you know you already have head count to work on numa in placement support for nova in ussuri? | |
| 14:50:50 | bauzas | mriedem: if I don't have other things like internal crisises, then yes | |
| 14:50:59 | bauzas | or escalations if you prefer | |
| 14:51:00 | sean-k-mooney | at this point not having it block so much of our roadmap since we cant do it the non plamcent way i dont see we have any choice but to dedicate headcount to it | |
| 14:51:29 | mriedem | bauzas: so that means no | |
| 14:51:41 | bauzas | sean-k-mooney: yeah and I went out of steam trying to find some other way for vGPU affinity | |
| 14:52:16 | mriedem | let's just not bullshit ourselves here is all i'm saying | |
| 14:52:21 | bauzas | mriedem: nah, it just means I'm telling the truth : we can't continue implementing NUMA related features without having NUMA support in placement | |
| 14:52:30 | mriedem | sure you can! | |
| 14:52:36 | bauzas | and that's something I'll make it clea to who it could be | |
| 14:52:39 | mriedem | stephenfin doesn't see why ont | |
| 14:52:40 | mriedem | *not | |
| 14:52:47 | stephenfin | jaypipes | |
| 14:52:54 | stephenfin | that was the main reason why, in the past :) | |
| 14:52:56 | sean-k-mooney | no not jaypies | |
| 14:53:04 | mriedem | god rest his soul | |
| 14:53:08 | stephenfin | RIP | |
| 14:53:08 | stephenfin | placement will solve <X> | |
| 14:53:11 | sean-k-mooney | we can implemntate we jsut cant get it merged | |
| 14:53:50 | sean-k-mooney | anyone here if his house/partner/dogs were safe after the storm | |
| 14:53:53 | mriedem | so what's the actual decision that's blocking the pmems stuff at this point? whether numa is implied or not? | |
| 14:54:27 | stephenfin | by specifying VPMEM, you get an implicit NUMA topology but that implicit NUMA topology works differently to every other implicit NUMA topology | |
| 14:54:39 | mriedem | differently how | |
| 14:54:41 | dansmith | that's BS | |
| 14:54:44 | dansmith | don't do that | |
| 14:54:54 | alex_xu | dansmith: you said yes in PTG :) | |
| 14:54:57 | sean-k-mooney | it does not give numa affinity fo the cpus and memory | |
| 14:55:02 | stephenfin | normally, each guest NUMA node is mapped to a unique host NUMA node | |
| 14:55:07 | stephenfin | with this, they'll float | |
| 14:55:10 | sean-k-mooney | so its not the same as hw:numa_nodes=1 | |
| 14:55:15 | dansmith | alex_xu: sean-k-mooney said I said that too, but I don't agree that I did | |
| 14:55:34 | stephenfin | so the guest has a NUMA topology but it's literally there to work around a quirk in libvirt | |
| 14:55:38 | dansmith | alex_xu: I know that because I'm quite sure I didn't say anything positive about vpmems | |
| 14:55:52 | sean-k-mooney | dansmith: it proably was not you but apprently it was form redhat an i know stephn and i relly hate that idea | |
| 14:55:58 | alex_xu | dansmith: haha | |
| 14:56:15 | stephenfin | I really don't want to blow something out of the water, but I feel by doing this we're really just masking a problem with libvirt (or QEMU) | |
| 14:56:35 | stephenfin | and we're also creating yet another special case that we'll have to clean up later | |
| 14:57:04 | mriedem | so with pmem do you have to have an implicit numa topology? | |
| 14:57:12 | stephenfin | yup, see here | |
| 14:57:26 | stephenfin | https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_domain.c#L11604-L11615 | |
| 14:57:28 | mriedem | i mean, will libvirt fail to boot the guest w/o numa defined in the xml? | |
| 14:57:31 | stephenfin | yup | |
| 14:57:32 | mriedem | ah ok | |
| 14:57:34 | alex_xu | sean-k-mooney: stephenfin ok, let me try to remove that, maybe it is super fast and simple than our discussion :) | |
| 14:57:50 | mriedem | alex_xu: remove what? qemu code? | |
| 14:57:58 | stephenfin | alex_xu: Remove the implicit NUMA topology? Doesn't look like that's an option. See the above link | |
| 14:58:03 | mriedem | that is likely years before it hits LTS | |
| 14:58:33 | stephenfin | libvirt is telling us we need a NUMA topology, so we must give it a NUMA topology | |
| 14:58:46 | alex_xu | stephenfin: add numa topology explicityly | |
| 14:58:54 | mriedem | ok so if qemu requires implicit numa, then our options are hack it and create a unicorn (bad for nova), change qemu and defer the feature (bad for intel), or do numa affinity as usual in nova, | |
| 14:58:55 | bauzas | oh gosh | |
| 14:58:55 | mriedem | right? | |
| 14:59:14 | stephenfin | mriedem: Right. But | |
| 14:59:17 | sean-k-mooney | mriedem: yes | |
| 14:59:21 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP Include error 'details' in dynamic vendordata log https://review.opendev.org/681329 | |
| 14:59:39 | stephenfin | the reason they avoided doing it that way was because they wanted to avoid doing more stuff in NUMATopologyFilter | |
| 14:59:42 | dansmith | artom: commented on the top numa functional patch | |
| 14:59:46 | bauzas | ... and all Hatters technically need to bail out for attending some meeting :) | |