| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 14:52:43 | dansmith | stephenfin: agree query arg feels more right-er | |
| 15:10:00 | efried | stephenfin: I vote qparam too. | |
| 15:10:13 | stephenfin | sweet. qparam it is | |
| 16:16:32 | gibi | stephenfin: I vote for query arg as Sean stated it is not part of the entity you actually create or modify | |
| 16:21:10 | efried | bauzas: I'm very close on the NUMA RP spec; +2 if you just flip the defaults as noted. But (despite him saying he's not blocking) I want to convince stephenfin that this is the way we should go. | |
| 16:21:46 | efried | sean-k-mooney: do you agree with my notes on the default for implicit numa nodes? | |
| 16:22:00 | efried | https://review.opendev.org/#/c/552924/20/specs/ussuri/approved/numa-topology-with-rps.rst@269 | |
| 16:24:31 | stephenfin | efried: So I'm clear, what's the objection to a "I want this host to report/not report NUMA"? | |
| 16:24:35 | stephenfin | dansmith too ^ | |
| 16:24:57 | efried | stephenfin: no objection. That's being provided. But I want the default to be "report NUMA". | |
| 16:25:39 | stephenfin | It's being provided temporarily though, not long term | |
| 16:25:44 | stephenfin | Why not do this long-term | |
| 16:25:50 | efried | ah, okay: | |
| 16:27:24 | efried | The way dansmith explained it, the only reason we don't always report NUMA and create real NUMA topologies for guests is because it's hard. But no consumer *actually* wants a guest that doesn't affine its memory to CPUs; they're taking a significant performance hit because we haven't solved this problem in nova. | |
| 16:27:38 | efried | I'm... paraphrasing. Dan was more eloquent about it. | |
| 16:27:53 | dansmith | doubtful, but yeah | |
| 16:27:56 | efried | But we have to balance that against fitting. | |
| 16:28:24 | dansmith | stephenfin: I don't want to have two "modes" for the compute service to operate in long term | |
| 16:28:47 | efried | again paraphrasing dansmith, nova explicitly disclaims the ability to fit that last VM on that last almost-full host. So if we're compromising, that's where we're compromising. | |
| 16:29:04 | stephenfin | But we could provide NUMA information to the guest. It would just match the topology of the host | |
| 16:29:13 | efried | that's exactly what we're doing. | |
| 16:29:24 | efried | the U proposal does it imperfectly, 80/20. | |
| 16:29:33 | efried | For V we can work on can_split to get that other 20% | |
| 16:29:39 | efried | and then we can remove the [workaround]. | |
| 16:30:46 | stephenfin | From the the libvirt XML perspective, yeah, but not from placement perspective | |
| 16:31:08 | efried | sorry, wha? | |
| 16:34:18 | stephenfin | The only guests that *needs* NUMA affinity are pinned instances, yeah? | |
| 16:34:39 | stephenfin | and those with hugepage, but that's a self-inflicted wound | |
| 16:35:15 | stephenfin | the pinned ones need it because their cores are pinned to host cores from a specific NUMA node | |
| 16:35:42 | stephenfin | whereas unpinned instances are floating across all (enabled) host cores | |
| 16:36:01 | stephenfin | so they naturally have affinity to everything, even though we don't properly expose that information to the guest | |
| 16:37:16 | efried | does memory float too? | |
| 16:37:37 | stephenfin | if you don't provide a pagesize, yes | |
| 16:37:48 | stephenfin | but otherwise, no. that's the self-inflicted wound I talked about above | |
| 16:38:24 | efried | Then why are we modeling 4k pages under NUMA nodes? | |
| 16:38:47 | efried | Wait, *can* you pin 4k pages? | |
| 16:39:03 | stephenfin | I _think_ so, yeah | |
| 16:39:12 | stephenfin | if you use the strict mem policy | |
| 16:39:32 | stephenfin | actually, I don't think it's pinning in the traditional sense | |
| 16:39:36 | stephenfin | because they can be shared | |
| 16:40:15 | stephenfin | sorry, I'm in a meeting so I can't formulate my thoughts properly. gimme 20 | |
| 16:42:50 | efried | Ight. I don't have the depth of understanding to refute the "unpinned/floating" argument, which I was kinda advocating the other day (albeit probably without specifics). Going to need dansmith to take that on. | |
| 16:45:03 | dansmith | I think maybe he's talking about the case where you're overcommitting memory | |
| 16:47:33 | dansmith | I'm also really not an expert on the low-level details, so maybe we've gotten lost in the woods a bit, | |
| 16:49:52 | efried | tobiash: What's the word on https://review.opendev.org/#/c/572805/ ? Spec freeze is today. | |
| 16:50:03 | dansmith | I'm trying to translate what I know of reports of what people do, vs. what they would like to do, and what makes sense into what we should be doing | |
| 16:51:19 | tobiash | efried: sorry, I was busy with other other things in the meantime, I fear it has to be postponed to the next release :( | |
| 16:51:34 | efried | tobiash: Okay, thanks, I'll do that. | |
| 16:51:41 | tobiash | thanks a lot | |
| 16:59:51 | bauzas | efried: sorry, was on meeting | |
| 17:00:42 | efried | bauzas: I really just want to know what you think of my French | |
| 17:01:03 | bauzas | LOL | |
| 17:01:29 | gibi | efried: based on Tushar's comment on the spec bp/support-shared-storage-resource-provider can be deferred out from U | |
| 17:02:05 | bauzas | efried: we have some french continuous present but not really like yours | |
| 17:02:39 | efried | no, like I said, I don't see anybody ever saying or writing anything like that. | |
| 17:02:41 | bauzas | efried: anyway, I see your -1 but I intentionnally flipped the default to *not* reshape as discussed between sean-k-mooney, stephenfin and I | |
| 17:02:59 | efried | bauzas: to me, that's the crux | |
| 17:03:10 | efried | If we don't reshape by default, everything changes. | |
| 17:03:13 | bauzas | efried: because of the potentiality of the regressions we could get | |
| 17:03:15 | bauzas | efried: I know | |
| 17:03:23 | bauzas | efried: and I wanted to discuss this with you | |
| 17:03:52 | bauzas | because I'm very afraid of any potential issue we would have in Ussuri | |
| 17:03:59 | bauzas | if we flip to changing the world | |
| 17:03:59 | efried | stephenfin and dansmith need to be involved, but I think they're are on calls rn | |
| 17:04:22 | bauzas | my point is, I don't know the figure but not all clouds care about NUMA | |
| 17:04:37 | bauzas | for those clouds, I'd prefer us to not change their lifes | |
| 17:04:51 | bauzas | and pretending there will be no regressions | |
| 17:05:07 | efried | So dansmith's argument was "don't care about NUMA" does *not* mean "give me shitty performance". | |
| 17:05:24 | bauzas | on the other hand, if we allow a default to be "no-op", then we can work on the NUMA implementation seamlessly and iteratively like we did for Cells v2 | |
| 17:05:27 | efried | The [workaround] and reversible reshape gives you the way to deal with regressions. | |
| 17:05:29 | dansmith | I don't really understand.. I thought the agreement was to default the new behavior off for U, not reshape by default, let people opt-in during U and then flip the default (or remove it) for V? | |
| 17:05:55 | efried | ugh, no, if we were doing that, there would be no good motivation to hack the splitting thing in. | |
| 17:06:02 | efried | And also no motivation for operators to opt in. | |
| 17:06:07 | efried | so it would be a waste of effort. | |
| 17:06:17 | bauzas | dansmith: that's what i wrote but looks like the outcome of tuesday's discussion between you, sean-k-mooney and efried was the other way | |
| 17:06:26 | dansmith | bauzas: not that I remember | |
| 17:06:37 | dansmith | but I'll admit to being completely exhausted by this conversation | |
| 17:06:41 | bauzas | efried: dansmith: floor is your | |
| 17:06:52 | bauzas | dansmith: and tbh, me too | |
| 17:07:04 | efried | you people need to do more cardio | |
| 17:07:11 | bauzas | I do gym twice a week | |
| 17:07:16 | stephenfin | lyarwood: comments on https://review.opendev.org/#/c/706880/ | |
| 17:07:30 | bauzas | (and skiing, but that's irrelevant) | |
| 17:07:49 | bauzas | efried: anyway, my point is, | |
| 17:07:59 | stephenfin | dansmith, efried, bauzas: so, from the top | |
| 17:08:14 | bauzas | we can't ask operators to modify their configs *before* they upgrade or *before* they restart their clouds | |
| 17:08:20 | stephenfin | from what CERN are saying, they're already dividing their hosts into those for NUMA and those for not NUMA | |
| 17:08:23 | efried | nope. We're not asking that. | |
| 17:08:42 | bauzas | efried: I know, but what you propose will frighten them | |
| 17:08:52 | stephenfin | so I'm not sure why we can't do the same in placement | |
| 17:08:55 | bauzas | because, we flip to NUMA everywhere | |
| 17:09:18 | bauzas | for CERN, that would mean non-NUMA cells would be NUMA-speaking from Ussuri | |
| 17:09:26 | efried | which will happen at some point anyway. | |
| 17:09:37 | bauzas | and they would have to let them speak... what? after this | |
| 17:09:53 | bauzas | efried: I don't disagree with you, and I think this could be Victoria | |
| 17:09:59 | efried | if we default 'off', nobody is going to switch it on. Then in V (or whenever) we switch the default and have this issue. | |
| 17:10:01 | stephenfin | if you care about NUMA affinity, configure things so placement speaks NUMA, otherwise YAGNI | |
| 17:10:06 | bauzas | but not Ussuri | |
| 17:10:23 | bauzas | stephenfin: that's what I propose | |
| 17:10:31 | stephenfin | it seems so much simpler | |