| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 16:34:39 | stephenfin | and those with hugepage, but that's a self-inflicted wound | |
| 16:35:15 | stephenfin | the pinned ones need it because their cores are pinned to host cores from a specific NUMA node | |
| 16:35:42 | stephenfin | whereas unpinned instances are floating across all (enabled) host cores | |
| 16:36:01 | stephenfin | so they naturally have affinity to everything, even though we don't properly expose that information to the guest | |
| 16:37:16 | efried | does memory float too? | |
| 16:37:37 | stephenfin | if you don't provide a pagesize, yes | |
| 16:37:48 | stephenfin | but otherwise, no. that's the self-inflicted wound I talked about above | |
| 16:38:24 | efried | Then why are we modeling 4k pages under NUMA nodes? | |
| 16:38:47 | efried | Wait, *can* you pin 4k pages? | |
| 16:39:03 | stephenfin | I _think_ so, yeah | |
| 16:39:12 | stephenfin | if you use the strict mem policy | |
| 16:39:32 | stephenfin | actually, I don't think it's pinning in the traditional sense | |
| 16:39:36 | stephenfin | because they can be shared | |
| 16:40:15 | stephenfin | sorry, I'm in a meeting so I can't formulate my thoughts properly. gimme 20 | |
| 16:42:50 | efried | Ight. I don't have the depth of understanding to refute the "unpinned/floating" argument, which I was kinda advocating the other day (albeit probably without specifics). Going to need dansmith to take that on. | |
| 16:45:03 | dansmith | I think maybe he's talking about the case where you're overcommitting memory | |
| 16:47:33 | dansmith | I'm also really not an expert on the low-level details, so maybe we've gotten lost in the woods a bit, | |
| 16:49:52 | efried | tobiash: What's the word on https://review.opendev.org/#/c/572805/ ? Spec freeze is today. | |
| 16:50:03 | dansmith | I'm trying to translate what I know of reports of what people do, vs. what they would like to do, and what makes sense into what we should be doing | |
| 16:51:19 | tobiash | efried: sorry, I was busy with other other things in the meantime, I fear it has to be postponed to the next release :( | |
| 16:51:34 | efried | tobiash: Okay, thanks, I'll do that. | |
| 16:51:41 | tobiash | thanks a lot | |
| 16:59:51 | bauzas | efried: sorry, was on meeting | |
| 17:00:42 | efried | bauzas: I really just want to know what you think of my French | |
| 17:01:03 | bauzas | LOL | |
| 17:01:29 | gibi | efried: based on Tushar's comment on the spec bp/support-shared-storage-resource-provider can be deferred out from U | |
| 17:02:05 | bauzas | efried: we have some french continuous present but not really like yours | |
| 17:02:39 | efried | no, like I said, I don't see anybody ever saying or writing anything like that. | |
| 17:02:41 | bauzas | efried: anyway, I see your -1 but I intentionnally flipped the default to *not* reshape as discussed between sean-k-mooney, stephenfin and I | |
| 17:02:59 | efried | bauzas: to me, that's the crux | |
| 17:03:10 | efried | If we don't reshape by default, everything changes. | |
| 17:03:13 | bauzas | efried: because of the potentiality of the regressions we could get | |
| 17:03:15 | bauzas | efried: I know | |
| 17:03:23 | bauzas | efried: and I wanted to discuss this with you | |
| 17:03:52 | bauzas | because I'm very afraid of any potential issue we would have in Ussuri | |
| 17:03:59 | bauzas | if we flip to changing the world | |
| 17:03:59 | efried | stephenfin and dansmith need to be involved, but I think they're are on calls rn | |
| 17:04:22 | bauzas | my point is, I don't know the figure but not all clouds care about NUMA | |
| 17:04:37 | bauzas | for those clouds, I'd prefer us to not change their lifes | |
| 17:04:51 | bauzas | and pretending there will be no regressions | |
| 17:05:07 | efried | So dansmith's argument was "don't care about NUMA" does *not* mean "give me shitty performance". | |
| 17:05:24 | bauzas | on the other hand, if we allow a default to be "no-op", then we can work on the NUMA implementation seamlessly and iteratively like we did for Cells v2 | |
| 17:05:27 | efried | The [workaround] and reversible reshape gives you the way to deal with regressions. | |
| 17:05:29 | dansmith | I don't really understand.. I thought the agreement was to default the new behavior off for U, not reshape by default, let people opt-in during U and then flip the default (or remove it) for V? | |
| 17:05:55 | efried | ugh, no, if we were doing that, there would be no good motivation to hack the splitting thing in. | |
| 17:06:02 | efried | And also no motivation for operators to opt in. | |
| 17:06:07 | efried | so it would be a waste of effort. | |
| 17:06:17 | bauzas | dansmith: that's what i wrote but looks like the outcome of tuesday's discussion between you, sean-k-mooney and efried was the other way | |
| 17:06:26 | dansmith | bauzas: not that I remember | |
| 17:06:37 | dansmith | but I'll admit to being completely exhausted by this conversation | |
| 17:06:41 | bauzas | efried: dansmith: floor is your | |
| 17:06:52 | bauzas | dansmith: and tbh, me too | |
| 17:07:04 | efried | you people need to do more cardio | |
| 17:07:11 | bauzas | I do gym twice a week | |
| 17:07:16 | stephenfin | lyarwood: comments on https://review.opendev.org/#/c/706880/ | |
| 17:07:30 | bauzas | (and skiing, but that's irrelevant) | |
| 17:07:49 | bauzas | efried: anyway, my point is, | |
| 17:07:59 | stephenfin | dansmith, efried, bauzas: so, from the top | |
| 17:08:14 | bauzas | we can't ask operators to modify their configs *before* they upgrade or *before* they restart their clouds | |
| 17:08:20 | stephenfin | from what CERN are saying, they're already dividing their hosts into those for NUMA and those for not NUMA | |
| 17:08:23 | efried | nope. We're not asking that. | |
| 17:08:42 | bauzas | efried: I know, but what you propose will frighten them | |
| 17:08:52 | stephenfin | so I'm not sure why we can't do the same in placement | |
| 17:08:55 | bauzas | because, we flip to NUMA everywhere | |
| 17:09:18 | bauzas | for CERN, that would mean non-NUMA cells would be NUMA-speaking from Ussuri | |
| 17:09:26 | efried | which will happen at some point anyway. | |
| 17:09:37 | bauzas | and they would have to let them speak... what? after this | |
| 17:09:53 | bauzas | efried: I don't disagree with you, and I think this could be Victoria | |
| 17:09:59 | efried | if we default 'off', nobody is going to switch it on. Then in V (or whenever) we switch the default and have this issue. | |
| 17:10:01 | stephenfin | if you care about NUMA affinity, configure things so placement speaks NUMA, otherwise YAGNI | |
| 17:10:06 | bauzas | but not Ussuri | |
| 17:10:23 | bauzas | stephenfin: that's what I propose | |
| 17:10:31 | stephenfin | it seems so much simpler | |
| 17:11:13 | bauzas | honestly, my vision of the work to do is : | |
| 17:11:48 | bauzas | 'plumb, plumb, plumb things on one side, and mark this feature as opt-in' | |
| 17:12:33 | bauzas | in the eventuality of a very bad situation close to RC1, then we just add an 'EXPERIMENTAL' flag on the option | |
| 17:12:38 | efried | stephenfin: so in that scenario, you upgrade your control plane, and then any hw:numa*-havin flavors will simply refuse to land until you've upgraded *and* opted-in some hosts. | |
| 17:12:38 | bauzas | boom, problem solved. | |
| 17:12:52 | stephenfin | no | |
| 17:12:59 | efried | or that's what the 'fallback' query is for | |
| 17:13:12 | stephenfin | yeah, short term fallback query like we do for PCPU | |
| 17:13:24 | efried | so, still you're doing two queries and either merging the results or violating pack/spread and server affinity groups. | |
| 17:14:12 | stephenfin | with a big ass warning saying "you're using this host for NUMA instances - update configuration now or perish in a future release" | |
| 17:14:22 | stephenfin | yeah, but some hosts can choose to never opt-in | |
| 17:14:25 | efried | And you can land NUMA-aware flavors on either kind of host | |
| 17:14:25 | stephenfin | because they don't care | |
| 17:14:35 | bauzas | yeah, that's my thoughts | |
| 17:14:44 | efried | What about NUMA-agnostic flavors? Those can only land on un-upgraded or un-reshaped hosts, right? | |
| 17:14:47 | efried | So one-way segregation? | |
| 17:15:02 | stephenfin | they never boot pinned instances and their instance floats across all (enabled) host cores as before | |
| 17:15:07 | bauzas | efried: non-NUMA would stick with non-NUMA hosts | |
| 17:15:15 | stephenfin | non-NUMA or non-upgraded | |
| 17:15:19 | bauzas | right | |
| 17:15:22 | stephenfin | because we can't distinguish | |
| 17:15:38 | bauzas | correct, we just say "not those hosts" | |
| 17:15:45 | efried | well, we could distinguish if we wanted to. | |
| 17:15:49 | bauzas | thru a forbidden triat | |
| 17:16:05 | stephenfin | not without operator intervention | |
| 17:16:33 | stephenfin | the operator would have to do something to say "this host is intended to be a non-NUMA host" | |
| 17:16:39 | efried | we could make the segregation complete by simply adding a trait to (even unreshaped) U hosts. | |