| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-26 | |||
| 13:40:01 | mriedem | https://docs.openstack.org/nova/latest/user/block-device-mapping.html#faqs | |
| 13:41:11 | kashyap | Ah, interesting | |
| 13:41:28 | kashyap | mriedem: Yeah, I could actually add some more - expand it to cover the other two modes | |
| 13:41:36 | kashyap | Because, this is going to become a question for more and more | |
| 13:41:56 | kashyap | As they slowly realize and start upgrading microcodes ... | |
| 13:42:58 | kashyap | ... "hey, live migration is broken, what now? :D" | |
| 13:45:40 | kashyap | mriedem: A quick q. on above -- since we're so close up to GA, is changing such defaults frowned upon? (Have a gander at the commit message / release note) | |
| 13:45:51 | efried | mriedem: ack, are we waiting on anything else to cut rc1 today? | |
| 13:47:06 | kashyap | In short: it's a legit issue, also fixed in the kernel. But relaxing the 'strict' default in Nova will definitely avoid surprises like: https://lkml.org/lkml/2018/7/24/152 ("VM boot failure on nodes not having DMA32 zone") | |
| 13:50:47 | mriedem | kashyap: i can't really speak to that change but yeah that's pretty non-trivial for the day of RC1 | |
| 13:50:53 | mriedem | efried: i don't think so | |
| 13:51:20 | efried | ight, I'll go (update and) ack the release patch I guess. | |
| 13:51:23 | mriedem | kashyap: also, "fixed in the kernel" might be something for rhosp but could be years for someone using suse or canonical openstack to pick up i'd think | |
| 13:51:28 | mriedem | efried: i already did | |
| 13:51:35 | mriedem | update htat is | |
| 13:51:37 | mriedem | just needs an ack | |
| 13:51:39 | efried | coo | |
| 13:52:35 | efried | done | |
| 13:52:37 | kashyap | mriedem: I'm not one of those who think "downstream first" ;-) When I say "fixed in the kernel" - it's in 4.19, and many of those said distros will backport it | |
| 13:52:46 | kashyap | It's a super small, isloated change | |
| 13:54:48 | kashyap | mriedem: Okay, ACK - for the RC1. But FWIW, the change is quite _safe_ and self-contained: and makes Nova instances work in more NUMA setups | |
| 13:55:41 | efried | kashyap: If you wanted to be super safe, you could make a [workaround] conf opt, defaulting to "prefer", to allow consumers to revert to "strict" if they want. | |
| 13:56:05 | efried | deprecate it immediately (assuming we don't want to continue to allow it) and remove it in U. | |
| 13:56:44 | kashyap | (Also, to preempt a thought: no, it's not "urgent for RHOS". I just think since Train also has NUMA-aware live migration; this change only helps. I did a crap load background reading on this, to understand the issue) | |
| 13:56:53 | kashyap | efried: Ah, good point on 'workarounds' | |
| 13:57:59 | kashyap | I'll mull on it today; stephenfin - might want to have a look at the above, it's in your wheelhouse - https://review.opendev.org/#/c/684375/ ("libvirt: Use the `preferred` memory allocation mode for NUMA | |
| 13:58:03 | kashyap | ") | |
| 13:58:13 | kashyap | efried: Thanks for the input. | |
| 13:58:26 | efried | yahyoubetcha | |
| 13:58:56 | efried | kashyap: in any case I would think you would want a nova lp bug for that guy. | |
| 13:59:16 | efried | (if that hasn't already been said) | |
| 13:59:39 | kashyap | efried: Yes, yes. Already said; will be filing | |
| 13:59:42 | mriedem | +1 on the workarounds idea, | |
| 13:59:51 | kashyap | I have a v2 sitting; and I wrote a long explanatory comment on _why_ we're doing that change | |
| 13:59:52 | mriedem | that also allows people to opt out if they dont' have the kernel fix | |
| 14:07:46 | dansmith | efried: kashyap: I know we've strayed a bit, but I always wanted the workarounds to be booleans that all default to False, opting into different behavior, not multi-value tunables | |
| 14:08:04 | dansmith | efried: kashyap: so enable_old_memory_somethingorother=True if they need the old behavior or whatever it is | |
| 14:08:37 | dansmith | setting some regular-looking conf tunable to some value is less like a workaround and more like ... config | |
| 14:09:17 | efried | Yeah, I'm not totally convinced this should be a [workaround] in the first place -- perhaps it should be an actual conf opt. | |
| 14:10:11 | dansmith | if it's actually a conf thing (setting to some value) then not a workaround, IMHO | |
| 14:10:30 | dansmith | but if it's "I don't have a modified kernel to fix X" then that can be true/false | |
| 14:11:13 | efried | Either way, the opt is for someone who understands the issue, and (IMO) it's an unnecessary extra step making them understand how True/False maps to "prefer"/"strict". But meh, not going to put in my mouthguard for this one. | |
| 14:11:52 | efried | I don't understand enough about the issue to know whether there's a chance this will need to be worked around at all / some nontrivial percentage of the time or for how long. | |
| 14:11:58 | dansmith | sounds like it's config and not really a workaround then, which is fine | |
| 14:14:34 | bauzas | gibi: /me scratches his head when he sees a lot of good helper methods https://github.com/openstack/nova/blob/master/nova/tests/functional/integrated_helpers.py#L582 and the likes | |
| 14:15:38 | bauzas | gibi: I'll passing a new revision today or tomorrow for https://review.opendev.org/#/c/670112 but please tell me whether we could use the above $ | |
| 14:15:39 | bauzas | ^ | |
| 14:31:50 | kashyap | dansmith: Hiya, just back. Reading your / Eric's remarks | |
| 14:32:42 | kashyap | dansmith: Yeah, I agree with your intention on a workaroud being: "a bool that makes it possible to opt-in to behaviour X". | |
| 14:33:35 | kashyap | efried: And yes, I'm still not convinced either to have a workaroud yet; however, making _everything_ configurable is not desirable either. "Death by a thousand config cuts" | |
| 14:36:07 | gibi | bauzas: I think you can use assertRequestMatchesUsage and the rest of the helpers. There are some limitations documented there about the structure though | |
| 14:36:13 | kashyap | efried: On the issue, the tl;dr is: today, if you boot a Nova instance on a NUMA host without a "DMA32" zone (one of the zones into which the kernel divides the memory) on the NUMA node, then your instance hard-fails to boot. | |
| 14:36:27 | kashyap | (And you can detect this "DMA32" thing by running `grep DMA /proc/zoneinfo` on the host.) | |
| 14:37:41 | kashyap | efried: And no, please remove that mouthguard - you _do_ make a good point: for someone to understand the mapping of bool to 'preferred'/'strict', it's needless gory details | |
| 14:38:11 | kashyap | I'll mull on it more and comment on the change. | |
| 14:38:23 | kashyap | Thanks for the input, both. | |
| 14:42:07 | mriedem | oh donnyd donnyd donnyd | |
| 14:42:12 | mriedem | you've opened a can of worms | |
| 14:42:29 | donnyd | Is it a good or bad can? | |
| 14:42:50 | donnyd | like fresh worms or worms that have been sitting in the sun for 3 weeks | |
| 14:44:11 | mriedem | moldy worms | |
| 14:44:33 | mriedem | i, of course, can't help but take that bait | |
| 14:44:38 | donnyd | LOL | |
| 14:45:11 | donnyd | Well they are just observations.. not saying anything is right, wrong.. indifferent.. | |
| 14:46:56 | mriedem | it's fun to see how the sausage gets made though huh | |
| 14:47:25 | donnyd | its quite interesting from my perspective - i have been building these things for quite a long time | |
| 14:47:27 | mriedem | fwiw every closed source project i worked on before openstack was also a clusterf*ck | |
| 14:47:46 | donnyd | openstack isn't a charlie foxtrot. | |
| 14:47:46 | mriedem | come release time anyway | |
| 14:47:56 | donnyd | I don't think that is the case at all... | |
| 14:48:04 | mriedem | the thing i mostly enjoy is the foundation marketing that says "20 release ON TIME" | |
| 14:48:16 | mriedem | *releases | |
| 14:48:19 | donnyd | I just think humans have a tendency to delay until they cannot anymore | |
| 14:48:31 | mriedem | yup, i noted that in my reply | |
| 14:48:34 | mriedem | and that's our MO here as well | |
| 14:49:37 | mriedem | and the squeeky wheel gets the grease... | |
| 14:49:51 | donnyd | that is a true story in literally all things | |
| 14:51:08 | mriedem | i've left some squeeky hinges in my house go ungreased out of spite | |
| 14:51:17 | donnyd | I also think nobody consumes the milestones because they are perceived to be "beta" | |
| 14:51:22 | mriedem | "gdi i *just* greased you" | |
| 14:52:03 | donnyd | and maybe changing around the perception would increase the consumption | |
| 14:52:12 | mriedem | at this point in the game i don't think that's likely | |
| 14:52:45 | mriedem | if anything, | |
| 14:52:54 | mriedem | people have wanted to consume at a much slower pace as time as gone on | |
| 14:53:03 | mriedem | hence all the demands for LTS branches and stuff | |
| 14:53:12 | donnyd | hence the 9 month major cycle | |
| 14:53:17 | donnyd | or even 12 month | |
| 14:53:23 | mriedem | blech | |
| 14:53:33 | mriedem | we'll just wait until the last minute to cram even less stuff in :) | |
| 14:53:55 | mriedem | and, | |
| 14:54:06 | mriedem | it will be double the shit that someone has to consumer and grok from the release notes when they do | |
| 14:54:12 | mriedem | *consume | |
| 14:54:48 | donnyd | how is that different from say an ubuntu or centos release though? | |
| 14:55:19 | donnyd | centos/rhel | |
| 14:56:08 | donnyd | Also thank you for replying | |
| 14:56:28 | mriedem | i can't say how those distro dev cycles are different, | |
| 14:56:33 | mriedem | but i imagine there much more strictly controlled | |
| 14:56:36 | mriedem | *they're | |
| 14:57:18 | mriedem | and when rhel i'm paying a support license so i expect more rigor before the thing is released than just "here it is, here are the release notes, have fun" | |
| 14:57:21 | mriedem | *with rhel | |