| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-26 | |||
| 09:55:17 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020 | |
| 09:56:29 | kashyap | ozzzo: Also responded on e-mail | |
| 10:00:53 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020 | |
| 11:27:55 | openstackgerrit | Merged openstack/nova master: doc: attaching virtual persistent memory to guests https://review.opendev.org/680300 | |
| 11:28:02 | openstackgerrit | Merged openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327 | |
| 11:41:14 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999 | |
| 11:46:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Functional reproduction for bug 1845291 https://review.opendev.org/684545 | |
| 11:46:30 | openstack | bug 1845291 in OpenStack Compute (nova) "migration is not recheduled if the server originally booted with --availability-zone |
|
| 11:46:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reset forced_destination before migration at a proper time https://review.opendev.org/684546 | |
| 13:12:40 | mriedem | efried: we might want to consider https://bugs.launchpad.net/nova/+bug/1845243 as train-rc-potential for rc2 since the regression was introduced in train, it was also backported and released on stein and rocy | |
| 13:12:40 | openstack | Launchpad bug 1845243 in OpenStack Compute (nova) stein "Nested 'path' query param in console URL breaks serialproxy" [High,Confirmed] | |
| 13:12:41 | mriedem | *rocky | |
| 13:13:26 | mriedem | so i think we have 2 potential bugs for rc2 https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential but i think we could still release rc1 today | |
| 13:16:48 | mriedem | i've updated the rc1 release patch https://review.opendev.org/#/c/684086/ | |
| 13:17:53 | mriedem | yaawang: were you able to test this fix for the numa live migration bug you reported? https://review.opendev.org/#/c/684409/ | |
| 13:28:21 | kashyap | sean-k-mooney: efried This was my answer to ozzzo, in an "FAQ form" :D - https://kashyapc.fedorapeople.org/CPU-microcode-host-passthrough-and-live-migration.txt | |
| 13:28:41 | kashyap | mriedem: ^ (But you might already know that answer.) | |
| 13:32:24 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Ignore sqla-migrate inspect.getargspec deprecation warnings on py36 https://review.opendev.org/685072 | |
| 13:39:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1841481 https://review.opendev.org/684840 | |
| 13:39:18 | openstack | bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] https://launchpad.net/bugs/1841481 - Assigned to Matt Riedemann (mriedem) | |
| 13:39:18 | mriedem | kashyap: lgtm, i remember you talking about microcode in your presentation | |
| 13:39:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Clear RT.compute_nodes entry if re-balance update fails https://review.opendev.org/684849 | |
| 13:39:38 | mriedem | if a FAQs style thing is useful for that docs page i'm fine with adding one | |
| 13:39:47 | mriedem | we recently did that for the bdms docs | |
| 13:40:01 | mriedem | https://docs.openstack.org/nova/latest/user/block-device-mapping.html#faqs | |
| 13:41:11 | kashyap | Ah, interesting | |
| 13:41:28 | kashyap | mriedem: Yeah, I could actually add some more - expand it to cover the other two modes | |
| 13:41:36 | kashyap | Because, this is going to become a question for more and more | |
| 13:41:56 | kashyap | As they slowly realize and start upgrading microcodes ... | |
| 13:42:58 | kashyap | ... "hey, live migration is broken, what now? :D" | |
| 13:45:40 | kashyap | mriedem: A quick q. on above -- since we're so close up to GA, is changing such defaults frowned upon? (Have a gander at the commit message / release note) | |
| 13:45:51 | efried | mriedem: ack, are we waiting on anything else to cut rc1 today? | |
| 13:47:06 | kashyap | In short: it's a legit issue, also fixed in the kernel. But relaxing the 'strict' default in Nova will definitely avoid surprises like: https://lkml.org/lkml/2018/7/24/152 ("VM boot failure on nodes not having DMA32 zone") | |
| 13:50:47 | mriedem | kashyap: i can't really speak to that change but yeah that's pretty non-trivial for the day of RC1 | |
| 13:50:53 | mriedem | efried: i don't think so | |
| 13:51:20 | efried | ight, I'll go (update and) ack the release patch I guess. | |
| 13:51:23 | mriedem | kashyap: also, "fixed in the kernel" might be something for rhosp but could be years for someone using suse or canonical openstack to pick up i'd think | |
| 13:51:28 | mriedem | efried: i already did | |
| 13:51:35 | mriedem | update htat is | |
| 13:51:37 | mriedem | just needs an ack | |
| 13:51:39 | efried | coo | |
| 13:52:35 | efried | done | |
| 13:52:37 | kashyap | mriedem: I'm not one of those who think "downstream first" ;-) When I say "fixed in the kernel" - it's in 4.19, and many of those said distros will backport it | |
| 13:52:46 | kashyap | It's a super small, isloated change | |
| 13:54:48 | kashyap | mriedem: Okay, ACK - for the RC1. But FWIW, the change is quite _safe_ and self-contained: and makes Nova instances work in more NUMA setups | |
| 13:55:41 | efried | kashyap: If you wanted to be super safe, you could make a [workaround] conf opt, defaulting to "prefer", to allow consumers to revert to "strict" if they want. | |
| 13:56:05 | efried | deprecate it immediately (assuming we don't want to continue to allow it) and remove it in U. | |
| 13:56:44 | kashyap | (Also, to preempt a thought: no, it's not "urgent for RHOS". I just think since Train also has NUMA-aware live migration; this change only helps. I did a crap load background reading on this, to understand the issue) | |
| 13:56:53 | kashyap | efried: Ah, good point on 'workarounds' | |
| 13:57:59 | kashyap | I'll mull on it today; stephenfin - might want to have a look at the above, it's in your wheelhouse - https://review.opendev.org/#/c/684375/ ("libvirt: Use the `preferred` memory allocation mode for NUMA | |
| 13:58:03 | kashyap | ") | |
| 13:58:13 | kashyap | efried: Thanks for the input. | |
| 13:58:26 | efried | yahyoubetcha | |
| 13:58:56 | efried | kashyap: in any case I would think you would want a nova lp bug for that guy. | |
| 13:59:16 | efried | (if that hasn't already been said) | |
| 13:59:39 | kashyap | efried: Yes, yes. Already said; will be filing | |
| 13:59:42 | mriedem | +1 on the workarounds idea, | |
| 13:59:51 | kashyap | I have a v2 sitting; and I wrote a long explanatory comment on _why_ we're doing that change | |
| 13:59:52 | mriedem | that also allows people to opt out if they dont' have the kernel fix | |
| 14:07:46 | dansmith | efried: kashyap: I know we've strayed a bit, but I always wanted the workarounds to be booleans that all default to False, opting into different behavior, not multi-value tunables | |
| 14:08:04 | dansmith | efried: kashyap: so enable_old_memory_somethingorother=True if they need the old behavior or whatever it is | |
| 14:08:37 | dansmith | setting some regular-looking conf tunable to some value is less like a workaround and more like ... config | |
| 14:09:17 | efried | Yeah, I'm not totally convinced this should be a [workaround] in the first place -- perhaps it should be an actual conf opt. | |
| 14:10:11 | dansmith | if it's actually a conf thing (setting to some value) then not a workaround, IMHO | |
| 14:10:30 | dansmith | but if it's "I don't have a modified kernel to fix X" then that can be true/false | |
| 14:11:13 | efried | Either way, the opt is for someone who understands the issue, and (IMO) it's an unnecessary extra step making them understand how True/False maps to "prefer"/"strict". But meh, not going to put in my mouthguard for this one. | |
| 14:11:52 | efried | I don't understand enough about the issue to know whether there's a chance this will need to be worked around at all / some nontrivial percentage of the time or for how long. | |
| 14:11:58 | dansmith | sounds like it's config and not really a workaround then, which is fine | |
| 14:14:34 | bauzas | gibi: /me scratches his head when he sees a lot of good helper methods https://github.com/openstack/nova/blob/master/nova/tests/functional/integrated_helpers.py#L582 and the likes | |
| 14:15:38 | bauzas | gibi: I'll passing a new revision today or tomorrow for https://review.opendev.org/#/c/670112 but please tell me whether we could use the above $ | |
| 14:15:39 | bauzas | ^ | |
| 14:31:50 | kashyap | dansmith: Hiya, just back. Reading your / Eric's remarks | |
| 14:32:42 | kashyap | dansmith: Yeah, I agree with your intention on a workaroud being: "a bool that makes it possible to opt-in to behaviour X". | |
| 14:33:35 | kashyap | efried: And yes, I'm still not convinced either to have a workaroud yet; however, making _everything_ configurable is not desirable either. "Death by a thousand config cuts" | |
| 14:36:07 | gibi | bauzas: I think you can use assertRequestMatchesUsage and the rest of the helpers. There are some limitations documented there about the structure though | |
| 14:36:13 | kashyap | efried: On the issue, the tl;dr is: today, if you boot a Nova instance on a NUMA host without a "DMA32" zone (one of the zones into which the kernel divides the memory) on the NUMA node, then your instance hard-fails to boot. | |
| 14:36:27 | kashyap | (And you can detect this "DMA32" thing by running `grep DMA /proc/zoneinfo` on the host.) | |
| 14:37:41 | kashyap | efried: And no, please remove that mouthguard - you _do_ make a good point: for someone to understand the mapping of bool to 'preferred'/'strict', it's needless gory details | |
| 14:38:11 | kashyap | I'll mull on it more and comment on the change. | |
| 14:38:23 | kashyap | Thanks for the input, both. | |
| 14:42:07 | mriedem | oh donnyd donnyd donnyd | |
| 14:42:12 | mriedem | you've opened a can of worms | |
| 14:42:29 | donnyd | Is it a good or bad can? | |
| 14:42:50 | donnyd | like fresh worms or worms that have been sitting in the sun for 3 weeks | |
| 14:44:11 | mriedem | moldy worms | |
| 14:44:33 | mriedem | i, of course, can't help but take that bait | |
| 14:44:38 | donnyd | LOL | |
| 14:45:11 | donnyd | Well they are just observations.. not saying anything is right, wrong.. indifferent.. | |
| 14:46:56 | mriedem | it's fun to see how the sausage gets made though huh | |
| 14:47:25 | donnyd | its quite interesting from my perspective - i have been building these things for quite a long time | |
| 14:47:27 | mriedem | fwiw every closed source project i worked on before openstack was also a clusterf*ck | |
| 14:47:46 | mriedem | come release time anyway | |
| 14:47:46 | donnyd | openstack isn't a charlie foxtrot. | |
| 14:47:56 | donnyd | I don't think that is the case at all... | |
| 14:48:04 | mriedem | the thing i mostly enjoy is the foundation marketing that says "20 release ON TIME" | |
| 14:48:16 | mriedem | *releases | |
| 14:48:19 | donnyd | I just think humans have a tendency to delay until they cannot anymore | |
| 14:48:31 | mriedem | yup, i noted that in my reply | |
| 14:48:34 | mriedem | and that's our MO here as well | |
| 14:49:37 | mriedem | and the squeeky wheel gets the grease... | |