Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-22
16:58:51 bauzas I have a 7-yr old daughter just learning French grammar
16:59:12 bauzas never say English is full of silly, or I'll just compare with French
16:59:24 efried Most Americans never learn English grammar, so...
17:00:18 efried bauzas this ~= celui-çi; that ~= celui-là
17:00:35 openstackgerrit Jim Rollenhagen proposed openstack/nova master: Ironic: negotiate microversion to allow downgrade to Pike https://review.openstack.org/535786
17:00:35 bauzas this = person, that = object ?
17:00:40 efried no
17:00:49 efried just ceci vs cela
17:00:55 bauzas mmmm
17:01:10 jroll hey (again) nova friends, the ironic rolling upgrade testing is down. while we try to track down why nova-conductor segfaults after upgrading ironic (without restarting n-cond), this patch will help us work around it by allowing nova queens to work with ironic pike (which is a good thing for users anyway). reviews would be super helpful, thank you :) https://review.openstack.org/#/c/535786/
17:01:50 efried bauzas Mais je peux voir que "ça" peut servir pour les deux
17:02:00 bauzas efried: 'Remove that code' looks to be wrong
17:02:06 bauzas that means "ce" to me
17:02:22 bauzas anyway
17:02:29 bauzas just rebasing now
17:02:53 mriedem edleafe: ok comments inline
17:03:17 efried bauzas Depends on the context. And it's definitely subtle. If I highlighted a chunk and made a gerrit comment on it, I would sai "Remove this code". If I was talking about some code somewhere else, or in the future, I would say "Remove that code".
17:05:50 bauzas efried: I see, it's more the intent itself
17:06:12 bauzas I'll try to play with this
17:06:17 bauzas (and not that)
17:06:39 bauzas thanks
17:06:48 efried Bonne chance.
17:06:55 edleafe mriedem: will review in a bit
17:07:00 efried I'll argue English-is-sillier-than-French any day.
17:11:24 jaypipes lyarwood: thx. just wanted to be sure.
17:16:26 mriedem stvnoyes: your swap volume multiattach tempest test patch is a merge conflict with the resize test, i'm going to rebase those so they are in a series, otherwise i can't test them together
17:17:05 stvnoyes ok
17:19:45 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216
17:22:31 esberglu gibi: Removed the nova-net references from ^ so it's ready whenever you have a chance. Thanks for the reviews!
17:25:16 openstackgerrit Matt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api https://review.openstack.org/271047
17:25:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Add the nova-multiattach job https://review.openstack.org/532689
17:25:22 mriedem ah crap
17:25:45 mriedem gibi: accidental rebase on https://review.openstack.org/#/c/271047/ - want to +W again?
17:32:47 dansmith jroll: is that a thing you want separate from working around the segv issue? because I thought we always expect ironic to be newer and this changes that
17:34:24 jroll dansmith: yes, we want to be able to go one release either way between ironic and nova. click the bug there, there's a plan where we drop most of the negotiation in the client to do so, this is just the quick way to do it
17:35:07 openstackgerrit Merged openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544
17:35:17 mriedem efried: thanks for running the nova meeting last week
17:35:49 efried mriedem Sure thing, for what it was worth :) I didn't update the agenda or anything, so it was a bit slipshod.
17:36:10 mriedem was fine
17:36:35 mriedem you are now, incidentally, the PTL for Rocky, congratulations; it was a trap.
17:37:00 dansmith woohoo!
17:37:02 efried hahahaha, so THAT's all it takes?
17:37:36 dansmith jroll: mkay, I'm not sure how I feel about this.. the mechanism hasn't changed since this was titled "hacky negotiation to let us do a thing" and ... I agree
17:38:38 jroll dansmith: I mean, it's a bit of a hack but it works. the client patches are nearly done to do the better way, but it likely won't make this cycle. our options are basically do this or release without tested rolling upgrades, AIUI
17:38:58 dansmith mriedem: you marked this bug as wishlist so I assume you're okay with it? (having a random previous ironic microversion the driver is willing to use)
17:39:19 dansmith jroll: because some library is crashing conductor?
17:39:30 jroll dansmith: suggestions welcome. especially if you can figure out the conductor exploding
17:40:05 dansmith jroll: everything we add after this will have to either bump that "previous" thing or add infrastructure and handling for the newer feature, IFF the version is new enough
17:40:44 edleafe efried: it's so cute that you think mriedem is kidding.
17:40:57 dansmith jroll: is it some action that causes conductor to crash and is a restart enough? It's odd since conductor shouldn't be related to ironic at all
17:41:09 mriedem dansmith: which bug?
17:41:55 dansmith mriedem: well, it's a combo thing.. they want "N-1 ironic support in the driver" as well as this configdrive optionality which uses it, which I guess works around some other unrelated issue by chance
17:42:02 dansmith mriedem: https://review.openstack.org/#/c/535786/5
17:42:23 jroll dansmith: debugging notes so far are at line 47 here: https://etherpad.openstack.org/p/IronicWhiteBoard . as far as we can tell, upon upgrade, some shared library is being deleted, that is being loaded at runtime in the conductor
17:42:32 jroll see also https://bugs.launchpad.net/ironic/+bug/1744139
17:42:33 openstack Launchpad bug 1744139 in Ironic "Grenade job fails due to systemd stopping n-cpu" [Critical,In progress] - Assigned to Julia Kreger (juliaashleykreger)
17:42:54 dansmith jroll: right, so just a restart of the service because of that issue would be all we need right?
17:43:12 dansmith that seems like a reasonable thing to have in a from-foo script in grenade with a note about the shared library thing
17:43:16 jroll dansmith: pike conductor won't start with queens version of global-reqs
17:43:49 dansmith jroll: yikes, well that kinda breaks upgrades across the board for everyone then
17:44:09 jroll dansmith: yeah, you're the first to respond to cries for help here :P
17:45:10 mriedem jroll: note that dan and i were out last week too on unrelated adventures
17:45:24 sean-k-mooney dansmith: only if you use systemd. if you use kolla you should be fine
17:45:46 sean-k-mooney dansmith: well i guess it breaks the gate for everyone
17:45:55 dansmith jroll: this N-1 patch is a really bad way to work around this grenade breakage, IMHO, because it sounds like you're suggesting changing the ordering to "fix" things with help from this
17:46:13 jroll sean-k-mooney: s/use systemd/run other services co-located with nova conductor without some isolation/
17:47:42 jroll dansmith: don't disagree - I know smart people spent a lot of time debugging the real issue and didn't get very far. I jumped in to help write this patch, which is something we want anyway. (of course we'll need to document this special case as well)
17:48:08 mriedem has grenade ever supported n-1 nova-conductor with n requirements? i thought with multinode grenade, we always brought down the controller services and brought everything up with N code and dependencies, and left the subnode n-cpu at N-1?
17:48:25 dansmith mriedem: right, conductor can't be backlevel
17:48:48 jroll ironic's grenade has always supported n-1 nova with n ironic
17:48:49 dansmith zero detail on the reason for the segv in the logs it looks like
17:48:53 sean-k-mooney johnthetubaguy: oh ya i had not read the full bug but i see its really a python lib issue so ya unless you use contaienrs or other isolation mechanisum(venvs ...) your kindo of screwed
17:49:05 sean-k-mooney * jroll not johnthetubaguy
17:49:10 mriedem jroll: sure but that's different,
17:49:12 dansmith jroll: we're talking about nova-conductor and the rest of nova
17:49:35 mriedem and the release notes in nova have always said, "make sure you upgrade ironic first before nova" which is how we always hedged against version discovery/negotiation
17:49:53 dansmith so I'm pretty hard -2 on working around this issue, which will break lots of people, by changing the upgrade order rules in the eleventh hour
17:50:01 dansmith independent of whether or not we support an n-1 ironic
17:50:05 jroll dansmith: mriedem: right, we haven't upgraded nova at all. but upgrading ironic upgrades a number of libraries. we don't restart n-cond with the new libraries, but somehow at runtime it's causing these segfaults
17:50:20 jroll sure, if someone can help fix the actual issue, I'm happy to kill this patch
17:50:21 dansmith jroll: yep, I get that as the cause here
17:50:47 dansmith we probably need gate/devstack/grenade people help here to capture a core or something
17:51:09 dansmith jroll: is there any temporal locality about when this started happening related to some bump in g-reqs?
17:51:10 jroll sorry, was pointing out that how nova does grenade is irrelevant here
17:51:30 dansmith jroll: sure, the ordering of services within grenade shouldn't change to cover this up, is what Imeant
17:51:32 jroll dansmith: not sure offhand, TheJulia do you know?
17:51:56 jroll :)
17:52:11 dansmith I see now why it affects conductor, because of the worker forking, which likely means it could affect other things the same way under load
17:52:38 TheJulia ironic is a plugin, aiui, we're stuck with the order of ironic after nova
17:52:53 openstackgerrit Jay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits https://review.openstack.org/531899
17:52:54 openstackgerrit Jay Pipes proposed openstack/nova master: _get_trees_matching_all_resources() sum children https://review.openstack.org/534866
17:52:54 openstackgerrit Jay Pipes proposed openstack/nova master: add test for scenario with sum of child resources https://review.openstack.org/534339
17:52:55 openstackgerrit Jay Pipes proposed openstack/nova master: WIP - populate alloc cands for trees https://review.openstack.org/534968
17:52:55 openstackgerrit Jay Pipes proposed openstack/nova master: add _get_usages_from_trees() function https://review.openstack.org/534967
17:52:56 openstackgerrit Jay Pipes proposed openstack/nova master: validate provider summary structure https://review.openstack.org/536516
17:52:59 dansmith no, ironic goes before nova, which is why this is happening, right?
17:53:09 TheJulia no
17:53:17 jroll we disable the nova upgrade altogether in our rolling upgrades test
17:53:37 dansmith ah, well, then the same difference
17:53:41 TheJulia the operational order of upgade as defined in grenade is nova before ironic can be even upgraded, however things go sideways in the logs starting around swift/glance being updated

Earlier   Later