Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-22
17:25:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Add the nova-multiattach job https://review.openstack.org/532689
17:25:22 mriedem ah crap
17:25:45 mriedem gibi: accidental rebase on https://review.openstack.org/#/c/271047/ - want to +W again?
17:32:47 dansmith jroll: is that a thing you want separate from working around the segv issue? because I thought we always expect ironic to be newer and this changes that
17:34:24 jroll dansmith: yes, we want to be able to go one release either way between ironic and nova. click the bug there, there's a plan where we drop most of the negotiation in the client to do so, this is just the quick way to do it
17:35:07 openstackgerrit Merged openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544
17:35:17 mriedem efried: thanks for running the nova meeting last week
17:35:49 efried mriedem Sure thing, for what it was worth :) I didn't update the agenda or anything, so it was a bit slipshod.
17:36:10 mriedem was fine
17:36:35 mriedem you are now, incidentally, the PTL for Rocky, congratulations; it was a trap.
17:37:00 dansmith woohoo!
17:37:02 efried hahahaha, so THAT's all it takes?
17:37:36 dansmith jroll: mkay, I'm not sure how I feel about this.. the mechanism hasn't changed since this was titled "hacky negotiation to let us do a thing" and ... I agree
17:38:38 jroll dansmith: I mean, it's a bit of a hack but it works. the client patches are nearly done to do the better way, but it likely won't make this cycle. our options are basically do this or release without tested rolling upgrades, AIUI
17:38:58 dansmith mriedem: you marked this bug as wishlist so I assume you're okay with it? (having a random previous ironic microversion the driver is willing to use)
17:39:19 dansmith jroll: because some library is crashing conductor?
17:39:30 jroll dansmith: suggestions welcome. especially if you can figure out the conductor exploding
17:40:05 dansmith jroll: everything we add after this will have to either bump that "previous" thing or add infrastructure and handling for the newer feature, IFF the version is new enough
17:40:44 edleafe efried: it's so cute that you think mriedem is kidding.
17:40:57 dansmith jroll: is it some action that causes conductor to crash and is a restart enough? It's odd since conductor shouldn't be related to ironic at all
17:41:09 mriedem dansmith: which bug?
17:41:55 dansmith mriedem: well, it's a combo thing.. they want "N-1 ironic support in the driver" as well as this configdrive optionality which uses it, which I guess works around some other unrelated issue by chance
17:42:02 dansmith mriedem: https://review.openstack.org/#/c/535786/5
17:42:23 jroll dansmith: debugging notes so far are at line 47 here: https://etherpad.openstack.org/p/IronicWhiteBoard . as far as we can tell, upon upgrade, some shared library is being deleted, that is being loaded at runtime in the conductor
17:42:32 jroll see also https://bugs.launchpad.net/ironic/+bug/1744139
17:42:33 openstack Launchpad bug 1744139 in Ironic "Grenade job fails due to systemd stopping n-cpu" [Critical,In progress] - Assigned to Julia Kreger (juliaashleykreger)
17:42:54 dansmith jroll: right, so just a restart of the service because of that issue would be all we need right?
17:43:12 dansmith that seems like a reasonable thing to have in a from-foo script in grenade with a note about the shared library thing
17:43:16 jroll dansmith: pike conductor won't start with queens version of global-reqs
17:43:49 dansmith jroll: yikes, well that kinda breaks upgrades across the board for everyone then
17:44:09 jroll dansmith: yeah, you're the first to respond to cries for help here :P
17:45:10 mriedem jroll: note that dan and i were out last week too on unrelated adventures
17:45:24 sean-k-mooney dansmith: only if you use systemd. if you use kolla you should be fine
17:45:46 sean-k-mooney dansmith: well i guess it breaks the gate for everyone
17:45:55 dansmith jroll: this N-1 patch is a really bad way to work around this grenade breakage, IMHO, because it sounds like you're suggesting changing the ordering to "fix" things with help from this
17:46:13 jroll sean-k-mooney: s/use systemd/run other services co-located with nova conductor without some isolation/
17:47:42 jroll dansmith: don't disagree - I know smart people spent a lot of time debugging the real issue and didn't get very far. I jumped in to help write this patch, which is something we want anyway. (of course we'll need to document this special case as well)
17:48:08 mriedem has grenade ever supported n-1 nova-conductor with n requirements? i thought with multinode grenade, we always brought down the controller services and brought everything up with N code and dependencies, and left the subnode n-cpu at N-1?
17:48:25 dansmith mriedem: right, conductor can't be backlevel
17:48:48 jroll ironic's grenade has always supported n-1 nova with n ironic
17:48:49 dansmith zero detail on the reason for the segv in the logs it looks like
17:48:53 sean-k-mooney johnthetubaguy: oh ya i had not read the full bug but i see its really a python lib issue so ya unless you use contaienrs or other isolation mechanisum(venvs ...) your kindo of screwed
17:49:05 sean-k-mooney * jroll not johnthetubaguy
17:49:10 mriedem jroll: sure but that's different,
17:49:12 dansmith jroll: we're talking about nova-conductor and the rest of nova
17:49:35 mriedem and the release notes in nova have always said, "make sure you upgrade ironic first before nova" which is how we always hedged against version discovery/negotiation
17:49:53 dansmith so I'm pretty hard -2 on working around this issue, which will break lots of people, by changing the upgrade order rules in the eleventh hour
17:50:01 dansmith independent of whether or not we support an n-1 ironic
17:50:05 jroll dansmith: mriedem: right, we haven't upgraded nova at all. but upgrading ironic upgrades a number of libraries. we don't restart n-cond with the new libraries, but somehow at runtime it's causing these segfaults
17:50:20 jroll sure, if someone can help fix the actual issue, I'm happy to kill this patch
17:50:21 dansmith jroll: yep, I get that as the cause here
17:50:47 dansmith we probably need gate/devstack/grenade people help here to capture a core or something
17:51:09 dansmith jroll: is there any temporal locality about when this started happening related to some bump in g-reqs?
17:51:10 jroll sorry, was pointing out that how nova does grenade is irrelevant here
17:51:30 dansmith jroll: sure, the ordering of services within grenade shouldn't change to cover this up, is what Imeant
17:51:32 jroll dansmith: not sure offhand, TheJulia do you know?
17:51:56 jroll :)
17:52:11 dansmith I see now why it affects conductor, because of the worker forking, which likely means it could affect other things the same way under load
17:52:38 TheJulia ironic is a plugin, aiui, we're stuck with the order of ironic after nova
17:52:53 openstackgerrit Jay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits https://review.openstack.org/531899
17:52:54 openstackgerrit Jay Pipes proposed openstack/nova master: _get_trees_matching_all_resources() sum children https://review.openstack.org/534866
17:52:54 openstackgerrit Jay Pipes proposed openstack/nova master: add test for scenario with sum of child resources https://review.openstack.org/534339
17:52:55 openstackgerrit Jay Pipes proposed openstack/nova master: WIP - populate alloc cands for trees https://review.openstack.org/534968
17:52:55 openstackgerrit Jay Pipes proposed openstack/nova master: add _get_usages_from_trees() function https://review.openstack.org/534967
17:52:56 openstackgerrit Jay Pipes proposed openstack/nova master: validate provider summary structure https://review.openstack.org/536516
17:52:59 dansmith no, ironic goes before nova, which is why this is happening, right?
17:53:09 TheJulia no
17:53:17 jroll we disable the nova upgrade altogether in our rolling upgrades test
17:53:37 dansmith ah, well, then the same difference
17:53:41 TheJulia the operational order of upgade as defined in grenade is nova before ironic can be even upgraded, however things go sideways in the logs starting around swift/glance being updated
17:53:44 openstackgerrit Sylvain Bauza proposed openstack/nova master: libvirt: pass the mdevs when rebooting the guest https://review.openstack.org/533818
17:53:44 openstackgerrit Sylvain Bauza proposed openstack/nova master: Recreate mediated devices on reboot https://review.openstack.org/533642
17:53:45 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: Avoid suspending guest with attached vGPUs https://review.openstack.org/535693
17:54:05 dansmith going before isn't a solution because it changes the ordering, which you're forcing by just disabling the nova upgrade
17:54:29 TheJulia going before requires retooling grenade _or_ moving ironic's upgrade scripting into grenade
17:54:41 TheJulia both have been objected to in the past
17:54:54 dansmith and changing the expectations of the operators, documentation, and anything else that may be different as a result
17:55:30 jroll sorry, we got off-track a bit - TheJulia, I meant to ask if you know if there are any g-r bumps around the same time as this started happening? (I'm attempting to find that right now)
17:55:35 TheJulia Ironic has always stated that ironic must be upgraded prior to nova, however some nova contributors have expressed desire that ironic can be upgrded earlier. The API pin is what kills us
17:56:07 TheJulia jroll: I don't believe
17:56:11 dansmith you said "goes before" and "people want earlier" which doesn't seem different to me
17:56:24 TheJulia I don't believe so, however we had two different grenade breaks in about 24 hours
17:56:25 dansmith why is nova-conductor not segfaulting like crazy after the thing that upgrades g-reqs (swift/glance) in other jobs?
17:56:49 TheJulia I think because it gets restarted
17:56:55 dansmith or is it and we don't notice because we don't care until after?
17:57:12 jroll I think nova gets upgraded first in most grenade runs, right?
17:57:17 dansmith okay and so it matters for you because you never upgrade nova at the end?
17:57:23 TheJulia the state it is left in prior to that restart is referring to a bunch of modules/libraries/paths that have been deleted due to prior python module upgrades
17:57:30 dansmith jroll: no I don't think so
17:57:53 jroll hm
17:58:25 TheJulia just restarting didn't work because of a sqlalchemy change
17:58:54 TheJulia since python requirements no longer match up because some processes have upgraded parts and pieces of the libraries
17:59:09 sean-k-mooney TheJulia: ie ironic installed new sqlalcamy and since nova is not upgraded its not compatible?
17:59:15 TheJulia sean-k-mooney: no
17:59:21 TheJulia Ironic has not yet upgraded at that point
17:59:48 TheJulia it is earlier on in the glance execution
17:59:55 dansmith what is the sqlalchemy fail? because that seems like a bad thing
18:00:31 TheJulia yes, an argument was removed from sqlalchemy and oslo_db changed accordingly
18:00:37 TheJulia I believe
18:01:55 TheJulia https://github.com/openstack/oslo.db/commit/4a727233dc2b7d1021ed7932d2cd27d8053dca31

Earlier   Later