| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 17:39:19 | dansmith | jroll: because some library is crashing conductor? | |
| 17:39:30 | jroll | dansmith: suggestions welcome. especially if you can figure out the conductor exploding | |
| 17:40:05 | dansmith | jroll: everything we add after this will have to either bump that "previous" thing or add infrastructure and handling for the newer feature, IFF the version is new enough | |
| 17:40:44 | edleafe | efried: it's so cute that you think mriedem is kidding. | |
| 17:40:57 | dansmith | jroll: is it some action that causes conductor to crash and is a restart enough? It's odd since conductor shouldn't be related to ironic at all | |
| 17:41:09 | mriedem | dansmith: which bug? | |
| 17:41:55 | dansmith | mriedem: well, it's a combo thing.. they want "N-1 ironic support in the driver" as well as this configdrive optionality which uses it, which I guess works around some other unrelated issue by chance | |
| 17:42:02 | dansmith | mriedem: https://review.openstack.org/#/c/535786/5 | |
| 17:42:23 | jroll | dansmith: debugging notes so far are at line 47 here: https://etherpad.openstack.org/p/IronicWhiteBoard . as far as we can tell, upon upgrade, some shared library is being deleted, that is being loaded at runtime in the conductor | |
| 17:42:32 | jroll | see also https://bugs.launchpad.net/ironic/+bug/1744139 | |
| 17:42:33 | openstack | Launchpad bug 1744139 in Ironic "Grenade job fails due to systemd stopping n-cpu" [Critical,In progress] - Assigned to Julia Kreger (juliaashleykreger) | |
| 17:42:54 | dansmith | jroll: right, so just a restart of the service because of that issue would be all we need right? | |
| 17:43:12 | dansmith | that seems like a reasonable thing to have in a from-foo script in grenade with a note about the shared library thing | |
| 17:43:16 | jroll | dansmith: pike conductor won't start with queens version of global-reqs | |
| 17:43:49 | dansmith | jroll: yikes, well that kinda breaks upgrades across the board for everyone then | |
| 17:44:09 | jroll | dansmith: yeah, you're the first to respond to cries for help here :P | |
| 17:45:10 | mriedem | jroll: note that dan and i were out last week too on unrelated adventures | |
| 17:45:24 | sean-k-mooney | dansmith: only if you use systemd. if you use kolla you should be fine | |
| 17:45:46 | sean-k-mooney | dansmith: well i guess it breaks the gate for everyone | |
| 17:45:55 | dansmith | jroll: this N-1 patch is a really bad way to work around this grenade breakage, IMHO, because it sounds like you're suggesting changing the ordering to "fix" things with help from this | |
| 17:46:13 | jroll | sean-k-mooney: s/use systemd/run other services co-located with nova conductor without some isolation/ | |
| 17:47:42 | jroll | dansmith: don't disagree - I know smart people spent a lot of time debugging the real issue and didn't get very far. I jumped in to help write this patch, which is something we want anyway. (of course we'll need to document this special case as well) | |
| 17:48:08 | mriedem | has grenade ever supported n-1 nova-conductor with n requirements? i thought with multinode grenade, we always brought down the controller services and brought everything up with N code and dependencies, and left the subnode n-cpu at N-1? | |
| 17:48:25 | dansmith | mriedem: right, conductor can't be backlevel | |
| 17:48:48 | jroll | ironic's grenade has always supported n-1 nova with n ironic | |
| 17:48:49 | dansmith | zero detail on the reason for the segv in the logs it looks like | |
| 17:48:53 | sean-k-mooney | johnthetubaguy: oh ya i had not read the full bug but i see its really a python lib issue so ya unless you use contaienrs or other isolation mechanisum(venvs ...) your kindo of screwed | |
| 17:49:05 | sean-k-mooney | * jroll not johnthetubaguy | |
| 17:49:10 | mriedem | jroll: sure but that's different, | |
| 17:49:12 | dansmith | jroll: we're talking about nova-conductor and the rest of nova | |
| 17:49:35 | mriedem | and the release notes in nova have always said, "make sure you upgrade ironic first before nova" which is how we always hedged against version discovery/negotiation | |
| 17:49:53 | dansmith | so I'm pretty hard -2 on working around this issue, which will break lots of people, by changing the upgrade order rules in the eleventh hour | |
| 17:50:01 | dansmith | independent of whether or not we support an n-1 ironic | |
| 17:50:05 | jroll | dansmith: mriedem: right, we haven't upgraded nova at all. but upgrading ironic upgrades a number of libraries. we don't restart n-cond with the new libraries, but somehow at runtime it's causing these segfaults | |
| 17:50:20 | jroll | sure, if someone can help fix the actual issue, I'm happy to kill this patch | |
| 17:50:21 | dansmith | jroll: yep, I get that as the cause here | |
| 17:50:47 | dansmith | we probably need gate/devstack/grenade people help here to capture a core or something | |
| 17:51:09 | dansmith | jroll: is there any temporal locality about when this started happening related to some bump in g-reqs? | |
| 17:51:10 | jroll | sorry, was pointing out that how nova does grenade is irrelevant here | |
| 17:51:30 | dansmith | jroll: sure, the ordering of services within grenade shouldn't change to cover this up, is what Imeant | |
| 17:51:32 | jroll | dansmith: not sure offhand, TheJulia do you know? | |
| 17:51:56 | jroll | :) | |
| 17:52:11 | dansmith | I see now why it affects conductor, because of the worker forking, which likely means it could affect other things the same way under load | |
| 17:52:38 | TheJulia | ironic is a plugin, aiui, we're stuck with the order of ironic after nova | |
| 17:52:53 | openstackgerrit | Jay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits https://review.openstack.org/531899 | |
| 17:52:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: _get_trees_matching_all_resources() sum children https://review.openstack.org/534866 | |
| 17:52:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: add test for scenario with sum of child resources https://review.openstack.org/534339 | |
| 17:52:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: WIP - populate alloc cands for trees https://review.openstack.org/534968 | |
| 17:52:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: add _get_usages_from_trees() function https://review.openstack.org/534967 | |
| 17:52:56 | openstackgerrit | Jay Pipes proposed openstack/nova master: validate provider summary structure https://review.openstack.org/536516 | |
| 17:52:59 | dansmith | no, ironic goes before nova, which is why this is happening, right? | |
| 17:53:09 | TheJulia | no | |
| 17:53:17 | jroll | we disable the nova upgrade altogether in our rolling upgrades test | |
| 17:53:37 | dansmith | ah, well, then the same difference | |
| 17:53:41 | TheJulia | the operational order of upgade as defined in grenade is nova before ironic can be even upgraded, however things go sideways in the logs starting around swift/glance being updated | |
| 17:53:44 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: pass the mdevs when rebooting the guest https://review.openstack.org/533818 | |
| 17:53:44 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Recreate mediated devices on reboot https://review.openstack.org/533642 | |
| 17:53:45 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Avoid suspending guest with attached vGPUs https://review.openstack.org/535693 | |
| 17:54:05 | dansmith | going before isn't a solution because it changes the ordering, which you're forcing by just disabling the nova upgrade | |
| 17:54:29 | TheJulia | going before requires retooling grenade _or_ moving ironic's upgrade scripting into grenade | |
| 17:54:41 | TheJulia | both have been objected to in the past | |
| 17:54:54 | dansmith | and changing the expectations of the operators, documentation, and anything else that may be different as a result | |
| 17:55:30 | jroll | sorry, we got off-track a bit - TheJulia, I meant to ask if you know if there are any g-r bumps around the same time as this started happening? (I'm attempting to find that right now) | |
| 17:55:35 | TheJulia | Ironic has always stated that ironic must be upgraded prior to nova, however some nova contributors have expressed desire that ironic can be upgrded earlier. The API pin is what kills us | |
| 17:56:07 | TheJulia | jroll: I don't believe | |
| 17:56:11 | dansmith | you said "goes before" and "people want earlier" which doesn't seem different to me | |
| 17:56:24 | TheJulia | I don't believe so, however we had two different grenade breaks in about 24 hours | |
| 17:56:25 | dansmith | why is nova-conductor not segfaulting like crazy after the thing that upgrades g-reqs (swift/glance) in other jobs? | |
| 17:56:49 | TheJulia | I think because it gets restarted | |
| 17:56:55 | dansmith | or is it and we don't notice because we don't care until after? | |
| 17:57:12 | jroll | I think nova gets upgraded first in most grenade runs, right? | |
| 17:57:17 | dansmith | okay and so it matters for you because you never upgrade nova at the end? | |
| 17:57:23 | TheJulia | the state it is left in prior to that restart is referring to a bunch of modules/libraries/paths that have been deleted due to prior python module upgrades | |
| 17:57:30 | dansmith | jroll: no I don't think so | |
| 17:57:53 | jroll | hm | |
| 17:58:25 | TheJulia | just restarting didn't work because of a sqlalchemy change | |
| 17:58:54 | TheJulia | since python requirements no longer match up because some processes have upgraded parts and pieces of the libraries | |
| 17:59:09 | sean-k-mooney | TheJulia: ie ironic installed new sqlalcamy and since nova is not upgraded its not compatible? | |
| 17:59:15 | TheJulia | sean-k-mooney: no | |
| 17:59:21 | TheJulia | Ironic has not yet upgraded at that point | |
| 17:59:48 | TheJulia | it is earlier on in the glance execution | |
| 17:59:55 | dansmith | what is the sqlalchemy fail? because that seems like a bad thing | |
| 18:00:31 | TheJulia | yes, an argument was removed from sqlalchemy and oslo_db changed accordingly | |
| 18:00:37 | TheJulia | I believe | |
| 18:01:55 | TheJulia | https://github.com/openstack/oslo.db/commit/4a727233dc2b7d1021ed7932d2cd27d8053dca31 | |
| 18:03:02 | jroll | looks like oslo.middleware 3.34.0, oslo.utils 3.35.0, and osprofiler 1.15.1 are the most relevant g-r updates | |
| 18:03:03 | TheJulia | jroll: but also, our grenade jobs were failing due to a neutron change for about 24-36 hours prior to that | |
| 18:03:09 | sean-k-mooney | is this for master? g-r for SQLAlchemy on master has a minium of 1.0.10 and pike has an upperconstaint of 1.1.12 so you should be able to install master without needing to upgrade SQLAlchemy | |
| 18:03:09 | dansmith | so some glue around this change might get us out of the jam? https://review.openstack.org/#/c/501073/1/nova/db/sqlalchemy/api.py | |
| 18:03:38 | jroll | TheJulia: oh right, were they getting into the upgrade bits? | |
| 18:03:40 | sean-k-mooney | i guess it upgrades because we allway install the newest requirement allowed in greade | |
| 18:03:46 | dansmith | I'd guess that's just a library API breakage in oslo.db | |
| 18:04:15 | TheJulia | jroll: what do you mean? | |
| 18:04:39 | jroll | TheJulia: with the neutron failure, was that before or after the upgrade, do you remember? | |
| 18:05:03 | dansmith | TheJulia: yeah, so that commit you linked from oslo.db needs to be graceful I think as it's an API breakage, AFAICT | |
| 18:05:18 | TheJulia | before, neutron's upgrade would fail | |
| 18:05:27 | jroll | TheJulia: ah, thanks. | |
| 18:05:28 | TheJulia | because of networking-generic-swtich | |
| 18:05:49 | dansmith | or, maybe since it was deprecated we need to backport a fix to nova pike to stop passing it? | |
| 18:06:38 | jroll | oslo.concurrency feels like a likely suspect in that range then, too (believe that touches some C libraries) | |