| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 17:46:13 | jroll | sean-k-mooney: s/use systemd/run other services co-located with nova conductor without some isolation/ | |
| 17:47:42 | jroll | dansmith: don't disagree - I know smart people spent a lot of time debugging the real issue and didn't get very far. I jumped in to help write this patch, which is something we want anyway. (of course we'll need to document this special case as well) | |
| 17:48:08 | mriedem | has grenade ever supported n-1 nova-conductor with n requirements? i thought with multinode grenade, we always brought down the controller services and brought everything up with N code and dependencies, and left the subnode n-cpu at N-1? | |
| 17:48:25 | dansmith | mriedem: right, conductor can't be backlevel | |
| 17:48:48 | jroll | ironic's grenade has always supported n-1 nova with n ironic | |
| 17:48:49 | dansmith | zero detail on the reason for the segv in the logs it looks like | |
| 17:48:53 | sean-k-mooney | johnthetubaguy: oh ya i had not read the full bug but i see its really a python lib issue so ya unless you use contaienrs or other isolation mechanisum(venvs ...) your kindo of screwed | |
| 17:49:05 | sean-k-mooney | * jroll not johnthetubaguy | |
| 17:49:10 | mriedem | jroll: sure but that's different, | |
| 17:49:12 | dansmith | jroll: we're talking about nova-conductor and the rest of nova | |
| 17:49:35 | mriedem | and the release notes in nova have always said, "make sure you upgrade ironic first before nova" which is how we always hedged against version discovery/negotiation | |
| 17:49:53 | dansmith | so I'm pretty hard -2 on working around this issue, which will break lots of people, by changing the upgrade order rules in the eleventh hour | |
| 17:50:01 | dansmith | independent of whether or not we support an n-1 ironic | |
| 17:50:05 | jroll | dansmith: mriedem: right, we haven't upgraded nova at all. but upgrading ironic upgrades a number of libraries. we don't restart n-cond with the new libraries, but somehow at runtime it's causing these segfaults | |
| 17:50:20 | jroll | sure, if someone can help fix the actual issue, I'm happy to kill this patch | |
| 17:50:21 | dansmith | jroll: yep, I get that as the cause here | |
| 17:50:47 | dansmith | we probably need gate/devstack/grenade people help here to capture a core or something | |
| 17:51:09 | dansmith | jroll: is there any temporal locality about when this started happening related to some bump in g-reqs? | |
| 17:51:10 | jroll | sorry, was pointing out that how nova does grenade is irrelevant here | |
| 17:51:30 | dansmith | jroll: sure, the ordering of services within grenade shouldn't change to cover this up, is what Imeant | |
| 17:51:32 | jroll | dansmith: not sure offhand, TheJulia do you know? | |
| 17:51:56 | jroll | :) | |
| 17:52:11 | dansmith | I see now why it affects conductor, because of the worker forking, which likely means it could affect other things the same way under load | |
| 17:52:38 | TheJulia | ironic is a plugin, aiui, we're stuck with the order of ironic after nova | |
| 17:52:53 | openstackgerrit | Jay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits https://review.openstack.org/531899 | |
| 17:52:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: add test for scenario with sum of child resources https://review.openstack.org/534339 | |
| 17:52:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: _get_trees_matching_all_resources() sum children https://review.openstack.org/534866 | |
| 17:52:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: add _get_usages_from_trees() function https://review.openstack.org/534967 | |
| 17:52:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: WIP - populate alloc cands for trees https://review.openstack.org/534968 | |
| 17:52:56 | openstackgerrit | Jay Pipes proposed openstack/nova master: validate provider summary structure https://review.openstack.org/536516 | |
| 17:52:59 | dansmith | no, ironic goes before nova, which is why this is happening, right? | |
| 17:53:09 | TheJulia | no | |
| 17:53:17 | jroll | we disable the nova upgrade altogether in our rolling upgrades test | |
| 17:53:37 | dansmith | ah, well, then the same difference | |
| 17:53:41 | TheJulia | the operational order of upgade as defined in grenade is nova before ironic can be even upgraded, however things go sideways in the logs starting around swift/glance being updated | |
| 17:53:44 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Recreate mediated devices on reboot https://review.openstack.org/533642 | |
| 17:53:44 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: pass the mdevs when rebooting the guest https://review.openstack.org/533818 | |
| 17:53:45 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Avoid suspending guest with attached vGPUs https://review.openstack.org/535693 | |
| 17:54:05 | dansmith | going before isn't a solution because it changes the ordering, which you're forcing by just disabling the nova upgrade | |
| 17:54:29 | TheJulia | going before requires retooling grenade _or_ moving ironic's upgrade scripting into grenade | |
| 17:54:41 | TheJulia | both have been objected to in the past | |
| 17:54:54 | dansmith | and changing the expectations of the operators, documentation, and anything else that may be different as a result | |
| 17:55:30 | jroll | sorry, we got off-track a bit - TheJulia, I meant to ask if you know if there are any g-r bumps around the same time as this started happening? (I'm attempting to find that right now) | |
| 17:55:35 | TheJulia | Ironic has always stated that ironic must be upgraded prior to nova, however some nova contributors have expressed desire that ironic can be upgrded earlier. The API pin is what kills us | |
| 17:56:07 | TheJulia | jroll: I don't believe | |
| 17:56:11 | dansmith | you said "goes before" and "people want earlier" which doesn't seem different to me | |
| 17:56:24 | TheJulia | I don't believe so, however we had two different grenade breaks in about 24 hours | |
| 17:56:25 | dansmith | why is nova-conductor not segfaulting like crazy after the thing that upgrades g-reqs (swift/glance) in other jobs? | |
| 17:56:49 | TheJulia | I think because it gets restarted | |
| 17:56:55 | dansmith | or is it and we don't notice because we don't care until after? | |
| 17:57:12 | jroll | I think nova gets upgraded first in most grenade runs, right? | |
| 17:57:17 | dansmith | okay and so it matters for you because you never upgrade nova at the end? | |
| 17:57:23 | TheJulia | the state it is left in prior to that restart is referring to a bunch of modules/libraries/paths that have been deleted due to prior python module upgrades | |
| 17:57:30 | dansmith | jroll: no I don't think so | |
| 17:57:53 | jroll | hm | |
| 17:58:25 | TheJulia | just restarting didn't work because of a sqlalchemy change | |
| 17:58:54 | TheJulia | since python requirements no longer match up because some processes have upgraded parts and pieces of the libraries | |
| 17:59:09 | sean-k-mooney | TheJulia: ie ironic installed new sqlalcamy and since nova is not upgraded its not compatible? | |
| 17:59:15 | TheJulia | sean-k-mooney: no | |
| 17:59:21 | TheJulia | Ironic has not yet upgraded at that point | |
| 17:59:48 | TheJulia | it is earlier on in the glance execution | |
| 17:59:55 | dansmith | what is the sqlalchemy fail? because that seems like a bad thing | |
| 18:00:31 | TheJulia | yes, an argument was removed from sqlalchemy and oslo_db changed accordingly | |
| 18:00:37 | TheJulia | I believe | |
| 18:01:55 | TheJulia | https://github.com/openstack/oslo.db/commit/4a727233dc2b7d1021ed7932d2cd27d8053dca31 | |
| 18:03:02 | jroll | looks like oslo.middleware 3.34.0, oslo.utils 3.35.0, and osprofiler 1.15.1 are the most relevant g-r updates | |
| 18:03:03 | TheJulia | jroll: but also, our grenade jobs were failing due to a neutron change for about 24-36 hours prior to that | |
| 18:03:09 | dansmith | so some glue around this change might get us out of the jam? https://review.openstack.org/#/c/501073/1/nova/db/sqlalchemy/api.py | |
| 18:03:09 | sean-k-mooney | is this for master? g-r for SQLAlchemy on master has a minium of 1.0.10 and pike has an upperconstaint of 1.1.12 so you should be able to install master without needing to upgrade SQLAlchemy | |
| 18:03:38 | jroll | TheJulia: oh right, were they getting into the upgrade bits? | |
| 18:03:40 | sean-k-mooney | i guess it upgrades because we allway install the newest requirement allowed in greade | |
| 18:03:46 | dansmith | I'd guess that's just a library API breakage in oslo.db | |
| 18:04:15 | TheJulia | jroll: what do you mean? | |
| 18:04:39 | jroll | TheJulia: with the neutron failure, was that before or after the upgrade, do you remember? | |
| 18:05:03 | dansmith | TheJulia: yeah, so that commit you linked from oslo.db needs to be graceful I think as it's an API breakage, AFAICT | |
| 18:05:18 | TheJulia | before, neutron's upgrade would fail | |
| 18:05:27 | jroll | TheJulia: ah, thanks. | |
| 18:05:28 | TheJulia | because of networking-generic-swtich | |
| 18:05:49 | dansmith | or, maybe since it was deprecated we need to backport a fix to nova pike to stop passing it? | |
| 18:06:38 | jroll | oslo.concurrency feels like a likely suspect in that range then, too (believe that touches some C libraries) | |
| 18:06:41 | mriedem | how is nova-conductor even picking up the new oslo.db if it's not being restarted? | |
| 18:06:52 | jroll | dansmith: that would let us restart it, yeah, though that isn't ideal | |
| 18:06:52 | dansmith | mriedem: worker forking | |
| 18:07:00 | jroll | well, there's two things going on | |
| 18:07:09 | jroll | the worker fork segfaults | |
| 18:07:17 | dansmith | jroll: ah right I got lost that this can't be the segv issue, it's the breakage that prevents the restart, correct | |
| 18:07:18 | jroll | if we restart the main process, it fails due to oslo.db | |
| 18:07:23 | dansmith | right right | |
| 18:07:33 | mriedem | ah ok, so not an intentional restart | |
| 18:07:44 | mriedem | something triggers a failure and restart, which then fails | |
| 18:08:10 | dansmith | mriedem: it's just workers being cycled in and out I think, not failure related initially | |
| 18:09:11 | mriedem | i'll go ahead and say i don't understand | |
| 18:09:17 | mriedem | i welcome the ridicule | |
| 18:09:37 | sean-k-mooney | dansmith: the minium version of SQLAlchemy on master is below the max on pike currently. the commit TheJulia referenced does not seem to indicate what version of SQLAlchemy removed the retry arg. it sound like there is a min version bump missing also if that change is not graceful | |
| 18:09:43 | jroll | mriedem: it looks like this http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/screen-n-cond.txt.gz#_Jan_18_05_52_41_241366 | |
| 18:09:46 | TheJulia | if the fork causes a dynamic library to be referenced that hasn't already been opened by the parent process, that would explain the segfault in that some of the things the parent was still running with that spanws the worker is gone because pip deleted them | |
| 18:09:55 | jroll | and then systemd starts killing n-cpu and such, because insanity | |
| 18:10:19 | TheJulia | and then people begin drinking fine spirits | |
| 18:11:30 | dansmith | I have to run to a thing for a bit, back in a bit | |
| 18:12:30 | jroll | I feel like this is related but I can't prove it https://github.com/openstack/oslo.concurrency/commit/55e06261aa86c87c7c059fbddc97cdbaae06e8dd | |