| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 17:02:53 | mriedem | edleafe: ok comments inline | |
| 17:03:17 | efried | bauzas Depends on the context. And it's definitely subtle. If I highlighted a chunk and made a gerrit comment on it, I would sai "Remove this code". If I was talking about some code somewhere else, or in the future, I would say "Remove that code". | |
| 17:05:50 | bauzas | efried: I see, it's more the intent itself | |
| 17:06:12 | bauzas | I'll try to play with this | |
| 17:06:17 | bauzas | (and not that) | |
| 17:06:39 | bauzas | thanks | |
| 17:06:48 | efried | Bonne chance. | |
| 17:06:55 | edleafe | mriedem: will review in a bit | |
| 17:07:00 | efried | I'll argue English-is-sillier-than-French any day. | |
| 17:11:24 | jaypipes | lyarwood: thx. just wanted to be sure. | |
| 17:16:26 | mriedem | stvnoyes: your swap volume multiattach tempest test patch is a merge conflict with the resize test, i'm going to rebase those so they are in a series, otherwise i can't test them together | |
| 17:17:05 | stvnoyes | ok | |
| 17:19:45 | openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: SEA https://review.openstack.org/523216 | |
| 17:22:31 | esberglu | gibi: Removed the nova-net references from ^ so it's ready whenever you have a chance. Thanks for the reviews! | |
| 17:25:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 17:25:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add the nova-multiattach job https://review.openstack.org/532689 | |
| 17:25:22 | mriedem | ah crap | |
| 17:25:45 | mriedem | gibi: accidental rebase on https://review.openstack.org/#/c/271047/ - want to +W again? | |
| 17:32:47 | dansmith | jroll: is that a thing you want separate from working around the segv issue? because I thought we always expect ironic to be newer and this changes that | |
| 17:34:24 | jroll | dansmith: yes, we want to be able to go one release either way between ironic and nova. click the bug there, there's a plan where we drop most of the negotiation in the client to do so, this is just the quick way to do it | |
| 17:35:07 | openstackgerrit | Merged openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544 | |
| 17:35:17 | mriedem | efried: thanks for running the nova meeting last week | |
| 17:35:49 | efried | mriedem Sure thing, for what it was worth :) I didn't update the agenda or anything, so it was a bit slipshod. | |
| 17:36:10 | mriedem | was fine | |
| 17:36:35 | mriedem | you are now, incidentally, the PTL for Rocky, congratulations; it was a trap. | |
| 17:37:00 | dansmith | woohoo! | |
| 17:37:02 | efried | hahahaha, so THAT's all it takes? | |
| 17:37:36 | dansmith | jroll: mkay, I'm not sure how I feel about this.. the mechanism hasn't changed since this was titled "hacky negotiation to let us do a thing" and ... I agree | |
| 17:38:38 | jroll | dansmith: I mean, it's a bit of a hack but it works. the client patches are nearly done to do the better way, but it likely won't make this cycle. our options are basically do this or release without tested rolling upgrades, AIUI | |
| 17:38:58 | dansmith | mriedem: you marked this bug as wishlist so I assume you're okay with it? (having a random previous ironic microversion the driver is willing to use) | |
| 17:39:19 | dansmith | jroll: because some library is crashing conductor? | |
| 17:39:30 | jroll | dansmith: suggestions welcome. especially if you can figure out the conductor exploding | |
| 17:40:05 | dansmith | jroll: everything we add after this will have to either bump that "previous" thing or add infrastructure and handling for the newer feature, IFF the version is new enough | |
| 17:40:44 | edleafe | efried: it's so cute that you think mriedem is kidding. | |
| 17:40:57 | dansmith | jroll: is it some action that causes conductor to crash and is a restart enough? It's odd since conductor shouldn't be related to ironic at all | |
| 17:41:09 | mriedem | dansmith: which bug? | |
| 17:41:55 | dansmith | mriedem: well, it's a combo thing.. they want "N-1 ironic support in the driver" as well as this configdrive optionality which uses it, which I guess works around some other unrelated issue by chance | |
| 17:42:02 | dansmith | mriedem: https://review.openstack.org/#/c/535786/5 | |
| 17:42:23 | jroll | dansmith: debugging notes so far are at line 47 here: https://etherpad.openstack.org/p/IronicWhiteBoard . as far as we can tell, upon upgrade, some shared library is being deleted, that is being loaded at runtime in the conductor | |
| 17:42:32 | jroll | see also https://bugs.launchpad.net/ironic/+bug/1744139 | |
| 17:42:33 | openstack | Launchpad bug 1744139 in Ironic "Grenade job fails due to systemd stopping n-cpu" [Critical,In progress] - Assigned to Julia Kreger (juliaashleykreger) | |
| 17:42:54 | dansmith | jroll: right, so just a restart of the service because of that issue would be all we need right? | |
| 17:43:12 | dansmith | that seems like a reasonable thing to have in a from-foo script in grenade with a note about the shared library thing | |
| 17:43:16 | jroll | dansmith: pike conductor won't start with queens version of global-reqs | |
| 17:43:49 | dansmith | jroll: yikes, well that kinda breaks upgrades across the board for everyone then | |
| 17:44:09 | jroll | dansmith: yeah, you're the first to respond to cries for help here :P | |
| 17:45:10 | mriedem | jroll: note that dan and i were out last week too on unrelated adventures | |
| 17:45:24 | sean-k-mooney | dansmith: only if you use systemd. if you use kolla you should be fine | |
| 17:45:46 | sean-k-mooney | dansmith: well i guess it breaks the gate for everyone | |
| 17:45:55 | dansmith | jroll: this N-1 patch is a really bad way to work around this grenade breakage, IMHO, because it sounds like you're suggesting changing the ordering to "fix" things with help from this | |
| 17:46:13 | jroll | sean-k-mooney: s/use systemd/run other services co-located with nova conductor without some isolation/ | |
| 17:47:42 | jroll | dansmith: don't disagree - I know smart people spent a lot of time debugging the real issue and didn't get very far. I jumped in to help write this patch, which is something we want anyway. (of course we'll need to document this special case as well) | |
| 17:48:08 | mriedem | has grenade ever supported n-1 nova-conductor with n requirements? i thought with multinode grenade, we always brought down the controller services and brought everything up with N code and dependencies, and left the subnode n-cpu at N-1? | |
| 17:48:25 | dansmith | mriedem: right, conductor can't be backlevel | |
| 17:48:48 | jroll | ironic's grenade has always supported n-1 nova with n ironic | |
| 17:48:49 | dansmith | zero detail on the reason for the segv in the logs it looks like | |
| 17:48:53 | sean-k-mooney | johnthetubaguy: oh ya i had not read the full bug but i see its really a python lib issue so ya unless you use contaienrs or other isolation mechanisum(venvs ...) your kindo of screwed | |
| 17:49:05 | sean-k-mooney | * jroll not johnthetubaguy | |
| 17:49:10 | mriedem | jroll: sure but that's different, | |
| 17:49:12 | dansmith | jroll: we're talking about nova-conductor and the rest of nova | |
| 17:49:35 | mriedem | and the release notes in nova have always said, "make sure you upgrade ironic first before nova" which is how we always hedged against version discovery/negotiation | |
| 17:49:53 | dansmith | so I'm pretty hard -2 on working around this issue, which will break lots of people, by changing the upgrade order rules in the eleventh hour | |
| 17:50:01 | dansmith | independent of whether or not we support an n-1 ironic | |
| 17:50:05 | jroll | dansmith: mriedem: right, we haven't upgraded nova at all. but upgrading ironic upgrades a number of libraries. we don't restart n-cond with the new libraries, but somehow at runtime it's causing these segfaults | |
| 17:50:20 | jroll | sure, if someone can help fix the actual issue, I'm happy to kill this patch | |
| 17:50:21 | dansmith | jroll: yep, I get that as the cause here | |
| 17:50:47 | dansmith | we probably need gate/devstack/grenade people help here to capture a core or something | |
| 17:51:09 | dansmith | jroll: is there any temporal locality about when this started happening related to some bump in g-reqs? | |
| 17:51:10 | jroll | sorry, was pointing out that how nova does grenade is irrelevant here | |
| 17:51:30 | dansmith | jroll: sure, the ordering of services within grenade shouldn't change to cover this up, is what Imeant | |
| 17:51:32 | jroll | dansmith: not sure offhand, TheJulia do you know? | |
| 17:51:56 | jroll | :) | |
| 17:52:11 | dansmith | I see now why it affects conductor, because of the worker forking, which likely means it could affect other things the same way under load | |
| 17:52:38 | TheJulia | ironic is a plugin, aiui, we're stuck with the order of ironic after nova | |
| 17:52:53 | openstackgerrit | Jay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits https://review.openstack.org/531899 | |
| 17:52:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: _get_trees_matching_all_resources() sum children https://review.openstack.org/534866 | |
| 17:52:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: add test for scenario with sum of child resources https://review.openstack.org/534339 | |
| 17:52:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: WIP - populate alloc cands for trees https://review.openstack.org/534968 | |
| 17:52:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: add _get_usages_from_trees() function https://review.openstack.org/534967 | |
| 17:52:56 | openstackgerrit | Jay Pipes proposed openstack/nova master: validate provider summary structure https://review.openstack.org/536516 | |
| 17:52:59 | dansmith | no, ironic goes before nova, which is why this is happening, right? | |
| 17:53:09 | TheJulia | no | |
| 17:53:17 | jroll | we disable the nova upgrade altogether in our rolling upgrades test | |
| 17:53:37 | dansmith | ah, well, then the same difference | |
| 17:53:41 | TheJulia | the operational order of upgade as defined in grenade is nova before ironic can be even upgraded, however things go sideways in the logs starting around swift/glance being updated | |
| 17:53:44 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: pass the mdevs when rebooting the guest https://review.openstack.org/533818 | |
| 17:53:44 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Recreate mediated devices on reboot https://review.openstack.org/533642 | |
| 17:53:45 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Avoid suspending guest with attached vGPUs https://review.openstack.org/535693 | |
| 17:54:05 | dansmith | going before isn't a solution because it changes the ordering, which you're forcing by just disabling the nova upgrade | |
| 17:54:29 | TheJulia | going before requires retooling grenade _or_ moving ironic's upgrade scripting into grenade | |
| 17:54:41 | TheJulia | both have been objected to in the past | |
| 17:54:54 | dansmith | and changing the expectations of the operators, documentation, and anything else that may be different as a result | |
| 17:55:30 | jroll | sorry, we got off-track a bit - TheJulia, I meant to ask if you know if there are any g-r bumps around the same time as this started happening? (I'm attempting to find that right now) | |
| 17:55:35 | TheJulia | Ironic has always stated that ironic must be upgraded prior to nova, however some nova contributors have expressed desire that ironic can be upgrded earlier. The API pin is what kills us | |
| 17:56:07 | TheJulia | jroll: I don't believe | |
| 17:56:11 | dansmith | you said "goes before" and "people want earlier" which doesn't seem different to me | |
| 17:56:24 | TheJulia | I don't believe so, however we had two different grenade breaks in about 24 hours | |
| 17:56:25 | dansmith | why is nova-conductor not segfaulting like crazy after the thing that upgrades g-reqs (swift/glance) in other jobs? | |
| 17:56:49 | TheJulia | I think because it gets restarted | |
| 17:56:55 | dansmith | or is it and we don't notice because we don't care until after? | |