| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-25 | |||
| 18:56:47 | mriedem | i meant you hurting chet | |
| 18:56:56 | dansmith | heh | |
| 19:02:03 | cfriesen | would there be value in having "nova-manage db online_data_migrations" return a value indicating whether or not it needs to be called again? (when specifying --max-count) | |
| 19:02:45 | cfriesen | seems like currently a calling script needs to parse the output | |
| 19:03:21 | melwitt | I thought it already returns 1 if there's more to do and 0 when there's no more to do. or are you saying it returns 0 when you use --max-count and there is still more to do? | |
| 19:05:01 | cfriesen | melwitt: it currently has "return ran and 1 or 0" and I think an exception in one of the migration routines will give a 0 | |
| 19:05:15 | cfriesen | I want success/partial-success/error return codes | |
| 19:05:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle conflict from neutron when addFloatingIP fails https://review.openstack.org/468136 | |
| 19:05:29 | mriedem | cfriesen: sounds like a bug | |
| 19:06:30 | cfriesen | mriedem: what should it return in the error case? 127 like we have if you specify a nonpositive integer for max-count? or something else | |
| 19:06:55 | mriedem | idk, sometimes we return 2 for things | |
| 19:07:00 | mriedem | it's not really defined | |
| 19:07:35 | sdague | cfriesen: so the problem with that is it means you have to turn off set -e in scripts | |
| 19:07:56 | mriedem | sdague: that's what we do in grenade for the nova-status upgrade check | |
| 19:08:06 | sdague | mriedem: yeh, which is a little suck | |
| 19:08:33 | mriedem | well maybe we don't https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/upgrade.sh#L97 | |
| 19:08:38 | mriedem | i must be thinking of something in d-g | |
| 19:08:56 | mriedem | ah we use || | |
| 19:08:59 | sdague | yeh | |
| 19:09:04 | sdague | which is just mask the world | |
| 19:09:23 | cfriesen | sdague: you mean distinguishing between success-and-done vs success-and-need-to-call-again? | |
| 19:10:15 | sdague | yeh, but more importantly best practice for shell scripts is set -e, and things are only supposed to be non zero on failure | |
| 19:11:11 | cfriesen | sdague: so you're saying the caller must always parse the output to determine if it's actually done? | |
| 19:11:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle conflict from neutron when addFloatingIP fails https://review.openstack.org/468136 | |
| 19:11:19 | cfriesen | because that seems kind of lame | |
| 19:11:55 | sdague | cfriesen: you're just calling this in a loop from another program right? | |
| 19:12:32 | cfriesen | sdague: no guarantee it'll be in a loop...what if someone wanted to process X migrations each maintenance window | |
| 19:12:45 | sdague | ok | |
| 19:13:19 | mriedem | sdague: i think that grenade-dsvm-neutron-multinode-ubuntu-trusty failure in stable/newton wasn't what you thought it was | |
| 19:13:23 | mriedem | stable/mitaka isn't deleted yet | |
| 19:13:30 | mriedem | plus this job passed yesterday | |
| 19:13:33 | mriedem | it's an ssh issue with ansible | |
| 19:13:40 | mriedem | http://logs.openstack.org/50/420750/2/gate/gate-grenade-dsvm-neutron-multinode-ubuntu-trusty/7fd8a27/logs/devstack-gate-setup-host.txt | |
| 19:14:13 | sdague | mriedem: how is stable/mitaka still a thing? | |
| 19:14:24 | mriedem | sdague: rax laid off tonyb | |
| 19:14:38 | mriedem | if i had twitter, i'd tweet it, that if you want LTS in openstack, don't hire tonyb | |
| 19:14:52 | sdague | heh | |
| 19:19:57 | mriedem | huh, this is weird http://logs.openstack.org/74/467674/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/414bb96/logs/screen-n-api.txt.gz?level=TRACE#_May_25_02_46_02_139728 | |
| 19:20:09 | mriedem | if you run tempest with cinder.cross_az_attach=False, | |
| 19:20:15 | mriedem | 2 bfv tests fail, | |
| 19:20:25 | mriedem | but they are failing on some weird instance.host lazy load issue in the api | |
| 19:20:27 | mriedem | which i wouldn't expect | |
| 19:20:50 | mriedem | host wouldn't be set at this point | |
| 19:20:55 | mriedem | but i'd think it's None | |
| 19:22:47 | mriedem | hmm, or is it the BuildRequest object? | |
| 19:23:04 | mriedem | dansmith: ^ methinks that's what is blowing up | |
| 19:23:33 | mriedem | we create an Instance object w/o a host field set, and serialize that into the BuildRequest | |
| 19:23:50 | dansmith | well, no host at that point | |
| 19:23:55 | mriedem | right, | |
| 19:24:06 | mriedem | but this az check is looking for the host on the instance, which isn't set | |
| 19:24:09 | mriedem | and kablamo | |
| 19:24:17 | dansmith | they're doing a .get o it which doesn't really make sense | |
| 19:24:25 | mriedem | that code is super old, | |
| 19:24:28 | mriedem | probably when it was a dict | |
| 19:24:38 | dansmith | from dict era? | |
| 19:24:39 | dansmith | okay | |
| 19:24:40 | dansmith | so maybe just needs updating for using 'in' ? | |
| 19:24:51 | mriedem | savvas: i'm surprised you didn't hit this when you set cross_az_attach=False ^ | |
| 19:24:55 | mriedem | dansmith: yeah | |
| 19:25:24 | mriedem | dansmith: and/or set instance.host = None for the temp instance that goes into the BuildRequest | |
| 19:26:25 | dansmith | well, we wouldn't want to write it into the database that way unintentionally.. I dunno, .get() was clearly looking for "if set or not" | |
| 19:26:39 | dansmith | and today that's done with in | |
| 19:26:39 | mriedem | i just worry about other places that could hit this, | |
| 19:26:48 | mriedem | but i'd think those would have been found by now | |
| 19:27:45 | dansmith | I'd just hate to have a situation where we've scheduled but something pulled the instance out of buildrequest and does a save() which overwrites the real host | |
| 19:28:01 | mriedem | sure i can be a cautious cat for now | |
| 19:28:15 | dansmith | call me nervous nellie | |
| 19:28:49 | savvas | mriedem: well most of the testing I've done was by creating an instance via horizon. Horizon asks you to create a volume when spawning an instance | |
| 19:29:06 | savvas | so I didn't realize that was happening until I started messing with the system, trying to break it basically | |
| 19:29:17 | melwitt | mriedem: tonyb got hired, so stable/mitaka's days are probably numbered | |
| 19:29:24 | mriedem | melwitt: i heard | |
| 19:29:27 | mriedem | i'll let stable/mitaka know | |
| 19:29:35 | melwitt | cool, thanks | |
| 19:29:36 | dansmith | well, just go announcing his news for him why don'tcha? | |
| 19:29:54 | melwitt | he announced it on twitter, get with the program! | |
| 19:30:02 | dansmith | oh, yeah I see, okay :) | |
| 19:30:08 | dansmith | guns lowered | |
| 19:30:14 | melwitt | lol | |
| 19:30:36 | melwitt | can't believe you're not up-to-speed with the twittersphere, SIGH | |
| 19:30:59 | mriedem | well someone better make an LTS reference in there | |
| 19:31:01 | mriedem | for me | |
| 19:31:02 | mriedem | by proxy | |
| 19:31:39 | melwitt | @mriedem_proxy | |
| 19:31:42 | dansmith | I don't follow him so I didn't notice | |
| 19:46:01 | cfriesen | mriedem: in commit ab05b902 you introduced an online migration to fix an issue....shouldn't that have been ported back to stable/mitaka to fix the issue before the Newton code would ever see it? | |
| 19:47:38 | cfriesen | or are we really saying that online data migrations should be run ASAP after an upgrade is done rather than at the operator's leisure? | |
| 19:50:24 | cdent | edleafe: is a list of list of hosts being returned the only option? Would a single dimension longer list work? | |
| 19:50:38 | cdent | that is: can the grouping happen on the conductor side? | |
| 19:51:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Avoid lazy-load error when getting instance AZ https://review.openstack.org/468147 | |
| 19:51:38 | mriedem | dansmith: easy stat pad ^ | |
| 19:54:10 | mriedem | cfriesen: that one was really odd | |
| 19:54:12 | cfriesen | mriedem: never mind, I see we introduced the change in Newton so no way to backport the migration | |
| 19:54:16 | mriedem | correct | |
| 19:55:21 | cfriesen | mriedem: I'm assuming it'd be okay if we deleted those entries separately prior to the upgrade | |
| 19:55:38 | cfriesen | the failed BuildRequest I mean | |
| 19:55:38 | mriedem | the old busted build requests? | |
| 19:55:39 | mriedem | i think so | |
| 19:57:46 | cfriesen | mriedem: what would have happened to successful old build requests? | |
| 20:00:43 | mriedem | happened where? when? | |
| 20:01:00 | mriedem | successful build requests are cleaned up when the instance is created | |