| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-25 | |||
| 19:11:55 | sdague | cfriesen: you're just calling this in a loop from another program right? | |
| 19:12:32 | cfriesen | sdague: no guarantee it'll be in a loop...what if someone wanted to process X migrations each maintenance window | |
| 19:12:45 | sdague | ok | |
| 19:13:19 | mriedem | sdague: i think that grenade-dsvm-neutron-multinode-ubuntu-trusty failure in stable/newton wasn't what you thought it was | |
| 19:13:23 | mriedem | stable/mitaka isn't deleted yet | |
| 19:13:30 | mriedem | plus this job passed yesterday | |
| 19:13:33 | mriedem | it's an ssh issue with ansible | |
| 19:13:40 | mriedem | http://logs.openstack.org/50/420750/2/gate/gate-grenade-dsvm-neutron-multinode-ubuntu-trusty/7fd8a27/logs/devstack-gate-setup-host.txt | |
| 19:14:13 | sdague | mriedem: how is stable/mitaka still a thing? | |
| 19:14:24 | mriedem | sdague: rax laid off tonyb | |
| 19:14:38 | mriedem | if i had twitter, i'd tweet it, that if you want LTS in openstack, don't hire tonyb | |
| 19:14:52 | sdague | heh | |
| 19:19:57 | mriedem | huh, this is weird http://logs.openstack.org/74/467674/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/414bb96/logs/screen-n-api.txt.gz?level=TRACE#_May_25_02_46_02_139728 | |
| 19:20:09 | mriedem | if you run tempest with cinder.cross_az_attach=False, | |
| 19:20:15 | mriedem | 2 bfv tests fail, | |
| 19:20:25 | mriedem | but they are failing on some weird instance.host lazy load issue in the api | |
| 19:20:27 | mriedem | which i wouldn't expect | |
| 19:20:50 | mriedem | host wouldn't be set at this point | |
| 19:20:55 | mriedem | but i'd think it's None | |
| 19:22:47 | mriedem | hmm, or is it the BuildRequest object? | |
| 19:23:04 | mriedem | dansmith: ^ methinks that's what is blowing up | |
| 19:23:33 | mriedem | we create an Instance object w/o a host field set, and serialize that into the BuildRequest | |
| 19:23:50 | dansmith | well, no host at that point | |
| 19:23:55 | mriedem | right, | |
| 19:24:06 | mriedem | but this az check is looking for the host on the instance, which isn't set | |
| 19:24:09 | mriedem | and kablamo | |
| 19:24:17 | dansmith | they're doing a .get o it which doesn't really make sense | |
| 19:24:25 | mriedem | that code is super old, | |
| 19:24:28 | mriedem | probably when it was a dict | |
| 19:24:38 | dansmith | from dict era? | |
| 19:24:39 | dansmith | okay | |
| 19:24:40 | dansmith | so maybe just needs updating for using 'in' ? | |
| 19:24:51 | mriedem | savvas: i'm surprised you didn't hit this when you set cross_az_attach=False ^ | |
| 19:24:55 | mriedem | dansmith: yeah | |
| 19:25:24 | mriedem | dansmith: and/or set instance.host = None for the temp instance that goes into the BuildRequest | |
| 19:26:25 | dansmith | well, we wouldn't want to write it into the database that way unintentionally.. I dunno, .get() was clearly looking for "if set or not" | |
| 19:26:39 | dansmith | and today that's done with in | |
| 19:26:39 | mriedem | i just worry about other places that could hit this, | |
| 19:26:48 | mriedem | but i'd think those would have been found by now | |
| 19:27:45 | dansmith | I'd just hate to have a situation where we've scheduled but something pulled the instance out of buildrequest and does a save() which overwrites the real host | |
| 19:28:01 | mriedem | sure i can be a cautious cat for now | |
| 19:28:15 | dansmith | call me nervous nellie | |
| 19:28:49 | savvas | mriedem: well most of the testing I've done was by creating an instance via horizon. Horizon asks you to create a volume when spawning an instance | |
| 19:29:06 | savvas | so I didn't realize that was happening until I started messing with the system, trying to break it basically | |
| 19:29:17 | melwitt | mriedem: tonyb got hired, so stable/mitaka's days are probably numbered | |
| 19:29:24 | mriedem | melwitt: i heard | |
| 19:29:27 | mriedem | i'll let stable/mitaka know | |
| 19:29:35 | melwitt | cool, thanks | |
| 19:29:36 | dansmith | well, just go announcing his news for him why don'tcha? | |
| 19:29:54 | melwitt | he announced it on twitter, get with the program! | |
| 19:30:02 | dansmith | oh, yeah I see, okay :) | |
| 19:30:08 | dansmith | guns lowered | |
| 19:30:14 | melwitt | lol | |
| 19:30:36 | melwitt | can't believe you're not up-to-speed with the twittersphere, SIGH | |
| 19:30:59 | mriedem | well someone better make an LTS reference in there | |
| 19:31:01 | mriedem | for me | |
| 19:31:02 | mriedem | by proxy | |
| 19:31:39 | melwitt | @mriedem_proxy | |
| 19:31:42 | dansmith | I don't follow him so I didn't notice | |
| 19:46:01 | cfriesen | mriedem: in commit ab05b902 you introduced an online migration to fix an issue....shouldn't that have been ported back to stable/mitaka to fix the issue before the Newton code would ever see it? | |
| 19:47:38 | cfriesen | or are we really saying that online data migrations should be run ASAP after an upgrade is done rather than at the operator's leisure? | |
| 19:50:24 | cdent | edleafe: is a list of list of hosts being returned the only option? Would a single dimension longer list work? | |
| 19:50:38 | cdent | that is: can the grouping happen on the conductor side? | |
| 19:51:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Avoid lazy-load error when getting instance AZ https://review.openstack.org/468147 | |
| 19:51:38 | mriedem | dansmith: easy stat pad ^ | |
| 19:54:10 | mriedem | cfriesen: that one was really odd | |
| 19:54:12 | cfriesen | mriedem: never mind, I see we introduced the change in Newton so no way to backport the migration | |
| 19:54:16 | mriedem | correct | |
| 19:55:21 | cfriesen | mriedem: I'm assuming it'd be okay if we deleted those entries separately prior to the upgrade | |
| 19:55:38 | cfriesen | the failed BuildRequest I mean | |
| 19:55:38 | mriedem | the old busted build requests? | |
| 19:55:39 | mriedem | i think so | |
| 19:57:46 | cfriesen | mriedem: what would have happened to successful old build requests? | |
| 20:00:43 | mriedem | happened where? when? | |
| 20:01:00 | mriedem | successful build requests are cleaned up when the instance is created | |
| 20:01:17 | mriedem | failed ones weren't getting cleaned up at some point, which i think was part of the issue | |
| 20:01:26 | cfriesen | ah, okay. so at steady-state there would only be failed ones left in the DB | |
| 20:01:29 | mriedem | so you'd have these old busted build requests with the old data format, | |
| 20:01:34 | mriedem | and they'd revive somehow and blow up | |
| 20:01:52 | mriedem | i believe they'd be pulled back when listing instances | |
| 20:01:55 | mriedem | because they weren't deleted | |
| 20:02:08 | mriedem | and the from_db_object or whatever assumed the new data model format | |
| 20:02:32 | mriedem | as i said, it was a weird one | |
| 20:02:54 | mriedem | dansmith: +2 on https://review.openstack.org/#/c/460616/ - time to get someone to pay you back | |
| 20:08:44 | edleafe | cd | |
| 20:08:47 | edleafe | ugh | |
| 20:11:01 | mriedem | dansmith: on this cells rpcapi thing https://review.openstack.org/#/c/461532/6/nova/cells/rpcapi.py - i'm not sure what type of exception should be raised here | |
| 20:11:17 | mriedem | just anything? | |
| 20:11:28 | dansmith | mriedem: yeah, we've done it a couple other places | |
| 20:11:30 | mriedem | ServiceTooOld ? | |
| 20:11:45 | dansmith | mriedem: since that would be an upgrade concern, and it's cellsv1, I'd just raise anything to call attention | |
| 20:11:49 | dansmith | sure, sounds fine tome | |
| 20:13:24 | gyee | mriedem, jaypipes, can you guys please take a look? https://review.hpcloud.net/135895 | |
| 20:13:31 | gyee | I am trying to get a resolution on this one | |
| 20:13:54 | mriedem | gyee: i don't have access to your internal gerrit :) | |
| 20:14:01 | mriedem | i can request access of course | |
| 20:14:53 | gyee | oh shit, wrong link | |
| 20:14:55 | gyee | my bad | |
| 20:15:28 | gyee | https://review.openstack.org/#/c/466143/ | |
| 20:16:05 | gyee | my brain is fried trying to context switch upstream/downstream | |