| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-06 | |||
| 14:50:10 | mriedem | 2. you can run nova-manage api_db sync unconditionally | |
| 14:50:48 | odyssey4me | we're very happy to shorten and simplify the steps taken, so option 2 sounds good to me | |
| 14:51:29 | odyssey4me | I'll push a patch up shortly. I'll ping you to verify the review if that's ok? | |
| 14:51:31 | mriedem | fwiw, grenade also runs map_cell0 on upgrade and cell0 is already mapped http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_34_277 | |
| 14:51:48 | mriedem | sure. i'm not ansible guy, but i can see if the non-voting upgrade job is green or not :) | |
| 14:52:33 | odyssey4me | :) hopefully we can make it as readable as a shell task procedure | |
| 14:52:42 | mriedem | oh looky here http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_25_031 | |
| 14:52:49 | mriedem | grenade hits the same thing but doesn't fail for some reason | |
| 14:56:11 | tssurya | sorry was afk, that doesn't look goo | |
| 14:56:14 | tssurya | good* | |
| 14:57:40 | odyssey4me | mriedem maybe someone couldn't figure it out and added || true ;) | |
| 14:58:41 | odyssey4me | jokes aside, that's quite serious... but I'm glad we're not alone, and that we could find a red flag :) | |
| 14:58:48 | mriedem | it's hitting htis https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/upgrade.sh#L74 | |
| 14:59:09 | mriedem | oh i know why it's not failing | |
| 14:59:17 | mriedem | the nova-manage command doesn't return a non-0 | |
| 14:59:27 | mriedem | that's why it dumps out the breadcrumb | |
| 14:59:33 | mriedem | "Has "nova-manage api_db sync" been run?" | |
| 15:00:03 | openstackgerrit | Merged openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | |
| 15:00:05 | tssurya | mriedem: is this because api DB was not synced before running db sync ? | |
| 15:00:26 | mriedem | yes | |
| 15:00:46 | tssurya | ah ok | |
| 15:04:47 | mriedem | the order in grenade for the db sync matches what we have in our upgrade docs too https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | |
| 15:07:50 | mriedem | tssurya: odyssey4me: superdan: jgwentworth: https://bugs.launchpad.net/grenade/+bug/1761775 | |
| 15:07:50 | openstack | Launchpad bug 1761775 in grenade "InternalError during nova-manage db sync when upgrading to Rocky" [Undecided,Confirmed] | |
| 15:07:58 | mriedem | i listed 2 options in there for fixing grenade | |
| 15:08:13 | mriedem | trying to channel what sdague would say... | |
| 15:08:40 | fried_rice | mriedem: Care to +W https://review.openstack.org/#/c/555961/ (lower-constraints.txt) quick before we break it again? | |
| 15:08:40 | tssurya | hmm question.. why wasn't this an issue before me adding a new column to cell_mappings ? | |
| 15:09:12 | tssurya | I mean no other column change in api_db before has needed a api sync before db sync situation ? | |
| 15:09:29 | odyssey4me | mriedem yeah, that does seem a bit chicken-and-egg | |
| 15:09:55 | cdent | finucannot: +many on the -W stuff. pain now for less pain later | |
| 15:10:09 | mriedem | tssurya: we just haven't modified the cell_mappings table in forever | |
| 15:10:41 | tssurya | mriedem: cell_mappings yes, what about other tables ? | |
| 15:10:57 | mriedem | db sync doesn't hit other tables | |
| 15:11:01 | mriedem | in the api db | |
| 15:11:11 | jgwentworth | mriedem: interesting, I always thought the order was api_db first in the upgrade docs. I would have thought that one should be done first and I can't immediately think of why it's not | |
| 15:11:17 | superdan | mriedem: oh is it trying to read the cell mappings to do the cell databases before it has sync'd the api db? | |
| 15:11:27 | superdan | yeah I'm not sure why api isn't first | |
| 15:11:32 | finucannot | cdent: Yup, only way to do it, unfortunately | |
| 15:11:35 | mriedem | superdan: nova-manage db sync by default tries to also sync cell0- | |
| 15:11:37 | tssurya | mriedem: ah heh yes, anyways I thought api_sync was run before db sync until now | |
| 15:11:59 | superdan | mriedem: and it looks up the mapping for it before it has sync'd the api db? | |
| 15:12:18 | tssurya | superdan: yes | |
| 15:12:32 | mriedem | yeah | |
| 15:12:58 | superdan | er, I guess t's just that we call db sync before api_db sync in grenade you mean | |
| 15:13:27 | mriedem | right | |
| 15:13:31 | mriedem | that's one possible fix | |
| 15:13:31 | superdan | mriedem: tbh, I imagine that it's second because the main db sync was there first, and we were adding stuff, so appending it to the end | |
| 15:13:52 | superdan | I don't think that was an intentional decision | |
| 15:14:00 | mriedem | superdan: maybe https://review.openstack.org/#/c/190399/ | |
| 15:14:15 | mriedem | wow mitaka | |
| 15:14:23 | superdan | I'm kinda surprised actually since you would have to sync the api db at day zero to get cell0 to work at all, unless it's just using config for that one | |
| 15:14:49 | mriedem | ocata->pike would use devstack to setup ocata first though wouldn't it? | |
| 15:14:55 | mriedem | and devstack does the base install with the api db sync first | |
| 15:15:11 | superdan | ah, right, right | |
| 15:15:17 | mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L724 | |
| 15:15:31 | mriedem | then later: https://github.com/openstack-dev/devstack/blob/master/lib/nova#L741 | |
| 15:15:44 | superdan | yeah I was thinking about my superconductor changes and was like "I was sure api goes first" | |
| 15:15:51 | superdan | but this is grenade, which matters less | |
| 15:15:55 | superdan | because devstack has already run | |
| 15:15:57 | superdan | so that makes sense | |
| 15:16:27 | mriedem | fried_rice: someone is goign to have to tl;dr that for me - will that lower-constraints.txt file get synced automagically by the reqs bot? | |
| 15:16:44 | superdan | someone else should write the grenade patch so I can +2 it | |
| 15:16:47 | fried_rice | mriedem: No, it's manual. | |
| 15:17:02 | fried_rice | mriedem: There was a ML thread by dhellmann | |
| 15:17:11 | mriedem | yeah i saw the novel | |
| 15:17:13 | tssurya | superdan: I can.. | |
| 15:17:14 | mriedem | again, tl;dr | |
| 15:17:28 | mriedem | fried_rice: also, is that job supposed to actually run tests? http://logs.openstack.org/61/555961/3/check/openstack-tox-lower-constraints/91b6d17/job-output.txt.gz#_2018-04-06_13_19_18_865706 | |
| 15:17:30 | mriedem | b/c it doesn't | |
| 15:17:35 | superdan | and we should change those docs, I imagine they're in that order for the same reason | |
| 15:18:23 | mriedem | fried_rice: so -1 on that | |
| 15:19:24 | superdan | tssurya: okay I'll tweak the doc | |
| 15:19:29 | fried_rice | mriedem: Okay. The requirements-check job sure works (cause that's what held up PS1 and PS2). But yeah, I don't know anything beyond that. | |
| 15:19:36 | tssurya | although mriedem says he prefers a "Add a from-queens script" to swapping | |
| 15:19:52 | mriedem | no i don't prefer that | |
| 15:20:04 | mriedem | i said, "Option #1 seems a bit shady, but option #2 seems excessive, especially if we're going to be changing things in the API DB in future releases." | |
| 15:20:19 | mriedem | in other words, i'm hedging | |
| 15:20:35 | tssurya | okay sorry so its the shady one then | |
| 15:21:16 | superdan | I don't think it's shady, I think it's been wrong for a long time, hidden by the fact that we run devstack (in the proper order) first before grenade | |
| 15:21:18 | mriedem | if we added a from-queens script, per grenade guidelines, we'd also need a nova release note in rocky saying you need to run nova-manage api_db sync before nova-manage db sync | |
| 15:21:20 | jgwentworth | heh, I think it was shady assuming there was a good reason for the current ordering, which there apparently isn't | |
| 15:21:31 | superdan | yeah | |
| 15:21:43 | odyssey4me | mriedem alright, I've pushed up an OSA patch to test out the shady option: https://review.openstack.org/#/c/559354/1/tasks/nova_db_setup.yml | |
| 15:21:54 | tssurya | superdan : should I do the docs in the same one then ? | |
| 15:22:03 | odyssey4me | that'll make api_db sync before before db | |
| 15:22:06 | superdan | tssurya: I'm doing the doc patch, if you can do grenade | |
| 15:22:16 | tssurya | superdan: okay :) | |
| 15:22:28 | odyssey4me | go.... TEAM! ;) | |
| 15:22:36 | mriedem | odyssey4me: that will affect both base install and upgrade? | |
| 15:22:46 | odyssey4me | mriedem yep | |
| 15:23:08 | jgwentworth | \o/ | |
| 15:24:28 | fried_rice | mriedem: Looking over dhellmann's email again, it's possible that the no-op-ness of that job is intentional; to be switched on at some point in the future once the dust has settled. This is just setting up the framework. | |
| 15:24:56 | mriedem | fried_rice: well the commit message is clearly misleading then | |
| 15:25:17 | fried_rice | mriedem: But... I guess I'll let dhellmann respond to your -1. I was just trying to take over since he's got a few hundred of these up in the air. | |
| 15:25:29 | jgwentworth | so we set our constraints manually and the job makes sure all of the components work together with the versions they've required? | |
| 15:25:34 | odyssey4me | mriedem if install must do a different order to upgrade, then that's not awesome - but we can adjust... but as it stands now it looks like our api_db sync happened first on upgrade only, but the db sync didn't happen before the cell mapping... so I can adjust the sequence to just move when the db sync happens if that's more sensible | |
| 15:25:59 | mriedem | odyssey4me: this might be fine, we'll see; i was wondering why there were redundant api db syncs in there before | |
| 15:26:32 | mriedem | i guess the first one was trying to be sly | |
| 15:26:40 | mriedem | and a one time only thing on install | |