| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 19:30:03 | mnaser | till nova-compute creates to db directly in the future | |
| 19:30:45 | melwitt | nova-compute creates directly to db in rocky. just obviously your older computes will not and those instances will be supported by nova-consoleauth | |
| 19:31:11 | mriedem | mnaser: melwitt: dansmith: cdent: efried: ^ should be good now, covers both cases | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 19:31:11 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:31:27 | melwitt | once all your computes are on rocky, then you wouldn't need nova-consoleauth once this backport lands https://review.openstack.org/610673 | |
| 19:32:31 | mnaser | melwitt: im seeing consoleauth get a token, but the traceback that says token validation failed is resulted from a method that does db.console_auth_token_get | |
| 19:33:21 | mnaser | ugh | |
| 19:33:27 | mnaser | [workarounds] enable_consoleauth=True | |
| 19:33:40 | melwitt | mnaser: and you have a mix of rocky computes and older than rocky computes? in that case, you'll need to set [workarounds]enable_consoleauth = True on your console proxy host | |
| 19:36:48 | mnaser | melwitt: that was it, thank you | |
| 19:37:11 | mnaser | part of me wants to document all this in some "heads up" way, but also i worry "look how bad it is" messaging :\ | |
| 19:37:25 | cdent | thanks mriedem, is the expectation on that stuff that since that code is "remove in stein" or "called from online migrations" that we can remove it in openstack/placement instead of using those changes (I've not caught up fully on the irc log) | |
| 19:37:31 | melwitt | mnaser: it defaults to False, but for a rolling upgrade you would need it to be True. yet more information I missed in the upgrade release notes :( | |
| 19:37:41 | mriedem | cdent: i don't expect it will be removed in stein, | |
| 19:37:45 | mriedem | we have to have a blocker migration first | |
| 19:37:56 | mriedem | but haven't thought through it all yet | |
| 19:38:08 | mnaser | melwitt: np, want me to push up a bug or something or you'll write it down? | |
| 19:38:09 | mriedem | openstack/placement doesn't have a "placement-manage db online_data_migrations" yet | |
| 19:38:14 | mriedem | or does it? | |
| 19:38:15 | mnaser | maybe good to add to the upgrade doc | |
| 19:38:17 | mriedem | cdent: ^ | |
| 19:38:42 | cdent | it doesn't have _any_ placement-manage yet | |
| 19:38:50 | melwitt | mnaser: alternatively, could switch to defaulting to True and then let operators turn it off and decommission nova-consoleauth intentionally once they've rolled everything to rocky | |
| 19:38:51 | cdent | that's part of that message I sent earlier today | |
| 19:39:13 | cdent | mriedem: but my thinking was: we don't need to do any online db migrations, yet, either | |
| 19:39:52 | cdent | that is: Isn't the database in the correct state when someone gets to using openstack/placement? | |
| 19:40:42 | mnaser | melwitt: that feels like a better user experience | |
| 19:40:50 | mnaser | a lot of users probably will do rolling upgrades | |
| 19:41:06 | mriedem | cdent: nope | |
| 19:41:07 | mnaser | and you can keep it =True for rocky only anyways and remove it after | |
| 19:41:18 | mriedem | cdent: not if you're upgrading from <pike to stein | |
| 19:41:29 | mriedem | like mnaser is doing with going from juno to rocky | |
| 19:41:47 | dansmith | melwitt: workarounds are supposed to default to off | |
| 19:41:56 | melwitt | mnaser: you can create a bug, that would be most visible I think | |
| 19:41:58 | dansmith | melwitt: so really we should have landed it where false meant what we wanted | |
| 19:42:31 | cdent | mriedem: I'm confused on how that's supposed to work: don' t you stop on the rocky _code_ when doing a FFU? | |
| 19:42:32 | dansmith | changing it again is kindof the suck too, IMHO | |
| 19:42:34 | melwitt | dansmith: ok :( I see | |
| 19:42:57 | mnaser | melwitt: dansmith i guess i'm pretty busy these times but i | |
| 19:43:10 | mnaser | i'll put up a bug and leave it for the team to decide whats best :> | |
| 19:43:31 | mriedem | cdent: you mean upgrade to rocky where the create_incomplete_consumeres online migration runs as part of the rocky nova-manage db online_data_migrations, and then extract placement while upgrading to stein? | |
| 19:43:39 | mriedem | and assume create_incomplete_consumers is done already? | |
| 19:43:58 | mriedem | that might happen | |
| 19:44:09 | mriedem | it's just nice to have a blocker migration to prevent you from upgrading if you didn't do the homework | |
| 19:44:25 | mriedem | we don't always have those though b/c sometimes they span multiple DBs | |
| 19:44:29 | cdent | the term "blocker migration" has never been sufficiently defined for me | |
| 19:44:32 | mriedem | we've used nova-status upgrade check for that though | |
| 19:44:38 | mriedem | as in db sync fails | |
| 19:44:54 | mriedem | db sync in N fails b/c you didn't complete the online migrations in N-1 | |
| 19:45:11 | melwitt | mnaser: thanks. at the very least, we can add more info to the upgrade reno, I think. but it sounds like I messed up the [workarounds] option too much to fix | |
| 19:45:36 | mnaser | melwitt: nah, it's fine, i think the messaging needs to be more clear as in like | |
| 19:45:42 | mnaser | yo shit will be broken if you havent completed the upgrade | |
| 19:45:54 | mnaser | because to me i saw some stuff related to it but it didnt really feel like "that was my issue" | |
| 19:46:10 | dansmith | melwitt: just MHO of course. We've deviated from my initial proposal of workarounds in the past, but I do think that adding another possible "how it is if you didn't change it" case at this point is probably less helpful | |
| 19:46:22 | melwitt | well, I was thinking "if you're doing a rolling upgrade, set [workarounds]enable_consoleauth = True | |
| 19:46:50 | melwitt | mnaser: ^ rather than, yo shit will be broken | |
| 19:46:52 | mnaser | yeah i'd be in favour of enabling it given that it cant really do much | |
| 19:47:04 | mriedem | cdent: i couldn't do a block migration for this request spec thing, so added it to nova-status upgrade check https://review.openstack.org/#/c/581813/ | |
| 19:47:10 | mriedem | *blocker migration | |
| 19:47:14 | mnaser | melwitt: much better delivered. | |
| 19:47:15 | mnaser | :p | |
| 19:47:27 | openstack | Launchpad bug 1798188 in OpenStack Compute (nova) "VNC stops working in rolling upgrade by default" [Undecided,New] | |
| 19:47:27 | mnaser | melwitt: and my sh.... rolling upgrades were affected by https://bugs.launchpad.net/nova/+bug/1798188 | |
| 19:47:32 | melwitt | lol, no I mean it _won't_ be broken if you set it | |
| 19:47:49 | mnaser | oh | |
| 19:47:55 | mnaser | ah my brain has potato'd | |
| 19:48:00 | cdent | mriedem perhaps it would be good/ideal if we can instead of a suite of N blocker migrations we have a sanity check of some kind in a placement status check, which effectively does the same kind of "You're database isn't ready" thing. | |
| 19:48:09 | mnaser | i did ocata => pike => queens => rocky in 2 days | |
| 19:48:21 | mriedem | cdent: that's an optoin | |
| 19:48:22 | mriedem | *option | |
| 19:48:27 | melwitt | dansmith: I didn't understand the "how it is if you didn't change it" are you saying you think changing the default would be less helpful than adding more words to the upgrade reno? | |
| 19:48:29 | mriedem | for dropping create_incomplete_consumers | |
| 19:48:29 | mnaser | this one was by far the toughest but as expected i guess | |
| 19:49:06 | dansmith | melwitt: I'm saying if you change it now, then people reading the renos will see "This was deprecated. Oopps, undeprecated set this workaround, Oops Oops, nevermind, it's set by default now" | |
| 19:49:26 | dansmith | melwitt: and it just seems like we're piling on the confusion if we keep making that a moving target | |
| 19:49:49 | dansmith | melwitt: if anything, add something to nova-status and backport it to help make sure people are warned to pay attention to this | |
| 19:49:57 | dansmith | and get that released before people have a chance to stumble over this | |
| 19:50:56 | melwitt | dansmith: I see, yeah. that's true, if we change the default we have to change all of the words related to how the workaround works, that would be confusing if someone's seen it before | |
| 19:51:26 | mnaser | (but also how many people went through this document already given the issue i ran into today :p) | |
| 19:51:35 | dansmith | melwitt: and I don't think we get to alter the older renos, if I'm not mistaken, but even still it's out there so if someone is looking at X.1 docs and then they're similar but different in X.2... | |
| 19:52:14 | melwitt | argh, yeah. | |
| 19:52:19 | dansmith | mnaser: I'm talking about in a year when most people are deploying rocky and trying to figure out what the story is now | |
| 19:52:37 | mnaser | dansmith: makes sense | |
| 19:53:11 | melwitt | I have cursed nova-consoleauth :( | |
| 19:53:41 | melwitt | ok, so add something to nova-status. I hope everyone uses nova-status | |
| 19:55:20 | dansmith | of course not everyone does.. OSA does I think, and hopefully all the buzz around making this a generic thing will mean in a year people are looking at it | |
| 19:55:35 | dansmith | I thought you were also suggesting clarifying words in renos to help understand | |
| 19:55:56 | dansmith | I was just saying flipping the default behavior now and trying to document _that_ is the confusing part | |
| 19:55:59 | melwitt | yeah but IIUC that doesn't help someone upgrading to rocky if I can't backport those words | |
| 19:56:08 | melwitt | oh | |
| 19:56:12 | dansmith | you can backport the words, | |
| 19:56:33 | dansmith | I just don't think you should change the behavior and backport more words explaining how it's changed for the third time | |
| 19:56:45 | melwitt | got it, ok | |
| 19:57:30 | mriedem | what would the nova-status upgrade check look for? that [workarounds]/enable_consoleauth is False and return a warning? | |
| 19:58:02 | dansmith | yeah, and maybe check the services table or current tokens to see if you even use that stuff | |
| 19:58:09 | dansmith | if you don't use console, then you don't need to warn, | |
| 19:58:26 | dansmith | but if you do and you're rolling, pretty much should have that set right? | |
| 19:58:33 | melwitt | yeah | |