| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 19:38:50 | melwitt | mnaser: alternatively, could switch to defaulting to True and then let operators turn it off and decommission nova-consoleauth intentionally once they've rolled everything to rocky | |
| 19:38:51 | cdent | that's part of that message I sent earlier today | |
| 19:39:13 | cdent | mriedem: but my thinking was: we don't need to do any online db migrations, yet, either | |
| 19:39:52 | cdent | that is: Isn't the database in the correct state when someone gets to using openstack/placement? | |
| 19:40:42 | mnaser | melwitt: that feels like a better user experience | |
| 19:40:50 | mnaser | a lot of users probably will do rolling upgrades | |
| 19:41:06 | mriedem | cdent: nope | |
| 19:41:07 | mnaser | and you can keep it =True for rocky only anyways and remove it after | |
| 19:41:18 | mriedem | cdent: not if you're upgrading from <pike to stein | |
| 19:41:29 | mriedem | like mnaser is doing with going from juno to rocky | |
| 19:41:47 | dansmith | melwitt: workarounds are supposed to default to off | |
| 19:41:56 | melwitt | mnaser: you can create a bug, that would be most visible I think | |
| 19:41:58 | dansmith | melwitt: so really we should have landed it where false meant what we wanted | |
| 19:42:31 | cdent | mriedem: I'm confused on how that's supposed to work: don' t you stop on the rocky _code_ when doing a FFU? | |
| 19:42:32 | dansmith | changing it again is kindof the suck too, IMHO | |
| 19:42:34 | melwitt | dansmith: ok :( I see | |
| 19:42:57 | mnaser | melwitt: dansmith i guess i'm pretty busy these times but i | |
| 19:43:10 | mnaser | i'll put up a bug and leave it for the team to decide whats best :> | |
| 19:43:31 | mriedem | cdent: you mean upgrade to rocky where the create_incomplete_consumeres online migration runs as part of the rocky nova-manage db online_data_migrations, and then extract placement while upgrading to stein? | |
| 19:43:39 | mriedem | and assume create_incomplete_consumers is done already? | |
| 19:43:58 | mriedem | that might happen | |
| 19:44:09 | mriedem | it's just nice to have a blocker migration to prevent you from upgrading if you didn't do the homework | |
| 19:44:25 | mriedem | we don't always have those though b/c sometimes they span multiple DBs | |
| 19:44:29 | cdent | the term "blocker migration" has never been sufficiently defined for me | |
| 19:44:32 | mriedem | we've used nova-status upgrade check for that though | |
| 19:44:38 | mriedem | as in db sync fails | |
| 19:44:54 | mriedem | db sync in N fails b/c you didn't complete the online migrations in N-1 | |
| 19:45:11 | melwitt | mnaser: thanks. at the very least, we can add more info to the upgrade reno, I think. but it sounds like I messed up the [workarounds] option too much to fix | |
| 19:45:36 | mnaser | melwitt: nah, it's fine, i think the messaging needs to be more clear as in like | |
| 19:45:42 | mnaser | yo shit will be broken if you havent completed the upgrade | |
| 19:45:54 | mnaser | because to me i saw some stuff related to it but it didnt really feel like "that was my issue" | |
| 19:46:10 | dansmith | melwitt: just MHO of course. We've deviated from my initial proposal of workarounds in the past, but I do think that adding another possible "how it is if you didn't change it" case at this point is probably less helpful | |
| 19:46:22 | melwitt | well, I was thinking "if you're doing a rolling upgrade, set [workarounds]enable_consoleauth = True | |
| 19:46:50 | melwitt | mnaser: ^ rather than, yo shit will be broken | |
| 19:46:52 | mnaser | yeah i'd be in favour of enabling it given that it cant really do much | |
| 19:47:04 | mriedem | cdent: i couldn't do a block migration for this request spec thing, so added it to nova-status upgrade check https://review.openstack.org/#/c/581813/ | |
| 19:47:10 | mriedem | *blocker migration | |
| 19:47:14 | mnaser | melwitt: much better delivered. | |
| 19:47:15 | mnaser | :p | |
| 19:47:27 | openstack | Launchpad bug 1798188 in OpenStack Compute (nova) "VNC stops working in rolling upgrade by default" [Undecided,New] | |
| 19:47:27 | mnaser | melwitt: and my sh.... rolling upgrades were affected by https://bugs.launchpad.net/nova/+bug/1798188 | |
| 19:47:32 | melwitt | lol, no I mean it _won't_ be broken if you set it | |
| 19:47:49 | mnaser | oh | |
| 19:47:55 | mnaser | ah my brain has potato'd | |
| 19:48:00 | cdent | mriedem perhaps it would be good/ideal if we can instead of a suite of N blocker migrations we have a sanity check of some kind in a placement status check, which effectively does the same kind of "You're database isn't ready" thing. | |
| 19:48:09 | mnaser | i did ocata => pike => queens => rocky in 2 days | |
| 19:48:21 | mriedem | cdent: that's an optoin | |
| 19:48:22 | mriedem | *option | |
| 19:48:27 | melwitt | dansmith: I didn't understand the "how it is if you didn't change it" are you saying you think changing the default would be less helpful than adding more words to the upgrade reno? | |
| 19:48:29 | mriedem | for dropping create_incomplete_consumers | |
| 19:48:29 | mnaser | this one was by far the toughest but as expected i guess | |
| 19:49:06 | dansmith | melwitt: I'm saying if you change it now, then people reading the renos will see "This was deprecated. Oopps, undeprecated set this workaround, Oops Oops, nevermind, it's set by default now" | |
| 19:49:26 | dansmith | melwitt: and it just seems like we're piling on the confusion if we keep making that a moving target | |
| 19:49:49 | dansmith | melwitt: if anything, add something to nova-status and backport it to help make sure people are warned to pay attention to this | |
| 19:49:57 | dansmith | and get that released before people have a chance to stumble over this | |
| 19:50:56 | melwitt | dansmith: I see, yeah. that's true, if we change the default we have to change all of the words related to how the workaround works, that would be confusing if someone's seen it before | |
| 19:51:26 | mnaser | (but also how many people went through this document already given the issue i ran into today :p) | |
| 19:51:35 | dansmith | melwitt: and I don't think we get to alter the older renos, if I'm not mistaken, but even still it's out there so if someone is looking at X.1 docs and then they're similar but different in X.2... | |
| 19:52:14 | melwitt | argh, yeah. | |
| 19:52:19 | dansmith | mnaser: I'm talking about in a year when most people are deploying rocky and trying to figure out what the story is now | |
| 19:52:37 | mnaser | dansmith: makes sense | |
| 19:53:11 | melwitt | I have cursed nova-consoleauth :( | |
| 19:53:41 | melwitt | ok, so add something to nova-status. I hope everyone uses nova-status | |
| 19:55:20 | dansmith | of course not everyone does.. OSA does I think, and hopefully all the buzz around making this a generic thing will mean in a year people are looking at it | |
| 19:55:35 | dansmith | I thought you were also suggesting clarifying words in renos to help understand | |
| 19:55:56 | dansmith | I was just saying flipping the default behavior now and trying to document _that_ is the confusing part | |
| 19:55:59 | melwitt | yeah but IIUC that doesn't help someone upgrading to rocky if I can't backport those words | |
| 19:56:08 | melwitt | oh | |
| 19:56:12 | dansmith | you can backport the words, | |
| 19:56:33 | dansmith | I just don't think you should change the behavior and backport more words explaining how it's changed for the third time | |
| 19:56:45 | melwitt | got it, ok | |
| 19:57:30 | mriedem | what would the nova-status upgrade check look for? that [workarounds]/enable_consoleauth is False and return a warning? | |
| 19:58:02 | dansmith | yeah, and maybe check the services table or current tokens to see if you even use that stuff | |
| 19:58:09 | dansmith | if you don't use console, then you don't need to warn, | |
| 19:58:26 | dansmith | but if you do and you're rolling, pretty much should have that set right? | |
| 19:58:33 | melwitt | yeah | |
| 19:58:45 | mriedem | you can also tell if there are no console auth entries in the db right? | |
| 19:58:50 | dansmith | I said that | |
| 19:59:08 | mriedem | i said it with an accent | |
| 19:59:19 | dansmith | fancy | |
| 19:59:21 | mriedem | dansmith: btw, https://review.openstack.org/#/c/611094/ needs an assertion on it | |
| 19:59:23 | melwitt | there wouldn't be, before rocky though. they'd be in the nova-consoleauth service | |
| 19:59:31 | mriedem | melwitt: well, that's the point right? | |
| 19:59:45 | mriedem | if you're using nova-consoleauth in queens, and upgrading to rocky, you want the workaround enabled | |
| 20:00:05 | mriedem | and nova-consoleauth would show up in the services table in....one of the dbs | |
| 20:00:05 | dansmith | mriedem: ack, I have to run off for a bit but will hit that when I get back | |
| 20:00:08 | melwitt | yeah, I mean, if you are checking a queens deployment for whether they use consoles at all, you'd have to check the nova-consoleauth service | |
| 20:00:26 | mriedem | we don't want to make an rpc call from the status check | |
| 20:00:32 | mriedem | but we could check the services table to see if it's been started | |
| 20:00:35 | melwitt | oh, you're thinking if they don't use consoles they won't run the service at all. that makes sense too | |
| 20:00:37 | melwitt | yeah | |
| 20:00:51 | mriedem | i just don't know which db that'd be in | |
| 20:00:58 | mriedem | api? | |
| 20:01:00 | mriedem | no, | |
| 20:01:02 | mriedem | wrong schema | |
| 20:01:09 | mriedem | i guess just iterate the cell dbs | |
| 20:01:40 | mriedem | if you find a non-deleted nova-consoleauth service record in that db, but no console auth tokens in the db, and workarounds is false, then fail | |
| 20:02:56 | melwitt | yeah, or warn like dansmith said. only matters if you're rolling | |
| 20:03:33 | melwitt | i.e. it will only mess you up if you're rolling | |
| 20:04:43 | mriedem | i left a comment on the bug with the status ugprade check idea | |