| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 19:13:02 | mnaser | yeah besides nova | |
| 19:13:04 | mnaser | no api interaction | |
| 19:15:56 | efried | If you see a repro, lmk. Otherwise I'm going to pretend it didn't happen. | |
| 19:17:35 | mnaser | efried: i can get you a stacktrace if you want, but i dont think id be able to reproduce it given i deleted stuff | |
| 19:18:45 | efried | mnaser: The stack trace won't tell me much. Logs up to that point might help a bit. | |
| 19:19:15 | efried | especially if they've got our fun new debug messages | |
| 19:20:08 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 19:20:33 | edmondsw | efried ^ this should address the fd open issue | |
| 19:23:41 | mriedem | fudge, | |
| 19:23:48 | mriedem | this unique constraint error is in 3 f'ing places | |
| 19:24:35 | melwitt | I wondered if there were more. and I had thought they'd call through the same method to create missing consumers but I guess all of the queries are different | |
| 19:24:58 | mriedem | well maybe not | |
| 19:24:58 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/resource_provider.py@1973 | |
| 19:25:04 | mriedem | that's not doing the insert-from-select | |
| 19:25:06 | mriedem | like the others | |
| 19:26:22 | mnaser | um | |
| 19:26:29 | mnaser | in rocky we moved to console auth tokens stored in db, right? | |
| 19:27:16 | melwitt | yes, in addition to nova-consoleauth until this lands https://review.openstack.org/610673 | |
| 19:28:21 | mnaser | melwitt: what service creates the auth tokens? | |
| 19:28:59 | melwitt | mnaser: nova-compute creates them for the database, nova-consoleauth creates them for nova-consoleauth | |
| 19:29:15 | mnaser | so if your nova-compute is not on rocky | |
| 19:29:29 | mnaser | ..does that mean no console? | |
| 19:29:29 | melwitt | then you get nova-consoleauth tokens | |
| 19:29:40 | mnaser | ok i see | |
| 19:29:41 | melwitt | no, you get console | |
| 19:29:54 | mnaser | so just an extra indirection right now | |
| 19:30:03 | mnaser | till nova-compute creates to db directly in the future | |
| 19:30:45 | melwitt | nova-compute creates directly to db in rocky. just obviously your older computes will not and those instances will be supported by nova-consoleauth | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:31:11 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 19:31:11 | mriedem | mnaser: melwitt: dansmith: cdent: efried: ^ should be good now, covers both cases | |
| 19:31:27 | melwitt | once all your computes are on rocky, then you wouldn't need nova-consoleauth once this backport lands https://review.openstack.org/610673 | |
| 19:32:31 | mnaser | melwitt: im seeing consoleauth get a token, but the traceback that says token validation failed is resulted from a method that does db.console_auth_token_get | |
| 19:33:21 | mnaser | ugh | |
| 19:33:27 | mnaser | [workarounds] enable_consoleauth=True | |
| 19:33:40 | melwitt | mnaser: and you have a mix of rocky computes and older than rocky computes? in that case, you'll need to set [workarounds]enable_consoleauth = True on your console proxy host | |
| 19:36:48 | mnaser | melwitt: that was it, thank you | |
| 19:37:11 | mnaser | part of me wants to document all this in some "heads up" way, but also i worry "look how bad it is" messaging :\ | |
| 19:37:25 | cdent | thanks mriedem, is the expectation on that stuff that since that code is "remove in stein" or "called from online migrations" that we can remove it in openstack/placement instead of using those changes (I've not caught up fully on the irc log) | |
| 19:37:31 | melwitt | mnaser: it defaults to False, but for a rolling upgrade you would need it to be True. yet more information I missed in the upgrade release notes :( | |
| 19:37:41 | mriedem | cdent: i don't expect it will be removed in stein, | |
| 19:37:45 | mriedem | we have to have a blocker migration first | |
| 19:37:56 | mriedem | but haven't thought through it all yet | |
| 19:38:08 | mnaser | melwitt: np, want me to push up a bug or something or you'll write it down? | |
| 19:38:09 | mriedem | openstack/placement doesn't have a "placement-manage db online_data_migrations" yet | |
| 19:38:14 | mriedem | or does it? | |
| 19:38:15 | mnaser | maybe good to add to the upgrade doc | |
| 19:38:17 | mriedem | cdent: ^ | |
| 19:38:42 | cdent | it doesn't have _any_ placement-manage yet | |
| 19:38:50 | melwitt | mnaser: alternatively, could switch to defaulting to True and then let operators turn it off and decommission nova-consoleauth intentionally once they've rolled everything to rocky | |
| 19:38:51 | cdent | that's part of that message I sent earlier today | |
| 19:39:13 | cdent | mriedem: but my thinking was: we don't need to do any online db migrations, yet, either | |
| 19:39:52 | cdent | that is: Isn't the database in the correct state when someone gets to using openstack/placement? | |
| 19:40:42 | mnaser | melwitt: that feels like a better user experience | |
| 19:40:50 | mnaser | a lot of users probably will do rolling upgrades | |
| 19:41:06 | mriedem | cdent: nope | |
| 19:41:07 | mnaser | and you can keep it =True for rocky only anyways and remove it after | |
| 19:41:18 | mriedem | cdent: not if you're upgrading from <pike to stein | |
| 19:41:29 | mriedem | like mnaser is doing with going from juno to rocky | |
| 19:41:47 | dansmith | melwitt: workarounds are supposed to default to off | |
| 19:41:56 | melwitt | mnaser: you can create a bug, that would be most visible I think | |
| 19:41:58 | dansmith | melwitt: so really we should have landed it where false meant what we wanted | |
| 19:42:31 | cdent | mriedem: I'm confused on how that's supposed to work: don' t you stop on the rocky _code_ when doing a FFU? | |
| 19:42:32 | dansmith | changing it again is kindof the suck too, IMHO | |
| 19:42:34 | melwitt | dansmith: ok :( I see | |
| 19:42:57 | mnaser | melwitt: dansmith i guess i'm pretty busy these times but i | |
| 19:43:10 | mnaser | i'll put up a bug and leave it for the team to decide whats best :> | |
| 19:43:31 | mriedem | cdent: you mean upgrade to rocky where the create_incomplete_consumeres online migration runs as part of the rocky nova-manage db online_data_migrations, and then extract placement while upgrading to stein? | |
| 19:43:39 | mriedem | and assume create_incomplete_consumers is done already? | |
| 19:43:58 | mriedem | that might happen | |
| 19:44:09 | mriedem | it's just nice to have a blocker migration to prevent you from upgrading if you didn't do the homework | |
| 19:44:25 | mriedem | we don't always have those though b/c sometimes they span multiple DBs | |
| 19:44:29 | cdent | the term "blocker migration" has never been sufficiently defined for me | |
| 19:44:32 | mriedem | we've used nova-status upgrade check for that though | |
| 19:44:38 | mriedem | as in db sync fails | |
| 19:44:54 | mriedem | db sync in N fails b/c you didn't complete the online migrations in N-1 | |
| 19:45:11 | melwitt | mnaser: thanks. at the very least, we can add more info to the upgrade reno, I think. but it sounds like I messed up the [workarounds] option too much to fix | |
| 19:45:36 | mnaser | melwitt: nah, it's fine, i think the messaging needs to be more clear as in like | |
| 19:45:42 | mnaser | yo shit will be broken if you havent completed the upgrade | |
| 19:45:54 | mnaser | because to me i saw some stuff related to it but it didnt really feel like "that was my issue" | |
| 19:46:10 | dansmith | melwitt: just MHO of course. We've deviated from my initial proposal of workarounds in the past, but I do think that adding another possible "how it is if you didn't change it" case at this point is probably less helpful | |
| 19:46:22 | melwitt | well, I was thinking "if you're doing a rolling upgrade, set [workarounds]enable_consoleauth = True | |
| 19:46:50 | melwitt | mnaser: ^ rather than, yo shit will be broken | |
| 19:46:52 | mnaser | yeah i'd be in favour of enabling it given that it cant really do much | |
| 19:47:04 | mriedem | cdent: i couldn't do a block migration for this request spec thing, so added it to nova-status upgrade check https://review.openstack.org/#/c/581813/ | |
| 19:47:10 | mriedem | *blocker migration | |
| 19:47:14 | mnaser | melwitt: much better delivered. | |
| 19:47:15 | mnaser | :p | |
| 19:47:27 | mnaser | melwitt: and my sh.... rolling upgrades were affected by https://bugs.launchpad.net/nova/+bug/1798188 | |
| 19:47:27 | openstack | Launchpad bug 1798188 in OpenStack Compute (nova) "VNC stops working in rolling upgrade by default" [Undecided,New] | |
| 19:47:32 | melwitt | lol, no I mean it _won't_ be broken if you set it | |
| 19:47:49 | mnaser | oh | |
| 19:47:55 | mnaser | ah my brain has potato'd | |
| 19:48:00 | cdent | mriedem perhaps it would be good/ideal if we can instead of a suite of N blocker migrations we have a sanity check of some kind in a placement status check, which effectively does the same kind of "You're database isn't ready" thing. | |
| 19:48:09 | mnaser | i did ocata => pike => queens => rocky in 2 days | |
| 19:48:21 | mriedem | cdent: that's an optoin | |
| 19:48:22 | mriedem | *option | |
| 19:48:27 | melwitt | dansmith: I didn't understand the "how it is if you didn't change it" are you saying you think changing the default would be less helpful than adding more words to the upgrade reno? | |
| 19:48:29 | mnaser | this one was by far the toughest but as expected i guess | |