| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 18:55:29 | melwitt | I thought we started using 1.8 in pike, that's when it was added | |
| 18:56:27 | mriedem | yeah i guess https://review.openstack.org/#/c/469634/ | |
| 18:58:13 | mriedem | ok i guess that solves the grenade mystery | |
| 18:58:29 | mriedem | geez when is someone going to add an FFU job that runs from ocata-em to master?! | |
| 19:01:00 | mnaser | issues like this is why ffu upgrades terrify me | |
| 19:01:00 | mnaser | lol | |
| 19:01:15 | mnaser | "when did this break? here's 2 years worth of code to go through!" | |
| 19:01:43 | dansmith | mnaser: it's way easier than the alternative, IMHO | |
| 19:01:54 | dansmith | of not knowing if the data set has been transformed since juno or not | |
| 19:02:05 | mnaser | dansmith: i'll agree on that statement | |
| 19:03:14 | mriedem | well, i was right about one thing | |
| 19:03:15 | mriedem | http://logs.openstack.org/00/607600/1/check/ironic-grenade-dsvm/4d493b1/logs/screen-n-cpu.txt.gz#_Oct_03_18_33_59_072341 | |
| 19:04:05 | melwitt | hm, I just realized, we're going to need to dupe these patches and use the same change-id to propose them to placement as well | |
| 19:04:13 | mriedem | yes | |
| 19:04:16 | mriedem | thta's what we've been doing | |
| 19:04:23 | melwitt | ok | |
| 19:04:28 | mnaser | forward porting | |
| 19:04:31 | mnaser | is that what we call it | |
| 19:05:00 | melwitt | on the second patch, it looks like there's at least one additional place we need to add the group_by, right? | |
| 19:05:15 | mriedem | yes | |
| 19:05:30 | melwitt | and should correspondingly test it too. the recreate test patch is already approved though | |
| 19:05:46 | mriedem | not for long | |
| 19:05:51 | melwitt | k | |
| 19:05:57 | efried | mriedem: That KeyError. Is that part of the existing bugs you've been talking about, or has it not yet been investigated? | |
| 19:06:13 | mnaser | the keyerror is not related | |
| 19:06:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:06:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:06:22 | mnaser | it was just some weird leftovers | |
| 19:06:45 | mriedem | mnaser: i'll rev my functional test patch and yours on top | |
| 19:06:52 | efried | Is it a bug that needs to be fixed, or was it a user error? | |
| 19:07:09 | mnaser | the keyerror? i dunno, but i dont think it should have been an issue because said user doesnt touch placement | |
| 19:07:13 | efried | I wouldn't have thought it should be possible no matter what abuse you lavish on the placement db. | |
| 19:07:27 | mnaser | fwiw the resource provider had nothing allocated | |
| 19:07:29 | mnaser | no usage that is | |
| 19:08:06 | efried | mnaser: Any sharing providers in this mess? | |
| 19:08:24 | mnaser | efried: sorry, not sure what you mean by that | |
| 19:08:47 | efried | Providers with the MISC_SHARES_VIA_AGGREGATE trait | |
| 19:10:00 | mnaser | efried: i am not sure hoenstly, i didn't dig in that much | |
| 19:12:42 | mriedem | then no | |
| 19:12:46 | mriedem | b/c you'd have to create them yourself | |
| 19:13:02 | mnaser | yeah besides nova | |
| 19:13:04 | mnaser | no api interaction | |
| 19:15:56 | efried | If you see a repro, lmk. Otherwise I'm going to pretend it didn't happen. | |
| 19:17:35 | mnaser | efried: i can get you a stacktrace if you want, but i dont think id be able to reproduce it given i deleted stuff | |
| 19:18:45 | efried | mnaser: The stack trace won't tell me much. Logs up to that point might help a bit. | |
| 19:19:15 | efried | especially if they've got our fun new debug messages | |
| 19:20:08 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 19:20:33 | edmondsw | efried ^ this should address the fd open issue | |
| 19:23:41 | mriedem | fudge, | |
| 19:23:48 | mriedem | this unique constraint error is in 3 f'ing places | |
| 19:24:35 | melwitt | I wondered if there were more. and I had thought they'd call through the same method to create missing consumers but I guess all of the queries are different | |
| 19:24:58 | mriedem | well maybe not | |
| 19:24:58 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/resource_provider.py@1973 | |
| 19:25:04 | mriedem | that's not doing the insert-from-select | |
| 19:25:06 | mriedem | like the others | |
| 19:26:22 | mnaser | um | |
| 19:26:29 | mnaser | in rocky we moved to console auth tokens stored in db, right? | |
| 19:27:16 | melwitt | yes, in addition to nova-consoleauth until this lands https://review.openstack.org/610673 | |
| 19:28:21 | mnaser | melwitt: what service creates the auth tokens? | |
| 19:28:59 | melwitt | mnaser: nova-compute creates them for the database, nova-consoleauth creates them for nova-consoleauth | |
| 19:29:15 | mnaser | so if your nova-compute is not on rocky | |
| 19:29:29 | mnaser | ..does that mean no console? | |
| 19:29:29 | melwitt | then you get nova-consoleauth tokens | |
| 19:29:40 | mnaser | ok i see | |
| 19:29:41 | melwitt | no, you get console | |
| 19:29:54 | mnaser | so just an extra indirection right now | |
| 19:30:03 | mnaser | till nova-compute creates to db directly in the future | |
| 19:30:45 | melwitt | nova-compute creates directly to db in rocky. just obviously your older computes will not and those instances will be supported by nova-consoleauth | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:31:11 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 19:31:11 | mriedem | mnaser: melwitt: dansmith: cdent: efried: ^ should be good now, covers both cases | |
| 19:31:27 | melwitt | once all your computes are on rocky, then you wouldn't need nova-consoleauth once this backport lands https://review.openstack.org/610673 | |
| 19:32:31 | mnaser | melwitt: im seeing consoleauth get a token, but the traceback that says token validation failed is resulted from a method that does db.console_auth_token_get | |
| 19:33:21 | mnaser | ugh | |
| 19:33:27 | mnaser | [workarounds] enable_consoleauth=True | |
| 19:33:40 | melwitt | mnaser: and you have a mix of rocky computes and older than rocky computes? in that case, you'll need to set [workarounds]enable_consoleauth = True on your console proxy host | |
| 19:36:48 | mnaser | melwitt: that was it, thank you | |
| 19:37:11 | mnaser | part of me wants to document all this in some "heads up" way, but also i worry "look how bad it is" messaging :\ | |
| 19:37:25 | cdent | thanks mriedem, is the expectation on that stuff that since that code is "remove in stein" or "called from online migrations" that we can remove it in openstack/placement instead of using those changes (I've not caught up fully on the irc log) | |
| 19:37:31 | melwitt | mnaser: it defaults to False, but for a rolling upgrade you would need it to be True. yet more information I missed in the upgrade release notes :( | |
| 19:37:41 | mriedem | cdent: i don't expect it will be removed in stein, | |
| 19:37:45 | mriedem | we have to have a blocker migration first | |
| 19:37:56 | mriedem | but haven't thought through it all yet | |
| 19:38:08 | mnaser | melwitt: np, want me to push up a bug or something or you'll write it down? | |
| 19:38:09 | mriedem | openstack/placement doesn't have a "placement-manage db online_data_migrations" yet | |
| 19:38:14 | mriedem | or does it? | |
| 19:38:15 | mnaser | maybe good to add to the upgrade doc | |
| 19:38:17 | mriedem | cdent: ^ | |
| 19:38:42 | cdent | it doesn't have _any_ placement-manage yet | |
| 19:38:50 | melwitt | mnaser: alternatively, could switch to defaulting to True and then let operators turn it off and decommission nova-consoleauth intentionally once they've rolled everything to rocky | |
| 19:38:51 | cdent | that's part of that message I sent earlier today | |
| 19:39:13 | cdent | mriedem: but my thinking was: we don't need to do any online db migrations, yet, either | |
| 19:39:52 | cdent | that is: Isn't the database in the correct state when someone gets to using openstack/placement? | |
| 19:40:42 | mnaser | melwitt: that feels like a better user experience | |
| 19:40:50 | mnaser | a lot of users probably will do rolling upgrades | |
| 19:41:06 | mriedem | cdent: nope | |
| 19:41:07 | mnaser | and you can keep it =True for rocky only anyways and remove it after | |
| 19:41:18 | mriedem | cdent: not if you're upgrading from <pike to stein | |
| 19:41:29 | mriedem | like mnaser is doing with going from juno to rocky | |