| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 18:50:51 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/consumer.py@29 | |
| 18:51:03 | dansmith | mnaser: yeah, can't really blame us for this one :) | |
| 18:51:11 | mnaser | dansmith: ill take that one :P | |
| 18:51:18 | mnaser | but yeah, i think the issue is upgrading across releases | |
| 18:51:50 | mriedem | the upgrade check CLI doesn't look to see if your minimum compute version is > N-1 | |
| 18:51:51 | mriedem | fwiw | |
| 18:52:14 | melwitt | so the microversion that started creating consumers was 1.8, pike https://docs.openstack.org/nova/latest/user/placement.html#require-placement-project-id-user-id-in-put-allocations | |
| 18:52:15 | mriedem | no one has requested it check for that | |
| 18:52:25 | melwitt | now when was the online data migration added... | |
| 18:52:33 | mnaser | fwiw | |
| 18:52:41 | mnaser | this cloud exists since juno to as far as i know | |
| 18:52:43 | mnaser | it's seen some shit | |
| 18:52:53 | melwitt | create_incomplete consumers was added in rocky | |
| 18:53:08 | melwitt | so allocations without consumers would be from before pike | |
| 18:53:52 | mriedem | hmm, i do seem to recall online data migrations for some placement stuff not working | |
| 18:53:57 | mriedem | b/c we were hitting the wrong db config | |
| 18:54:03 | mriedem | making it think nothing needed to be migrated | |
| 18:54:04 | melwitt | so any grenade that covered create_incomplete_consumers would be testing queens => rocky and never see any consumerless allocations | |
| 18:55:04 | mriedem | i suppose we were using at least 1.8 when creating allocations in queens | |
| 18:55:11 | mriedem | b/c of dansmith's migratoin allocation stuff | |
| 18:55:29 | melwitt | I thought we started using 1.8 in pike, that's when it was added | |
| 18:56:27 | mriedem | yeah i guess https://review.openstack.org/#/c/469634/ | |
| 18:58:13 | mriedem | ok i guess that solves the grenade mystery | |
| 18:58:29 | mriedem | geez when is someone going to add an FFU job that runs from ocata-em to master?! | |
| 19:01:00 | mnaser | lol | |
| 19:01:00 | mnaser | issues like this is why ffu upgrades terrify me | |
| 19:01:15 | mnaser | "when did this break? here's 2 years worth of code to go through!" | |
| 19:01:43 | dansmith | mnaser: it's way easier than the alternative, IMHO | |
| 19:01:54 | dansmith | of not knowing if the data set has been transformed since juno or not | |
| 19:02:05 | mnaser | dansmith: i'll agree on that statement | |
| 19:03:14 | mriedem | well, i was right about one thing | |
| 19:03:15 | mriedem | http://logs.openstack.org/00/607600/1/check/ironic-grenade-dsvm/4d493b1/logs/screen-n-cpu.txt.gz#_Oct_03_18_33_59_072341 | |
| 19:04:05 | melwitt | hm, I just realized, we're going to need to dupe these patches and use the same change-id to propose them to placement as well | |
| 19:04:13 | mriedem | yes | |
| 19:04:16 | mriedem | thta's what we've been doing | |
| 19:04:23 | melwitt | ok | |
| 19:04:28 | mnaser | forward porting | |
| 19:04:31 | mnaser | is that what we call it | |
| 19:05:00 | melwitt | on the second patch, it looks like there's at least one additional place we need to add the group_by, right? | |
| 19:05:15 | mriedem | yes | |
| 19:05:30 | melwitt | and should correspondingly test it too. the recreate test patch is already approved though | |
| 19:05:46 | mriedem | not for long | |
| 19:05:51 | melwitt | k | |
| 19:05:57 | efried | mriedem: That KeyError. Is that part of the existing bugs you've been talking about, or has it not yet been investigated? | |
| 19:06:13 | mnaser | the keyerror is not related | |
| 19:06:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:06:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:06:22 | mnaser | it was just some weird leftovers | |
| 19:06:45 | mriedem | mnaser: i'll rev my functional test patch and yours on top | |
| 19:06:52 | efried | Is it a bug that needs to be fixed, or was it a user error? | |
| 19:07:09 | mnaser | the keyerror? i dunno, but i dont think it should have been an issue because said user doesnt touch placement | |
| 19:07:13 | efried | I wouldn't have thought it should be possible no matter what abuse you lavish on the placement db. | |
| 19:07:27 | mnaser | fwiw the resource provider had nothing allocated | |
| 19:07:29 | mnaser | no usage that is | |
| 19:08:06 | efried | mnaser: Any sharing providers in this mess? | |
| 19:08:24 | mnaser | efried: sorry, not sure what you mean by that | |
| 19:08:47 | efried | Providers with the MISC_SHARES_VIA_AGGREGATE trait | |
| 19:10:00 | mnaser | efried: i am not sure hoenstly, i didn't dig in that much | |
| 19:12:42 | mriedem | then no | |
| 19:12:46 | mriedem | b/c you'd have to create them yourself | |
| 19:13:02 | mnaser | yeah besides nova | |
| 19:13:04 | mnaser | no api interaction | |
| 19:15:56 | efried | If you see a repro, lmk. Otherwise I'm going to pretend it didn't happen. | |
| 19:17:35 | mnaser | efried: i can get you a stacktrace if you want, but i dont think id be able to reproduce it given i deleted stuff | |
| 19:18:45 | efried | mnaser: The stack trace won't tell me much. Logs up to that point might help a bit. | |
| 19:19:15 | efried | especially if they've got our fun new debug messages | |
| 19:20:08 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 19:20:33 | edmondsw | efried ^ this should address the fd open issue | |
| 19:23:41 | mriedem | fudge, | |
| 19:23:48 | mriedem | this unique constraint error is in 3 f'ing places | |
| 19:24:35 | melwitt | I wondered if there were more. and I had thought they'd call through the same method to create missing consumers but I guess all of the queries are different | |
| 19:24:58 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/resource_provider.py@1973 | |
| 19:24:58 | mriedem | well maybe not | |
| 19:25:04 | mriedem | that's not doing the insert-from-select | |
| 19:25:06 | mriedem | like the others | |
| 19:26:22 | mnaser | um | |
| 19:26:29 | mnaser | in rocky we moved to console auth tokens stored in db, right? | |
| 19:27:16 | melwitt | yes, in addition to nova-consoleauth until this lands https://review.openstack.org/610673 | |
| 19:28:21 | mnaser | melwitt: what service creates the auth tokens? | |
| 19:28:59 | melwitt | mnaser: nova-compute creates them for the database, nova-consoleauth creates them for nova-consoleauth | |
| 19:29:15 | mnaser | so if your nova-compute is not on rocky | |
| 19:29:29 | melwitt | then you get nova-consoleauth tokens | |
| 19:29:29 | mnaser | ..does that mean no console? | |
| 19:29:40 | mnaser | ok i see | |
| 19:29:41 | melwitt | no, you get console | |
| 19:29:54 | mnaser | so just an extra indirection right now | |
| 19:30:03 | mnaser | till nova-compute creates to db directly in the future | |
| 19:30:45 | melwitt | nova-compute creates directly to db in rocky. just obviously your older computes will not and those instances will be supported by nova-consoleauth | |
| 19:31:11 | mriedem | mnaser: melwitt: dansmith: cdent: efried: ^ should be good now, covers both cases | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 19:31:11 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:31:27 | melwitt | once all your computes are on rocky, then you wouldn't need nova-consoleauth once this backport lands https://review.openstack.org/610673 | |
| 19:32:31 | mnaser | melwitt: im seeing consoleauth get a token, but the traceback that says token validation failed is resulted from a method that does db.console_auth_token_get | |
| 19:33:21 | mnaser | ugh | |
| 19:33:27 | mnaser | [workarounds] enable_consoleauth=True | |
| 19:33:40 | melwitt | mnaser: and you have a mix of rocky computes and older than rocky computes? in that case, you'll need to set [workarounds]enable_consoleauth = True on your console proxy host | |
| 19:36:48 | mnaser | melwitt: that was it, thank you | |
| 19:37:11 | mnaser | part of me wants to document all this in some "heads up" way, but also i worry "look how bad it is" messaging :\ | |
| 19:37:25 | cdent | thanks mriedem, is the expectation on that stuff that since that code is "remove in stein" or "called from online migrations" that we can remove it in openstack/placement instead of using those changes (I've not caught up fully on the irc log) | |