| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 18:50:29 | melwitt | dansmith: true, but I'm pretty sure the online migration to create missing consumers was created later on, i.e. not in the same release where we started created consumers with allocations. so maybe that's how it missed it. by the time the online data migration existed, grenade was no longer testing the old way that didn't create consumers | |
| 18:50:46 | mriedem | melwitt: different code path | |
| 18:50:51 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/consumer.py@29 | |
| 18:51:03 | dansmith | mnaser: yeah, can't really blame us for this one :) | |
| 18:51:11 | mnaser | dansmith: ill take that one :P | |
| 18:51:18 | mnaser | but yeah, i think the issue is upgrading across releases | |
| 18:51:50 | mriedem | the upgrade check CLI doesn't look to see if your minimum compute version is > N-1 | |
| 18:51:51 | mriedem | fwiw | |
| 18:52:14 | melwitt | so the microversion that started creating consumers was 1.8, pike https://docs.openstack.org/nova/latest/user/placement.html#require-placement-project-id-user-id-in-put-allocations | |
| 18:52:15 | mriedem | no one has requested it check for that | |
| 18:52:25 | melwitt | now when was the online data migration added... | |
| 18:52:33 | mnaser | fwiw | |
| 18:52:41 | mnaser | this cloud exists since juno to as far as i know | |
| 18:52:43 | mnaser | it's seen some shit | |
| 18:52:53 | melwitt | create_incomplete consumers was added in rocky | |
| 18:53:08 | melwitt | so allocations without consumers would be from before pike | |
| 18:53:52 | mriedem | hmm, i do seem to recall online data migrations for some placement stuff not working | |
| 18:53:57 | mriedem | b/c we were hitting the wrong db config | |
| 18:54:03 | mriedem | making it think nothing needed to be migrated | |
| 18:54:04 | melwitt | so any grenade that covered create_incomplete_consumers would be testing queens => rocky and never see any consumerless allocations | |
| 18:55:04 | mriedem | i suppose we were using at least 1.8 when creating allocations in queens | |
| 18:55:11 | mriedem | b/c of dansmith's migratoin allocation stuff | |
| 18:55:29 | melwitt | I thought we started using 1.8 in pike, that's when it was added | |
| 18:56:27 | mriedem | yeah i guess https://review.openstack.org/#/c/469634/ | |
| 18:58:13 | mriedem | ok i guess that solves the grenade mystery | |
| 18:58:29 | mriedem | geez when is someone going to add an FFU job that runs from ocata-em to master?! | |
| 19:01:00 | mnaser | lol | |
| 19:01:00 | mnaser | issues like this is why ffu upgrades terrify me | |
| 19:01:15 | mnaser | "when did this break? here's 2 years worth of code to go through!" | |
| 19:01:43 | dansmith | mnaser: it's way easier than the alternative, IMHO | |
| 19:01:54 | dansmith | of not knowing if the data set has been transformed since juno or not | |
| 19:02:05 | mnaser | dansmith: i'll agree on that statement | |
| 19:03:14 | mriedem | well, i was right about one thing | |
| 19:03:15 | mriedem | http://logs.openstack.org/00/607600/1/check/ironic-grenade-dsvm/4d493b1/logs/screen-n-cpu.txt.gz#_Oct_03_18_33_59_072341 | |
| 19:04:05 | melwitt | hm, I just realized, we're going to need to dupe these patches and use the same change-id to propose them to placement as well | |
| 19:04:13 | mriedem | yes | |
| 19:04:16 | mriedem | thta's what we've been doing | |
| 19:04:23 | melwitt | ok | |
| 19:04:28 | mnaser | forward porting | |
| 19:04:31 | mnaser | is that what we call it | |
| 19:05:00 | melwitt | on the second patch, it looks like there's at least one additional place we need to add the group_by, right? | |
| 19:05:15 | mriedem | yes | |
| 19:05:30 | melwitt | and should correspondingly test it too. the recreate test patch is already approved though | |
| 19:05:46 | mriedem | not for long | |
| 19:05:51 | melwitt | k | |
| 19:05:57 | efried | mriedem: That KeyError. Is that part of the existing bugs you've been talking about, or has it not yet been investigated? | |
| 19:06:13 | mnaser | the keyerror is not related | |
| 19:06:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:06:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:06:22 | mnaser | it was just some weird leftovers | |
| 19:06:45 | mriedem | mnaser: i'll rev my functional test patch and yours on top | |
| 19:06:52 | efried | Is it a bug that needs to be fixed, or was it a user error? | |
| 19:07:09 | mnaser | the keyerror? i dunno, but i dont think it should have been an issue because said user doesnt touch placement | |
| 19:07:13 | efried | I wouldn't have thought it should be possible no matter what abuse you lavish on the placement db. | |
| 19:07:27 | mnaser | fwiw the resource provider had nothing allocated | |
| 19:07:29 | mnaser | no usage that is | |
| 19:08:06 | efried | mnaser: Any sharing providers in this mess? | |
| 19:08:24 | mnaser | efried: sorry, not sure what you mean by that | |
| 19:08:47 | efried | Providers with the MISC_SHARES_VIA_AGGREGATE trait | |
| 19:10:00 | mnaser | efried: i am not sure hoenstly, i didn't dig in that much | |
| 19:12:42 | mriedem | then no | |
| 19:12:46 | mriedem | b/c you'd have to create them yourself | |
| 19:13:02 | mnaser | yeah besides nova | |
| 19:13:04 | mnaser | no api interaction | |
| 19:15:56 | efried | If you see a repro, lmk. Otherwise I'm going to pretend it didn't happen. | |
| 19:17:35 | mnaser | efried: i can get you a stacktrace if you want, but i dont think id be able to reproduce it given i deleted stuff | |
| 19:18:45 | efried | mnaser: The stack trace won't tell me much. Logs up to that point might help a bit. | |
| 19:19:15 | efried | especially if they've got our fun new debug messages | |
| 19:20:08 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 19:20:33 | edmondsw | efried ^ this should address the fd open issue | |
| 19:23:41 | mriedem | fudge, | |
| 19:23:48 | mriedem | this unique constraint error is in 3 f'ing places | |
| 19:24:35 | melwitt | I wondered if there were more. and I had thought they'd call through the same method to create missing consumers but I guess all of the queries are different | |
| 19:24:58 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/resource_provider.py@1973 | |
| 19:24:58 | mriedem | well maybe not | |
| 19:25:04 | mriedem | that's not doing the insert-from-select | |
| 19:25:06 | mriedem | like the others | |
| 19:26:22 | mnaser | um | |
| 19:26:29 | mnaser | in rocky we moved to console auth tokens stored in db, right? | |
| 19:27:16 | melwitt | yes, in addition to nova-consoleauth until this lands https://review.openstack.org/610673 | |
| 19:28:21 | mnaser | melwitt: what service creates the auth tokens? | |
| 19:28:59 | melwitt | mnaser: nova-compute creates them for the database, nova-consoleauth creates them for nova-consoleauth | |
| 19:29:15 | mnaser | so if your nova-compute is not on rocky | |
| 19:29:29 | melwitt | then you get nova-consoleauth tokens | |
| 19:29:29 | mnaser | ..does that mean no console? | |
| 19:29:40 | mnaser | ok i see | |
| 19:29:41 | melwitt | no, you get console | |
| 19:29:54 | mnaser | so just an extra indirection right now | |
| 19:30:03 | mnaser | till nova-compute creates to db directly in the future | |
| 19:30:45 | melwitt | nova-compute creates directly to db in rocky. just obviously your older computes will not and those instances will be supported by nova-consoleauth | |
| 19:31:11 | mriedem | mnaser: melwitt: dansmith: cdent: efried: ^ should be good now, covers both cases | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 19:31:11 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:31:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:31:27 | melwitt | once all your computes are on rocky, then you wouldn't need nova-consoleauth once this backport lands https://review.openstack.org/610673 | |
| 19:32:31 | mnaser | melwitt: im seeing consoleauth get a token, but the traceback that says token validation failed is resulted from a method that does db.console_auth_token_get | |
| 19:33:21 | mnaser | ugh | |
| 19:33:27 | mnaser | [workarounds] enable_consoleauth=True | |
| 19:33:40 | melwitt | mnaser: and you have a mix of rocky computes and older than rocky computes? in that case, you'll need to set [workarounds]enable_consoleauth = True on your console proxy host | |
| 19:36:48 | mnaser | melwitt: that was it, thank you | |