| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 18:43:37 | mnaser | its ok, i do it for the people | |
| 18:43:51 | mnaser | openstack upgrades are easy, i promise, i already helped fixed most of it | |
| 18:45:37 | mnaser | where are the versions listed again? in the rpc api file? | |
| 18:46:08 | mriedem | dansmith: so on upgrade to rocky we would have listed allocations for a provider here https://github.com/openstack/nova/blob/237bfcfd82fc28a955574b588fbce1d2392c9e45/nova/compute/resource_tracker.py#L1298 which should have hit the unique constraint | |
| 18:46:09 | mriedem | so idk | |
| 18:46:17 | dansmith | mriedem: yeah | |
| 18:46:17 | mriedem | i mean in a grenade run | |
| 18:46:27 | dansmith | mnaser: that rpc error looks like you have something old | |
| 18:46:41 | dansmith | like an old conductor still running maybe? | |
| 18:46:43 | mriedem | maybe we didn't hit it in grenade because the queens allocations already have project_id/user_id created? | |
| 18:46:45 | mriedem | s/created/set/ | |
| 18:47:04 | mriedem | or.... because we ran the online data migrations? | |
| 18:47:20 | mnaser | dansmith: oh shit, i missed a compute. | |
| 18:47:36 | mnaser | in pike=>queens | |
| 18:47:40 | melwitt | mriedem: well, normally allocations are required to be created with project/user and that's what creates a consumer. this bug is about the online data migration for allocations with missing consumers, prior to the microversion where we required project/user, right? | |
| 18:47:58 | dansmith | mnaser: mm, I dunno, if that's coming from a conductor node I think it's probably an old conductor, but I'd have to see more about where exactly | |
| 18:48:06 | melwitt | so grenade probably doesn't cover that, I wouldn't think | |
| 18:48:20 | dansmith | mnaser: or are you saying the auto stuff is calculating a 4.x when everything has moved past 5.0 because of a very old compute? | |
| 18:48:53 | dansmith | melwitt: but you don't create consumers directly and before some version, no consumers were created for you | |
| 18:49:11 | mnaser | dansmith: there is a compute that is active running at version 22 (service table), rest of services are 30 (for queens computes) and 35 (control plaen) | |
| 18:49:13 | dansmith | melwitt: so grenade should hit this at some point, unless we run online migrations and fix them up before we run or whatever | |
| 18:49:25 | mnaser | so i guess the auto stuff is calculating based on the fact that the oldest compute (which is active) | |
| 18:49:25 | dansmith | mnaser: ah okay yeah | |
| 18:49:36 | mnaser | so | |
| 18:49:39 | mnaser | working as intended | |
| 18:49:57 | mnaser | i think the upgrade check would have probably warned me if i used it oops | |
| 18:50:29 | melwitt | dansmith: true, but I'm pretty sure the online migration to create missing consumers was created later on, i.e. not in the same release where we started created consumers with allocations. so maybe that's how it missed it. by the time the online data migration existed, grenade was no longer testing the old way that didn't create consumers | |
| 18:50:46 | mriedem | melwitt: different code path | |
| 18:50:51 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/consumer.py@29 | |
| 18:51:03 | dansmith | mnaser: yeah, can't really blame us for this one :) | |
| 18:51:11 | mnaser | dansmith: ill take that one :P | |
| 18:51:18 | mnaser | but yeah, i think the issue is upgrading across releases | |
| 18:51:50 | mriedem | the upgrade check CLI doesn't look to see if your minimum compute version is > N-1 | |
| 18:51:51 | mriedem | fwiw | |
| 18:52:14 | melwitt | so the microversion that started creating consumers was 1.8, pike https://docs.openstack.org/nova/latest/user/placement.html#require-placement-project-id-user-id-in-put-allocations | |
| 18:52:15 | mriedem | no one has requested it check for that | |
| 18:52:25 | melwitt | now when was the online data migration added... | |
| 18:52:33 | mnaser | fwiw | |
| 18:52:41 | mnaser | this cloud exists since juno to as far as i know | |
| 18:52:43 | mnaser | it's seen some shit | |
| 18:52:53 | melwitt | create_incomplete consumers was added in rocky | |
| 18:53:08 | melwitt | so allocations without consumers would be from before pike | |
| 18:53:52 | mriedem | hmm, i do seem to recall online data migrations for some placement stuff not working | |
| 18:53:57 | mriedem | b/c we were hitting the wrong db config | |
| 18:54:03 | mriedem | making it think nothing needed to be migrated | |
| 18:54:04 | melwitt | so any grenade that covered create_incomplete_consumers would be testing queens => rocky and never see any consumerless allocations | |
| 18:55:04 | mriedem | i suppose we were using at least 1.8 when creating allocations in queens | |
| 18:55:11 | mriedem | b/c of dansmith's migratoin allocation stuff | |
| 18:55:29 | melwitt | I thought we started using 1.8 in pike, that's when it was added | |
| 18:56:27 | mriedem | yeah i guess https://review.openstack.org/#/c/469634/ | |
| 18:58:13 | mriedem | ok i guess that solves the grenade mystery | |
| 18:58:29 | mriedem | geez when is someone going to add an FFU job that runs from ocata-em to master?! | |
| 19:01:00 | mnaser | issues like this is why ffu upgrades terrify me | |
| 19:01:00 | mnaser | lol | |
| 19:01:15 | mnaser | "when did this break? here's 2 years worth of code to go through!" | |
| 19:01:43 | dansmith | mnaser: it's way easier than the alternative, IMHO | |
| 19:01:54 | dansmith | of not knowing if the data set has been transformed since juno or not | |
| 19:02:05 | mnaser | dansmith: i'll agree on that statement | |
| 19:03:14 | mriedem | well, i was right about one thing | |
| 19:03:15 | mriedem | http://logs.openstack.org/00/607600/1/check/ironic-grenade-dsvm/4d493b1/logs/screen-n-cpu.txt.gz#_Oct_03_18_33_59_072341 | |
| 19:04:05 | melwitt | hm, I just realized, we're going to need to dupe these patches and use the same change-id to propose them to placement as well | |
| 19:04:13 | mriedem | yes | |
| 19:04:16 | mriedem | thta's what we've been doing | |
| 19:04:23 | melwitt | ok | |
| 19:04:28 | mnaser | forward porting | |
| 19:04:31 | mnaser | is that what we call it | |
| 19:05:00 | melwitt | on the second patch, it looks like there's at least one additional place we need to add the group_by, right? | |
| 19:05:15 | mriedem | yes | |
| 19:05:30 | melwitt | and should correspondingly test it too. the recreate test patch is already approved though | |
| 19:05:46 | mriedem | not for long | |
| 19:05:51 | melwitt | k | |
| 19:05:57 | efried | mriedem: That KeyError. Is that part of the existing bugs you've been talking about, or has it not yet been investigated? | |
| 19:06:13 | mnaser | the keyerror is not related | |
| 19:06:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:06:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:06:22 | mnaser | it was just some weird leftovers | |
| 19:06:45 | mriedem | mnaser: i'll rev my functional test patch and yours on top | |
| 19:06:52 | efried | Is it a bug that needs to be fixed, or was it a user error? | |
| 19:07:09 | mnaser | the keyerror? i dunno, but i dont think it should have been an issue because said user doesnt touch placement | |
| 19:07:13 | efried | I wouldn't have thought it should be possible no matter what abuse you lavish on the placement db. | |
| 19:07:27 | mnaser | fwiw the resource provider had nothing allocated | |
| 19:07:29 | mnaser | no usage that is | |
| 19:08:06 | efried | mnaser: Any sharing providers in this mess? | |
| 19:08:24 | mnaser | efried: sorry, not sure what you mean by that | |
| 19:08:47 | efried | Providers with the MISC_SHARES_VIA_AGGREGATE trait | |
| 19:10:00 | mnaser | efried: i am not sure hoenstly, i didn't dig in that much | |
| 19:12:42 | mriedem | then no | |
| 19:12:46 | mriedem | b/c you'd have to create them yourself | |
| 19:13:02 | mnaser | yeah besides nova | |
| 19:13:04 | mnaser | no api interaction | |
| 19:15:56 | efried | If you see a repro, lmk. Otherwise I'm going to pretend it didn't happen. | |
| 19:17:35 | mnaser | efried: i can get you a stacktrace if you want, but i dont think id be able to reproduce it given i deleted stuff | |
| 19:18:45 | efried | mnaser: The stack trace won't tell me much. Logs up to that point might help a bit. | |
| 19:19:15 | efried | especially if they've got our fun new debug messages | |
| 19:20:08 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: Use tempfile for powervm config drive https://review.openstack.org/610174 | |
| 19:20:33 | edmondsw | efried ^ this should address the fd open issue | |
| 19:23:41 | mriedem | fudge, | |
| 19:23:48 | mriedem | this unique constraint error is in 3 f'ing places | |
| 19:24:35 | melwitt | I wondered if there were more. and I had thought they'd call through the same method to create missing consumers but I guess all of the queries are different | |
| 19:24:58 | mriedem | well maybe not | |