| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 18:40:03 | mriedem | "of course" | |
| 18:40:04 | mriedem | nice | |
| 18:40:06 | mriedem | :( | |
| 18:40:14 | melwitt | it's eyes, looking at the linked patches | |
| 18:40:25 | melwitt | but pile-o-poo could have been good | |
| 18:41:49 | dansmith | mriedem: so this has to be broken for anyone right? | |
| 18:41:59 | mriedem | dansmith: which thing? | |
| 18:42:01 | mriedem | placement? yes. | |
| 18:42:07 | dansmith | yeah | |
| 18:42:14 | mriedem | i guess mnaser is first to rocky ever | |
| 18:42:27 | mnaser | world first | |
| 18:42:32 | mnaser | but now | |
| 18:42:35 | mnaser | we have MORE | |
| 18:42:36 | mnaser | Exception during message handling: RPCVersionCapError: Requested message version, 5.0 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 4.17. | |
| 18:42:37 | mriedem | honestly i'm not sure how it wouldn't have been a problem caught in grenade | |
| 18:42:39 | dansmith | looks straightforward | |
| 18:42:47 | mnaser | nova-conductor when scheduling new instances | |
| 18:42:50 | melwitt | mnaser: need to get a windshield sticker for your car WORLD FIRST TO ROCKY | |
| 18:42:50 | mriedem | b/c in grenade we have existing instances with allocations | |
| 18:43:16 | mnaser | more like a military medal for the stuff i have to go through y'all | |
| 18:43:17 | mnaser | lol | |
| 18:43:27 | melwitt | :***( | |
| 18:43:33 | mnaser | all my packages are up to date, control plane is all rocky with queen computes | |
| 18:43:35 | melwitt | (those are tears) | |
| 18:43:37 | mnaser | its ok, i do it for the people | |
| 18:43:51 | mnaser | openstack upgrades are easy, i promise, i already helped fixed most of it | |
| 18:45:37 | mnaser | where are the versions listed again? in the rpc api file? | |
| 18:46:08 | mriedem | dansmith: so on upgrade to rocky we would have listed allocations for a provider here https://github.com/openstack/nova/blob/237bfcfd82fc28a955574b588fbce1d2392c9e45/nova/compute/resource_tracker.py#L1298 which should have hit the unique constraint | |
| 18:46:09 | mriedem | so idk | |
| 18:46:17 | mriedem | i mean in a grenade run | |
| 18:46:17 | dansmith | mriedem: yeah | |
| 18:46:27 | dansmith | mnaser: that rpc error looks like you have something old | |
| 18:46:41 | dansmith | like an old conductor still running maybe? | |
| 18:46:43 | mriedem | maybe we didn't hit it in grenade because the queens allocations already have project_id/user_id created? | |
| 18:46:45 | mriedem | s/created/set/ | |
| 18:47:04 | mriedem | or.... because we ran the online data migrations? | |
| 18:47:20 | mnaser | dansmith: oh shit, i missed a compute. | |
| 18:47:36 | mnaser | in pike=>queens | |
| 18:47:40 | melwitt | mriedem: well, normally allocations are required to be created with project/user and that's what creates a consumer. this bug is about the online data migration for allocations with missing consumers, prior to the microversion where we required project/user, right? | |
| 18:47:58 | dansmith | mnaser: mm, I dunno, if that's coming from a conductor node I think it's probably an old conductor, but I'd have to see more about where exactly | |
| 18:48:06 | melwitt | so grenade probably doesn't cover that, I wouldn't think | |
| 18:48:20 | dansmith | mnaser: or are you saying the auto stuff is calculating a 4.x when everything has moved past 5.0 because of a very old compute? | |
| 18:48:53 | dansmith | melwitt: but you don't create consumers directly and before some version, no consumers were created for you | |
| 18:49:11 | mnaser | dansmith: there is a compute that is active running at version 22 (service table), rest of services are 30 (for queens computes) and 35 (control plaen) | |
| 18:49:13 | dansmith | melwitt: so grenade should hit this at some point, unless we run online migrations and fix them up before we run or whatever | |
| 18:49:25 | dansmith | mnaser: ah okay yeah | |
| 18:49:25 | mnaser | so i guess the auto stuff is calculating based on the fact that the oldest compute (which is active) | |
| 18:49:36 | mnaser | so | |
| 18:49:39 | mnaser | working as intended | |
| 18:49:57 | mnaser | i think the upgrade check would have probably warned me if i used it oops | |
| 18:50:29 | melwitt | dansmith: true, but I'm pretty sure the online migration to create missing consumers was created later on, i.e. not in the same release where we started created consumers with allocations. so maybe that's how it missed it. by the time the online data migration existed, grenade was no longer testing the old way that didn't create consumers | |
| 18:50:46 | mriedem | melwitt: different code path | |
| 18:50:51 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/consumer.py@29 | |
| 18:51:03 | dansmith | mnaser: yeah, can't really blame us for this one :) | |
| 18:51:11 | mnaser | dansmith: ill take that one :P | |
| 18:51:18 | mnaser | but yeah, i think the issue is upgrading across releases | |
| 18:51:50 | mriedem | the upgrade check CLI doesn't look to see if your minimum compute version is > N-1 | |
| 18:51:51 | mriedem | fwiw | |
| 18:52:14 | melwitt | so the microversion that started creating consumers was 1.8, pike https://docs.openstack.org/nova/latest/user/placement.html#require-placement-project-id-user-id-in-put-allocations | |
| 18:52:15 | mriedem | no one has requested it check for that | |
| 18:52:25 | melwitt | now when was the online data migration added... | |
| 18:52:33 | mnaser | fwiw | |
| 18:52:41 | mnaser | this cloud exists since juno to as far as i know | |
| 18:52:43 | mnaser | it's seen some shit | |
| 18:52:53 | melwitt | create_incomplete consumers was added in rocky | |
| 18:53:08 | melwitt | so allocations without consumers would be from before pike | |
| 18:53:52 | mriedem | hmm, i do seem to recall online data migrations for some placement stuff not working | |
| 18:53:57 | mriedem | b/c we were hitting the wrong db config | |
| 18:54:03 | mriedem | making it think nothing needed to be migrated | |
| 18:54:04 | melwitt | so any grenade that covered create_incomplete_consumers would be testing queens => rocky and never see any consumerless allocations | |
| 18:55:04 | mriedem | i suppose we were using at least 1.8 when creating allocations in queens | |
| 18:55:11 | mriedem | b/c of dansmith's migratoin allocation stuff | |
| 18:55:29 | melwitt | I thought we started using 1.8 in pike, that's when it was added | |
| 18:56:27 | mriedem | yeah i guess https://review.openstack.org/#/c/469634/ | |
| 18:58:13 | mriedem | ok i guess that solves the grenade mystery | |
| 18:58:29 | mriedem | geez when is someone going to add an FFU job that runs from ocata-em to master?! | |
| 19:01:00 | mnaser | lol | |
| 19:01:00 | mnaser | issues like this is why ffu upgrades terrify me | |
| 19:01:15 | mnaser | "when did this break? here's 2 years worth of code to go through!" | |
| 19:01:43 | dansmith | mnaser: it's way easier than the alternative, IMHO | |
| 19:01:54 | dansmith | of not knowing if the data set has been transformed since juno or not | |
| 19:02:05 | mnaser | dansmith: i'll agree on that statement | |
| 19:03:14 | mriedem | well, i was right about one thing | |
| 19:03:15 | mriedem | http://logs.openstack.org/00/607600/1/check/ironic-grenade-dsvm/4d493b1/logs/screen-n-cpu.txt.gz#_Oct_03_18_33_59_072341 | |
| 19:04:05 | melwitt | hm, I just realized, we're going to need to dupe these patches and use the same change-id to propose them to placement as well | |
| 19:04:13 | mriedem | yes | |
| 19:04:16 | mriedem | thta's what we've been doing | |
| 19:04:23 | melwitt | ok | |
| 19:04:28 | mnaser | forward porting | |
| 19:04:31 | mnaser | is that what we call it | |
| 19:05:00 | melwitt | on the second patch, it looks like there's at least one additional place we need to add the group_by, right? | |
| 19:05:15 | mriedem | yes | |
| 19:05:30 | melwitt | and should correspondingly test it too. the recreate test patch is already approved though | |
| 19:05:46 | mriedem | not for long | |
| 19:05:51 | melwitt | k | |
| 19:05:57 | efried | mriedem: That KeyError. Is that part of the existing bugs you've been talking about, or has it not yet been investigated? | |
| 19:06:13 | mnaser | the keyerror is not related | |
| 19:06:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Critical,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 19:06:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 19:06:22 | mnaser | it was just some weird leftovers | |