| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 18:20:51 | mnaser | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L3522 | |
| 18:20:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 18:21:16 | mriedem | tssurya: hmm, maybe that's hitting after an upgrade? | |
| 18:21:43 | mriedem | but, | |
| 18:21:54 | mriedem | if that were the case, ironic's grenade job should be broken | |
| 18:22:01 | mriedem | is the value changing? | |
| 18:22:51 | tssurya | so at the moment the only info I have is that we cherry-picked this to queens | |
| 18:23:06 | tssurya | maybe something is wrong because of that | |
| 18:24:04 | mriedem | tssurya: ok it's called from here in the RT https://review.openstack.org/#/c/571535/2/nova/compute/resource_tracker.py@617 | |
| 18:24:37 | mriedem | my guess is the compute node record already existed with a random uuid, and on restart of the compute service with the new code, it's trying to update the uuid in the compute node record using the ironic node uuid | |
| 18:24:53 | mriedem | i don't know why ironic's grenade job wouldn't fail for the same reason, but it seems like an obvious oversight in that patch | |
| 18:24:55 | tssurya | mriedem: yea | |
| 18:25:05 | tssurya | that's exactly what's happening | |
| 18:25:12 | tssurya | so this is for the existing nodes.. | |
| 18:25:15 | mriedem | \o/ | |
| 18:25:54 | mriedem | mnaser: as for https://github.com/openstack/nova/blob/stable/rocky/nova/api/openstack/placement/objects/resource_provider.py#L3486 | |
| 18:25:56 | mriedem | and the KeyError | |
| 18:26:00 | mriedem | i don't understand any of that code | |
| 18:26:10 | mriedem | that's all efried, tetsuro and jaypipes | |
| 18:27:06 | mriedem | https://review.openstack.org/#/c/559480/ | |
| 18:28:17 | tssurya | heh | |
| 18:28:34 | mriedem | tssurya: report a bug for the ironic node uuid thing i guess | |
| 18:28:49 | mriedem | jroll: do you know anything about the ironic grenade job? | |
| 18:28:52 | tssurya | mriedem: yea I will | |
| 18:28:54 | mriedem | does it restart n-cpu across releases? | |
| 18:29:34 | mnaser | mriedem: i think this is a weird env-related corner case where those havent done much reporting to placement | |
| 18:30:06 | mnaser | going to delete those stale rps | |
| 18:32:18 | mnaser | mriedem: every time i use osc-placement i have to say thank you | |
| 18:32:20 | mnaser | its such a life saver | |
| 18:34:06 | mriedem | don't thank me, thank avolkov | |
| 18:34:25 | mriedem | and rpodolykia | |
| 18:34:34 | mriedem | i know i butchered that irc | |
| 18:35:11 | mriedem | tssurya: yeah so you can set a readonly field while it's never been set | |
| 18:35:18 | mriedem | once it's set though, it's stuck | |
| 18:35:40 | tssurya | mriedem: ah right, thanks | |
| 18:35:45 | tssurya | I am filing a bug now | |
| 18:35:57 | mriedem | cdent: efried: https://review.openstack.org/#/q/topic:bug/1798163+(status:open+OR+status:merged) | |
| 18:36:02 | mriedem | dansmith: melwitt: ^ | |
| 18:39:08 | mriedem | starting on the placement repo cherry picks | |
| 18:39:39 | melwitt | 👀 | |
| 18:39:51 | openstack | Launchpad bug 1798172 in OpenStack Compute (nova) "Ironic driver tries to update the compute_node's UUID which of course fails in case of existing compute_nodes" [Undecided,New] | |
| 18:39:51 | tssurya | mriedem: https://bugs.launchpad.net/nova/+bug/1798172 | |
| 18:39:56 | mriedem | i'll assume that lego block is a pile-o-poo | |
| 18:40:03 | mriedem | "of course" | |
| 18:40:04 | mriedem | nice | |
| 18:40:06 | mriedem | :( | |
| 18:40:14 | melwitt | it's eyes, looking at the linked patches | |
| 18:40:25 | melwitt | but pile-o-poo could have been good | |
| 18:41:49 | dansmith | mriedem: so this has to be broken for anyone right? | |
| 18:41:59 | mriedem | dansmith: which thing? | |
| 18:42:01 | mriedem | placement? yes. | |
| 18:42:07 | dansmith | yeah | |
| 18:42:14 | mriedem | i guess mnaser is first to rocky ever | |
| 18:42:27 | mnaser | world first | |
| 18:42:32 | mnaser | but now | |
| 18:42:35 | mnaser | we have MORE | |
| 18:42:36 | mnaser | Exception during message handling: RPCVersionCapError: Requested message version, 5.0 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 4.17. | |
| 18:42:37 | mriedem | honestly i'm not sure how it wouldn't have been a problem caught in grenade | |
| 18:42:39 | dansmith | looks straightforward | |
| 18:42:47 | mnaser | nova-conductor when scheduling new instances | |
| 18:42:50 | melwitt | mnaser: need to get a windshield sticker for your car WORLD FIRST TO ROCKY | |
| 18:42:50 | mriedem | b/c in grenade we have existing instances with allocations | |
| 18:43:16 | mnaser | more like a military medal for the stuff i have to go through y'all | |
| 18:43:17 | mnaser | lol | |
| 18:43:27 | melwitt | :***( | |
| 18:43:33 | mnaser | all my packages are up to date, control plane is all rocky with queen computes | |
| 18:43:35 | melwitt | (those are tears) | |
| 18:43:37 | mnaser | its ok, i do it for the people | |
| 18:43:51 | mnaser | openstack upgrades are easy, i promise, i already helped fixed most of it | |
| 18:45:37 | mnaser | where are the versions listed again? in the rpc api file? | |
| 18:46:08 | mriedem | dansmith: so on upgrade to rocky we would have listed allocations for a provider here https://github.com/openstack/nova/blob/237bfcfd82fc28a955574b588fbce1d2392c9e45/nova/compute/resource_tracker.py#L1298 which should have hit the unique constraint | |
| 18:46:09 | mriedem | so idk | |
| 18:46:17 | mriedem | i mean in a grenade run | |
| 18:46:17 | dansmith | mriedem: yeah | |
| 18:46:27 | dansmith | mnaser: that rpc error looks like you have something old | |
| 18:46:41 | dansmith | like an old conductor still running maybe? | |
| 18:46:43 | mriedem | maybe we didn't hit it in grenade because the queens allocations already have project_id/user_id created? | |
| 18:46:45 | mriedem | s/created/set/ | |
| 18:47:04 | mriedem | or.... because we ran the online data migrations? | |
| 18:47:20 | mnaser | dansmith: oh shit, i missed a compute. | |
| 18:47:36 | mnaser | in pike=>queens | |
| 18:47:40 | melwitt | mriedem: well, normally allocations are required to be created with project/user and that's what creates a consumer. this bug is about the online data migration for allocations with missing consumers, prior to the microversion where we required project/user, right? | |
| 18:47:58 | dansmith | mnaser: mm, I dunno, if that's coming from a conductor node I think it's probably an old conductor, but I'd have to see more about where exactly | |
| 18:48:06 | melwitt | so grenade probably doesn't cover that, I wouldn't think | |
| 18:48:20 | dansmith | mnaser: or are you saying the auto stuff is calculating a 4.x when everything has moved past 5.0 because of a very old compute? | |
| 18:48:53 | dansmith | melwitt: but you don't create consumers directly and before some version, no consumers were created for you | |
| 18:49:11 | mnaser | dansmith: there is a compute that is active running at version 22 (service table), rest of services are 30 (for queens computes) and 35 (control plaen) | |
| 18:49:13 | dansmith | melwitt: so grenade should hit this at some point, unless we run online migrations and fix them up before we run or whatever | |
| 18:49:25 | dansmith | mnaser: ah okay yeah | |
| 18:49:25 | mnaser | so i guess the auto stuff is calculating based on the fact that the oldest compute (which is active) | |
| 18:49:36 | mnaser | so | |
| 18:49:39 | mnaser | working as intended | |
| 18:49:57 | mnaser | i think the upgrade check would have probably warned me if i used it oops | |
| 18:50:29 | melwitt | dansmith: true, but I'm pretty sure the online migration to create missing consumers was created later on, i.e. not in the same release where we started created consumers with allocations. so maybe that's how it missed it. by the time the online data migration existed, grenade was no longer testing the old way that didn't create consumers | |
| 18:50:46 | mriedem | melwitt: different code path | |
| 18:50:51 | mriedem | https://review.openstack.org/#/c/611115/3/nova/api/openstack/placement/objects/consumer.py@29 | |
| 18:51:03 | dansmith | mnaser: yeah, can't really blame us for this one :) | |
| 18:51:11 | mnaser | dansmith: ill take that one :P | |
| 18:51:18 | mnaser | but yeah, i think the issue is upgrading across releases | |
| 18:51:50 | mriedem | the upgrade check CLI doesn't look to see if your minimum compute version is > N-1 | |