| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 18:13:36 | tssurya | mriedem: question about https://review.openstack.org/#/c/571535/ . How are we able to set the compute_node.uuid which is a read-only attribute ? Am I missing something here ? | |
| 18:14:30 | mriedem | please hold dear caller | |
| 18:14:36 | tssurya | ack :) | |
| 18:14:51 | mriedem | we create compute node records with uuids in tests all the time | |
| 18:15:58 | tssurya | true.. I am somehow hitting https://pastebin.com/kqyu66wB | |
| 18:16:02 | tssurya | will look closer | |
| 18:16:13 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Undecided,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 18:16:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 18:17:08 | mnaser | mriedem: woo, fix worked here | |
| 18:17:16 | mnaser | confirmed with the LOG.info message showed up as expected | |
| 18:17:34 | mnaser | with 3 consumer records for rp #4 as the one that was broken which we were looking at | |
| 18:18:30 | mriedem | rebasing | |
| 18:19:03 | mnaser | lol | |
| 18:19:08 | mnaser | i'm getting a KeyError now | |
| 18:19:12 | mnaser | (in placement) | |
| 18:19:57 | mnaser | http://paste.openstack.org/show/732269/ this time | |
| 18:20:51 | mnaser | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L3522 | |
| 18:20:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 18:21:16 | mriedem | tssurya: hmm, maybe that's hitting after an upgrade? | |
| 18:21:43 | mriedem | but, | |
| 18:21:54 | mriedem | if that were the case, ironic's grenade job should be broken | |
| 18:22:01 | mriedem | is the value changing? | |
| 18:22:51 | tssurya | so at the moment the only info I have is that we cherry-picked this to queens | |
| 18:23:06 | tssurya | maybe something is wrong because of that | |
| 18:24:04 | mriedem | tssurya: ok it's called from here in the RT https://review.openstack.org/#/c/571535/2/nova/compute/resource_tracker.py@617 | |
| 18:24:37 | mriedem | my guess is the compute node record already existed with a random uuid, and on restart of the compute service with the new code, it's trying to update the uuid in the compute node record using the ironic node uuid | |
| 18:24:53 | mriedem | i don't know why ironic's grenade job wouldn't fail for the same reason, but it seems like an obvious oversight in that patch | |
| 18:24:55 | tssurya | mriedem: yea | |
| 18:25:05 | tssurya | that's exactly what's happening | |
| 18:25:12 | tssurya | so this is for the existing nodes.. | |
| 18:25:15 | mriedem | \o/ | |
| 18:25:54 | mriedem | mnaser: as for https://github.com/openstack/nova/blob/stable/rocky/nova/api/openstack/placement/objects/resource_provider.py#L3486 | |
| 18:25:56 | mriedem | and the KeyError | |
| 18:26:00 | mriedem | i don't understand any of that code | |
| 18:26:10 | mriedem | that's all efried, tetsuro and jaypipes | |
| 18:27:06 | mriedem | https://review.openstack.org/#/c/559480/ | |
| 18:28:17 | tssurya | heh | |
| 18:28:34 | mriedem | tssurya: report a bug for the ironic node uuid thing i guess | |
| 18:28:49 | mriedem | jroll: do you know anything about the ironic grenade job? | |
| 18:28:52 | tssurya | mriedem: yea I will | |
| 18:28:54 | mriedem | does it restart n-cpu across releases? | |
| 18:29:34 | mnaser | mriedem: i think this is a weird env-related corner case where those havent done much reporting to placement | |
| 18:30:06 | mnaser | going to delete those stale rps | |
| 18:32:18 | mnaser | mriedem: every time i use osc-placement i have to say thank you | |
| 18:32:20 | mnaser | its such a life saver | |
| 18:34:06 | mriedem | don't thank me, thank avolkov | |
| 18:34:25 | mriedem | and rpodolykia | |
| 18:34:34 | mriedem | i know i butchered that irc | |
| 18:35:11 | mriedem | tssurya: yeah so you can set a readonly field while it's never been set | |
| 18:35:18 | mriedem | once it's set though, it's stuck | |
| 18:35:40 | tssurya | mriedem: ah right, thanks | |
| 18:35:45 | tssurya | I am filing a bug now | |
| 18:35:57 | mriedem | cdent: efried: https://review.openstack.org/#/q/topic:bug/1798163+(status:open+OR+status:merged) | |
| 18:36:02 | mriedem | dansmith: melwitt: ^ | |
| 18:39:08 | mriedem | starting on the placement repo cherry picks | |
| 18:39:39 | melwitt | 👀 | |
| 18:39:51 | openstack | Launchpad bug 1798172 in OpenStack Compute (nova) "Ironic driver tries to update the compute_node's UUID which of course fails in case of existing compute_nodes" [Undecided,New] | |
| 18:39:51 | tssurya | mriedem: https://bugs.launchpad.net/nova/+bug/1798172 | |
| 18:39:56 | mriedem | i'll assume that lego block is a pile-o-poo | |
| 18:40:03 | mriedem | "of course" | |
| 18:40:04 | mriedem | nice | |
| 18:40:06 | mriedem | :( | |
| 18:40:14 | melwitt | it's eyes, looking at the linked patches | |
| 18:40:25 | melwitt | but pile-o-poo could have been good | |
| 18:41:49 | dansmith | mriedem: so this has to be broken for anyone right? | |
| 18:41:59 | mriedem | dansmith: which thing? | |
| 18:42:01 | mriedem | placement? yes. | |
| 18:42:07 | dansmith | yeah | |
| 18:42:14 | mriedem | i guess mnaser is first to rocky ever | |
| 18:42:27 | mnaser | world first | |
| 18:42:32 | mnaser | but now | |
| 18:42:35 | mnaser | we have MORE | |
| 18:42:36 | mnaser | Exception during message handling: RPCVersionCapError: Requested message version, 5.0 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 4.17. | |
| 18:42:37 | mriedem | honestly i'm not sure how it wouldn't have been a problem caught in grenade | |
| 18:42:39 | dansmith | looks straightforward | |
| 18:42:47 | mnaser | nova-conductor when scheduling new instances | |
| 18:42:50 | melwitt | mnaser: need to get a windshield sticker for your car WORLD FIRST TO ROCKY | |
| 18:42:50 | mriedem | b/c in grenade we have existing instances with allocations | |
| 18:43:16 | mnaser | more like a military medal for the stuff i have to go through y'all | |
| 18:43:17 | mnaser | lol | |
| 18:43:27 | melwitt | :***( | |
| 18:43:33 | mnaser | all my packages are up to date, control plane is all rocky with queen computes | |
| 18:43:35 | melwitt | (those are tears) | |
| 18:43:37 | mnaser | its ok, i do it for the people | |
| 18:43:51 | mnaser | openstack upgrades are easy, i promise, i already helped fixed most of it | |
| 18:45:37 | mnaser | where are the versions listed again? in the rpc api file? | |
| 18:46:08 | mriedem | dansmith: so on upgrade to rocky we would have listed allocations for a provider here https://github.com/openstack/nova/blob/237bfcfd82fc28a955574b588fbce1d2392c9e45/nova/compute/resource_tracker.py#L1298 which should have hit the unique constraint | |
| 18:46:09 | mriedem | so idk | |
| 18:46:17 | mriedem | i mean in a grenade run | |
| 18:46:17 | dansmith | mriedem: yeah | |
| 18:46:27 | dansmith | mnaser: that rpc error looks like you have something old | |
| 18:46:41 | dansmith | like an old conductor still running maybe? | |
| 18:46:43 | mriedem | maybe we didn't hit it in grenade because the queens allocations already have project_id/user_id created? | |
| 18:46:45 | mriedem | s/created/set/ | |
| 18:47:04 | mriedem | or.... because we ran the online data migrations? | |
| 18:47:20 | mnaser | dansmith: oh shit, i missed a compute. | |
| 18:47:36 | mnaser | in pike=>queens | |
| 18:47:40 | melwitt | mriedem: well, normally allocations are required to be created with project/user and that's what creates a consumer. this bug is about the online data migration for allocations with missing consumers, prior to the microversion where we required project/user, right? | |
| 18:47:58 | dansmith | mnaser: mm, I dunno, if that's coming from a conductor node I think it's probably an old conductor, but I'd have to see more about where exactly | |
| 18:48:06 | melwitt | so grenade probably doesn't cover that, I wouldn't think | |