| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 17:24:45 | mnaser | mriedem: on a nova note im thinking maybe live migrate those machines and cheat | |
| 17:24:46 | mnaser | lol | |
| 17:35:57 | openstackgerrit | melanie witt proposed openstack/nova master: Bump os-brick version to 2.6.1 https://review.openstack.org/611109 | |
| 17:49:16 | mriedem | mnaser: ^ fugly but works | |
| 17:49:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Undecided,New] https://launchpad.net/bugs/1798163 | |
| 17:49:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 17:50:08 | mnaser | mriedem: i have a fix | |
| 17:50:20 | mnaser | do you want me to squash into yours or rather two patch? | |
| 17:55:37 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 17:57:32 | mnaser | oops missed an uncomment | |
| 17:57:37 | mnaser | testing functional tests again locally | |
| 17:59:42 | melwitt | mriedem: should we wait for the tempest tests to merge before marking https://blueprints.launchpad.net/nova/+spec/boot-instance-specific-storage-backend as complete? | |
| 18:00:43 | mnaser | bleh | |
| 18:00:44 | mnaser | another thing broke | |
| 18:05:52 | mnaser | ok yay | |
| 18:05:54 | mnaser | i was missing somethig | |
| 18:06:37 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 18:09:43 | mriedem | mnaser: i think i might clean mine up to be a simple new test, and then you can stack on top | |
| 18:10:10 | mnaser | mriedem: feel free to checkout mine locally i you want | |
| 18:10:10 | mriedem | i was rushing b/c my pizza was getting cold | |
| 18:10:20 | mnaser | mriedem: very valid reason tbh | |
| 18:10:23 | mriedem | melwitt: no, i was going to mark it today but forgot | |
| 18:10:32 | mnaser | i was rushing because i have a broken cloud :-) | |
| 18:10:40 | melwitt | ok, I can mark it | |
| 18:10:41 | mnaser | but i'm comfortable pushing that one liner now | |
| 18:10:48 | mriedem | i'm glad i deleted my vexxhost vm the other day then | |
| 18:10:49 | mriedem | :) | |
| 18:11:36 | mnaser | its not our public cloud | |
| 18:11:41 | mnaser | i wouldnt trust those nova people and the code they ship | |
| 18:11:43 | mnaser | :p | |
| 18:11:49 | mriedem | good plan | |
| 18:13:36 | tssurya | mriedem: question about https://review.openstack.org/#/c/571535/ . How are we able to set the compute_node.uuid which is a read-only attribute ? Am I missing something here ? | |
| 18:14:30 | mriedem | please hold dear caller | |
| 18:14:36 | tssurya | ack :) | |
| 18:14:51 | mriedem | we create compute node records with uuids in tests all the time | |
| 18:15:58 | tssurya | true.. I am somehow hitting https://pastebin.com/kqyu66wB | |
| 18:16:02 | tssurya | will look closer | |
| 18:16:13 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Undecided,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 18:16:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 18:17:08 | mnaser | mriedem: woo, fix worked here | |
| 18:17:16 | mnaser | confirmed with the LOG.info message showed up as expected | |
| 18:17:34 | mnaser | with 3 consumer records for rp #4 as the one that was broken which we were looking at | |
| 18:18:30 | mriedem | rebasing | |
| 18:19:03 | mnaser | lol | |
| 18:19:08 | mnaser | i'm getting a KeyError now | |
| 18:19:12 | mnaser | (in placement) | |
| 18:19:57 | mnaser | http://paste.openstack.org/show/732269/ this time | |
| 18:20:51 | mnaser | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L3522 | |
| 18:20:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 18:21:16 | mriedem | tssurya: hmm, maybe that's hitting after an upgrade? | |
| 18:21:43 | mriedem | but, | |
| 18:21:54 | mriedem | if that were the case, ironic's grenade job should be broken | |
| 18:22:01 | mriedem | is the value changing? | |
| 18:22:51 | tssurya | so at the moment the only info I have is that we cherry-picked this to queens | |
| 18:23:06 | tssurya | maybe something is wrong because of that | |
| 18:24:04 | mriedem | tssurya: ok it's called from here in the RT https://review.openstack.org/#/c/571535/2/nova/compute/resource_tracker.py@617 | |
| 18:24:37 | mriedem | my guess is the compute node record already existed with a random uuid, and on restart of the compute service with the new code, it's trying to update the uuid in the compute node record using the ironic node uuid | |
| 18:24:53 | mriedem | i don't know why ironic's grenade job wouldn't fail for the same reason, but it seems like an obvious oversight in that patch | |
| 18:24:55 | tssurya | mriedem: yea | |
| 18:25:05 | tssurya | that's exactly what's happening | |
| 18:25:12 | tssurya | so this is for the existing nodes.. | |
| 18:25:15 | mriedem | \o/ | |
| 18:25:54 | mriedem | mnaser: as for https://github.com/openstack/nova/blob/stable/rocky/nova/api/openstack/placement/objects/resource_provider.py#L3486 | |
| 18:25:56 | mriedem | and the KeyError | |
| 18:26:00 | mriedem | i don't understand any of that code | |
| 18:26:10 | mriedem | that's all efried, tetsuro and jaypipes | |
| 18:27:06 | mriedem | https://review.openstack.org/#/c/559480/ | |
| 18:28:17 | tssurya | heh | |
| 18:28:34 | mriedem | tssurya: report a bug for the ironic node uuid thing i guess | |
| 18:28:49 | mriedem | jroll: do you know anything about the ironic grenade job? | |
| 18:28:52 | tssurya | mriedem: yea I will | |
| 18:28:54 | mriedem | does it restart n-cpu across releases? | |
| 18:29:34 | mnaser | mriedem: i think this is a weird env-related corner case where those havent done much reporting to placement | |
| 18:30:06 | mnaser | going to delete those stale rps | |
| 18:32:18 | mnaser | mriedem: every time i use osc-placement i have to say thank you | |
| 18:32:20 | mnaser | its such a life saver | |
| 18:34:06 | mriedem | don't thank me, thank avolkov | |
| 18:34:25 | mriedem | and rpodolykia | |
| 18:34:34 | mriedem | i know i butchered that irc | |
| 18:35:11 | mriedem | tssurya: yeah so you can set a readonly field while it's never been set | |
| 18:35:18 | mriedem | once it's set though, it's stuck | |
| 18:35:40 | tssurya | mriedem: ah right, thanks | |
| 18:35:45 | tssurya | I am filing a bug now | |
| 18:35:57 | mriedem | cdent: efried: https://review.openstack.org/#/q/topic:bug/1798163+(status:open+OR+status:merged) | |
| 18:36:02 | mriedem | dansmith: melwitt: ^ | |
| 18:39:08 | mriedem | starting on the placement repo cherry picks | |
| 18:39:39 | melwitt | 👀 | |
| 18:39:51 | openstack | Launchpad bug 1798172 in OpenStack Compute (nova) "Ironic driver tries to update the compute_node's UUID which of course fails in case of existing compute_nodes" [Undecided,New] | |
| 18:39:51 | tssurya | mriedem: https://bugs.launchpad.net/nova/+bug/1798172 | |
| 18:39:56 | mriedem | i'll assume that lego block is a pile-o-poo | |
| 18:40:03 | mriedem | "of course" | |
| 18:40:04 | mriedem | nice | |
| 18:40:06 | mriedem | :( | |
| 18:40:14 | melwitt | it's eyes, looking at the linked patches | |
| 18:40:25 | melwitt | but pile-o-poo could have been good | |
| 18:41:49 | dansmith | mriedem: so this has to be broken for anyone right? | |
| 18:41:59 | mriedem | dansmith: which thing? | |
| 18:42:01 | mriedem | placement? yes. | |
| 18:42:07 | dansmith | yeah | |
| 18:42:14 | mriedem | i guess mnaser is first to rocky ever | |