| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 17:07:45 | mriedem | there aren't consumer records until rocky | |
| 17:08:19 | mriedem | and the data migration function that is blowing up is trying to populate that table from existing allocations records | |
| 17:08:46 | dansmith | tssurya: np | |
| 17:08:51 | mnaser | mriedem: right, but the traceback seems to say: Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid' | |
| 17:09:02 | mnaser | yet -- select * from consumers where uuid='fbee657b-6a60-4525-b7c8-b070643404ec'; -- returns nothing | |
| 17:09:38 | mriedem | hmm it's doing an insert from select, | |
| 17:09:46 | mriedem | so the select results probably have duplicates | |
| 17:09:50 | mriedem | and those aren't being trimmed | |
| 17:09:58 | mriedem | and i bet the test for this only had 1 allocation against 1 provider | |
| 17:10:02 | mriedem | or something like that | |
| 17:10:25 | mnaser | lets test that out | |
| 17:10:33 | mriedem | would be nice to see what the select query results are | |
| 17:10:50 | mriedem | the sql-fu in here is hard for me to grok | |
| 17:11:15 | mnaser | mriedem: you're right | |
| 17:11:19 | mnaser | 9 rows returned from that | |
| 17:11:27 | mriedem | what's the select query? | |
| 17:11:59 | mnaser | mriedem: http://paste.openstack.org/show/732263/ | |
| 17:12:02 | mnaser | stole this from the traceback | |
| 17:12:16 | mnaser | it was right next to the error | |
| 17:14:03 | mnaser | mriedem: yup.. i see 9 records but really 3 unique ones | |
| 17:15:20 | mriedem | ok, so i bet 3 instances with allocations against a single provider, and each instance has 3 resource class allocations (VCPU, MEMORY_MB and DISK_GB) | |
| 17:15:34 | mriedem | and we're not collapsing those 3 allocations for the same consumer into a single consumer entry | |
| 17:16:21 | mriedem | let me see if i can dig up what is supposed to be testing this | |
| 17:17:04 | mnaser | mriedem: thats exactly the case | |
| 17:17:13 | mriedem | \o/ | |
| 17:17:19 | mnaser | i can confirm same resource provider each, with 3 resource classes | |
| 17:17:34 | mnaser | how come the others didnt break when migrating | |
| 17:17:48 | mnaser | i mean this isn't exactly an outlier | |
| 17:17:59 | mriedem | don't know | |
| 17:18:13 | mriedem | https://review.openstack.org/#/c/565405/26/nova/tests/functional/api/openstack/placement/db/test_consumer.py is only testing with 3 unique allocations each with a single resource class | |
| 17:18:18 | mriedem | so that's why i guess tests didn't catch it | |
| 17:19:00 | mnaser | poop | |
| 17:19:07 | mnaser | well | |
| 17:19:13 | mnaser | i guess i gotta find a fix | |
| 17:19:44 | mriedem | i'm having a hard f'ing time understanding these test | |
| 17:19:45 | mriedem | *tests | |
| 17:19:50 | mnaser | yeah :\ | |
| 17:19:55 | mnaser | and the whole logic too | |
| 17:20:56 | mriedem | well for the select query, i'd think we need to group the allocations records results by consumer_id | |
| 17:21:49 | mriedem | really need a recreate in a test to see how to fix this | |
| 17:21:52 | mnaser | mriedem: or maybe just even a select distinct? | |
| 17:21:59 | mnaser | but yes, i agree | |
| 17:22:12 | mriedem | yeah true, | |
| 17:22:16 | mriedem | again, wish jay was here | |
| 17:22:39 | mriedem | also, this is a placement bug so maybe i can just kick you over to that channel and let those guys fix it :P | |
| 17:23:06 | mnaser | lolll | |
| 17:23:17 | mnaser | i mean you're not wrong | |
| 17:23:29 | mnaser | i'll take this to #openstack-placement | |
| 17:24:45 | mnaser | mriedem: on a nova note im thinking maybe live migrate those machines and cheat | |
| 17:24:46 | mnaser | lol | |
| 17:35:57 | openstackgerrit | melanie witt proposed openstack/nova master: Bump os-brick version to 2.6.1 https://review.openstack.org/611109 | |
| 17:49:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 17:49:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Undecided,New] https://launchpad.net/bugs/1798163 | |
| 17:49:16 | mriedem | mnaser: ^ fugly but works | |
| 17:50:08 | mnaser | mriedem: i have a fix | |
| 17:50:20 | mnaser | do you want me to squash into yours or rather two patch? | |
| 17:55:37 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 17:57:32 | mnaser | oops missed an uncomment | |
| 17:57:37 | mnaser | testing functional tests again locally | |
| 17:59:42 | melwitt | mriedem: should we wait for the tempest tests to merge before marking https://blueprints.launchpad.net/nova/+spec/boot-instance-specific-storage-backend as complete? | |
| 18:00:43 | mnaser | bleh | |
| 18:00:44 | mnaser | another thing broke | |
| 18:05:52 | mnaser | ok yay | |
| 18:05:54 | mnaser | i was missing somethig | |
| 18:06:37 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 18:09:43 | mriedem | mnaser: i think i might clean mine up to be a simple new test, and then you can stack on top | |
| 18:10:10 | mriedem | i was rushing b/c my pizza was getting cold | |
| 18:10:10 | mnaser | mriedem: feel free to checkout mine locally i you want | |
| 18:10:20 | mnaser | mriedem: very valid reason tbh | |
| 18:10:23 | mriedem | melwitt: no, i was going to mark it today but forgot | |
| 18:10:32 | mnaser | i was rushing because i have a broken cloud :-) | |
| 18:10:40 | melwitt | ok, I can mark it | |
| 18:10:41 | mnaser | but i'm comfortable pushing that one liner now | |
| 18:10:48 | mriedem | i'm glad i deleted my vexxhost vm the other day then | |
| 18:10:49 | mriedem | :) | |
| 18:11:36 | mnaser | its not our public cloud | |
| 18:11:41 | mnaser | i wouldnt trust those nova people and the code they ship | |
| 18:11:43 | mnaser | :p | |
| 18:11:49 | mriedem | good plan | |
| 18:13:36 | tssurya | mriedem: question about https://review.openstack.org/#/c/571535/ . How are we able to set the compute_node.uuid which is a read-only attribute ? Am I missing something here ? | |
| 18:14:30 | mriedem | please hold dear caller | |
| 18:14:36 | tssurya | ack :) | |
| 18:14:51 | mriedem | we create compute node records with uuids in tests all the time | |
| 18:15:58 | tssurya | true.. I am somehow hitting https://pastebin.com/kqyu66wB | |
| 18:16:02 | tssurya | will look closer | |
| 18:16:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 18:16:13 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Undecided,In progress] https://launchpad.net/bugs/1798163 - Assigned to Mohammed Naser (mnaser) | |
| 18:17:08 | mnaser | mriedem: woo, fix worked here | |
| 18:17:16 | mnaser | confirmed with the LOG.info message showed up as expected | |
| 18:17:34 | mnaser | with 3 consumer records for rp #4 as the one that was broken which we were looking at | |
| 18:18:30 | mriedem | rebasing | |
| 18:19:03 | mnaser | lol | |
| 18:19:08 | mnaser | i'm getting a KeyError now | |
| 18:19:12 | mnaser | (in placement) | |
| 18:19:57 | mnaser | http://paste.openstack.org/show/732269/ this time | |
| 18:20:51 | mnaser | https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L3522 | |
| 18:20:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 18:21:16 | mriedem | tssurya: hmm, maybe that's hitting after an upgrade? | |
| 18:21:43 | mriedem | but, | |
| 18:21:54 | mriedem | if that were the case, ironic's grenade job should be broken | |