| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 16:59:53 | mriedem | tssurya: have you reported a bug? | |
| 17:00:03 | dansmith | mriedem: I got one already | |
| 17:00:06 | dansmith | coming | |
| 17:00:08 | tssurya | mriedem: no | |
| 17:00:09 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611094 | |
| 17:00:11 | tssurya | dansmith: ah thanks :) | |
| 17:00:13 | dansmith | mriedem: tssurya ^ | |
| 17:00:26 | mriedem | mnaser: well i'm not sure how it's happening, but clearly we could race if two requests are listing resource providers at the same time, | |
| 17:00:41 | mriedem | mnaser: so likely just bug report it and we can add a try/except for the duplicate entry error | |
| 17:00:59 | mnaser | mriedem: but i mean this is constantly happening and i think i cant provision any more vms on the cloud with a hostnotfound type of thing | |
| 17:00:59 | dansmith | mriedem: I meant devstack-based tests, but it's because we always have a transport_url set I think.. | |
| 17:01:04 | dansmith | and yeah, this'll be a rocky backport | |
| 17:01:51 | mriedem | mnaser: so i wonder if https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/api/openstack/placement/objects/resource_provider.py#L1940 is always True? | |
| 17:02:11 | mnaser | mriedem: thats what im trying to decipher | |
| 17:02:32 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611094 | |
| 17:04:12 | mnaser | mriedem: i think this is a cloud with some brokenness that is being exposed | |
| 17:04:38 | mnaser | right now, there are records in allocations for that specific uuid under resource_provider_id=4 | |
| 17:04:53 | mnaser | but in that traceback, it tries to add allocations for resource_provider_id=5 | |
| 17:05:08 | mnaser | sorry, no, i lied | |
| 17:05:12 | mnaser | it actaully tries to add for 4 | |
| 17:05:39 | mnaser | let me see if there is anything in cosnumers | |
| 17:06:08 | tssurya | dansmith: thanks a lot | |
| 17:07:25 | mnaser | i think this sql server is f'd | |
| 17:07:37 | mnaser | select * from consumers where uuid='fbee657b-6a60-4525-b7c8-b070643404ec'; returns nothing | |
| 17:07:45 | mriedem | there aren't consumer records until rocky | |
| 17:08:19 | mriedem | and the data migration function that is blowing up is trying to populate that table from existing allocations records | |
| 17:08:46 | dansmith | tssurya: np | |
| 17:08:51 | mnaser | mriedem: right, but the traceback seems to say: Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid' | |
| 17:09:02 | mnaser | yet -- select * from consumers where uuid='fbee657b-6a60-4525-b7c8-b070643404ec'; -- returns nothing | |
| 17:09:38 | mriedem | hmm it's doing an insert from select, | |
| 17:09:46 | mriedem | so the select results probably have duplicates | |
| 17:09:50 | mriedem | and those aren't being trimmed | |
| 17:09:58 | mriedem | and i bet the test for this only had 1 allocation against 1 provider | |
| 17:10:02 | mriedem | or something like that | |
| 17:10:25 | mnaser | lets test that out | |
| 17:10:33 | mriedem | would be nice to see what the select query results are | |
| 17:10:50 | mriedem | the sql-fu in here is hard for me to grok | |
| 17:11:15 | mnaser | mriedem: you're right | |
| 17:11:19 | mnaser | 9 rows returned from that | |
| 17:11:27 | mriedem | what's the select query? | |
| 17:11:59 | mnaser | mriedem: http://paste.openstack.org/show/732263/ | |
| 17:12:02 | mnaser | stole this from the traceback | |
| 17:12:16 | mnaser | it was right next to the error | |
| 17:14:03 | mnaser | mriedem: yup.. i see 9 records but really 3 unique ones | |
| 17:15:20 | mriedem | ok, so i bet 3 instances with allocations against a single provider, and each instance has 3 resource class allocations (VCPU, MEMORY_MB and DISK_GB) | |
| 17:15:34 | mriedem | and we're not collapsing those 3 allocations for the same consumer into a single consumer entry | |
| 17:16:21 | mriedem | let me see if i can dig up what is supposed to be testing this | |
| 17:17:04 | mnaser | mriedem: thats exactly the case | |
| 17:17:13 | mriedem | \o/ | |
| 17:17:19 | mnaser | i can confirm same resource provider each, with 3 resource classes | |
| 17:17:34 | mnaser | how come the others didnt break when migrating | |
| 17:17:48 | mnaser | i mean this isn't exactly an outlier | |
| 17:17:59 | mriedem | don't know | |
| 17:18:13 | mriedem | https://review.openstack.org/#/c/565405/26/nova/tests/functional/api/openstack/placement/db/test_consumer.py is only testing with 3 unique allocations each with a single resource class | |
| 17:18:18 | mriedem | so that's why i guess tests didn't catch it | |
| 17:19:00 | mnaser | poop | |
| 17:19:07 | mnaser | well | |
| 17:19:13 | mnaser | i guess i gotta find a fix | |
| 17:19:44 | mriedem | i'm having a hard f'ing time understanding these test | |
| 17:19:45 | mriedem | *tests | |
| 17:19:50 | mnaser | yeah :\ | |
| 17:19:55 | mnaser | and the whole logic too | |
| 17:20:56 | mriedem | well for the select query, i'd think we need to group the allocations records results by consumer_id | |
| 17:21:49 | mriedem | really need a recreate in a test to see how to fix this | |
| 17:21:52 | mnaser | mriedem: or maybe just even a select distinct? | |
| 17:21:59 | mnaser | but yes, i agree | |
| 17:22:12 | mriedem | yeah true, | |
| 17:22:16 | mriedem | again, wish jay was here | |
| 17:22:39 | mriedem | also, this is a placement bug so maybe i can just kick you over to that channel and let those guys fix it :P | |
| 17:23:06 | mnaser | lolll | |
| 17:23:17 | mnaser | i mean you're not wrong | |
| 17:23:29 | mnaser | i'll take this to #openstack-placement | |
| 17:24:45 | mnaser | mriedem: on a nova note im thinking maybe live migrate those machines and cheat | |
| 17:24:46 | mnaser | lol | |
| 17:35:57 | openstackgerrit | melanie witt proposed openstack/nova master: Bump os-brick version to 2.6.1 https://review.openstack.org/611109 | |
| 17:49:16 | mriedem | mnaser: ^ fugly but works | |
| 17:49:16 | openstack | bug 1798163 in OpenStack Compute (nova) "Placement incomplete consumers online migration fails" [Undecided,New] https://launchpad.net/bugs/1798163 | |
| 17:49:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add recreate test for bug 1798163 https://review.openstack.org/611113 | |
| 17:50:08 | mnaser | mriedem: i have a fix | |
| 17:50:20 | mnaser | do you want me to squash into yours or rather two patch? | |
| 17:55:37 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 17:57:32 | mnaser | oops missed an uncomment | |
| 17:57:37 | mnaser | testing functional tests again locally | |
| 17:59:42 | melwitt | mriedem: should we wait for the tempest tests to merge before marking https://blueprints.launchpad.net/nova/+spec/boot-instance-specific-storage-backend as complete? | |
| 18:00:43 | mnaser | bleh | |
| 18:00:44 | mnaser | another thing broke | |
| 18:05:52 | mnaser | ok yay | |
| 18:05:54 | mnaser | i was missing somethig | |
| 18:06:37 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Use unique consumer_id when doing online data migration https://review.openstack.org/611115 | |
| 18:09:43 | mriedem | mnaser: i think i might clean mine up to be a simple new test, and then you can stack on top | |
| 18:10:10 | mnaser | mriedem: feel free to checkout mine locally i you want | |
| 18:10:10 | mriedem | i was rushing b/c my pizza was getting cold | |
| 18:10:20 | mnaser | mriedem: very valid reason tbh | |
| 18:10:23 | mriedem | melwitt: no, i was going to mark it today but forgot | |
| 18:10:32 | mnaser | i was rushing because i have a broken cloud :-) | |
| 18:10:40 | melwitt | ok, I can mark it | |
| 18:10:41 | mnaser | but i'm comfortable pushing that one liner now | |
| 18:10:48 | mriedem | i'm glad i deleted my vexxhost vm the other day then | |
| 18:10:49 | mriedem | :) | |
| 18:11:36 | mnaser | its not our public cloud | |