Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-16
16:52:24 mnaser [placement] incomplete_consumer_project_id and incomplete_consumer_user_id are a thing, i guess
16:52:45 mriedem mnaser: oh i guess adding/removing host aggregates in rocky would do it b/c we have to find the provider by name which we use GET /resource_providers?name=foo for that
16:52:45 tssurya no not in the database. but maybe we hit it here ? https://github.com/openstack/nova/blob/a53e46a75936b55c93face840764a67f2186cb11/nova/objects/cell_mapping.py#L150
16:52:50 mriedem and when building the "provider tree"
16:53:24 mriedem when reporting inventory
16:53:27 mriedem from the compute
16:53:42 mnaser mriedem: the 500 is coming on /resource_providers/foo/allocations"
16:53:52 dansmith tssurya: so there is already a check for None-ness, but I guess if the value in the db isn't a template we'll fail to exit
16:54:13 mnaser and what it seems like almost any time its requesting allocations
16:54:22 mriedem mnaser: yeah that happens in the resource tracker on the compute
16:54:24 tssurya dansmith: ah yea was just wondering why that check didn't catch the Noneness
16:54:26 mriedem _remove_deleted_instances_allocations method
16:54:41 mnaser yep, i see compute ips sending in that request
16:54:44 mriedem so on startup of the compute, it's going to list allocations for the given compute ndoe provider
16:55:04 mriedem and try to create consumers table records for that provider and any allocations against it
16:55:32 mriedem i'm not sure how we race to hit _create_incomplete_consumers_for_provider though
16:55:41 mriedem because that should be idempotent
16:55:46 mriedem and jaypipes isn't around
16:55:55 dansmith tssurya: I'm shocked we haven't seen this in the regular tests
16:56:27 tssurya dansmith: yea we should have seen this somewhere
16:56:43 tssurya not sure if people didn't hit this when moving to rocky ?
16:57:34 dansmith well, I'm not sure why devstack doesn't hit it
16:57:40 dansmith I guess because we always have those defined in config
16:57:58 tssurya probably yea, but its easily reproducible
16:58:04 dansmith yep
16:58:05 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093
16:58:18 tssurya will you also backport this please ? we might need this in queens
16:58:19 mriedem our base test case uses the rpc fixture https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/test.py#L238
16:58:25 mriedem so that's probably why we'd never hit it?
16:58:42 mriedem tssurya: the template stuff wasn't in queens
16:59:18 mriedem https://github.com/openstack/nova/blob/396156eb13521a0e7af4488a8cd4693aa65a0da2/nova/tests/fixtures.py#L728
16:59:38 mriedem all of our tests at least configure this: transport_url = 'fake:/'
16:59:44 tssurya mriedem: oh yea sorry rocky then
16:59:53 mriedem tssurya: have you reported a bug?
17:00:03 dansmith mriedem: I got one already
17:00:06 dansmith coming
17:00:08 tssurya mriedem: no
17:00:09 openstackgerrit Dan Smith proposed openstack/nova master: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611094
17:00:11 tssurya dansmith: ah thanks :)
17:00:13 dansmith mriedem: tssurya ^
17:00:26 mriedem mnaser: well i'm not sure how it's happening, but clearly we could race if two requests are listing resource providers at the same time,
17:00:41 mriedem mnaser: so likely just bug report it and we can add a try/except for the duplicate entry error
17:00:59 mnaser mriedem: but i mean this is constantly happening and i think i cant provision any more vms on the cloud with a hostnotfound type of thing
17:00:59 dansmith mriedem: I meant devstack-based tests, but it's because we always have a transport_url set I think..
17:01:04 dansmith and yeah, this'll be a rocky backport
17:01:51 mriedem mnaser: so i wonder if https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/api/openstack/placement/objects/resource_provider.py#L1940 is always True?
17:02:11 mnaser mriedem: thats what im trying to decipher
17:02:32 openstackgerrit Dan Smith proposed openstack/nova master: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611094
17:04:12 mnaser mriedem: i think this is a cloud with some brokenness that is being exposed
17:04:38 mnaser right now, there are records in allocations for that specific uuid under resource_provider_id=4
17:04:53 mnaser but in that traceback, it tries to add allocations for resource_provider_id=5
17:05:08 mnaser sorry, no, i lied
17:05:12 mnaser it actaully tries to add for 4
17:05:39 mnaser let me see if there is anything in cosnumers
17:06:08 tssurya dansmith: thanks a lot
17:07:25 mnaser i think this sql server is f'd
17:07:37 mnaser select * from consumers where uuid='fbee657b-6a60-4525-b7c8-b070643404ec'; returns nothing
17:07:45 mriedem there aren't consumer records until rocky
17:08:19 mriedem and the data migration function that is blowing up is trying to populate that table from existing allocations records
17:08:46 dansmith tssurya: np
17:08:51 mnaser mriedem: right, but the traceback seems to say: Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid'
17:09:02 mnaser yet -- select * from consumers where uuid='fbee657b-6a60-4525-b7c8-b070643404ec'; -- returns nothing
17:09:38 mriedem hmm it's doing an insert from select,
17:09:46 mriedem so the select results probably have duplicates
17:09:50 mriedem and those aren't being trimmed
17:09:58 mriedem and i bet the test for this only had 1 allocation against 1 provider
17:10:02 mriedem or something like that
17:10:25 mnaser lets test that out
17:10:33 mriedem would be nice to see what the select query results are
17:10:50 mriedem the sql-fu in here is hard for me to grok
17:11:15 mnaser mriedem: you're right
17:11:19 mnaser 9 rows returned from that
17:11:27 mriedem what's the select query?
17:11:59 mnaser mriedem: http://paste.openstack.org/show/732263/
17:12:02 mnaser stole this from the traceback
17:12:16 mnaser it was right next to the error
17:14:03 mnaser mriedem: yup.. i see 9 records but really 3 unique ones
17:15:20 mriedem ok, so i bet 3 instances with allocations against a single provider, and each instance has 3 resource class allocations (VCPU, MEMORY_MB and DISK_GB)
17:15:34 mriedem and we're not collapsing those 3 allocations for the same consumer into a single consumer entry
17:16:21 mriedem let me see if i can dig up what is supposed to be testing this
17:17:04 mnaser mriedem: thats exactly the case
17:17:13 mriedem \o/
17:17:19 mnaser i can confirm same resource provider each, with 3 resource classes
17:17:34 mnaser how come the others didnt break when migrating
17:17:48 mnaser i mean this isn't exactly an outlier
17:17:59 mriedem don't know
17:18:13 mriedem https://review.openstack.org/#/c/565405/26/nova/tests/functional/api/openstack/placement/db/test_consumer.py is only testing with 3 unique allocations each with a single resource class
17:18:18 mriedem so that's why i guess tests didn't catch it
17:19:00 mnaser poop
17:19:07 mnaser well
17:19:13 mnaser i guess i gotta find a fix
17:19:44 mriedem i'm having a hard f'ing time understanding these test
17:19:45 mriedem *tests
17:19:50 mnaser yeah :\
17:19:55 mnaser and the whole logic too
17:20:56 mriedem well for the select query, i'd think we need to group the allocations records results by consumer_id
17:21:49 mriedem really need a recreate in a test to see how to fix this
17:21:52 mnaser mriedem: or maybe just even a select distinct?
17:21:59 mnaser but yes, i agree
17:22:12 mriedem yeah true,

Earlier   Later