Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-16
16:43:18 openstackgerrit Artom Lifshitz proposed openstack/nova stable/rocky: Handle volume API failure in _post_live_migration https://review.openstack.org/611083
16:44:01 mriedem mnaser: traceback in a paste?
16:44:41 tssurya dansmith: had a question about the cell templating stuff,
16:44:43 openstackgerrit Artom Lifshitz proposed openstack/nova stable/queens: Handle volume API failure in _post_live_migration https://review.openstack.org/611084
16:44:51 dansmith tssurya: yah?
16:45:00 mnaser mriedem: http://paste.openstack.org/show/732260/
16:45:02 tssurya shouldn't we consider the cell0's transport_url here : https://github.com/openstack/nova/blob/a53e46a75936b55c93face840764a67f2186cb11/nova/objects/cell_mapping.py#L144 ?
16:45:05 mnaser OH also fun little thing i found out about today
16:45:08 openstackgerrit Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088
16:45:14 stephenfin artom: Thoughts? https://review.openstack.org/611088
16:45:22 mnaser Q=>R upgrades requires you to run api_db sync first then db sync after (i dont think this is documented)
16:45:26 tssurya right now running db sync without local_cell parameter gives out errors
16:45:43 mnaser because cell disabled field is missing from api database so the db sync fails
16:45:46 stephenfin artom: I'd personally like to backport that as far as we can go. I'm kind of sick of explaining how broken this is to people
16:46:00 mnaser api_db sync first adds that field, which then lets db sync do it after
16:46:03 mriedem mnaser: i think it's ordered that way in the upgrade docs
16:46:11 melwitt
16:46:12 mnaser really, let me double check
16:46:24 dansmith tssurya: not sure what you mean.. pastebin an error?
16:46:25 mriedem https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process
16:46:28 tssurya mnaser, mriedem: yea someone ran into the same issue and we changed the order
16:46:31 mriedem "Using the newly installed nova code, run the DB sync. (nova-manage api_db sync; nova-manage db sync). These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail."
16:46:32 tssurya dansmith: ok
16:46:37 artom stephenfin, I don't know the full history, but I feel like it's opening a can of worms
16:47:00 mnaser mriedem: serves me right for looking at the queens docs thinking it hasnt change because it "looks" the same
16:47:06 artom stephenfin, also, I could imagine a scenario where an operator really pinky swears the destination host is fine, and wants to live migrate regardless
16:47:10 mnaser you're right, the order was swapped in rocky, my bad
16:47:23 artom stephenfin, so I'm not sure I'm comfortable with such a heavy handed approach
16:47:31 mnaser but anyways, back to that gigantic traceback
16:47:32 mriedem mnaser: i think grenade was doing it the right way before that docs change,
16:47:34 mriedem our docs were just old
16:47:37 tssurya dansmith: https://pastebin.com/7cQKv0fz
16:47:41 artom stephenfin, totally get where you're coming from though :)
16:47:44 stephenfin artom: Fair point. Wanna stick your thoughts in that review?
16:48:01 dansmith tssurya: oh because cell0's transport_url column can be NULL ?
16:48:03 artom stephenfin, yep, will do
16:48:15 stephenfin ta
16:48:25 tssurya yea
16:48:34 dansmith tssurya: add "and val" here: https://github.com/openstack/nova/blob/a53e46a75936b55c93face840764a67f2186cb11/nova/objects/cell_mapping.py#L162
16:48:51 dansmith tssurya: you gonna cook up a patch or do you want me to?
16:49:09 dansmith tssurya: I wonder why/how we're not hitting that in the gate?
16:49:15 dansmith do we set it to something bogus?
16:49:29 tssurya dansmith: would be nice if you do it..
16:49:33 dansmith tssurya: sure
16:49:46 mriedem mnaser: so this is being triggered when listing resource providers, what is doing that?
16:49:46 tssurya we were just doing the upgrade checks for rocky and saw this
16:50:03 mnaser mriedem: i'm assuming nova-scheduler?
16:50:26 mnaser https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/api/openstack/placement/objects/resource_provider.py#L1923-L1965
16:50:27 mnaser this is interesting
16:50:56 dansmith tssurya: hmm, actually, it's not nullable on the object, so I'm not sure how it'd be doing the right thing if you have NULL in the db
16:51:16 tssurya dansmith: but we can have NULL on the conf file
16:51:38 tssurya technically there is no requirement to produce a transport_url for db sync right ?
16:52:15 tssurya s/to produce/to supply
16:52:16 dansmith tssurya: ohh, I see, I thought you were saying it was NULL in the database
16:52:24 mnaser [placement] incomplete_consumer_project_id and incomplete_consumer_user_id are a thing, i guess
16:52:45 mriedem mnaser: oh i guess adding/removing host aggregates in rocky would do it b/c we have to find the provider by name which we use GET /resource_providers?name=foo for that
16:52:45 tssurya no not in the database. but maybe we hit it here ? https://github.com/openstack/nova/blob/a53e46a75936b55c93face840764a67f2186cb11/nova/objects/cell_mapping.py#L150
16:52:50 mriedem and when building the "provider tree"
16:53:24 mriedem when reporting inventory
16:53:27 mriedem from the compute
16:53:42 mnaser mriedem: the 500 is coming on /resource_providers/foo/allocations"
16:53:52 dansmith tssurya: so there is already a check for None-ness, but I guess if the value in the db isn't a template we'll fail to exit
16:54:13 mnaser and what it seems like almost any time its requesting allocations
16:54:22 mriedem mnaser: yeah that happens in the resource tracker on the compute
16:54:24 tssurya dansmith: ah yea was just wondering why that check didn't catch the Noneness
16:54:26 mriedem _remove_deleted_instances_allocations method
16:54:41 mnaser yep, i see compute ips sending in that request
16:54:44 mriedem so on startup of the compute, it's going to list allocations for the given compute ndoe provider
16:55:04 mriedem and try to create consumers table records for that provider and any allocations against it
16:55:32 mriedem i'm not sure how we race to hit _create_incomplete_consumers_for_provider though
16:55:41 mriedem because that should be idempotent
16:55:46 mriedem and jaypipes isn't around
16:55:55 dansmith tssurya: I'm shocked we haven't seen this in the regular tests
16:56:27 tssurya dansmith: yea we should have seen this somewhere
16:56:43 tssurya not sure if people didn't hit this when moving to rocky ?
16:57:34 dansmith well, I'm not sure why devstack doesn't hit it
16:57:40 dansmith I guess because we always have those defined in config
16:57:58 tssurya probably yea, but its easily reproducible
16:58:04 dansmith yep
16:58:05 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093
16:58:18 tssurya will you also backport this please ? we might need this in queens
16:58:19 mriedem our base test case uses the rpc fixture https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/test.py#L238
16:58:25 mriedem so that's probably why we'd never hit it?
16:58:42 mriedem tssurya: the template stuff wasn't in queens
16:59:18 mriedem https://github.com/openstack/nova/blob/396156eb13521a0e7af4488a8cd4693aa65a0da2/nova/tests/fixtures.py#L728
16:59:38 mriedem all of our tests at least configure this: transport_url = 'fake:/'
16:59:44 tssurya mriedem: oh yea sorry rocky then
16:59:53 mriedem tssurya: have you reported a bug?
17:00:03 dansmith mriedem: I got one already
17:00:06 dansmith coming
17:00:08 tssurya mriedem: no
17:00:09 openstackgerrit Dan Smith proposed openstack/nova master: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611094
17:00:11 tssurya dansmith: ah thanks :)
17:00:13 dansmith mriedem: tssurya ^
17:00:26 mriedem mnaser: well i'm not sure how it's happening, but clearly we could race if two requests are listing resource providers at the same time,
17:00:41 mriedem mnaser: so likely just bug report it and we can add a try/except for the duplicate entry error
17:00:59 mnaser mriedem: but i mean this is constantly happening and i think i cant provision any more vms on the cloud with a hostnotfound type of thing
17:00:59 dansmith mriedem: I meant devstack-based tests, but it's because we always have a transport_url set I think..
17:01:04 dansmith and yeah, this'll be a rocky backport
17:01:51 mriedem mnaser: so i wonder if https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/api/openstack/placement/objects/resource_provider.py#L1940 is always True?

Earlier   Later