| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-16 | |||
| 16:46:03 | mriedem | mnaser: i think it's ordered that way in the upgrade docs | |
| 16:46:11 | melwitt | ||
| 16:46:12 | mnaser | really, let me double check | |
| 16:46:24 | dansmith | tssurya: not sure what you mean.. pastebin an error? | |
| 16:46:25 | mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | |
| 16:46:28 | tssurya | mnaser, mriedem: yea someone ran into the same issue and we changed the order | |
| 16:46:31 | mriedem | "Using the newly installed nova code, run the DB sync. (nova-manage api_db sync; nova-manage db sync). These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail." | |
| 16:46:32 | tssurya | dansmith: ok | |
| 16:46:37 | artom | stephenfin, I don't know the full history, but I feel like it's opening a can of worms | |
| 16:47:00 | mnaser | mriedem: serves me right for looking at the queens docs thinking it hasnt change because it "looks" the same | |
| 16:47:06 | artom | stephenfin, also, I could imagine a scenario where an operator really pinky swears the destination host is fine, and wants to live migrate regardless | |
| 16:47:10 | mnaser | you're right, the order was swapped in rocky, my bad | |
| 16:47:23 | artom | stephenfin, so I'm not sure I'm comfortable with such a heavy handed approach | |
| 16:47:31 | mnaser | but anyways, back to that gigantic traceback | |
| 16:47:32 | mriedem | mnaser: i think grenade was doing it the right way before that docs change, | |
| 16:47:34 | mriedem | our docs were just old | |
| 16:47:37 | tssurya | dansmith: https://pastebin.com/7cQKv0fz | |
| 16:47:41 | artom | stephenfin, totally get where you're coming from though :) | |
| 16:47:44 | stephenfin | artom: Fair point. Wanna stick your thoughts in that review? | |
| 16:48:01 | dansmith | tssurya: oh because cell0's transport_url column can be NULL ? | |
| 16:48:03 | artom | stephenfin, yep, will do | |
| 16:48:15 | stephenfin | ta | |
| 16:48:25 | tssurya | yea | |
| 16:48:34 | dansmith | tssurya: add "and val" here: https://github.com/openstack/nova/blob/a53e46a75936b55c93face840764a67f2186cb11/nova/objects/cell_mapping.py#L162 | |
| 16:48:51 | dansmith | tssurya: you gonna cook up a patch or do you want me to? | |
| 16:49:09 | dansmith | tssurya: I wonder why/how we're not hitting that in the gate? | |
| 16:49:15 | dansmith | do we set it to something bogus? | |
| 16:49:29 | tssurya | dansmith: would be nice if you do it.. | |
| 16:49:33 | dansmith | tssurya: sure | |
| 16:49:46 | mriedem | mnaser: so this is being triggered when listing resource providers, what is doing that? | |
| 16:49:46 | tssurya | we were just doing the upgrade checks for rocky and saw this | |
| 16:50:03 | mnaser | mriedem: i'm assuming nova-scheduler? | |
| 16:50:26 | mnaser | https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/api/openstack/placement/objects/resource_provider.py#L1923-L1965 | |
| 16:50:27 | mnaser | this is interesting | |
| 16:50:56 | dansmith | tssurya: hmm, actually, it's not nullable on the object, so I'm not sure how it'd be doing the right thing if you have NULL in the db | |
| 16:51:16 | tssurya | dansmith: but we can have NULL on the conf file | |
| 16:51:38 | tssurya | technically there is no requirement to produce a transport_url for db sync right ? | |
| 16:52:15 | tssurya | s/to produce/to supply | |
| 16:52:16 | dansmith | tssurya: ohh, I see, I thought you were saying it was NULL in the database | |
| 16:52:24 | mnaser | [placement] incomplete_consumer_project_id and incomplete_consumer_user_id are a thing, i guess | |
| 16:52:45 | mriedem | mnaser: oh i guess adding/removing host aggregates in rocky would do it b/c we have to find the provider by name which we use GET /resource_providers?name=foo for that | |
| 16:52:45 | tssurya | no not in the database. but maybe we hit it here ? https://github.com/openstack/nova/blob/a53e46a75936b55c93face840764a67f2186cb11/nova/objects/cell_mapping.py#L150 | |
| 16:52:50 | mriedem | and when building the "provider tree" | |
| 16:53:24 | mriedem | when reporting inventory | |
| 16:53:27 | mriedem | from the compute | |
| 16:53:42 | mnaser | mriedem: the 500 is coming on /resource_providers/foo/allocations" | |
| 16:53:52 | dansmith | tssurya: so there is already a check for None-ness, but I guess if the value in the db isn't a template we'll fail to exit | |
| 16:54:13 | mnaser | and what it seems like almost any time its requesting allocations | |
| 16:54:22 | mriedem | mnaser: yeah that happens in the resource tracker on the compute | |
| 16:54:24 | tssurya | dansmith: ah yea was just wondering why that check didn't catch the Noneness | |
| 16:54:26 | mriedem | _remove_deleted_instances_allocations method | |
| 16:54:41 | mnaser | yep, i see compute ips sending in that request | |
| 16:54:44 | mriedem | so on startup of the compute, it's going to list allocations for the given compute ndoe provider | |
| 16:55:04 | mriedem | and try to create consumers table records for that provider and any allocations against it | |
| 16:55:32 | mriedem | i'm not sure how we race to hit _create_incomplete_consumers_for_provider though | |
| 16:55:41 | mriedem | because that should be idempotent | |
| 16:55:46 | mriedem | and jaypipes isn't around | |
| 16:55:55 | dansmith | tssurya: I'm shocked we haven't seen this in the regular tests | |
| 16:56:27 | tssurya | dansmith: yea we should have seen this somewhere | |
| 16:56:43 | tssurya | not sure if people didn't hit this when moving to rocky ? | |
| 16:57:34 | dansmith | well, I'm not sure why devstack doesn't hit it | |
| 16:57:40 | dansmith | I guess because we always have those defined in config | |
| 16:57:58 | tssurya | probably yea, but its easily reproducible | |
| 16:58:04 | dansmith | yep | |
| 16:58:05 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Handle volume API failure in _post_live_migration https://review.openstack.org/611093 | |
| 16:58:18 | tssurya | will you also backport this please ? we might need this in queens | |
| 16:58:19 | mriedem | our base test case uses the rpc fixture https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/test.py#L238 | |
| 16:58:25 | mriedem | so that's probably why we'd never hit it? | |
| 16:58:42 | mriedem | tssurya: the template stuff wasn't in queens | |
| 16:59:18 | mriedem | https://github.com/openstack/nova/blob/396156eb13521a0e7af4488a8cd4693aa65a0da2/nova/tests/fixtures.py#L728 | |
| 16:59:38 | mriedem | all of our tests at least configure this: transport_url = 'fake:/' | |
| 16:59:44 | tssurya | mriedem: oh yea sorry rocky then | |
| 16:59:53 | mriedem | tssurya: have you reported a bug? | |
| 17:00:03 | dansmith | mriedem: I got one already | |
| 17:00:06 | dansmith | coming | |
| 17:00:08 | tssurya | mriedem: no | |
| 17:00:09 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611094 | |
| 17:00:11 | tssurya | dansmith: ah thanks :) | |
| 17:00:13 | dansmith | mriedem: tssurya ^ | |
| 17:00:26 | mriedem | mnaser: well i'm not sure how it's happening, but clearly we could race if two requests are listing resource providers at the same time, | |
| 17:00:41 | mriedem | mnaser: so likely just bug report it and we can add a try/except for the duplicate entry error | |
| 17:00:59 | mnaser | mriedem: but i mean this is constantly happening and i think i cant provision any more vms on the cloud with a hostnotfound type of thing | |
| 17:00:59 | dansmith | mriedem: I meant devstack-based tests, but it's because we always have a transport_url set I think.. | |
| 17:01:04 | dansmith | and yeah, this'll be a rocky backport | |
| 17:01:51 | mriedem | mnaser: so i wonder if https://github.com/openstack/nova/blob/377921103121bc62a3f7fce60c63e30815406851/nova/api/openstack/placement/objects/resource_provider.py#L1940 is always True? | |
| 17:02:11 | mnaser | mriedem: thats what im trying to decipher | |
| 17:02:32 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix formatting non-templated cell URLs with no config https://review.openstack.org/611094 | |
| 17:04:12 | mnaser | mriedem: i think this is a cloud with some brokenness that is being exposed | |
| 17:04:38 | mnaser | right now, there are records in allocations for that specific uuid under resource_provider_id=4 | |
| 17:04:53 | mnaser | but in that traceback, it tries to add allocations for resource_provider_id=5 | |
| 17:05:08 | mnaser | sorry, no, i lied | |
| 17:05:12 | mnaser | it actaully tries to add for 4 | |
| 17:05:39 | mnaser | let me see if there is anything in cosnumers | |
| 17:06:08 | tssurya | dansmith: thanks a lot | |
| 17:07:25 | mnaser | i think this sql server is f'd | |
| 17:07:37 | mnaser | select * from consumers where uuid='fbee657b-6a60-4525-b7c8-b070643404ec'; returns nothing | |
| 17:07:45 | mriedem | there aren't consumer records until rocky | |
| 17:08:19 | mriedem | and the data migration function that is blowing up is trying to populate that table from existing allocations records | |
| 17:08:46 | dansmith | tssurya: np | |
| 17:08:51 | mnaser | mriedem: right, but the traceback seems to say: Duplicate entry 'fbee657b-6a60-4525-b7c8-b070643404ec' for key 'uniq_consumers0uuid' | |