Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-04
10:21:37 openstackgerrit Boxiang Zhu proposed openstack/nova master: Use the functional test test_parallel_evacuate_with_server_group https://review.openstack.org/649963
11:17:35 awalende following problem: I had a semi-failed migration. The instance has been migrated to another host and is working so far. However, not everything has been adjusted in the db and I get the following log repeatedly in the source host:
11:17:53 awalende Y_MB': 2048, u'DISK_GB': 21}}
11:17:53 awalende Instance a3fcb5a8-74e3-42f9-9659-428ffe710c19 has been moved to another host schwalm. There are allocations remaining against the source host that might need to be removed: {u'resources': {u'VCPU': 2, u'MEMOR
11:18:20 awalende Where can I find the responsible parts in the db to clean up this mess?
11:24:26 tssurya awalende: depending on the release you are running this would be the "allocations" table in the nova_api or placement database
11:24:43 awalende It's queens :)
11:24:57 tssurya you could use the https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete CLI to remove the allocation against the migration record on the source host
11:25:06 tssurya ah if queens its nova_api database
11:25:09 tssurya allocations table
11:26:11 tssurya awalende: you should be able to find records for that migration_uuid as the consumer against the source resource_provider
11:28:10 awalende select * from allocations where consumer_id='a3fcb5a8-74e3-42f9-9659-428ffe710c19';
11:28:34 awalende this gave me 3 rows, so I simply have to overwrite the resource_provider_id with the id from the new host?
11:29:11 tssurya awalende: the consumer_id is name of your instance ? or the migration_uuid ?
11:29:28 awalende instance id
11:29:46 tssurya ah so these are the allocations against the new host for your instance
11:30:02 awalende ah okay
11:30:10 tssurya could you verify by checking the resource_provider_id ? these three records should be the ones you should NOT delete
11:30:40 tssurya what you should do is query with consumer_id=migration_uuid
11:30:45 awalende is there a smart way to find out the resource_provider_id of each computehost?
11:38:43 awalende Ah I believe I got it
11:39:02 awalende So the 3 rows in my db still have the resource_provider_id from the "old" host
11:39:25 tssurya awalende: ah no allocations against new host ?
11:40:06 awalende For now, I'm not finding anyone. As I mentioned before, I had to finish the failed migration manually by myself
11:40:36 awalende something has been really messed up by this
11:40:50 tssurya ah ok that's bad then no allocations were formed against the target host and we don't know how far the migration went
11:40:52 tssurya yea
11:41:12 awalende for example I had to manually transfer the swap partition disk
11:42:05 openstackgerrit Lee Yarwood proposed openstack/nova master: Use migration_status during volume migrating and retyping https://review.openstack.org/637224
11:42:18 awalende Would probably the best idea to scrap this instance completely, but my customer will try to assassinate me
11:42:39 sean-k-mooney jaypipes: thanks jay for your comments on https://review.openstack.org/#/c/620115/29 can you confim my understanding of what you want changed is correct
11:44:44 awalende but anyway, thanks for the help tssurya. much appreciated
11:45:06 tssurya awalende: np :)
11:58:15 awalende ha, it worked
11:58:48 tssurya awalende: you just changed the allocations manually ? :)
11:59:12 awalende exactly, the warnings in the compute logs went away and the instance is still working :D
12:00:07 tssurya :) let's hope for the best then
12:00:33 awalende fingers crossed
12:03:14 jaypipes sean-k-mooney: I really don't know why a new "utils.lock" variable was introduced.
12:03:27 jaypipes sean-k-mooney: instead of just using utils.synchronized
12:04:10 sean-k-mooney ya neither do i which is why i was going to change it as you suggested
12:04:53 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Reject resize requests when the source host is down https://review.openstack.org/623489
12:06:23 jaypipes sean-k-mooney: answered on review
12:06:31 sean-k-mooney jaypipes: thanks
12:06:36 jaypipes sean-k-mooney: remove the utils.lock thing and I'm +2
12:07:34 sean-k-mooney perfect ill start work on that now so. if you have time to review the follow up patch its pretty short but if not that ok too.
12:07:47 jaypipes sean-k-mooney: yep, already on it.
12:08:03 jaypipes sean-k-mooney: I had to spend time this morning correcting your comma misuse. :P
12:08:27 jaypipes sean-k-mooney: and don't get me started on your abuse of the apostrophe.
12:09:08 sean-k-mooney :) i hear punction is a think people find useful.
12:09:14 sean-k-mooney isnt my use of it novel
12:09:36 jaypipes "punction", eh? :)
12:12:14 sean-k-mooney punctuation, i was 72% there.
12:12:26 artom ... that's how they take cerebro-spinal fluid samples, no?
12:32:32 sean-k-mooney i may have lost a tab in firefox and realised its still open in one of the 14 firefox windows i currently have open.
12:32:44 sean-k-mooney proably
12:34:13 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove DiffieHellmann tests https://review.openstack.org/649974
13:02:45 ganso hi alex_xu, could you please take a look at https://review.openstack.org/#/c/647566 ? The other part of that bugfix has already merged, we are waiting on this fix to land so we can backport. Thanks in advance!
13:03:14 alex_xu ganso: let me check
13:03:51 adrianc sean-k-mooney: Hey, are you planning to work on : https://review.openstack.org/#/c/620115/29 ?
13:04:51 openstackgerrit Alex Xu proposed openstack/nova master: [DNM] debug the LM test https://review.openstack.org/649807
13:04:59 sean-k-mooney im in a meeting for the next hour but after
13:05:09 sean-k-mooney if you want to do it fell free
13:06:12 efried ileixe: It appears to be abandoned. It looks like dansmith was going to make it work some other way, you may want to follow up with him.
13:06:14 adrianc sean-k-mooney, jaypipes : also replied on the utils.lock()
13:06:26 efried ileixe: I also seem to recall there was going to be a forum session following on from one we had in Berlin on this topic...
13:07:08 efried ileixe: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23675/change-ownership-of-resources-followup
13:08:01 adrianc sean-k-mooney: have some meetings and prior engagements will probably get to it on Sunday.
13:08:54 sean-k-mooney adrianc: ill read your comments if you have no objections to useing syncronised instead of util.lock then i can make the changes today
13:09:42 adrianc sean-k-mooney: well i intentionally defined the utils.lock() so ill be able to use as a context manager and save the wrapping
13:09:46 openstackgerrit Takashi NATSUME proposed openstack/nova master: Change a log level for overwriting allocation https://review.openstack.org/649788
13:10:49 adrianc sean-k-mooney: but if decorators are preferred then I have no objection.
13:13:21 mriedem anyone hit this grenade failure yet? http://logs.openstack.org/12/648912/4/check/grenade-py3/19a6549/logs/grenade.sh.txt.gz#_2019-04-03_03_50_29_324
13:14:21 mriedem efried: cdent: ^?
13:15:31 cdent mriedem: I remember hitting that some months ago and thinking I fixed it
13:15:59 cdent It happens when the openstack client is reinstalled, at a different python version, so the plugins are no longer visible
13:16:45 mriedem hmm, only happening on stable/stein and master http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cval%3D'openstack%3A%20'%5C%5C''resource%20provider%20inventory%20show%20openstack%3A%20VCPU%20-f%20value%20-c%20total'%5C%5C''%20is%20not%20an%20openstack%20command.%20See%20'%5C%5C''openstack%20--help'%5C%5C''.%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
13:16:50 mriedem grenade was maybe branched?
13:17:04 cdent so, somewhere in that log will be something causing whichever openstack command is on the path to become a different one
13:18:30 cdent this was the original bug https://bugs.launchpad.net/grenade/+bug/1805156
13:18:31 openstack Launchpad bug 1805156 in grenade "In a python3 grenade run, openstack client plugins are lost after swift install" [Undecided,Fix released] - Assigned to Chris Dent (cdent)
13:18:32 openstackgerrit Hamdy Khader proposed openstack/nova master: Do not perform port update in case of baremetal instance. https://review.openstack.org/649345
13:18:56 mriedem cdent: looks like osc-placement is installed into py2.7 and python-openstackclient is installed into py3.6
13:19:42 efried mriedem: Yes, I was looking at that last night.
13:20:20 efried okay, looks like you guys are further along than I got
13:20:46 mriedem WARNING: Did not find python 3 classifier for remote package osc-placement
13:20:52 mriedem that's why osc-placement isn't installed on py3
13:20:59 cdent mriedem: swift is reinstalling python-openstackclient here: http://logs.openstack.org/12/648912/4/check/grenade-py3/19a6549/logs/grenade.sh.txt.gz#_2019-04-03_03_26_57_820
13:21:18 mriedem https://github.com/openstack/osc-placement/blob/master/setup.cfg#L18
13:21:26 mriedem assuming this is a rax-dfw node...
13:22:01 cdent are you sure that's more than a warning?
13:23:28 mriedem that makes me think it's bug 1820892
13:23:29 openstack bug 1820892 in devstack "Intermittent "Error starting thread.: ModuleNotFoundError: No module named 'etcd3gw'" in grenade-py3 jobs since March 14" [High,In progress] https://launchpad.net/bugs/1820892 - Assigned to Dr. Jens Harbott (j-harbott)
13:23:53 mriedem devstack failing to parse the py3 classifier on pypi for rax-dfw nodes and thinks it's a py2.7 package and installs there
13:23:57 mriedem https://review.openstack.org/#/c/649096/
13:24:42 cdent I agree with that fix
13:25:06 cdent I'll fix osc-placement in the meantime
13:25:33 mriedem osc-placement looks OK, except it doesn't have a 3.6 classifier, but devstack isn't looking for that
13:25:41 frickler cdent: mriedem: hmm, maybe there's more to it with the "swift reinstalling osc" thing. might be swift shouldn't be installed at all for a py3 job as long as they don't do py3
13:26:26 cdent frickler: I've wondered that too

Earlier   Later