Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-21
19:40:24 efried is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug :(
19:40:33 mriedem efried: MigrationTask.execute runs in conductor, not the api, but otherwise yes same idea
19:41:01 efried mriedem: so okay, I was doing all those things, and not seeing the error in the logs.
19:41:24 mriedem are you running the periodic on the source or dest compute service?
19:41:27 efried both
19:41:44 efried because self._run_periodics() is easier
19:41:53 efried than digging up just the dest.
19:41:58 efried but that shouldn't matter, should it?
19:42:03 mriedem it shouldn't no
19:42:11 efried let me push it and maybe something jumps out.
19:42:38 mriedem "is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug" - i was hitting that a week or two ago on my "mega boot from volume request" test and had to change some stuff, i thought about pushing that up but never did
19:42:45 mriedem iow to make OS_DEBUG=True work
19:43:43 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Don't populate resources for not-yet-migrated instance https://review.opendev.org/689842
19:43:45 openstack bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried)
19:43:45 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866
19:43:52 efried mriedem: ^
19:48:58 openstackgerrit Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842
19:49:22 efried that'll be copacetic once the test works at least ^
19:49:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224
19:49:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591
19:49:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253
19:49:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450
19:49:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592
19:49:52 openstackgerrit Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012
19:49:52 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451
19:49:53 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058
19:49:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013
19:49:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075
19:49:54 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070
19:49:55 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630
19:49:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316
19:49:56 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add finish_revert_snapshot_based_resize_at_source compute method https://review.opendev.org/637647
19:49:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453
19:49:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.opendev.org/638047
19:49:57 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.opendev.org/638046
19:49:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.opendev.org/638048
19:50:06 efried (It'll work fine now since the repro... doesn't.)
19:52:10 mriedem efried: ok i think i see the problem,
19:52:19 mriedem RT._claim_existing_migration is what sets Migration.dest_compute, not conductor,
19:52:36 mriedem so you need to mock that, run the original and then run periodics before returning from _claim_existing_migration
19:53:14 mriedem the live migration task in conductor sets Migration.dest_compute so that's where i got confused
19:53:54 mriedem i guess we do that for the live migration task because we used to never do claims in the compute for live migration
19:54:08 mriedem so we'd never call RT._claim_existing_migration for live migration before train
19:54:20 mriedem well, and we still don't if the instance doesn't have a numa topology
19:54:56 efried mriedem: I tried doing as you suggest and ran into the deadlock.
19:55:04 efried but
19:55:23 efried why does Migration.dest_compute make the difference?
19:55:42 mriedem because RT._update_available_resource calls MigrationList.get_in_progress_by_host_and_node
19:56:00 efried okay
19:56:05 mriedem without having Migration.dest_compute set the migration won't be returning for that dest host during the periodic
19:56:07 efried so I was barking up the right tree
19:56:22 efried to get around the deadlock, should I ... stub out the lock?
19:56:30 efried that seems pretty dangerous.
19:57:17 mriedem hmm
19:59:37 efried Perhaps I can stub out the lock just from within my stub
20:00:13 mriedem well, if we have to monkey with locks we're likely going about this the wrong way
20:00:30 mriedem so maybe you can't reproduce with a cold migration resize_claim and instead need to do a live migration in the test,
20:00:37 mriedem because in that case Migration.dest_compute is set in the conductor
20:00:55 efried okay.
20:01:00 efried still _prep_resize tho?
20:01:11 mriedem no, that's not called for live migration
20:01:16 mriedem you could probably stub pre_live_migration
20:01:21 mriedem that runs on the dest
20:01:52 mriedem unless the instance has a numa topology i don't think it'll ever have a migration context so it doesn't matter too much
20:04:12 efried woot, got repro
20:05:16 mriedem yass
20:05:21 mriedem now i have to go pick up some contacts
20:08:57 openstackgerrit Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842
20:08:57 openstack bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried)
20:08:57 openstackgerrit Eric Fried proposed openstack/nova master: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866
20:09:05 efried mriedem_afk, luyao: ^
20:13:28 openstackgerrit Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842
20:13:28 openstack bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried)
20:13:28 openstackgerrit Eric Fried proposed openstack/nova master: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866
20:13:32 efried now with pep8 fixed!
20:56:49 openstackgerrit Florian Haas proposed openstack/nova stable/queens: Explain nested guest support https://review.opendev.org/609790
20:58:02 mriedem efried: +2 on both
20:58:26 efried thanks mriedem. I think gibi is to blame for the formatting -- I copy/pasted test_bug_1845291.py
20:58:36 mriedem tarnations
20:59:29 efried We should get someone like wangfaxin to propose a patch to reformat it.
20:59:41 efried https://review.opendev.org/#/q/owner:wangfaxin%2540inspur.com+status:open
21:12:47 mriedem whatever helps the new contributor
21:13:04 mriedem efried: i replied in https://review.opendev.org/#/c/689049/ - i'm not sure how much we're on the same page, but i think i'm thinking more we need to do your "eventually" case
21:13:59 mriedem you said, "TL;DR the only thing that should change right now is: don't retry on consumer 409." but that's not what we do today anyway in move_allocations
21:14:00 efried oh
21:14:04 mriedem unless i don't understand how that code works
21:14:04 efried yeah
21:14:15 efried no, you're right, I forgot we were doing the text scraping thing
21:14:25 mriedem ok
21:14:25 efried in that case, we shouldn't "fix" this at all.
21:16:47 efried I'll need to swap it back in to be more thoroughly convinced of that ^ but at least last week when I was thinking this through, I convinced myself that making these narrow/local changes was not going to help anything (given that we're not retrying on consumer alloc 409)
21:17:24 mriedem i agree that we don't have the context for figuring out what to do within move_allocations,
21:17:37 mriedem which is why i think we need a separate method that wraps move_allocations and has the necessary logic
21:17:45 mriedem which only gets used in the revert case
21:18:51 mriedem i've got about 45 min to kill so maybe i can hack something together for what i'm thinking
21:19:19 lifeless mriedem: constraints was parallel to solver work FWIW
21:19:33 lifeless mriedem: even with a solver we want stability and precise controls :)
21:20:16 lifeless mriedem: I did have a solver branch, but yeah, job changes at the wrong time stalled that work then someone else popped up so ... its in a branch on github somewhere but the other volunteers thing will eventuate eventually I hope
21:35:31 mordred lifeless: you're a solver branch
22:02:05 mriedem lifeless: ack

Earlier   Later