| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-21 | |||
| 19:18:50 | efried | oh, jmlowe left, nm | |
| 19:21:42 | efried | mriedem: is the theory that MigrationTask._execute (n-api) creates the Migration record, which should thenceforth be able to produce an instance via the .instance @property; and then casts to prep_resize on the compute? So by hijacking _prep_resize and doing update_available_resource first, that should hit the window. | |
| 19:23:18 | efried | jmlowe: you're back! | |
| 19:23:48 | jmlowe | sometimes you eat the wifi sometimes the wifi eats you | |
| 19:24:02 | efried | see https://review.opendev.org/#/c/602201/ which melwitt says she and johnthetubaguy will be working on this cycle. | |
| 19:24:16 | efried | that would allow you to quota placement-based resources like vgpu | |
| 19:24:44 | efried | btw, I'm assuming you were talking about post-vgpu-in-placement. Otherwise I think it's just a PCI device and can be quotaed like any other PCI device. | |
| 19:24:47 | jmlowe | exactly what I was hoping for | |
| 19:25:15 | jmlowe | slapping an arbitrary quota on something tracked by placement | |
| 19:25:55 | jmlowe | now I just need to hope placement can understand different flavors of vgpu's | |
| 19:26:52 | efried | jmlowe: what do you mean by "flavors"? | |
| 19:27:03 | jmlowe | a 8GB slice of a 16GB framebuffer != a 4GB slice of a 16GB frame buffer | |
| 19:27:32 | efried | Those are distinctions for which traits would be appropriate | |
| 19:27:36 | jmlowe | the nvidia vgpu stuff slices up the frame buffer in powers of 2 | |
| 19:27:55 | efried | which of them are done by nova and which you would have to do manually, I couldn't say off the top. | |
| 19:27:57 | jmlowe | 1/2, 1/4, 1/8, etc | |
| 19:28:19 | efried | If you want to hang out during euro business hours, bauzas might be able to answer those off the top. | |
| 19:30:04 | efried | ...I can't see where we're assigning any traits to VGPU providers at the moment. | |
| 19:30:19 | jmlowe | I'm currently on rocky, so we are looking at the U release for the real stuff, I've got 24 GPU's now so I can manage for the time being by hand (by hand I mean sharp whacks on the back of the user's hands with a ruler) | |
| 19:31:19 | jmlowe | a year from now there's a chance I'll have a few gpu's hundred to manage | |
| 19:40:24 | efried | is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug :( | |
| 19:40:33 | mriedem | efried: MigrationTask.execute runs in conductor, not the api, but otherwise yes same idea | |
| 19:41:01 | efried | mriedem: so okay, I was doing all those things, and not seeing the error in the logs. | |
| 19:41:24 | mriedem | are you running the periodic on the source or dest compute service? | |
| 19:41:27 | efried | both | |
| 19:41:44 | efried | because self._run_periodics() is easier | |
| 19:41:53 | efried | than digging up just the dest. | |
| 19:41:58 | efried | but that shouldn't matter, should it? | |
| 19:42:03 | mriedem | it shouldn't no | |
| 19:42:11 | efried | let me push it and maybe something jumps out. | |
| 19:42:38 | mriedem | "is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug" - i was hitting that a week or two ago on my "mega boot from volume request" test and had to change some stuff, i thought about pushing that up but never did | |
| 19:42:45 | mriedem | iow to make OS_DEBUG=True work | |
| 19:43:43 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Don't populate resources for not-yet-migrated instance https://review.opendev.org/689842 | |
| 19:43:45 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866 | |
| 19:43:45 | openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 19:43:52 | efried | mriedem: ^ | |
| 19:48:58 | openstackgerrit | Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842 | |
| 19:49:22 | efried | that'll be copacetic once the test works at least ^ | |
| 19:49:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | |
| 19:49:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | |
| 19:49:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | |
| 19:49:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | |
| 19:49:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | |
| 19:49:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | |
| 19:49:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | |
| 19:49:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | |
| 19:49:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | |
| 19:49:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070 | |
| 19:49:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075 | |
| 19:49:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316 | |
| 19:49:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630 | |
| 19:49:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453 | |
| 19:49:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add finish_revert_snapshot_based_resize_at_source compute method https://review.opendev.org/637647 | |
| 19:49:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.opendev.org/638046 | |
| 19:49:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.opendev.org/638047 | |
| 19:49:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.opendev.org/638048 | |
| 19:50:06 | efried | (It'll work fine now since the repro... doesn't.) | |
| 19:52:10 | mriedem | efried: ok i think i see the problem, | |
| 19:52:19 | mriedem | RT._claim_existing_migration is what sets Migration.dest_compute, not conductor, | |
| 19:52:36 | mriedem | so you need to mock that, run the original and then run periodics before returning from _claim_existing_migration | |
| 19:53:14 | mriedem | the live migration task in conductor sets Migration.dest_compute so that's where i got confused | |
| 19:53:54 | mriedem | i guess we do that for the live migration task because we used to never do claims in the compute for live migration | |
| 19:54:08 | mriedem | so we'd never call RT._claim_existing_migration for live migration before train | |
| 19:54:20 | mriedem | well, and we still don't if the instance doesn't have a numa topology | |
| 19:54:56 | efried | mriedem: I tried doing as you suggest and ran into the deadlock. | |
| 19:55:04 | efried | but | |
| 19:55:23 | efried | why does Migration.dest_compute make the difference? | |
| 19:55:42 | mriedem | because RT._update_available_resource calls MigrationList.get_in_progress_by_host_and_node | |
| 19:56:00 | efried | okay | |
| 19:56:05 | mriedem | without having Migration.dest_compute set the migration won't be returning for that dest host during the periodic | |
| 19:56:07 | efried | so I was barking up the right tree | |
| 19:56:22 | efried | to get around the deadlock, should I ... stub out the lock? | |
| 19:56:30 | efried | that seems pretty dangerous. | |
| 19:57:17 | mriedem | hmm | |
| 19:59:37 | efried | Perhaps I can stub out the lock just from within my stub | |
| 20:00:13 | mriedem | well, if we have to monkey with locks we're likely going about this the wrong way | |
| 20:00:30 | mriedem | so maybe you can't reproduce with a cold migration resize_claim and instead need to do a live migration in the test, | |
| 20:00:37 | mriedem | because in that case Migration.dest_compute is set in the conductor | |
| 20:00:55 | efried | okay. | |
| 20:01:00 | efried | still _prep_resize tho? | |
| 20:01:11 | mriedem | no, that's not called for live migration | |
| 20:01:16 | mriedem | you could probably stub pre_live_migration | |
| 20:01:21 | mriedem | that runs on the dest | |
| 20:01:52 | mriedem | unless the instance has a numa topology i don't think it'll ever have a migration context so it doesn't matter too much | |
| 20:04:12 | efried | woot, got repro | |
| 20:05:16 | mriedem | yass | |
| 20:05:21 | mriedem | now i have to go pick up some contacts | |
| 20:08:57 | openstackgerrit | Eric Fried proposed openstack/nova master: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866 | |
| 20:08:57 | openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 20:08:57 | openstackgerrit | Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842 | |
| 20:09:05 | efried | mriedem_afk, luyao: ^ | |
| 20:13:28 | openstackgerrit | Eric Fried proposed openstack/nova master: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866 | |
| 20:13:28 | openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 20:13:28 | openstackgerrit | Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842 | |
| 20:13:32 | efried | now with pep8 fixed! | |
| 20:56:49 | openstackgerrit | Florian Haas proposed openstack/nova stable/queens: Explain nested guest support https://review.opendev.org/609790 | |
| 20:58:02 | mriedem | efried: +2 on both | |
| 20:58:26 | efried | thanks mriedem. I think gibi is to blame for the formatting -- I copy/pasted test_bug_1845291.py | |
| 20:58:36 | mriedem | tarnations | |
| 20:59:29 | efried | We should get someone like wangfaxin to propose a patch to reformat it. | |