| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-21 | |||
| 17:48:15 | efried | So I tried mocking _claim_existing_migration to do its thing first, followed by running periodics. | |
| 17:48:15 | efried | AFAICT I need to get inside _move_claim itself, between _claim_existing_migration (at the top) and the init of mig_context (at the bottom). | |
| 17:48:15 | efried | Doing the mock at _prep_resize wasn't working because both the migration and the migration context are happening underneath that. | |
| 17:48:15 | efried | mriedem: I'm close-ish. | |
| 18:11:51 | mriedem | efried: you don't need to replace _prep_resize entirely, just wrap it, | |
| 18:12:08 | mriedem | i.e. stub it out and inside the stub run the periodics and then call the original _prep_resize | |
| 18:12:28 | mriedem | like this https://review.opendev.org/#/c/689013/1/nova/tests/functional/regressions/test_bug_1848343.py | |
| 18:19:34 | mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 | |
| 18:19:34 | mriedem | looks like the ceph job is not happy again | |
| 18:19:53 | mriedem | and the graphs are outdated on master since we don't run tempest-full-py3 in master (we run tempest-integrated-compute) | |
| 18:20:36 | melwitt | dangit, I'll look and also get the grafana page issue fixed | |
| 18:20:56 | mriedem | i looked at a ceph failure last week and it looked like some setup issue in devstack | |
| 18:21:48 | melwitt | ok | |
| 18:23:53 | openstackgerrit | sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861 | |
| 18:23:53 | openstackgerrit | sean mooney proposed openstack/nova master: block rebuild when numa topology changed https://review.opendev.org/687957 | |
| 18:26:59 | openstackgerrit | Merged openstack/nova stable/train: Update compute rpc version alias for train https://review.opendev.org/689164 | |
| 18:27:05 | openstackgerrit | Merged openstack/nova stable/train: Error out interrupted builds https://review.opendev.org/687216 | |
| 18:28:23 | melwitt | mriedem: can you link me a failure? I don't see any recent failure when I open a few sample patches | |
| 18:33:48 | dansmith | melwitt: could probably convince kibana to show you just that job with failed status | |
| 18:34:17 | melwitt | oh yeah, kibana. my old friend | |
| 18:35:53 | dansmith | something like build_status:FAILURE build_name:"ceph" | |
| 18:35:57 | openstackgerrit | Merged openstack/nova stable/train: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686226 | |
| 18:36:17 | melwitt | yeah, thanks | |
| 18:36:39 | melwitt | can do that | |
| 18:50:21 | openstackgerrit | Merged openstack/nova stable/train: Fix exception translation when creating volume https://review.opendev.org/688072 | |
| 19:12:57 | efried | mriedem: yes, that's what I did, but it didn't trigger the problem. | |
| 19:14:10 | jmlowe | Is anybody aware off the top of their heads of a mechanism to quota vgpu's ? | |
| 19:17:29 | efried | melwitt: ^ did we get anywhere with the placement-based quota thing? | |
| 19:17:56 | melwitt | efried: not yet. that's johnthetubaguy's unified limits spec | |
| 19:18:24 | melwitt | I had a chat with him about it and he'd like to get started on it this cycle and will re-propose the spec | |
| 19:18:34 | efried | https://review.opendev.org/#/c/602201/ | |
| 19:18:38 | melwitt | (and I will help with the work) | |
| 19:18:39 | efried | that un ^ ? | |
| 19:18:49 | melwitt | yes that's the one | |
| 19:18:50 | efried | oh, jmlowe left, nm | |
| 19:21:42 | efried | mriedem: is the theory that MigrationTask._execute (n-api) creates the Migration record, which should thenceforth be able to produce an instance via the .instance @property; and then casts to prep_resize on the compute? So by hijacking _prep_resize and doing update_available_resource first, that should hit the window. | |
| 19:23:18 | efried | jmlowe: you're back! | |
| 19:23:48 | jmlowe | sometimes you eat the wifi sometimes the wifi eats you | |
| 19:24:02 | efried | see https://review.opendev.org/#/c/602201/ which melwitt says she and johnthetubaguy will be working on this cycle. | |
| 19:24:16 | efried | that would allow you to quota placement-based resources like vgpu | |
| 19:24:44 | efried | btw, I'm assuming you were talking about post-vgpu-in-placement. Otherwise I think it's just a PCI device and can be quotaed like any other PCI device. | |
| 19:24:47 | jmlowe | exactly what I was hoping for | |
| 19:25:15 | jmlowe | slapping an arbitrary quota on something tracked by placement | |
| 19:25:55 | jmlowe | now I just need to hope placement can understand different flavors of vgpu's | |
| 19:26:52 | efried | jmlowe: what do you mean by "flavors"? | |
| 19:27:03 | jmlowe | a 8GB slice of a 16GB framebuffer != a 4GB slice of a 16GB frame buffer | |
| 19:27:32 | efried | Those are distinctions for which traits would be appropriate | |
| 19:27:36 | jmlowe | the nvidia vgpu stuff slices up the frame buffer in powers of 2 | |
| 19:27:55 | efried | which of them are done by nova and which you would have to do manually, I couldn't say off the top. | |
| 19:27:57 | jmlowe | 1/2, 1/4, 1/8, etc | |
| 19:28:19 | efried | If you want to hang out during euro business hours, bauzas might be able to answer those off the top. | |
| 19:30:04 | efried | ...I can't see where we're assigning any traits to VGPU providers at the moment. | |
| 19:30:19 | jmlowe | I'm currently on rocky, so we are looking at the U release for the real stuff, I've got 24 GPU's now so I can manage for the time being by hand (by hand I mean sharp whacks on the back of the user's hands with a ruler) | |
| 19:31:19 | jmlowe | a year from now there's a chance I'll have a few gpu's hundred to manage | |
| 19:40:24 | efried | is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug :( | |
| 19:40:33 | mriedem | efried: MigrationTask.execute runs in conductor, not the api, but otherwise yes same idea | |
| 19:41:01 | efried | mriedem: so okay, I was doing all those things, and not seeing the error in the logs. | |
| 19:41:24 | mriedem | are you running the periodic on the source or dest compute service? | |
| 19:41:27 | efried | both | |
| 19:41:44 | efried | because self._run_periodics() is easier | |
| 19:41:53 | efried | than digging up just the dest. | |
| 19:41:58 | efried | but that shouldn't matter, should it? | |
| 19:42:03 | mriedem | it shouldn't no | |
| 19:42:11 | efried | let me push it and maybe something jumps out. | |
| 19:42:38 | mriedem | "is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug" - i was hitting that a week or two ago on my "mega boot from volume request" test and had to change some stuff, i thought about pushing that up but never did | |
| 19:42:45 | mriedem | iow to make OS_DEBUG=True work | |
| 19:43:43 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Don't populate resources for not-yet-migrated instance https://review.opendev.org/689842 | |
| 19:43:45 | openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 19:43:45 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866 | |
| 19:43:52 | efried | mriedem: ^ | |
| 19:48:58 | openstackgerrit | Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842 | |
| 19:49:22 | efried | that'll be copacetic once the test works at least ^ | |
| 19:49:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | |
| 19:49:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | |
| 19:49:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | |
| 19:49:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | |
| 19:49:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | |
| 19:49:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | |
| 19:49:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | |
| 19:49:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | |
| 19:49:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | |
| 19:49:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075 | |
| 19:49:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070 | |
| 19:49:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630 | |
| 19:49:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316 | |
| 19:49:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add finish_revert_snapshot_based_resize_at_source compute method https://review.opendev.org/637647 | |
| 19:49:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453 | |
| 19:49:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.opendev.org/638047 | |
| 19:49:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.opendev.org/638046 | |
| 19:49:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.opendev.org/638048 | |
| 19:50:06 | efried | (It'll work fine now since the repro... doesn't.) | |
| 19:52:10 | mriedem | efried: ok i think i see the problem, | |
| 19:52:19 | mriedem | RT._claim_existing_migration is what sets Migration.dest_compute, not conductor, | |
| 19:52:36 | mriedem | so you need to mock that, run the original and then run periodics before returning from _claim_existing_migration | |
| 19:53:14 | mriedem | the live migration task in conductor sets Migration.dest_compute so that's where i got confused | |
| 19:53:54 | mriedem | i guess we do that for the live migration task because we used to never do claims in the compute for live migration | |
| 19:54:08 | mriedem | so we'd never call RT._claim_existing_migration for live migration before train | |
| 19:54:20 | mriedem | well, and we still don't if the instance doesn't have a numa topology | |
| 19:54:56 | efried | mriedem: I tried doing as you suggest and ran into the deadlock. | |
| 19:55:04 | efried | but | |