Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-21
17:13:06 dansmith to me, confidence in soakage comes from being in the firehose, not a few rechecks on a single patch
17:13:10 dansmith yeah, that ^
17:16:06 efried no, my point is, if we merge both of those things in master now, they're in the firehose because the second patch puts them in the way of I think three functional tests. Then we can soak them in master until you're comfortable before doing anything further.
17:26:22 openstackgerrit Merged openstack/nova stable/rocky: Stop sending bad values from libosinfo to libvirt https://review.opendev.org/688068
17:26:27 openstackgerrit Merged openstack/nova stable/stein: Fix unit of hw_rng:rate_period https://review.opendev.org/689153
17:42:35 artom sean-k-mooney, btw, do we care about https://review.opendev.org/#/c/465783/ not being in OSP10 Neutron?
17:42:56 artom (Sorry to make you context-switch - it's a dependency of https://code.engineering.redhat.com/gerrit/#/c/183568/1)
17:43:08 artom Doh, was meant for the internal channel
17:48:15 efried mriedem: I'm close-ish.
17:48:15 efried Doing the mock at _prep_resize wasn't working because both the migration and the migration context are happening underneath that.
17:48:15 efried AFAICT I need to get inside _move_claim itself, between _claim_existing_migration (at the top) and the init of mig_context (at the bottom).
17:48:15 efried So I tried mocking _claim_existing_migration to do its thing first, followed by running periodics.
17:48:15 efried But that deadlocks because *in test* they're both running in the same worker, so they're under the same COMPUTE_RESOURCE_SEMAPHORE.
17:48:15 efried Where would you go from here?
18:11:51 mriedem efried: you don't need to replace _prep_resize entirely, just wrap it,
18:12:08 mriedem i.e. stub it out and inside the stub run the periodics and then call the original _prep_resize
18:12:28 mriedem like this https://review.opendev.org/#/c/689013/1/nova/tests/functional/regressions/test_bug_1848343.py
18:19:34 mriedem looks like the ceph job is not happy again
18:19:34 mriedem http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1
18:19:53 mriedem and the graphs are outdated on master since we don't run tempest-full-py3 in master (we run tempest-integrated-compute)
18:20:36 melwitt dangit, I'll look and also get the grafana page issue fixed
18:20:56 mriedem i looked at a ceph failure last week and it looked like some setup issue in devstack
18:21:48 melwitt ok
18:23:53 openstackgerrit sean mooney proposed openstack/nova master: block rebuild when numa topology changed https://review.opendev.org/687957
18:23:53 openstackgerrit sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861
18:26:59 openstackgerrit Merged openstack/nova stable/train: Update compute rpc version alias for train https://review.opendev.org/689164
18:27:05 openstackgerrit Merged openstack/nova stable/train: Error out interrupted builds https://review.opendev.org/687216
18:28:23 melwitt mriedem: can you link me a failure? I don't see any recent failure when I open a few sample patches
18:33:48 dansmith melwitt: could probably convince kibana to show you just that job with failed status
18:34:17 melwitt oh yeah, kibana. my old friend
18:35:53 dansmith something like build_status:FAILURE build_name:"ceph"
18:35:57 openstackgerrit Merged openstack/nova stable/train: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686226
18:36:17 melwitt yeah, thanks
18:36:39 melwitt can do that
18:50:21 openstackgerrit Merged openstack/nova stable/train: Fix exception translation when creating volume https://review.opendev.org/688072
19:12:57 efried mriedem: yes, that's what I did, but it didn't trigger the problem.
19:14:10 jmlowe Is anybody aware off the top of their heads of a mechanism to quota vgpu's ?
19:17:29 efried melwitt: ^ did we get anywhere with the placement-based quota thing?
19:17:56 melwitt efried: not yet. that's johnthetubaguy's unified limits spec
19:18:24 melwitt I had a chat with him about it and he'd like to get started on it this cycle and will re-propose the spec
19:18:34 efried https://review.opendev.org/#/c/602201/
19:18:38 melwitt (and I will help with the work)
19:18:39 efried that un ^ ?
19:18:49 melwitt yes that's the one
19:18:50 efried oh, jmlowe left, nm
19:21:42 efried mriedem: is the theory that MigrationTask._execute (n-api) creates the Migration record, which should thenceforth be able to produce an instance via the .instance @property; and then casts to prep_resize on the compute? So by hijacking _prep_resize and doing update_available_resource first, that should hit the window.
19:23:18 efried jmlowe: you're back!
19:23:48 jmlowe sometimes you eat the wifi sometimes the wifi eats you
19:24:02 efried see https://review.opendev.org/#/c/602201/ which melwitt says she and johnthetubaguy will be working on this cycle.
19:24:16 efried that would allow you to quota placement-based resources like vgpu
19:24:44 efried btw, I'm assuming you were talking about post-vgpu-in-placement. Otherwise I think it's just a PCI device and can be quotaed like any other PCI device.
19:24:47 jmlowe exactly what I was hoping for
19:25:15 jmlowe slapping an arbitrary quota on something tracked by placement
19:25:55 jmlowe now I just need to hope placement can understand different flavors of vgpu's
19:26:52 efried jmlowe: what do you mean by "flavors"?
19:27:03 jmlowe a 8GB slice of a 16GB framebuffer != a 4GB slice of a 16GB frame buffer
19:27:32 efried Those are distinctions for which traits would be appropriate
19:27:36 jmlowe the nvidia vgpu stuff slices up the frame buffer in powers of 2
19:27:55 efried which of them are done by nova and which you would have to do manually, I couldn't say off the top.
19:27:57 jmlowe 1/2, 1/4, 1/8, etc
19:28:19 efried If you want to hang out during euro business hours, bauzas might be able to answer those off the top.
19:30:04 efried ...I can't see where we're assigning any traits to VGPU providers at the moment.
19:30:19 jmlowe I'm currently on rocky, so we are looking at the U release for the real stuff, I've got 24 GPU's now so I can manage for the time being by hand (by hand I mean sharp whacks on the back of the user's hands with a ruler)
19:31:19 jmlowe a year from now there's a chance I'll have a few gpu's hundred to manage
19:40:24 efried is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug :(
19:40:33 mriedem efried: MigrationTask.execute runs in conductor, not the api, but otherwise yes same idea
19:41:01 efried mriedem: so okay, I was doing all those things, and not seeing the error in the logs.
19:41:24 mriedem are you running the periodic on the source or dest compute service?
19:41:27 efried both
19:41:44 efried because self._run_periodics() is easier
19:41:53 efried than digging up just the dest.
19:41:58 efried but that shouldn't matter, should it?
19:42:03 mriedem it shouldn't no
19:42:11 efried let me push it and maybe something jumps out.
19:42:38 mriedem "is there any workaround for this f'in subunit parser bs, hitting it locally and it's making things tough to debug" - i was hitting that a week or two ago on my "mega boot from volume request" test and had to change some stuff, i thought about pushing that up but never did
19:42:45 mriedem iow to make OS_DEBUG=True work
19:43:43 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Don't populate resources for not-yet-migrated instance https://review.opendev.org/689842
19:43:45 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/689866
19:43:45 openstack bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried)
19:43:52 efried mriedem: ^
19:48:58 openstackgerrit Eric Fried proposed openstack/nova master: Don't populate resources for not-yet-migrated inst https://review.opendev.org/689842
19:49:22 efried that'll be copacetic once the test works at least ^
19:49:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224
19:49:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253
19:49:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591
19:49:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592
19:49:51 openstackgerrit Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450
19:49:52 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451
19:49:52 openstackgerrit Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012
19:49:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013
19:49:53 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058
19:49:54 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070
19:49:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075
19:49:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316
19:49:55 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630
19:49:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453
19:49:56 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add finish_revert_snapshot_based_resize_at_source compute method https://review.opendev.org/637647
19:49:57 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.opendev.org/638046
19:49:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.opendev.org/638047
19:49:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.opendev.org/638048

Earlier   Later