Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-13
15:10:14 mriedem lyarwood: ok +2 on that one; want to get https://review.openstack.org/#/c/541036/ and then i'll cut the release?
15:11:47 lyarwood mriedem: yup will do, thanks
15:23:03 mriedem lyarwood: this is re-introducing a regression from queens https://review.openstack.org/#/c/543569/
15:23:06 mriedem i'm pretty sure anyway
15:24:13 lyarwood mriedem: How so? We also block swap_volume for encrypted volumes when using native LUKS in >=Queens
15:24:26 mriedem lyarwood: https://github.com/openstack/nova/commit/cafe3d066ef7021c18961d4b239a10f61db23f2d#diff-f4019782d93a196a0d026479e6aa61b1
15:24:41 mriedem you're not backporting that regression fix, introduced by the native luks support patch
15:24:46 mriedem that broke several cinder backends
15:25:16 openstackgerrit Boris Bobrov proposed openstack/python-novaclient stable/queens: Fix listing of instances above API max_limit https://review.openstack.org/543968
15:26:13 lyarwood mriedem: crap, yeah my bad
15:31:52 mriedem stephenfin: you would probably enjoy the honors here https://review.openstack.org/#/c/513160/
15:31:58 mriedem nova-next job + tls proxy enabled
15:32:14 stephenfin mriedem: Ooh, snazzy
15:32:57 openstackgerrit Dan Smith proposed openstack/nova master: Lazy-load instance attributes with read_deleted=yes https://review.openstack.org/543970
15:33:11 mriedem i think once we drop the legacy nova-next job that's defined in openstack-zuul-jobs, we should look at making nova-next (defined in tree) voting and gating
15:33:25 mriedem and we should probably also make it run with py3
15:33:29 openstackgerrit Claudiu Belu proposed openstack/nova master: compute: Makes the resize_instance RPC call synchronous https://review.openstack.org/543971
15:33:33 cdent \o/
15:33:51 mriedem claudiub: re your allocation + resize delete thing, did you report a bug for that?
15:33:55 mriedem if not, can you?
15:34:18 claudiub I did. I just sent a patch for it as well
15:34:20 claudiub ^
15:34:40 claudiub basically, the Except branch in ComputeManager's prep_resize is never executed.
15:34:45 mriedem claudiub: i don't think that's the kind of fix we want
15:35:00 claudiub in which the allocations gets cleaned up
15:35:03 mriedem we likely need to handle the allocation cleanup in resize_instance
15:35:31 stephenfin mriedem: Both the commit message and comment in 'nova-next/run.yaml' say the 'tls-proxy' service is enabled when it isn't. If you want to drop that, I can +W
15:35:43 stephenfin Or vice versa. It's confusing otherwise :)
15:36:00 claudiub that could work too
15:36:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160
15:37:21 mriedem stephenfin: done
15:37:21 claudiub we might have to do the rescheduling there too, then.
15:37:40 stephenfin mriedem: likewise
15:37:43 mriedem claudiub: if resize_instance fails, we're dead
15:38:36 claudiub what do you mean?
15:38:53 claudiub you mean that the resize operation should fail, and be put in error state?
15:38:54 mriedem i don't think we need to try and make every part of a resize reschedulable
15:38:56 mriedem yes
15:40:08 mriedem changing the rpc cast to call might fix this issue, but likely introduces an rpc messaging timeout problem
15:40:15 mriedem if resize_instance takes >1 minute
15:40:17 claudiub well, not necessarely always. for example, if an InstanceFaultRollback is raised by the driver (at least in our scenario - if the new flavor's disk is smaller than the original), then the resize didn't happen, and the instance will be put back into Running state
15:40:37 claudiub hm, I see.
15:42:50 mriedem i see that the libvirt driver's migrate_disk_and_power_off method, called from resize_instance, is doing some checks like if you're trying to resize the disk down or if you're using lvm, it fails,
15:42:52 jaypipes dims: https://twitter.com/jaypipes/status/963437624852668419
15:42:57 mriedem in that case we'd fail and not reschedule
15:43:06 mriedem but in those cases, we don't want to reschedule
15:43:26 claudiub fair enough
15:43:32 dims jaypipes, y i was reading all about it yesterday ... :(
15:43:56 claudiub well then, I'll send another PS.
15:48:13 mriedem claudiub: you can maybe build a functional test similar to test_resize_to_same_host_prep_resize_fails
15:48:25 mriedem where resize_instance fails and we make sure to cleanup allocations created by the scheduler for the destination host
15:49:50 claudiub will do
15:59:38 mriedem dansmith: gibi: claudiub's bug got me thinking about how we might leak migration allocations during a failed cold migrate https://bugs.launchpad.net/nova/+bug/1749215/comments/2
15:59:40 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
15:59:55 mriedem normally deleting an instance removes it's allocations from placement and we're happy, that's the last resort kind of cleanup thing,
16:00:10 mriedem but we don't cleanup any allocations tracked against migration records for the instance when we delete the instance,
16:00:19 mriedem granted if we rollback at the point of failure correctly, we shouldn't need to
16:03:00 dansmith mriedem: well, the bug would be not in the instance delete path, but in the path where we left a migration allocation in place from the failed resize yeah?
16:03:18 mriedem yeah that's what i mean by "if we rollback at the point of failure correctly, we shouldn't need to"
16:03:48 mriedem looking at all of the resize flows, and where errors_out_migration_ctxt is used, it could be a lot of places
16:04:06 mriedem which is what kind of worries me
16:04:56 mriedem anyway, it would likely break my brain to try and audit all of those places right now
16:18:23 mnaser melwitt: dansmith https://review.openstack.org/#/c/539005/ thanks in advance, we just ran into this right now
16:18:47 cdent mriedem: have you tried putting the contents of verify_placement.sh in the parent file in a ( subshell ) and then checking the exit code of the whole thing?
16:19:20 cdent I'm not tracking what you've already tried, just saw the recent "external file not gonna work" report
16:23:05 mriedem cdent: nope
16:23:31 mriedem i think the recent failure is due to the legacy nova-next job running from a symlink in nova/tools/hooks
16:33:48 openstackgerrit Eric Berglund proposed openstack/nova master: Add check for redundant import aliases https://review.openstack.org/543995
16:36:24 mriedem i wonder why the nova admin guide docs on schedulers gives a reference to the volume scheduler in cinder https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html
16:36:37 mriedem the link is broken, but i don't know if i should fix it or just remove that part
16:37:22 gibi mriedem: does this trace mean that nova does not handle the exception? https://bugs.launchpad.net/nova/+bug/1749215/comments/3
16:37:23 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
16:37:53 mriedem gibi: depends on what you mean by handle
16:38:54 mriedem that's in the _error_out_instance_on_exception context manager,
16:39:09 mriedem which is setting the instance to ERROR due to an unexpected exception
16:39:19 mriedem that's likely a bug, the instance shouldn't be in ERROR
16:39:32 mriedem for "Resize error: Unable to resize disk down."
16:39:33 gibi mriedem: in my log this is the last place nova mentions the ResizeError
16:39:33 openstackgerrit Merged openstack/nova stable/pike: Make eventlet hub use a monotonic clock https://review.openstack.org/537811
16:40:08 mriedem gibi: sure, that's because resize_instance is running on the source host, via rpc cast from prep_resize on the dest host
16:40:09 gibi mriedem: I in that context manager this is coming from we does not delete claims
16:40:14 mriedem so if that fails, we should cleanup and exit
16:40:47 mriedem the claim is made in prep_resize on the dest
16:41:02 mriedem before we rpc cast to resize_instance
16:41:38 gibi mriedem: ahh that was the missing piece
16:41:50 gibi mriedem: thanks, now I have a place to look at
16:42:53 mriedem what i'm not sure about is if/when the dest host RT updates its usage after the migration fails to remove the claim
16:43:52 mriedem # Grab all in-progress migrations:
16:43:52 mriedem migrations = objects.MigrationList.get_in_progress_by_host_and_node(
16:43:53 mriedem context, self.host, nodename)
16:43:53 mriedem self._pair_instances_to_migrations(migrations, instances)
16:43:53 mriedem self._update_usage_from_migrations(context, migrations, nodename)
16:43:59 mriedem that should update the dest host RT at some point
16:44:08 mriedem once the migration fails, it will no longer be in that in-progress list
16:46:37 gibi mriedem: in my env I don't see that this Exception handler is called during the failed resize: https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4124
16:47:00 mriedem gibi: it's not,
16:47:08 mriedem because we've already rpc cast to the source host
16:47:26 mriedem https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4085
16:47:39 mriedem that's why claudiub's original fix was to change that to an rpc call
16:47:51 mriedem so the failure would come back to prep_resize on the dest and we'd run that cleanup code

Earlier   Later