Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-13
15:33:55 mriedem if not, can you?
15:34:18 claudiub I did. I just sent a patch for it as well
15:34:20 claudiub ^
15:34:40 claudiub basically, the Except branch in ComputeManager's prep_resize is never executed.
15:34:45 mriedem claudiub: i don't think that's the kind of fix we want
15:35:00 claudiub in which the allocations gets cleaned up
15:35:03 mriedem we likely need to handle the allocation cleanup in resize_instance
15:35:31 stephenfin mriedem: Both the commit message and comment in 'nova-next/run.yaml' say the 'tls-proxy' service is enabled when it isn't. If you want to drop that, I can +W
15:35:43 stephenfin Or vice versa. It's confusing otherwise :)
15:36:00 claudiub that could work too
15:36:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/513160
15:37:21 claudiub we might have to do the rescheduling there too, then.
15:37:21 mriedem stephenfin: done
15:37:40 stephenfin mriedem: likewise
15:37:43 mriedem claudiub: if resize_instance fails, we're dead
15:38:36 claudiub what do you mean?
15:38:53 claudiub you mean that the resize operation should fail, and be put in error state?
15:38:54 mriedem i don't think we need to try and make every part of a resize reschedulable
15:38:56 mriedem yes
15:40:08 mriedem changing the rpc cast to call might fix this issue, but likely introduces an rpc messaging timeout problem
15:40:15 mriedem if resize_instance takes >1 minute
15:40:17 claudiub well, not necessarely always. for example, if an InstanceFaultRollback is raised by the driver (at least in our scenario - if the new flavor's disk is smaller than the original), then the resize didn't happen, and the instance will be put back into Running state
15:40:37 claudiub hm, I see.
15:42:50 mriedem i see that the libvirt driver's migrate_disk_and_power_off method, called from resize_instance, is doing some checks like if you're trying to resize the disk down or if you're using lvm, it fails,
15:42:52 jaypipes dims: https://twitter.com/jaypipes/status/963437624852668419
15:42:57 mriedem in that case we'd fail and not reschedule
15:43:06 mriedem but in those cases, we don't want to reschedule
15:43:26 claudiub fair enough
15:43:32 dims jaypipes, y i was reading all about it yesterday ... :(
15:43:56 claudiub well then, I'll send another PS.
15:48:13 mriedem claudiub: you can maybe build a functional test similar to test_resize_to_same_host_prep_resize_fails
15:48:25 mriedem where resize_instance fails and we make sure to cleanup allocations created by the scheduler for the destination host
15:49:50 claudiub will do
15:59:38 mriedem dansmith: gibi: claudiub's bug got me thinking about how we might leak migration allocations during a failed cold migrate https://bugs.launchpad.net/nova/+bug/1749215/comments/2
15:59:40 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
15:59:55 mriedem normally deleting an instance removes it's allocations from placement and we're happy, that's the last resort kind of cleanup thing,
16:00:10 mriedem but we don't cleanup any allocations tracked against migration records for the instance when we delete the instance,
16:00:19 mriedem granted if we rollback at the point of failure correctly, we shouldn't need to
16:03:00 dansmith mriedem: well, the bug would be not in the instance delete path, but in the path where we left a migration allocation in place from the failed resize yeah?
16:03:18 mriedem yeah that's what i mean by "if we rollback at the point of failure correctly, we shouldn't need to"
16:03:48 mriedem looking at all of the resize flows, and where errors_out_migration_ctxt is used, it could be a lot of places
16:04:06 mriedem which is what kind of worries me
16:04:56 mriedem anyway, it would likely break my brain to try and audit all of those places right now
16:18:23 mnaser melwitt: dansmith https://review.openstack.org/#/c/539005/ thanks in advance, we just ran into this right now
16:18:47 cdent mriedem: have you tried putting the contents of verify_placement.sh in the parent file in a ( subshell ) and then checking the exit code of the whole thing?
16:19:20 cdent I'm not tracking what you've already tried, just saw the recent "external file not gonna work" report
16:23:05 mriedem cdent: nope
16:23:31 mriedem i think the recent failure is due to the legacy nova-next job running from a symlink in nova/tools/hooks
16:33:48 openstackgerrit Eric Berglund proposed openstack/nova master: Add check for redundant import aliases https://review.openstack.org/543995
16:36:24 mriedem i wonder why the nova admin guide docs on schedulers gives a reference to the volume scheduler in cinder https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html
16:36:37 mriedem the link is broken, but i don't know if i should fix it or just remove that part
16:37:22 gibi mriedem: does this trace mean that nova does not handle the exception? https://bugs.launchpad.net/nova/+bug/1749215/comments/3
16:37:23 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
16:37:53 mriedem gibi: depends on what you mean by handle
16:38:54 mriedem that's in the _error_out_instance_on_exception context manager,
16:39:09 mriedem which is setting the instance to ERROR due to an unexpected exception
16:39:19 mriedem that's likely a bug, the instance shouldn't be in ERROR
16:39:32 mriedem for "Resize error: Unable to resize disk down."
16:39:33 openstackgerrit Merged openstack/nova stable/pike: Make eventlet hub use a monotonic clock https://review.openstack.org/537811
16:39:33 gibi mriedem: in my log this is the last place nova mentions the ResizeError
16:40:08 mriedem gibi: sure, that's because resize_instance is running on the source host, via rpc cast from prep_resize on the dest host
16:40:09 gibi mriedem: I in that context manager this is coming from we does not delete claims
16:40:14 mriedem so if that fails, we should cleanup and exit
16:40:47 mriedem the claim is made in prep_resize on the dest
16:41:02 mriedem before we rpc cast to resize_instance
16:41:38 gibi mriedem: ahh that was the missing piece
16:41:50 gibi mriedem: thanks, now I have a place to look at
16:42:53 mriedem what i'm not sure about is if/when the dest host RT updates its usage after the migration fails to remove the claim
16:43:52 mriedem migrations = objects.MigrationList.get_in_progress_by_host_and_node(
16:43:52 mriedem # Grab all in-progress migrations:
16:43:53 mriedem self._update_usage_from_migrations(context, migrations, nodename)
16:43:53 mriedem self._pair_instances_to_migrations(migrations, instances)
16:43:53 mriedem context, self.host, nodename)
16:43:59 mriedem that should update the dest host RT at some point
16:44:08 mriedem once the migration fails, it will no longer be in that in-progress list
16:46:37 gibi mriedem: in my env I don't see that this Exception handler is called during the failed resize: https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4124
16:47:00 mriedem gibi: it's not,
16:47:08 mriedem because we've already rpc cast to the source host
16:47:26 mriedem https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4085
16:47:39 mriedem that's why claudiub's original fix was to change that to an rpc call
16:47:51 mriedem so the failure would come back to prep_resize on the dest and we'd run that cleanup code
16:48:03 mriedem but rpc calls just mean messaging timeouts
16:48:06 mriedem so i'd rather not do that
16:48:08 openstackgerrit Merged openstack/python-novaclient stable/queens: Updated from global requirements https://review.openstack.org/543741
16:48:09 gibi mriedem: OK, then I understand what I see.
16:48:18 gibi mriedem: thanks again
16:48:45 gibi mriedem: I have to run now but if the problem still open then I can look at it again tomorrow
17:04:50 hegemoOn hello there
17:05:04 hegemoOn is it possible to define the number of queue in an image metadata
17:05:15 hegemoOn when using virtion-scsi ?
17:09:05 mriedem stephenfin: on https://review.openstack.org/#/c/531413/ - i think i might decouple the re-wording part so we can backport that, since i see some other config guide docs that reference that option
17:11:00 stephenfin mriedem: Sure thing. The reservation was because we haven't removed deprecated aliases before, that I'm aware of (there's little cost in keeping them). Worth making sure it was intentional
17:11:13 mriedem i'm sure we've removed deprecated aliases before
17:11:29 mriedem maybe not recently, but it was pretty common cleanup back in the day
17:12:01 stephenfin :D
17:12:15 stephenfin numa-aware-vswitches has me plenty busy :)
17:14:44 hegemoOn in virsh xml you have <driver queues='n'>
17:14:56 hegemoOn for virtio-scsi
17:15:09 hegemoOn i cannot see a way to define n in nova
17:16:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010

Earlier   Later