Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-13
15:40:15 mriedem if resize_instance takes >1 minute
15:40:17 claudiub well, not necessarely always. for example, if an InstanceFaultRollback is raised by the driver (at least in our scenario - if the new flavor's disk is smaller than the original), then the resize didn't happen, and the instance will be put back into Running state
15:40:37 claudiub hm, I see.
15:42:50 mriedem i see that the libvirt driver's migrate_disk_and_power_off method, called from resize_instance, is doing some checks like if you're trying to resize the disk down or if you're using lvm, it fails,
15:42:52 jaypipes dims: https://twitter.com/jaypipes/status/963437624852668419
15:42:57 mriedem in that case we'd fail and not reschedule
15:43:06 mriedem but in those cases, we don't want to reschedule
15:43:26 claudiub fair enough
15:43:32 dims jaypipes, y i was reading all about it yesterday ... :(
15:43:56 claudiub well then, I'll send another PS.
15:48:13 mriedem claudiub: you can maybe build a functional test similar to test_resize_to_same_host_prep_resize_fails
15:48:25 mriedem where resize_instance fails and we make sure to cleanup allocations created by the scheduler for the destination host
15:49:50 claudiub will do
15:59:38 mriedem dansmith: gibi: claudiub's bug got me thinking about how we might leak migration allocations during a failed cold migrate https://bugs.launchpad.net/nova/+bug/1749215/comments/2
15:59:40 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
15:59:55 mriedem normally deleting an instance removes it's allocations from placement and we're happy, that's the last resort kind of cleanup thing,
16:00:10 mriedem but we don't cleanup any allocations tracked against migration records for the instance when we delete the instance,
16:00:19 mriedem granted if we rollback at the point of failure correctly, we shouldn't need to
16:03:00 dansmith mriedem: well, the bug would be not in the instance delete path, but in the path where we left a migration allocation in place from the failed resize yeah?
16:03:18 mriedem yeah that's what i mean by "if we rollback at the point of failure correctly, we shouldn't need to"
16:03:48 mriedem looking at all of the resize flows, and where errors_out_migration_ctxt is used, it could be a lot of places
16:04:06 mriedem which is what kind of worries me
16:04:56 mriedem anyway, it would likely break my brain to try and audit all of those places right now
16:18:23 mnaser melwitt: dansmith https://review.openstack.org/#/c/539005/ thanks in advance, we just ran into this right now
16:18:47 cdent mriedem: have you tried putting the contents of verify_placement.sh in the parent file in a ( subshell ) and then checking the exit code of the whole thing?
16:19:20 cdent I'm not tracking what you've already tried, just saw the recent "external file not gonna work" report
16:23:05 mriedem cdent: nope
16:23:31 mriedem i think the recent failure is due to the legacy nova-next job running from a symlink in nova/tools/hooks
16:33:48 openstackgerrit Eric Berglund proposed openstack/nova master: Add check for redundant import aliases https://review.openstack.org/543995
16:36:24 mriedem i wonder why the nova admin guide docs on schedulers gives a reference to the volume scheduler in cinder https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html
16:36:37 mriedem the link is broken, but i don't know if i should fix it or just remove that part
16:37:22 gibi mriedem: does this trace mean that nova does not handle the exception? https://bugs.launchpad.net/nova/+bug/1749215/comments/3
16:37:23 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
16:37:53 mriedem gibi: depends on what you mean by handle
16:38:54 mriedem that's in the _error_out_instance_on_exception context manager,
16:39:09 mriedem which is setting the instance to ERROR due to an unexpected exception
16:39:19 mriedem that's likely a bug, the instance shouldn't be in ERROR
16:39:32 mriedem for "Resize error: Unable to resize disk down."
16:39:33 gibi mriedem: in my log this is the last place nova mentions the ResizeError
16:39:33 openstackgerrit Merged openstack/nova stable/pike: Make eventlet hub use a monotonic clock https://review.openstack.org/537811
16:40:08 mriedem gibi: sure, that's because resize_instance is running on the source host, via rpc cast from prep_resize on the dest host
16:40:09 gibi mriedem: I in that context manager this is coming from we does not delete claims
16:40:14 mriedem so if that fails, we should cleanup and exit
16:40:47 mriedem the claim is made in prep_resize on the dest
16:41:02 mriedem before we rpc cast to resize_instance
16:41:38 gibi mriedem: ahh that was the missing piece
16:41:50 gibi mriedem: thanks, now I have a place to look at
16:42:53 mriedem what i'm not sure about is if/when the dest host RT updates its usage after the migration fails to remove the claim
16:43:52 mriedem # Grab all in-progress migrations:
16:43:52 mriedem migrations = objects.MigrationList.get_in_progress_by_host_and_node(
16:43:53 mriedem context, self.host, nodename)
16:43:53 mriedem self._pair_instances_to_migrations(migrations, instances)
16:43:53 mriedem self._update_usage_from_migrations(context, migrations, nodename)
16:43:59 mriedem that should update the dest host RT at some point
16:44:08 mriedem once the migration fails, it will no longer be in that in-progress list
16:46:37 gibi mriedem: in my env I don't see that this Exception handler is called during the failed resize: https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4124
16:47:00 mriedem gibi: it's not,
16:47:08 mriedem because we've already rpc cast to the source host
16:47:26 mriedem https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4085
16:47:39 mriedem that's why claudiub's original fix was to change that to an rpc call
16:47:51 mriedem so the failure would come back to prep_resize on the dest and we'd run that cleanup code
16:48:03 mriedem but rpc calls just mean messaging timeouts
16:48:06 mriedem so i'd rather not do that
16:48:08 openstackgerrit Merged openstack/python-novaclient stable/queens: Updated from global requirements https://review.openstack.org/543741
16:48:09 gibi mriedem: OK, then I understand what I see.
16:48:18 gibi mriedem: thanks again
16:48:45 gibi mriedem: I have to run now but if the problem still open then I can look at it again tomorrow
17:04:50 hegemoOn hello there
17:05:04 hegemoOn is it possible to define the number of queue in an image metadata
17:05:15 hegemoOn when using virtion-scsi ?
17:09:05 mriedem stephenfin: on https://review.openstack.org/#/c/531413/ - i think i might decouple the re-wording part so we can backport that, since i see some other config guide docs that reference that option
17:11:00 stephenfin mriedem: Sure thing. The reservation was because we haven't removed deprecated aliases before, that I'm aware of (there's little cost in keeping them). Worth making sure it was intentional
17:11:13 mriedem i'm sure we've removed deprecated aliases before
17:11:29 mriedem maybe not recently, but it was pretty common cleanup back in the day
17:12:01 stephenfin :D
17:12:15 stephenfin numa-aware-vswitches has me plenty busy :)
17:14:44 hegemoOn in virsh xml you have <driver queues='n'>
17:14:56 hegemoOn for virtio-scsi
17:15:09 hegemoOn i cannot see a way to define n in nova
17:16:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010
17:19:28 openstackgerrit Chris Dent proposed openstack/nova master: Move db MAX constants to own file https://review.openstack.org/543469
17:21:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove the deprecated scheduler_driver_task_period option https://review.openstack.org/531413
17:21:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Clarify the help text for [scheduler]periodic_task_interval https://review.openstack.org/544015
17:24:50 stephenfin mriedem: Should I have +Wd this, given that you've a -2 on the following patch? https://review.openstack.org/#/c/539738/
17:25:00 stephenfin I can rebase and take it out of the gate if not
17:25:32 mriedem it's a bug fix
17:25:35 mriedem so i don't think so
17:25:48 stephenfin Phew. Okidok
17:25:53 mriedem i mean, it's not a problem
17:27:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove warning in feature support matrix page https://review.openstack.org/544017
17:30:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Check for leaked server resource allocations in post_test_hook https://review.openstack.org/538510
17:31:14 openstackgerrit James E. Blair proposed openstack/python-novaclient stable/ocata: Zuul: Remove project name https://review.openstack.org/544018
17:32:00 openstackgerrit Merged openstack/nova stable/pike: Query all cells for service version in _validate_bdm https://review.openstack.org/541036
17:33:03 openstackgerrit Eric Berglund proposed openstack/nova master: Use correct arguments in task inits https://review.openstack.org/543571
17:34:42 mriedem lyarwood: dansmith: bauzas: so before i request the next pike release, do i have to bump the minor version to 16.1.0 for https://review.openstack.org/#/c/528330/ because it's a schema migration; i will forever remember the red hat product team lashing i got for *not* doing something like that back in newton
17:35:49 dansmith um
17:35:51 dansmith okay
17:36:02 dansmith bump the minor _to_ what?
17:36:10 mriedem 16.0.4 -> 16.1.0
17:36:15 dansmith oh I see

Earlier   Later