Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-13
16:03:00 dansmith mriedem: well, the bug would be not in the instance delete path, but in the path where we left a migration allocation in place from the failed resize yeah?
16:03:18 mriedem yeah that's what i mean by "if we rollback at the point of failure correctly, we shouldn't need to"
16:03:48 mriedem looking at all of the resize flows, and where errors_out_migration_ctxt is used, it could be a lot of places
16:04:06 mriedem which is what kind of worries me
16:04:56 mriedem anyway, it would likely break my brain to try and audit all of those places right now
16:18:23 mnaser melwitt: dansmith https://review.openstack.org/#/c/539005/ thanks in advance, we just ran into this right now
16:18:47 cdent mriedem: have you tried putting the contents of verify_placement.sh in the parent file in a ( subshell ) and then checking the exit code of the whole thing?
16:19:20 cdent I'm not tracking what you've already tried, just saw the recent "external file not gonna work" report
16:23:05 mriedem cdent: nope
16:23:31 mriedem i think the recent failure is due to the legacy nova-next job running from a symlink in nova/tools/hooks
16:33:48 openstackgerrit Eric Berglund proposed openstack/nova master: Add check for redundant import aliases https://review.openstack.org/543995
16:36:24 mriedem i wonder why the nova admin guide docs on schedulers gives a reference to the volume scheduler in cinder https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html
16:36:37 mriedem the link is broken, but i don't know if i should fix it or just remove that part
16:37:22 gibi mriedem: does this trace mean that nova does not handle the exception? https://bugs.launchpad.net/nova/+bug/1749215/comments/3
16:37:23 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
16:37:53 mriedem gibi: depends on what you mean by handle
16:38:54 mriedem that's in the _error_out_instance_on_exception context manager,
16:39:09 mriedem which is setting the instance to ERROR due to an unexpected exception
16:39:19 mriedem that's likely a bug, the instance shouldn't be in ERROR
16:39:32 mriedem for "Resize error: Unable to resize disk down."
16:39:33 openstackgerrit Merged openstack/nova stable/pike: Make eventlet hub use a monotonic clock https://review.openstack.org/537811
16:39:33 gibi mriedem: in my log this is the last place nova mentions the ResizeError
16:40:08 mriedem gibi: sure, that's because resize_instance is running on the source host, via rpc cast from prep_resize on the dest host
16:40:09 gibi mriedem: I in that context manager this is coming from we does not delete claims
16:40:14 mriedem so if that fails, we should cleanup and exit
16:40:47 mriedem the claim is made in prep_resize on the dest
16:41:02 mriedem before we rpc cast to resize_instance
16:41:38 gibi mriedem: ahh that was the missing piece
16:41:50 gibi mriedem: thanks, now I have a place to look at
16:42:53 mriedem what i'm not sure about is if/when the dest host RT updates its usage after the migration fails to remove the claim
16:43:52 mriedem migrations = objects.MigrationList.get_in_progress_by_host_and_node(
16:43:52 mriedem # Grab all in-progress migrations:
16:43:53 mriedem self._update_usage_from_migrations(context, migrations, nodename)
16:43:53 mriedem self._pair_instances_to_migrations(migrations, instances)
16:43:53 mriedem context, self.host, nodename)
16:43:59 mriedem that should update the dest host RT at some point
16:44:08 mriedem once the migration fails, it will no longer be in that in-progress list
16:46:37 gibi mriedem: in my env I don't see that this Exception handler is called during the failed resize: https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4124
16:47:00 mriedem gibi: it's not,
16:47:08 mriedem because we've already rpc cast to the source host
16:47:26 mriedem https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4085
16:47:39 mriedem that's why claudiub's original fix was to change that to an rpc call
16:47:51 mriedem so the failure would come back to prep_resize on the dest and we'd run that cleanup code
16:48:03 mriedem but rpc calls just mean messaging timeouts
16:48:06 mriedem so i'd rather not do that
16:48:08 openstackgerrit Merged openstack/python-novaclient stable/queens: Updated from global requirements https://review.openstack.org/543741
16:48:09 gibi mriedem: OK, then I understand what I see.
16:48:18 gibi mriedem: thanks again
16:48:45 gibi mriedem: I have to run now but if the problem still open then I can look at it again tomorrow
17:04:50 hegemoOn hello there
17:05:04 hegemoOn is it possible to define the number of queue in an image metadata
17:05:15 hegemoOn when using virtion-scsi ?
17:09:05 mriedem stephenfin: on https://review.openstack.org/#/c/531413/ - i think i might decouple the re-wording part so we can backport that, since i see some other config guide docs that reference that option
17:11:00 stephenfin mriedem: Sure thing. The reservation was because we haven't removed deprecated aliases before, that I'm aware of (there's little cost in keeping them). Worth making sure it was intentional
17:11:13 mriedem i'm sure we've removed deprecated aliases before
17:11:29 mriedem maybe not recently, but it was pretty common cleanup back in the day
17:12:01 stephenfin :D
17:12:15 stephenfin numa-aware-vswitches has me plenty busy :)
17:14:44 hegemoOn in virsh xml you have <driver queues='n'>
17:14:56 hegemoOn for virtio-scsi
17:15:09 hegemoOn i cannot see a way to define n in nova
17:16:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010
17:19:28 openstackgerrit Chris Dent proposed openstack/nova master: Move db MAX constants to own file https://review.openstack.org/543469
17:21:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove the deprecated scheduler_driver_task_period option https://review.openstack.org/531413
17:21:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Clarify the help text for [scheduler]periodic_task_interval https://review.openstack.org/544015
17:24:50 stephenfin mriedem: Should I have +Wd this, given that you've a -2 on the following patch? https://review.openstack.org/#/c/539738/
17:25:00 stephenfin I can rebase and take it out of the gate if not
17:25:32 mriedem it's a bug fix
17:25:35 mriedem so i don't think so
17:25:48 stephenfin Phew. Okidok
17:25:53 mriedem i mean, it's not a problem
17:27:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove warning in feature support matrix page https://review.openstack.org/544017
17:30:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Check for leaked server resource allocations in post_test_hook https://review.openstack.org/538510
17:31:14 openstackgerrit James E. Blair proposed openstack/python-novaclient stable/ocata: Zuul: Remove project name https://review.openstack.org/544018
17:32:00 openstackgerrit Merged openstack/nova stable/pike: Query all cells for service version in _validate_bdm https://review.openstack.org/541036
17:33:03 openstackgerrit Eric Berglund proposed openstack/nova master: Use correct arguments in task inits https://review.openstack.org/543571
17:34:42 mriedem lyarwood: dansmith: bauzas: so before i request the next pike release, do i have to bump the minor version to 16.1.0 for https://review.openstack.org/#/c/528330/ because it's a schema migration; i will forever remember the red hat product team lashing i got for *not* doing something like that back in newton
17:35:49 dansmith um
17:35:51 dansmith okay
17:36:02 dansmith bump the minor _to_ what?
17:36:10 mriedem 16.0.4 -> 16.1.0
17:36:15 dansmith oh I see
17:36:18 mriedem i would normally just do 16.0.5
17:36:23 mriedem but caught hell for doing that once
17:36:24 dansmith I don't think I know anything about that
17:40:47 mriedem alright 16.1.0 it is https://review.openstack.org/#/c/544020/
18:29:43 lyarwood mriedem: 16.1.0 is fine by me, this is still an optional schema migration until queens anyway right so anyone not running migrations after updating nova during stable/pike will pick it up when they upgrade to queens.
18:31:15 lyarwood mriedem: FWIW with TripleO/RDO I can't see us ever running schema migrations with each minor update (16.0.4 to 16.1.0 etc)
18:31:48 dansmith lyarwood: apparently a minor version change kicks the "do the db sync" flag on
18:32:21 lyarwood dansmith: not in any of our tooling
18:32:32 lyarwood dansmith: just the poor ops guy who notices the change
18:32:46 dansmith lyarwood: supposedly bauzas beat up mriedem about it last time on that basis
18:34:29 lyarwood owalsh: ^ re minor updates on TripleO, we don't run schema migrations at all right?
18:35:30 dansmith if not then bauzas should pay for mriedem's therapist bills
18:35:50 lyarwood dansmith: I can only assume his point was that we should still highlight to ops etc that the update contains a schema migration by bumping the minor release or something
18:39:49 owalsh lyarwood: correct, just restart services
18:40:11 lyarwood owalsh: cool thanks
18:40:24 dansmith death match round 2, dublin
18:53:36 mriedem fork in the kidneys, check
19:55:15 openstackgerrit Merged openstack/nova master: Replace Chinese quotes to English quotes https://review.openstack.org/543349

Earlier   Later