Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-13
16:37:23 openstack Launchpad bug 1749215 in OpenStack Compute (nova) "Allocations not deleted on failed resize_instance" [Medium,In progress] - Assigned to Claudiu Belu (cbelu)
16:37:53 mriedem gibi: depends on what you mean by handle
16:38:54 mriedem that's in the _error_out_instance_on_exception context manager,
16:39:09 mriedem which is setting the instance to ERROR due to an unexpected exception
16:39:19 mriedem that's likely a bug, the instance shouldn't be in ERROR
16:39:32 mriedem for "Resize error: Unable to resize disk down."
16:39:33 openstackgerrit Merged openstack/nova stable/pike: Make eventlet hub use a monotonic clock https://review.openstack.org/537811
16:39:33 gibi mriedem: in my log this is the last place nova mentions the ResizeError
16:40:08 mriedem gibi: sure, that's because resize_instance is running on the source host, via rpc cast from prep_resize on the dest host
16:40:09 gibi mriedem: I in that context manager this is coming from we does not delete claims
16:40:14 mriedem so if that fails, we should cleanup and exit
16:40:47 mriedem the claim is made in prep_resize on the dest
16:41:02 mriedem before we rpc cast to resize_instance
16:41:38 gibi mriedem: ahh that was the missing piece
16:41:50 gibi mriedem: thanks, now I have a place to look at
16:42:53 mriedem what i'm not sure about is if/when the dest host RT updates its usage after the migration fails to remove the claim
16:43:52 mriedem migrations = objects.MigrationList.get_in_progress_by_host_and_node(
16:43:52 mriedem # Grab all in-progress migrations:
16:43:53 mriedem self._update_usage_from_migrations(context, migrations, nodename)
16:43:53 mriedem self._pair_instances_to_migrations(migrations, instances)
16:43:53 mriedem context, self.host, nodename)
16:43:59 mriedem that should update the dest host RT at some point
16:44:08 mriedem once the migration fails, it will no longer be in that in-progress list
16:46:37 gibi mriedem: in my env I don't see that this Exception handler is called during the failed resize: https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4124
16:47:00 mriedem gibi: it's not,
16:47:08 mriedem because we've already rpc cast to the source host
16:47:26 mriedem https://github.com/openstack/nova/blob/fa6c0f9cb14f1b4ce4d9b1dbacb1743173089986/nova/compute/manager.py#L4085
16:47:39 mriedem that's why claudiub's original fix was to change that to an rpc call
16:47:51 mriedem so the failure would come back to prep_resize on the dest and we'd run that cleanup code
16:48:03 mriedem but rpc calls just mean messaging timeouts
16:48:06 mriedem so i'd rather not do that
16:48:08 openstackgerrit Merged openstack/python-novaclient stable/queens: Updated from global requirements https://review.openstack.org/543741
16:48:09 gibi mriedem: OK, then I understand what I see.
16:48:18 gibi mriedem: thanks again
16:48:45 gibi mriedem: I have to run now but if the problem still open then I can look at it again tomorrow
17:04:50 hegemoOn hello there
17:05:04 hegemoOn is it possible to define the number of queue in an image metadata
17:05:15 hegemoOn when using virtion-scsi ?
17:09:05 mriedem stephenfin: on https://review.openstack.org/#/c/531413/ - i think i might decouple the re-wording part so we can backport that, since i see some other config guide docs that reference that option
17:11:00 stephenfin mriedem: Sure thing. The reservation was because we haven't removed deprecated aliases before, that I'm aware of (there's little cost in keeping them). Worth making sure it was intentional
17:11:13 mriedem i'm sure we've removed deprecated aliases before
17:11:29 mriedem maybe not recently, but it was pretty common cleanup back in the day
17:12:01 stephenfin :D
17:12:15 stephenfin numa-aware-vswitches has me plenty busy :)
17:14:44 hegemoOn in virsh xml you have <driver queues='n'>
17:14:56 hegemoOn for virtio-scsi
17:15:09 hegemoOn i cannot see a way to define n in nova
17:16:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix and update compute schedulers config guide https://review.openstack.org/544010
17:19:28 openstackgerrit Chris Dent proposed openstack/nova master: Move db MAX constants to own file https://review.openstack.org/543469
17:21:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove the deprecated scheduler_driver_task_period option https://review.openstack.org/531413
17:21:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Clarify the help text for [scheduler]periodic_task_interval https://review.openstack.org/544015
17:24:50 stephenfin mriedem: Should I have +Wd this, given that you've a -2 on the following patch? https://review.openstack.org/#/c/539738/
17:25:00 stephenfin I can rebase and take it out of the gate if not
17:25:32 mriedem it's a bug fix
17:25:35 mriedem so i don't think so
17:25:48 stephenfin Phew. Okidok
17:25:53 mriedem i mean, it's not a problem
17:27:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove warning in feature support matrix page https://review.openstack.org/544017
17:30:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Check for leaked server resource allocations in post_test_hook https://review.openstack.org/538510
17:31:14 openstackgerrit James E. Blair proposed openstack/python-novaclient stable/ocata: Zuul: Remove project name https://review.openstack.org/544018
17:32:00 openstackgerrit Merged openstack/nova stable/pike: Query all cells for service version in _validate_bdm https://review.openstack.org/541036
17:33:03 openstackgerrit Eric Berglund proposed openstack/nova master: Use correct arguments in task inits https://review.openstack.org/543571
17:34:42 mriedem lyarwood: dansmith: bauzas: so before i request the next pike release, do i have to bump the minor version to 16.1.0 for https://review.openstack.org/#/c/528330/ because it's a schema migration; i will forever remember the red hat product team lashing i got for *not* doing something like that back in newton
17:35:49 dansmith um
17:35:51 dansmith okay
17:36:02 dansmith bump the minor _to_ what?
17:36:10 mriedem 16.0.4 -> 16.1.0
17:36:15 dansmith oh I see
17:36:18 mriedem i would normally just do 16.0.5
17:36:23 mriedem but caught hell for doing that once
17:36:24 dansmith I don't think I know anything about that
17:40:47 mriedem alright 16.1.0 it is https://review.openstack.org/#/c/544020/
18:29:43 lyarwood mriedem: 16.1.0 is fine by me, this is still an optional schema migration until queens anyway right so anyone not running migrations after updating nova during stable/pike will pick it up when they upgrade to queens.
18:31:15 lyarwood mriedem: FWIW with TripleO/RDO I can't see us ever running schema migrations with each minor update (16.0.4 to 16.1.0 etc)
18:31:48 dansmith lyarwood: apparently a minor version change kicks the "do the db sync" flag on
18:32:21 lyarwood dansmith: not in any of our tooling
18:32:32 lyarwood dansmith: just the poor ops guy who notices the change
18:32:46 dansmith lyarwood: supposedly bauzas beat up mriedem about it last time on that basis
18:34:29 lyarwood owalsh: ^ re minor updates on TripleO, we don't run schema migrations at all right?
18:35:30 dansmith if not then bauzas should pay for mriedem's therapist bills
18:35:50 lyarwood dansmith: I can only assume his point was that we should still highlight to ops etc that the update contains a schema migration by bumping the minor release or something
18:39:49 owalsh lyarwood: correct, just restart services
18:40:11 lyarwood owalsh: cool thanks
18:40:24 dansmith death match round 2, dublin
18:53:36 mriedem fork in the kidneys, check
19:55:15 openstackgerrit Merged openstack/nova master: Replace Chinese quotes to English quotes https://review.openstack.org/543349
19:56:31 dansmith efried: jaypipes: seen the question on this? https://review.openstack.org/#/c/540111/3
19:56:45 dansmith I was about to reply, but then realized I was misunderstanding his concern and I think it's probably valid
19:57:06 dansmith hoping that there's some detail of how you see that working that wouldn't actually break it
19:57:46 efried dansmith: It's been on my list to look at, but was rapidly getting buried. Thanks for bringing it back to the top. Looking....
19:58:58 mriedem been wondering the same type of thing with traits,
19:59:12 mriedem the ironic driver will blow away any traits that aren't on the ironic node
19:59:16 mriedem rather than try to merge the
19:59:18 mriedem *them
19:59:27 dansmith well, there needs to be some amount of that I think,
19:59:36 dansmith although we can't blow them all away in this case I think
19:59:49 dansmith was hoping there was some "only blows away at the given level" detail or something
20:00:09 efried mriedem: I remember that being discussed at length (for ironic traits), and the conclusion in that case was that the ironic inspector was the Source Of Truth, so it was kosher to blow away anything that crept in from elsewhere.
20:00:26 mriedem idk, it seems quite limiting
20:00:41 dansmith efried: it is for sure until the compute service starts needing to do some too, like for capabilities

Earlier   Later