Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-03
17:52:10 hansmoleman "Hm, this actually makes the previously passing ceph job fail. And thinking about it, this is backwards -- we shouldn't delete the volume snapshot first if the created volume depends on it. The created volume should be deleted first, and then that would allow the volume snapshot to be deleted (in the case of ceph)."
17:52:16 jgwentworth ah, only failed for ceph
17:52:43 jgwentworth okay, so do we need different behavior depending on whether it's ceph?
17:53:25 hansmoleman the issue is that we have to delete the volume snapshot before attempting to delete the volume, and nova-compute deletes the volume,
17:53:39 hansmoleman was there an issue with trying to just delete the volume snapshot before the server with ceph?
17:54:07 hansmoleman https://review.openstack.org/#/c/565601/5 is what i was +1 on
17:54:36 hansmoleman and you said that made ceph fail?
17:54:38 jgwentworth yes, it failed every time with ceph. so I thought maybe the comment was backward
17:54:43 hansmoleman ceph is backward
17:54:58 hansmoleman the cinder api shouldn't really behave differently depending on the volume type
17:55:19 hansmoleman otherwise client side tooling would always need a "if volume_type=='rbd'" condition
17:55:22 jgwentworth it has something to do with the references in ceph. trying to see what it was exactly
17:55:29 jgwentworth yeah
17:57:11 hansmoleman idk how you delete a ceph volume that has snapshots then,
17:57:18 hansmoleman because if you can't remove the snapshots until the volume is gone,
17:57:25 jgwentworth you have to delete the servers that reference it first, I think
17:57:25 hansmoleman and you can't delete the volume while it has snapshots,
17:57:26 hansmoleman then wtf
17:57:30 hansmoleman ugh
17:57:54 hansmoleman so for rbd volume-backed servers with snapshots, you just always orphan the volumes?
17:58:40 jgwentworth well, in the case of the test, what happens is the server is deleted and then nova deletes the volume snapshot after
17:59:12 hansmoleman nova doesn't delete volume snapshots
17:59:24 hansmoleman nova-compute will attempt to delete the volume because bdm.delete_on_termination,
17:59:25 jgwentworth I mean, the virt domain gets destroyed and then compute deletes the underlying volume if delete_on_termination=True
17:59:30 hansmoleman which fails if the volume has snapshots
18:01:47 jgwentworth okay, I think I'm lacking on knowledge of volume snapshots. let me dig into how this works (non-ceph vs ceph) and propose a change to put the comment back and explain the ceph part
18:02:48 jgwentworth I didn't realize they were different and thought I had made a mistake with trying to delete the snapshot first because that change failed the ceph job, whereas leaving it alone had both jobs passing
18:49:14 openstackgerrit Merged openstack/nova master: Reno for notification-transformation-rocky https://review.openstack.org/588403
18:55:33 openstackgerrit Chris Dent proposed openstack/nova master: [placement] ensure_rc_cache only at start of process https://review.openstack.org/584086
19:07:48 hansmoleman cfriesen: what does the VIM do for a "crashed" instance in order to recover it?
19:13:08 hansmoleman seems this could go upstream in some form https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R7696 - looks like it cleans up old files when a previously evacuated source host comes back online and the instances are off it now (and were being resized when the source was evacuated?)
19:16:02 hansmoleman i thought we already had something like this upstream https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R7967
19:17:03 openstackgerrit Merged openstack/nova master: [placement] Move resource_class_cache into placement hierarchy https://review.openstack.org/584085
19:19:53 cfriesen hansmoleman: for a crashed instance it just does a "stop/start" sequence I think. More generally for an ERROR instance it will cycle through gradually more extreme options (reboot, rebuild, evacuate, etc.) depending on the node state
19:20:36 hansmoleman ok and looking at the _cleanup_running_orphan_instances periodic, and the upstream ComputeManager.init_host, it looks like we don't have something for that,
19:20:55 hansmoleman b/c on compute startup we'll only work with instances still in the db and only destroy guests from the hypervisor that have been evacuated to another host
19:21:12 hansmoleman so i case in that case, the compute host went down when the user deleted the instance from the db
19:21:25 hansmoleman then the compute comes back up and the instance isn't in the db but it's consuming resources on the hypervisor
19:23:44 cfriesen hansmoleman: I think we also got running orphan instance from things like failures during migration
19:24:01 openstack Launchpad bug 1285000 in OpenStack Compute (nova) pike "instance data resides on destination node when vm is deleted during live-migration" [Medium,Fix released] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)
19:24:01 hansmoleman so https://bugs.launchpad.net/nova/+bug/1285000
19:26:04 cfriesen that'd be one possibility. you could also get something like what mdbooth commented on where a live migration never runs "post live migration at destination" and the system gets into a weird state
19:28:56 cfriesen the orphan audit dates from havana, when things were not quite as robust as they are now
19:30:01 hansmoleman ack
19:39:20 hansmoleman cfriesen: interesting https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-afb9c0c0ca5276c7eacd987bbf51d8e6R447
19:39:48 hansmoleman does upstream retrieve the volume_image_metadata properly for volume-backed scheduling with things like the NUMATopologyFilter?
19:41:53 hansmoleman looks like we should, compute API _get_bdm_image_metadata
19:41:59 hansmoleman gets the volume image metadata from the volume
19:53:02 openstack Launchpad bug 1785318 in OpenStack Compute (nova) "evacuate rebuild claim will not use any image_meta for volume-backed instances" [Medium,Triaged]
19:53:02 hansmoleman https://bugs.launchpad.net/nova/+bug/1785318
20:05:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old check_attach version check in API https://review.openstack.org/588348
20:09:43 hansmoleman fried_rice: ^ updated the commit message on that for the del instance.id thing in the tests
20:09:53 hansmoleman also updated some really really old comment
20:14:31 cfriesen hansmoleman: thanks for opening that bug report. I believe we have evacuate working reliably with NumaTopology.
20:14:42 hansmoleman yeah i don't see why it wouldn't
20:14:49 hansmoleman i was thinking of the isolated hosts filter,
20:14:58 hansmoleman which relies on the image id which we don't have in the request spec for bfv
20:15:33 hansmoleman https://review.openstack.org/#/c/543263/
20:15:37 cfriesen hansmoleman: there will likely be other cases where we fixed something and then it's been fixed upstream and we ported the change without retesting against upstream first.
20:15:54 hansmoleman ^ still not sure *why* or if it was intentional that we don't store the RequestSpec.image.id for bfv instances
20:16:22 hansmoleman we also just don't really test the isolated hosts filter
20:48:21 hansmoleman cfriesen: i'll have a patch up for that shortly, slightly different than what's in starlingx
20:59:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix image-defined numa claims during evacuate https://review.openstack.org/588657
21:22:25 openstackgerrit Eric Fried proposed openstack/nova master: Remove redundant _update()s https://review.openstack.org/588091
21:29:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Optimize AZ lookup during schedule_and_build_instances https://review.openstack.org/588665
21:37:05 hansmoleman dansmith: looks like another place for the long rpc timeout https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-2a50b2dbeb123b515ebb4b917ae1cb2bR751
21:47:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Use CONF.long_rpc_timeout in post_live_migration_at_destination https://review.openstack.org/588668
21:50:53 hansmoleman cfriesen: i thought the api change to return the server group that each server is in was kind of interesting,
21:51:05 hansmoleman but that kind of sucks for performance if you're listing servers with details for 1000 servers
21:51:11 hansmoleman since it's an api db query per instance
21:51:32 hansmoleman alternatively that could be done by storing the instance group info in instance_extra with the instance,
21:51:39 hansmoleman or adding a member filter to GET /os-server-groups
21:51:52 hansmoleman so you could just get server groups by a given member server
21:52:14 hansmoleman that list would always return at most 1 since a server can't be in more than one group
21:55:35 cfriesen hansmoleman: that implementation was driven partly by trying to make it as easy to port as possible.
21:56:12 hansmoleman yeah i get that
21:56:22 hansmoleman i think the server group is only returned if the wrs-header is present too...
21:56:26 cfriesen yes]
21:57:24 hansmoleman was also thinking we could do like a GET /servers/{id}/group but that is pretty boring, it would just return the server group subresource
21:57:46 hansmoleman anywho
21:59:08 cfriesen heading out?
21:59:10 openstack Launchpad bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,Triaged]
21:59:10 hansmoleman jgwentworth: on https://bugs.launchpad.net/nova/+bug/1781286 i just remembered,
21:59:15 hansmoleman cfriesen: of course not
21:59:38 hansmoleman jgwentworth: one way i thought about fixing that was adding a different migrate_server (or whatever it's called) method in conductor that's not trying to target a cell,
21:59:59 hansmoleman because that's how build_resources works, it's not using the @targets_cell decorator because it's called from the api for scheduling and from the compute for reschedules
22:00:10 hansmoleman so really we should do similar for rescheduling a resize/cold migrate
22:00:22 cfriesen hansmoleman: as a heads-up, I'm on vacation next week. you'll have to save up your questions. :)
22:00:31 hansmoleman cfriesen: damn
22:00:40 hansmoleman i'm leaving for china on friday
22:00:48 cfriesen you can try asking Dean
22:01:19 hansmoleman given i'm gotten through the api extension and nova/compute/* changes so far, i think i'm pretty good
22:01:30 hansmoleman i expect some scheduler stuff but mostly linked to what i've already seen
22:01:37 hansmoleman *i've
22:01:55 cfriesen there are some changes around the server group affinity to close some races in the scheduler
22:02:49 hansmoleman jgwentworth: reason i bring that up now, is if we fixed the bug that way, it's a new rpc api method which isn't backportable,
22:02:57 hansmoleman so we'd have to get that done before rc if we want it in rocky
22:04:49 hansmoleman i also thought about putting the az per host on the Selection object that we pass down for alternates,
22:04:58 hansmoleman but that's an rcp api version bump on the Selection object, so again, not backportable
22:06:36 hansmoleman ew it's actually 2 separate but very similar bugs and we'd need both

Earlier   Later