Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-03
17:43:19 hansmoleman seeing that in CI logs now, it's hitting on TestVolumeBootPattern,
17:43:32 hansmoleman and i think the race is that tempest doesn't wait to cleanup the volume snapshots first
17:43:35 hansmoleman before deleting theserver
17:43:39 hansmoleman i thought jgwentworth had a patch for that
17:43:57 hansmoleman looking at the cinder API, os-detach is synchronous, it's an rpc call from volume api to volume manager
17:44:16 jgwentworth sounds familiar. let me check
17:45:02 jgwentworth the one I'm thinking of is this, not sure if that's the same thing you're talking about https://review.openstack.org/571336
17:45:27 jgwentworth no, mine is about test_volume_backup
17:46:13 jgwentworth I had a different one that got merged, let me find that
17:47:04 cfriesen hansmoleman: did cinder change os-detach from async to sync? your bug report said it was async.
17:47:21 hansmoleman cfriesen: not sure, but that might have been faulty triage by me at the time, not sure
17:47:25 hansmoleman it's an old bug
17:47:35 hansmoleman i might have assumed it was async b/c it's async in nova
17:47:40 hansmoleman but lots of the cinder api is synchronous
17:48:38 cfriesen I'm going to have to go through irc history and start opening starlingx storyboard bugs. :)
17:49:10 jgwentworth this one https://review.openstack.org/#/c/565601/8/tempest/scenario/test_volume_boot_pattern.py but I didn't change anything about deletion of the server. because for rbd, you have to delete the server first before you delete the volume, the server booted from the volume is dependent on the volume
17:49:49 hansmoleman ugh
17:49:52 jgwentworth is it backward for non-rbd perhaps?
17:50:15 hansmoleman http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20delete%20volume%5C%22%20AND%20message%3A%5C%22due%20to%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d shows up in non-ceph jobs though
17:50:58 jgwentworth I see. my brain is getting confused trying to think of how the dependency works for non-ceph, but it might be the case that the order has to be different depending on ceph vs non-ceph
17:51:36 jgwentworth I can't remember it now but I thought when I tried to delete the volume snapshot first, the test failed
17:52:09 hansmoleman yes you said that in there,
17:52:10 hansmoleman "Hm, this actually makes the previously passing ceph job fail. And thinking about it, this is backwards -- we shouldn't delete the volume snapshot first if the created volume depends on it. The created volume should be deleted first, and then that would allow the volume snapshot to be deleted (in the case of ceph)."
17:52:16 jgwentworth ah, only failed for ceph
17:52:43 jgwentworth okay, so do we need different behavior depending on whether it's ceph?
17:53:25 hansmoleman the issue is that we have to delete the volume snapshot before attempting to delete the volume, and nova-compute deletes the volume,
17:53:39 hansmoleman was there an issue with trying to just delete the volume snapshot before the server with ceph?
17:54:07 hansmoleman https://review.openstack.org/#/c/565601/5 is what i was +1 on
17:54:36 hansmoleman and you said that made ceph fail?
17:54:38 jgwentworth yes, it failed every time with ceph. so I thought maybe the comment was backward
17:54:43 hansmoleman ceph is backward
17:54:58 hansmoleman the cinder api shouldn't really behave differently depending on the volume type
17:55:19 hansmoleman otherwise client side tooling would always need a "if volume_type=='rbd'" condition
17:55:22 jgwentworth it has something to do with the references in ceph. trying to see what it was exactly
17:55:29 jgwentworth yeah
17:57:11 hansmoleman idk how you delete a ceph volume that has snapshots then,
17:57:18 hansmoleman because if you can't remove the snapshots until the volume is gone,
17:57:25 jgwentworth you have to delete the servers that reference it first, I think
17:57:25 hansmoleman and you can't delete the volume while it has snapshots,
17:57:26 hansmoleman then wtf
17:57:30 hansmoleman ugh
17:57:54 hansmoleman so for rbd volume-backed servers with snapshots, you just always orphan the volumes?
17:58:40 jgwentworth well, in the case of the test, what happens is the server is deleted and then nova deletes the volume snapshot after
17:59:12 hansmoleman nova doesn't delete volume snapshots
17:59:24 hansmoleman nova-compute will attempt to delete the volume because bdm.delete_on_termination,
17:59:25 jgwentworth I mean, the virt domain gets destroyed and then compute deletes the underlying volume if delete_on_termination=True
17:59:30 hansmoleman which fails if the volume has snapshots
18:01:47 jgwentworth okay, I think I'm lacking on knowledge of volume snapshots. let me dig into how this works (non-ceph vs ceph) and propose a change to put the comment back and explain the ceph part
18:02:48 jgwentworth I didn't realize they were different and thought I had made a mistake with trying to delete the snapshot first because that change failed the ceph job, whereas leaving it alone had both jobs passing
18:49:14 openstackgerrit Merged openstack/nova master: Reno for notification-transformation-rocky https://review.openstack.org/588403
18:55:33 openstackgerrit Chris Dent proposed openstack/nova master: [placement] ensure_rc_cache only at start of process https://review.openstack.org/584086
19:07:48 hansmoleman cfriesen: what does the VIM do for a "crashed" instance in order to recover it?
19:13:08 hansmoleman seems this could go upstream in some form https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R7696 - looks like it cleans up old files when a previously evacuated source host comes back online and the instances are off it now (and were being resized when the source was evacuated?)
19:16:02 hansmoleman i thought we already had something like this upstream https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R7967
19:17:03 openstackgerrit Merged openstack/nova master: [placement] Move resource_class_cache into placement hierarchy https://review.openstack.org/584085
19:19:53 cfriesen hansmoleman: for a crashed instance it just does a "stop/start" sequence I think. More generally for an ERROR instance it will cycle through gradually more extreme options (reboot, rebuild, evacuate, etc.) depending on the node state
19:20:36 hansmoleman ok and looking at the _cleanup_running_orphan_instances periodic, and the upstream ComputeManager.init_host, it looks like we don't have something for that,
19:20:55 hansmoleman b/c on compute startup we'll only work with instances still in the db and only destroy guests from the hypervisor that have been evacuated to another host
19:21:12 hansmoleman so i case in that case, the compute host went down when the user deleted the instance from the db
19:21:25 hansmoleman then the compute comes back up and the instance isn't in the db but it's consuming resources on the hypervisor
19:23:44 cfriesen hansmoleman: I think we also got running orphan instance from things like failures during migration
19:24:01 openstack Launchpad bug 1285000 in OpenStack Compute (nova) pike "instance data resides on destination node when vm is deleted during live-migration" [Medium,Fix released] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)
19:24:01 hansmoleman so https://bugs.launchpad.net/nova/+bug/1285000
19:26:04 cfriesen that'd be one possibility. you could also get something like what mdbooth commented on where a live migration never runs "post live migration at destination" and the system gets into a weird state
19:28:56 cfriesen the orphan audit dates from havana, when things were not quite as robust as they are now
19:30:01 hansmoleman ack
19:39:20 hansmoleman cfriesen: interesting https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-afb9c0c0ca5276c7eacd987bbf51d8e6R447
19:39:48 hansmoleman does upstream retrieve the volume_image_metadata properly for volume-backed scheduling with things like the NUMATopologyFilter?
19:41:53 hansmoleman looks like we should, compute API _get_bdm_image_metadata
19:41:59 hansmoleman gets the volume image metadata from the volume
19:53:02 openstack Launchpad bug 1785318 in OpenStack Compute (nova) "evacuate rebuild claim will not use any image_meta for volume-backed instances" [Medium,Triaged]
19:53:02 hansmoleman https://bugs.launchpad.net/nova/+bug/1785318
20:05:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old check_attach version check in API https://review.openstack.org/588348
20:09:43 hansmoleman fried_rice: ^ updated the commit message on that for the del instance.id thing in the tests
20:09:53 hansmoleman also updated some really really old comment
20:14:31 cfriesen hansmoleman: thanks for opening that bug report. I believe we have evacuate working reliably with NumaTopology.
20:14:42 hansmoleman yeah i don't see why it wouldn't
20:14:49 hansmoleman i was thinking of the isolated hosts filter,
20:14:58 hansmoleman which relies on the image id which we don't have in the request spec for bfv
20:15:33 hansmoleman https://review.openstack.org/#/c/543263/
20:15:37 cfriesen hansmoleman: there will likely be other cases where we fixed something and then it's been fixed upstream and we ported the change without retesting against upstream first.
20:15:54 hansmoleman ^ still not sure *why* or if it was intentional that we don't store the RequestSpec.image.id for bfv instances
20:16:22 hansmoleman we also just don't really test the isolated hosts filter
20:48:21 hansmoleman cfriesen: i'll have a patch up for that shortly, slightly different than what's in starlingx
20:59:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix image-defined numa claims during evacuate https://review.openstack.org/588657
21:22:25 openstackgerrit Eric Fried proposed openstack/nova master: Remove redundant _update()s https://review.openstack.org/588091
21:29:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Optimize AZ lookup during schedule_and_build_instances https://review.openstack.org/588665
21:37:05 hansmoleman dansmith: looks like another place for the long rpc timeout https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-2a50b2dbeb123b515ebb4b917ae1cb2bR751
21:47:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Use CONF.long_rpc_timeout in post_live_migration_at_destination https://review.openstack.org/588668
21:50:53 hansmoleman cfriesen: i thought the api change to return the server group that each server is in was kind of interesting,
21:51:05 hansmoleman but that kind of sucks for performance if you're listing servers with details for 1000 servers
21:51:11 hansmoleman since it's an api db query per instance
21:51:32 hansmoleman alternatively that could be done by storing the instance group info in instance_extra with the instance,
21:51:39 hansmoleman or adding a member filter to GET /os-server-groups
21:51:52 hansmoleman so you could just get server groups by a given member server
21:52:14 hansmoleman that list would always return at most 1 since a server can't be in more than one group
21:55:35 cfriesen hansmoleman: that implementation was driven partly by trying to make it as easy to port as possible.
21:56:12 hansmoleman yeah i get that
21:56:22 hansmoleman i think the server group is only returned if the wrs-header is present too...
21:56:26 cfriesen yes]

Earlier   Later