| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-03 | |||
| 16:29:00 | cdent | have I got names right? hansmoleman is matt, jgwentworth is mel? | |
| 16:29:10 | jgwentworth | correct | |
| 16:40:12 | leakypipes | jgwentworth: weird... why do we have stable versions of dependent libraries when we follow a semver release model? | |
| 16:44:33 | openstack | Launchpad bug 1628606 in OpenStack Compute (nova) "live migration does not clean up at target node if a failure occurs during post migration" [Low,Confirmed] | |
| 16:44:33 | mdbooth | dansmith: For interest, the fun LM bug we discussed earlier was sorta filed before https://bugs.launchpad.net/nova/+bug/1628606 . I don't think we appreciated the potential for the thing ending up running in 2 places at once, though. | |
| 16:45:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused flavor_delete_info() method https://review.openstack.org/588621 | |
| 16:50:32 | leakypipes | jgwentworth: I suppose so we can backport stuff... | |
| 16:50:44 | leakypipes | (shows you how much I keep up with stable stuff...) | |
| 16:53:28 | mdbooth | I just added a comment to https://bugs.launchpad.net/nova/+bug/1628606 . I think it's pretty serious. | |
| 16:53:29 | openstack | Launchpad bug 1628606 in OpenStack Compute (nova) "live migration does not clean up at target node if a failure occurs during post migration" [Low,Confirmed] | |
| 16:53:52 | mdbooth | I can't update the importance, though, not that it matters all that much I guess. | |
| 16:56:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013 | |
| 16:56:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014 | |
| 16:56:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Recreate volume attachments during a reschedule https://review.openstack.org/587071 | |
| 17:03:41 | hansmoleman | lyarwood: did you talk to dansmith at all about re-creating the volume attachment record in conductor vs compute when rescheduling? ^ | |
| 17:04:05 | hansmoleman | https://review.openstack.org/#/c/587071/3/nova/compute/manager.py@1611 | |
| 17:05:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix host validity check for live-migration https://review.openstack.org/401009 | |
| 17:07:09 | jgwentworth | leakypipes: yeah, I don't know the nitty gritty on how or why it works but yeah, if we backport fixes to stable and release a new lib version from stable, people can receive fixes via the upper-constraints requirement bump if they're on an older release. I guess maybe it's more an artifact of how the deployment tools usually work? not sure | |
| 17:07:35 | lyarwood | hansmoleman: I did not, just had 30 mins to work on this now at the end of the day | |
| 17:09:03 | jgwentworth | mdbooth: will take a look. fyi to update importance of a bug, all you have to do is join the bugs team (open team) https://launchpad.net/~nova-bugs | |
| 17:09:25 | jgwentworth | being a team member gives permission to update importance | |
| 17:10:35 | jgwentworth | bug from 2016, still a thing? yeesh | |
| 17:25:32 | hansmoleman | cfriesen_afk: it's interesting you added this _await_volume_detached thing in compute https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R2649 - i thought os-detach in cinder was a synchronous operation, so what issues were you hitting that required that waiter? | |
| 17:26:39 | hansmoleman | unless maybe you made detach async on the cinder side...? | |
| 17:27:54 | cfriesen | hansmoleman: Isn't https://review.openstack.org/#/c/401009/ just about checking whether the specified host exists, not whether it satisfies filters? | |
| 17:29:00 | hansmoleman | cfriesen: correct, but it does it before changing the instance task_state to 'migrating' | |
| 17:29:22 | hansmoleman | as opposed to where it was before https://review.openstack.org/#/c/401009/13/nova/compute/api.py@a4369 | |
| 17:29:33 | hansmoleman | if we failed at ^ we'd leave the instance task_state stuck in 'migrating' | |
| 17:29:54 | hansmoleman | the scheduler (or conductor in the case of force) if the specific host is valid | |
| 17:30:37 | hansmoleman | seems like this should go upstream https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R3245 | |
| 17:31:42 | hansmoleman | for https://review.openstack.org/#/c/401009/ we should really probably just have a @reverts_task_state decorator in compute api like we do in the manger | |
| 17:31:43 | hansmoleman | *manager | |
| 17:36:45 | hansmoleman | yeah so for evacuate, | |
| 17:36:45 | cfriesen | hansmoleman: about the live migration stuff...we wanted to ensure that any operation involving node selection actually went through the scheduler | |
| 17:36:47 | hansmoleman | https://github.com/openstack/nova/blob/3ac6deb94c5a07e42611158a680bd26febe79d6d/nova/compute/manager.py#L3170 | |
| 17:37:04 | hansmoleman | cfriesen: yeah i realize - that's not what that bug fix above is about | |
| 17:37:45 | hansmoleman | for evacuate we'll update the port's binding profile to set the migrating_to attr (which is really supposed to only be for live migration with DVR i think...), | |
| 17:39:32 | hansmoleman | heh and then we'll wipe that out in _update_port_binding_for_instance immediately after | |
| 17:40:02 | hansmoleman | yeah so it probably makes more sense to get the refreshed nw info during evacuate since we update the port binding for the dest host right before | |
| 17:40:18 | hansmoleman | https://github.com/openstack/nova/blob/3ac6deb94c5a07e42611158a680bd26febe79d6d/nova/compute/manager.py#L3182 | |
| 17:40:22 | openstack | Launchpad bug 1527623 in OpenStack Compute (nova) "Nova might orphan volumes when it's racing to delete a volume-backed instance" [Medium,In progress] - Assigned to ChangBo Guo(gcb) (glongwave) | |
| 17:40:22 | cfriesen | the _await_volume_detached() thing was actually grabbed from https://bugs.launchpad.net/nova/+bug/1527623 | |
| 17:43:06 | hansmoleman | i think that might be old / bogus now, | |
| 17:43:19 | hansmoleman | seeing that in CI logs now, it's hitting on TestVolumeBootPattern, | |
| 17:43:32 | hansmoleman | and i think the race is that tempest doesn't wait to cleanup the volume snapshots first | |
| 17:43:35 | hansmoleman | before deleting theserver | |
| 17:43:39 | hansmoleman | i thought jgwentworth had a patch for that | |
| 17:43:57 | hansmoleman | looking at the cinder API, os-detach is synchronous, it's an rpc call from volume api to volume manager | |
| 17:44:16 | jgwentworth | sounds familiar. let me check | |
| 17:45:02 | jgwentworth | the one I'm thinking of is this, not sure if that's the same thing you're talking about https://review.openstack.org/571336 | |
| 17:45:27 | jgwentworth | no, mine is about test_volume_backup | |
| 17:46:13 | jgwentworth | I had a different one that got merged, let me find that | |
| 17:47:04 | cfriesen | hansmoleman: did cinder change os-detach from async to sync? your bug report said it was async. | |
| 17:47:21 | hansmoleman | cfriesen: not sure, but that might have been faulty triage by me at the time, not sure | |
| 17:47:25 | hansmoleman | it's an old bug | |
| 17:47:35 | hansmoleman | i might have assumed it was async b/c it's async in nova | |
| 17:47:40 | hansmoleman | but lots of the cinder api is synchronous | |
| 17:48:38 | cfriesen | I'm going to have to go through irc history and start opening starlingx storyboard bugs. :) | |
| 17:49:10 | jgwentworth | this one https://review.openstack.org/#/c/565601/8/tempest/scenario/test_volume_boot_pattern.py but I didn't change anything about deletion of the server. because for rbd, you have to delete the server first before you delete the volume, the server booted from the volume is dependent on the volume | |
| 17:49:49 | hansmoleman | ugh | |
| 17:49:52 | jgwentworth | is it backward for non-rbd perhaps? | |
| 17:50:15 | hansmoleman | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20delete%20volume%5C%22%20AND%20message%3A%5C%22due%20to%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d shows up in non-ceph jobs though | |
| 17:50:58 | jgwentworth | I see. my brain is getting confused trying to think of how the dependency works for non-ceph, but it might be the case that the order has to be different depending on ceph vs non-ceph | |
| 17:51:36 | jgwentworth | I can't remember it now but I thought when I tried to delete the volume snapshot first, the test failed | |
| 17:52:09 | hansmoleman | yes you said that in there, | |
| 17:52:10 | hansmoleman | "Hm, this actually makes the previously passing ceph job fail. And thinking about it, this is backwards -- we shouldn't delete the volume snapshot first if the created volume depends on it. The created volume should be deleted first, and then that would allow the volume snapshot to be deleted (in the case of ceph)." | |
| 17:52:16 | jgwentworth | ah, only failed for ceph | |
| 17:52:43 | jgwentworth | okay, so do we need different behavior depending on whether it's ceph? | |
| 17:53:25 | hansmoleman | the issue is that we have to delete the volume snapshot before attempting to delete the volume, and nova-compute deletes the volume, | |
| 17:53:39 | hansmoleman | was there an issue with trying to just delete the volume snapshot before the server with ceph? | |
| 17:54:07 | hansmoleman | https://review.openstack.org/#/c/565601/5 is what i was +1 on | |
| 17:54:36 | hansmoleman | and you said that made ceph fail? | |
| 17:54:38 | jgwentworth | yes, it failed every time with ceph. so I thought maybe the comment was backward | |
| 17:54:43 | hansmoleman | ceph is backward | |
| 17:54:58 | hansmoleman | the cinder api shouldn't really behave differently depending on the volume type | |
| 17:55:19 | hansmoleman | otherwise client side tooling would always need a "if volume_type=='rbd'" condition | |
| 17:55:22 | jgwentworth | it has something to do with the references in ceph. trying to see what it was exactly | |
| 17:55:29 | jgwentworth | yeah | |
| 17:57:11 | hansmoleman | idk how you delete a ceph volume that has snapshots then, | |
| 17:57:18 | hansmoleman | because if you can't remove the snapshots until the volume is gone, | |
| 17:57:25 | jgwentworth | you have to delete the servers that reference it first, I think | |
| 17:57:25 | hansmoleman | and you can't delete the volume while it has snapshots, | |
| 17:57:26 | hansmoleman | then wtf | |
| 17:57:30 | hansmoleman | ugh | |
| 17:57:54 | hansmoleman | so for rbd volume-backed servers with snapshots, you just always orphan the volumes? | |
| 17:58:40 | jgwentworth | well, in the case of the test, what happens is the server is deleted and then nova deletes the volume snapshot after | |
| 17:59:12 | hansmoleman | nova doesn't delete volume snapshots | |
| 17:59:24 | hansmoleman | nova-compute will attempt to delete the volume because bdm.delete_on_termination, | |
| 17:59:25 | jgwentworth | I mean, the virt domain gets destroyed and then compute deletes the underlying volume if delete_on_termination=True | |
| 17:59:30 | hansmoleman | which fails if the volume has snapshots | |
| 18:01:47 | jgwentworth | okay, I think I'm lacking on knowledge of volume snapshots. let me dig into how this works (non-ceph vs ceph) and propose a change to put the comment back and explain the ceph part | |
| 18:02:48 | jgwentworth | I didn't realize they were different and thought I had made a mistake with trying to delete the snapshot first because that change failed the ceph job, whereas leaving it alone had both jobs passing | |
| 18:49:14 | openstackgerrit | Merged openstack/nova master: Reno for notification-transformation-rocky https://review.openstack.org/588403 | |
| 18:55:33 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] ensure_rc_cache only at start of process https://review.openstack.org/584086 | |
| 19:07:48 | hansmoleman | cfriesen: what does the VIM do for a "crashed" instance in order to recover it? | |
| 19:13:08 | hansmoleman | seems this could go upstream in some form https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R7696 - looks like it cleans up old files when a previously evacuated source host comes back online and the instances are off it now (and were being resized when the source was evacuated?) | |
| 19:16:02 | hansmoleman | i thought we already had something like this upstream https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R7967 | |
| 19:17:03 | openstackgerrit | Merged openstack/nova master: [placement] Move resource_class_cache into placement hierarchy https://review.openstack.org/584085 | |
| 19:19:53 | cfriesen | hansmoleman: for a crashed instance it just does a "stop/start" sequence I think. More generally for an ERROR instance it will cycle through gradually more extreme options (reboot, rebuild, evacuate, etc.) depending on the node state | |
| 19:20:36 | hansmoleman | ok and looking at the _cleanup_running_orphan_instances periodic, and the upstream ComputeManager.init_host, it looks like we don't have something for that, | |