| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-03 | |||
| 15:36:45 | cfriesen | gotta step out for a bit....back later. | |
| 15:38:03 | mriedem | cfriesen_afk: i see you guys removed the force flag for live migrate so you can't do that, which means you'd go through the scheduler, but that *doesnt* apply to live migrations before the microversion that added the force flag because in those cases, simply specifying a host bypasses the scheduler | |
| 15:38:46 | mriedem | RUN_ON_REBUILD is only b/c we don't actually move hosts on rebuild | |
| 15:41:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate upgrade_levels options for deprecated/removed services https://review.openstack.org/588607 | |
| 15:41:52 | mriedem | dansmith: might want to get that into rocky ^ | |
| 15:41:54 | mriedem | to start the timer | |
| 15:41:59 | mriedem | melwitt: ^ you too given nova-consoleauth | |
| 15:43:51 | melwitt | ack | |
| 15:46:18 | dansmith | yar | |
| 15:49:55 | melwitt | mriedem: TYPO | |
| 15:50:03 | melwitt | in the reno | |
| 15:50:24 | dansmith | WUT? NO. | |
| 15:50:29 | melwitt | YUH HUH | |
| 15:50:31 | dansmith | zomg | |
| 15:50:43 | dansmith | let it be known on the third day of august, the year of our lord 2018... | |
| 15:53:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate upgrade_levels options for deprecated/removed services https://review.openstack.org/588607 | |
| 15:53:59 | mriedem | dansmith: you want to hit jichen's change below that also? | |
| 15:55:25 | dansmith | meh | |
| 15:56:13 | mriedem | i thought it was useful b/c at least one person thought "auto" could be applied to all | |
| 15:56:47 | dansmith | meh | |
| 15:57:10 | mriedem | MEH?! | |
| 15:57:16 | dansmith | MEH. | |
| 15:57:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Reno for notification-transformation-rocky https://review.openstack.org/588403 | |
| 15:57:24 | mriedem | so i guess +r has killed casual nick friday? | |
| 15:57:47 | dansmith | no | |
| 15:57:48 | dansmith | it has killed other things, but not casual friday | |
| 16:03:36 | leakypipes | sean-k-mooney: I'm good with https://review.openstack.org/#/c/587378 going into os-vif. I will leave it to the submitter to try and get a stable branch of nova to bring it in via requirements.txt though (0.00001% chance of that happening) | |
| 16:10:08 | fried_rice | finucannot: If [testenv] defines `commands` and [myenv] defines `commands`, does [myenv] actually execute testenv.commands + myenv.commands?? | |
| 16:10:47 | finucannot | fried_rice: No, [myenv] commands will override [testenv] commands | |
| 16:11:05 | fried_rice | finucannot: So that's what I thought, but snot the behavior I'm seeing :( | |
| 16:11:18 | finucannot | Got a paste? | |
| 16:12:25 | hansmoleman | cfriesen_afk: found it https://review.openstack.org/#/c/401009/ | |
| 16:13:22 | fried_rice | finucannot: agh, ignore me, pebcak | |
| 16:15:30 | fried_rice | jgwentworth: How did you manage that? Got password? | |
| 16:15:51 | jgwentworth | yeah | |
| 16:29:00 | cdent | have I got names right? hansmoleman is matt, jgwentworth is mel? | |
| 16:29:10 | jgwentworth | correct | |
| 16:40:12 | leakypipes | jgwentworth: weird... why do we have stable versions of dependent libraries when we follow a semver release model? | |
| 16:44:33 | mdbooth | dansmith: For interest, the fun LM bug we discussed earlier was sorta filed before https://bugs.launchpad.net/nova/+bug/1628606 . I don't think we appreciated the potential for the thing ending up running in 2 places at once, though. | |
| 16:44:33 | openstack | Launchpad bug 1628606 in OpenStack Compute (nova) "live migration does not clean up at target node if a failure occurs during post migration" [Low,Confirmed] | |
| 16:45:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused flavor_delete_info() method https://review.openstack.org/588621 | |
| 16:50:32 | leakypipes | jgwentworth: I suppose so we can backport stuff... | |
| 16:50:44 | leakypipes | (shows you how much I keep up with stable stuff...) | |
| 16:53:28 | mdbooth | I just added a comment to https://bugs.launchpad.net/nova/+bug/1628606 . I think it's pretty serious. | |
| 16:53:29 | openstack | Launchpad bug 1628606 in OpenStack Compute (nova) "live migration does not clean up at target node if a failure occurs during post migration" [Low,Confirmed] | |
| 16:53:52 | mdbooth | I can't update the importance, though, not that it matters all that much I guess. | |
| 16:56:43 | openstackgerrit | Lee Yarwood proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow https://review.openstack.org/587013 | |
| 16:56:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: Add regression test for bug#1784353 https://review.openstack.org/587014 | |
| 16:56:46 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Recreate volume attachments during a reschedule https://review.openstack.org/587071 | |
| 17:03:41 | hansmoleman | lyarwood: did you talk to dansmith at all about re-creating the volume attachment record in conductor vs compute when rescheduling? ^ | |
| 17:04:05 | hansmoleman | https://review.openstack.org/#/c/587071/3/nova/compute/manager.py@1611 | |
| 17:05:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix host validity check for live-migration https://review.openstack.org/401009 | |
| 17:07:09 | jgwentworth | leakypipes: yeah, I don't know the nitty gritty on how or why it works but yeah, if we backport fixes to stable and release a new lib version from stable, people can receive fixes via the upper-constraints requirement bump if they're on an older release. I guess maybe it's more an artifact of how the deployment tools usually work? not sure | |
| 17:07:35 | lyarwood | hansmoleman: I did not, just had 30 mins to work on this now at the end of the day | |
| 17:09:03 | jgwentworth | mdbooth: will take a look. fyi to update importance of a bug, all you have to do is join the bugs team (open team) https://launchpad.net/~nova-bugs | |
| 17:09:25 | jgwentworth | being a team member gives permission to update importance | |
| 17:10:35 | jgwentworth | bug from 2016, still a thing? yeesh | |
| 17:25:32 | hansmoleman | cfriesen_afk: it's interesting you added this _await_volume_detached thing in compute https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R2649 - i thought os-detach in cinder was a synchronous operation, so what issues were you hitting that required that waiter? | |
| 17:26:39 | hansmoleman | unless maybe you made detach async on the cinder side...? | |
| 17:27:54 | cfriesen | hansmoleman: Isn't https://review.openstack.org/#/c/401009/ just about checking whether the specified host exists, not whether it satisfies filters? | |
| 17:29:00 | hansmoleman | cfriesen: correct, but it does it before changing the instance task_state to 'migrating' | |
| 17:29:22 | hansmoleman | as opposed to where it was before https://review.openstack.org/#/c/401009/13/nova/compute/api.py@a4369 | |
| 17:29:33 | hansmoleman | if we failed at ^ we'd leave the instance task_state stuck in 'migrating' | |
| 17:29:54 | hansmoleman | the scheduler (or conductor in the case of force) if the specific host is valid | |
| 17:30:37 | hansmoleman | seems like this should go upstream https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-77f9348ab09642ba46409b6828af4af0R3245 | |
| 17:31:42 | hansmoleman | for https://review.openstack.org/#/c/401009/ we should really probably just have a @reverts_task_state decorator in compute api like we do in the manger | |
| 17:31:43 | hansmoleman | *manager | |
| 17:36:45 | cfriesen | hansmoleman: about the live migration stuff...we wanted to ensure that any operation involving node selection actually went through the scheduler | |
| 17:36:45 | hansmoleman | yeah so for evacuate, | |
| 17:36:47 | hansmoleman | https://github.com/openstack/nova/blob/3ac6deb94c5a07e42611158a680bd26febe79d6d/nova/compute/manager.py#L3170 | |
| 17:37:04 | hansmoleman | cfriesen: yeah i realize - that's not what that bug fix above is about | |
| 17:37:45 | hansmoleman | for evacuate we'll update the port's binding profile to set the migrating_to attr (which is really supposed to only be for live migration with DVR i think...), | |
| 17:39:32 | hansmoleman | heh and then we'll wipe that out in _update_port_binding_for_instance immediately after | |
| 17:40:02 | hansmoleman | yeah so it probably makes more sense to get the refreshed nw info during evacuate since we update the port binding for the dest host right before | |
| 17:40:18 | hansmoleman | https://github.com/openstack/nova/blob/3ac6deb94c5a07e42611158a680bd26febe79d6d/nova/compute/manager.py#L3182 | |
| 17:40:22 | cfriesen | the _await_volume_detached() thing was actually grabbed from https://bugs.launchpad.net/nova/+bug/1527623 | |
| 17:40:22 | openstack | Launchpad bug 1527623 in OpenStack Compute (nova) "Nova might orphan volumes when it's racing to delete a volume-backed instance" [Medium,In progress] - Assigned to ChangBo Guo(gcb) (glongwave) | |
| 17:43:06 | hansmoleman | i think that might be old / bogus now, | |
| 17:43:19 | hansmoleman | seeing that in CI logs now, it's hitting on TestVolumeBootPattern, | |
| 17:43:32 | hansmoleman | and i think the race is that tempest doesn't wait to cleanup the volume snapshots first | |
| 17:43:35 | hansmoleman | before deleting theserver | |
| 17:43:39 | hansmoleman | i thought jgwentworth had a patch for that | |
| 17:43:57 | hansmoleman | looking at the cinder API, os-detach is synchronous, it's an rpc call from volume api to volume manager | |
| 17:44:16 | jgwentworth | sounds familiar. let me check | |
| 17:45:02 | jgwentworth | the one I'm thinking of is this, not sure if that's the same thing you're talking about https://review.openstack.org/571336 | |
| 17:45:27 | jgwentworth | no, mine is about test_volume_backup | |
| 17:46:13 | jgwentworth | I had a different one that got merged, let me find that | |
| 17:47:04 | cfriesen | hansmoleman: did cinder change os-detach from async to sync? your bug report said it was async. | |
| 17:47:21 | hansmoleman | cfriesen: not sure, but that might have been faulty triage by me at the time, not sure | |
| 17:47:25 | hansmoleman | it's an old bug | |
| 17:47:35 | hansmoleman | i might have assumed it was async b/c it's async in nova | |
| 17:47:40 | hansmoleman | but lots of the cinder api is synchronous | |
| 17:48:38 | cfriesen | I'm going to have to go through irc history and start opening starlingx storyboard bugs. :) | |
| 17:49:10 | jgwentworth | this one https://review.openstack.org/#/c/565601/8/tempest/scenario/test_volume_boot_pattern.py but I didn't change anything about deletion of the server. because for rbd, you have to delete the server first before you delete the volume, the server booted from the volume is dependent on the volume | |
| 17:49:49 | hansmoleman | ugh | |
| 17:49:52 | jgwentworth | is it backward for non-rbd perhaps? | |
| 17:50:15 | hansmoleman | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20delete%20volume%5C%22%20AND%20message%3A%5C%22due%20to%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d shows up in non-ceph jobs though | |
| 17:50:58 | jgwentworth | I see. my brain is getting confused trying to think of how the dependency works for non-ceph, but it might be the case that the order has to be different depending on ceph vs non-ceph | |
| 17:51:36 | jgwentworth | I can't remember it now but I thought when I tried to delete the volume snapshot first, the test failed | |
| 17:52:09 | hansmoleman | yes you said that in there, | |