| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-21 | |||
| 16:21:02 | mriedem | ah nvm i see how we get this info in nova | |
| 16:21:05 | danpb | mriedem: yeah the job type field is what we're hooking off | |
| 16:21:09 | mriedem | yup | |
| 16:21:20 | mriedem | elif info.type == libvirt.VIR_DOMAIN_JOB_FAILED: | |
| 16:22:30 | mriedem | danpb: alright thanks i think i know what to do here, | |
| 16:22:40 | mriedem | sean-k-mooney: i probably won't have something for you to test by your eod | |
| 16:22:47 | mriedem | although your eod varies wildly | |
| 16:23:01 | mriedem | but i'm in serious need of a shower and lunch at this point....i'm devolving | |
| 16:23:50 | mdbooth | mriedem danpb: IIRC we encountered limitations with this in the block rebase operation. Isn't there a race with the job info disappearing? If the job is no longer present, we no longer know if it failed or not, and the solution was supposed to be to consume events? | |
| 16:23:51 | sean-k-mooney | haha yes it does today i need to drive home which is an hour and a half away so ill be leave shortly. if you have something ill test it as soon as im back online | |
| 16:24:51 | mdbooth | Yeah, I wrote one of my comment essays about it | |
| 16:25:42 | mdbooth | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L827-L838 | |
| 16:28:06 | sean-k-mooney | mdbooth: right. am can we check if the domain is still present on the source node? if it is it would mean it failed right? | |
| 16:31:55 | danpb | mdbooth: with new enough libvirt the job will stick around | |
| 16:32:27 | danpb | mdbooth: with older libvirt the _live_Migration_monitor code has heuristic to try to figure out if no-job == failed vs success | |
| 16:33:14 | mdbooth | danpb: I got the impression at the time that eric was piling on heuristics in there for us, but really we weren't supposed to be doing that. Sounds like that's out of date? | |
| 16:34:51 | danpb | mdbooth: what do you mean ? | |
| 16:36:10 | mdbooth | There was also the heuristic for status.end. | |
| 16:36:29 | mdbooth | I just got the strong impression at the time that consuming events was the intended approach here. | |
| 16:36:49 | mdbooth | If we've sat on the problem for that to be out of date... result :) | |
| 16:38:55 | mriedem | mdbooth: if i get no job, i'm going to not send the callback event to compute manager to trigger the port binding activation, | |
| 16:39:09 | mriedem | because worst case is the job failed and we're screwing up networking, which is what sean is seeing, | |
| 16:39:21 | mriedem | best case is we don't know, but post live migration will still activate the port bindings, | |
| 16:39:30 | mriedem | you just have a bigger window of network downtime | |
| 16:39:43 | mriedem | *plus*, if the job was successful and we go into post-copy, we activate the port bindings then too | |
| 16:40:17 | mriedem | i'm fairly certain this is 100% fool proof and will forever be bug free | |
| 16:47:31 | tssurya | dansmith: would you prefer me returnng (1) the failed_cell_uuids from get_instance_objects_sorted only if cell_down_support is set ? or (2) you don't want this flag creeping down even to that level and so we just return the tuple under all conditions ? | |
| 16:47:46 | tssurya | and deal with it in the api | |
| 16:49:06 | tssurya | I am asking because its called "get_instance_objects_sorted" and returning the tuple under all conditions kind of might be weird ? | |
| 16:49:34 | dansmith | tssurya: just return it always, and let the api decide what to do with it based on the version I think | |
| 16:49:42 | tssurya | dansmith: ack | |
| 16:49:58 | dansmith | tssurya: you can change the name if you think that's important | |
| 16:50:35 | tssurya | I will put it up for review and we can see | |
| 16:51:27 | tssurya | thanks | |
| 16:51:36 | dansmith | cool | |
| 16:52:20 | sean-k-mooney | mriedem: can we get that on a tee shirt. | |
| 16:57:20 | mriedem | sean-k-mooney: my bug free guarantee? | |
| 16:57:30 | mriedem | it only applies from today through labor day | |
| 17:02:13 | dansmith | oof, 329 in check | |
| 17:02:30 | melwitt | . | |
| 17:04:24 | mriedem | so ima also mark https://bugs.launchpad.net/nova/+bug/1788014 as rc potential | |
| 17:04:24 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:04:31 | mriedem | given it's a regression when live migration fails | |
| 17:05:00 | mriedem | my only question on that one is doing a tactical fix for the GA | |
| 17:06:50 | melwitt | ok, so rc3 now | |
| 17:36:50 | openstackgerrit | Merged openstack/nova master: Update resources once in update_available_resource https://review.openstack.org/520024 | |
| 17:36:57 | openstackgerrit | Merged openstack/nova master: Set policy_opt defaults in placement gabbi fixture https://review.openstack.org/594172 | |
| 17:39:38 | openstackgerrit | Merged openstack/nova master: Set policy_opt defaults in placement deploy unit test https://review.openstack.org/594334 | |
| 17:40:38 | dansmith | mriedem: what's the plan here? https://review.openstack.org/#/c/591735/ | |
| 17:41:23 | dansmith | we would like that to be in all current upstream stable, but will backport it ourselves if we're not going to do it upstream, so I just wanna know if I should hold off or not | |
| 17:53:05 | mriedem | dansmith: i was waiting for you to rebase it | |
| 17:53:21 | dansmith | oh heh sorry | |
| 17:53:28 | mriedem | sni | |
| 17:53:30 | mriedem | snip snap | |
| 17:53:51 | mriedem | melwitt: yeah so probably rc3 | |
| 17:54:16 | mriedem | these are the 2 as of today https://bugs.launchpad.net/nova/+bugs?field.tag=rocky-rc-potential | |
| 17:54:21 | mriedem | looks like final rc is thursday | |
| 17:54:31 | mriedem | that first one has a fix in thegate | |
| 17:54:44 | mriedem | i wouldn't mind bouncing of a few of you on the 2nd one | |
| 17:54:47 | mriedem | *off a few | |
| 17:55:35 | melwitt | ack | |
| 17:58:23 | mriedem | dansmith: melwitt: so tl;dr, the issue in https://bugs.launchpad.net/nova/+bug/1788014 is that live migration fails and we trigger a lifecycle event which activates the port binding on the dest host incorrectly, it shouldn't do that, | |
| 17:58:23 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:58:37 | mriedem | we're getting an event from libvirt but don't know if it's success or failure for the job, | |
| 17:59:14 | mriedem | so what i could do if we want to be low risk with the fix for rocky GA is just not listen on that event and we'll activate port bindings on success like we always did before the change, and we'll still do the early port activating on post-copy events if the live migration is successful | |
| 17:59:35 | mriedem | long-term we could check the actual job status and if failed, don't trigger our lifecycle event, but that's riskier for rocky GA at this point IMO | |
| 18:00:08 | mriedem | so i'd propose a 2-part fix, one partial that we backport and one for just stein | |
| 18:03:57 | melwitt | ok | |
| 18:04:28 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Fix cancel_all_events event name parsing https://review.openstack.org/592086 | |
| 18:04:29 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Wait for network-vif-plugged before starting live migration https://review.openstack.org/591735 | |
| 18:04:30 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: DNM: Debug patch to test live migration waiting https://review.openstack.org/591775 | |
| 18:05:23 | melwitt | I guess I don't understand what the lifecycle event gives us if we already know success or failure | |
| 18:05:55 | melwitt | without listening for it | |
| 18:07:06 | mriedem | in rocky we started listening on 2 new events, | |
| 18:07:12 | mriedem | one is post-copy and one is migration completed | |
| 18:07:31 | mriedem | the idea is that as soon as we switch we activate the port bindings on the dest host for minimal downtime | |
| 18:07:50 | mriedem | the problem is we get the latter event even if live migration fails | |
| 18:08:01 | mriedem | and we're not doing any conditional logic in that one to see if the job failed or not | |
| 18:09:26 | melwitt | I think I understand that part, it's just when you said "we'll activate port bindings on success like we always did before the change" it makes me not understand what gain the lifecycle event was supposed to give, if we already know success or failure without it | |
| 18:11:55 | mriedem | because if we can activate the network on the dest at the point the guest is paused to complete the transfer, it makes the network downtime window shorter | |
| 18:14:16 | melwitt | and without the lifecycle event, we activate the network later on after the guest is paused | |
| 18:14:19 | mriedem | we might still need my other fix for rollback, i'm not sure; what sean is hitting isn't a failure after post-copy | |
| 18:14:32 | mriedem | we activate the network after the guest transfer is complete and resumed on the dest | |
| 18:16:49 | melwitt | ok, got it | |
| 18:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508 | |
| 18:33:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508 | |
| 19:36:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.openstack.org/594527 | |
| 19:37:36 | mriedem | melwitt: dansmith: alright fyi ^ would need sean-k-mooney to test out the 2nd more complicated fix since he has the env that recreates the bug | |
| 19:37:43 | mriedem | and i need to run to an appt | |
| 19:39:36 | mriedem | also threw that stuff in the rc todo etherpad | |
| 19:43:27 | melwitt | ack | |
| 19:54:56 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 19:54:57 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 19:54:58 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider raise https://review.openstack.org/584598 | |
| 19:54:59 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 19:55:00 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 19:55:01 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 19:55:02 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 19:55:03 | openstackgerrit | Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 20:14:31 | dansmith | so I | |