| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-21 | |||
| 16:36:29 | mdbooth | I just got the strong impression at the time that consuming events was the intended approach here. | |
| 16:36:49 | mdbooth | If we've sat on the problem for that to be out of date... result :) | |
| 16:38:55 | mriedem | mdbooth: if i get no job, i'm going to not send the callback event to compute manager to trigger the port binding activation, | |
| 16:39:09 | mriedem | because worst case is the job failed and we're screwing up networking, which is what sean is seeing, | |
| 16:39:21 | mriedem | best case is we don't know, but post live migration will still activate the port bindings, | |
| 16:39:30 | mriedem | you just have a bigger window of network downtime | |
| 16:39:43 | mriedem | *plus*, if the job was successful and we go into post-copy, we activate the port bindings then too | |
| 16:40:17 | mriedem | i'm fairly certain this is 100% fool proof and will forever be bug free | |
| 16:47:31 | tssurya | dansmith: would you prefer me returnng (1) the failed_cell_uuids from get_instance_objects_sorted only if cell_down_support is set ? or (2) you don't want this flag creeping down even to that level and so we just return the tuple under all conditions ? | |
| 16:47:46 | tssurya | and deal with it in the api | |
| 16:49:06 | tssurya | I am asking because its called "get_instance_objects_sorted" and returning the tuple under all conditions kind of might be weird ? | |
| 16:49:34 | dansmith | tssurya: just return it always, and let the api decide what to do with it based on the version I think | |
| 16:49:42 | tssurya | dansmith: ack | |
| 16:49:58 | dansmith | tssurya: you can change the name if you think that's important | |
| 16:50:35 | tssurya | I will put it up for review and we can see | |
| 16:51:27 | tssurya | thanks | |
| 16:51:36 | dansmith | cool | |
| 16:52:20 | sean-k-mooney | mriedem: can we get that on a tee shirt. | |
| 16:57:20 | mriedem | sean-k-mooney: my bug free guarantee? | |
| 16:57:30 | mriedem | it only applies from today through labor day | |
| 17:02:13 | dansmith | oof, 329 in check | |
| 17:02:30 | melwitt | . | |
| 17:04:24 | mriedem | so ima also mark https://bugs.launchpad.net/nova/+bug/1788014 as rc potential | |
| 17:04:24 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:04:31 | mriedem | given it's a regression when live migration fails | |
| 17:05:00 | mriedem | my only question on that one is doing a tactical fix for the GA | |
| 17:06:50 | melwitt | ok, so rc3 now | |
| 17:36:50 | openstackgerrit | Merged openstack/nova master: Update resources once in update_available_resource https://review.openstack.org/520024 | |
| 17:36:57 | openstackgerrit | Merged openstack/nova master: Set policy_opt defaults in placement gabbi fixture https://review.openstack.org/594172 | |
| 17:39:38 | openstackgerrit | Merged openstack/nova master: Set policy_opt defaults in placement deploy unit test https://review.openstack.org/594334 | |
| 17:40:38 | dansmith | mriedem: what's the plan here? https://review.openstack.org/#/c/591735/ | |
| 17:41:23 | dansmith | we would like that to be in all current upstream stable, but will backport it ourselves if we're not going to do it upstream, so I just wanna know if I should hold off or not | |
| 17:53:05 | mriedem | dansmith: i was waiting for you to rebase it | |
| 17:53:21 | dansmith | oh heh sorry | |
| 17:53:28 | mriedem | sni | |
| 17:53:30 | mriedem | snip snap | |
| 17:53:51 | mriedem | melwitt: yeah so probably rc3 | |
| 17:54:16 | mriedem | these are the 2 as of today https://bugs.launchpad.net/nova/+bugs?field.tag=rocky-rc-potential | |
| 17:54:21 | mriedem | looks like final rc is thursday | |
| 17:54:31 | mriedem | that first one has a fix in thegate | |
| 17:54:44 | mriedem | i wouldn't mind bouncing of a few of you on the 2nd one | |
| 17:54:47 | mriedem | *off a few | |
| 17:55:35 | melwitt | ack | |
| 17:58:23 | mriedem | dansmith: melwitt: so tl;dr, the issue in https://bugs.launchpad.net/nova/+bug/1788014 is that live migration fails and we trigger a lifecycle event which activates the port binding on the dest host incorrectly, it shouldn't do that, | |
| 17:58:23 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:58:37 | mriedem | we're getting an event from libvirt but don't know if it's success or failure for the job, | |
| 17:59:14 | mriedem | so what i could do if we want to be low risk with the fix for rocky GA is just not listen on that event and we'll activate port bindings on success like we always did before the change, and we'll still do the early port activating on post-copy events if the live migration is successful | |
| 17:59:35 | mriedem | long-term we could check the actual job status and if failed, don't trigger our lifecycle event, but that's riskier for rocky GA at this point IMO | |
| 18:00:08 | mriedem | so i'd propose a 2-part fix, one partial that we backport and one for just stein | |
| 18:03:57 | melwitt | ok | |
| 18:04:28 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Fix cancel_all_events event name parsing https://review.openstack.org/592086 | |
| 18:04:29 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Wait for network-vif-plugged before starting live migration https://review.openstack.org/591735 | |
| 18:04:30 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: DNM: Debug patch to test live migration waiting https://review.openstack.org/591775 | |
| 18:05:23 | melwitt | I guess I don't understand what the lifecycle event gives us if we already know success or failure | |
| 18:05:55 | melwitt | without listening for it | |
| 18:07:06 | mriedem | in rocky we started listening on 2 new events, | |
| 18:07:12 | mriedem | one is post-copy and one is migration completed | |
| 18:07:31 | mriedem | the idea is that as soon as we switch we activate the port bindings on the dest host for minimal downtime | |
| 18:07:50 | mriedem | the problem is we get the latter event even if live migration fails | |
| 18:08:01 | mriedem | and we're not doing any conditional logic in that one to see if the job failed or not | |
| 18:09:26 | melwitt | I think I understand that part, it's just when you said "we'll activate port bindings on success like we always did before the change" it makes me not understand what gain the lifecycle event was supposed to give, if we already know success or failure without it | |
| 18:11:55 | mriedem | because if we can activate the network on the dest at the point the guest is paused to complete the transfer, it makes the network downtime window shorter | |
| 18:14:16 | melwitt | and without the lifecycle event, we activate the network later on after the guest is paused | |
| 18:14:19 | mriedem | we might still need my other fix for rollback, i'm not sure; what sean is hitting isn't a failure after post-copy | |
| 18:14:32 | mriedem | we activate the network after the guest transfer is complete and resumed on the dest | |
| 18:16:49 | melwitt | ok, got it | |
| 18:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508 | |
| 18:33:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508 | |
| 19:36:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.openstack.org/594527 | |
| 19:37:36 | mriedem | melwitt: dansmith: alright fyi ^ would need sean-k-mooney to test out the 2nd more complicated fix since he has the env that recreates the bug | |
| 19:37:43 | mriedem | and i need to run to an appt | |
| 19:39:36 | mriedem | also threw that stuff in the rc todo etherpad | |
| 19:43:27 | melwitt | ack | |
| 19:54:56 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 19:54:57 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 19:54:58 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider raise https://review.openstack.org/584598 | |
| 19:54:59 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 19:55:00 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 19:55:01 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 19:55:02 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 19:55:03 | openstackgerrit | Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 20:14:31 | dansmith | so I | |
| 20:14:48 | dansmith | am pretty sure my batching breaks tssurya's down cell in its current form | |
| 20:15:02 | dansmith | more specifically, it'll cause scatter gather to never notice failues | |
| 20:24:16 | melwitt | dansmith: it's specific to the batching? it looks like pre-batch the code still removes the "did not respond" and "raised exception" results | |
| 20:24:29 | dansmith | still removes? | |
| 20:25:21 | melwitt | dansmith: this part looks like it's removing "down cells" from results? https://review.openstack.org/#/c/592698/5/nova/compute/multi_cell_list.py@311 | |
| 20:26:00 | dansmith | right but with the batching we never hit that because we don't start executing the queries until the heapq | |
| 20:27:10 | melwitt | oh, I see | |
| 20:27:45 | melwitt | hmm | |
| 20:29:21 | dansmith | I'll just have to get a little more into the middle of that process and we just won't get the standard handlers from scatter gather | |
| 21:13:09 | mriedem | huh, you don't see instance.save() messaging timeouts in the gate very often http://logs.openstack.org/98/591898/3/check/nova-next/2d5e60c/logs/screen-n-cpu.txt.gz#_Aug_21_17_10_17_732263 | |
| 21:14:24 | mriedem | guessing this isn't good http://logs.openstack.org/98/591898/3/check/nova-next/2d5e60c/logs/screen-n-cpu.txt.gz#_Aug_21_17_10_17_426420 | |
| 21:16:54 | mriedem | seen here too http://logs.openstack.org/85/567785/7/check/nova-tox-functional-py35/d5a8036/job-output.txt#_2018-08-17_09_06_20_241427 | |
| 21:36:54 | mriedem | melwitt: what do you think is missing from placement for shared storage providers support? as far as i know, it's the nova stuff that's lacking as we identified ~2 weeks ago | |
| 21:37:05 | mriedem | for sure move operations are not ready for shared storage providers | |
| 21:37:18 | mriedem | the placement side of shared storage is pretty simple though, and has been done for a long time | |
| 21:40:28 | openstackgerrit | Dmitry Sutyagin proposed openstack/nova-specs master: Allow disabling KSM / mem-merge via extra spec https://review.openstack.org/593197 | |
| 21:41:23 | melwitt | mriedem: it's not that I think anything is missing, I'm pragmatically thinking of the integration work and if there is something unforeseen we need to fix. I expect bugs to shake out when we integrate something for the first time. I think not having bugs shake out will be the rarer case | |
| 21:41:38 | dansmith | mriedem: I think we were saying the same about aggregates being done in placement before we added the placement filter stuff and realized we needed tweaks | |