Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-21
16:40:17 mriedem i'm fairly certain this is 100% fool proof and will forever be bug free
16:47:31 tssurya dansmith: would you prefer me returnng (1) the failed_cell_uuids from get_instance_objects_sorted only if cell_down_support is set ? or (2) you don't want this flag creeping down even to that level and so we just return the tuple under all conditions ?
16:47:46 tssurya and deal with it in the api
16:49:06 tssurya I am asking because its called "get_instance_objects_sorted" and returning the tuple under all conditions kind of might be weird ?
16:49:34 dansmith tssurya: just return it always, and let the api decide what to do with it based on the version I think
16:49:42 tssurya dansmith: ack
16:49:58 dansmith tssurya: you can change the name if you think that's important
16:50:35 tssurya I will put it up for review and we can see
16:51:27 tssurya thanks
16:51:36 dansmith cool
16:52:20 sean-k-mooney mriedem: can we get that on a tee shirt.
16:57:20 mriedem sean-k-mooney: my bug free guarantee?
16:57:30 mriedem it only applies from today through labor day
17:02:13 dansmith oof, 329 in check
17:02:30 melwitt .
17:04:24 mriedem so ima also mark https://bugs.launchpad.net/nova/+bug/1788014 as rc potential
17:04:24 openstack Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
17:04:31 mriedem given it's a regression when live migration fails
17:05:00 mriedem my only question on that one is doing a tactical fix for the GA
17:06:50 melwitt ok, so rc3 now
17:36:50 openstackgerrit Merged openstack/nova master: Update resources once in update_available_resource https://review.openstack.org/520024
17:36:57 openstackgerrit Merged openstack/nova master: Set policy_opt defaults in placement gabbi fixture https://review.openstack.org/594172
17:39:38 openstackgerrit Merged openstack/nova master: Set policy_opt defaults in placement deploy unit test https://review.openstack.org/594334
17:40:38 dansmith mriedem: what's the plan here? https://review.openstack.org/#/c/591735/
17:41:23 dansmith we would like that to be in all current upstream stable, but will backport it ourselves if we're not going to do it upstream, so I just wanna know if I should hold off or not
17:53:05 mriedem dansmith: i was waiting for you to rebase it
17:53:21 dansmith oh heh sorry
17:53:28 mriedem sni
17:53:30 mriedem snip snap
17:53:51 mriedem melwitt: yeah so probably rc3
17:54:16 mriedem these are the 2 as of today https://bugs.launchpad.net/nova/+bugs?field.tag=rocky-rc-potential
17:54:21 mriedem looks like final rc is thursday
17:54:31 mriedem that first one has a fix in thegate
17:54:44 mriedem i wouldn't mind bouncing of a few of you on the 2nd one
17:54:47 mriedem *off a few
17:55:35 melwitt ack
17:58:23 mriedem dansmith: melwitt: so tl;dr, the issue in https://bugs.launchpad.net/nova/+bug/1788014 is that live migration fails and we trigger a lifecycle event which activates the port binding on the dest host incorrectly, it shouldn't do that,
17:58:23 openstack Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
17:58:37 mriedem we're getting an event from libvirt but don't know if it's success or failure for the job,
17:59:14 mriedem so what i could do if we want to be low risk with the fix for rocky GA is just not listen on that event and we'll activate port bindings on success like we always did before the change, and we'll still do the early port activating on post-copy events if the live migration is successful
17:59:35 mriedem long-term we could check the actual job status and if failed, don't trigger our lifecycle event, but that's riskier for rocky GA at this point IMO
18:00:08 mriedem so i'd propose a 2-part fix, one partial that we backport and one for just stein
18:03:57 melwitt ok
18:04:28 openstackgerrit Dan Smith proposed openstack/nova stable/queens: Fix cancel_all_events event name parsing https://review.openstack.org/592086
18:04:29 openstackgerrit Dan Smith proposed openstack/nova stable/queens: Wait for network-vif-plugged before starting live migration https://review.openstack.org/591735
18:04:30 openstackgerrit Dan Smith proposed openstack/nova stable/queens: DNM: Debug patch to test live migration waiting https://review.openstack.org/591775
18:05:23 melwitt I guess I don't understand what the lifecycle event gives us if we already know success or failure
18:05:55 melwitt without listening for it
18:07:06 mriedem in rocky we started listening on 2 new events,
18:07:12 mriedem one is post-copy and one is migration completed
18:07:31 mriedem the idea is that as soon as we switch we activate the port bindings on the dest host for minimal downtime
18:07:50 mriedem the problem is we get the latter event even if live migration fails
18:08:01 mriedem and we're not doing any conditional logic in that one to see if the job failed or not
18:09:26 melwitt I think I understand that part, it's just when you said "we'll activate port bindings on success like we always did before the change" it makes me not understand what gain the lifecycle event was supposed to give, if we already know success or failure without it
18:11:55 mriedem because if we can activate the network on the dest at the point the guest is paused to complete the transfer, it makes the network downtime window shorter
18:14:16 melwitt and without the lifecycle event, we activate the network later on after the guest is paused
18:14:19 mriedem we might still need my other fix for rollback, i'm not sure; what sean is hitting isn't a failure after post-copy
18:14:32 mriedem we activate the network after the guest transfer is complete and resumed on the dest
18:16:49 melwitt ok, got it
18:30:58 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508
18:33:11 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508
19:36:26 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.openstack.org/594527
19:37:36 mriedem melwitt: dansmith: alright fyi ^ would need sean-k-mooney to test out the 2nd more complicated fix since he has the env that recreates the bug
19:37:43 mriedem and i need to run to an appt
19:39:36 mriedem also threw that stuff in the rc todo etherpad
19:43:27 melwitt ack
19:54:56 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
19:54:57 openstackgerrit Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033
19:54:58 openstackgerrit Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider raise https://review.openstack.org/584598
19:54:59 openstackgerrit Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
19:55:00 openstackgerrit Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648
19:55:01 openstackgerrit Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034
19:55:02 openstackgerrit Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049
19:55:03 openstackgerrit Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236
20:14:31 dansmith so I
20:14:48 dansmith am pretty sure my batching breaks tssurya's down cell in its current form
20:15:02 dansmith more specifically, it'll cause scatter gather to never notice failues
20:24:16 melwitt dansmith: it's specific to the batching? it looks like pre-batch the code still removes the "did not respond" and "raised exception" results
20:24:29 dansmith still removes?
20:25:21 melwitt dansmith: this part looks like it's removing "down cells" from results? https://review.openstack.org/#/c/592698/5/nova/compute/multi_cell_list.py@311
20:26:00 dansmith right but with the batching we never hit that because we don't start executing the queries until the heapq
20:27:10 melwitt oh, I see
20:27:45 melwitt hmm
20:29:21 dansmith I'll just have to get a little more into the middle of that process and we just won't get the standard handlers from scatter gather
21:13:09 mriedem huh, you don't see instance.save() messaging timeouts in the gate very often http://logs.openstack.org/98/591898/3/check/nova-next/2d5e60c/logs/screen-n-cpu.txt.gz#_Aug_21_17_10_17_732263
21:14:24 mriedem guessing this isn't good http://logs.openstack.org/98/591898/3/check/nova-next/2d5e60c/logs/screen-n-cpu.txt.gz#_Aug_21_17_10_17_426420
21:16:54 mriedem seen here too http://logs.openstack.org/85/567785/7/check/nova-tox-functional-py35/d5a8036/job-output.txt#_2018-08-17_09_06_20_241427
21:36:54 mriedem melwitt: what do you think is missing from placement for shared storage providers support? as far as i know, it's the nova stuff that's lacking as we identified ~2 weeks ago
21:37:05 mriedem for sure move operations are not ready for shared storage providers
21:37:18 mriedem the placement side of shared storage is pretty simple though, and has been done for a long time
21:40:28 openstackgerrit Dmitry Sutyagin proposed openstack/nova-specs master: Allow disabling KSM / mem-merge via extra spec https://review.openstack.org/593197
21:41:23 melwitt mriedem: it's not that I think anything is missing, I'm pragmatically thinking of the integration work and if there is something unforeseen we need to fix. I expect bugs to shake out when we integrate something for the first time. I think not having bugs shake out will be the rarer case
21:41:38 dansmith mriedem: I think we were saying the same about aggregates being done in placement before we added the placement filter stuff and realized we needed tweaks
21:41:56 dansmith and surely thought what was being done in placement for NRPs was going to be usable by nova until we thought about it
21:46:01 mriedem dansmith: the member_of thing with aggs right?
21:46:17 dansmith we needed member_of with and and or
21:46:33 dansmith and I meant prefilter above
21:46:37 dansmith or request filter
21:46:40 dansmith or whatever that guy called it
21:47:29 dansmith the granular request stuff is another similar example, where we try to use what we think is just clean resource requests for actual nova stuff and realize we need this giant complex syntax instead

Earlier   Later