Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-21
16:07:51 mriedem https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L179-L184
16:07:59 mriedem mdbooth: no
16:08:03 mriedem not for this
16:08:26 danpb oh, so you're just looking at the lifecycle events
16:08:40 mdbooth mriedem: ack. Was looking at _live_migration_monitor.
16:08:59 danpb i'm not convinced that's not a desirable way to determine success vs failure
16:09:09 mriedem we're getting VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED
16:09:17 mriedem and assuming it's success
16:09:23 danpb the the job status from the _live_migration_monitor is better way to check for failure
16:09:31 mriedem yeah, but we're on different threads here
16:09:43 mriedem we have the domain, could we get the jobState from that?
16:09:54 danpb mriedem: that VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED just says that the guest has been paused, as a result of the live migration operation
16:10:03 danpb it doesn't say anything about the operation being success or failure
16:10:14 mriedem right, and that's our bug :)
16:10:18 danpb so you definitely can't assume success from that
16:11:22 mriedem right,
16:11:26 mriedem so i can remove that to fix this quick
16:11:39 mriedem or try to find the jobState from the domain and check the status?
16:11:42 dansmith Kevin_Zheng: okay I've changed my mind for the moment.. the api code is so generator-unfriendly that a quick hack to test this is more involved than I thought
16:11:59 danpb mriedem: if there's some action that needs to take place during the migration operation
16:12:18 danpb mriedem: then my gut feeling would be to hav the _live_migration_monitor thread take care of it
16:12:26 sean-k-mooney mriedem: well im not sure we need to change that code. where is the EVENT_LIFECYCLE_MIGRATION_COMPLETED event consumed because we have stopped moving stuff at this point we jsut dont know if it succeded
16:12:41 mriedem danpb: yeah most likely - and that's inline with what dansmith said on the review for this change
16:12:53 mriedem since it was baking libvirt logic into the compute manager lifecycle callback handler
16:13:09 danpb if the lifecycle events are needed, then forward those onto that thread too
16:13:09 mriedem sean-k-mooney: ComputeManager.handle_lifecycle_event
16:15:51 sean-k-mooney mriedem: so what we actully need to do is check the job status here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1126-L1139
16:16:09 mriedem we're not going to do that in the compute manager
16:16:25 openstackgerrit Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698
16:16:26 openstackgerrit Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717
16:16:27 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131
16:16:58 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
16:16:59 openstackgerrit Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033
16:17:00 openstackgerrit Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598
16:17:01 openstackgerrit Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
16:17:02 openstackgerrit Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648
16:17:03 openstackgerrit Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034
16:17:04 openstackgerrit Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049
16:17:05 openstackgerrit Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236
16:17:06 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Regex consts for placement schema https://review.openstack.org/591863
16:17:21 danpb mriedem: yeah you'd want to check status in the libvirt driver, and if some action is required in the compute manager, trigger some callout for the compute manager to act on i guess
16:18:00 mriedem how does one even determine job status based on https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainJobInfo ?
16:18:55 mriedem https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainJobType ?
16:19:58 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Regex consts for placement schema https://review.openstack.org/591863
16:19:59 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
16:20:00 openstackgerrit Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033
16:20:01 openstackgerrit Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598
16:20:02 openstackgerrit Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
16:20:03 openstackgerrit Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648
16:20:04 openstackgerrit Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034
16:20:05 openstackgerrit Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049
16:20:06 openstackgerrit Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236
16:21:02 mriedem ah nvm i see how we get this info in nova
16:21:05 danpb mriedem: yeah the job type field is what we're hooking off
16:21:09 mriedem yup
16:21:20 mriedem elif info.type == libvirt.VIR_DOMAIN_JOB_FAILED:
16:22:30 mriedem danpb: alright thanks i think i know what to do here,
16:22:40 mriedem sean-k-mooney: i probably won't have something for you to test by your eod
16:22:47 mriedem although your eod varies wildly
16:23:01 mriedem but i'm in serious need of a shower and lunch at this point....i'm devolving
16:23:50 mdbooth mriedem danpb: IIRC we encountered limitations with this in the block rebase operation. Isn't there a race with the job info disappearing? If the job is no longer present, we no longer know if it failed or not, and the solution was supposed to be to consume events?
16:23:51 sean-k-mooney haha yes it does today i need to drive home which is an hour and a half away so ill be leave shortly. if you have something ill test it as soon as im back online
16:24:51 mdbooth Yeah, I wrote one of my comment essays about it
16:25:42 mdbooth https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L827-L838
16:28:06 sean-k-mooney mdbooth: right. am can we check if the domain is still present on the source node? if it is it would mean it failed right?
16:31:55 danpb mdbooth: with new enough libvirt the job will stick around
16:32:27 danpb mdbooth: with older libvirt the _live_Migration_monitor code has heuristic to try to figure out if no-job == failed vs success
16:33:14 mdbooth danpb: I got the impression at the time that eric was piling on heuristics in there for us, but really we weren't supposed to be doing that. Sounds like that's out of date?
16:34:51 danpb mdbooth: what do you mean ?
16:36:10 mdbooth There was also the heuristic for status.end.
16:36:29 mdbooth I just got the strong impression at the time that consuming events was the intended approach here.
16:36:49 mdbooth If we've sat on the problem for that to be out of date... result :)
16:38:55 mriedem mdbooth: if i get no job, i'm going to not send the callback event to compute manager to trigger the port binding activation,
16:39:09 mriedem because worst case is the job failed and we're screwing up networking, which is what sean is seeing,
16:39:21 mriedem best case is we don't know, but post live migration will still activate the port bindings,
16:39:30 mriedem you just have a bigger window of network downtime
16:39:43 mriedem *plus*, if the job was successful and we go into post-copy, we activate the port bindings then too
16:40:17 mriedem i'm fairly certain this is 100% fool proof and will forever be bug free
16:47:31 tssurya dansmith: would you prefer me returnng (1) the failed_cell_uuids from get_instance_objects_sorted only if cell_down_support is set ? or (2) you don't want this flag creeping down even to that level and so we just return the tuple under all conditions ?
16:47:46 tssurya and deal with it in the api
16:49:06 tssurya I am asking because its called "get_instance_objects_sorted" and returning the tuple under all conditions kind of might be weird ?
16:49:34 dansmith tssurya: just return it always, and let the api decide what to do with it based on the version I think
16:49:42 tssurya dansmith: ack
16:49:58 dansmith tssurya: you can change the name if you think that's important
16:50:35 tssurya I will put it up for review and we can see
16:51:27 tssurya thanks
16:51:36 dansmith cool
16:52:20 sean-k-mooney mriedem: can we get that on a tee shirt.
16:57:20 mriedem sean-k-mooney: my bug free guarantee?
16:57:30 mriedem it only applies from today through labor day
17:02:13 dansmith oof, 329 in check
17:02:30 melwitt .
17:04:24 openstack Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
17:04:24 mriedem so ima also mark https://bugs.launchpad.net/nova/+bug/1788014 as rc potential
17:04:31 mriedem given it's a regression when live migration fails
17:05:00 mriedem my only question on that one is doing a tactical fix for the GA
17:06:50 melwitt ok, so rc3 now
17:36:50 openstackgerrit Merged openstack/nova master: Update resources once in update_available_resource https://review.openstack.org/520024
17:36:57 openstackgerrit Merged openstack/nova master: Set policy_opt defaults in placement gabbi fixture https://review.openstack.org/594172
17:39:38 openstackgerrit Merged openstack/nova master: Set policy_opt defaults in placement deploy unit test https://review.openstack.org/594334

Earlier   Later