Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-21
16:01:29 danpb sean-k-mooney: you have any more context than just that log file ?
16:01:56 sean-k-mooney danpb: yes im testing live migration between ovs to ovs-dpdk in this case
16:02:07 sean-k-mooney that is causing qemu to have an internal error
16:02:30 danpb yep, looks like QEMU on target saw error in expected state & exited
16:02:30 mdbooth mriedem: I think if we've reached the point of post-copy, we shouldn't rollback.
16:02:40 sean-k-mooney nova is assuming that when we get the live migration complete event that everything worked fine bug in this case qemu explodes and the migration failes
16:02:41 danpb whcih should have caused libvirt to abort migration & nova to rollback
16:02:46 mdbooth mriedem: Because the guest was actually running on the destination at that point.
16:03:19 mdbooth Guessing that could be tricky with the current code structure, though...
16:03:23 danpb mdbooth: this logfile isn't showing post-coyp is it ? looks like normal pre-copy to me
16:03:38 sean-k-mooney mdbooth: the guest is still running fine after this. is netwroking is messed up but its still running on the host
16:03:43 mriedem mdbooth: we aren't hitting post-copy
16:03:44 mdbooth danpb: We could be talking about different things. I was referring to https://review.openstack.org/#/c/594139/
16:04:11 danpb oh fun, two different live migration discussions in parallel :-)
16:04:22 sean-k-mooney mdbooth: danpb its the same one
16:04:25 mriedem danpb: so this is all new code since you've been in nova
16:04:28 sean-k-mooney i reproduced it
16:04:52 mriedem we're just assuming "migration completed" means it was successful, which is wrong in this case
16:05:11 mriedem so just need to not send that event callback up to the compute manager if the job failed
16:05:19 mriedem which i think we can glean from the jobState object
16:05:35 mriedem assuming that info is available to us in the params to _event_lifecycle_callback
16:06:19 mriedem i'm not sure that it does though, we get event and detail
16:06:22 mriedem but not the job status
16:07:07 danpb mriedem: which migration events are you referring to ?
16:07:45 mdbooth I don't think we're actually consuming events. We're polling the migration job.
16:07:51 mriedem https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L179-L184
16:07:59 mriedem mdbooth: no
16:08:03 mriedem not for this
16:08:26 danpb oh, so you're just looking at the lifecycle events
16:08:40 mdbooth mriedem: ack. Was looking at _live_migration_monitor.
16:08:59 danpb i'm not convinced that's not a desirable way to determine success vs failure
16:09:09 mriedem we're getting VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED
16:09:17 mriedem and assuming it's success
16:09:23 danpb the the job status from the _live_migration_monitor is better way to check for failure
16:09:31 mriedem yeah, but we're on different threads here
16:09:43 mriedem we have the domain, could we get the jobState from that?
16:09:54 danpb mriedem: that VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED just says that the guest has been paused, as a result of the live migration operation
16:10:03 danpb it doesn't say anything about the operation being success or failure
16:10:14 mriedem right, and that's our bug :)
16:10:18 danpb so you definitely can't assume success from that
16:11:22 mriedem right,
16:11:26 mriedem so i can remove that to fix this quick
16:11:39 mriedem or try to find the jobState from the domain and check the status?
16:11:42 dansmith Kevin_Zheng: okay I've changed my mind for the moment.. the api code is so generator-unfriendly that a quick hack to test this is more involved than I thought
16:11:59 danpb mriedem: if there's some action that needs to take place during the migration operation
16:12:18 danpb mriedem: then my gut feeling would be to hav the _live_migration_monitor thread take care of it
16:12:26 sean-k-mooney mriedem: well im not sure we need to change that code. where is the EVENT_LIFECYCLE_MIGRATION_COMPLETED event consumed because we have stopped moving stuff at this point we jsut dont know if it succeded
16:12:41 mriedem danpb: yeah most likely - and that's inline with what dansmith said on the review for this change
16:12:53 mriedem since it was baking libvirt logic into the compute manager lifecycle callback handler
16:13:09 mriedem sean-k-mooney: ComputeManager.handle_lifecycle_event
16:13:09 danpb if the lifecycle events are needed, then forward those onto that thread too
16:15:51 sean-k-mooney mriedem: so what we actully need to do is check the job status here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1126-L1139
16:16:09 mriedem we're not going to do that in the compute manager
16:16:25 openstackgerrit Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698
16:16:26 openstackgerrit Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717
16:16:27 openstackgerrit Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131
16:16:58 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
16:16:59 openstackgerrit Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033
16:17:00 openstackgerrit Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598
16:17:01 openstackgerrit Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
16:17:02 openstackgerrit Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648
16:17:03 openstackgerrit Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034
16:17:04 openstackgerrit Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049
16:17:05 openstackgerrit Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236
16:17:06 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Regex consts for placement schema https://review.openstack.org/591863
16:17:21 danpb mriedem: yeah you'd want to check status in the libvirt driver, and if some action is required in the compute manager, trigger some callout for the compute manager to act on i guess
16:18:00 mriedem how does one even determine job status based on https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainJobInfo ?
16:18:55 mriedem https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainJobType ?
16:19:58 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Regex consts for placement schema https://review.openstack.org/591863
16:19:59 openstackgerrit Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927
16:20:00 openstackgerrit Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033
16:20:01 openstackgerrit Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598
16:20:02 openstackgerrit Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599
16:20:03 openstackgerrit Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648
16:20:04 openstackgerrit Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034
16:20:05 openstackgerrit Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049
16:20:06 openstackgerrit Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236
16:21:02 mriedem ah nvm i see how we get this info in nova
16:21:05 danpb mriedem: yeah the job type field is what we're hooking off
16:21:09 mriedem yup
16:21:20 mriedem elif info.type == libvirt.VIR_DOMAIN_JOB_FAILED:
16:22:30 mriedem danpb: alright thanks i think i know what to do here,
16:22:40 mriedem sean-k-mooney: i probably won't have something for you to test by your eod
16:22:47 mriedem although your eod varies wildly
16:23:01 mriedem but i'm in serious need of a shower and lunch at this point....i'm devolving
16:23:50 mdbooth mriedem danpb: IIRC we encountered limitations with this in the block rebase operation. Isn't there a race with the job info disappearing? If the job is no longer present, we no longer know if it failed or not, and the solution was supposed to be to consume events?
16:23:51 sean-k-mooney haha yes it does today i need to drive home which is an hour and a half away so ill be leave shortly. if you have something ill test it as soon as im back online
16:24:51 mdbooth Yeah, I wrote one of my comment essays about it
16:25:42 mdbooth https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L827-L838
16:28:06 sean-k-mooney mdbooth: right. am can we check if the domain is still present on the source node? if it is it would mean it failed right?
16:31:55 danpb mdbooth: with new enough libvirt the job will stick around
16:32:27 danpb mdbooth: with older libvirt the _live_Migration_monitor code has heuristic to try to figure out if no-job == failed vs success
16:33:14 mdbooth danpb: I got the impression at the time that eric was piling on heuristics in there for us, but really we weren't supposed to be doing that. Sounds like that's out of date?
16:34:51 danpb mdbooth: what do you mean ?
16:36:10 mdbooth There was also the heuristic for status.end.
16:36:29 mdbooth I just got the strong impression at the time that consuming events was the intended approach here.
16:36:49 mdbooth If we've sat on the problem for that to be out of date... result :)
16:38:55 mriedem mdbooth: if i get no job, i'm going to not send the callback event to compute manager to trigger the port binding activation,
16:39:09 mriedem because worst case is the job failed and we're screwing up networking, which is what sean is seeing,
16:39:21 mriedem best case is we don't know, but post live migration will still activate the port bindings,

Earlier   Later