| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-21 | |||
| 16:09:31 | mriedem | yeah, but we're on different threads here | |
| 16:09:43 | mriedem | we have the domain, could we get the jobState from that? | |
| 16:09:54 | danpb | mriedem: that VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED just says that the guest has been paused, as a result of the live migration operation | |
| 16:10:03 | danpb | it doesn't say anything about the operation being success or failure | |
| 16:10:14 | mriedem | right, and that's our bug :) | |
| 16:10:18 | danpb | so you definitely can't assume success from that | |
| 16:11:22 | mriedem | right, | |
| 16:11:26 | mriedem | so i can remove that to fix this quick | |
| 16:11:39 | mriedem | or try to find the jobState from the domain and check the status? | |
| 16:11:42 | dansmith | Kevin_Zheng: okay I've changed my mind for the moment.. the api code is so generator-unfriendly that a quick hack to test this is more involved than I thought | |
| 16:11:59 | danpb | mriedem: if there's some action that needs to take place during the migration operation | |
| 16:12:18 | danpb | mriedem: then my gut feeling would be to hav the _live_migration_monitor thread take care of it | |
| 16:12:26 | sean-k-mooney | mriedem: well im not sure we need to change that code. where is the EVENT_LIFECYCLE_MIGRATION_COMPLETED event consumed because we have stopped moving stuff at this point we jsut dont know if it succeded | |
| 16:12:41 | mriedem | danpb: yeah most likely - and that's inline with what dansmith said on the review for this change | |
| 16:12:53 | mriedem | since it was baking libvirt logic into the compute manager lifecycle callback handler | |
| 16:13:09 | danpb | if the lifecycle events are needed, then forward those onto that thread too | |
| 16:13:09 | mriedem | sean-k-mooney: ComputeManager.handle_lifecycle_event | |
| 16:15:51 | sean-k-mooney | mriedem: so what we actully need to do is check the job status here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1126-L1139 | |
| 16:16:09 | mriedem | we're not going to do that in the compute manager | |
| 16:16:25 | openstackgerrit | Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698 | |
| 16:16:26 | openstackgerrit | Dan Smith proposed openstack/nova master: List instances from all cells explicitly https://review.openstack.org/593717 | |
| 16:16:27 | openstackgerrit | Dan Smith proposed openstack/nova master: Make instance_list perform per-cell batching https://review.openstack.org/593131 | |
| 16:16:58 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 16:16:59 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 16:17:00 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598 | |
| 16:17:01 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 16:17:02 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 16:17:03 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 16:17:04 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 16:17:05 | openstackgerrit | Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 16:17:06 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Regex consts for placement schema https://review.openstack.org/591863 | |
| 16:17:21 | danpb | mriedem: yeah you'd want to check status in the libvirt driver, and if some action is required in the compute manager, trigger some callout for the compute manager to act on i guess | |
| 16:18:00 | mriedem | how does one even determine job status based on https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainJobInfo ? | |
| 16:18:55 | mriedem | https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainJobType ? | |
| 16:19:58 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Regex consts for placement schema https://review.openstack.org/591863 | |
| 16:19:59 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 16:20:00 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 16:20:01 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598 | |
| 16:20:02 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 16:20:03 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 16:20:04 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 16:20:05 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 16:20:06 | openstackgerrit | Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 16:21:02 | mriedem | ah nvm i see how we get this info in nova | |
| 16:21:05 | danpb | mriedem: yeah the job type field is what we're hooking off | |
| 16:21:09 | mriedem | yup | |
| 16:21:20 | mriedem | elif info.type == libvirt.VIR_DOMAIN_JOB_FAILED: | |
| 16:22:30 | mriedem | danpb: alright thanks i think i know what to do here, | |
| 16:22:40 | mriedem | sean-k-mooney: i probably won't have something for you to test by your eod | |
| 16:22:47 | mriedem | although your eod varies wildly | |
| 16:23:01 | mriedem | but i'm in serious need of a shower and lunch at this point....i'm devolving | |
| 16:23:50 | mdbooth | mriedem danpb: IIRC we encountered limitations with this in the block rebase operation. Isn't there a race with the job info disappearing? If the job is no longer present, we no longer know if it failed or not, and the solution was supposed to be to consume events? | |
| 16:23:51 | sean-k-mooney | haha yes it does today i need to drive home which is an hour and a half away so ill be leave shortly. if you have something ill test it as soon as im back online | |
| 16:24:51 | mdbooth | Yeah, I wrote one of my comment essays about it | |
| 16:25:42 | mdbooth | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L827-L838 | |
| 16:28:06 | sean-k-mooney | mdbooth: right. am can we check if the domain is still present on the source node? if it is it would mean it failed right? | |
| 16:31:55 | danpb | mdbooth: with new enough libvirt the job will stick around | |
| 16:32:27 | danpb | mdbooth: with older libvirt the _live_Migration_monitor code has heuristic to try to figure out if no-job == failed vs success | |
| 16:33:14 | mdbooth | danpb: I got the impression at the time that eric was piling on heuristics in there for us, but really we weren't supposed to be doing that. Sounds like that's out of date? | |
| 16:34:51 | danpb | mdbooth: what do you mean ? | |
| 16:36:10 | mdbooth | There was also the heuristic for status.end. | |
| 16:36:29 | mdbooth | I just got the strong impression at the time that consuming events was the intended approach here. | |
| 16:36:49 | mdbooth | If we've sat on the problem for that to be out of date... result :) | |
| 16:38:55 | mriedem | mdbooth: if i get no job, i'm going to not send the callback event to compute manager to trigger the port binding activation, | |
| 16:39:09 | mriedem | because worst case is the job failed and we're screwing up networking, which is what sean is seeing, | |
| 16:39:21 | mriedem | best case is we don't know, but post live migration will still activate the port bindings, | |
| 16:39:30 | mriedem | you just have a bigger window of network downtime | |
| 16:39:43 | mriedem | *plus*, if the job was successful and we go into post-copy, we activate the port bindings then too | |
| 16:40:17 | mriedem | i'm fairly certain this is 100% fool proof and will forever be bug free | |
| 16:47:31 | tssurya | dansmith: would you prefer me returnng (1) the failed_cell_uuids from get_instance_objects_sorted only if cell_down_support is set ? or (2) you don't want this flag creeping down even to that level and so we just return the tuple under all conditions ? | |
| 16:47:46 | tssurya | and deal with it in the api | |
| 16:49:06 | tssurya | I am asking because its called "get_instance_objects_sorted" and returning the tuple under all conditions kind of might be weird ? | |
| 16:49:34 | dansmith | tssurya: just return it always, and let the api decide what to do with it based on the version I think | |
| 16:49:42 | tssurya | dansmith: ack | |
| 16:49:58 | dansmith | tssurya: you can change the name if you think that's important | |
| 16:50:35 | tssurya | I will put it up for review and we can see | |
| 16:51:27 | tssurya | thanks | |
| 16:51:36 | dansmith | cool | |
| 16:52:20 | sean-k-mooney | mriedem: can we get that on a tee shirt. | |
| 16:57:20 | mriedem | sean-k-mooney: my bug free guarantee? | |
| 16:57:30 | mriedem | it only applies from today through labor day | |
| 17:02:13 | dansmith | oof, 329 in check | |
| 17:02:30 | melwitt | . | |
| 17:04:24 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:04:24 | mriedem | so ima also mark https://bugs.launchpad.net/nova/+bug/1788014 as rc potential | |
| 17:04:31 | mriedem | given it's a regression when live migration fails | |
| 17:05:00 | mriedem | my only question on that one is doing a tactical fix for the GA | |
| 17:06:50 | melwitt | ok, so rc3 now | |
| 17:36:50 | openstackgerrit | Merged openstack/nova master: Update resources once in update_available_resource https://review.openstack.org/520024 | |
| 17:36:57 | openstackgerrit | Merged openstack/nova master: Set policy_opt defaults in placement gabbi fixture https://review.openstack.org/594172 | |
| 17:39:38 | openstackgerrit | Merged openstack/nova master: Set policy_opt defaults in placement deploy unit test https://review.openstack.org/594334 | |
| 17:40:38 | dansmith | mriedem: what's the plan here? https://review.openstack.org/#/c/591735/ | |
| 17:41:23 | dansmith | we would like that to be in all current upstream stable, but will backport it ourselves if we're not going to do it upstream, so I just wanna know if I should hold off or not | |
| 17:53:05 | mriedem | dansmith: i was waiting for you to rebase it | |
| 17:53:21 | dansmith | oh heh sorry | |
| 17:53:28 | mriedem | sni | |
| 17:53:30 | mriedem | snip snap | |
| 17:53:51 | mriedem | melwitt: yeah so probably rc3 | |
| 17:54:16 | mriedem | these are the 2 as of today https://bugs.launchpad.net/nova/+bugs?field.tag=rocky-rc-potential | |
| 17:54:21 | mriedem | looks like final rc is thursday | |