| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-19 | |||
| 15:14:53 | mriedem | for the deleted pci device | |
| 15:15:31 | sean-k-mooney | mriedem: mgariepy ill write this up in the bug and i might go fix it. that said im going on vaction today so i might not get to it untill january | |
| 15:15:46 | sean-k-mooney | if other want to work on it in the interim feel free. | |
| 15:16:46 | mriedem | sean-k-mooney: i just did https://bugs.launchpad.net/nova/+bug/1633120 | |
| 15:16:46 | openstack | Launchpad bug 1633120 in OpenStack Compute (nova) "Nova scheduler tries to assign an already-in-use SRIOV QAT VF to a new instance" [Undecided,Confirmed] | |
| 15:17:29 | sean-k-mooney | mriedem: ok in that case ill close https://bugs.launchpad.net/nova/+bug/1809040 as a duplicate of https://bugs.launchpad.net/nova/+bug/1633120 | |
| 15:17:29 | openstack | Launchpad bug 1633120 in OpenStack Compute (nova) "duplicate for #1809040 Nova scheduler tries to assign an already-in-use SRIOV QAT VF to a new instance" [Undecided,Confirmed] | |
| 15:17:57 | mriedem | sean-k-mooney: already did | |
| 15:18:07 | mriedem | stephenfin: yes bug 1784579 is about os-vif port binding failed errors right? | |
| 15:18:08 | openstack | bug 1784579 in OpenStack Compute (nova) queens "unable to live migrate instance after update to queens" [Medium,Confirmed] https://launchpad.net/bugs/1784579 | |
| 15:18:20 | jaypipes | mriedem: apologies. internet down for last 15 minutes in Sarasota... last thing I got from you was "is that from the db or the config" | |
| 15:18:29 | sean-k-mooney | mriedem: in that case ill pay sean-k-mooney 1 million dollars :) | |
| 15:19:00 | mriedem | jaypipes: oh just this terrible code https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L177 | |
| 15:19:14 | stephenfin | mriedem: Yup, but it's to do with live migration and the fix is only for the service startup code path | |
| 15:19:18 | mriedem | sean-k-mooney: if you want, throw up a patch that changes ^ to a continue, fix whatever test hits that and i'll +2 | |
| 15:19:40 | stephenfin | At least, assuming I'm reading it right. I'll do some digging but just wanted to sanity check it before I dived down the rabbit hole :) | |
| 15:19:56 | mriedem | stephenfin: the live migratoin fails because of the port binding failures | |
| 15:20:06 | jaypipes | mriedem: I think you meant this code: https://github.com/openstack/nova/blob/a1dba961f0018a4995d208a290f4a859ce295840/nova/pci/manager.py#L1-L357 | |
| 15:20:22 | mriedem | i see what you did there | |
| 15:20:28 | jaypipes | u like that? | |
| 15:21:03 | sean-k-mooney | jaypipes: your not wronge | |
| 15:21:15 | mriedem | stephenfin: comment | |
| 15:21:16 | mriedem | 2 | |
| 15:21:17 | mriedem | "To summarize, it looks like the pre_live_migration method on the destination host fails to plug vifs and you end up with the "binding_failed" error, which is raised and makes the source live_migration method fail as expected. The failure is on the dest host. As a result, the info cache is updated with "binding_failed" which causes the source compute restart to fail here:" | |
| 15:21:23 | jaypipes | sean-k-mooney: but you are. | |
| 15:21:30 | jaypipes | wronge that is. | |
| 15:21:36 | jaypipes | sean-k-mooney: :P | |
| 15:22:13 | sean-k-mooney | jaypipes: it is currently more functional then cyborge's pci passthough support at least ... | |
| 15:22:19 | mriedem | stephenfin: so no i didn't fix the original reason for the port binding failure in pre_live_migration, because that could have been for any number of reasons (neutron agent was down on the dest host?) | |
| 15:22:38 | mriedem | i fixed a symptom of that failure, which was nova-compute failed to restart after that failure | |
| 15:22:54 | mriedem | as the commit message says, "Admittedly this isn't the smartest thing and doesn't attempt | |
| 15:22:54 | mriedem | to recover / fix the instance networking info" | |
| 15:22:59 | stephenfin | mriedem: I'm missing something. Why make changes to 'ComputeManager.init_host' (via '_init_instance') in that commit? The exception was being seen in the live migration flow | |
| 15:23:01 | stephenfin | ahhhhh | |
| 15:23:21 | mriedem | 1. live migratoin fails, port binding failed - that gets saved in the info cache | |
| 15:23:32 | mriedem | 2. restart source compute - that blows up because it wasn't handling binding_failed vif types in the os-vif conversion code | |
| 15:23:39 | mriedem | i handle #2 | |
| 15:23:46 | mriedem | #1 is sort of out of my control | |
| 15:23:51 | stephenfin | Your fix would inadvertently resolve https://bugs.launchpad.net/nova/+bug/1738373 so | |
| 15:23:51 | openstack | Launchpad bug 1738373 in OpenStack Compute (nova) "nova-compute cannot restart if _init_host failed" [Undecided,In progress] - Assigned to Xiao Gong (gongxiao) | |
| 15:24:12 | mriedem | i mean, we probably shouldn't be saving off busted port binding information when pre_live_migration fails, | |
| 15:24:27 | mriedem | since that overwrites the previously good port binding information from the source host | |
| 15:25:03 | mriedem | i would have to dig into where we save off the bad port binding information | |
| 15:25:08 | stephenfin | Yup, there's a related fix (also for live migration) that you worked on which looks more involved https://bugs.launchpad.net/nova/+bug/1783917 | |
| 15:25:08 | openstack | Launchpad bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [High,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 15:25:18 | stephenfin | Wait, wrong link? | |
| 15:26:02 | mriedem | ^ was a regression in rocky | |
| 15:26:46 | mriedem | so i suppose my fix should have been related to bug 1784579 | |
| 15:26:46 | openstack | bug 1784579 in OpenStack Compute (nova) queens "unable to live migrate instance after update to queens" [Medium,Confirmed] https://launchpad.net/bugs/1784579 | |
| 15:26:49 | mriedem | not closes it | |
| 15:27:27 | stephenfin | Probably, yeah | |
| 15:27:32 | stephenfin | OK, there's just a lot of overlap on these and I'm just trying to unravel it. We also have https://bugzilla.redhat.com/show_bug.cgi?id=1578028 (which I'm filing upstream now) which seems related too | |
| 15:27:32 | openstack | bugzilla.redhat.com bug 1578028 in openstack-nova "ovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [Urgent,Assigned] - Assigned to nova-maint | |
| 15:28:09 | stephenfin | That os_vif conversion code could probably do with some tweaks. I'll see what I can do | |
| 15:29:20 | mriedem | looks like i need to backport https://review.openstack.org/#/c/595317/ as well - i had marked the bug for queens but forgot to keep going i guess | |
| 15:29:43 | sean-k-mooney | stephenfin mriedem fixed that in rocky rc phase | |
| 15:30:30 | mriedem | stephenfin: sorting out where we save off bogus port binding failed information during pre_live_migration is probably worthwhile | |
| 15:30:40 | mriedem | i'm sure there is probably some update db decorator involved that automatically does it | |
| 15:31:12 | sean-k-mooney | mriedem: i thik its related to the network opdate events we get form neutron | |
| 15:31:22 | mriedem | oh yeah that might have been it | |
| 15:31:27 | stephenfin | mriedem: Ack. I just need to get a reproducer. Easier said than done | |
| 15:32:12 | mriedem | sean-k-mooney: b/c we'll get a network-changed event for that i believe | |
| 15:32:24 | stephenfin | sean-k-mooney: I assume you're referring to https://bugs.launchpad.net/nova/+bug/1783917 which I think is different to https://bugzilla.redhat.com/show_bug.cgi?id=1578028 | |
| 15:32:24 | openstack | Launchpad bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [High,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 15:32:25 | openstack | bugzilla.redhat.com bug 1578028 in openstack-nova "ovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [Urgent,Assigned] - Assigned to nova-maint | |
| 15:33:08 | stephenfin | The latter is an issue all the way back to newton, assuming that BZ information is correct. Couldn't possibly be a Rocky regression | |
| 15:33:15 | mriedem | stephenfin: https://bugzilla.redhat.com/show_bug.cgi?id=1578028 is newton, so yes | |
| 15:33:54 | mriedem | https://review.openstack.org/#/c/595317/1/nova/network/os_vif_util.py wouldn't fix that since it's specifically handling vif_type of 'binding_failed' | |
| 15:33:57 | mriedem | not 'unbound' | |
| 15:34:33 | sean-k-mooney | unbound is the vif type before we set the host in the binding profile | |
| 15:34:34 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/626218 | |
| 15:34:46 | stephenfin | mriedem: Aye, the dumb fix would be to handle unbound. Reading that bug, sahid suggested refreshing the info_cache but I'm thinking we can't do that on service startup due to the cost? | |
| 15:35:21 | sean-k-mooney | stephenfin: well https://review.openstack.org/#/c/591607/ will help with that | |
| 15:35:35 | mriedem | stephenfin: there is a lot of conversation about that in https://review.openstack.org/#/c/587498/ | |
| 15:35:52 | stephenfin | mriedem++ | |
| 15:35:54 | mriedem | this https://review.openstack.org/#/c/603844/ would also be related | |
| 15:36:04 | mriedem | as a forced recovery action | |
| 15:37:26 | mriedem | sean correctly pointed out i should have used related/partial bug https://review.openstack.org/#/c/587498/3//COMMIT_MSG@35 | |
| 15:37:30 | mriedem | i must have missed that | |
| 15:37:50 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=1645316 needs https://review.openstack.org/#/c/591607/ | |
| 15:37:51 | openstack | bugzilla.redhat.com bug 1645316 in openstack-nova "Nova fails to attach both interfaces to VM after hypervisor reboot" [High,On_dev] - Assigned to smooney | |
| 15:38:04 | mriedem | stephenfin: i think the heal conversation is here https://review.openstack.org/#/c/587498/1/nova/compute/manager.py@956 | |
| 15:38:24 | mriedem | i have to come back on https://review.openstack.org/#/c/591607/ | |
| 15:38:28 | mriedem | too much gd stuff going on | |
| 15:38:42 | mriedem | sean-k-mooney: are you working on that pci device bogus removal patch? | |
| 15:38:44 | mriedem | or should i? | |
| 15:39:10 | stephenfin | mriedem: Yeah, sorry. Let me draft a bug report and a patch and we can discuss later. As you were | |
| 15:39:21 | sean-k-mooney | am i just booted up my sriov env so ill make the change now and see if that works | |
| 15:39:45 | mriedem | stephenfin: bug report for the unbound thing? yeah i guess just refer to these other 20 bugs for binding_failed and it's the same issue | |
| 15:40:20 | stephenfin | Yeah, maybe I should just make the existing bug more generic and re-open it | |
| 15:40:44 | mriedem | stephenfin: btw, PS1 of my patch would have handled unbound https://review.openstack.org/#/c/587498/1/nova/network/os_vif_util.py | |
| 15:40:57 | mriedem | since it raised a new UnsupportedVifTypeConversion exception for anything we can't convert | |
| 15:41:09 | mriedem | and handled in init_host https://review.openstack.org/#/c/587498/1/nova/compute/manager.py | |
| 15:41:36 | mriedem | eric convinced me to do something more targeted | |
| 15:41:40 | stephenfin | mriedem: Any idea why you changed? | |
| 15:41:41 | stephenfin | ah | |
| 15:42:13 | mriedem | https://review.openstack.org/#/c/587498/1/nova/network/os_vif_util.py | |
| 15:42:25 | mriedem | so your fix is just do the same pattern but for vif_type=unbound | |
| 15:42:35 | openstackgerrit | Martin Midolesov proposed openstack/nova master: vmware:add support for the hw_video_ram image property https://review.openstack.org/564193 | |
| 15:42:49 | stephenfin | yup | |