Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-19
15:21:30 jaypipes wronge that is.
15:21:36 jaypipes sean-k-mooney: :P
15:22:13 sean-k-mooney jaypipes: it is currently more functional then cyborge's pci passthough support at least ...
15:22:19 mriedem stephenfin: so no i didn't fix the original reason for the port binding failure in pre_live_migration, because that could have been for any number of reasons (neutron agent was down on the dest host?)
15:22:38 mriedem i fixed a symptom of that failure, which was nova-compute failed to restart after that failure
15:22:54 mriedem as the commit message says, "Admittedly this isn't the smartest thing and doesn't attempt
15:22:54 mriedem     to recover / fix the instance networking info"
15:22:59 stephenfin mriedem: I'm missing something. Why make changes to 'ComputeManager.init_host' (via '_init_instance') in that commit? The exception was being seen in the live migration flow
15:23:01 stephenfin ahhhhh
15:23:21 mriedem 1. live migratoin fails, port binding failed - that gets saved in the info cache
15:23:32 mriedem 2. restart source compute - that blows up because it wasn't handling binding_failed vif types in the os-vif conversion code
15:23:39 mriedem i handle #2
15:23:46 mriedem #1 is sort of out of my control
15:23:51 stephenfin Your fix would inadvertently resolve https://bugs.launchpad.net/nova/+bug/1738373 so
15:23:51 openstack Launchpad bug 1738373 in OpenStack Compute (nova) "nova-compute cannot restart if _init_host failed" [Undecided,In progress] - Assigned to Xiao Gong (gongxiao)
15:24:12 mriedem i mean, we probably shouldn't be saving off busted port binding information when pre_live_migration fails,
15:24:27 mriedem since that overwrites the previously good port binding information from the source host
15:25:03 mriedem i would have to dig into where we save off the bad port binding information
15:25:08 stephenfin Yup, there's a related fix (also for live migration) that you worked on which looks more involved https://bugs.launchpad.net/nova/+bug/1783917
15:25:08 openstack Launchpad bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [High,Fix released] - Assigned to Matt Riedemann (mriedem)
15:25:18 stephenfin Wait, wrong link?
15:26:02 mriedem ^ was a regression in rocky
15:26:46 mriedem so i suppose my fix should have been related to bug 1784579
15:26:46 openstack bug 1784579 in OpenStack Compute (nova) queens "unable to live migrate instance after update to queens" [Medium,Confirmed] https://launchpad.net/bugs/1784579
15:26:49 mriedem not closes it
15:27:27 stephenfin Probably, yeah
15:27:32 stephenfin OK, there's just a lot of overlap on these and I'm just trying to unravel it. We also have https://bugzilla.redhat.com/show_bug.cgi?id=1578028 (which I'm filing upstream now) which seems related too
15:27:32 openstack bugzilla.redhat.com bug 1578028 in openstack-nova "ovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [Urgent,Assigned] - Assigned to nova-maint
15:28:09 stephenfin That os_vif conversion code could probably do with some tweaks. I'll see what I can do
15:29:20 mriedem looks like i need to backport https://review.openstack.org/#/c/595317/ as well - i had marked the bug for queens but forgot to keep going i guess
15:29:43 sean-k-mooney stephenfin mriedem fixed that in rocky rc phase
15:30:30 mriedem stephenfin: sorting out where we save off bogus port binding failed information during pre_live_migration is probably worthwhile
15:30:40 mriedem i'm sure there is probably some update db decorator involved that automatically does it
15:31:12 sean-k-mooney mriedem: i thik its related to the network opdate events we get form neutron
15:31:22 mriedem oh yeah that might have been it
15:31:27 stephenfin mriedem: Ack. I just need to get a reproducer. Easier said than done
15:32:12 mriedem sean-k-mooney: b/c we'll get a network-changed event for that i believe
15:32:24 stephenfin sean-k-mooney: I assume you're referring to https://bugs.launchpad.net/nova/+bug/1783917 which I think is different to https://bugzilla.redhat.com/show_bug.cgi?id=1578028
15:32:24 openstack Launchpad bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [High,Fix released] - Assigned to Matt Riedemann (mriedem)
15:32:25 openstack bugzilla.redhat.com bug 1578028 in openstack-nova "ovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [Urgent,Assigned] - Assigned to nova-maint
15:33:08 stephenfin The latter is an issue all the way back to newton, assuming that BZ information is correct. Couldn't possibly be a Rocky regression
15:33:15 mriedem stephenfin: https://bugzilla.redhat.com/show_bug.cgi?id=1578028 is newton, so yes
15:33:54 mriedem https://review.openstack.org/#/c/595317/1/nova/network/os_vif_util.py wouldn't fix that since it's specifically handling vif_type of 'binding_failed'
15:33:57 mriedem not 'unbound'
15:34:33 sean-k-mooney unbound is the vif type before we set the host in the binding profile
15:34:34 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/626218
15:34:46 stephenfin mriedem: Aye, the dumb fix would be to handle unbound. Reading that bug, sahid suggested refreshing the info_cache but I'm thinking we can't do that on service startup due to the cost?
15:35:21 sean-k-mooney stephenfin: well https://review.openstack.org/#/c/591607/ will help with that
15:35:35 mriedem stephenfin: there is a lot of conversation about that in https://review.openstack.org/#/c/587498/
15:35:52 stephenfin mriedem++
15:35:54 mriedem this https://review.openstack.org/#/c/603844/ would also be related
15:36:04 mriedem as a forced recovery action
15:37:26 mriedem sean correctly pointed out i should have used related/partial bug https://review.openstack.org/#/c/587498/3//COMMIT_MSG@35
15:37:30 mriedem i must have missed that
15:37:50 sean-k-mooney https://bugzilla.redhat.com/show_bug.cgi?id=1645316 needs https://review.openstack.org/#/c/591607/
15:37:51 openstack bugzilla.redhat.com bug 1645316 in openstack-nova "Nova fails to attach both interfaces to VM after hypervisor reboot" [High,On_dev] - Assigned to smooney
15:38:04 mriedem stephenfin: i think the heal conversation is here https://review.openstack.org/#/c/587498/1/nova/compute/manager.py@956
15:38:24 mriedem i have to come back on https://review.openstack.org/#/c/591607/
15:38:28 mriedem too much gd stuff going on
15:38:42 mriedem sean-k-mooney: are you working on that pci device bogus removal patch?
15:38:44 mriedem or should i?
15:39:10 stephenfin mriedem: Yeah, sorry. Let me draft a bug report and a patch and we can discuss later. As you were
15:39:21 sean-k-mooney am i just booted up my sriov env so ill make the change now and see if that works
15:39:45 mriedem stephenfin: bug report for the unbound thing? yeah i guess just refer to these other 20 bugs for binding_failed and it's the same issue
15:40:20 stephenfin Yeah, maybe I should just make the existing bug more generic and re-open it
15:40:44 mriedem stephenfin: btw, PS1 of my patch would have handled unbound https://review.openstack.org/#/c/587498/1/nova/network/os_vif_util.py
15:40:57 mriedem since it raised a new UnsupportedVifTypeConversion exception for anything we can't convert
15:41:09 mriedem and handled in init_host https://review.openstack.org/#/c/587498/1/nova/compute/manager.py
15:41:36 mriedem eric convinced me to do something more targeted
15:41:40 stephenfin mriedem: Any idea why you changed?
15:41:41 stephenfin ah
15:42:13 mriedem https://review.openstack.org/#/c/587498/1/nova/network/os_vif_util.py
15:42:25 mriedem so your fix is just do the same pattern but for vif_type=unbound
15:42:35 openstackgerrit Martin Midolesov proposed openstack/nova master: vmware:add support for the hw_video_ram image property https://review.openstack.org/564193
15:42:49 stephenfin yup
15:43:08 canori01 mriedem: Is it possible to force the resize of an instance to the same host? I have allow_resize_to_same_host, but it seems like it still tries to pick different hosts
15:43:31 mriedem stephenfin: btw it goes back to newton https://review.openstack.org/#/c/350595/
15:44:31 mriedem canori01: i know you can force a cold migrate to a specific host with a newer microversion but i'm not sure if that applies to resize as well
15:44:35 stephenfin backporting fun \o/
15:44:43 mriedem https://developer.openstack.org/api-ref/compute/?expanded=migrate-server-migrate-action-detail#migrate-server-migrate-action
15:44:53 mriedem canori01: ^ host param was added in 2.56
15:45:50 mriedem canori01: ah looks like that won't work for cold migrate at least https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3542
15:46:41 mriedem and you can't specify a host for resize https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L806
15:46:53 mriedem https://developer.openstack.org/api-ref/compute/?expanded=migrate-server-migrate-action-detail,resize-server-resize-action-detail#resize-server-resize-action
15:47:27 mriedem canori01: so the answer is no, and the scheduler is likely picking another host because one is available even though you could use the original host, but maybe weighers or something is picking the other host
15:48:03 mriedem canori01: or the source host is filtered out because of bug https://bugs.launchpad.net/nova/+bug/1790204
15:48:03 openstack Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [Medium,Triaged]
15:49:11 openstackgerrit Balazs Gibizer proposed openstack/nova master: Create RequestGroup from neutron port https://review.openstack.org/625941
15:49:11 openstackgerrit Balazs Gibizer proposed openstack/nova master: Include requested_resources to allocation candidate query https://review.openstack.org/625942
15:49:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268
15:49:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527
15:49:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239
15:49:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528
15:49:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240
15:49:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529
15:49:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459
15:49:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
15:49:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: Reject interface attach with QoS aware port https://review.openstack.org/570078
15:49:16 openstackgerrit Balazs Gibizer proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079
15:49:17 openstackgerrit Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421

Earlier   Later