Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-31
15:25:08 dansmith but it's not an actual type of vif
15:25:16 mriedem unbound could also screw us
15:25:27 dansmith it also seems really wrong that they ever used that field as a signal for failure
15:25:46 sean-k-mooney dansmith: yes like unbound it indicate that its not valid for this port to be used with a guest in its current state
15:25:54 mriedem sean-k-mooney: so i'm not sure what external event we're getting for this
15:26:14 mriedem because in queens we wouldn't be waiting for a vif plugged event so we'd just log "Received unexpected event %(event)s for ..."
15:26:17 mriedem and not update our cache
15:26:25 dansmith network-changed?
15:26:38 mriedem maybe, but not sure why
15:26:44 mriedem unless the port status changes?
15:26:51 mriedem if so, then yeah i think it's network-changed
15:26:58 dansmith LOG.debug('Refreshing instance network info cache due to '
15:26:58 dansmith 'event %s.', event.key, instance=instance)
15:27:02 dansmith should see that if so
15:27:34 mriedem right, so in that case, we are getting fresh port info from neutron,
15:27:42 mriedem and if it's telling us the vif type is 'binding_failed' then i'm not sure what we can do
15:27:45 sean-k-mooney mriedem: the port status changed because we changed the host_id when we tried to migrate it
15:28:22 mriedem sean-k-mooney: i don't think that happens in pre_live_migration
15:28:25 mriedem that happens in post
15:29:05 mriedem pre_live_migration would, if we got past vif plugging, set the migrating_to field in the port binding profile
15:29:07 sean-k-mooney mriedem: we call plug vifs on the dest in pre livemigrate no? pre_live_migration
15:29:10 mriedem but that doesn't change the port's host binding
15:29:13 mriedem yes
15:29:17 openstackgerrit Merged openstack/nova stable/queens: Disable limits if force_hosts or force_nodes is set https://review.openstack.org/584616
15:29:17 dansmith mriedem: yeah, I guess I was thinking more of a single bind failing meaning we shouldn't persist that, but I guess we're doing a wholesale update and it's telling us failed for that one
15:29:23 openstackgerrit Merged openstack/nova stable/queens: Add functional regressions tests for server_group_members OverQuota https://review.openstack.org/581845
15:29:34 dansmith which I dunno, seems dumb for us to overwrite our data with that
15:29:43 mriedem dansmith: yeah that's what i said in the bug
15:29:58 mriedem if we can detect that type is changing from "ovs" to "binding_failed", don't save that n stuff
15:30:17 mriedem "here let me corrupt your internal state for you, you're welcome"
15:31:39 sean-k-mooney well the fact we are storing the vif_type persently in the cache is likely a bug in itself.
15:31:46 mriedem i don't have a setup for testing this, but it seems it would be easy enough to recreate by just stopping the neutron agent on a dest host and trying to live migrate to that host
15:32:03 dansmith sean-k-mooney: why?
15:32:17 mriedem that's the only way we know how to plug/unplug the thing,
15:32:26 mriedem similar to caching off the volume type in the bdm.connection_info
15:32:32 dansmith unless we always hit neutron always, but that's the point of the cache
15:32:37 sean-k-mooney well the fact that we dont ask neutron to bind the port again wen we start the instance and fail becasue we have a chagned bad value seams wrong
15:33:16 dansmith mriedem: will your "just reboot the instance" thing re-bind to the existing host?
15:33:30 mriedem if neutron gives us the proper information it should,
15:33:31 sean-k-mooney i dont think haveing the cache is a bad thing but the fact we dont retry with uncached data is what i thnk is weird
15:33:32 mriedem which leads me to https://review.openstack.org/#/c/587498/1/nova/compute/manager.py@956
15:33:47 mriedem ^ is the cached info, but we could change that to get a fresh copy from neutron,
15:34:01 mriedem but that's why i'm asking if neutron is sending us binding_failed continually b/c if so, that won't help
15:35:28 sean-k-mooney mriedem: without the neutron logs its hard to tell but it seam more likely to me that the newly deoployed host is misconfigured then neutron is broken for all nodes
15:35:45 dansmith yes, obviously the root cause is in the neutron logs,
15:35:55 mriedem right,
15:36:03 mriedem but what gets us back to getting a real vif type from neutron?
15:36:05 dansmith but presumably it was bound before and just restoring the info makes it work on the original host again,
15:36:06 dansmith so presumably if we re-bind on the source it will work
15:36:07 mriedem re-plugging on the source?
15:36:17 dansmith mriedem: well, restarting the compute node for sure :)
15:36:26 dansmith just not positive about the reboot case
15:36:32 dansmith although I think it probably will
15:36:51 mriedem re-plug fails on the source when restarting the compute
15:36:58 mriedem that's what https://review.openstack.org/#/c/587498/1/nova/compute/manager.py is for
15:37:13 mriedem because it's trying to plug using a binding_failed vif type
15:37:18 sean-k-mooney mriedem: repluging is not the same as rebinding however.
15:37:38 sean-k-mooney its using the binidng_failed vif form the info cache to replug right?
15:37:46 mriedem on compute restart yes
15:37:52 mriedem which causes compute to blow up and fail to start
15:38:19 sean-k-mooney ya so we could have it try to rebind the port when that happens for the current host
15:38:48 dansmith oh right,
15:38:52 mriedem i'm failing to see what ever changed the port binding on the dest
15:38:57 dansmith it'll re-plug on restart but likely not bind
15:39:08 dansmith mriedem: the live migration, no?
15:39:24 mriedem pre_live_migration doesn't change the port's host binding before it tries to plug
15:39:38 mriedem the port binding only changes to the dest in post live migration
15:40:33 mriedem he's failing here https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6163
15:41:04 mriedem that's an rpc call from the source host and then the source catches that here https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6327 and triggers rollback
15:41:34 mriedem rollback calls this https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6859 but it doesn't do anything for neutron
15:43:13 mriedem starting in rocky, https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6903 will clear the port's migrating_to profile entry and delete any inactive dest host port bindings
15:43:36 dansmith mriedem: any chance this is the second migration attempt after the first failed in post?
15:43:50 openstackgerrit Surya Seetharaman proposed openstack/nova master: Cleanup comp_node, res_prov, services, aggregate_hosts during cell deletion https://review.openstack.org/546660
15:44:00 mriedem dansmith: possibly
15:44:13 mriedem the info cache could have already had binding_failed in it before they started the live migration
15:44:17 mriedem which makes pre_live_migration fail
15:44:29 dansmith right
15:45:15 sean-k-mooney hum that sound plausible
15:45:40 openstackgerrit Surya Seetharaman proposed openstack/nova master: Cleanup comp_node, res_prov, services, aggregate_hosts during cell deletion https://review.openstack.org/546660
15:45:50 sean-k-mooney is the fix in that case still to just not update the info_cache woith binding_vailed
15:46:08 mriedem idk
15:46:10 sean-k-mooney *binding_failed
15:46:19 mriedem kind of throwing things at the wall at that point trying to see what works
15:46:23 mriedem w/o a real recreate i can tinker with
15:46:46 sean-k-mooney i guess we need to create a regression test for it to figure it out
15:46:53 sean-k-mooney or get logs
15:47:11 mriedem we can't really test this very easily...
15:47:14 mriedem tempest won't do this
15:47:25 mriedem and an in-tree regressoin test would just stub out neutron
15:48:00 mriedem this is why for now i just did handle the error like the original bug for the exact same thing
15:48:10 mriedem https://review.openstack.org/#/c/129158/
15:48:18 mriedem exact same issue, just pre-os-vif
15:49:52 sean-k-mooney well catching the error and preventing the agent from crashing is proably correct in any case.
15:56:11 cdent is there a tag for nova-api
15:56:49 cdent or does one just say "api"?
15:57:55 sean-k-mooney cdent: i had assumed [openstack-dev][nova][api] ...
15:58:16 sean-k-mooney im probably wrong however
15:58:24 mriedem i think he's talking about bug tags
15:58:37 sean-k-mooney oh ok
15:58:41 cdent sean-k-mooney: yeah, sorry, mriedem is right, I wasn't context switching well

Earlier   Later