Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-02
21:43:46 mriedem "all control nodes and net nodes are running OSA (Rocky), some compute are running RDO (Queens), some are RDO (Rocky) and the remaining are OSA (Rocky)."
21:44:01 NewBruce ill double check some other regions and see if they are diferent there
21:44:13 mnaser NewBruce: by any chance maybe the other regions aren’t pinned to lower version maybe?
21:44:21 mnaser And so this is why we don’t catch it?
21:44:30 NewBruce mriedem correct / mnaser ill double check
21:46:08 NewBruce quick (not exhaustive) check, upgrade levels is auto
21:47:48 NewBruce we have the same mix of service versions there as well (30 / 35)
21:49:08 mnaser Yet that issue somehow doesn’t happen there
21:49:14 mnaser So off
21:49:17 mnaser Odd
21:50:21 mriedem NewBruce: it fails in _post_live_migration right/
21:50:22 mriedem ?
21:51:43 mnaser I think so. Rather than deleting the old binding and activating the new one, it tries to update the port binding
21:51:44 NewBruce mriedem correct
21:51:52 NewBruce mnaser correct
21:52:13 mnaser Which is what it would be with old port binding method.
21:52:49 mriedem right in the old method there is just one port binding
21:52:52 mriedem and we change the host on it
21:55:06 NewBruce mriedem and thats the exact behavior we see in RDO - RDO
21:55:57 mnaser So rocky to rocky and it does old port binding
21:56:32 mnaser NewBruce: can you restart a nova-compute with rdo and debug=true and double check the value of upgrade_levels in the output on startup of Oslo CFC
21:56:37 mnaser Cfg
21:56:43 mnaser In case rdo is doing weird stuff
21:57:14 NewBruce sure
21:57:36 sean-k-mooney sound like ye are making some progress on this
21:57:51 NewBruce [upgrade_levels]
21:57:51 NewBruce compute = auto
21:58:14 NewBruce debug=true
22:00:21 sean-k-mooney is the theory that RDO and OSA are not using the same compute RPC version due to there configs eventhough they are running more or less the same code
22:00:43 sean-k-mooney and as a result RDO is useing the old mechaniusm while osa is usign the new mechanisium?
22:03:10 NewBruce mnaser
22:03:11 NewBruce nova-compute.log:2019-04-02 23:59:08.448 10483 DEBUG oslo_service.service [req-4f4874a7-a967-49d1-a643-c59b856c5c61 - - - - -] upgrade_levels.compute = auto log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3032
22:03:58 mnaser I mean I dug through the conductor code a lot and it checks the source and dest to be above or at a certain later
22:04:03 mnaser Level
22:04:19 mnaser As far as I know the conductor creates the port bindings
22:04:36 sean-k-mooney not in all cases
22:04:42 sean-k-mooney it can be created by the compute node
22:05:37 mnaser Oh really? So I think in the new flow, the new compute node creates it right?
22:05:38 mriedem there is already an active port binding for the source host when you start the live migration,
22:05:52 mriedem in the new flow, conductor will create an inactive port binding for the dest host
22:06:20 mriedem and saves information about that new dest host port binding on the LiveMigrateData.vifs field that gets passed around to the computes
22:06:25 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L280-L284
22:06:51 mnaser Right. That’s what I thought. I even asked NewBruce to set the “dont live migrate until vif is plugged” option that was introduced in rocky as false and moved to true in Stein
22:06:52 sean-k-mooney yes we do the version check the bind the ports on teh destiation
22:07:11 mriedem mnaser: that's unrelated to this
22:07:30 mriedem i mean it was part of the same bp
22:07:40 mriedem but doesn't rely on the active/inactive port bindings
22:07:46 sean-k-mooney mnaser: in the old flow in post livemigate on dest the compute node updated the port binding
22:08:10 mriedem correct that is the call here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6836
22:08:42 mnaser NewBruce: can you update conductor to perhaps output the content of every statement in the if that sean-k-mooney linked?
22:09:08 mnaser And do and rdo to rdo migration and see which one of them is evaluating to false and not doing port bindings?
22:09:25 mriedem that code in post_live_migration_at_destination won't update the port binding host if it's already the specified host https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088
22:09:30 mnaser Cause rdo to rdo seems to do old school bindings from what I understand
22:09:32 mriedem which is why it's a no-op with the new flow
22:09:41 mriedem because we've already activated the dest host port binding before we get to https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088
22:10:46 NewBruce mnaser yeah, thats no problem (to update the conductor and test)
22:11:06 sean-k-mooney right be if we create an inactive binding on the dest and then somehow end ups runint the old code in post live migrate the host it gets back from neturon will be the source node as that will still be the active binding right
22:11:25 mriedem NewBruce: in case you're not familar, this is the code on the source compute that activates the dest port binding once we've switched to post-copy
22:11:25 mriedem https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L1136-L1160
22:11:47 sean-k-mooney so if we are corssing the streams here that check might be a little wierd
22:12:03 NewBruce mriedem right; yeah i have filled that with a ton of debug messages as well :D (have the logs saved down in my diaries if they become useful)
22:13:45 mnaser I think once we find out why rdo-rdo uses old flow even if upgrade levels are auto.. it might help
22:14:13 cfriesen when doing spec re-approvals for T for stuff that didn't make it in S, do I just copy from stein/approved into train/approved?
22:14:20 mordred mriedem: cool - thanks for the context on the migrate
22:15:09 sean-k-mooney mriedem: that code assumes we get the lifecycle event
22:15:15 mriedem cfriesen: https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications
22:15:20 sean-k-mooney if we dont we wont activate it.
22:15:49 sean-k-mooney until post live migration so may that could be whats happening in the RDO to OSA flow
22:15:50 cfriesen mriedem: you mean I'm supposed to read the readme?
22:16:05 cfriesen :)
22:16:05 mriedem sean-k-mooney: that code is just a best effort to try and activate the dest host port binding to reduce network downtime, as the comment says, " Otherwise the ports are bound to the destination host # in post_live_migration_at_destination."
22:16:45 sean-k-mooney ya but if we dont activate here then https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 will be true
22:17:13 mnaser not gonna lie, bugs like this makes me wish we had one agent that did it all :P
22:17:22 mnaser I’ll walk myself out
22:17:35 mriedem sean-k-mooney: sure, and honestly i thought that was still supposed to work
22:17:49 mriedem i don't know enough about what's happening within neutron to cause that duplicate primary key error
22:18:05 sean-k-mooney it proably shoudl work but i dont think we have testeded it.
22:18:35 mriedem the new flow deals with port bindings on the port bindings resources in the port bindings API, the old flow just deals with the port resource and its binding:host_id attribute
22:18:38 NewBruce i will post a large log dump into the launchpad… brb
22:18:42 mnaser mriedem: nova triés to update the old binding to point to the new host, but because a new binding already exists, it blows up
22:18:59 mnaser Cause you can’t have a binding with same port/host combo
22:19:00 sean-k-mooney we might be abel to reproduce the neutron issue with a functional test
22:19:11 mriedem mnaser: yeah, i just thought neutron was handling that for us
22:19:45 mnaser i guess maybe that’s where the bug lives
22:19:57 mriedem i.e. i thought if we changed the ports binding:host_id value and there was already a port binding for that host, but was inactive, neutron would automatically activate it
22:19:58 sean-k-mooney create a port binding, create an inactive port binding on another host, try to update the orginal binding to the destiation instead of activating
22:20:16 sean-k-mooney mriedem: i dont think it does that
22:20:29 mnaser Yep that would be a reproducer sean-k-mooney
22:20:37 mriedem sean-k-mooney: hell i could just push a patch to comment this out https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L1155 and our live migration job should blow up
22:21:56 mriedem NewBruce: while you're at it, dump the libvirtd and qemu-kvm package versions for both rdo and osa rocky nodes in the bu
22:21:58 mriedem *bug
22:22:17 NewBruce (i’ve just added to the ml2_port_binding trace to the launchpad as well as a debug version of the logs)
22:25:37 openstackgerrit Chris Friesen proposed openstack/nova-specs master: Re-propose emulated virtual TPM spec to train https://review.openstack.org/649463
22:28:06 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Test theory about bug 1822884 https://review.openstack.org/649464
22:28:07 openstack bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] https://launchpad.net/bugs/1822884
22:28:07 mriedem mnaser: NewBruce: sean-k-mooney: ^ we'll find out
22:29:06 mnaser Let’s wait and see
22:29:41 NewBruce mriedem posted version info
22:30:32 sean-k-mooney i might be able to reporduce somethign on the neutron side too if i extend https://github.com/openstack/neutron/blob/master/neutron/tests/fullstack/test_ports_rebind.py but honestly im not sure that neutorn has mulit service functional test like nova has where we use a fake message bus and run everything in the one process
22:30:46 NewBruce mriedem will test it out

Earlier   Later