| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-02 | |||
| 21:39:03 | mriedem | first i've heard of it :/ | |
| 21:39:15 | mriedem | mnaser: ok but that's why you didn't hit it | |
| 21:39:21 | mnaser | I’ve been trying to help NewBruce nail it down forever. | |
| 21:39:36 | mnaser | But I’ve kinda struggled at finding the code path to replicate it | |
| 21:40:01 | mnaser | There are other environments where there is mixed compute AND Rocky network and this doesn’t happen | |
| 21:40:04 | mnaser | Right NewBruce ? | |
| 21:40:27 | NewBruce | yeah thats right mnaser | |
| 21:40:29 | mriedem | right he said, "2. OSA -> OSA uses the new flow (two entries which are cleaned up)" | |
| 21:40:40 | NewBruce | this is the first site we’ve seen it on... | |
| 21:41:43 | NewBruce | mriedem right; so if i grab a VM on an OSA node, and migrate it to an OSA node, watch ml2_port_bindings - ill see one port, briefly two ports, the profiles change, and then the second port entry removed | |
| 21:41:50 | NewBruce | as described in the bluebrint | |
| 21:42:23 | NewBruce | testing RDO - RDO, i don’t see that behaviour - just a single port entry in ml2_port_bindings throughout the entire migration | |
| 21:42:28 | mnaser | mriedem: NewBruce runs a few regions and this one is the only one where it works. With both queens to queens | |
| 21:42:41 | mnaser | Shit, what’s upgrade levels set to in the rdo notes? | |
| 21:42:57 | mnaser | Nodes | |
| 21:43:01 | mnaser | Sorry. I’m on mobile. | |
| 21:43:13 | NewBruce | mnaser auto | |
| 21:43:38 | NewBruce | mriedem and i chatted above that will pin it to the lowest version, as you suspected previously | |
| 21:43:41 | mnaser | ok, so yeah, I’m pretty torn on why it would actually do that in one environment and not another. | |
| 21:43:46 | mriedem | "all control nodes and net nodes are running OSA (Rocky), some compute are running RDO (Queens), some are RDO (Rocky) and the remaining are OSA (Rocky)." | |
| 21:44:01 | NewBruce | ill double check some other regions and see if they are diferent there | |
| 21:44:13 | mnaser | NewBruce: by any chance maybe the other regions aren’t pinned to lower version maybe? | |
| 21:44:21 | mnaser | And so this is why we don’t catch it? | |
| 21:44:30 | NewBruce | mriedem correct / mnaser ill double check | |
| 21:46:08 | NewBruce | quick (not exhaustive) check, upgrade levels is auto | |
| 21:47:48 | NewBruce | we have the same mix of service versions there as well (30 / 35) | |
| 21:49:08 | mnaser | Yet that issue somehow doesn’t happen there | |
| 21:49:14 | mnaser | So off | |
| 21:49:17 | mnaser | Odd | |
| 21:50:21 | mriedem | NewBruce: it fails in _post_live_migration right/ | |
| 21:50:22 | mriedem | ? | |
| 21:51:43 | mnaser | I think so. Rather than deleting the old binding and activating the new one, it tries to update the port binding | |
| 21:51:44 | NewBruce | mriedem correct | |
| 21:51:52 | NewBruce | mnaser correct | |
| 21:52:13 | mnaser | Which is what it would be with old port binding method. | |
| 21:52:49 | mriedem | right in the old method there is just one port binding | |
| 21:52:52 | mriedem | and we change the host on it | |
| 21:55:06 | NewBruce | mriedem and thats the exact behavior we see in RDO - RDO | |
| 21:55:57 | mnaser | So rocky to rocky and it does old port binding | |
| 21:56:32 | mnaser | NewBruce: can you restart a nova-compute with rdo and debug=true and double check the value of upgrade_levels in the output on startup of Oslo CFC | |
| 21:56:37 | mnaser | Cfg | |
| 21:56:43 | mnaser | In case rdo is doing weird stuff | |
| 21:57:14 | NewBruce | sure | |
| 21:57:36 | sean-k-mooney | sound like ye are making some progress on this | |
| 21:57:51 | NewBruce | [upgrade_levels] | |
| 21:57:51 | NewBruce | compute = auto | |
| 21:58:14 | NewBruce | debug=true | |
| 22:00:21 | sean-k-mooney | is the theory that RDO and OSA are not using the same compute RPC version due to there configs eventhough they are running more or less the same code | |
| 22:00:43 | sean-k-mooney | and as a result RDO is useing the old mechaniusm while osa is usign the new mechanisium? | |
| 22:03:10 | NewBruce | mnaser | |
| 22:03:11 | NewBruce | nova-compute.log:2019-04-02 23:59:08.448 10483 DEBUG oslo_service.service [req-4f4874a7-a967-49d1-a643-c59b856c5c61 - - - - -] upgrade_levels.compute = auto log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3032 | |
| 22:03:58 | mnaser | I mean I dug through the conductor code a lot and it checks the source and dest to be above or at a certain later | |
| 22:04:03 | mnaser | Level | |
| 22:04:19 | mnaser | As far as I know the conductor creates the port bindings | |
| 22:04:36 | sean-k-mooney | not in all cases | |
| 22:04:42 | sean-k-mooney | it can be created by the compute node | |
| 22:05:37 | mnaser | Oh really? So I think in the new flow, the new compute node creates it right? | |
| 22:05:38 | mriedem | there is already an active port binding for the source host when you start the live migration, | |
| 22:05:52 | mriedem | in the new flow, conductor will create an inactive port binding for the dest host | |
| 22:06:20 | mriedem | and saves information about that new dest host port binding on the LiveMigrateData.vifs field that gets passed around to the computes | |
| 22:06:25 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L280-L284 | |
| 22:06:51 | mnaser | Right. That’s what I thought. I even asked NewBruce to set the “dont live migrate until vif is plugged” option that was introduced in rocky as false and moved to true in Stein | |
| 22:06:52 | sean-k-mooney | yes we do the version check the bind the ports on teh destiation | |
| 22:07:11 | mriedem | mnaser: that's unrelated to this | |
| 22:07:30 | mriedem | i mean it was part of the same bp | |
| 22:07:40 | mriedem | but doesn't rely on the active/inactive port bindings | |
| 22:07:46 | sean-k-mooney | mnaser: in the old flow in post livemigate on dest the compute node updated the port binding | |
| 22:08:10 | mriedem | correct that is the call here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6836 | |
| 22:08:42 | mnaser | NewBruce: can you update conductor to perhaps output the content of every statement in the if that sean-k-mooney linked? | |
| 22:09:08 | mnaser | And do and rdo to rdo migration and see which one of them is evaluating to false and not doing port bindings? | |
| 22:09:25 | mriedem | that code in post_live_migration_at_destination won't update the port binding host if it's already the specified host https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 | |
| 22:09:30 | mnaser | Cause rdo to rdo seems to do old school bindings from what I understand | |
| 22:09:32 | mriedem | which is why it's a no-op with the new flow | |
| 22:09:41 | mriedem | because we've already activated the dest host port binding before we get to https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 | |
| 22:10:46 | NewBruce | mnaser yeah, thats no problem (to update the conductor and test) | |
| 22:11:06 | sean-k-mooney | right be if we create an inactive binding on the dest and then somehow end ups runint the old code in post live migrate the host it gets back from neturon will be the source node as that will still be the active binding right | |
| 22:11:25 | mriedem | NewBruce: in case you're not familar, this is the code on the source compute that activates the dest port binding once we've switched to post-copy | |
| 22:11:25 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L1136-L1160 | |
| 22:11:47 | sean-k-mooney | so if we are corssing the streams here that check might be a little wierd | |
| 22:12:03 | NewBruce | mriedem right; yeah i have filled that with a ton of debug messages as well :D (have the logs saved down in my diaries if they become useful) | |
| 22:13:45 | mnaser | I think once we find out why rdo-rdo uses old flow even if upgrade levels are auto.. it might help | |
| 22:14:13 | cfriesen | when doing spec re-approvals for T for stuff that didn't make it in S, do I just copy from stein/approved into train/approved? | |
| 22:14:20 | mordred | mriedem: cool - thanks for the context on the migrate | |
| 22:15:09 | sean-k-mooney | mriedem: that code assumes we get the lifecycle event | |
| 22:15:15 | mriedem | cfriesen: https://specs.openstack.org/openstack/nova-specs/readme.html#previously-approved-specifications | |
| 22:15:20 | sean-k-mooney | if we dont we wont activate it. | |
| 22:15:49 | sean-k-mooney | until post live migration so may that could be whats happening in the RDO to OSA flow | |
| 22:15:50 | cfriesen | mriedem: you mean I'm supposed to read the readme? | |
| 22:16:05 | cfriesen | :) | |
| 22:16:05 | mriedem | sean-k-mooney: that code is just a best effort to try and activate the dest host port binding to reduce network downtime, as the comment says, " Otherwise the ports are bound to the destination host # in post_live_migration_at_destination." | |
| 22:16:45 | sean-k-mooney | ya but if we dont activate here then https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 will be true | |
| 22:17:13 | mnaser | not gonna lie, bugs like this makes me wish we had one agent that did it all :P | |
| 22:17:22 | mnaser | I’ll walk myself out | |
| 22:17:35 | mriedem | sean-k-mooney: sure, and honestly i thought that was still supposed to work | |
| 22:17:49 | mriedem | i don't know enough about what's happening within neutron to cause that duplicate primary key error | |
| 22:18:05 | sean-k-mooney | it proably shoudl work but i dont think we have testeded it. | |
| 22:18:35 | mriedem | the new flow deals with port bindings on the port bindings resources in the port bindings API, the old flow just deals with the port resource and its binding:host_id attribute | |
| 22:18:38 | NewBruce | i will post a large log dump into the launchpad… brb | |
| 22:18:42 | mnaser | mriedem: nova triés to update the old binding to point to the new host, but because a new binding already exists, it blows up | |
| 22:18:59 | mnaser | Cause you can’t have a binding with same port/host combo | |