| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-02 | |||
| 21:32:37 | mriedem | ultimately it looks like https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L41 is faulty in that it doesn't account for pinned RPC versions | |
| 21:32:39 | NewBruce | i have a test machine, so i can override to at least test it | |
| 21:32:40 | mriedem | dansmith: yeah ^? | |
| 21:33:03 | mriedem | NewBruce: the kill switch would be to disable the 'Port Bindings Extended' neutron API extension until you're fully upgraded to rocky | |
| 21:33:08 | dansmith | mriedem: if conductor is new enough then the new objects are fine | |
| 21:33:23 | NewBruce | which would force into the old flow | |
| 21:33:24 | mriedem | dansmith: sounds like conductor in this case is rocky but some computes are queens | |
| 21:33:31 | dansmith | mriedem: which is fine | |
| 21:33:36 | NewBruce | dansmith correct | |
| 21:33:37 | mriedem | because of the bounce back thingy? | |
| 21:33:40 | dansmith | yes | |
| 21:33:46 | dansmith | the rpc pin has nothing to do with the objects | |
| 21:33:54 | dansmith | it's only really method signatures | |
| 21:33:55 | mriedem | sure | |
| 21:34:01 | mriedem | but, | |
| 21:34:16 | mriedem | conductor is doing some stuff in neutron based on what the compute service versions are, | |
| 21:34:28 | mriedem | and then setting fields in the objects that get passed to compute, | |
| 21:34:42 | mriedem | and then compute has logic that is based on if those new fields are set or not, | |
| 21:34:45 | mriedem | which it sounds like they might not be | |
| 21:35:18 | mriedem | tldr compute is probably not doing the right thing | |
| 21:35:58 | dansmith | well, if the backport is right then it should be okay, unless the conductor is doing something against neutron that the compute can't possibly handle properly | |
| 21:36:02 | dansmith | but if so, we broke upgrades | |
| 21:38:13 | mriedem | mnaser: did you hit https://bugs.launchpad.net/nova/+bug/1822884 when upgrading to rocky and doing live migrations with mixed computes (queens and rocky?) | |
| 21:38:14 | openstack | Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] | |
| 21:38:24 | mriedem | mnaser: or do you upgrade neutron after nova? | |
| 21:38:40 | mnaser | we upgrade neutron after nova usually mriedem | |
| 21:38:55 | mnaser | I’m very intimately familiar with that bug though. | |
| 21:39:03 | mriedem | first i've heard of it :/ | |
| 21:39:15 | mriedem | mnaser: ok but that's why you didn't hit it | |
| 21:39:21 | mnaser | I’ve been trying to help NewBruce nail it down forever. | |
| 21:39:36 | mnaser | But I’ve kinda struggled at finding the code path to replicate it | |
| 21:40:01 | mnaser | There are other environments where there is mixed compute AND Rocky network and this doesn’t happen | |
| 21:40:04 | mnaser | Right NewBruce ? | |
| 21:40:27 | NewBruce | yeah thats right mnaser | |
| 21:40:29 | mriedem | right he said, "2. OSA -> OSA uses the new flow (two entries which are cleaned up)" | |
| 21:40:40 | NewBruce | this is the first site we’ve seen it on... | |
| 21:41:43 | NewBruce | mriedem right; so if i grab a VM on an OSA node, and migrate it to an OSA node, watch ml2_port_bindings - ill see one port, briefly two ports, the profiles change, and then the second port entry removed | |
| 21:41:50 | NewBruce | as described in the bluebrint | |
| 21:42:23 | NewBruce | testing RDO - RDO, i don’t see that behaviour - just a single port entry in ml2_port_bindings throughout the entire migration | |
| 21:42:28 | mnaser | mriedem: NewBruce runs a few regions and this one is the only one where it works. With both queens to queens | |
| 21:42:41 | mnaser | Shit, what’s upgrade levels set to in the rdo notes? | |
| 21:42:57 | mnaser | Nodes | |
| 21:43:01 | mnaser | Sorry. I’m on mobile. | |
| 21:43:13 | NewBruce | mnaser auto | |
| 21:43:38 | NewBruce | mriedem and i chatted above that will pin it to the lowest version, as you suspected previously | |
| 21:43:41 | mnaser | ok, so yeah, I’m pretty torn on why it would actually do that in one environment and not another. | |
| 21:43:46 | mriedem | "all control nodes and net nodes are running OSA (Rocky), some compute are running RDO (Queens), some are RDO (Rocky) and the remaining are OSA (Rocky)." | |
| 21:44:01 | NewBruce | ill double check some other regions and see if they are diferent there | |
| 21:44:13 | mnaser | NewBruce: by any chance maybe the other regions aren’t pinned to lower version maybe? | |
| 21:44:21 | mnaser | And so this is why we don’t catch it? | |
| 21:44:30 | NewBruce | mriedem correct / mnaser ill double check | |
| 21:46:08 | NewBruce | quick (not exhaustive) check, upgrade levels is auto | |
| 21:47:48 | NewBruce | we have the same mix of service versions there as well (30 / 35) | |
| 21:49:08 | mnaser | Yet that issue somehow doesn’t happen there | |
| 21:49:14 | mnaser | So off | |
| 21:49:17 | mnaser | Odd | |
| 21:50:21 | mriedem | NewBruce: it fails in _post_live_migration right/ | |
| 21:50:22 | mriedem | ? | |
| 21:51:43 | mnaser | I think so. Rather than deleting the old binding and activating the new one, it tries to update the port binding | |
| 21:51:44 | NewBruce | mriedem correct | |
| 21:51:52 | NewBruce | mnaser correct | |
| 21:52:13 | mnaser | Which is what it would be with old port binding method. | |
| 21:52:49 | mriedem | right in the old method there is just one port binding | |
| 21:52:52 | mriedem | and we change the host on it | |
| 21:55:06 | NewBruce | mriedem and thats the exact behavior we see in RDO - RDO | |
| 21:55:57 | mnaser | So rocky to rocky and it does old port binding | |
| 21:56:32 | mnaser | NewBruce: can you restart a nova-compute with rdo and debug=true and double check the value of upgrade_levels in the output on startup of Oslo CFC | |
| 21:56:37 | mnaser | Cfg | |
| 21:56:43 | mnaser | In case rdo is doing weird stuff | |
| 21:57:14 | NewBruce | sure | |
| 21:57:36 | sean-k-mooney | sound like ye are making some progress on this | |
| 21:57:51 | NewBruce | [upgrade_levels] | |
| 21:57:51 | NewBruce | compute = auto | |
| 21:58:14 | NewBruce | debug=true | |
| 22:00:21 | sean-k-mooney | is the theory that RDO and OSA are not using the same compute RPC version due to there configs eventhough they are running more or less the same code | |
| 22:00:43 | sean-k-mooney | and as a result RDO is useing the old mechaniusm while osa is usign the new mechanisium? | |
| 22:03:10 | NewBruce | mnaser | |
| 22:03:11 | NewBruce | nova-compute.log:2019-04-02 23:59:08.448 10483 DEBUG oslo_service.service [req-4f4874a7-a967-49d1-a643-c59b856c5c61 - - - - -] upgrade_levels.compute = auto log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:3032 | |
| 22:03:58 | mnaser | I mean I dug through the conductor code a lot and it checks the source and dest to be above or at a certain later | |
| 22:04:03 | mnaser | Level | |
| 22:04:19 | mnaser | As far as I know the conductor creates the port bindings | |
| 22:04:36 | sean-k-mooney | not in all cases | |
| 22:04:42 | sean-k-mooney | it can be created by the compute node | |
| 22:05:37 | mnaser | Oh really? So I think in the new flow, the new compute node creates it right? | |
| 22:05:38 | mriedem | there is already an active port binding for the source host when you start the live migration, | |
| 22:05:52 | mriedem | in the new flow, conductor will create an inactive port binding for the dest host | |
| 22:06:20 | mriedem | and saves information about that new dest host port binding on the LiveMigrateData.vifs field that gets passed around to the computes | |
| 22:06:25 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L280-L284 | |
| 22:06:51 | mnaser | Right. That’s what I thought. I even asked NewBruce to set the “dont live migrate until vif is plugged” option that was introduced in rocky as false and moved to true in Stein | |
| 22:06:52 | sean-k-mooney | yes we do the version check the bind the ports on teh destiation | |
| 22:07:11 | mriedem | mnaser: that's unrelated to this | |
| 22:07:30 | mriedem | i mean it was part of the same bp | |
| 22:07:40 | mriedem | but doesn't rely on the active/inactive port bindings | |
| 22:07:46 | sean-k-mooney | mnaser: in the old flow in post livemigate on dest the compute node updated the port binding | |
| 22:08:10 | mriedem | correct that is the call here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6836 | |
| 22:08:42 | mnaser | NewBruce: can you update conductor to perhaps output the content of every statement in the if that sean-k-mooney linked? | |
| 22:09:08 | mnaser | And do and rdo to rdo migration and see which one of them is evaluating to false and not doing port bindings? | |
| 22:09:25 | mriedem | that code in post_live_migration_at_destination won't update the port binding host if it's already the specified host https://github.com/openstack/nova/blob/stable/rocky/nova/network/neutronv2/api.py#L3088 | |
| 22:09:30 | mnaser | Cause rdo to rdo seems to do old school bindings from what I understand | |
| 22:09:32 | mriedem | which is why it's a no-op with the new flow | |