| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-02 | |||
| 21:23:48 | NewBruce | i can confirm that on the target it is enabled | |
| 21:23:52 | mriedem | the live migration task should only do the new flow with the 2nd deactivated port binding if (1) the neutron 'Port Bindings Extended' API extension is available, and (2) both the source and dest compute service versions are >= 37 | |
| 21:23:58 | mriedem | *35 | |
| 21:24:27 | eandersson | mriedem, solution was to just list all VMs in the project and then just filter on metadata :p | |
| 21:24:42 | eandersson | until tags is supported everywhere (openstackclient, terraform etc) | |
| 21:25:30 | NewBruce | yeah, so we looked at that as well; | |
| 21:25:31 | NewBruce | | cc-compute10-kna1 | nova-compute | 35 | | |
| 21:25:31 | NewBruce | | cc-compute29-kna1 | nova-compute | 35 | | |
| 21:25:31 | NewBruce | source = 10 (RDO Rocky) Target = 29 (OSA Rocky) | |
| 21:26:05 | NewBruce | however we do have other compute nodes in the same environment which have not been upgraded to Rocky yet; (service version 30) | |
| 21:27:38 | mriedem | NewBruce: do you have [upgrade_levels]/compute pinned to queens or something? | |
| 21:28:09 | openstackgerrit | Jared Winborne proposed openstack/nova master: Leave the brackets on Ceph Monitor IPv6 addresses for libguestfs https://review.openstack.org/649405 | |
| 21:29:21 | NewBruce | [upgrade_levels] | |
| 21:29:21 | NewBruce | compute = auto | |
| 21:29:43 | mriedem | so that will pin to the lowest nova-compute service version in the deployment while you're upgrading, | |
| 21:29:56 | mriedem | so if you have queens computes, it will be using the queens rpc versions and backlevel the migrate_data object | |
| 21:30:11 | mriedem | which likely is dropping the vifs information for the 2nd deactivated port binding | |
| 21:30:27 | NewBruce | Aha…. | |
| 21:30:27 | mriedem | that's my guess anyway | |
| 21:30:36 | NewBruce | ok, so can we override that? | |
| 21:31:43 | NewBruce | again, mnaser / sean-k-mooney suggested that might be the case and to upgrade the entire compute - which we were in the process of, however anothe error in a node caused a bit of strife due to broken libevent so’s in OVS so we halted | |
| 21:32:06 | mriedem | well you can pin the compute rpc api version to a specific release (queens) or even rpc api version (5.0) but that could break things where the controller is sending versions of objects to older queens computes that won't understand those chnages | |
| 21:32:11 | NewBruce | also, we have other sites where we have performed the same procedure, without issue | |
| 21:32:37 | mriedem | ultimately it looks like https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L41 is faulty in that it doesn't account for pinned RPC versions | |
| 21:32:39 | NewBruce | i have a test machine, so i can override to at least test it | |
| 21:32:40 | mriedem | dansmith: yeah ^? | |
| 21:33:03 | mriedem | NewBruce: the kill switch would be to disable the 'Port Bindings Extended' neutron API extension until you're fully upgraded to rocky | |
| 21:33:08 | dansmith | mriedem: if conductor is new enough then the new objects are fine | |
| 21:33:23 | NewBruce | which would force into the old flow | |
| 21:33:24 | mriedem | dansmith: sounds like conductor in this case is rocky but some computes are queens | |
| 21:33:31 | dansmith | mriedem: which is fine | |
| 21:33:36 | NewBruce | dansmith correct | |
| 21:33:37 | mriedem | because of the bounce back thingy? | |
| 21:33:40 | dansmith | yes | |
| 21:33:46 | dansmith | the rpc pin has nothing to do with the objects | |
| 21:33:54 | dansmith | it's only really method signatures | |
| 21:33:55 | mriedem | sure | |
| 21:34:01 | mriedem | but, | |
| 21:34:16 | mriedem | conductor is doing some stuff in neutron based on what the compute service versions are, | |
| 21:34:28 | mriedem | and then setting fields in the objects that get passed to compute, | |
| 21:34:42 | mriedem | and then compute has logic that is based on if those new fields are set or not, | |
| 21:34:45 | mriedem | which it sounds like they might not be | |
| 21:35:18 | mriedem | tldr compute is probably not doing the right thing | |
| 21:35:58 | dansmith | well, if the backport is right then it should be okay, unless the conductor is doing something against neutron that the compute can't possibly handle properly | |
| 21:36:02 | dansmith | but if so, we broke upgrades | |
| 21:38:13 | mriedem | mnaser: did you hit https://bugs.launchpad.net/nova/+bug/1822884 when upgrading to rocky and doing live migrations with mixed computes (queens and rocky?) | |
| 21:38:14 | openstack | Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] | |
| 21:38:24 | mriedem | mnaser: or do you upgrade neutron after nova? | |
| 21:38:40 | mnaser | we upgrade neutron after nova usually mriedem | |
| 21:38:55 | mnaser | I’m very intimately familiar with that bug though. | |
| 21:39:03 | mriedem | first i've heard of it :/ | |
| 21:39:15 | mriedem | mnaser: ok but that's why you didn't hit it | |
| 21:39:21 | mnaser | I’ve been trying to help NewBruce nail it down forever. | |
| 21:39:36 | mnaser | But I’ve kinda struggled at finding the code path to replicate it | |
| 21:40:01 | mnaser | There are other environments where there is mixed compute AND Rocky network and this doesn’t happen | |
| 21:40:04 | mnaser | Right NewBruce ? | |
| 21:40:27 | NewBruce | yeah thats right mnaser | |
| 21:40:29 | mriedem | right he said, "2. OSA -> OSA uses the new flow (two entries which are cleaned up)" | |
| 21:40:40 | NewBruce | this is the first site we’ve seen it on... | |
| 21:41:43 | NewBruce | mriedem right; so if i grab a VM on an OSA node, and migrate it to an OSA node, watch ml2_port_bindings - ill see one port, briefly two ports, the profiles change, and then the second port entry removed | |
| 21:41:50 | NewBruce | as described in the bluebrint | |
| 21:42:23 | NewBruce | testing RDO - RDO, i don’t see that behaviour - just a single port entry in ml2_port_bindings throughout the entire migration | |
| 21:42:28 | mnaser | mriedem: NewBruce runs a few regions and this one is the only one where it works. With both queens to queens | |
| 21:42:41 | mnaser | Shit, what’s upgrade levels set to in the rdo notes? | |
| 21:42:57 | mnaser | Nodes | |
| 21:43:01 | mnaser | Sorry. I’m on mobile. | |
| 21:43:13 | NewBruce | mnaser auto | |
| 21:43:38 | NewBruce | mriedem and i chatted above that will pin it to the lowest version, as you suspected previously | |
| 21:43:41 | mnaser | ok, so yeah, I’m pretty torn on why it would actually do that in one environment and not another. | |
| 21:43:46 | mriedem | "all control nodes and net nodes are running OSA (Rocky), some compute are running RDO (Queens), some are RDO (Rocky) and the remaining are OSA (Rocky)." | |
| 21:44:01 | NewBruce | ill double check some other regions and see if they are diferent there | |
| 21:44:13 | mnaser | NewBruce: by any chance maybe the other regions aren’t pinned to lower version maybe? | |
| 21:44:21 | mnaser | And so this is why we don’t catch it? | |
| 21:44:30 | NewBruce | mriedem correct / mnaser ill double check | |
| 21:46:08 | NewBruce | quick (not exhaustive) check, upgrade levels is auto | |
| 21:47:48 | NewBruce | we have the same mix of service versions there as well (30 / 35) | |
| 21:49:08 | mnaser | Yet that issue somehow doesn’t happen there | |
| 21:49:14 | mnaser | So off | |
| 21:49:17 | mnaser | Odd | |
| 21:50:21 | mriedem | NewBruce: it fails in _post_live_migration right/ | |
| 21:50:22 | mriedem | ? | |
| 21:51:43 | mnaser | I think so. Rather than deleting the old binding and activating the new one, it tries to update the port binding | |
| 21:51:44 | NewBruce | mriedem correct | |
| 21:51:52 | NewBruce | mnaser correct | |
| 21:52:13 | mnaser | Which is what it would be with old port binding method. | |
| 21:52:49 | mriedem | right in the old method there is just one port binding | |
| 21:52:52 | mriedem | and we change the host on it | |
| 21:55:06 | NewBruce | mriedem and thats the exact behavior we see in RDO - RDO | |
| 21:55:57 | mnaser | So rocky to rocky and it does old port binding | |
| 21:56:32 | mnaser | NewBruce: can you restart a nova-compute with rdo and debug=true and double check the value of upgrade_levels in the output on startup of Oslo CFC | |
| 21:56:37 | mnaser | Cfg | |
| 21:56:43 | mnaser | In case rdo is doing weird stuff | |
| 21:57:14 | NewBruce | sure | |
| 21:57:36 | sean-k-mooney | sound like ye are making some progress on this | |
| 21:57:51 | NewBruce | [upgrade_levels] | |
| 21:57:51 | NewBruce | compute = auto | |
| 21:58:14 | NewBruce | debug=true | |
| 22:00:21 | sean-k-mooney | is the theory that RDO and OSA are not using the same compute RPC version due to there configs eventhough they are running more or less the same code | |
| 22:00:43 | sean-k-mooney | and as a result RDO is useing the old mechaniusm while osa is usign the new mechanisium? | |
| 22:03:10 | NewBruce | mnaser | |