Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-02
20:29:06 eandersson > search_options["metadata"] = '{"my_key" : "bla" }'
20:29:12 eandersson https://review.openstack.org/#/c/408571/
20:29:18 eandersson Also, is the alternative to use tags?
20:43:31 efried eandersson: I have no idea, but did you look at the spec? http://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/add-whitelist-for-server-list-filter-sort-parameters.html
20:44:32 efried That appears to answer the first question. alex_xu could probably answer the second.
20:45:12 mriedem mnaser: i only see ca-ymq-2
20:45:48 mnaser huh? sjc1 is a separate region
20:46:05 mnaser OS_REGION_NAME=sjc1
20:46:51 eandersson I can't find any good documentation for tags, but maybe I just suck at googling :D
20:48:36 eandersson nova cli has things like > nova server-tag-add
20:48:57 eandersson but openstackcli has no mention of tags
20:49:00 eandersson > https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-create
20:49:10 mriedem mnaser: oh i thouht it was a zone
20:49:27 mriedem eandersson: openstack server set i think
20:49:35 mriedem https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-set
20:49:44 mriedem oh nvm
20:49:49 mriedem no tags support there either yet
20:49:55 mriedem consulting https://etherpad.openstack.org/p/compute-api-microversion-gap-in-osc
20:50:22 mriedem https://review.openstack.org/#/c/569386/
20:50:51 eandersson That explains it
20:51:51 mriedem eandersson: if you're going to be in denver https://www.openstack.org/summit/denver-2019/summit-schedule/events/23665/closing-compute-api-feature-gaps-in-the-openstack-cli
20:52:02 eandersson I will
20:52:19 mriedem i've started the etherpad for that session https://etherpad.openstack.org/p/DEN-osc-compute-api-gaps
21:09:50 NewBruce hej mriedem - we look to have stumbled on a bug in Neutrons Port Binding API, in a particular area of the code which you will probably be more intimately familliar with than I
21:10:03 NewBruce https://bugs.launchpad.net/nova/+bug/1822884 (cc sean-k-mooney)
21:10:04 openstack Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New]
21:12:04 openstackgerrit melanie witt proposed openstack/nova stable/stein: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/649454
21:12:51 NewBruce if you have a minute, id love your opinion - im trying to decide how much more debugging is worth before we move over to cold migrations
21:16:40 openstackgerrit Artom Lifshitz proposed openstack/nova master: Docs: emulator threads: clarify expected behavior https://review.openstack.org/649416
21:17:51 mriedem NewBruce: i don't know what the differences are between RDO and OSA which would cause issues here
21:18:18 mriedem as you said, "This is unexpected, as even in the RDO to RDO case, both nodes are Rocky and so the new process should be in use."
21:18:31 NewBruce hrmm, no - that seems to be our sticking point - i was chatting with sean-k-mooney and also mnaser on this
21:18:31 mriedem are you sure you're not hitting a case where one node is really queens?
21:19:17 NewBruce yeah, absolutely sure - as I am using a single test source node which is absolutely Rocky as well as the target
21:21:17 mriedem is 'Port Bindings Extended' showing up in the neutron api extension list in both the rdo and osa case?
21:22:07 mriedem if rdo is rocky, i don't know why rdo->rdo would use the old flow
21:22:16 mriedem unless the 'Port Bindings Extended' neutron extension is not showing up
21:23:48 NewBruce i can confirm that on the target it is enabled
21:23:52 mriedem the live migration task should only do the new flow with the 2nd deactivated port binding if (1) the neutron 'Port Bindings Extended' API extension is available, and (2) both the source and dest compute service versions are >= 37
21:23:58 mriedem *35
21:24:27 eandersson mriedem, solution was to just list all VMs in the project and then just filter on metadata :p
21:24:42 eandersson until tags is supported everywhere (openstackclient, terraform etc)
21:25:30 NewBruce yeah, so we looked at that as well;
21:25:31 NewBruce | cc-compute10-kna1 | nova-compute | 35 |
21:25:31 NewBruce | cc-compute29-kna1 | nova-compute | 35 |
21:25:31 NewBruce source = 10 (RDO Rocky) Target = 29 (OSA Rocky)
21:26:05 NewBruce however we do have other compute nodes in the same environment which have not been upgraded to Rocky yet; (service version 30)
21:27:38 mriedem NewBruce: do you have [upgrade_levels]/compute pinned to queens or something?
21:28:09 openstackgerrit Jared Winborne proposed openstack/nova master: Leave the brackets on Ceph Monitor IPv6 addresses for libguestfs https://review.openstack.org/649405
21:29:21 NewBruce [upgrade_levels]
21:29:21 NewBruce compute = auto
21:29:43 mriedem so that will pin to the lowest nova-compute service version in the deployment while you're upgrading,
21:29:56 mriedem so if you have queens computes, it will be using the queens rpc versions and backlevel the migrate_data object
21:30:11 mriedem which likely is dropping the vifs information for the 2nd deactivated port binding
21:30:27 NewBruce Aha….
21:30:27 mriedem that's my guess anyway
21:30:36 NewBruce ok, so can we override that?
21:31:43 NewBruce again, mnaser / sean-k-mooney suggested that might be the case and to upgrade the entire compute - which we were in the process of, however anothe error in a node caused a bit of strife due to broken libevent so’s in OVS so we halted
21:32:06 mriedem well you can pin the compute rpc api version to a specific release (queens) or even rpc api version (5.0) but that could break things where the controller is sending versions of objects to older queens computes that won't understand those chnages
21:32:11 NewBruce also, we have other sites where we have performed the same procedure, without issue
21:32:37 mriedem ultimately it looks like https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L41 is faulty in that it doesn't account for pinned RPC versions
21:32:39 NewBruce i have a test machine, so i can override to at least test it
21:32:40 mriedem dansmith: yeah ^?
21:33:03 mriedem NewBruce: the kill switch would be to disable the 'Port Bindings Extended' neutron API extension until you're fully upgraded to rocky
21:33:08 dansmith mriedem: if conductor is new enough then the new objects are fine
21:33:23 NewBruce which would force into the old flow
21:33:24 mriedem dansmith: sounds like conductor in this case is rocky but some computes are queens
21:33:31 dansmith mriedem: which is fine
21:33:36 NewBruce dansmith correct
21:33:37 mriedem because of the bounce back thingy?
21:33:40 dansmith yes
21:33:46 dansmith the rpc pin has nothing to do with the objects
21:33:54 dansmith it's only really method signatures
21:33:55 mriedem sure
21:34:01 mriedem but,
21:34:16 mriedem conductor is doing some stuff in neutron based on what the compute service versions are,
21:34:28 mriedem and then setting fields in the objects that get passed to compute,
21:34:42 mriedem and then compute has logic that is based on if those new fields are set or not,
21:34:45 mriedem which it sounds like they might not be
21:35:18 mriedem tldr compute is probably not doing the right thing
21:35:58 dansmith well, if the backport is right then it should be okay, unless the conductor is doing something against neutron that the compute can't possibly handle properly
21:36:02 dansmith but if so, we broke upgrades
21:38:13 mriedem mnaser: did you hit https://bugs.launchpad.net/nova/+bug/1822884 when upgrading to rocky and doing live migrations with mixed computes (queens and rocky?)
21:38:14 openstack Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New]
21:38:24 mriedem mnaser: or do you upgrade neutron after nova?
21:38:40 mnaser we upgrade neutron after nova usually mriedem
21:38:55 mnaser I’m very intimately familiar with that bug though.
21:39:03 mriedem first i've heard of it :/
21:39:15 mriedem mnaser: ok but that's why you didn't hit it
21:39:21 mnaser I’ve been trying to help NewBruce nail it down forever.
21:39:36 mnaser But I’ve kinda struggled at finding the code path to replicate it
21:40:01 mnaser There are other environments where there is mixed compute AND Rocky network and this doesn’t happen
21:40:04 mnaser Right NewBruce ?
21:40:27 NewBruce yeah thats right mnaser
21:40:29 mriedem right he said, "2. OSA -> OSA uses the new flow (two entries which are cleaned up)"
21:40:40 NewBruce this is the first site we’ve seen it on...
21:41:43 NewBruce mriedem right; so if i grab a VM on an OSA node, and migrate it to an OSA node, watch ml2_port_bindings - ill see one port, briefly two ports, the profiles change, and then the second port entry removed
21:41:50 NewBruce as described in the bluebrint
21:42:23 NewBruce testing RDO - RDO, i don’t see that behaviour - just a single port entry in ml2_port_bindings throughout the entire migration
21:42:28 mnaser mriedem: NewBruce runs a few regions and this one is the only one where it works. With both queens to queens

Earlier   Later