Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-19
18:50:12 mriedem also https://github.com/dpeschman
18:50:17 mriedem https://avatars3.githubusercontent.com/u/145098?s=400&v=4
18:50:26 mriedem i would like to know what's going on there
18:50:36 melwitt lol
18:51:59 dansmith even still,
18:52:11 dansmith we're out of the db context at that point, so I'm not sure why this is happening anyway
18:52:34 dansmith out of the pick_context_manager I mean
19:11:51 openstackgerrit Merged openstack/nova-specs master: Update placement aggregates spec to clarify generation handling https://review.openstack.org/548237
19:23:12 openstackgerrit Dan Smith proposed openstack/nova master: Add AggregateList.get_by_metadata() query method https://review.openstack.org/544728
19:23:13 openstackgerrit Dan Smith proposed openstack/nova master: Add aggregates list to Destination object https://review.openstack.org/544729
19:23:13 openstackgerrit Dan Smith proposed openstack/nova master: Add request filter functionality to scheduler https://review.openstack.org/544730
19:23:14 openstackgerrit Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990
19:23:14 openstackgerrit Dan Smith proposed openstack/nova master: [WIP] Add require_tenant_aggregate request filter https://review.openstack.org/545002
19:23:15 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282
19:35:52 mriedem debugging weirdness during live migratoin would be a lot easier if we had thread ids in the logs...
19:40:47 mriedem sean-k-mooney: do you know if unbinding a port makes it automatically go to status=DOWN?
19:44:19 mriedem sean-k-mooney: https://review.openstack.org/#/c/434870/11/nova/compute/manager.py@1126
19:44:55 mriedem seeing issues during live migration where we handle the migration completed lifecycle event from the hypervisor and start unbinding the port while we're also going through _post_live_migration
19:45:02 mriedem we blow up trying to unplug an unbound VIF
19:45:18 sean-k-mooney[m] Im not sure. The status of an unbound port does not really have a meaning as it is effectively just a db entry when unbound. E. G. It dose not exist on the datapath when unbound.
19:45:34 mriedem i ask because of https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186
19:45:50 mriedem i'm seeing mysterious network-vif-plugged and network-vif-unplugged events around the time of the failure
19:46:00 mriedem but they are coming before we unplug vifs on the source host
19:47:49 mriedem might be getting network-vif-plugged events on the source host from when they were plugged on the dest host in pre_live_migration,
19:48:00 mriedem the events are going to be routed to where the instance currrently lives, which is the source host until we switch
19:48:57 mriedem then we live migrate the guest,
19:49:10 mriedem get the 'migration completed' lifecycle event from the hypervisor and start unbinding the port,
19:49:11 dansmith mriedem: once there is a migration record, they'll be routed to both hosts
19:49:50 sean-k-mooney[m] Well in the old workflow we don't unbind the port. We just update the host in the binding details. Yes i think the vif plugged events are from when the destination vif is plugged
19:49:51 mriedem ok, i was only checking the source host logs, can check the dest host logs
19:50:18 mriedem sean-k-mooney[m]: this is the patch causing problems right now https://review.openstack.org/#/c/434870/
19:50:31 mriedem the thing that tried to do the early switch once were in post-copy
19:50:37 mriedem *once we were in
19:50:56 jroll jaypipes: dansmith: this is roughly what I recall from PTG about ironic RT stuff, does it make sense to you? https://review.openstack.org/#/c/545479/
19:53:33 mriedem dansmith: hmm, not seeing the same event routed to both hosts
19:53:48 mriedem but we should have a migration record at this point because i see on the dest host logs that we've plugged the vifs there in pre_live_migration
19:55:30 dansmith mriedem: hang on I'm piecing this back together in my head just a sec
19:55:36 dansmith jroll: ack, will look ater
19:56:00 dansmith mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4547-L4559
19:56:01 jroll thanks
19:56:04 mriedem fwiw, i don't see https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4556 in the logs
19:56:05 dansmith mriedem: it will require migration_context to be set
19:56:19 mriedem yeah that doesn't show up in the n-api logs http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-n-api.txt.gz
19:56:24 mriedem do we not lazy-load migration_context?
19:56:29 dansmith okay, so maybe we haven't set migration context yet?
19:56:35 dansmith mriedem: we query for it specifically in the api side:
19:56:46 dansmith https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_external_events.py#L59-L61
19:56:52 mriedem yup
19:57:24 dansmith more likely we haven't .save()d it yet or something
19:58:16 mriedem i only see apply_migration_context called in evacuate and resize flows, not live migration,
19:58:19 mriedem is it set in the RT?
19:58:37 dansmith not that I know of, so maybe we never set it for live migration?
19:59:09 sean-k-mooney[m] Right so it kindof makes sense when you activate the binding on the destination it deactivates the binding for the source which i guess results in vif unbound for the old host. How are we getting the vif on the source node that we pass to _nova_to_osvif_vif_unbound we should probably be cache the original vif object in the migration object
19:59:35 mriedem right we don't do a move_claim for live migration,
19:59:41 mriedem and RT.move_claim is what sets migration_context on the instance
20:00:22 mriedem sean-k-mooney[m]: heh way ahead of you https://review.openstack.org/#/c/515423/10/nova/objects/migrate_data.py@47 :)
20:01:43 sean-k-mooney[m] :)
20:02:13 mriedem artom: fyi re the migration_context thing, since you wrote https://review.openstack.org/#/c/371048/
20:02:28 mriedem but i think ^ was just for evac cases,
20:02:34 mriedem where the source is dead so we can't rpc there
20:02:42 dansmith blast from ye olde past
20:03:37 mriedem reminds me i need to touch up some dings in the woodwork with olde english
20:12:22 SamYaple other than a --hard reboot, is there a way to regenerate the libvirt.xml config? we changed the vnc listening address but we cannot migrate/live-migrate some instances due to pci passthrough of a sata controller
20:14:13 sean-k-mooney2 SamYaple: there is no way via nova to regenerate the xml that wont be equivalent to a reboot if you cant live migrate
20:14:44 sean-k-mooney2 SamYaple: you could likely just do a domain xml update out of band.
20:15:02 SamYaple yea unfortunately not, but not for technical reasons
20:15:07 SamYaple sean-k-mooney2: awesome. thought not. thanks for confirming
20:16:13 sean-k-mooney2 SamYaple: if you update the vnc adress in the nova.conf you will also need to restart nova compute as i dont think that is one of the mutable config options either.
20:17:35 SamYaple oh no worries. its all rolled out, just in the "applying" phase
20:17:42 SamYaple we already live-mgirated 90% of the instances
20:26:54 mriedem sean-k-mooney[m]: yeah so i think changing the port binding changes the status,
20:27:07 mriedem this is the source host neutron agent 2 seconds after nova updated the port binding to point at the dest host
20:27:09 mriedem http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_08_374523
20:27:15 mriedem binding host and status changes on the port in there
20:28:25 mriedem which according to this: https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186 will trigger a network-vif-unplugged event
20:29:39 sean-k-mooney2 right so looking at that ports binding_details its an ovs vif with ovs_hybrid_plug": true so os-vif will plug the vif on the destination ovs during pre livemigrate and neutron will emit the vif plugged events as it wires them up
20:30:01 mnaser does anyone know if there's some sort of variable you can use in docs to output the current version/release
20:30:11 mnaser reason - https://docs.openstack.org/nova/queens/cli/nova-manage.html -- this doc and many others have old versions listed
20:30:21 mnaser i can update it manually but it'll probably be forgotten again :>
20:31:34 mriedem mnaser: https://review.openstack.org/#/c/553903/
20:32:45 mnaser mriedem: oh well there you go, that's a solution
20:33:58 sean-k-mooney2 mriedem: ok so the vif plugged event is coming from the pre livemigrate when os-vif adds the port to ovs and the unplug event is coming form activateing the binding on the destenation host. is that the order it happens in
20:35:14 openstackgerrit Merged openstack/nova master: Don't filter out sibling sets with one core https://review.openstack.org/537361
20:37:11 mriedem sean-k-mooney2: yes i think so,
20:37:25 mriedem sean-k-mooney2: i'm confused as to why neutron thinks the port is unbound though
20:37:34 mriedem this is on both the source and dest neutron agent
20:37:35 mriedem http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_08_821754
20:37:38 mriedem http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/subnode-2/screen-q-agt.txt.gz#_Mar_19_14_25_09_150483
20:38:33 sean-k-mooney2 subnode-2 is the source node correct
20:38:44 mriedem yes
20:38:59 sean-k-mooney2 this is where the vif is pulled on the destination in prelivemigrate http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-n-cpu.txt.gz#_Mar_19_14_25_03_105245
20:41:35 mriedem this is where the source n-cpu changes the port binding host_id to the dest host http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/subnode-2/screen-n-cpu.txt.gz#_Mar_19_14_25_06_838681
20:41:44 mriedem in q-svc, i see this http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-svc.txt.gz#_Mar_19_14_25_06_991106
20:41:52 mriedem Device cd0afcf3-c654-488e-b578-6d3c45f3a5a5 not bound to the agent host ubuntu-xenial-rax-dfw-0003055550
20:42:09 mriedem so i guess that's why the vif type is 'unbound'?
20:42:22 mriedem but i'm not sure why that is because we plugged the vif on the dest host before all that
20:48:04 sean-k-mooney2 well this is where we plug it on the destination in the q-agt log http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_03_126952
20:49:08 sean-k-mooney2 and then right after it updates the vif_type to unbound http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_04_008234
20:49:50 openstackgerrit Merged openstack/nova master: Ensure emulator threads are always calculated https://review.openstack.org/537363
20:50:13 openstackgerrit Merged openstack/nova master: [libvirt] Add _get_numa_memnode() https://review.openstack.org/529906

Earlier   Later