Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-19
18:52:34 dansmith out of the pick_context_manager I mean
19:11:51 openstackgerrit Merged openstack/nova-specs master: Update placement aggregates spec to clarify generation handling https://review.openstack.org/548237
19:23:12 openstackgerrit Dan Smith proposed openstack/nova master: Add AggregateList.get_by_metadata() query method https://review.openstack.org/544728
19:23:13 openstackgerrit Dan Smith proposed openstack/nova master: Add request filter functionality to scheduler https://review.openstack.org/544730
19:23:13 openstackgerrit Dan Smith proposed openstack/nova master: Add aggregates list to Destination object https://review.openstack.org/544729
19:23:14 openstackgerrit Dan Smith proposed openstack/nova master: [WIP] Add require_tenant_aggregate request filter https://review.openstack.org/545002
19:23:14 openstackgerrit Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990
19:23:15 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282
19:35:52 mriedem debugging weirdness during live migratoin would be a lot easier if we had thread ids in the logs...
19:40:47 mriedem sean-k-mooney: do you know if unbinding a port makes it automatically go to status=DOWN?
19:44:19 mriedem sean-k-mooney: https://review.openstack.org/#/c/434870/11/nova/compute/manager.py@1126
19:44:55 mriedem seeing issues during live migration where we handle the migration completed lifecycle event from the hypervisor and start unbinding the port while we're also going through _post_live_migration
19:45:02 mriedem we blow up trying to unplug an unbound VIF
19:45:18 sean-k-mooney[m] Im not sure. The status of an unbound port does not really have a meaning as it is effectively just a db entry when unbound. E. G. It dose not exist on the datapath when unbound.
19:45:34 mriedem i ask because of https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186
19:45:50 mriedem i'm seeing mysterious network-vif-plugged and network-vif-unplugged events around the time of the failure
19:46:00 mriedem but they are coming before we unplug vifs on the source host
19:47:49 mriedem might be getting network-vif-plugged events on the source host from when they were plugged on the dest host in pre_live_migration,
19:48:00 mriedem the events are going to be routed to where the instance currrently lives, which is the source host until we switch
19:48:57 mriedem then we live migrate the guest,
19:49:10 mriedem get the 'migration completed' lifecycle event from the hypervisor and start unbinding the port,
19:49:11 dansmith mriedem: once there is a migration record, they'll be routed to both hosts
19:49:50 sean-k-mooney[m] Well in the old workflow we don't unbind the port. We just update the host in the binding details. Yes i think the vif plugged events are from when the destination vif is plugged
19:49:51 mriedem ok, i was only checking the source host logs, can check the dest host logs
19:50:18 mriedem sean-k-mooney[m]: this is the patch causing problems right now https://review.openstack.org/#/c/434870/
19:50:31 mriedem the thing that tried to do the early switch once were in post-copy
19:50:37 mriedem *once we were in
19:50:56 jroll jaypipes: dansmith: this is roughly what I recall from PTG about ironic RT stuff, does it make sense to you? https://review.openstack.org/#/c/545479/
19:53:33 mriedem dansmith: hmm, not seeing the same event routed to both hosts
19:53:48 mriedem but we should have a migration record at this point because i see on the dest host logs that we've plugged the vifs there in pre_live_migration
19:55:30 dansmith mriedem: hang on I'm piecing this back together in my head just a sec
19:55:36 dansmith jroll: ack, will look ater
19:56:00 dansmith mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4547-L4559
19:56:01 jroll thanks
19:56:04 mriedem fwiw, i don't see https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4556 in the logs
19:56:05 dansmith mriedem: it will require migration_context to be set
19:56:19 mriedem yeah that doesn't show up in the n-api logs http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-n-api.txt.gz
19:56:24 mriedem do we not lazy-load migration_context?
19:56:29 dansmith okay, so maybe we haven't set migration context yet?
19:56:35 dansmith mriedem: we query for it specifically in the api side:
19:56:46 dansmith https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_external_events.py#L59-L61
19:56:52 mriedem yup
19:57:24 dansmith more likely we haven't .save()d it yet or something
19:58:16 mriedem i only see apply_migration_context called in evacuate and resize flows, not live migration,
19:58:19 mriedem is it set in the RT?
19:58:37 dansmith not that I know of, so maybe we never set it for live migration?
19:59:09 sean-k-mooney[m] Right so it kindof makes sense when you activate the binding on the destination it deactivates the binding for the source which i guess results in vif unbound for the old host. How are we getting the vif on the source node that we pass to _nova_to_osvif_vif_unbound we should probably be cache the original vif object in the migration object
19:59:35 mriedem right we don't do a move_claim for live migration,
19:59:41 mriedem and RT.move_claim is what sets migration_context on the instance
20:00:22 mriedem sean-k-mooney[m]: heh way ahead of you https://review.openstack.org/#/c/515423/10/nova/objects/migrate_data.py@47 :)
20:01:43 sean-k-mooney[m] :)
20:02:13 mriedem artom: fyi re the migration_context thing, since you wrote https://review.openstack.org/#/c/371048/
20:02:28 mriedem but i think ^ was just for evac cases,
20:02:34 mriedem where the source is dead so we can't rpc there
20:02:42 dansmith blast from ye olde past
20:03:37 mriedem reminds me i need to touch up some dings in the woodwork with olde english
20:12:22 SamYaple other than a --hard reboot, is there a way to regenerate the libvirt.xml config? we changed the vnc listening address but we cannot migrate/live-migrate some instances due to pci passthrough of a sata controller
20:14:13 sean-k-mooney2 SamYaple: there is no way via nova to regenerate the xml that wont be equivalent to a reboot if you cant live migrate
20:14:44 sean-k-mooney2 SamYaple: you could likely just do a domain xml update out of band.
20:15:02 SamYaple yea unfortunately not, but not for technical reasons
20:15:07 SamYaple sean-k-mooney2: awesome. thought not. thanks for confirming
20:16:13 sean-k-mooney2 SamYaple: if you update the vnc adress in the nova.conf you will also need to restart nova compute as i dont think that is one of the mutable config options either.
20:17:35 SamYaple oh no worries. its all rolled out, just in the "applying" phase
20:17:42 SamYaple we already live-mgirated 90% of the instances
20:26:54 mriedem sean-k-mooney[m]: yeah so i think changing the port binding changes the status,
20:27:07 mriedem this is the source host neutron agent 2 seconds after nova updated the port binding to point at the dest host
20:27:09 mriedem http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_08_374523
20:27:15 mriedem binding host and status changes on the port in there
20:28:25 mriedem which according to this: https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186 will trigger a network-vif-unplugged event
20:29:39 sean-k-mooney2 right so looking at that ports binding_details its an ovs vif with ovs_hybrid_plug": true so os-vif will plug the vif on the destination ovs during pre livemigrate and neutron will emit the vif plugged events as it wires them up
20:30:01 mnaser does anyone know if there's some sort of variable you can use in docs to output the current version/release
20:30:11 mnaser reason - https://docs.openstack.org/nova/queens/cli/nova-manage.html -- this doc and many others have old versions listed
20:30:21 mnaser i can update it manually but it'll probably be forgotten again :>
20:31:34 mriedem mnaser: https://review.openstack.org/#/c/553903/
20:32:45 mnaser mriedem: oh well there you go, that's a solution
20:33:58 sean-k-mooney2 mriedem: ok so the vif plugged event is coming from the pre livemigrate when os-vif adds the port to ovs and the unplug event is coming form activateing the binding on the destenation host. is that the order it happens in
20:35:14 openstackgerrit Merged openstack/nova master: Don't filter out sibling sets with one core https://review.openstack.org/537361
20:37:11 mriedem sean-k-mooney2: yes i think so,
20:37:25 mriedem sean-k-mooney2: i'm confused as to why neutron thinks the port is unbound though
20:37:34 mriedem this is on both the source and dest neutron agent
20:37:35 mriedem http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_08_821754
20:37:38 mriedem http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/subnode-2/screen-q-agt.txt.gz#_Mar_19_14_25_09_150483
20:38:33 sean-k-mooney2 subnode-2 is the source node correct
20:38:44 mriedem yes
20:38:59 sean-k-mooney2 this is where the vif is pulled on the destination in prelivemigrate http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-n-cpu.txt.gz#_Mar_19_14_25_03_105245
20:41:35 mriedem this is where the source n-cpu changes the port binding host_id to the dest host http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/subnode-2/screen-n-cpu.txt.gz#_Mar_19_14_25_06_838681
20:41:44 mriedem in q-svc, i see this http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-svc.txt.gz#_Mar_19_14_25_06_991106
20:41:52 mriedem Device cd0afcf3-c654-488e-b578-6d3c45f3a5a5 not bound to the agent host ubuntu-xenial-rax-dfw-0003055550
20:42:09 mriedem so i guess that's why the vif type is 'unbound'?
20:42:22 mriedem but i'm not sure why that is because we plugged the vif on the dest host before all that
20:48:04 sean-k-mooney2 well this is where we plug it on the destination in the q-agt log http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_03_126952
20:49:08 sean-k-mooney2 and then right after it updates the vif_type to unbound http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_04_008234
20:49:50 openstackgerrit Merged openstack/nova master: Ensure emulator threads are always calculated https://review.openstack.org/537363
20:50:13 openstackgerrit Merged openstack/nova master: [libvirt] Add _get_numa_memnode() https://review.openstack.org/529906
20:50:41 mriedem sean-k-mooney2: hmm yeah wtf
20:50:46 sean-k-mooney2 so its the neutron agent on the destination that is seting the status to down and vif_type to unbound.
20:51:25 sean-k-mooney2 it kind of makes sense. at this point the vif is still bound on the host and we have not activated the vif on the dest but we just plugged it
20:51:29 mriedem this is all before nova hits the lifecycle post-copy event and switches the port binding, so presumably this already all happens today during live migration
20:51:57 openstackgerrit Ed Leafe proposed openstack/nova master: Address issues raised in adding member_of to GET /a-c https://review.openstack.org/554357
20:51:59 edleafe efried: ^^ gift-wrapped just for you

Earlier   Later