Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-19
18:11:22 melwitt I don't think it's different. I think this is a separate different bug, but was highlighting the comment where someone explained what was going on
18:11:54 dansmith yeah, that explanation makes sense but I'm not sure why we weren't hitting that immediately is what I mean
18:11:59 melwitt I think we fixed the scatter gather problem but this service record thing is separate and different
18:12:03 melwitt oh
18:13:22 melwitt I assume we were ... trying to think if this scenario is covered by grenade or how it wouldn't be
18:13:59 dansmith well, it should have manifested even in functional tests I would think
18:14:04 dansmith we must be in a nested call where we're already in a reader transaction and we call to do a write, right?
18:14:19 melwitt yeah, that would be the condition for that error being raised
18:14:21 dansmith blanking out the context dumps that so we get a new transaction for the write
18:18:08 melwitt you mean the way that the commenter patched nova/objects/service.py? yeah
18:18:47 dansmith yeah they used a new context
18:19:24 melwitt yeah
18:31:03 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553818
18:35:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317
18:37:34 mriedem dansmith: http://lists.openstack.org/pipermail/openstack-dev/2018-March/128487.html for the setuptools thing you were talking about before
18:37:37 mriedem pkg_version
18:37:57 dansmith ah cool
18:37:59 dansmith migi: ^
18:38:30 dansmith mriedem: did you see my comment on that patch for the obj_relationships thing?
18:39:13 mriedem not yet
18:44:31 melwitt dansmith: from what I can tell, we only have unit test coverage for the uuid generation for service where things are mocked (and would anyway be a write to begin with), or calling _from_db_object selectively https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_service.py#L145
18:44:54 melwitt to hit this, we'd have to have a functional test scenario where we read an already existing service record that has no uuid
18:45:03 melwitt I can try writing one
18:47:24 dansmith hmm, I thought we did
18:47:30 dansmith because that was mriedem
18:47:36 dansmith and he looooves functional tests
18:47:56 dansmith I would also have expected to hear more noise about this
18:48:20 dansmith like, wouldn't the first compute startup after that patch merged cause nova-compute to not update its service record?
18:49:43 mriedem it was dtp with me helping, but it's a pretty established pattern,
18:49:53 mriedem i.e. he copied it from compute_nodes.uuid, and aggregates.uuid
18:49:57 dansmith yeah
18:50:04 dansmith was just lookin gup the commit
18:50:12 mriedem also https://github.com/dpeschman
18:50:17 mriedem https://avatars3.githubusercontent.com/u/145098?s=400&v=4
18:50:26 mriedem i would like to know what's going on there
18:50:36 melwitt lol
18:51:59 dansmith even still,
18:52:11 dansmith we're out of the db context at that point, so I'm not sure why this is happening anyway
18:52:34 dansmith out of the pick_context_manager I mean
19:11:51 openstackgerrit Merged openstack/nova-specs master: Update placement aggregates spec to clarify generation handling https://review.openstack.org/548237
19:23:12 openstackgerrit Dan Smith proposed openstack/nova master: Add AggregateList.get_by_metadata() query method https://review.openstack.org/544728
19:23:13 openstackgerrit Dan Smith proposed openstack/nova master: Add aggregates list to Destination object https://review.openstack.org/544729
19:23:13 openstackgerrit Dan Smith proposed openstack/nova master: Add request filter functionality to scheduler https://review.openstack.org/544730
19:23:14 openstackgerrit Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990
19:23:14 openstackgerrit Dan Smith proposed openstack/nova master: [WIP] Add require_tenant_aggregate request filter https://review.openstack.org/545002
19:23:15 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282
19:35:52 mriedem debugging weirdness during live migratoin would be a lot easier if we had thread ids in the logs...
19:40:47 mriedem sean-k-mooney: do you know if unbinding a port makes it automatically go to status=DOWN?
19:44:19 mriedem sean-k-mooney: https://review.openstack.org/#/c/434870/11/nova/compute/manager.py@1126
19:44:55 mriedem seeing issues during live migration where we handle the migration completed lifecycle event from the hypervisor and start unbinding the port while we're also going through _post_live_migration
19:45:02 mriedem we blow up trying to unplug an unbound VIF
19:45:18 sean-k-mooney[m] Im not sure. The status of an unbound port does not really have a meaning as it is effectively just a db entry when unbound. E. G. It dose not exist on the datapath when unbound.
19:45:34 mriedem i ask because of https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186
19:45:50 mriedem i'm seeing mysterious network-vif-plugged and network-vif-unplugged events around the time of the failure
19:46:00 mriedem but they are coming before we unplug vifs on the source host
19:47:49 mriedem might be getting network-vif-plugged events on the source host from when they were plugged on the dest host in pre_live_migration,
19:48:00 mriedem the events are going to be routed to where the instance currrently lives, which is the source host until we switch
19:48:57 mriedem then we live migrate the guest,
19:49:10 mriedem get the 'migration completed' lifecycle event from the hypervisor and start unbinding the port,
19:49:11 dansmith mriedem: once there is a migration record, they'll be routed to both hosts
19:49:50 sean-k-mooney[m] Well in the old workflow we don't unbind the port. We just update the host in the binding details. Yes i think the vif plugged events are from when the destination vif is plugged
19:49:51 mriedem ok, i was only checking the source host logs, can check the dest host logs
19:50:18 mriedem sean-k-mooney[m]: this is the patch causing problems right now https://review.openstack.org/#/c/434870/
19:50:31 mriedem the thing that tried to do the early switch once were in post-copy
19:50:37 mriedem *once we were in
19:50:56 jroll jaypipes: dansmith: this is roughly what I recall from PTG about ironic RT stuff, does it make sense to you? https://review.openstack.org/#/c/545479/
19:53:33 mriedem dansmith: hmm, not seeing the same event routed to both hosts
19:53:48 mriedem but we should have a migration record at this point because i see on the dest host logs that we've plugged the vifs there in pre_live_migration
19:55:30 dansmith mriedem: hang on I'm piecing this back together in my head just a sec
19:55:36 dansmith jroll: ack, will look ater
19:56:00 dansmith mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4547-L4559
19:56:01 jroll thanks
19:56:04 mriedem fwiw, i don't see https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4556 in the logs
19:56:05 dansmith mriedem: it will require migration_context to be set
19:56:19 mriedem yeah that doesn't show up in the n-api logs http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-n-api.txt.gz
19:56:24 mriedem do we not lazy-load migration_context?
19:56:29 dansmith okay, so maybe we haven't set migration context yet?
19:56:35 dansmith mriedem: we query for it specifically in the api side:
19:56:46 dansmith https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_external_events.py#L59-L61
19:56:52 mriedem yup
19:57:24 dansmith more likely we haven't .save()d it yet or something
19:58:16 mriedem i only see apply_migration_context called in evacuate and resize flows, not live migration,
19:58:19 mriedem is it set in the RT?
19:58:37 dansmith not that I know of, so maybe we never set it for live migration?
19:59:09 sean-k-mooney[m] Right so it kindof makes sense when you activate the binding on the destination it deactivates the binding for the source which i guess results in vif unbound for the old host. How are we getting the vif on the source node that we pass to _nova_to_osvif_vif_unbound we should probably be cache the original vif object in the migration object
19:59:35 mriedem right we don't do a move_claim for live migration,
19:59:41 mriedem and RT.move_claim is what sets migration_context on the instance
20:00:22 mriedem sean-k-mooney[m]: heh way ahead of you https://review.openstack.org/#/c/515423/10/nova/objects/migrate_data.py@47 :)
20:01:43 sean-k-mooney[m] :)
20:02:13 mriedem artom: fyi re the migration_context thing, since you wrote https://review.openstack.org/#/c/371048/
20:02:28 mriedem but i think ^ was just for evac cases,
20:02:34 mriedem where the source is dead so we can't rpc there
20:02:42 dansmith blast from ye olde past
20:03:37 mriedem reminds me i need to touch up some dings in the woodwork with olde english
20:12:22 SamYaple other than a --hard reboot, is there a way to regenerate the libvirt.xml config? we changed the vnc listening address but we cannot migrate/live-migrate some instances due to pci passthrough of a sata controller
20:14:13 sean-k-mooney2 SamYaple: there is no way via nova to regenerate the xml that wont be equivalent to a reboot if you cant live migrate
20:14:44 sean-k-mooney2 SamYaple: you could likely just do a domain xml update out of band.
20:15:02 SamYaple yea unfortunately not, but not for technical reasons
20:15:07 SamYaple sean-k-mooney2: awesome. thought not. thanks for confirming
20:16:13 sean-k-mooney2 SamYaple: if you update the vnc adress in the nova.conf you will also need to restart nova compute as i dont think that is one of the mutable config options either.

Earlier   Later