| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-19 | |||
| 18:11:59 | melwitt | I think we fixed the scatter gather problem but this service record thing is separate and different | |
| 18:12:03 | melwitt | oh | |
| 18:13:22 | melwitt | I assume we were ... trying to think if this scenario is covered by grenade or how it wouldn't be | |
| 18:13:59 | dansmith | well, it should have manifested even in functional tests I would think | |
| 18:14:04 | dansmith | we must be in a nested call where we're already in a reader transaction and we call to do a write, right? | |
| 18:14:19 | melwitt | yeah, that would be the condition for that error being raised | |
| 18:14:21 | dansmith | blanking out the context dumps that so we get a new transaction for the write | |
| 18:18:08 | melwitt | you mean the way that the commenter patched nova/objects/service.py? yeah | |
| 18:18:47 | dansmith | yeah they used a new context | |
| 18:19:24 | melwitt | yeah | |
| 18:31:03 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553818 | |
| 18:35:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | |
| 18:37:34 | mriedem | dansmith: http://lists.openstack.org/pipermail/openstack-dev/2018-March/128487.html for the setuptools thing you were talking about before | |
| 18:37:37 | mriedem | pkg_version | |
| 18:37:57 | dansmith | ah cool | |
| 18:37:59 | dansmith | migi: ^ | |
| 18:38:30 | dansmith | mriedem: did you see my comment on that patch for the obj_relationships thing? | |
| 18:39:13 | mriedem | not yet | |
| 18:44:31 | melwitt | dansmith: from what I can tell, we only have unit test coverage for the uuid generation for service where things are mocked (and would anyway be a write to begin with), or calling _from_db_object selectively https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_service.py#L145 | |
| 18:44:54 | melwitt | to hit this, we'd have to have a functional test scenario where we read an already existing service record that has no uuid | |
| 18:45:03 | melwitt | I can try writing one | |
| 18:47:24 | dansmith | hmm, I thought we did | |
| 18:47:30 | dansmith | because that was mriedem | |
| 18:47:36 | dansmith | and he looooves functional tests | |
| 18:47:56 | dansmith | I would also have expected to hear more noise about this | |
| 18:48:20 | dansmith | like, wouldn't the first compute startup after that patch merged cause nova-compute to not update its service record? | |
| 18:49:43 | mriedem | it was dtp with me helping, but it's a pretty established pattern, | |
| 18:49:53 | mriedem | i.e. he copied it from compute_nodes.uuid, and aggregates.uuid | |
| 18:49:57 | dansmith | yeah | |
| 18:50:04 | dansmith | was just lookin gup the commit | |
| 18:50:12 | mriedem | also https://github.com/dpeschman | |
| 18:50:17 | mriedem | https://avatars3.githubusercontent.com/u/145098?s=400&v=4 | |
| 18:50:26 | mriedem | i would like to know what's going on there | |
| 18:50:36 | melwitt | lol | |
| 18:51:59 | dansmith | even still, | |
| 18:52:11 | dansmith | we're out of the db context at that point, so I'm not sure why this is happening anyway | |
| 18:52:34 | dansmith | out of the pick_context_manager I mean | |
| 19:11:51 | openstackgerrit | Merged openstack/nova-specs master: Update placement aggregates spec to clarify generation handling https://review.openstack.org/548237 | |
| 19:23:12 | openstackgerrit | Dan Smith proposed openstack/nova master: Add AggregateList.get_by_metadata() query method https://review.openstack.org/544728 | |
| 19:23:13 | openstackgerrit | Dan Smith proposed openstack/nova master: Add request filter functionality to scheduler https://review.openstack.org/544730 | |
| 19:23:13 | openstackgerrit | Dan Smith proposed openstack/nova master: Add aggregates list to Destination object https://review.openstack.org/544729 | |
| 19:23:14 | openstackgerrit | Dan Smith proposed openstack/nova master: [WIP] Add require_tenant_aggregate request filter https://review.openstack.org/545002 | |
| 19:23:14 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990 | |
| 19:23:15 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282 | |
| 19:35:52 | mriedem | debugging weirdness during live migratoin would be a lot easier if we had thread ids in the logs... | |
| 19:40:47 | mriedem | sean-k-mooney: do you know if unbinding a port makes it automatically go to status=DOWN? | |
| 19:44:19 | mriedem | sean-k-mooney: https://review.openstack.org/#/c/434870/11/nova/compute/manager.py@1126 | |
| 19:44:55 | mriedem | seeing issues during live migration where we handle the migration completed lifecycle event from the hypervisor and start unbinding the port while we're also going through _post_live_migration | |
| 19:45:02 | mriedem | we blow up trying to unplug an unbound VIF | |
| 19:45:18 | sean-k-mooney[m] | Im not sure. The status of an unbound port does not really have a meaning as it is effectively just a db entry when unbound. E. G. It dose not exist on the datapath when unbound. | |
| 19:45:34 | mriedem | i ask because of https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186 | |
| 19:45:50 | mriedem | i'm seeing mysterious network-vif-plugged and network-vif-unplugged events around the time of the failure | |
| 19:46:00 | mriedem | but they are coming before we unplug vifs on the source host | |
| 19:47:49 | mriedem | might be getting network-vif-plugged events on the source host from when they were plugged on the dest host in pre_live_migration, | |
| 19:48:00 | mriedem | the events are going to be routed to where the instance currrently lives, which is the source host until we switch | |
| 19:48:57 | mriedem | then we live migrate the guest, | |
| 19:49:10 | mriedem | get the 'migration completed' lifecycle event from the hypervisor and start unbinding the port, | |
| 19:49:11 | dansmith | mriedem: once there is a migration record, they'll be routed to both hosts | |
| 19:49:50 | sean-k-mooney[m] | Well in the old workflow we don't unbind the port. We just update the host in the binding details. Yes i think the vif plugged events are from when the destination vif is plugged | |
| 19:49:51 | mriedem | ok, i was only checking the source host logs, can check the dest host logs | |
| 19:50:18 | mriedem | sean-k-mooney[m]: this is the patch causing problems right now https://review.openstack.org/#/c/434870/ | |
| 19:50:31 | mriedem | the thing that tried to do the early switch once were in post-copy | |
| 19:50:37 | mriedem | *once we were in | |
| 19:50:56 | jroll | jaypipes: dansmith: this is roughly what I recall from PTG about ironic RT stuff, does it make sense to you? https://review.openstack.org/#/c/545479/ | |
| 19:53:33 | mriedem | dansmith: hmm, not seeing the same event routed to both hosts | |
| 19:53:48 | mriedem | but we should have a migration record at this point because i see on the dest host logs that we've plugged the vifs there in pre_live_migration | |
| 19:55:30 | dansmith | mriedem: hang on I'm piecing this back together in my head just a sec | |
| 19:55:36 | dansmith | jroll: ack, will look ater | |
| 19:56:00 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4547-L4559 | |
| 19:56:01 | jroll | thanks | |
| 19:56:04 | mriedem | fwiw, i don't see https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4556 in the logs | |
| 19:56:05 | dansmith | mriedem: it will require migration_context to be set | |
| 19:56:19 | mriedem | yeah that doesn't show up in the n-api logs http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-n-api.txt.gz | |
| 19:56:24 | mriedem | do we not lazy-load migration_context? | |
| 19:56:29 | dansmith | okay, so maybe we haven't set migration context yet? | |
| 19:56:35 | dansmith | mriedem: we query for it specifically in the api side: | |
| 19:56:46 | dansmith | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_external_events.py#L59-L61 | |
| 19:56:52 | mriedem | yup | |
| 19:57:24 | dansmith | more likely we haven't .save()d it yet or something | |
| 19:58:16 | mriedem | i only see apply_migration_context called in evacuate and resize flows, not live migration, | |
| 19:58:19 | mriedem | is it set in the RT? | |
| 19:58:37 | dansmith | not that I know of, so maybe we never set it for live migration? | |
| 19:59:09 | sean-k-mooney[m] | Right so it kindof makes sense when you activate the binding on the destination it deactivates the binding for the source which i guess results in vif unbound for the old host. How are we getting the vif on the source node that we pass to _nova_to_osvif_vif_unbound we should probably be cache the original vif object in the migration object | |
| 19:59:35 | mriedem | right we don't do a move_claim for live migration, | |
| 19:59:41 | mriedem | and RT.move_claim is what sets migration_context on the instance | |
| 20:00:22 | mriedem | sean-k-mooney[m]: heh way ahead of you https://review.openstack.org/#/c/515423/10/nova/objects/migrate_data.py@47 :) | |
| 20:01:43 | sean-k-mooney[m] | :) | |
| 20:02:13 | mriedem | artom: fyi re the migration_context thing, since you wrote https://review.openstack.org/#/c/371048/ | |
| 20:02:28 | mriedem | but i think ^ was just for evac cases, | |
| 20:02:34 | mriedem | where the source is dead so we can't rpc there | |
| 20:02:42 | dansmith | blast from ye olde past | |
| 20:03:37 | mriedem | reminds me i need to touch up some dings in the woodwork with olde english | |
| 20:12:22 | SamYaple | other than a --hard reboot, is there a way to regenerate the libvirt.xml config? we changed the vnc listening address but we cannot migrate/live-migrate some instances due to pci passthrough of a sata controller | |
| 20:14:13 | sean-k-mooney2 | SamYaple: there is no way via nova to regenerate the xml that wont be equivalent to a reboot if you cant live migrate | |
| 20:14:44 | sean-k-mooney2 | SamYaple: you could likely just do a domain xml update out of band. | |
| 20:15:02 | SamYaple | yea unfortunately not, but not for technical reasons | |
| 20:15:07 | SamYaple | sean-k-mooney2: awesome. thought not. thanks for confirming | |
| 20:16:13 | sean-k-mooney2 | SamYaple: if you update the vnc adress in the nova.conf you will also need to restart nova compute as i dont think that is one of the mutable config options either. | |
| 20:17:35 | SamYaple | oh no worries. its all rolled out, just in the "applying" phase | |
| 20:17:42 | SamYaple | we already live-mgirated 90% of the instances | |