| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-19 | |||
| 18:38:30 | dansmith | mriedem: did you see my comment on that patch for the obj_relationships thing? | |
| 18:39:13 | mriedem | not yet | |
| 18:44:31 | melwitt | dansmith: from what I can tell, we only have unit test coverage for the uuid generation for service where things are mocked (and would anyway be a write to begin with), or calling _from_db_object selectively https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_service.py#L145 | |
| 18:44:54 | melwitt | to hit this, we'd have to have a functional test scenario where we read an already existing service record that has no uuid | |
| 18:45:03 | melwitt | I can try writing one | |
| 18:47:24 | dansmith | hmm, I thought we did | |
| 18:47:30 | dansmith | because that was mriedem | |
| 18:47:36 | dansmith | and he looooves functional tests | |
| 18:47:56 | dansmith | I would also have expected to hear more noise about this | |
| 18:48:20 | dansmith | like, wouldn't the first compute startup after that patch merged cause nova-compute to not update its service record? | |
| 18:49:43 | mriedem | it was dtp with me helping, but it's a pretty established pattern, | |
| 18:49:53 | mriedem | i.e. he copied it from compute_nodes.uuid, and aggregates.uuid | |
| 18:49:57 | dansmith | yeah | |
| 18:50:04 | dansmith | was just lookin gup the commit | |
| 18:50:12 | mriedem | also https://github.com/dpeschman | |
| 18:50:17 | mriedem | https://avatars3.githubusercontent.com/u/145098?s=400&v=4 | |
| 18:50:26 | mriedem | i would like to know what's going on there | |
| 18:50:36 | melwitt | lol | |
| 18:51:59 | dansmith | even still, | |
| 18:52:11 | dansmith | we're out of the db context at that point, so I'm not sure why this is happening anyway | |
| 18:52:34 | dansmith | out of the pick_context_manager I mean | |
| 19:11:51 | openstackgerrit | Merged openstack/nova-specs master: Update placement aggregates spec to clarify generation handling https://review.openstack.org/548237 | |
| 19:23:12 | openstackgerrit | Dan Smith proposed openstack/nova master: Add AggregateList.get_by_metadata() query method https://review.openstack.org/544728 | |
| 19:23:13 | openstackgerrit | Dan Smith proposed openstack/nova master: Add request filter functionality to scheduler https://review.openstack.org/544730 | |
| 19:23:13 | openstackgerrit | Dan Smith proposed openstack/nova master: Add aggregates list to Destination object https://review.openstack.org/544729 | |
| 19:23:14 | openstackgerrit | Dan Smith proposed openstack/nova master: [WIP] Add require_tenant_aggregate request filter https://review.openstack.org/545002 | |
| 19:23:14 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990 | |
| 19:23:15 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282 | |
| 19:35:52 | mriedem | debugging weirdness during live migratoin would be a lot easier if we had thread ids in the logs... | |
| 19:40:47 | mriedem | sean-k-mooney: do you know if unbinding a port makes it automatically go to status=DOWN? | |
| 19:44:19 | mriedem | sean-k-mooney: https://review.openstack.org/#/c/434870/11/nova/compute/manager.py@1126 | |
| 19:44:55 | mriedem | seeing issues during live migration where we handle the migration completed lifecycle event from the hypervisor and start unbinding the port while we're also going through _post_live_migration | |
| 19:45:02 | mriedem | we blow up trying to unplug an unbound VIF | |
| 19:45:18 | sean-k-mooney[m] | Im not sure. The status of an unbound port does not really have a meaning as it is effectively just a db entry when unbound. E. G. It dose not exist on the datapath when unbound. | |
| 19:45:34 | mriedem | i ask because of https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186 | |
| 19:45:50 | mriedem | i'm seeing mysterious network-vif-plugged and network-vif-unplugged events around the time of the failure | |
| 19:46:00 | mriedem | but they are coming before we unplug vifs on the source host | |
| 19:47:49 | mriedem | might be getting network-vif-plugged events on the source host from when they were plugged on the dest host in pre_live_migration, | |
| 19:48:00 | mriedem | the events are going to be routed to where the instance currrently lives, which is the source host until we switch | |
| 19:48:57 | mriedem | then we live migrate the guest, | |
| 19:49:10 | mriedem | get the 'migration completed' lifecycle event from the hypervisor and start unbinding the port, | |
| 19:49:11 | dansmith | mriedem: once there is a migration record, they'll be routed to both hosts | |
| 19:49:50 | sean-k-mooney[m] | Well in the old workflow we don't unbind the port. We just update the host in the binding details. Yes i think the vif plugged events are from when the destination vif is plugged | |
| 19:49:51 | mriedem | ok, i was only checking the source host logs, can check the dest host logs | |
| 19:50:18 | mriedem | sean-k-mooney[m]: this is the patch causing problems right now https://review.openstack.org/#/c/434870/ | |
| 19:50:31 | mriedem | the thing that tried to do the early switch once were in post-copy | |
| 19:50:37 | mriedem | *once we were in | |
| 19:50:56 | jroll | jaypipes: dansmith: this is roughly what I recall from PTG about ironic RT stuff, does it make sense to you? https://review.openstack.org/#/c/545479/ | |
| 19:53:33 | mriedem | dansmith: hmm, not seeing the same event routed to both hosts | |
| 19:53:48 | mriedem | but we should have a migration record at this point because i see on the dest host logs that we've plugged the vifs there in pre_live_migration | |
| 19:55:30 | dansmith | mriedem: hang on I'm piecing this back together in my head just a sec | |
| 19:55:36 | dansmith | jroll: ack, will look ater | |
| 19:56:00 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4547-L4559 | |
| 19:56:01 | jroll | thanks | |
| 19:56:04 | mriedem | fwiw, i don't see https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4556 in the logs | |
| 19:56:05 | dansmith | mriedem: it will require migration_context to be set | |
| 19:56:19 | mriedem | yeah that doesn't show up in the n-api logs http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-n-api.txt.gz | |
| 19:56:24 | mriedem | do we not lazy-load migration_context? | |
| 19:56:29 | dansmith | okay, so maybe we haven't set migration context yet? | |
| 19:56:35 | dansmith | mriedem: we query for it specifically in the api side: | |
| 19:56:46 | dansmith | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_external_events.py#L59-L61 | |
| 19:56:52 | mriedem | yup | |
| 19:57:24 | dansmith | more likely we haven't .save()d it yet or something | |
| 19:58:16 | mriedem | i only see apply_migration_context called in evacuate and resize flows, not live migration, | |
| 19:58:19 | mriedem | is it set in the RT? | |
| 19:58:37 | dansmith | not that I know of, so maybe we never set it for live migration? | |
| 19:59:09 | sean-k-mooney[m] | Right so it kindof makes sense when you activate the binding on the destination it deactivates the binding for the source which i guess results in vif unbound for the old host. How are we getting the vif on the source node that we pass to _nova_to_osvif_vif_unbound we should probably be cache the original vif object in the migration object | |
| 19:59:35 | mriedem | right we don't do a move_claim for live migration, | |
| 19:59:41 | mriedem | and RT.move_claim is what sets migration_context on the instance | |
| 20:00:22 | mriedem | sean-k-mooney[m]: heh way ahead of you https://review.openstack.org/#/c/515423/10/nova/objects/migrate_data.py@47 :) | |
| 20:01:43 | sean-k-mooney[m] | :) | |
| 20:02:13 | mriedem | artom: fyi re the migration_context thing, since you wrote https://review.openstack.org/#/c/371048/ | |
| 20:02:28 | mriedem | but i think ^ was just for evac cases, | |
| 20:02:34 | mriedem | where the source is dead so we can't rpc there | |
| 20:02:42 | dansmith | blast from ye olde past | |
| 20:03:37 | mriedem | reminds me i need to touch up some dings in the woodwork with olde english | |
| 20:12:22 | SamYaple | other than a --hard reboot, is there a way to regenerate the libvirt.xml config? we changed the vnc listening address but we cannot migrate/live-migrate some instances due to pci passthrough of a sata controller | |
| 20:14:13 | sean-k-mooney2 | SamYaple: there is no way via nova to regenerate the xml that wont be equivalent to a reboot if you cant live migrate | |
| 20:14:44 | sean-k-mooney2 | SamYaple: you could likely just do a domain xml update out of band. | |
| 20:15:02 | SamYaple | yea unfortunately not, but not for technical reasons | |
| 20:15:07 | SamYaple | sean-k-mooney2: awesome. thought not. thanks for confirming | |
| 20:16:13 | sean-k-mooney2 | SamYaple: if you update the vnc adress in the nova.conf you will also need to restart nova compute as i dont think that is one of the mutable config options either. | |
| 20:17:35 | SamYaple | oh no worries. its all rolled out, just in the "applying" phase | |
| 20:17:42 | SamYaple | we already live-mgirated 90% of the instances | |
| 20:26:54 | mriedem | sean-k-mooney[m]: yeah so i think changing the port binding changes the status, | |
| 20:27:07 | mriedem | this is the source host neutron agent 2 seconds after nova updated the port binding to point at the dest host | |
| 20:27:09 | mriedem | http://logs.openstack.org/71/551371/6/check/legacy-tempest-dsvm-multinode-live-migration/4d466b2/logs/screen-q-agt.txt.gz#_Mar_19_14_25_08_374523 | |
| 20:27:15 | mriedem | binding host and status changes on the port in there | |
| 20:28:25 | mriedem | which according to this: https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186 will trigger a network-vif-unplugged event | |
| 20:29:39 | sean-k-mooney2 | right so looking at that ports binding_details its an ovs vif with ovs_hybrid_plug": true so os-vif will plug the vif on the destination ovs during pre livemigrate and neutron will emit the vif plugged events as it wires them up | |
| 20:30:01 | mnaser | does anyone know if there's some sort of variable you can use in docs to output the current version/release | |
| 20:30:11 | mnaser | reason - https://docs.openstack.org/nova/queens/cli/nova-manage.html -- this doc and many others have old versions listed | |
| 20:30:21 | mnaser | i can update it manually but it'll probably be forgotten again :> | |
| 20:31:34 | mriedem | mnaser: https://review.openstack.org/#/c/553903/ | |
| 20:32:45 | mnaser | mriedem: oh well there you go, that's a solution | |
| 20:33:58 | sean-k-mooney2 | mriedem: ok so the vif plugged event is coming from the pre livemigrate when os-vif adds the port to ovs and the unplug event is coming form activateing the binding on the destenation host. is that the order it happens in | |
| 20:35:14 | openstackgerrit | Merged openstack/nova master: Don't filter out sibling sets with one core https://review.openstack.org/537361 | |
| 20:37:11 | mriedem | sean-k-mooney2: yes i think so, | |
| 20:37:25 | mriedem | sean-k-mooney2: i'm confused as to why neutron thinks the port is unbound though | |
| 20:37:34 | mriedem | this is on both the source and dest neutron agent | |