Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-12
16:08:42 sean-k-mooney mriedem: so in this case if i detect that the bindings differ e.g. source linux bridge and dest ovs should i fail the migration at that point since
16:08:48 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Reset the instance keypair while rebuilding (spec) https://review.openstack.org/375221
16:08:49 mriedem sdague: updated ^
16:08:51 sean-k-mooney i know the xml will not be updated correctly
16:10:32 mriedem sean-k-mooney: when going from old to new, if you create the port binding, is it just for the dest host or both the source and dest?
16:11:29 mriedem i was kind of hoping to avoid the retype complexity in this, because that makes things weird
16:11:44 mriedem one reason for doing this is to simply cut down on network downtime during live migration,
16:11:47 sean-k-mooney well there will always be the source portbinding. so i would create the destination port binding on the destination host in that case and clean up the source binding if migration succeeded
16:11:52 mriedem another reason is to change vif types, yes?
16:12:18 sean-k-mooney mriedem: yes they are the 2 main usecases
16:12:25 mriedem how is there always a source port binding? i thought we didn't create port binding resources today at all? or you just mean the binding:profile in the port we already have?
16:12:42 mriedem like,
16:12:52 sean-k-mooney the port binding profile in the port we already have
16:13:00 mriedem i thought there is literally going to be a new neutron api which is like POST /ports/{uuid}/bindings
16:13:13 sean-k-mooney i need to double check but i taught that would be expsed via the new api automatically
16:13:23 openstackgerrit David Rabel proposed openstack/nova master: VMware: add support for graceful shutdown of instances https://review.openstack.org/494169
16:13:44 sean-k-mooney mriedem: yes there will be https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html#list-bindings
16:13:48 mriedem so if i do GET /ports/{id}/bindings, for existing ports it will give me at least one result based on the existing port's binding profile?
16:14:12 mriedem is there some data migration that neutron is going to do for that? or just a fallback lookup in the api code?
16:14:42 mriedem maybe this is already modeled and the API is just exposing it? https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html#data-model-changes
16:15:36 mriedem sean-k-mooney: so going back to your question,
16:16:00 mriedem what happens today if you try live migrating an instance with a linuxbridge vif on the source host to a dest host which is using ovs?
16:16:04 mriedem does vif plugging explode?
16:16:23 sean-k-mooney no everything works perfectly with no error... in that direction
16:16:30 sean-k-mooney but you have no network connectivity
16:16:49 mriedem ok so it doesn't work
16:16:57 mriedem it doesn't blow up, but it doesn't work, right?
16:17:03 sean-k-mooney what libvirt did undder the hood was creat a linux bridge an plug the tap into it and neutron never knew about it
16:17:27 sean-k-mooney so the live migration succeeds
16:17:43 sean-k-mooney if you do a hard reboot everything gets fixed
16:17:53 sean-k-mooney but the bridge does not get cleaned up
16:18:24 sean-k-mooney going the other way os-vif explodes if your linux bridge node does not have ovs-vsctl available
16:19:05 sean-k-mooney if it does same thing. we create and ovs bridge called br-int and add the tap to it and the linux bridge agent never know about it so it never get wired up
16:19:57 mriedem huh, seems like something we should be explicitly checking for and failing on during pre live migration
16:20:38 mriedem LB and OVS aside, i mean you also have like macvtap and vhostuser and all of the other vif types
16:20:46 mriedem so who know what goes wrong when migrating between those right?
16:20:50 sean-k-mooney mriedem: you would think so but this actully used to work before hevana but it was a undocument "feature" so it got broken in icehouse
16:21:27 sean-k-mooney mriedem: provided the vif model is still virtio then it actully works correctly if we can update the xml
16:21:54 mriedem can't we just assert that if both nodes in the live migration are not new enough to do the port binding thing that we don't allow it?
16:22:03 mriedem and the vif type changes i mean
16:22:23 sean-k-mooney yes we can that was what i was going to suggest
16:22:35 mriedem if the vif type doesn't change, then maybe we can still support the port binding thing for mixed version computes?
16:23:08 mriedem although if the dest host is old we won't have a port binding to flip to active in post live migrate
16:23:14 sean-k-mooney if the vif_type and vif binding_details does not change we should not need to update the xml either so we could activate the binding before live migrate
16:23:21 mriedem so we can really only do the port binding if going from old to new yes?
16:24:01 mriedem to reset, we plan to create the dest host port binding during pre live migration yes?
16:24:18 sean-k-mooney there are basically 3 cases where we have old+new. i am going to document them in the spec
16:24:21 mriedem that may or may not happen if we're doing that on an old ode
16:24:22 mriedem *node
16:24:30 mriedem that would be helpful :)
16:24:55 mriedem btw, https://docs.openstack.org/nova/latest/reference/live-migration.html is uber helpful
16:25:24 sean-k-mooney yes but to reset if its an old source node it wont create the binding but we could do that in do_migrate on the destination.
16:25:39 mriedem pre_live_migration runs on the dest host and i think that's where the dest host port binding was going to be created, but i'm not sure
16:25:42 sean-k-mooney if you have a new source node and old destination there is no way to update the xml
16:26:01 mriedem ok so we create the dest host port binding from the source node?
16:26:48 mriedem live_migrate runs on the source node, calls pre_live_migration on the dest node, and then live_migrate on the source starts doing it's thing
16:26:54 sean-k-mooney mriedem: ill duble check i have been confusing myself with this most of the afternoon so kindo of mixed up what happens where
16:27:02 mriedem use https://docs.openstack.org/nova/latest/reference/live-migration.html
16:28:29 kashyap kaisers: When you're back; glad you noted the correction in the bug: "1) correction regarding my last comment (#6): block commit -> blockRebase"
16:28:37 kashyap kaisers: They're both completely opposite.
16:29:32 sean-k-mooney right ok ill work on the spec and use https://docs.openstack.org/nova/latest/reference/live-migration.html to keep it straight
16:31:39 sean-k-mooney mriedem: basically what i wanted to know was is it ok to change the sematic in the event that we need an xml update but cant do one and fail the migrate instead of migrating and haveing broken networkin that is then fixed with a hard reboot
16:32:21 mriedem sean-k-mooney: it seems to me we shouldn't be allowing that today anyway
16:32:25 mriedem so yes i'd be ok with that
16:33:27 sean-k-mooney ok ill capture that in the spec with the different combinations. thanks
16:35:07 openstackgerrit Eric Fried proposed openstack/nova-specs master: Remove barbican from scope of ksa adapter work https://review.openstack.org/511524
16:35:10 efried sdague FYI, crickets in -barbican
16:35:23 efried sdague mriedem ^ updated spec accordingly.
17:02:37 openstackgerrit Elod Illes proposed openstack/nova master: WIP: Transform scheduler.select_destinations notification https://review.openstack.org/508506
17:06:00 openstackgerrit Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
17:17:53 openstackgerrit Merged openstack/nova master: api-ref: add warning about force evacuate for ironic https://review.openstack.org/511477
17:42:21 openstackgerrit melanie witt proposed openstack/nova stable/pike: Fix target_cell usage for scatter_gather_cells https://review.openstack.org/511538
17:54:38 mriedem dansmith: on this migrations API paging spec https://review.openstack.org/#/c/506030/ i don't think we need to do the elaborate scatter/gather thing like you did for instance listing probably, with the built in merge sorting and all
17:55:07 dansmith mriedem: migrations are per instance right?
17:55:11 mriedem we could build in the scatter/gather on it's own if we wanted, but it seems we could just merge sort the results once we pull them out of the db, the user can't pass in sort keys and this is admin-only anyway so performance shouldn't be a huge concern
17:55:12 dansmith if so, there's no scattering
17:56:02 mriedem not sure i follow,
17:56:13 mriedem couldn't we use the scatter/gather thing on this today? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3995
17:56:29 dansmith mriedem: you list migrations for an instance, right?
17:56:29 mriedem it would just run the db query concurrently per cell and munge the results
17:56:37 mriedem not this api
17:56:46 mriedem this is all migrations regardless of instance
17:56:49 dansmith oh
17:57:14 dansmith don't we have one that is per instance?
17:57:17 mriedem we do
17:57:24 dansmith *awesome*
17:57:33 mriedem but paging that one would be more like paging simple tenant usage i think,
17:57:36 mriedem which was weird
17:57:46 dansmith anyway,
17:58:46 dansmith I would think just doing linear paging for that would be fine yeah
17:58:49 dansmith until someone complains
17:59:41 mriedem yup cool
18:00:01 mriedem only cburgess would have that many migration records to page anyway
18:00:07 cburgess Do what?
18:00:21 mriedem migrating like a mofo
18:00:24 cburgess yes
18:00:26 cburgess yes we do..
18:00:28 cburgess *sigh*
18:00:39 mriedem well then do we have the spec for you! https://review.openstack.org/#/c/506030/
18:10:02 _ix Hello friends. I can think of a way to do this, but I was hoping there might be a better way. I'm looking to determine whether some of our nodes that are believed to be running in an HA capacity are in fact running on the same hypervisors.

Earlier   Later