Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-27
02:20:18 melwitt yeah, events are sketch depending on which networking backend too, right
02:20:30 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Change deprecated policies to policy https://review.openstack.org/583434
02:20:30 melwitt like ovs vs other
02:20:36 mriedem melwitt: shouldn't be in this case,
02:20:44 mriedem odl should send the event on host binding changes
02:20:46 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Fix all invalid obj_make_compatible test case https://review.openstack.org/574240
02:20:48 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Fix all invalid obj_make_compatible test case https://review.openstack.org/574240
02:20:49 mriedem just not plug/unplug
02:20:51 melwitt oh, because neutron knows about it and not relying on anything else? ok
02:21:20 melwitt just remember getting burned by the whole plug event thing for reboot
02:21:50 melwitt but that was because we so os-vif plug only, not any call to neutron and the agent (or something) has to notice it
02:22:27 sean-k-mooney melwitt: the binding change is handeld in the common ml2 layer if i rember corrrectly yes. the port wire up/tear down event however has to come form the backend not the common layer hence the delta between odl/ovs in that case
02:22:53 melwitt sean-k-mooney: yeah, I was having trouble remembering what the deal was. thanks
02:24:20 sean-k-mooney melwitt: the reason it did not work with linux bridge is its pools. the reason it did not work for odl was they were missing the handeler for the event in odl to send it to the websocket creeated by netowrking odl. i think they have fixed that. maybe
02:27:54 sean-k-mooney any way nova is reciving the port update event in this case from neutron and its updating the network info cacche so by the time we call nova_to_osvif_vif the vif_type is set to unbound and boom. if we still have the migration data object at this point we should have a copy of the original vif object that we could use instead of the info_cache versions to work around it.
02:28:23 mriedem so migrate_instance_start() was always a noop before this series,
02:28:49 mriedem so its order in _post_live_migration would have never mattered except for nova-network
02:29:14 mriedem given we already call migrate_instance_start during post-copy, i don't think moving the order of those calls in _post_live_migration will matter,
02:29:33 mriedem because from these logs, i can see that when we call migrate_instance_start from _post_live_migration, it's a noop b/c the dest port binding is already active
02:30:03 mriedem http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_00_974248
02:30:19 mriedem so i would think it means, we need to handle unbound vifs during unplug in the driver?
02:30:29 mriedem or just not call unplug_vifs in certain cases
02:30:32 mriedem not totally sure though
02:31:33 mriedem all the libvirt driver does in post_live_migration_at_source is unplug_vifs
02:32:11 sean-k-mooney if we dont call unplug_vif we could leak the linux bridges we create for ovs hybrid plug
02:33:06 mriedem umm...
02:33:15 mriedem oh i see what you were saying about storing off the vif_type then
02:33:44 mriedem b/c i was going to say, we could just not call unplug_vifs if the vif type (after refreshing the network info cache from neutron) was now 'unbound'
02:33:59 mriedem if it is, we can temporarily heal that using migrate_data.vifs
02:34:01 mriedem that has the vif type in it
02:34:15 sean-k-mooney mriedem: yep
02:34:34 mriedem ok i could try cooking something up real quick,
02:34:39 mriedem my wife is going to kill me though
02:34:59 melwitt you could do tomorrow morning?
02:35:12 sean-k-mooney i can try this in the morning too. i just need a 2 node vanila devstack install right
02:35:13 melwitt unless you were thinking to fast-approve this tonight
02:36:26 mriedem why would the vif type be unbound?
02:36:30 mriedem shouldn't it be bound to the dest host?
02:36:35 mriedem since we activated it there?
02:37:13 sean-k-mooney mriedem: it is. each host has its own binding now. only one will be in the bound state all the rest will be unbound
02:38:36 mriedem but i think the port in our info cache is not host-aware...
02:38:39 mriedem i need to check
02:39:56 mriedem http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_00_726935
02:40:04 mriedem that's where we refresh the info cache in _post_live_migration
02:40:10 mriedem after activating the dest host port binding
02:40:23 mriedem ed": false, "tenant_id": "7dbeedd7076e472091193779ebbcf887", "mtu": 1400}, "id": "1d8de970-331e-46b5-8c7b-574821e891e5", "label": "tempest-LiveMigrationTest-411356071-network"}, "devname": "tap21095ff0-6b", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {}, "address": "fa:16:3e:34:c9:90", "active": false, "type": "unbound", "id": "21095ff0-6bcd-414b-9d6f-b63e03aacb23", "qbg_params": null}]
02:40:23 mriedem [{"profile": {"migrating_to": "ubuntu-xenial-rax-dfw-0001002004"}, "ovs_interfaceid": null, "preserve_on_delete": false, "network": {"bridge": null, "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [], "address": "10.1.0.10"}], "version": 4, "meta": {}, "dns": [], "routes": [], "cidr": "10.1.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.1.0.1"}}], "meta": {"in
02:40:50 mriedem yeah...that's wrong
02:40:55 mriedem it should be bound to the dest host
02:44:42 sean-k-mooney well it was bound shortly before http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_59_311896
02:46:49 mriedem yup we hit post-copy callback here and activate the dest host port binding http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_58_561391
02:47:34 mriedem refresh nw info cache here http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_59_310738
02:48:16 mriedem then we get an unplugged vif event from neutron
02:48:19 sean-k-mooney whats happening is liekly that when the ovs neutron agent sees the tap device disapear it is sending an update to notify us the port state has changed on the souce node.
02:48:19 mriedem could be concurrently
02:48:50 mriedem yeah we get the unplugged event and refresh the cache and it's unbound http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_00_726935
02:49:14 mriedem post live migrate the dest host port binding is already active http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_44_00_974248
02:50:01 mriedem then we unplug and kablammo
02:50:36 mriedem doesn't help that we route all of these plug/unplug neutron events to the source host only, that's a nova limitation during live migration right now
02:50:57 mriedem and there might be some kind of delay in the state updates or something in the neutron db?
02:51:09 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix create_all() to replace_all() in comments https://review.openstack.org/586396
02:51:16 mriedem anyway, i can hack around this a bit i think but kind of sucks
02:51:53 sean-k-mooney mriedem: well there is a delay in the neutron agent sendign the update over the rabbit rpc bus to the neutron-server and then the rest call to nova.
02:52:37 mriedem i just worry the port isn't wired up on the dest or something, but that shouldn't be the case b/c we plug_vifs on the dest host during pre_live_migration now
02:52:41 mriedem it's just inactive until post-copy
02:52:48 sean-k-mooney we could prabably hack in a filter to ignore nay info cache updates where teh vif type is unbound and the port profile containts a migrating_to field
02:53:11 mriedem yeah...
02:53:49 mriedem that would coincide with this http://logs.openstack.org/63/585163/1/check/nova-live-migration/1b2aebb/logs/screen-n-cpu.txt#_Jul_27_01_43_59_310738
02:54:04 sean-k-mooney mriedem: yes if the pulgin fails in pre_live_migration we bail out early and try another host so at this point the dest networking shoudl be fully set up
02:55:59 mriedem also, if we get the info cache based on what's setup for the dest host, we could have changed vif types, so unplugging on the source could be a different vif type...couldn't it?
02:56:03 mriedem this gets a bit wonky
02:56:24 mriedem we do have an exact copy of the source_vif in the migrate data vifs
02:56:33 sean-k-mooney yes it could have changed.
02:56:35 sean-k-mooney yep
02:56:49 sean-k-mooney the migrate data has everything you need.
02:57:18 sean-k-mooney just look up the vif by the port uuid and unplug or better yet just loop over all the vifs in migrate data instead of instance
02:57:50 mriedem that's kind of what i'm going to do, will hack something up quick and post it then flesh it out more in the morning
02:57:59 mriedem sean-k-mooney: and for the love of toast go to bed
02:58:11 bzhao__ Sorry for a nic break, I have a brief in the neutron log from the link shows. For the failure test instance, seem It works correct in Neutron side.
02:58:36 sean-k-mooney haha its only 4 am. but ya. il be back only in 6-8 hours and ill take a look at it then. nighto/
03:01:55 openstack Launchpad bug 1783917 in OpenStack Compute (nova) "live migration fails with NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" [High,Confirmed]
03:01:55 melwitt bzhao__: thanks. feel free to add a comment to explain about the neutron side in https://bugs.launchpad.net/neutron/+bug/1783917 see comment #6
03:02:39 bzhao__ melwitt: Thanks, I will. ;-)
03:07:32 mriedem got a patch, pretty simple, no tests but can be easily added by someone else tonight or in the morning
03:09:16 sapd Hi everyone. I got this error when attach a SR-IOV port to instance http://paste.openstack.org/show/726723/ Please help me
03:10:35 mriedem sapd: read through https://docs.openstack.org/neutron/latest/admin/config-sriov.html and check everything in there
03:10:54 melwitt mriedem: coolness, sounds good
03:14:12 sapd mriedem: yep. I have read it. And follow the guide to config. Everything I setup is correct. Because I already launched an instance using SR-IOV successful. But It did not receive DHCP. So I launched another instance using Openvswitch then add SR-IOV port to the instance. But got above error.
03:16:57 openstack Launchpad bug 1708433 in OpenStack Compute (nova) "Attaching sriov nic VM fail with keyError pci_slot" [Undecided,Expired]
03:16:57 melwitt sapd: looks like the bug has been around for awhile and still not resolved https://bugs.launchpad.net/nova/+bug/1708433 they say you can boot with the port if you pass it during server create, but that attaching port separately is broken
03:18:17 melwitt sapd: what release of nova are you using?
03:18:32 sapd melwitt: I'm using queens version. 17.0.4
03:18:34 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Use source vifs when unplugging on source during post live migrate https://review.openstack.org/586402
03:18:35 mriedem melwitt: bzhao__: Kevin_Zheng: sean-k-mooney: ^ just needs unit tests
03:19:03 melwitt sapd: okay, I'm going to re-open that bug and mention what version you saw it in. it will need to be worked on
03:19:13 Kevin_Zheng mriedem, got it, just finish reading launchpad report
03:19:15 mriedem ask sahid to look at it
03:19:19 mriedem the sriov bug i mean
03:19:54 melwitt k
03:20:33 sean-k-mooney[m] Melwitt we used ti have an api check at one point to expresly forbid attach sriov port to existing instances.

Earlier   Later