Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-20
20:00:23 sean-k-mooney im not sure that its fair to describe kolla image as like system image e.g. lxc style but they are not that light weight either
20:04:30 zigo What's blocking this backport patch ? https://review.opendev.org/#/c/711233/
20:04:55 zigo The bug https://bugs.launchpad.net/nova/+bug/1788014 is causing real life troubles and a fix would be really nice.
20:04:55 openstack Launchpad bug 1788014 in OpenStack Compute (nova) rocky "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Elod Illes (elod-illes)
20:05:38 zigo We had all sorts of down time due to it, lots of head scratching until we understood what was going on...
20:08:37 melwitt elod: question for your morrow ^
22:47:20 abaindur Hello, I have a question about post copy live migration. What happens if live_migration_permit_post_copy is only set on nova compute on some hypervisors? Does it need to be the same across every host?
22:47:31 sean-k-mooney mnaser: this is my counter proposal https://review.opendev.org/#/c/720107/3/goals/proposed/container-images.rst@14
23:11:53 sean-k-mooney abaindur: i think it is based on teh source node
23:12:17 sean-k-mooney but we dont test it so it shoudl be the same on all node but it might work if its different
23:12:19 abaindur would there be any issues if we migrated from a source host that had post copy enabled, but a destination host that didnt?
23:12:48 abaindur we want to give it a shot - but only wanted to run it on a subset of hypervisors
23:14:50 abaindur sean-k-mooney: one other question about live migration: reason we are going to post-copy is because we're seeing significant downtime (15 - 30+ seconds) during live migration. Seems to always start when VM is Paused on source/Resumed on dest, then start working shortly after port-binding activate call is made, and port is plugged on the host
23:15:31 abaindur We thought that maybe giving post-copy a shot would help, since it would give us the benefit of this fix: https://opendev.org/openstack/nova/commit/1f48d3d83b4d5f6f9cd96ee06d2fc005635c1ff9
23:16:54 abaindur But are there any known issues around pre-copy live migration? Bulk of the time seems to be taken up in _post_copy_live_migration() function on the source host. For example it took 18+ seconds from statr of that function until the port-binding activate call was sent to neutron
23:18:37 abaindur sorry, not _post_copy_live_migration(). I meant _post_live_migration() function
23:22:30 sean-k-mooney abaindur: libvirt will check if the qemu and libvirt on each host support it
23:22:39 sean-k-mooney and only enable it if both do i belive
23:23:38 sean-k-mooney so in principal i dont think it would have a negitive effect just be aware that you would see different behavior migration too a host with it enabled vs migrating form a host with it enabled
23:24:23 sean-k-mooney i dont recall off the top of my head which config we check to enable it but i belive it would have a asymetric behavior as i think we only check one of them
23:25:52 sean-k-mooney abaindur: for what it si worth the port binding events shoudl work with or without post copy
23:27:12 sean-k-mooney _post_live_migration is the function that cleans up the image on the source node and finishes and work reqiured on the dest
23:28:05 sean-k-mooney if you are using a nova and neutron that do not support multiple port bining _post_live_migration_at_dest is where we will do the port binding
23:29:06 sean-k-mooney but in a nova that support neutron multiple port bindign api we will prebind the port on the souce and activate it either in responce to the live migrion even tor the start of _post_live_migration
23:29:16 abaindur yea, but we noticed that in pre-copy LV mode, the port binding activate call is sent towards the end of that function. And connectivity is disrupted when VM is Paused.
23:29:18 abaindur https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6931
23:29:45 sean-k-mooney ya so are you using ovs with the ovs firewall driver by any chance
23:29:51 abaindur and here is where the port binding activation call is invoekd: https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:29:52 sean-k-mooney or are you using iptables
23:30:42 abaindur between those 2 lines of code, we noticed nova taking 18+ seconds - 2 seocnds for bolume cleanup, few more seconds for disconnection, 5 sec for get_instance_nw_info, 5 more sec for compute_utils.notify_about_instance_action, etc...
23:31:01 abaindur One the port binding activate call came, another 3-5 seconds for port to be plugged by OVS-agent
23:31:04 abaindur we're using iptables
23:31:26 sean-k-mooney on what release
23:31:33 abaindur Rocky
23:31:49 sean-k-mooney the iptable firewall should have less downtime the openvswithc
23:32:34 sean-k-mooney as we can pre plug the port and have neutron wire it up while we are waiting for migration too happen at the libvirt level.
23:33:08 sean-k-mooney when using the ovs firewall driver because libvirt recreate the ovs port it takes longer as neutron has to do it twice
23:33:14 abaindur as mentioned, we're seeing non-trivial downtime in that _post_live_migration() function on the src host, between the log "'_post_live_migration() is started.." (or when VM is Paused), and when neutron receives the port-binding /activate call at https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:33:40 sean-k-mooney abaindur: yes but before the multiple port binding change it used to be even longer
23:34:04 abaindur so is that down time then to be expected or unavoidable? :(
23:34:23 sean-k-mooney no its avoidable
23:34:31 sean-k-mooney so this is the feature you are trying to use https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html
23:34:52 sean-k-mooney i did not think this required post-copy but let me double check
23:35:17 abaindur well thats why we were considering trying out post-copy - to see if it speeds up when nova makes the port binding call
23:35:30 sean-k-mooney ok so yes if you want the quick setup we are waiting for VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY
23:35:43 abaindur post-copy seems to trigger that call based on VIR_DOMAIN_EVENT states
23:36:14 abaindur pre-copy waits for that _post_live_migration() on the host to invoke network_api.migrate_instance_start(), which seems to take a while for us
23:36:29 sean-k-mooney that will allow us to activate the port bidning from the souce earlier then _post_live_migration
23:37:13 abaindur Right, thats why we will try post-copy mode. But was wondering if the downtime/delay we are seeing with pre-copy is expected or unavoidable?
23:37:19 sean-k-mooney abaindur: ya so it is a littl strange that your _post_live_migration function completion is os long
23:37:39 sean-k-mooney i would not expect _post_live_migration to take multiple seconds
23:38:41 abaindur what are downsides to post copy besides that VM needs to be rebooted if theres a live migration error?
23:39:06 abaindur and page faults may slow down the VM as memory needs to be copied over the network?
23:39:17 sean-k-mooney that is the main one. if there is a network outage while its still in post copy phase then the vm will crash
23:39:35 sean-k-mooney yes page falts acroos the network might beut all write happen locally
23:39:47 sean-k-mooney so the vm will only pause if it need to read un copied data
23:40:05 sean-k-mooney and once its copied local update to that will happen to the dest memmory
23:40:19 sean-k-mooney as you pointed out we are activating the port here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:40:54 sean-k-mooney which happen near the star of the function so the dely is likely related to cinder performance
23:41:10 sean-k-mooney abaindur: have you tried live migrating vms that dont have cinder volumes
23:41:28 abaindur yea, we tried VMs both volume and ephemeral based
23:41:42 sean-k-mooney did you see the same delay?
23:42:03 abaindur pretty much
23:42:24 abaindur timed the volume code with some logs of our own, heres what we observed:
23:42:36 abaindur 1.913 seconds for _get_instance_block_device_info and self.driver.post_live_migration
23:42:45 abaindur about 0.4 sec for self.driver.get_volume_connector(instance)
23:42:51 abaindur 2.659 seconds for self.volume_api.terminate_connection()
23:42:58 abaindur 3.28 seconds for network_info = self.network_api.get_instance_nw_info(ctxt, instance)
23:43:03 abaindur 5.019 seconds for self._notify_about_instance_usage(ctxt, instance, "live_migration._post.start", network_info=network_info)
23:43:16 abaindur another almost 5 seconds nova spent just making the port binding activate API call, seems to be spending time in keystone and oslo_concurrency.lockutils code
23:43:25 sean-k-mooney is that absolute time of time for each function
23:43:44 abaindur yea, we just added logs throughout that post_live_migration() function
23:43:56 abaindur before/after each of those calls to see what was taking so long
23:44:01 sean-k-mooney ok so i subtrac those numebr to get the time for each
23:44:07 sean-k-mooney that is still very slow
23:44:30 abaindur for example, in one live migration, we say: VM Paused at 25:06.192
23:44:39 sean-k-mooney abaindur: are you using memcache for you keysontone auth tokens
23:45:32 abaindur by time we saw nova-compute make the port binding call, it was at: 35:24.694
23:46:20 abaindur VM Paused at 35:06.192 *
23:46:49 sean-k-mooney abaindur: can you check your nova.conf and see if you have https://zuul.opendev.org/t/openstack/build/dcde79801a624c25b195a46ead7af562/log/controller/logs/etc/nova/nova-cpu_conf.txt#62-68
23:49:04 sean-k-mooney also 10 seconds to bind the port in precopy mode is too hight so there is something else hurting performance which is why im suspecign you do not have caching of keysotne configure correctly.
23:52:47 abaindur Ok no, memcache_servers is not set...
23:52:58 abaindur this is for nova compute on the hypervisor side right?
23:53:44 sean-k-mooney https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6982-L7008 could be safely moved to https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6939 by the way and in the libvirt case https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6933 woudl be fine too.
23:53:57 sean-k-mooney am it will be used both on the contoler and computes
23:54:13 sean-k-mooney so you are missing
23:54:16 sean-k-mooney [keystone_authtoken]
23:54:18 sean-k-mooney memcached_servers = localhost:11211
23:54:25 sean-k-mooney or well your actull memcache servers
23:54:45 abaindur yeah, I dont see that config opt ever ok
23:54:49 sean-k-mooney this will be used to cach auth tokens for every api call we make
23:55:16 abaindur i dont see it ever set*
23:58:21 sean-k-mooney abaindur: so at least on the contoler side it has a significnat impact on the api
23:58:25 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1836642
23:58:25 openstack Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Incomplete] - Assigned to Slawek Kaplonski (slaweq)
23:58:53 sean-k-mooney we adress this porblem in the gate by truning on caching with https://github.com/openstack/devstack/commit/d33cdd01f83b891b010e0fd238f1816910f3fd77
23:59:18 abaindur I dont see it on controller either
23:59:25 sean-k-mooney i am not sure if i tis use dby the compute node but i think it will be whenever it is calling cinder neutron or placment
23:59:35 sean-k-mooney abaindur: its optional
23:59:49 sean-k-mooney but it improves perfromacne alot

Earlier   Later