Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-20
23:37:13 abaindur Right, thats why we will try post-copy mode. But was wondering if the downtime/delay we are seeing with pre-copy is expected or unavoidable?
23:37:19 sean-k-mooney abaindur: ya so it is a littl strange that your _post_live_migration function completion is os long
23:37:39 sean-k-mooney i would not expect _post_live_migration to take multiple seconds
23:38:41 abaindur what are downsides to post copy besides that VM needs to be rebooted if theres a live migration error?
23:39:06 abaindur and page faults may slow down the VM as memory needs to be copied over the network?
23:39:17 sean-k-mooney that is the main one. if there is a network outage while its still in post copy phase then the vm will crash
23:39:35 sean-k-mooney yes page falts acroos the network might beut all write happen locally
23:39:47 sean-k-mooney so the vm will only pause if it need to read un copied data
23:40:05 sean-k-mooney and once its copied local update to that will happen to the dest memmory
23:40:19 sean-k-mooney as you pointed out we are activating the port here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L7005
23:40:54 sean-k-mooney which happen near the star of the function so the dely is likely related to cinder performance
23:41:10 sean-k-mooney abaindur: have you tried live migrating vms that dont have cinder volumes
23:41:28 abaindur yea, we tried VMs both volume and ephemeral based
23:41:42 sean-k-mooney did you see the same delay?
23:42:03 abaindur pretty much
23:42:24 abaindur timed the volume code with some logs of our own, heres what we observed:
23:42:36 abaindur 1.913 seconds for _get_instance_block_device_info and self.driver.post_live_migration
23:42:45 abaindur about 0.4 sec for self.driver.get_volume_connector(instance)
23:42:51 abaindur 2.659 seconds for self.volume_api.terminate_connection()
23:42:58 abaindur 3.28 seconds for network_info = self.network_api.get_instance_nw_info(ctxt, instance)
23:43:03 abaindur 5.019 seconds for self._notify_about_instance_usage(ctxt, instance, "live_migration._post.start", network_info=network_info)
23:43:16 abaindur another almost 5 seconds nova spent just making the port binding activate API call, seems to be spending time in keystone and oslo_concurrency.lockutils code
23:43:25 sean-k-mooney is that absolute time of time for each function
23:43:44 abaindur yea, we just added logs throughout that post_live_migration() function
23:43:56 abaindur before/after each of those calls to see what was taking so long
23:44:01 sean-k-mooney ok so i subtrac those numebr to get the time for each
23:44:07 sean-k-mooney that is still very slow
23:44:30 abaindur for example, in one live migration, we say: VM Paused at 25:06.192
23:44:39 sean-k-mooney abaindur: are you using memcache for you keysontone auth tokens
23:45:32 abaindur by time we saw nova-compute make the port binding call, it was at: 35:24.694
23:46:20 abaindur VM Paused at 35:06.192 *
23:46:49 sean-k-mooney abaindur: can you check your nova.conf and see if you have https://zuul.opendev.org/t/openstack/build/dcde79801a624c25b195a46ead7af562/log/controller/logs/etc/nova/nova-cpu_conf.txt#62-68
23:49:04 sean-k-mooney also 10 seconds to bind the port in precopy mode is too hight so there is something else hurting performance which is why im suspecign you do not have caching of keysotne configure correctly.
23:52:47 abaindur Ok no, memcache_servers is not set...
23:52:58 abaindur this is for nova compute on the hypervisor side right?
23:53:44 sean-k-mooney https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6982-L7008 could be safely moved to https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6939 by the way and in the libvirt case https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6933 woudl be fine too.
23:53:57 sean-k-mooney am it will be used both on the contoler and computes
23:54:13 sean-k-mooney so you are missing
23:54:16 sean-k-mooney [keystone_authtoken]
23:54:18 sean-k-mooney memcached_servers = localhost:11211
23:54:25 sean-k-mooney or well your actull memcache servers
23:54:45 abaindur yeah, I dont see that config opt ever ok
23:54:49 sean-k-mooney this will be used to cach auth tokens for every api call we make
23:55:16 abaindur i dont see it ever set*
23:58:21 sean-k-mooney abaindur: so at least on the contoler side it has a significnat impact on the api
23:58:25 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1836642
23:58:25 openstack Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Incomplete] - Assigned to Slawek Kaplonski (slaweq)
23:58:53 sean-k-mooney we adress this porblem in the gate by truning on caching with https://github.com/openstack/devstack/commit/d33cdd01f83b891b010e0fd238f1816910f3fd77
23:59:18 abaindur I dont see it on controller either
23:59:25 sean-k-mooney i am not sure if i tis use dby the compute node but i think it will be whenever it is calling cinder neutron or placment
23:59:35 sean-k-mooney abaindur: its optional
23:59:49 sean-k-mooney but it improves perfromacne alot
#openstack-nova - 2020-04-21
00:00:01 abaindur so for pre-copy, you are suggesting to try memcache for keystone tokens?
00:00:29 sean-k-mooney ya but im not saying dont try enableing post-copy
00:00:48 sean-k-mooney im just sugging that memcache might impove the performace enough to not need it
00:00:54 sean-k-mooney i woudl test both
00:01:12 sean-k-mooney abaindur: how did you deploy openstack by the way
00:01:23 abaindur thanks, yea we'll probably try both options then to see which leads to least downtime
00:01:51 sean-k-mooney for both options is basically just a compute agent restart and cofig update
00:03:24 sean-k-mooney abaindur: at least in the case of https://bugs.launchpad.net/nova/+bug/1836642 we were seeing edgcaces wher ethe metadta api respoce could take upwards of 12 - 16 seconds and that went away when we enabeld the cache
00:03:24 openstack Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Incomplete] - Assigned to Slawek Kaplonski (slaweq)
00:05:00 sean-k-mooney abaindur: although to be clear we enabel both the cache section and the memcahced_server in the keystone_authoken section
00:05:18 sean-k-mooney so all of the highlight section is how we adressed the isseu https://zuul.opendev.org/t/openstack/build/dcde79801a624c25b195a46ead7af562/log/controller/logs/etc/nova/nova-cpu_conf.txt#62-68
00:09:27 sean-k-mooney abaindur: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time might be of interst
00:11:30 sean-k-mooney i think that might only be relevent to the apis service however and might not apply to when we use the keystone auth client to talk to otehr apis but im not sure baout that. anyway its late here so im going to go o/
00:11:58 abaindur sean-k-mooney: oh wait, we have memcached_servers set in keystone_authtoken
00:12:05 abaindur but we dont have a [cache] section with the other configs
00:12:37 sean-k-mooney ah the cache section is novas own internal cache config
00:13:07 sean-k-mooney so in the metadata api case it is what allows cache copies fo the metadata to be shareed between processes
00:14:00 sean-k-mooney i think your best path forward in that cae is likely your original one of enabling post copy. you might also want o consider auto convergence
00:27:50 abaindur ok thanks for the help! will update here later if i find out more
00:42:47 openstackgerrit Merged openstack/nova master: Allocate mdevs when resizing or reverting resize https://review.opendev.org/712741
01:50:40 openstackgerrit Merged openstack/nova stable/train: Reject boot request for unsupported images https://review.opendev.org/708577
06:38:58 gibi stephenfin, gmann, artom: I'm +2 on the policy code changes.
06:48:53 gibi I'm tempted to squash the doc follow up into the original doc patch for more readable diff but meh I wait for others to show up
06:55:19 bauzas gibi: gmann: fwiw, I'll leave the prelude section change pretty small for the policy changes, but you are free to amend it for clarifying what we finally got for policy changes
06:55:37 bauzas hopefully, will be posted later this morning
06:55:43 gibi bauzas: thanks
06:55:46 bauzas (and good morning Nova, sorry for my manners)
06:57:07 bauzas gibi: tbh, I'm mostly paraphrasing the cycle highlights with a couple of additions (the BFV instance rescue and the vGPU multiple types support)
06:57:18 bauzas gibi: worth mentioning others ?
06:58:00 gibi I think that is plenty
06:58:59 bauzas gibi: well, I'm not afraid, that will still be an order of magnitude less than the rest of the relnotes
06:59:15 bauzas https://blueprints.launchpad.net/nova/+spec/remove-xvpvncproxy and https://blueprints.launchpad.net/nova/+spec/remove-nova-network-ussuri should also be commented I tihnk
06:59:40 bauzas gibi: fwiw, this is how it looks at the moment without a prelude section https://docs.openstack.org/releasenotes/nova/unreleased.html
06:59:43 bauzas good luck with this.
07:00:17 bauzas anyway, late breakfast, kids are up
07:01:03 gibi hm, based on that I even highlight the fact that we dropped supporting python 2.7
07:25:32 gibi sean-k-mooney: I guess you have SRIOV env at hand. Could you check if the bug https://bugs.launchpad.net/nova/+bug/1851545 is valid?
07:25:32 openstack Launchpad bug 1851545 in OpenStack Compute (nova) "Port update exception on nova unshelve for instance with PCI devices (part 2)" [Undecided,New]
07:26:01 gibi for me it would take a day or so to set one up
07:37:24 bauzas gibi: good point, okay, let's wrap this up and see what people think
07:38:27 gibi bauzas: yep, lets do the review on gerrit :)
07:59:58 elod zigo melwitt : about https://bugs.launchpad.net/nova/+bug/1788014 , I realized that it needs maybe some refactoring... then I forgot about looking at it. :/ Will do today. I'll need some thorough review for that i guess.
07:59:58 openstack Launchpad bug 1788014 in OpenStack Compute (nova) rocky "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Elod Illes (elod-illes)
08:00:26 zigo elod: Thanks !
08:06:07 bauzas gosh, the tox target for relnotes takes a while recalculating all the relnotes...
08:38:49 openstackgerrit Sylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section https://review.opendev.org/721548
08:38:59 bauzas wow, it took a while but I'm done
08:39:42 openstackgerrit Merged openstack/nova master: Add new default roles in quota class policies https://review.opendev.org/719100

Earlier   Later