| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-20 | |||
| 23:41:28 | abaindur | yea, we tried VMs both volume and ephemeral based | |
| 23:41:42 | sean-k-mooney | did you see the same delay? | |
| 23:42:03 | abaindur | pretty much | |
| 23:42:24 | abaindur | timed the volume code with some logs of our own, heres what we observed: | |
| 23:42:36 | abaindur | 1.913 seconds for _get_instance_block_device_info and self.driver.post_live_migration | |
| 23:42:45 | abaindur | about 0.4 sec for self.driver.get_volume_connector(instance) | |
| 23:42:51 | abaindur | 2.659 seconds for self.volume_api.terminate_connection() | |
| 23:42:58 | abaindur | 3.28 seconds for network_info = self.network_api.get_instance_nw_info(ctxt, instance) | |
| 23:43:03 | abaindur | 5.019 seconds for self._notify_about_instance_usage(ctxt, instance, "live_migration._post.start", network_info=network_info) | |
| 23:43:16 | abaindur | another almost 5 seconds nova spent just making the port binding activate API call, seems to be spending time in keystone and oslo_concurrency.lockutils code | |
| 23:43:25 | sean-k-mooney | is that absolute time of time for each function | |
| 23:43:44 | abaindur | yea, we just added logs throughout that post_live_migration() function | |
| 23:43:56 | abaindur | before/after each of those calls to see what was taking so long | |
| 23:44:01 | sean-k-mooney | ok so i subtrac those numebr to get the time for each | |
| 23:44:07 | sean-k-mooney | that is still very slow | |
| 23:44:30 | abaindur | for example, in one live migration, we say: VM Paused at 25:06.192 | |
| 23:44:39 | sean-k-mooney | abaindur: are you using memcache for you keysontone auth tokens | |
| 23:45:32 | abaindur | by time we saw nova-compute make the port binding call, it was at: 35:24.694 | |
| 23:46:20 | abaindur | VM Paused at 35:06.192 * | |
| 23:46:49 | sean-k-mooney | abaindur: can you check your nova.conf and see if you have https://zuul.opendev.org/t/openstack/build/dcde79801a624c25b195a46ead7af562/log/controller/logs/etc/nova/nova-cpu_conf.txt#62-68 | |
| 23:49:04 | sean-k-mooney | also 10 seconds to bind the port in precopy mode is too hight so there is something else hurting performance which is why im suspecign you do not have caching of keysotne configure correctly. | |
| 23:52:47 | abaindur | Ok no, memcache_servers is not set... | |
| 23:52:58 | abaindur | this is for nova compute on the hypervisor side right? | |
| 23:53:44 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6982-L7008 could be safely moved to https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6939 by the way and in the libvirt case https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L6933 woudl be fine too. | |
| 23:53:57 | sean-k-mooney | am it will be used both on the contoler and computes | |
| 23:54:13 | sean-k-mooney | so you are missing | |
| 23:54:16 | sean-k-mooney | [keystone_authtoken] | |
| 23:54:18 | sean-k-mooney | memcached_servers = localhost:11211 | |
| 23:54:25 | sean-k-mooney | or well your actull memcache servers | |
| 23:54:45 | abaindur | yeah, I dont see that config opt ever ok | |
| 23:54:49 | sean-k-mooney | this will be used to cach auth tokens for every api call we make | |
| 23:55:16 | abaindur | i dont see it ever set* | |
| 23:58:21 | sean-k-mooney | abaindur: so at least on the contoler side it has a significnat impact on the api | |
| 23:58:25 | openstack | Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Incomplete] - Assigned to Slawek Kaplonski (slaweq) | |
| 23:58:25 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1836642 | |
| 23:58:53 | sean-k-mooney | we adress this porblem in the gate by truning on caching with https://github.com/openstack/devstack/commit/d33cdd01f83b891b010e0fd238f1816910f3fd77 | |
| 23:59:18 | abaindur | I dont see it on controller either | |
| 23:59:25 | sean-k-mooney | i am not sure if i tis use dby the compute node but i think it will be whenever it is calling cinder neutron or placment | |
| 23:59:35 | sean-k-mooney | abaindur: its optional | |
| 23:59:49 | sean-k-mooney | but it improves perfromacne alot | |
| #openstack-nova - 2020-04-21 | |||
| 00:00:01 | abaindur | so for pre-copy, you are suggesting to try memcache for keystone tokens? | |
| 00:00:29 | sean-k-mooney | ya but im not saying dont try enableing post-copy | |
| 00:00:48 | sean-k-mooney | im just sugging that memcache might impove the performace enough to not need it | |
| 00:00:54 | sean-k-mooney | i woudl test both | |
| 00:01:12 | sean-k-mooney | abaindur: how did you deploy openstack by the way | |
| 00:01:23 | abaindur | thanks, yea we'll probably try both options then to see which leads to least downtime | |
| 00:01:51 | sean-k-mooney | for both options is basically just a compute agent restart and cofig update | |
| 00:03:24 | openstack | Launchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Incomplete] - Assigned to Slawek Kaplonski (slaweq) | |
| 00:03:24 | sean-k-mooney | abaindur: at least in the case of https://bugs.launchpad.net/nova/+bug/1836642 we were seeing edgcaces wher ethe metadta api respoce could take upwards of 12 - 16 seconds and that went away when we enabeld the cache | |
| 00:05:00 | sean-k-mooney | abaindur: although to be clear we enabel both the cache section and the memcahced_server in the keystone_authoken section | |
| 00:05:18 | sean-k-mooney | so all of the highlight section is how we adressed the isseu https://zuul.opendev.org/t/openstack/build/dcde79801a624c25b195a46ead7af562/log/controller/logs/etc/nova/nova-cpu_conf.txt#62-68 | |
| 00:09:27 | sean-k-mooney | abaindur: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time might be of interst | |
| 00:11:30 | sean-k-mooney | i think that might only be relevent to the apis service however and might not apply to when we use the keystone auth client to talk to otehr apis but im not sure baout that. anyway its late here so im going to go o/ | |
| 00:11:58 | abaindur | sean-k-mooney: oh wait, we have memcached_servers set in keystone_authtoken | |
| 00:12:05 | abaindur | but we dont have a [cache] section with the other configs | |
| 00:12:37 | sean-k-mooney | ah the cache section is novas own internal cache config | |
| 00:13:07 | sean-k-mooney | so in the metadata api case it is what allows cache copies fo the metadata to be shareed between processes | |
| 00:14:00 | sean-k-mooney | i think your best path forward in that cae is likely your original one of enabling post copy. you might also want o consider auto convergence | |
| 00:27:50 | abaindur | ok thanks for the help! will update here later if i find out more | |
| 00:42:47 | openstackgerrit | Merged openstack/nova master: Allocate mdevs when resizing or reverting resize https://review.opendev.org/712741 | |
| 01:50:40 | openstackgerrit | Merged openstack/nova stable/train: Reject boot request for unsupported images https://review.opendev.org/708577 | |
| 06:38:58 | gibi | stephenfin, gmann, artom: I'm +2 on the policy code changes. | |
| 06:48:53 | gibi | I'm tempted to squash the doc follow up into the original doc patch for more readable diff but meh I wait for others to show up | |
| 06:55:19 | bauzas | gibi: gmann: fwiw, I'll leave the prelude section change pretty small for the policy changes, but you are free to amend it for clarifying what we finally got for policy changes | |
| 06:55:37 | bauzas | hopefully, will be posted later this morning | |
| 06:55:43 | gibi | bauzas: thanks | |
| 06:55:46 | bauzas | (and good morning Nova, sorry for my manners) | |
| 06:57:07 | bauzas | gibi: tbh, I'm mostly paraphrasing the cycle highlights with a couple of additions (the BFV instance rescue and the vGPU multiple types support) | |
| 06:57:18 | bauzas | gibi: worth mentioning others ? | |
| 06:58:00 | gibi | I think that is plenty | |
| 06:58:59 | bauzas | gibi: well, I'm not afraid, that will still be an order of magnitude less than the rest of the relnotes | |
| 06:59:15 | bauzas | https://blueprints.launchpad.net/nova/+spec/remove-xvpvncproxy and https://blueprints.launchpad.net/nova/+spec/remove-nova-network-ussuri should also be commented I tihnk | |
| 06:59:40 | bauzas | gibi: fwiw, this is how it looks at the moment without a prelude section https://docs.openstack.org/releasenotes/nova/unreleased.html | |
| 06:59:43 | bauzas | good luck with this. | |
| 07:00:17 | bauzas | anyway, late breakfast, kids are up | |
| 07:01:03 | gibi | hm, based on that I even highlight the fact that we dropped supporting python 2.7 | |
| 07:25:32 | openstack | Launchpad bug 1851545 in OpenStack Compute (nova) "Port update exception on nova unshelve for instance with PCI devices (part 2)" [Undecided,New] | |
| 07:25:32 | gibi | sean-k-mooney: I guess you have SRIOV env at hand. Could you check if the bug https://bugs.launchpad.net/nova/+bug/1851545 is valid? | |
| 07:26:01 | gibi | for me it would take a day or so to set one up | |
| 07:37:24 | bauzas | gibi: good point, okay, let's wrap this up and see what people think | |
| 07:38:27 | gibi | bauzas: yep, lets do the review on gerrit :) | |
| 07:59:58 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) rocky "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Elod Illes (elod-illes) | |
| 07:59:58 | elod | zigo melwitt : about https://bugs.launchpad.net/nova/+bug/1788014 , I realized that it needs maybe some refactoring... then I forgot about looking at it. :/ Will do today. I'll need some thorough review for that i guess. | |
| 08:00:26 | zigo | elod: Thanks ! | |
| 08:06:07 | bauzas | gosh, the tox target for relnotes takes a while recalculating all the relnotes... | |
| 08:38:49 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section https://review.opendev.org/721548 | |
| 08:38:59 | bauzas | wow, it took a while but I'm done | |
| 08:39:42 | openstackgerrit | Merged openstack/nova master: Add new default roles in quota class policies https://review.opendev.org/719100 | |
| 08:39:48 | bauzas | gibi: stephenfin: dansmith: melwitt: alex_xu: gmann: others, the prelude section patch is up for reviews https://review.opendev.org/721548 | |
| 08:40:34 | bauzas | lyarwood: too ^ | |
| 08:41:51 | bauzas | gibi: can I hold the bugs lock ? | |
| 08:42:32 | openstack | Launchpad bug 1831315 in OpenStack Compute (nova) "nova-manage cell_v2 discover_hosts fails for IPv6 - ValueError: invalid literal for int() with base 10 - db connection URI gets its brackets eaten" [Undecided,New] - Assigned to Harald Jensås (harald-jensas) | |
| 08:42:32 | gibi | bauzas: I'm looking at https://bugs.launchpad.net/nova/+bug/1831315 right now but after that I have to jump to your prelude patch so you can have the lock | |
| 08:42:32 | openstack | Launchpad bug 1831315 in OpenStack Compute (nova) "nova-manage cell_v2 discover_hosts fails for IPv6 - ValueError: invalid literal for int() with base 10 - db connection URI gets its brackets eaten" [Undecided,New] - Assigned to Harald Jensås (harald-jensas) | |
| 08:43:19 | bauzas | gibi: you don't order bugs by Age, right? | |
| 08:47:02 | gibi | bauzas: I skip those that I cannot do anyithing with | |
| 08:47:27 | bauzas | ok, I'll try to look at them and ask for more if needed | |
| 08:47:50 | gibi | thanks | |
| 09:00:33 | openstack | Launchpad bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,New] | |
| 09:00:33 | bauzas | gibi: for example, https://bugs.launchpad.net/nova/+bug/1860312 looks unrelated to Nova | |