Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-05
21:20:21 spatel I am seeing strange performance issue, I am running erlang application on openstack instance which is giving me very poor performance but if i run same application on bare metal performance is really good
21:21:01 spatel I have created big VM and allocated all vCPU to instance so only single VM running on compute node..
21:21:17 spatel trying to understand where is the bottleneck
21:31:36 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Fetch 'Service' objects once when building AZs https://review.opendev.org/705653
21:31:37 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Bump minimum version of websockify https://review.opendev.org/705654
21:31:37 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Merge unnecessary 'NovaProxyRequestHandlerBase' separation https://review.opendev.org/705655
21:31:38 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Remove 'run_once' helper https://review.opendev.org/705656
21:31:38 openstackgerrit Stephen Finucane proposed openstack/nova master: tox: Integrate mypy https://review.opendev.org/676208
21:31:39 openstackgerrit Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci' https://review.opendev.org/676209
21:31:39 openstackgerrit Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/705657
21:31:40 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: mypy: Add type annotations to top-level modules https://review.opendev.org/705658
21:32:42 sean-k-mooney spatel: erlang is pretty memory sensitive so using hugepages in the gust and likely on the host for the vm should help
21:33:20 sean-k-mooney spatel: you shoudl also look into http://erlang.org/doc/man/HiPE_app.html
21:35:15 openstackgerrit Merged openstack/nova master: Handle neutron without the fip-port-details extension https://review.opendev.org/705760
21:38:42 spatel sean-k-mooney: i am using hugepage
21:38:52 spatel I have tried all possible best practices
21:39:56 spatel earlier i gave all CPU to vm (32 core) so thinking i should just create VM with 16 vCPU so it stay one side of NUMA zone to see how its performing..
21:42:30 sean-k-mooney ya numa affinitising the vm might help or create a dual numa guest
21:42:56 sean-k-mooney that would report the numa affintiy to erlang but if you are using hugepags it has a numa toplogy of 1 already
21:43:03 spatel I am only curious if i run same benchmark on bare metal then i get good result but if i run on VM then it gives poor result
21:43:16 sean-k-mooney what it could be is the cpu model
21:43:34 sean-k-mooney if you are not useing host-passthouhg you might not have all the cpu feature in the vm that you have in the host
21:43:36 spatel my CPU mode is passthrough
21:43:58 sean-k-mooney well if you have it numa affined and cpu passthohg
21:44:03 sean-k-mooney are you using cpu pinning
21:44:06 spatel Yes
21:44:11 spatel CPU pinning
21:44:27 sean-k-mooney then the only bottelnecs you have left are disk io or netwrok
21:44:52 sean-k-mooney what disk cache mode are you using
21:44:53 spatel I am using SR-IOV and not seeing any single packetloss or performance issue on network layer
21:45:08 spatel We have SSD
21:45:15 spatel not sure about disk cache mode?
21:45:47 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes
21:46:08 spatel I have noticed in SS command lost of packet getting in queue and increasing latency
21:46:29 sean-k-mooney file=writeback,block=writeback,network=writeback would be the highest performace setting in most cases
21:48:06 spatel hmm
21:48:27 spatel I will try that too
21:48:50 spatel is that kvm setting ?
21:49:34 sean-k-mooney ya its a setting for the libvirt dirver
21:49:38 spatel I have flavor setting numa_node=2
21:49:41 sean-k-mooney so in the [libvirt] section
21:49:58 spatel ok.. i will give it a try and see if that make any difference
21:50:22 spatel tomorrow i am going to run more erlang test and see where is the bottleneck.. i will keep you posted..
22:26:05 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Deprecate base rules in favor of new rules https://review.opendev.org/701624
22:26:31 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix os-attach-interfaces policy to be admin_or_owner https://review.opendev.org/705135
22:26:44 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Remove old policy enforcement in attach_interfaces https://review.opendev.org/705127
22:26:57 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing attach_interfaces policies https://review.opendev.org/705126
22:27:10 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in os-attach-interfaces https://review.opendev.org/705799
22:44:25 efried gibi: stephenfin, other: how many people were in the room at the Shanghai PTG?
23:36:01 alex_xu efried: 20 maybe
#openstack-nova - 2020-02-06
00:44:19 efried Thanks alex_xu
03:00:28 openstackgerrit Merged openstack/nova master: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/682523
03:38:39 openstackgerrit Qiu Fossen proposed openstack/nova-specs master: specify mac for creating instance https://review.opendev.org/700429
05:16:14 openstackgerrit Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to server actions API https://review.opendev.org/706179
06:50:59 openstackgerrit Merged openstack/nova master: Avoid calling neutron for N networks https://review.opendev.org/705784
07:02:33 openstackgerrit Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API https://review.opendev.org/706179
07:53:36 gibi efried: yeah max 20 I t was more on the 12 - 15 side in my memory
07:54:45 gibi efried, dansmith: I was avoiding cyborg integration code recently as I saw that both of you are actively handled that but now I feel you want me to read in. I hope I will not blow up things with my fresh oppinion
08:04:09 openstackgerrit Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430
09:30:56 ralonsoh stephenfin, sorry for bothering you again...
09:30:57 ralonsoh https://review.opendev.org/#/c/705760/2/nova/network/neutron.py@2615
09:31:12 ralonsoh I didn't realize this when I reviewed the patch
09:31:32 ralonsoh but a FIP can NOT be attached to a port
09:32:35 ralonsoh so this exception... is unnecessary. Actually we are still having some problems in the Neutron OVN CI
09:32:35 ralonsoh https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/703537/7/check/neutron-ovn-tempest-ovs-release/7b55444/testr_results.html
09:32:48 stephenfin damn. okay, let me fix that up
09:33:22 ralonsoh stephenfin, let me first check the test
09:33:33 ralonsoh I would like to confirm that in the logs
09:39:08 ralonsoh stephenfin, https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/703537/7/check/neutron-ovn-tempest-ovs-release/7b55444/controller/logs/screen-n-api.txt
09:39:27 ralonsoh we are creating IPs without any port associated
10:05:57 stephenfin ralonsoh: What happens if I do 'list_ports(ids=['foo'])' and 'foo' doesn't exist?
10:06:16 openstackgerrit Stephen Finucane proposed openstack/nova master: Don't error out on floating IPs without associated ports https://review.opendev.org/706213
10:06:19 stephenfin I assume I'll just get an empty response, i.e. {'ports': []}
10:06:34 stephenfin ralonsoh, gibi: That's the fix, btw ^
10:08:19 ralonsoh stephenfin, yes, you have an empty list
10:08:33 ralonsoh show_port will return an exception
10:08:39 ralonsoh list_ports don't
10:08:50 stephenfin cool, that's what I'd assumed in that patch
10:16:51 openstackgerrit Stephen Finucane proposed openstack/nova master: Rework how we check for extensions https://review.opendev.org/705792
11:09:01 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Rename _is_storage_shared_with to _is_path_shared_with https://review.opendev.org/693337
11:09:14 lyarwood stephenfin: ^ can you +W that again, had to rebase to clear the conflict.
11:09:30 stephenfin done
11:10:05 lyarwood ta
11:11:25 gibi stephenfin: a floating ip without a port is an unused floating ip?
11:11:55 openstackgerrit Guo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120
11:12:04 stephenfin gibi: Yup. We're using
11:13:11 stephenfin ...it for the 'GET /os-floating-ips' API, and that describes itself as "Return(ing) a list of floating IPs allocated to a project."
11:13:21 gibi cool, thanks
11:24:34 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" for Ussuri https://review.opendev.org/693844
11:25:45 brinzhang gibi, lyarwood: this is my spec's invalid link fix patch https://review.opendev.org/#/c/705933/, please review
11:27:19 brinzhang gibi, lyarwood: https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1663456 this is the bug 1663456 fix, one is +2 by stephenfin
11:27:19 openstack bug 1663456 in OpenStack Compute (nova) "Field 'updated_at' always 'None' when show aggregate" [Low,In progress] https://launchpad.net/bugs/1663456 - Assigned to Brin Zhang (zhangbailin)
12:09:25 openstackgerrit Brin Zhang proposed openstack/nova master: Add server actions v82 samples test https://review.opendev.org/706251
12:16:49 gibi stephenfin: fyi if you need I can reproduce the stestr bug https://github.com/mtreinish/stestr/pull/285 in a pretty stable way in nova
12:17:53 stephenfin gibi: I had this https://review.opendev.org/#/c/700522/
12:31:22 gibi stephenfin: you won :)
12:31:38 gibi stephenfin: is there a way I can help the stestr fix merged?
12:34:05 artom stephenfin, left you a drive-by comment on https://review.opendev.org/#/c/705792/3
12:37:12 openstackgerrit Arnaud Morin proposed openstack/nova master: Force refresh of instance info cache on event https://review.opendev.org/702394
12:55:01 openstackgerrit Martin Midolesov proposed openstack/nova master: Implementing graceful shutdown. https://review.opendev.org/666245

Earlier   Later