Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-06
06:58:13 frickler gibi: looks placement-y once again. maybe https://review.opendev.org/c/openstack/releases/+/806580
06:58:20 gibi frickler: yes
06:58:35 gibi yes placement-y
06:58:43 gibi I will look into it shortly
06:58:53 gibi could be the new osc-placement release, yes
07:14:56 lyarwood gibi: Morning, I'm going to be afk this morning looking after my sick kid once again, I'll try to help with the nova-next stuff this afternooon \o
07:15:20 gibi lyarwood: ack, hope the kid will be better soon
07:39:37 bauzas morning folks
07:39:53 bauzas gibi: ack, will look up, I need to write the prelude section :D
07:40:25 gibi bauzas: morning, I'm debugging the nova-next job now, so you are free to work with the prelude :)
07:44:30 bauzas :D
07:44:47 bauzas (good opportunity for looking at what we eventually merged during this cycle fwiw)
07:45:09 bauzas (in case people want to work on a prelude change for Yoga :p )
07:57:41 gibi frickler: indeed the new osc-placement release broke the test we have a bug in https://review.opendev.org/c/openstack/osc-placement/+/804458
07:58:06 gibi frickler: I will propose a fix for osc-placement. Can we release a new osc-placement lib for Xena?
07:58:19 gibi elodilles: ^^ ?
07:59:40 frickler gibi: I looked at that patch, but assumed it would only change behavior when actively requesting that version
07:59:59 frickler gibi: releasing bug-fixes should always be possible, too
08:00:08 gibi frickler: it moved some code around outside of the version guard
08:17:35 opendevreview Jorhson Deng proposed openstack/nova master: recheck the attachment_id after the reschedule successful https://review.opendev.org/c/openstack/nova/+/796209
08:52:53 opendevreview Pierre-Samuel Le Stang proposed openstack/nova master: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/807551
09:12:18 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
09:12:41 gibi bauzas: ^^
09:13:16 gibi sorry, I was too fast, it will fail ...
09:22:04 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
09:27:16 opendevreview Pierre-Samuel Le Stang proposed openstack/nova master: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/807551
09:27:16 opendevreview Pierre-Samuel Le Stang proposed openstack/nova master: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/807555
09:39:44 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
09:48:58 opendevreview Pierre-Samuel Le Stang proposed openstack/nova master: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/807551
09:51:45 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Fix allocation show / unset on empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556
09:53:14 gibi bauzas: please prioritize ^^
09:57:32 opendevreview Balazs Gibizer proposed openstack/nova master: DNM: check nova-next with osc-placement fix https://review.opendev.org/c/openstack/nova/+/807558
10:49:59 gibi bauzas: fyi, I've created the yoga series in launchpad https://launchpad.net/nova/yoga
10:50:25 gibi bauzas: I will set it to active once Xena RC1 is cut
10:59:28 sean-k-mooney hehe i proably shoudl do that for os-vif although i generally only do it when we need to backport something and the series does not exist
10:59:50 sean-k-mooney the last one i created was victoria...
10:59:54 gibi sean-k-mooney: I use the series and the milestones to track bps
11:00:19 sean-k-mooney ya for os-vif we just have bugs for RFEs so it tend to be less important
11:00:29 sean-k-mooney ill quickly add wallaby and yoga
11:02:27 gibi ack
11:02:31 viks__ hi, i'm doing some disk I/O testing in my openstack setup... In my test, i see that write speed in VM is very less compared to that of host.. So is there any way i can increase the VM disk I/O? is there any comparision or something which says what percentage ratio disk write speed gets degraded in VM comparison to host? Can some plz give some direction?
11:02:34 sean-k-mooney ok done
11:07:10 opendevreview Balazs Gibizer proposed openstack/nova master: [doc] port-resource-request-groups not landed in Xena https://review.opendev.org/c/openstack/nova/+/807564
11:41:32 sean-k-mooney viks__: it should be close to identical at least in the raw backend
11:42:19 sean-k-mooney for qcow it should also be close but we expect some overhead while the file is growing. we dont really maintian the lvm image_backend activly but it used to give slightly better write performnce the qcow but i think raw was on par
11:42:56 sean-k-mooney you may want to exlopre useing virtio-scsi instaed of virtio-blk which is our default
11:43:35 sean-k-mooney viks__: we do not support multiple io treads or multiqueue currently so if you have multiple disk the performace will not scale per disk
11:44:38 sean-k-mooney but for a vm with a singel root disk and preallcoated stroage the perfromance shoudl be close (within singel digit percent) to that of the host
11:47:25 viks__ sean-k-mooney: is `virtio-scsi` applies for vm with single root disk/preallcoated storage or cinder volumes attached?
11:48:03 sean-k-mooney when you enabel virtio-scsi via hw_disk_bus it is used for all storage
11:48:28 sean-k-mooney my suggestion to try it is because it supports a slightly different feature set like trim
11:49:07 sean-k-mooney so dpending on if your using ssd and your workload sometimes virtio-scsi perfroms better even for a vm with 1 root disk and no cinder volumes
11:49:23 sean-k-mooney virtio-scsi is our recomendation when ever using ceph
11:49:54 lyarwood that reminds me I need to read up on the virtio-blk trim support thread
11:50:00 sean-k-mooney but for local storage we generally suggest staying with virtio-blk unless you have messured a performace increase with virtio-scsi
11:51:06 sean-k-mooney virtio-scsi is general better when you use a large number of cinder volumes as it does not consume a pci device slot per volumn
11:51:30 sean-k-mooney so it scales better in that regard but again all io is handel by the qemu emulator thread since we do not use io threads
11:51:54 sean-k-mooney so that will be the bottle neck regardless of if you use virtio-blk or virtio-scsi
11:55:59 viks__ sean-k-mooney: ok thanks for the inputs.. i'll explore and test `virtio-scsi` also and see if i have some findings..
12:49:23 opendevreview Merged openstack/os-vif stable/ussuri: Refactor code of linux_net to more cleaner and increase performace https://review.opendev.org/c/openstack/os-vif/+/765419
12:50:14 viks__ sean-k-mooney: here is what i have tested... https://paste.openstack.org/show/808597/ These are from my default setup.. i see a huge difference... but as per you, it should not differ too much even in default setup.. am i doing something wrong or missing some configuration?
12:51:48 sean-k-mooney correct it should not differe much. so some questions what virt driverf are you using (libvirt?) what images_backend have you configured (qcow? raw? flat? lvm?)
12:52:13 sean-k-mooney have you set your disk cache mode or is it at the default and have you enabeld image preallocation our not
12:52:50 opendevreview Merged openstack/os-vif stable/ussuri: Fix - os-vif fails to get the correct UpLink Representor https://review.opendev.org/c/openstack/os-vif/+/765967
12:54:00 sean-k-mooney viks__: unless sysbench preallocates the file and then writes to it the large delta there may just be growing the root disk
12:54:10 sean-k-mooney viks__: so you might need to rerun that
12:54:24 sean-k-mooney for fio i know it preallocates and there we are seeign about a 2x delta
12:54:33 bauzas gibi: ok, so you need to create a new series for launchpad ?
12:54:43 bauzas I wasn"t knowing about it
12:55:39 bauzas gibi: could you please add this for https://docs.openstack.org/nova/latest/contributor/ptl-guide.html ?
12:56:24 bauzas ah, nevermind
12:56:25 bauzas https://docs.openstack.org/nova/latest/contributor/ptl-guide.html#immediately-after-rc
12:58:44 viks__ sean-k-mooney: when i rerun sysbench, then also i get the below, which is still way below the one on the host:
12:58:52 viks__ https://www.irccloud.com/pastebin/sf7VLtQi/
12:59:58 sean-k-mooney viks__: ack but without answering my ohter question i cant really help
13:00:56 sean-k-mooney viks__: the default parmaters that are used will depend on your deployment tool so that is why i am asking what you image_backend set to and cache modes also have you preallocated the image or not
13:02:40 viks__ sean-k-mooney: oh sorry... it's libvirt/qcow, `preallocate_images = none, disk_cachemodes = file=writeback,block=writeback`
13:05:32 sean-k-mooney ok so preallocat_image=none means tehre will be some degradation while the qcow is expanded but that does not account of rthe fio delta since it precallocates teh files in the guest to prevent that form being an issue
13:06:12 sean-k-mooney writeback is generally quite good for perfromance you coudl try settign the cache mode to none if you host suport O_DIRECT that might improve perfromance
13:06:49 sean-k-mooney if you leeve the cachemode unest we will default to none if O_DIRECT is support or use writeback which was our default until recently
13:09:20 viks__ sean-k-mooney: ok.. what is the easy way to check for `O_DIRECT` support in the host?
13:10:50 sean-k-mooney well you were using --direct=1 in your fio test so it should support it
13:11:06 sean-k-mooney O_DIRECT is the file mapping mode used for direct io
13:11:16 viks__ ok...
13:11:59 sean-k-mooney first thing i would try is setting the cachemode for file to none
13:12:07 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
13:12:10 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Fix allocation show / unset with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556
13:12:14 sean-k-mooney then hard reboot the vm and see if that improve the performance
13:12:57 viks__ sean-k-mooney: ok thanks... will try the same
13:14:29 sean-k-mooney if that has no effect i would try setting preallocate_images=space and then try changing images_backend to raw. you could also in parallel create a second vm with hw_disk_bus=virtio-scsi to compare side by side to virtio-blk
13:15:19 sean-k-mooney viks__: for really write heavy worklaods lvm used to slightly outperform raw but that requried extra setup and it is less well maintained
13:15:40 sean-k-mooney so if the other tuning dont help in your case that is what i woudl try last
13:16:46 sean-k-mooney viks__: outside of openstack there are some host level sysctl tunings you can do to improve guest io perfromance like changing the io schduler or dirty data writeback tresholds
13:18:31 sean-k-mooney tuned can be used as a erfernce for some of the commnly tuned settings
13:18:33 sean-k-mooney https://github.com/redhat-performance/tuned/blob/master/profiles/throughput-performance/tuned.conf#L24-L59
13:19:52 viks__ sean-k-mooney: thanks for all the inputs.. will do some testing based on these .... thanks again...
13:22:42 opendevreview Balazs Gibizer proposed openstack/nova master: DNM: check nova-next with osc-placement fix https://review.opendev.org/c/openstack/nova/+/807558
13:25:34 sean-k-mooney viks__: if you are testing virtio-scsi i the image options are hw_disk_bus=scsi hw_scsi_model=virtio-scsi by they way
13:25:51 sean-k-mooney i previously msityped it as hw_disk_bus=virtio-scsi
13:26:11 viks__ sean-k-mooney: ok...

Earlier   Later