Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-06
11:49:07 sean-k-mooney so dpending on if your using ssd and your workload sometimes virtio-scsi perfroms better even for a vm with 1 root disk and no cinder volumes
11:49:23 sean-k-mooney virtio-scsi is our recomendation when ever using ceph
11:49:54 lyarwood that reminds me I need to read up on the virtio-blk trim support thread
11:50:00 sean-k-mooney but for local storage we generally suggest staying with virtio-blk unless you have messured a performace increase with virtio-scsi
11:51:06 sean-k-mooney virtio-scsi is general better when you use a large number of cinder volumes as it does not consume a pci device slot per volumn
11:51:30 sean-k-mooney so it scales better in that regard but again all io is handel by the qemu emulator thread since we do not use io threads
11:51:54 sean-k-mooney so that will be the bottle neck regardless of if you use virtio-blk or virtio-scsi
11:55:59 viks__ sean-k-mooney: ok thanks for the inputs.. i'll explore and test `virtio-scsi` also and see if i have some findings..
12:49:23 opendevreview Merged openstack/os-vif stable/ussuri: Refactor code of linux_net to more cleaner and increase performace https://review.opendev.org/c/openstack/os-vif/+/765419
12:50:14 viks__ sean-k-mooney: here is what i have tested... https://paste.openstack.org/show/808597/ These are from my default setup.. i see a huge difference... but as per you, it should not differ too much even in default setup.. am i doing something wrong or missing some configuration?
12:51:48 sean-k-mooney correct it should not differe much. so some questions what virt driverf are you using (libvirt?) what images_backend have you configured (qcow? raw? flat? lvm?)
12:52:13 sean-k-mooney have you set your disk cache mode or is it at the default and have you enabeld image preallocation our not
12:52:50 opendevreview Merged openstack/os-vif stable/ussuri: Fix - os-vif fails to get the correct UpLink Representor https://review.opendev.org/c/openstack/os-vif/+/765967
12:54:00 sean-k-mooney viks__: unless sysbench preallocates the file and then writes to it the large delta there may just be growing the root disk
12:54:10 sean-k-mooney viks__: so you might need to rerun that
12:54:24 sean-k-mooney for fio i know it preallocates and there we are seeign about a 2x delta
12:54:33 bauzas gibi: ok, so you need to create a new series for launchpad ?
12:54:43 bauzas I wasn"t knowing about it
12:55:39 bauzas gibi: could you please add this for https://docs.openstack.org/nova/latest/contributor/ptl-guide.html ?
12:56:24 bauzas ah, nevermind
12:56:25 bauzas https://docs.openstack.org/nova/latest/contributor/ptl-guide.html#immediately-after-rc
12:58:44 viks__ sean-k-mooney: when i rerun sysbench, then also i get the below, which is still way below the one on the host:
12:58:52 viks__ https://www.irccloud.com/pastebin/sf7VLtQi/
12:59:58 sean-k-mooney viks__: ack but without answering my ohter question i cant really help
13:00:56 sean-k-mooney viks__: the default parmaters that are used will depend on your deployment tool so that is why i am asking what you image_backend set to and cache modes also have you preallocated the image or not
13:02:40 viks__ sean-k-mooney: oh sorry... it's libvirt/qcow, `preallocate_images = none, disk_cachemodes = file=writeback,block=writeback`
13:05:32 sean-k-mooney ok so preallocat_image=none means tehre will be some degradation while the qcow is expanded but that does not account of rthe fio delta since it precallocates teh files in the guest to prevent that form being an issue
13:06:12 sean-k-mooney writeback is generally quite good for perfromance you coudl try settign the cache mode to none if you host suport O_DIRECT that might improve perfromance
13:06:49 sean-k-mooney if you leeve the cachemode unest we will default to none if O_DIRECT is support or use writeback which was our default until recently
13:09:20 viks__ sean-k-mooney: ok.. what is the easy way to check for `O_DIRECT` support in the host?
13:10:50 sean-k-mooney well you were using --direct=1 in your fio test so it should support it
13:11:06 sean-k-mooney O_DIRECT is the file mapping mode used for direct io
13:11:16 viks__ ok...
13:11:59 sean-k-mooney first thing i would try is setting the cachemode for file to none
13:12:07 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
13:12:10 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Fix allocation show / unset with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556
13:12:14 sean-k-mooney then hard reboot the vm and see if that improve the performance
13:12:57 viks__ sean-k-mooney: ok thanks... will try the same
13:14:29 sean-k-mooney if that has no effect i would try setting preallocate_images=space and then try changing images_backend to raw. you could also in parallel create a second vm with hw_disk_bus=virtio-scsi to compare side by side to virtio-blk
13:15:19 sean-k-mooney viks__: for really write heavy worklaods lvm used to slightly outperform raw but that requried extra setup and it is less well maintained
13:15:40 sean-k-mooney so if the other tuning dont help in your case that is what i woudl try last
13:16:46 sean-k-mooney viks__: outside of openstack there are some host level sysctl tunings you can do to improve guest io perfromance like changing the io schduler or dirty data writeback tresholds
13:18:31 sean-k-mooney tuned can be used as a erfernce for some of the commnly tuned settings
13:18:33 sean-k-mooney https://github.com/redhat-performance/tuned/blob/master/profiles/throughput-performance/tuned.conf#L24-L59
13:19:52 viks__ sean-k-mooney: thanks for all the inputs.. will do some testing based on these .... thanks again...
13:22:42 opendevreview Balazs Gibizer proposed openstack/nova master: DNM: check nova-next with osc-placement fix https://review.opendev.org/c/openstack/nova/+/807558
13:25:34 sean-k-mooney viks__: if you are testing virtio-scsi i the image options are hw_disk_bus=scsi hw_scsi_model=virtio-scsi by they way
13:25:51 sean-k-mooney i previously msityped it as hw_disk_bus=virtio-scsi
13:26:11 viks__ sean-k-mooney: ok...
14:06:16 bauzas huzzah, can see my review priorities : https://review.opendev.org/q/label:Review-Priority%253E%253D1%252Csbauza
14:20:19 opendevreview Merged openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
14:20:20 opendevreview Merged openstack/osc-placement master: Fix allocation show / unset with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556
14:41:23 gibi lyarwood, bauzas: ^^ thanks for the quick approve. I will ask for a release freeze exception shortly
14:53:41 gibi RFE requested: http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024686.html
14:55:35 slaweq hi stable nova cores, can somebody check https://review.opendev.org/c/openstack/nova/+/791421 /.
14:55:38 slaweq ?
14:55:44 slaweq thx in advance for help :)
15:28:58 gibi do we have USA out today due to Labor Day?
15:31:33 kashyap gibi: Yeah
15:31:40 kashyap Also Canada, IIRC
15:31:46 gibi OK, then I don't wait for them today :)
15:57:54 bauzas gibi: sorry was on a meeting
15:58:59 bauzas gibi: yup, saw your FFE request but as kashyap said and you guessed, a whole portion of the world located between Pacific and Atlantic oceans and above a certain latitude is currently shutdown for the day
15:59:20 gibi bauzas: no worries the release now queued
15:59:35 bauzas said a French guy working
15:59:59 gibi bauzas: I looked at melwitt's repro patch https://review.opendev.org/c/openstack/placement/+/807493 and I left some ideas but no real solutions yet
16:01:51 bauzas gibi: honestly we face the limits of the single-commit design we have with Placement
16:03:26 gibi bauzas: this is now more about writing a sane a repro test that excersise transaction isolation with mysql (as sqlite does not have it)
16:03:57 gibi I think the fix melwitt's proposed on top is sane but the repro test is racy somehow actually forks processes
16:04:02 gibi whihc is scarry
16:04:28 bauzas gibi: yup, I saw the tox modification and I understood the reasoning
16:12:04 melwitt showing the effect of the consistent read problem inside the transaction
16:12:04 melwitt I will try to find a different way to do it. it's inherently problematic to try and repro this because once one path starts a transaction, trying to do something in the middle of it (to cause the race state) poses a record locking problem. at one point I was trying to fake it by just returning bogus things to make it think it hit a generation conflict (without doing any database write), which worked to make it retry but that wasn't
16:21:40 gibi melwitt: yes, exactly, that was the dead end I went down this afternoon :/
16:22:16 gibi melwitt: I think at some point we can accept that we cannot reliably test this in func env and simply land the fix and monitor the tempest jobs to see if it is resolved the race or not
16:22:54 melwitt gibi: I went down the dead end three days in a row 😑
16:23:04 melwitt gibi: yeah. I was thinking that too. I kept thinking there has to be a way but if there is I'm not clever enough to find it haha
16:24:06 gibi melwitt: but enjoy your day off, we can continue this tomorrow
16:24:53 melwitt kk, thanks
#openstack-nova - 2021-09-07
01:49:39 opendevreview Jorhson Deng proposed openstack/nova master: recheck the attachment_id after the reschedule successful https://review.opendev.org/c/openstack/nova/+/796209
03:15:06 opendevreview Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519
07:57:50 opendevreview Wenping Song proposed openstack/placement master: Dropping lower constraints testing https://review.opendev.org/c/openstack/placement/+/787863
08:45:57 bbezak Hi - Can we make a release os-vif for Victoria? 2.2.1? this bug is quite critical - https://bugs.launchpad.net/os-vif/+bug/1892132. And it is not part of 2.2.0 release
09:02:02 sean-k-mooney its not actully a bug it was a kernel abi break
09:02:31 sean-k-mooney but yes we can proably do a release we just need too propose it to the release repo
09:03:00 sean-k-mooney bbezak: the release managmen it automated via a git repo
09:04:04 sean-k-mooney so we just need to update this with the corerct sha https://github.com/openstack/releases/blob/master/deliverables/victoria/os-vif.yaml
09:04:31 sean-k-mooney there is a script in the release repo for doing that or you can do it by hand
09:05:59 sean-k-mooney https://github.com/openstack/os-vif/compare/2.2.0...stable/victoria
09:06:16 kashyap sean-k-mooney: What's the kernel ABI break?
09:06:25 bbezak indeed - OFED 5.4 also introduced that change in renumeration at some level.
09:06:25 sean-k-mooney looking at the delta it does not look like there is much else included but release are pretty cheap
09:06:31 bbezak https://bugzilla.redhat.com/show_bug.cgi?id=1918703
09:06:40 kashyap Ah, a commit is linked - https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1
09:06:42 bbezak kashyap: https://bugzilla.redhat.com/show_bug.cgi?id=1918703
09:06:51 kashyap Thanks
09:07:00 sean-k-mooney kashyap: melonox moved where the reprentor netdevs were advertised in /sys
09:07:26 sean-k-mooney we have fixed os-vif and libvirt to be able to handel both now
09:07:37 sean-k-mooney but we need to actully release os-vif with the fix
09:07:42 kashyap I see

Earlier   Later