Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-06
12:56:25 bauzas https://docs.openstack.org/nova/latest/contributor/ptl-guide.html#immediately-after-rc
12:58:44 viks__ sean-k-mooney: when i rerun sysbench, then also i get the below, which is still way below the one on the host:
12:58:52 viks__ https://www.irccloud.com/pastebin/sf7VLtQi/
12:59:58 sean-k-mooney viks__: ack but without answering my ohter question i cant really help
13:00:56 sean-k-mooney viks__: the default parmaters that are used will depend on your deployment tool so that is why i am asking what you image_backend set to and cache modes also have you preallocated the image or not
13:02:40 viks__ sean-k-mooney: oh sorry... it's libvirt/qcow, `preallocate_images = none, disk_cachemodes = file=writeback,block=writeback`
13:05:32 sean-k-mooney ok so preallocat_image=none means tehre will be some degradation while the qcow is expanded but that does not account of rthe fio delta since it precallocates teh files in the guest to prevent that form being an issue
13:06:12 sean-k-mooney writeback is generally quite good for perfromance you coudl try settign the cache mode to none if you host suport O_DIRECT that might improve perfromance
13:06:49 sean-k-mooney if you leeve the cachemode unest we will default to none if O_DIRECT is support or use writeback which was our default until recently
13:09:20 viks__ sean-k-mooney: ok.. what is the easy way to check for `O_DIRECT` support in the host?
13:10:50 sean-k-mooney well you were using --direct=1 in your fio test so it should support it
13:11:06 sean-k-mooney O_DIRECT is the file mapping mode used for direct io
13:11:16 viks__ ok...
13:11:59 sean-k-mooney first thing i would try is setting the cachemode for file to none
13:12:07 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
13:12:10 opendevreview Balazs Gibizer proposed openstack/osc-placement master: Fix allocation show / unset with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556
13:12:14 sean-k-mooney then hard reboot the vm and see if that improve the performance
13:12:57 viks__ sean-k-mooney: ok thanks... will try the same
13:14:29 sean-k-mooney if that has no effect i would try setting preallocate_images=space and then try changing images_backend to raw. you could also in parallel create a second vm with hw_disk_bus=virtio-scsi to compare side by side to virtio-blk
13:15:19 sean-k-mooney viks__: for really write heavy worklaods lvm used to slightly outperform raw but that requried extra setup and it is less well maintained
13:15:40 sean-k-mooney so if the other tuning dont help in your case that is what i woudl try last
13:16:46 sean-k-mooney viks__: outside of openstack there are some host level sysctl tunings you can do to improve guest io perfromance like changing the io schduler or dirty data writeback tresholds
13:18:31 sean-k-mooney tuned can be used as a erfernce for some of the commnly tuned settings
13:18:33 sean-k-mooney https://github.com/redhat-performance/tuned/blob/master/profiles/throughput-performance/tuned.conf#L24-L59
13:19:52 viks__ sean-k-mooney: thanks for all the inputs.. will do some testing based on these .... thanks again...
13:22:42 opendevreview Balazs Gibizer proposed openstack/nova master: DNM: check nova-next with osc-placement fix https://review.opendev.org/c/openstack/nova/+/807558
13:25:34 sean-k-mooney viks__: if you are testing virtio-scsi i the image options are hw_disk_bus=scsi hw_scsi_model=virtio-scsi by they way
13:25:51 sean-k-mooney i previously msityped it as hw_disk_bus=virtio-scsi
13:26:11 viks__ sean-k-mooney: ok...
14:06:16 bauzas huzzah, can see my review priorities : https://review.opendev.org/q/label:Review-Priority%253E%253D1%252Csbauza
14:20:19 opendevreview Merged openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553
14:20:20 opendevreview Merged openstack/osc-placement master: Fix allocation show / unset with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556
14:41:23 gibi lyarwood, bauzas: ^^ thanks for the quick approve. I will ask for a release freeze exception shortly
14:53:41 gibi RFE requested: http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024686.html
14:55:35 slaweq hi stable nova cores, can somebody check https://review.opendev.org/c/openstack/nova/+/791421 /.
14:55:38 slaweq ?
14:55:44 slaweq thx in advance for help :)
15:28:58 gibi do we have USA out today due to Labor Day?
15:31:33 kashyap gibi: Yeah
15:31:40 kashyap Also Canada, IIRC
15:31:46 gibi OK, then I don't wait for them today :)
15:57:54 bauzas gibi: sorry was on a meeting
15:58:59 bauzas gibi: yup, saw your FFE request but as kashyap said and you guessed, a whole portion of the world located between Pacific and Atlantic oceans and above a certain latitude is currently shutdown for the day
15:59:20 gibi bauzas: no worries the release now queued
15:59:35 bauzas said a French guy working
15:59:59 gibi bauzas: I looked at melwitt's repro patch https://review.opendev.org/c/openstack/placement/+/807493 and I left some ideas but no real solutions yet
16:01:51 bauzas gibi: honestly we face the limits of the single-commit design we have with Placement
16:03:26 gibi bauzas: this is now more about writing a sane a repro test that excersise transaction isolation with mysql (as sqlite does not have it)
16:03:57 gibi I think the fix melwitt's proposed on top is sane but the repro test is racy somehow actually forks processes
16:04:02 gibi whihc is scarry
16:04:28 bauzas gibi: yup, I saw the tox modification and I understood the reasoning
16:12:04 melwitt showing the effect of the consistent read problem inside the transaction
16:12:04 melwitt I will try to find a different way to do it. it's inherently problematic to try and repro this because once one path starts a transaction, trying to do something in the middle of it (to cause the race state) poses a record locking problem. at one point I was trying to fake it by just returning bogus things to make it think it hit a generation conflict (without doing any database write), which worked to make it retry but that wasn't
16:21:40 gibi melwitt: yes, exactly, that was the dead end I went down this afternoon :/
16:22:16 gibi melwitt: I think at some point we can accept that we cannot reliably test this in func env and simply land the fix and monitor the tempest jobs to see if it is resolved the race or not
16:22:54 melwitt gibi: I went down the dead end three days in a row 😑
16:23:04 melwitt gibi: yeah. I was thinking that too. I kept thinking there has to be a way but if there is I'm not clever enough to find it haha
16:24:06 gibi melwitt: but enjoy your day off, we can continue this tomorrow
16:24:53 melwitt kk, thanks
#openstack-nova - 2021-09-07
01:49:39 opendevreview Jorhson Deng proposed openstack/nova master: recheck the attachment_id after the reschedule successful https://review.opendev.org/c/openstack/nova/+/796209
03:15:06 opendevreview Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519
07:57:50 opendevreview Wenping Song proposed openstack/placement master: Dropping lower constraints testing https://review.opendev.org/c/openstack/placement/+/787863
08:45:57 bbezak Hi - Can we make a release os-vif for Victoria? 2.2.1? this bug is quite critical - https://bugs.launchpad.net/os-vif/+bug/1892132. And it is not part of 2.2.0 release
09:02:02 sean-k-mooney its not actully a bug it was a kernel abi break
09:02:31 sean-k-mooney but yes we can proably do a release we just need too propose it to the release repo
09:03:00 sean-k-mooney bbezak: the release managmen it automated via a git repo
09:04:04 sean-k-mooney so we just need to update this with the corerct sha https://github.com/openstack/releases/blob/master/deliverables/victoria/os-vif.yaml
09:04:31 sean-k-mooney there is a script in the release repo for doing that or you can do it by hand
09:05:59 sean-k-mooney https://github.com/openstack/os-vif/compare/2.2.0...stable/victoria
09:06:16 kashyap sean-k-mooney: What's the kernel ABI break?
09:06:25 bbezak indeed - OFED 5.4 also introduced that change in renumeration at some level.
09:06:25 sean-k-mooney looking at the delta it does not look like there is much else included but release are pretty cheap
09:06:31 bbezak https://bugzilla.redhat.com/show_bug.cgi?id=1918703
09:06:40 kashyap Ah, a commit is linked - https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1
09:06:42 bbezak kashyap: https://bugzilla.redhat.com/show_bug.cgi?id=1918703
09:06:51 kashyap Thanks
09:07:00 sean-k-mooney kashyap: melonox moved where the reprentor netdevs were advertised in /sys
09:07:26 sean-k-mooney we have fixed os-vif and libvirt to be able to handel both now
09:07:37 sean-k-mooney but we need to actully release os-vif with the fix
09:07:42 kashyap I see
09:08:02 kashyap This seems to be a "forced" ABI break - because of an external vendor
09:08:06 kashyap But yeah, a break nonetheless
09:08:33 sean-k-mooney well yes they chose to do this to enable a new feature in the future
09:09:02 sean-k-mooney we adapted on master a long time ago and the backports where held on review bandwith for some time
09:09:23 sean-k-mooney now that they have actully landed we can do a realase for them
09:09:51 bbezak yesterday ussuri was merged, but I think older ones are still waiting for +2s
09:10:12 bbezak and/or rechecks
09:10:16 sean-k-mooney i think train was reviewed. i should now have stable branch rights so ill take a look
09:10:50 sean-k-mooney ah you are right
09:10:54 sean-k-mooney they are still pending
09:12:31 sean-k-mooney its in the governace repo
09:12:43 bauzas ok, then it's not needed
09:13:02 sean-k-mooney https://github.com/openstack/governance/blob/master/reference/runtimes/xena.rst
09:13:41 sean-k-mooney technically i guess these are minimume but 3.6-3.8 is what is tested
09:14:04 sean-k-mooney technially i think we can mostly run on 3.5
09:14:57 sean-k-mooney bbezak: i have rechecked the train version ill take a look at them again later today
09:15:22 sean-k-mooney bbezak: are you going to porpos a release to the git repo or would you like me to do that
09:17:40 bbezak sean-k-mooney: thx, I will try to propose git release for victoria/ussuri
09:45:02 opendevreview Balazs Gibizer proposed openstack/nova master: DNM: check nova job results with placement transaction fix https://review.opendev.org/c/openstack/nova/+/807558
10:07:12 bbezak ussuri, victoria os-vif release has been proposed, as per docs - https://releases.openstack.org/reference/using.html#using-new-release-command. PTL needs to approve those, please take a look gibi:

Earlier   Later