| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-06 | |||
| 12:54:10 | sean-k-mooney | viks__: so you might need to rerun that | |
| 12:54:24 | sean-k-mooney | for fio i know it preallocates and there we are seeign about a 2x delta | |
| 12:54:33 | bauzas | gibi: ok, so you need to create a new series for launchpad ? | |
| 12:54:43 | bauzas | I wasn"t knowing about it | |
| 12:55:39 | bauzas | gibi: could you please add this for https://docs.openstack.org/nova/latest/contributor/ptl-guide.html ? | |
| 12:56:24 | bauzas | ah, nevermind | |
| 12:56:25 | bauzas | https://docs.openstack.org/nova/latest/contributor/ptl-guide.html#immediately-after-rc | |
| 12:58:44 | viks__ | sean-k-mooney: when i rerun sysbench, then also i get the below, which is still way below the one on the host: | |
| 12:58:52 | viks__ | https://www.irccloud.com/pastebin/sf7VLtQi/ | |
| 12:59:58 | sean-k-mooney | viks__: ack but without answering my ohter question i cant really help | |
| 13:00:56 | sean-k-mooney | viks__: the default parmaters that are used will depend on your deployment tool so that is why i am asking what you image_backend set to and cache modes also have you preallocated the image or not | |
| 13:02:40 | viks__ | sean-k-mooney: oh sorry... it's libvirt/qcow, `preallocate_images = none, disk_cachemodes = file=writeback,block=writeback` | |
| 13:05:32 | sean-k-mooney | ok so preallocat_image=none means tehre will be some degradation while the qcow is expanded but that does not account of rthe fio delta since it precallocates teh files in the guest to prevent that form being an issue | |
| 13:06:12 | sean-k-mooney | writeback is generally quite good for perfromance you coudl try settign the cache mode to none if you host suport O_DIRECT that might improve perfromance | |
| 13:06:49 | sean-k-mooney | if you leeve the cachemode unest we will default to none if O_DIRECT is support or use writeback which was our default until recently | |
| 13:09:20 | viks__ | sean-k-mooney: ok.. what is the easy way to check for `O_DIRECT` support in the host? | |
| 13:10:50 | sean-k-mooney | well you were using --direct=1 in your fio test so it should support it | |
| 13:11:06 | sean-k-mooney | O_DIRECT is the file mapping mode used for direct io | |
| 13:11:16 | viks__ | ok... | |
| 13:11:59 | sean-k-mooney | first thing i would try is setting the cachemode for file to none | |
| 13:12:07 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553 | |
| 13:12:10 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Fix allocation show / unset with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556 | |
| 13:12:14 | sean-k-mooney | then hard reboot the vm and see if that improve the performance | |
| 13:12:57 | viks__ | sean-k-mooney: ok thanks... will try the same | |
| 13:14:29 | sean-k-mooney | if that has no effect i would try setting preallocate_images=space and then try changing images_backend to raw. you could also in parallel create a second vm with hw_disk_bus=virtio-scsi to compare side by side to virtio-blk | |
| 13:15:19 | sean-k-mooney | viks__: for really write heavy worklaods lvm used to slightly outperform raw but that requried extra setup and it is less well maintained | |
| 13:15:40 | sean-k-mooney | so if the other tuning dont help in your case that is what i woudl try last | |
| 13:16:46 | sean-k-mooney | viks__: outside of openstack there are some host level sysctl tunings you can do to improve guest io perfromance like changing the io schduler or dirty data writeback tresholds | |
| 13:18:31 | sean-k-mooney | tuned can be used as a erfernce for some of the commnly tuned settings | |
| 13:18:33 | sean-k-mooney | https://github.com/redhat-performance/tuned/blob/master/profiles/throughput-performance/tuned.conf#L24-L59 | |
| 13:19:52 | viks__ | sean-k-mooney: thanks for all the inputs.. will do some testing based on these .... thanks again... | |
| 13:22:42 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: check nova-next with osc-placement fix https://review.opendev.org/c/openstack/nova/+/807558 | |
| 13:25:34 | sean-k-mooney | viks__: if you are testing virtio-scsi i the image options are hw_disk_bus=scsi hw_scsi_model=virtio-scsi by they way | |
| 13:25:51 | sean-k-mooney | i previously msityped it as hw_disk_bus=virtio-scsi | |
| 13:26:11 | viks__ | sean-k-mooney: ok... | |
| 14:06:16 | bauzas | huzzah, can see my review priorities : https://review.opendev.org/q/label:Review-Priority%253E%253D1%252Csbauza | |
| 14:20:19 | opendevreview | Merged openstack/osc-placement master: Repro allocation show bug with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807553 | |
| 14:20:20 | opendevreview | Merged openstack/osc-placement master: Fix allocation show / unset with empty allocation https://review.opendev.org/c/openstack/osc-placement/+/807556 | |
| 14:41:23 | gibi | lyarwood, bauzas: ^^ thanks for the quick approve. I will ask for a release freeze exception shortly | |
| 14:53:41 | gibi | RFE requested: http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024686.html | |
| 14:55:35 | slaweq | hi stable nova cores, can somebody check https://review.opendev.org/c/openstack/nova/+/791421 /. | |
| 14:55:38 | slaweq | ? | |
| 14:55:44 | slaweq | thx in advance for help :) | |
| 15:28:58 | gibi | do we have USA out today due to Labor Day? | |
| 15:31:33 | kashyap | gibi: Yeah | |
| 15:31:40 | kashyap | Also Canada, IIRC | |
| 15:31:46 | gibi | OK, then I don't wait for them today :) | |
| 15:57:54 | bauzas | gibi: sorry was on a meeting | |
| 15:58:59 | bauzas | gibi: yup, saw your FFE request but as kashyap said and you guessed, a whole portion of the world located between Pacific and Atlantic oceans and above a certain latitude is currently shutdown for the day | |
| 15:59:20 | gibi | bauzas: no worries the release now queued | |
| 15:59:35 | bauzas | said a French guy working | |
| 15:59:59 | gibi | bauzas: I looked at melwitt's repro patch https://review.opendev.org/c/openstack/placement/+/807493 and I left some ideas but no real solutions yet | |
| 16:01:51 | bauzas | gibi: honestly we face the limits of the single-commit design we have with Placement | |
| 16:03:26 | gibi | bauzas: this is now more about writing a sane a repro test that excersise transaction isolation with mysql (as sqlite does not have it) | |
| 16:03:57 | gibi | I think the fix melwitt's proposed on top is sane but the repro test is racy somehow actually forks processes | |
| 16:04:02 | gibi | whihc is scarry | |
| 16:04:28 | bauzas | gibi: yup, I saw the tox modification and I understood the reasoning | |
| 16:12:04 | melwitt | I will try to find a different way to do it. it's inherently problematic to try and repro this because once one path starts a transaction, trying to do something in the middle of it (to cause the race state) poses a record locking problem. at one point I was trying to fake it by just returning bogus things to make it think it hit a generation conflict (without doing any database write), which worked to make it retry but that wasn't | |
| 16:12:04 | melwitt | showing the effect of the consistent read problem inside the transaction | |
| 16:21:40 | gibi | melwitt: yes, exactly, that was the dead end I went down this afternoon :/ | |
| 16:22:16 | gibi | melwitt: I think at some point we can accept that we cannot reliably test this in func env and simply land the fix and monitor the tempest jobs to see if it is resolved the race or not | |
| 16:22:54 | melwitt | gibi: I went down the dead end three days in a row 😑 | |
| 16:23:04 | melwitt | gibi: yeah. I was thinking that too. I kept thinking there has to be a way but if there is I'm not clever enough to find it haha | |
| 16:24:06 | gibi | melwitt: but enjoy your day off, we can continue this tomorrow | |
| 16:24:53 | melwitt | kk, thanks | |
| #openstack-nova - 2021-09-07 | |||
| 01:49:39 | opendevreview | Jorhson Deng proposed openstack/nova master: recheck the attachment_id after the reschedule successful https://review.opendev.org/c/openstack/nova/+/796209 | |
| 03:15:06 | opendevreview | Ade Lee proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | |
| 07:57:50 | opendevreview | Wenping Song proposed openstack/placement master: Dropping lower constraints testing https://review.opendev.org/c/openstack/placement/+/787863 | |
| 08:45:57 | bbezak | Hi - Can we make a release os-vif for Victoria? 2.2.1? this bug is quite critical - https://bugs.launchpad.net/os-vif/+bug/1892132. And it is not part of 2.2.0 release | |
| 09:02:02 | sean-k-mooney | its not actully a bug it was a kernel abi break | |
| 09:02:31 | sean-k-mooney | but yes we can proably do a release we just need too propose it to the release repo | |
| 09:03:00 | sean-k-mooney | bbezak: the release managmen it automated via a git repo | |
| 09:04:04 | sean-k-mooney | so we just need to update this with the corerct sha https://github.com/openstack/releases/blob/master/deliverables/victoria/os-vif.yaml | |
| 09:04:31 | sean-k-mooney | there is a script in the release repo for doing that or you can do it by hand | |
| 09:05:59 | sean-k-mooney | https://github.com/openstack/os-vif/compare/2.2.0...stable/victoria | |
| 09:06:16 | kashyap | sean-k-mooney: What's the kernel ABI break? | |
| 09:06:25 | sean-k-mooney | looking at the delta it does not look like there is much else included but release are pretty cheap | |
| 09:06:25 | bbezak | indeed - OFED 5.4 also introduced that change in renumeration at some level. | |
| 09:06:31 | bbezak | https://bugzilla.redhat.com/show_bug.cgi?id=1918703 | |
| 09:06:40 | kashyap | Ah, a commit is linked - https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1 | |
| 09:06:42 | bbezak | kashyap: https://bugzilla.redhat.com/show_bug.cgi?id=1918703 | |
| 09:06:51 | kashyap | Thanks | |
| 09:07:00 | sean-k-mooney | kashyap: melonox moved where the reprentor netdevs were advertised in /sys | |
| 09:07:26 | sean-k-mooney | we have fixed os-vif and libvirt to be able to handel both now | |
| 09:07:37 | sean-k-mooney | but we need to actully release os-vif with the fix | |
| 09:07:42 | kashyap | I see | |
| 09:08:02 | kashyap | This seems to be a "forced" ABI break - because of an external vendor | |
| 09:08:06 | kashyap | But yeah, a break nonetheless | |
| 09:08:33 | sean-k-mooney | well yes they chose to do this to enable a new feature in the future | |
| 09:09:02 | sean-k-mooney | we adapted on master a long time ago and the backports where held on review bandwith for some time | |
| 09:09:23 | sean-k-mooney | now that they have actully landed we can do a realase for them | |
| 09:09:51 | bbezak | yesterday ussuri was merged, but I think older ones are still waiting for +2s | |
| 09:10:12 | bbezak | and/or rechecks | |
| 09:10:16 | sean-k-mooney | i think train was reviewed. i should now have stable branch rights so ill take a look | |
| 09:10:50 | sean-k-mooney | ah you are right | |
| 09:10:54 | sean-k-mooney | they are still pending | |
| 09:12:31 | sean-k-mooney | its in the governace repo | |
| 09:12:43 | bauzas | ok, then it's not needed | |
| 09:13:02 | sean-k-mooney | https://github.com/openstack/governance/blob/master/reference/runtimes/xena.rst | |
| 09:13:41 | sean-k-mooney | technically i guess these are minimume but 3.6-3.8 is what is tested | |