| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-06 | |||
| 23:17:18 | mriedem | we should probably have a "things to consider if you plan to use shared storage" section in the admin docs | |
| 23:17:28 | sean-k-mooney | assuming we dont already have a tempest senario test i think im going to write one and add it to the tempest-full-multinode job | |
| 23:17:57 | mriedem | there are no tempest tests for live migration + volume multiattach that i'm aware of | |
| 23:18:05 | mriedem | there are for resize though | |
| 23:19:17 | sean-k-mooney | mriedem: lyarwood was looking at a posibly race. i dont have the context to had but i was thinking a simple test that boots 2 vms with the same multi attach volume and migrate one of the vms back and forth | |
| 23:21:22 | sean-k-mooney | i think it was related to concurent update of the attachnets | |
| 23:21:40 | sean-k-mooney | mriedem: oh lyarwood filled an upstream bug https://bugs.launchpad.net/nova/+bug/1814245 | |
| 23:21:41 | openstack | Launchpad bug 1814245 in OpenStack Compute (nova) "_disconnect_volume incorrectly called for multiattach volumes during post_live_migration" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 23:24:26 | mriedem | so it disconnects the volume from the other server that wasn't live migrated | |
| 23:24:32 | mriedem | if both servers start on the same source host | |
| 23:24:33 | mriedem | yeah? | |
| 23:24:42 | sean-k-mooney | ya | |
| 23:25:27 | mriedem | i guess that bug says https://review.openstack.org/#/c/551302/ fixes it, | |
| 23:25:31 | mriedem | but ^ needs to be updated | |
| 23:27:23 | mriedem | sean-k-mooney: i don't know that a tempest test would catch this unless that test also ssh's into the guest and tries to verify the volume is still connected on the source host | |
| 23:27:40 | mriedem | and and connected to the live migrated server on the dest host | |
| 23:27:59 | sean-k-mooney | mriedem: yes i planned to ssh in touch a file migrate and check i could read the file on both vms | |
| 23:28:02 | mriedem | this is similar to https://review.openstack.org/#/c/548356/ | |
| 23:28:17 | mriedem | maybe you want to work on cleaning up ^ first | |
| 23:28:22 | mriedem | i'm sure steve is long gone by now | |
| 23:28:56 | mriedem | he was only on loan from oracle until they got multiattach support in queens | |
| 23:29:13 | sean-k-mooney | am i can take a look. i have not worked with tempest in a while so i kind of wanted to use this to get familar with it again | |
| 23:30:55 | sean-k-mooney | if that is isshing into the vm it really should be a senario test not an api test... but that is a different issue | |
| 23:31:56 | sean-k-mooney | the api tests "should" be able to run with the nova fake or libvirt fake driver | |
| 23:42:42 | openstackgerrit | Merged openstack/nova master: cleanup *.pyc files in docs tox envs https://review.openstack.org/635210 | |
| 23:51:01 | mshaikh99_ | I am using OpenStack Queens Release and running Ubuntu 16.04 VM on it, The disk IO seems to be very slow with default cache setting as none for Qemu based virt type | |
| 23:51:18 | mshaikh99_ | # time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB) copied, 296.352 s, 7.1 MB/s real 4m56.559s user 0m0.100s sys 4m19.716s | |
| 23:51:35 | mshaikh99_ | Where is the compute node has the IO as: root@compute1:~# time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 20.724 s, 101 MB/s | |
| 23:52:29 | mshaikh99_ | Based on the openstack forum link, I tried to chagne the default cache from none to writeback as suggested https://ask.openstack.org/en/question/6800/qemu-poor-disk-performance/ but it does not change and after the VM hardreboot I still see it as none | |
| 23:53:02 | mshaikh99_ | http://paste.openstack.org/show/744655/ | |
| 23:53:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change live_migration_wait_for_vif_plug=True by default https://review.openstack.org/635360 | |
| 23:54:48 | mshaikh99_ | which file and which node? | |
| 23:56:04 | sean-k-mooney | mshaikh99_: did you do the hard reboot after restarting nova compute | |
| 23:57:33 | mshaikh99 | Hi | |
| 23:57:44 | mshaikh99 | Is there an update on the question I asked | |
| 23:57:46 | sean-k-mooney | mshaikh99: if the qcow file is not preaccloated that could be part of the problem | |
| 23:58:28 | sean-k-mooney | mshaikh99: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.preallocate_images | |
| 23:59:29 | mshaikh99 | ok | |
| #openstack-nova - 2019-02-07 | |||
| 00:00:23 | sean-k-mooney | by the way did you do the hard reboot after you restart the nova compute service | |
| 00:01:17 | mshaikh99 | I did hard reboot of VM | |
| 00:01:25 | mshaikh99 | after nova compute service is restarted | |
| 00:02:26 | mshaikh99 | Although after hardreboot of VM, I do see time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 42.8637 s, 48.9 MB/s | |
| 00:03:25 | mshaikh99 | so I guess the option for disk_cachemodes=file=writebac is not used any more. I can remove it from /etc/nova/nova-compute.conf [DEFAULT] compute_driver=libvirt.LibvirtDriver disk_cachemodes=file=writebac | |
| 00:03:59 | sean-k-mooney | well it still exists | |
| 00:04:01 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes | |
| 00:04:06 | sean-k-mooney | what release are you using again | |
| 00:05:00 | mshaikh99 | Queens | |
| 00:05:03 | sean-k-mooney | file=directsync would be safter then writeback by the way and may give better performance then writethrough | |
| 00:05:08 | mshaikh99 | So I only need to hard reboot and that would have increased the IO | |
| 00:06:08 | mshaikh99 | or I should keep the disk_cachemodes=file=directsync option in DEFAULT section of nova compute config file? | |
| 00:06:51 | melwitt | we had an old bug around that, but it was fixed in queens, so you should have the fix as long as you're running >= 17.0.0.0b2 https://bugs.launchpad.net/nova/+bug/1727558 | |
| 00:06:52 | openstack | Launchpad bug 1727558 in OpenStack Compute (nova) pike "libvirt driver ignores 'disk_cachemodes' configuration setting" [High,Fix committed] - Assigned to melanie witt (melwitt) | |
| 00:07:24 | mshaikh99 | I don't see that writebac option was picked up by the nova | |
| 00:07:26 | mshaikh99 | libvirt+ 27386 1 16 15:01 ? 00:03:06 /usr/bin/qemu-system-x86_64 -name guest=instance-00000002,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-22-instance-00000002/master-key.aes -machine pc-i440fx-bionic,accel=tcg,usb=off,dump-guest-core=off -cpu EPYC,acpi=on,ss=on,hypervisor=on,erms=on,mpx=on,pcommit=on,clwb=on,pku=on,ospke=on,3dnowext=on,3dnow=on,vme=off,fma=off,avx | |
| 00:07:28 | sean-k-mooney | mshaikh99: disk_cachemode is part of the [libvirt] sechont not [DEFAULT] | |
| 00:08:16 | sean-k-mooney | mshaikh99: i would set [libvirt] disk_cachemodes=file=directsync | |
| 00:08:21 | mshaikh99 | ok let me try it then | |
| 00:09:40 | mshaikh99 | I have changed it and hard reboot my VM | |
| 00:09:58 | sean-k-mooney | and restarted nova-compute agent | |
| 00:11:49 | mshaikh99 | there is no nova-compute agent runnnig | |
| 00:12:20 | sean-k-mooney | the nova compute agent and nova compute service are the same thing | |
| 00:14:16 | mshaikh99 | ok now I see time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 44.5152 s, 47.1 MB/s real 0m44.655s user 0m0.072s sys 0m40.564s | |
| 00:14:30 | mshaikh99 | about 47.1 MB | |
| 00:14:39 | mshaikh99 | and cache=directsync | |
| 00:14:46 | mshaikh99 | for qemu process on compute node | |
| 00:14:53 | mshaikh99 | thanks for the help | |
| 00:20:12 | sean-k-mooney | mshaikh99: setting the cache mode to writeback will imporve it more | |
| 00:20:45 | sean-k-mooney | mshaikh99: but you need to make sure the servers are UPS backed or you could get data currption with a power outage | |
| 00:21:11 | sean-k-mooney | and all the other issues that go with writeback | |
| 00:21:32 | sean-k-mooney | o/ | |
| 00:28:07 | mshaikh99 | is directsync safe | |
| 00:28:17 | mshaikh99 | I mean my server are not UPS backed up | |
| 00:28:33 | sean-k-mooney | yes direct sync is safe | |
| 00:28:39 | mshaikh99 | otherwise I will revert back to none | |
| 00:28:59 | mshaikh99 | Cool | |
| 00:29:05 | mshaikh99 | Thanks Sean | |
| 00:29:25 | sean-k-mooney | cache mode none,writethrough and directsync are considerd eqally safe | |
| 00:29:53 | sean-k-mooney | writeback is less safe and unsafe well its in the name | |
| 00:31:33 | sean-k-mooney | none and directsync tend to have better write performance then writehtrough but writethrough has better read performace | |
| 00:32:15 | mshaikh99 | I will check with my team about the read v/s write use-case | |
| 00:32:23 | mshaikh99 | and will try to setup accordingly | |
| 00:32:41 | mshaikh99 | Can I setup a group of servers for one way and another group for another way | |
| 00:32:52 | mshaikh99 | or it is global parameter? | |
| 00:33:04 | sean-k-mooney | its per compute node | |
| 00:33:13 | sean-k-mooney | so you can use host aggrages to group them | |
| 00:33:20 | mshaikh99 | ok, so I can setup a host aggregate | |
| 00:33:23 | mshaikh99 | ok got it | |
| 00:57:10 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Add migration param to check_can_live_migrate_destination https://review.openstack.org/634605 | |
| 00:57:11 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] New objects to transmit NUMA config from dest to source https://review.openstack.org/634827 | |
| 00:57:11 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Create claims when live migrating https://review.openstack.org/634606 | |
| 00:57:12 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Send dest NUMA info inside LiveMigrateData https://review.openstack.org/634828 | |
| 00:57:13 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: [WIP] Update NUMA XML during live migration https://review.openstack.org/635229 | |
| 01:06:40 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 01:07:00 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | |
| 01:07:17 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | |
| 01:07:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113 | |
| 01:07:48 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974 | |
| 01:16:13 | openstackgerrit | Merged openstack/nova master: Switch tempest-slow to be run on python 3 https://review.openstack.org/634204 | |
| 02:20:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311 | |
| 02:20:24 | openstackgerrit | Merged openstack/nova stable/rocky: Use X-Forwarded-Proto as origin protocol if present https://review.openstack.org/634814 | |