Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-06
23:27:40 mriedem and and connected to the live migrated server on the dest host
23:27:59 sean-k-mooney mriedem: yes i planned to ssh in touch a file migrate and check i could read the file on both vms
23:28:02 mriedem this is similar to https://review.openstack.org/#/c/548356/
23:28:17 mriedem maybe you want to work on cleaning up ^ first
23:28:22 mriedem i'm sure steve is long gone by now
23:28:56 mriedem he was only on loan from oracle until they got multiattach support in queens
23:29:13 sean-k-mooney am i can take a look. i have not worked with tempest in a while so i kind of wanted to use this to get familar with it again
23:30:55 sean-k-mooney if that is isshing into the vm it really should be a senario test not an api test... but that is a different issue
23:31:56 sean-k-mooney the api tests "should" be able to run with the nova fake or libvirt fake driver
23:42:42 openstackgerrit Merged openstack/nova master: cleanup *.pyc files in docs tox envs https://review.openstack.org/635210
23:51:01 mshaikh99_ I am using OpenStack Queens Release and running Ubuntu 16.04 VM on it, The disk IO seems to be very slow with default cache setting as none for Qemu based virt type
23:51:18 mshaikh99_ # time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB) copied, 296.352 s, 7.1 MB/s real 4m56.559s user 0m0.100s sys 4m19.716s
23:51:35 mshaikh99_ Where is the compute node has the IO as: root@compute1:~# time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 20.724 s, 101 MB/s
23:52:29 mshaikh99_ Based on the openstack forum link, I tried to chagne the default cache from none to writeback as suggested https://ask.openstack.org/en/question/6800/qemu-poor-disk-performance/ but it does not change and after the VM hardreboot I still see it as none
23:53:02 mshaikh99_ http://paste.openstack.org/show/744655/
23:53:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Change live_migration_wait_for_vif_plug=True by default https://review.openstack.org/635360
23:54:48 mshaikh99_ which file and which node?
23:56:04 sean-k-mooney mshaikh99_: did you do the hard reboot after restarting nova compute
23:57:33 mshaikh99 Hi
23:57:44 mshaikh99 Is there an update on the question I asked
23:57:46 sean-k-mooney mshaikh99: if the qcow file is not preaccloated that could be part of the problem
23:58:28 sean-k-mooney mshaikh99: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.preallocate_images
23:59:29 mshaikh99 ok
#openstack-nova - 2019-02-07
00:00:23 sean-k-mooney by the way did you do the hard reboot after you restart the nova compute service
00:01:17 mshaikh99 I did hard reboot of VM
00:01:25 mshaikh99 after nova compute service is restarted
00:02:26 mshaikh99 Although after hardreboot of VM, I do see time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 42.8637 s, 48.9 MB/s
00:03:25 mshaikh99 so I guess the option for disk_cachemodes=file=writebac is not used any more. I can remove it from /etc/nova/nova-compute.conf [DEFAULT] compute_driver=libvirt.LibvirtDriver disk_cachemodes=file=writebac
00:03:59 sean-k-mooney well it still exists
00:04:01 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes
00:04:06 sean-k-mooney what release are you using again
00:05:00 mshaikh99 Queens
00:05:03 sean-k-mooney file=directsync would be safter then writeback by the way and may give better performance then writethrough
00:05:08 mshaikh99 So I only need to hard reboot and that would have increased the IO
00:06:08 mshaikh99 or I should keep the disk_cachemodes=file=directsync option in DEFAULT section of nova compute config file?
00:06:51 melwitt we had an old bug around that, but it was fixed in queens, so you should have the fix as long as you're running >= 17.0.0.0b2 https://bugs.launchpad.net/nova/+bug/1727558
00:06:52 openstack Launchpad bug 1727558 in OpenStack Compute (nova) pike "libvirt driver ignores 'disk_cachemodes' configuration setting" [High,Fix committed] - Assigned to melanie witt (melwitt)
00:07:24 mshaikh99 I don't see that writebac option was picked up by the nova
00:07:26 mshaikh99 libvirt+ 27386 1 16 15:01 ? 00:03:06 /usr/bin/qemu-system-x86_64 -name guest=instance-00000002,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-22-instance-00000002/master-key.aes -machine pc-i440fx-bionic,accel=tcg,usb=off,dump-guest-core=off -cpu EPYC,acpi=on,ss=on,hypervisor=on,erms=on,mpx=on,pcommit=on,clwb=on,pku=on,ospke=on,3dnowext=on,3dnow=on,vme=off,fma=off,avx
00:07:28 sean-k-mooney mshaikh99: disk_cachemode is part of the [libvirt] sechont not [DEFAULT]
00:08:16 sean-k-mooney mshaikh99: i would set [libvirt] disk_cachemodes=file=directsync
00:08:21 mshaikh99 ok let me try it then
00:09:40 mshaikh99 I have changed it and hard reboot my VM
00:09:58 sean-k-mooney and restarted nova-compute agent
00:11:49 mshaikh99 there is no nova-compute agent runnnig
00:12:20 sean-k-mooney the nova compute agent and nova compute service are the same thing
00:14:16 mshaikh99 ok now I see time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 44.5152 s, 47.1 MB/s real 0m44.655s user 0m0.072s sys 0m40.564s
00:14:30 mshaikh99 about 47.1 MB
00:14:39 mshaikh99 and cache=directsync
00:14:46 mshaikh99 for qemu process on compute node
00:14:53 mshaikh99 thanks for the help
00:20:12 sean-k-mooney mshaikh99: setting the cache mode to writeback will imporve it more
00:20:45 sean-k-mooney mshaikh99: but you need to make sure the servers are UPS backed or you could get data currption with a power outage
00:21:11 sean-k-mooney and all the other issues that go with writeback
00:21:32 sean-k-mooney o/
00:28:07 mshaikh99 is directsync safe
00:28:17 mshaikh99 I mean my server are not UPS backed up
00:28:33 sean-k-mooney yes direct sync is safe
00:28:39 mshaikh99 otherwise I will revert back to none
00:28:59 mshaikh99 Cool
00:29:05 mshaikh99 Thanks Sean
00:29:25 sean-k-mooney cache mode none,writethrough and directsync are considerd eqally safe
00:29:53 sean-k-mooney writeback is less safe and unsafe well its in the name
00:31:33 sean-k-mooney none and directsync tend to have better write performance then writehtrough but writethrough has better read performace
00:32:15 mshaikh99 I will check with my team about the read v/s write use-case
00:32:23 mshaikh99 and will try to setup accordingly
00:32:41 mshaikh99 Can I setup a group of servers for one way and another group for another way
00:32:52 mshaikh99 or it is global parameter?
00:33:04 sean-k-mooney its per compute node
00:33:13 sean-k-mooney so you can use host aggrages to group them
00:33:20 mshaikh99 ok, so I can setup a host aggregate
00:33:23 mshaikh99 ok got it
00:57:10 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] Add migration param to check_can_live_migrate_destination https://review.openstack.org/634605
00:57:11 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] Create claims when live migrating https://review.openstack.org/634606
00:57:11 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] New objects to transmit NUMA config from dest to source https://review.openstack.org/634827
00:57:12 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] Send dest NUMA info inside LiveMigrateData https://review.openstack.org/634828
00:57:13 openstackgerrit Artom Lifshitz proposed openstack/nova master: [WIP] Update NUMA XML during live migration https://review.openstack.org/635229
01:06:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104
01:07:00 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106
01:07:17 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110
01:07:32 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113
01:07:48 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974
01:16:13 openstackgerrit Merged openstack/nova master: Switch tempest-slow to be run on python 3 https://review.openstack.org/634204
02:20:20 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311
02:20:24 openstackgerrit Merged openstack/nova stable/rocky: Use X-Forwarded-Proto as origin protocol if present https://review.openstack.org/634814
02:20:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581
02:20:52 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017
02:21:09 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018
02:21:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019
03:52:32 openstackgerrit Sam Morrison proposed openstack/nova-specs master: add spec "flavor-classes" https://review.openstack.org/635408
03:55:36 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020
03:55:46 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027
03:56:00 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031
03:56:16 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299
03:56:33 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344
05:39:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673
05:39:49 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676
05:40:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689
05:40:20 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709
06:01:11 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Remove deprecated novaclient.v2.contrib modules https://review.openstack.org/635430

Earlier   Later