Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-16
15:57:19 cfriesen _disk_qcow2_to_raw() in nova/virt/libvirt/driver.py
15:57:30 cfriesen and vice-versa
15:57:57 mdbooth kashyap: Yes, cfriesen is essentially complaining that writeback isn't what he wants.
15:58:44 mdbooth cfriesen: What impact did this have on the machine?
15:59:26 kashyap mdboth: I see. In that case, I know cfriesen sometimes posts on 'qemu-devel'; so Chris if you have the perf data then post there and Cc 'qemu-block' list...
15:59:31 efried jaypipes Got local changes to the n-r-p series? I'm doing a restack, and can hit Tetsuro's comments on https://review.openstack.org/#/c/384807/ while I'm at it.
15:59:48 kashyap But yeah, I'll let him to describe the issue here
15:59:49 jaypipes efried: nope.
15:59:52 mdbooth I'm wondering if we give instances cache=none by default
15:59:55 efried jaypipes ight
15:59:59 cfriesen kashyap: mdbooth: digging up the details
16:00:01 mdbooth I thought we did?
16:00:14 kashyap mdbooth: Easy to check, let's examine a Gate job logs...
16:00:41 kashyap I thought we let the default be 'writeback' as QEMU gives us; because there's a reason they settled on it as the default
16:02:10 mdbooth kashyap: Are you sure writeback is the default?
16:02:13 mdbooth writeback is terrible.
16:02:17 kashyap mdbooth: Yes, I just checked
16:02:31 kashyap mdbooth: Just fire up `man qemu-system-x86` and look for 'writeback'
16:03:01 kashyap mdbooth: And...Here's the evidence from the gate jobs:
16:03:04 kashyap http://logs.openstack.org/14/407514/9/check/legacy-grenade-dsvm-neutron-multinode-live-migration/8d355c4/logs/libvirt/qemu/instance-00000019.txt.gz
16:05:10 cfriesen mdbooth: kashyap: from our bug notes it looks like we were actually interfering with other host processes, not the guests. (my bad, I misremembered) So we turned off the cache and limited it to two disk-intensive operations (glance downloads, image conversions, etc.) at a time.
16:06:14 kashyap cfriesen: BTW, what are your guests?
16:06:37 kashyap Are they something super old?
16:06:58 cfriesen kashyap: libvirt/kvm...no idea what the software in the guest was, they were customer tests
16:07:07 bauzas mriedem: sure, will look at the change asap
16:08:24 mdbooth cfriesen: Do you recall what the other host processes were?
16:09:01 kashyap cfriesen: Hmm. As a reminder to self: 'writeback' caches the writes in the host page cache, and 'none' avoids it (but it also doesn't take advantage of the host RAM, especially if you have lots of it)
16:10:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test allocation candidates: multiple aggregates https://review.openstack.org/518633
16:10:45 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test helper: validate provider summaries https://review.openstack.org/518982
16:11:39 kashyap cfriesen: The QEMU folks remind me that, you have to explicitly mention cache mode for both source _and_ destination images.
16:11:44 mdbooth kashyap: cfriesen So, writeback is terrible and we shouldn't be using it for anything, with the possible exception of CI because we don't care about data there and it performs better.
16:11:49 gibi efried, jaypipes: I fixed the comments in https://review.openstack.org/#/c/518633 and in its parent
16:11:57 cfriesen mdbooth: digging in the archives...but from what I remember it filled the cache and then used up all the available dirty pages on the system, slowing down other host "management" processes
16:12:00 mdbooth I suspect, in fact, that we explicitly configure writeback in our CI
16:12:00 efried gibi Thanks!
16:12:55 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:13:06 mriedem andreykurilin: I rebased ^ but am confused about the release note you had in there
16:13:23 mriedem andreykurilin: i removed it because the same release note was already in novaclient per https://review.openstack.org/#/c/425822/
16:13:24 kashyap mdbooth: Hmm, can we make a blanket comment that it is "terrible"? Because the Ceph folks explicitly seem to prefer 'writeback', from my recollection
16:13:31 mdbooth cfriesen: Meh. Tweaking this stuff is opening a can of worms.
16:13:46 mdbooth kashyap: Yep. writeback means something else in ceph :/
16:14:34 kashyap mdbooth: Oh dear, yeah
16:14:36 mdbooth To qemu's ceph driver, writeback means only to inform the ceph server that it can use writeback mode. Data is not cached locally.
16:15:42 kashyap Hmm
16:15:57 cfriesen mdbooth: yeah, that's why mriedem wanted me to talk to you. :) Agreed that it's mostly an issue if you're trying to minimize the amount of RAM used by the host (to maximize how much the guests can get).
16:16:34 mdbooth If it was affecting instance performance it would be different, because we control both of those things.
16:16:58 mdbooth If it's just affecting random other stuff... I'm not convinced we want to go there.
16:17:04 kashyap cfriesen: Can you confirm if were specifying the cache mode for both source & destination disk images separately?
16:17:14 kashyap (When doing 'qemu-img convert', i.e.)
16:17:40 mdbooth kashyap: We won't have been specifying either.
16:18:07 andreykurilin mriedem: hi! as far as you know, novaclient can be used by 2 different way (CLI and as a python-lib). novaclient.client.Client can accept a string with a Token and a proper keystoneauth object will be constructed. This work for python-lib way. I helped to fix this in another patch (see https://review.openstack.org/#/c/419441/) and we decided to add release notes in the separate patch to merge fix asap.
16:18:43 kashyap mdbooth: Yeah, you're right, we aren't specifying anything in Nova:
16:18:44 kashyap """Converts a qcow2 disk to raw."""
16:18:44 kashyap def _disk_qcow2_to_raw(path):
16:18:45 kashyap utils.execute('qemu-img', 'convert', '-f', 'qcow2',
16:18:45 kashyap path_raw = path + '_raw'
16:18:47 kashyap '-O', 'raw', path, path_raw)
16:18:48 cfriesen kashyap: we've been specifying just "-t" for the dest cache to avoid filling the available dirty pages. probably using -T for the source would also make sense but we haven't done that.
16:18:53 andreykurilin mriedem: As for CLI - the token method is still broken.
16:19:01 mriedem andreykurilin: ah ok,
16:19:02 kashyap cfriesen: Are you running the command manually?
16:19:11 mriedem so in the cli fix, i removed the reno because it was causing a merge conflict
16:19:15 cfriesen kashyap: no, we patched nova in that code you just showed
16:19:23 mriedem andreykurilin: so that cli fix would need a separate release note, specific to the CLI and the --os-token option
16:19:32 andreykurilin yup
16:20:36 kashyap cfriesen: I see, then you might want to patch Nova by specifying cache for both source _and_ dest
16:20:52 kashyap And propose the patch upstream, if you have time ;-)
16:21:21 cfriesen kashyap: I'm happy to propose the patch upstream...matt just wanted me to check with the two of you first as to whether it makes sense for the general case
16:22:26 kashyap Let me check with the QEMU guys, too. For the general case
16:22:28 jaypipes gibi, efried: ok, so https://review.openstack.org/#/c/519601/ needs a rebase/fix now that efried's patch that changed the call signature of AllocationCandidates.get_by_filters() was merged.
16:22:35 andreykurilin mriedem: there are 2 ways to fix CLI - "quick and dirty" (as far as I remeber proposed change worked for me) or start cleaning up https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L213 at first since it was designed mostly for HTTPClient imlementation which doesn't exist anymore
16:22:37 jaypipes gibi, efried: who's gonna do that one?
16:22:42 cfriesen mdbooth: in response to what it was affecting...the original problem was that a stress test with a number of heat stacks ended up starting VMs and one of them didn't have any NICs configured due to delays in openstack components on the compute node.
16:22:43 mdbooth cfriesen: As I say, I think it's a can of worms. The behaviour isn't clearly wrong. We're potentially massively penalising some workloads at the expense of others.
16:22:53 mdbooth cfriesen: Not yours, obviously.
16:22:57 jaypipes gibi, efried: need to make sure not to step on each other's feet, which is why I'm asking.
16:23:18 cfriesen mdbooth: heh...yeah, I get it. as long as you have lots of extra ram available for the host then using the cache is going to be faster.
16:23:24 gibi jaypipes, efried: I'm already started fixing https://review.openstack.org/#/c/519601/
16:23:45 jaypipes gibi: gotcha, good. was just making sure you were on it :)
16:23:45 efried gibi Ack, I'll leave it alone.
16:23:56 cfriesen mdbooth: but it doesn't degrade gracefully
16:24:15 mdbooth Nova isn't a real time system :)
16:24:16 gibi jaypipes, efried: cool. I have no other ongoing local change in that chain
16:24:39 kashyap mdbooth: Heh
16:24:50 mdbooth Honestly, I think we'd be better served working out why the delay resulted in a failure to launch.
16:25:00 kashyap cfriesen: I tend to agree with mdbooth's "can of worms" characterization
16:25:20 cfriesen mdbooth: I'm shocked, shocked I tell you. Incidentally, in a totally separate issue we're proposing the option of "fair" locks in oslo to fix a problem in nova.
16:25:52 mdbooth I can't find them right now, but I'm convinced there's a way to set hints on a file descriptor to tell the kernel: I'm only going to read this once.
16:26:14 openstackgerrit Jay Pipes proposed openstack/nova master: XenAPI: create vGPU for instance https://review.openstack.org/516899
16:26:29 cfriesen mdbooth: I'm familiar with madvise() for that purpose...perhaps we should propose changing qemu-img convert to use it.
16:26:32 jaypipes jianghuaw_: ^ just a rebase.
16:26:35 andreykurilin mriedem: I can spend some time to address simple comments, but ot sure that I'm ready now to go refactoring SecretHelper
16:26:38 mdbooth I wonder if qemu-img convert makes use of that, and if it would materially affect the kernel's cache behaviour
16:26:45 mdbooth cfriesen: That's it!
16:26:46 cfriesen mdbooth: but that would still consume dirty pages
16:26:54 mdbooth My memory and google-fu was failing me
16:27:04 mriedem andreykurilin: i'm not even sure what SecretsHelper is, something with a keyring?
16:27:11 mriedem definitely not tested or probably documented
16:27:30 cfriesen mdbooth: and there's the system limits on how many dirty pages are allowed
16:28:23 andreykurilin mriedem: I do not have a lot of info about it as well:) It was introduced long time ago and used for prompting password, caching(?!) or something like that

Earlier   Later