Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-16
16:10:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test allocation candidates: multiple aggregates https://review.openstack.org/518633
16:10:45 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test helper: validate provider summaries https://review.openstack.org/518982
16:11:39 kashyap cfriesen: The QEMU folks remind me that, you have to explicitly mention cache mode for both source _and_ destination images.
16:11:44 mdbooth kashyap: cfriesen So, writeback is terrible and we shouldn't be using it for anything, with the possible exception of CI because we don't care about data there and it performs better.
16:11:49 gibi efried, jaypipes: I fixed the comments in https://review.openstack.org/#/c/518633 and in its parent
16:11:57 cfriesen mdbooth: digging in the archives...but from what I remember it filled the cache and then used up all the available dirty pages on the system, slowing down other host "management" processes
16:12:00 mdbooth I suspect, in fact, that we explicitly configure writeback in our CI
16:12:00 efried gibi Thanks!
16:12:55 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:13:06 mriedem andreykurilin: I rebased ^ but am confused about the release note you had in there
16:13:23 mriedem andreykurilin: i removed it because the same release note was already in novaclient per https://review.openstack.org/#/c/425822/
16:13:24 kashyap mdbooth: Hmm, can we make a blanket comment that it is "terrible"? Because the Ceph folks explicitly seem to prefer 'writeback', from my recollection
16:13:31 mdbooth cfriesen: Meh. Tweaking this stuff is opening a can of worms.
16:13:46 mdbooth kashyap: Yep. writeback means something else in ceph :/
16:14:34 kashyap mdbooth: Oh dear, yeah
16:14:36 mdbooth To qemu's ceph driver, writeback means only to inform the ceph server that it can use writeback mode. Data is not cached locally.
16:15:42 kashyap Hmm
16:15:57 cfriesen mdbooth: yeah, that's why mriedem wanted me to talk to you. :) Agreed that it's mostly an issue if you're trying to minimize the amount of RAM used by the host (to maximize how much the guests can get).
16:16:34 mdbooth If it was affecting instance performance it would be different, because we control both of those things.
16:16:58 mdbooth If it's just affecting random other stuff... I'm not convinced we want to go there.
16:17:04 kashyap cfriesen: Can you confirm if were specifying the cache mode for both source & destination disk images separately?
16:17:14 kashyap (When doing 'qemu-img convert', i.e.)
16:17:40 mdbooth kashyap: We won't have been specifying either.
16:18:07 andreykurilin mriedem: hi! as far as you know, novaclient can be used by 2 different way (CLI and as a python-lib). novaclient.client.Client can accept a string with a Token and a proper keystoneauth object will be constructed. This work for python-lib way. I helped to fix this in another patch (see https://review.openstack.org/#/c/419441/) and we decided to add release notes in the separate patch to merge fix asap.
16:18:43 kashyap mdbooth: Yeah, you're right, we aren't specifying anything in Nova:
16:18:44 kashyap """Converts a qcow2 disk to raw."""
16:18:44 kashyap def _disk_qcow2_to_raw(path):
16:18:45 kashyap utils.execute('qemu-img', 'convert', '-f', 'qcow2',
16:18:45 kashyap path_raw = path + '_raw'
16:18:47 kashyap '-O', 'raw', path, path_raw)
16:18:48 cfriesen kashyap: we've been specifying just "-t" for the dest cache to avoid filling the available dirty pages. probably using -T for the source would also make sense but we haven't done that.
16:18:53 andreykurilin mriedem: As for CLI - the token method is still broken.
16:19:01 mriedem andreykurilin: ah ok,
16:19:02 kashyap cfriesen: Are you running the command manually?
16:19:11 mriedem so in the cli fix, i removed the reno because it was causing a merge conflict
16:19:15 cfriesen kashyap: no, we patched nova in that code you just showed
16:19:23 mriedem andreykurilin: so that cli fix would need a separate release note, specific to the CLI and the --os-token option
16:19:32 andreykurilin yup
16:20:36 kashyap cfriesen: I see, then you might want to patch Nova by specifying cache for both source _and_ dest
16:20:52 kashyap And propose the patch upstream, if you have time ;-)
16:21:21 cfriesen kashyap: I'm happy to propose the patch upstream...matt just wanted me to check with the two of you first as to whether it makes sense for the general case
16:22:26 kashyap Let me check with the QEMU guys, too. For the general case
16:22:28 jaypipes gibi, efried: ok, so https://review.openstack.org/#/c/519601/ needs a rebase/fix now that efried's patch that changed the call signature of AllocationCandidates.get_by_filters() was merged.
16:22:35 andreykurilin mriedem: there are 2 ways to fix CLI - "quick and dirty" (as far as I remeber proposed change worked for me) or start cleaning up https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L213 at first since it was designed mostly for HTTPClient imlementation which doesn't exist anymore
16:22:37 jaypipes gibi, efried: who's gonna do that one?
16:22:42 cfriesen mdbooth: in response to what it was affecting...the original problem was that a stress test with a number of heat stacks ended up starting VMs and one of them didn't have any NICs configured due to delays in openstack components on the compute node.
16:22:43 mdbooth cfriesen: As I say, I think it's a can of worms. The behaviour isn't clearly wrong. We're potentially massively penalising some workloads at the expense of others.
16:22:53 mdbooth cfriesen: Not yours, obviously.
16:22:57 jaypipes gibi, efried: need to make sure not to step on each other's feet, which is why I'm asking.
16:23:18 cfriesen mdbooth: heh...yeah, I get it. as long as you have lots of extra ram available for the host then using the cache is going to be faster.
16:23:24 gibi jaypipes, efried: I'm already started fixing https://review.openstack.org/#/c/519601/
16:23:45 jaypipes gibi: gotcha, good. was just making sure you were on it :)
16:23:45 efried gibi Ack, I'll leave it alone.
16:23:56 cfriesen mdbooth: but it doesn't degrade gracefully
16:24:15 mdbooth Nova isn't a real time system :)
16:24:16 gibi jaypipes, efried: cool. I have no other ongoing local change in that chain
16:24:39 kashyap mdbooth: Heh
16:24:50 mdbooth Honestly, I think we'd be better served working out why the delay resulted in a failure to launch.
16:25:00 kashyap cfriesen: I tend to agree with mdbooth's "can of worms" characterization
16:25:20 cfriesen mdbooth: I'm shocked, shocked I tell you. Incidentally, in a totally separate issue we're proposing the option of "fair" locks in oslo to fix a problem in nova.
16:25:52 mdbooth I can't find them right now, but I'm convinced there's a way to set hints on a file descriptor to tell the kernel: I'm only going to read this once.
16:26:14 openstackgerrit Jay Pipes proposed openstack/nova master: XenAPI: create vGPU for instance https://review.openstack.org/516899
16:26:29 cfriesen mdbooth: I'm familiar with madvise() for that purpose...perhaps we should propose changing qemu-img convert to use it.
16:26:32 jaypipes jianghuaw_: ^ just a rebase.
16:26:35 andreykurilin mriedem: I can spend some time to address simple comments, but ot sure that I'm ready now to go refactoring SecretHelper
16:26:38 mdbooth I wonder if qemu-img convert makes use of that, and if it would materially affect the kernel's cache behaviour
16:26:45 mdbooth cfriesen: That's it!
16:26:46 cfriesen mdbooth: but that would still consume dirty pages
16:26:54 mdbooth My memory and google-fu was failing me
16:27:04 mriedem andreykurilin: i'm not even sure what SecretsHelper is, something with a keyring?
16:27:11 mriedem definitely not tested or probably documented
16:27:30 cfriesen mdbooth: and there's the system limits on how many dirty pages are allowed
16:28:23 andreykurilin mriedem: I do not have a lot of info about it as well:) It was introduced long time ago and used for prompting password, caching(?!) or something like that
16:28:43 mriedem andreykurilin: ok i'd say fix the CLI for using --os-token, and we can deal with SecretsHelper later
16:28:58 mriedem https://github.com/openstack/python-novaclient/commit/7c6f538cf6bc86ecfd9cb25f1bf7f14a6652b592
16:29:19 mriedem i wonder if that even works anymore
16:29:24 mriedem harlowja: ^?
16:29:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test alloc_cands with indirectly sharing RPs https://review.openstack.org/519601
16:29:44 kashyap mdbooth: cfriesen There's some interesting discussion w/ the Block Layer folks on this on #qemu at the moment. So the suggestion is that, Nova should be opinionated based on the knowledge we have
16:29:57 andreykurilin I suppose that part of it still work
16:30:39 mriedem andreykurilin: but i'm not sure if the original reason for it is needed anymore
16:30:46 mriedem we don't require the keyring module in novaclient
16:30:52 mriedem so no one probably knows about this,
16:31:03 mriedem i seem to remember the annoying keyring prompts when i started working on nova back in folsom times
16:31:06 gibi jaypipes, efried: I just pushed the updated https://review.openstack.org/519601 I'm signing off for today and I will not be available on Friday-Monday. Happy hacking!
16:31:17 efried gibi Thank you sir.
16:31:31 jaypipes gibi: noooooooooooooooooooooooooooooooooooooo!
16:31:45 kashyap mdbooth: cfriesen: To summarize: the "opinionated" POV for Nova could be, when using 'qemu-img': for source use 'writeback' and for destination use 'unsafe'
16:31:50 jaypipes gibi: j/k enjoy your weekend. we'll take over your patch as needed.
16:32:05 gibi jaypipes: sorry and thank you
16:32:17 mdbooth kashyap: That wouldn't solve cfriesen 's problem though
16:32:19 kashyap cfriesen: (The QCOW2 developer himself says "I think unsafe caching is the best at least for qcow2 targets")
16:32:25 andreykurilin mriedem: I suppose we can try just remove it and, possibly, no one will notice :)
16:32:29 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Stop posting to os-volumes_boot https://review.openstack.org/473833
16:32:39 cfriesen mdbooth: kashyap: aha, found my notes. we pushed sdb to 100% util, and the global number of dirty pags hit /proc/sys/vm/dirty_bytes. This cause writes to sda to be very slow (which is where the openstack stuff is writing).
16:32:42 mdbooth He's not interested in the performance of qemu-img, but its impact on other processes
16:32:57 cfriesen kashyap: looks like newer qemu-img defaults to "unsafe" for convert
16:32:58 kashyap mdbooth: Yeah, I realize that, I should've phrased better
16:33:15 andreykurilin mriedem: ok, I'll revise the context of the patch and return to it
16:33:16 kashyap cfriesen: Indeed, they're the defaults, I just learnt

Earlier   Later