Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-16
16:17:40 mdbooth kashyap: We won't have been specifying either.
16:18:07 andreykurilin mriedem: hi! as far as you know, novaclient can be used by 2 different way (CLI and as a python-lib). novaclient.client.Client can accept a string with a Token and a proper keystoneauth object will be constructed. This work for python-lib way. I helped to fix this in another patch (see https://review.openstack.org/#/c/419441/) and we decided to add release notes in the separate patch to merge fix asap.
16:18:43 kashyap mdbooth: Yeah, you're right, we aren't specifying anything in Nova:
16:18:44 kashyap """Converts a qcow2 disk to raw."""
16:18:44 kashyap def _disk_qcow2_to_raw(path):
16:18:45 kashyap utils.execute('qemu-img', 'convert', '-f', 'qcow2',
16:18:45 kashyap path_raw = path + '_raw'
16:18:47 kashyap '-O', 'raw', path, path_raw)
16:18:48 cfriesen kashyap: we've been specifying just "-t" for the dest cache to avoid filling the available dirty pages. probably using -T for the source would also make sense but we haven't done that.
16:18:53 andreykurilin mriedem: As for CLI - the token method is still broken.
16:19:01 mriedem andreykurilin: ah ok,
16:19:02 kashyap cfriesen: Are you running the command manually?
16:19:11 mriedem so in the cli fix, i removed the reno because it was causing a merge conflict
16:19:15 cfriesen kashyap: no, we patched nova in that code you just showed
16:19:23 mriedem andreykurilin: so that cli fix would need a separate release note, specific to the CLI and the --os-token option
16:19:32 andreykurilin yup
16:20:36 kashyap cfriesen: I see, then you might want to patch Nova by specifying cache for both source _and_ dest
16:20:52 kashyap And propose the patch upstream, if you have time ;-)
16:21:21 cfriesen kashyap: I'm happy to propose the patch upstream...matt just wanted me to check with the two of you first as to whether it makes sense for the general case
16:22:26 kashyap Let me check with the QEMU guys, too. For the general case
16:22:28 jaypipes gibi, efried: ok, so https://review.openstack.org/#/c/519601/ needs a rebase/fix now that efried's patch that changed the call signature of AllocationCandidates.get_by_filters() was merged.
16:22:35 andreykurilin mriedem: there are 2 ways to fix CLI - "quick and dirty" (as far as I remeber proposed change worked for me) or start cleaning up https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L213 at first since it was designed mostly for HTTPClient imlementation which doesn't exist anymore
16:22:37 jaypipes gibi, efried: who's gonna do that one?
16:22:42 cfriesen mdbooth: in response to what it was affecting...the original problem was that a stress test with a number of heat stacks ended up starting VMs and one of them didn't have any NICs configured due to delays in openstack components on the compute node.
16:22:43 mdbooth cfriesen: As I say, I think it's a can of worms. The behaviour isn't clearly wrong. We're potentially massively penalising some workloads at the expense of others.
16:22:53 mdbooth cfriesen: Not yours, obviously.
16:22:57 jaypipes gibi, efried: need to make sure not to step on each other's feet, which is why I'm asking.
16:23:18 cfriesen mdbooth: heh...yeah, I get it. as long as you have lots of extra ram available for the host then using the cache is going to be faster.
16:23:24 gibi jaypipes, efried: I'm already started fixing https://review.openstack.org/#/c/519601/
16:23:45 jaypipes gibi: gotcha, good. was just making sure you were on it :)
16:23:45 efried gibi Ack, I'll leave it alone.
16:23:56 cfriesen mdbooth: but it doesn't degrade gracefully
16:24:15 mdbooth Nova isn't a real time system :)
16:24:16 gibi jaypipes, efried: cool. I have no other ongoing local change in that chain
16:24:39 kashyap mdbooth: Heh
16:24:50 mdbooth Honestly, I think we'd be better served working out why the delay resulted in a failure to launch.
16:25:00 kashyap cfriesen: I tend to agree with mdbooth's "can of worms" characterization
16:25:20 cfriesen mdbooth: I'm shocked, shocked I tell you. Incidentally, in a totally separate issue we're proposing the option of "fair" locks in oslo to fix a problem in nova.
16:25:52 mdbooth I can't find them right now, but I'm convinced there's a way to set hints on a file descriptor to tell the kernel: I'm only going to read this once.
16:26:14 openstackgerrit Jay Pipes proposed openstack/nova master: XenAPI: create vGPU for instance https://review.openstack.org/516899
16:26:29 cfriesen mdbooth: I'm familiar with madvise() for that purpose...perhaps we should propose changing qemu-img convert to use it.
16:26:32 jaypipes jianghuaw_: ^ just a rebase.
16:26:35 andreykurilin mriedem: I can spend some time to address simple comments, but ot sure that I'm ready now to go refactoring SecretHelper
16:26:38 mdbooth I wonder if qemu-img convert makes use of that, and if it would materially affect the kernel's cache behaviour
16:26:45 mdbooth cfriesen: That's it!
16:26:46 cfriesen mdbooth: but that would still consume dirty pages
16:26:54 mdbooth My memory and google-fu was failing me
16:27:04 mriedem andreykurilin: i'm not even sure what SecretsHelper is, something with a keyring?
16:27:11 mriedem definitely not tested or probably documented
16:27:30 cfriesen mdbooth: and there's the system limits on how many dirty pages are allowed
16:28:23 andreykurilin mriedem: I do not have a lot of info about it as well:) It was introduced long time ago and used for prompting password, caching(?!) or something like that
16:28:43 mriedem andreykurilin: ok i'd say fix the CLI for using --os-token, and we can deal with SecretsHelper later
16:28:58 mriedem https://github.com/openstack/python-novaclient/commit/7c6f538cf6bc86ecfd9cb25f1bf7f14a6652b592
16:29:19 mriedem i wonder if that even works anymore
16:29:24 mriedem harlowja: ^?
16:29:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test alloc_cands with indirectly sharing RPs https://review.openstack.org/519601
16:29:44 kashyap mdbooth: cfriesen There's some interesting discussion w/ the Block Layer folks on this on #qemu at the moment. So the suggestion is that, Nova should be opinionated based on the knowledge we have
16:29:57 andreykurilin I suppose that part of it still work
16:30:39 mriedem andreykurilin: but i'm not sure if the original reason for it is needed anymore
16:30:46 mriedem we don't require the keyring module in novaclient
16:30:52 mriedem so no one probably knows about this,
16:31:03 mriedem i seem to remember the annoying keyring prompts when i started working on nova back in folsom times
16:31:06 gibi jaypipes, efried: I just pushed the updated https://review.openstack.org/519601 I'm signing off for today and I will not be available on Friday-Monday. Happy hacking!
16:31:17 efried gibi Thank you sir.
16:31:31 jaypipes gibi: noooooooooooooooooooooooooooooooooooooo!
16:31:45 kashyap mdbooth: cfriesen: To summarize: the "opinionated" POV for Nova could be, when using 'qemu-img': for source use 'writeback' and for destination use 'unsafe'
16:31:50 jaypipes gibi: j/k enjoy your weekend. we'll take over your patch as needed.
16:32:05 gibi jaypipes: sorry and thank you
16:32:17 mdbooth kashyap: That wouldn't solve cfriesen 's problem though
16:32:19 kashyap cfriesen: (The QCOW2 developer himself says "I think unsafe caching is the best at least for qcow2 targets")
16:32:25 andreykurilin mriedem: I suppose we can try just remove it and, possibly, no one will notice :)
16:32:29 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Stop posting to os-volumes_boot https://review.openstack.org/473833
16:32:39 cfriesen mdbooth: kashyap: aha, found my notes. we pushed sdb to 100% util, and the global number of dirty pags hit /proc/sys/vm/dirty_bytes. This cause writes to sda to be very slow (which is where the openstack stuff is writing).
16:32:42 mdbooth He's not interested in the performance of qemu-img, but its impact on other processes
16:32:57 cfriesen kashyap: looks like newer qemu-img defaults to "unsafe" for convert
16:32:58 kashyap mdbooth: Yeah, I realize that, I should've phrased better
16:33:15 andreykurilin mriedem: ok, I'll revise the context of the patch and return to it
16:33:16 kashyap cfriesen: Indeed, they're the defaults, I just learnt
16:33:27 openstackgerrit Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/520516
16:33:35 mdbooth The question is whether to compromise the performance of qemu-img to limit its system-wide impact.
16:34:41 cfriesen mdbooth: are you aware of a way to limit what percentage of system-wide dirty page limit a given process can use?
16:34:43 mdbooth cfriesen: Oof, that sucks.
16:35:17 mriedem easy novaclient cleanup thing here https://review.openstack.org/#/c/473833/
16:35:19 mdbooth cfriesen: I'm not.
16:36:00 mriedem andreykurilin: i'm guessing the SecretsHelper isn't working,
16:36:13 mriedem because in my devstack, I unset OS_PASSWORD and ran 'nova list' and got a 401 response,
16:36:14 mdbooth cfriesen: I'm going to have to duck out as I have something else urgent to get done. Happy to continue this conversation tomorrow?
16:36:19 mriedem i wasn't prompted for a password
16:36:21 kashyap mdbooth: Are you saying using 'unsafe' for 'qemu-img convert' sucks?
16:36:34 cfriesen mdbooth: no worries, thanks for the chat
16:36:41 kashyap mdbooth: Just trying to ensure I'm not misparsing you. But yeah, I'll let you proceed.
16:36:59 mdbooth kashyap: No, that saturating sdb indirectly affects sda.
16:37:09 kashyap Ah, okay.
16:37:13 cfriesen kashyap: "unsafe" says that it's possible to keep it in memory and not force it to be written out. I suspect the OS will eventually write it out anyways.
16:37:37 openstackgerrit Merged openstack/python-novaclient master: Fix missing metavar in aggregate-update https://review.openstack.org/512485
16:37:43 kashyap cfriesen: Yeah, the guest has its own cache indeed.
16:37:44 cfriesen kashyap: but if you hit the system-wide dirty-page limit, it's going to affect writes for all devices on the system
16:38:27 cfriesen kashyap: remember we're talking about "qemu-img convert" here, there's no guest involved
16:39:11 kashyap cfriesen: Yes, sorry for the mixup. I was talking about the other scenario where why people tend to not use caching for guests.
16:39:30 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/520461

Earlier   Later