Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-16
16:24:50 mdbooth Honestly, I think we'd be better served working out why the delay resulted in a failure to launch.
16:25:00 kashyap cfriesen: I tend to agree with mdbooth's "can of worms" characterization
16:25:20 cfriesen mdbooth: I'm shocked, shocked I tell you. Incidentally, in a totally separate issue we're proposing the option of "fair" locks in oslo to fix a problem in nova.
16:25:52 mdbooth I can't find them right now, but I'm convinced there's a way to set hints on a file descriptor to tell the kernel: I'm only going to read this once.
16:26:14 openstackgerrit Jay Pipes proposed openstack/nova master: XenAPI: create vGPU for instance https://review.openstack.org/516899
16:26:29 cfriesen mdbooth: I'm familiar with madvise() for that purpose...perhaps we should propose changing qemu-img convert to use it.
16:26:32 jaypipes jianghuaw_: ^ just a rebase.
16:26:35 andreykurilin mriedem: I can spend some time to address simple comments, but ot sure that I'm ready now to go refactoring SecretHelper
16:26:38 mdbooth I wonder if qemu-img convert makes use of that, and if it would materially affect the kernel's cache behaviour
16:26:45 mdbooth cfriesen: That's it!
16:26:46 cfriesen mdbooth: but that would still consume dirty pages
16:26:54 mdbooth My memory and google-fu was failing me
16:27:04 mriedem andreykurilin: i'm not even sure what SecretsHelper is, something with a keyring?
16:27:11 mriedem definitely not tested or probably documented
16:27:30 cfriesen mdbooth: and there's the system limits on how many dirty pages are allowed
16:28:23 andreykurilin mriedem: I do not have a lot of info about it as well:) It was introduced long time ago and used for prompting password, caching(?!) or something like that
16:28:43 mriedem andreykurilin: ok i'd say fix the CLI for using --os-token, and we can deal with SecretsHelper later
16:28:58 mriedem https://github.com/openstack/python-novaclient/commit/7c6f538cf6bc86ecfd9cb25f1bf7f14a6652b592
16:29:19 mriedem i wonder if that even works anymore
16:29:24 mriedem harlowja: ^?
16:29:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test alloc_cands with indirectly sharing RPs https://review.openstack.org/519601
16:29:44 kashyap mdbooth: cfriesen There's some interesting discussion w/ the Block Layer folks on this on #qemu at the moment. So the suggestion is that, Nova should be opinionated based on the knowledge we have
16:29:57 andreykurilin I suppose that part of it still work
16:30:39 mriedem andreykurilin: but i'm not sure if the original reason for it is needed anymore
16:30:46 mriedem we don't require the keyring module in novaclient
16:30:52 mriedem so no one probably knows about this,
16:31:03 mriedem i seem to remember the annoying keyring prompts when i started working on nova back in folsom times
16:31:06 gibi jaypipes, efried: I just pushed the updated https://review.openstack.org/519601 I'm signing off for today and I will not be available on Friday-Monday. Happy hacking!
16:31:17 efried gibi Thank you sir.
16:31:31 jaypipes gibi: noooooooooooooooooooooooooooooooooooooo!
16:31:45 kashyap mdbooth: cfriesen: To summarize: the "opinionated" POV for Nova could be, when using 'qemu-img': for source use 'writeback' and for destination use 'unsafe'
16:31:50 jaypipes gibi: j/k enjoy your weekend. we'll take over your patch as needed.
16:32:05 gibi jaypipes: sorry and thank you
16:32:17 mdbooth kashyap: That wouldn't solve cfriesen 's problem though
16:32:19 kashyap cfriesen: (The QCOW2 developer himself says "I think unsafe caching is the best at least for qcow2 targets")
16:32:25 andreykurilin mriedem: I suppose we can try just remove it and, possibly, no one will notice :)
16:32:29 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Stop posting to os-volumes_boot https://review.openstack.org/473833
16:32:39 cfriesen mdbooth: kashyap: aha, found my notes. we pushed sdb to 100% util, and the global number of dirty pags hit /proc/sys/vm/dirty_bytes. This cause writes to sda to be very slow (which is where the openstack stuff is writing).
16:32:42 mdbooth He's not interested in the performance of qemu-img, but its impact on other processes
16:32:57 cfriesen kashyap: looks like newer qemu-img defaults to "unsafe" for convert
16:32:58 kashyap mdbooth: Yeah, I realize that, I should've phrased better
16:33:15 andreykurilin mriedem: ok, I'll revise the context of the patch and return to it
16:33:16 kashyap cfriesen: Indeed, they're the defaults, I just learnt
16:33:27 openstackgerrit Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/520516
16:33:35 mdbooth The question is whether to compromise the performance of qemu-img to limit its system-wide impact.
16:34:41 cfriesen mdbooth: are you aware of a way to limit what percentage of system-wide dirty page limit a given process can use?
16:34:43 mdbooth cfriesen: Oof, that sucks.
16:35:17 mriedem easy novaclient cleanup thing here https://review.openstack.org/#/c/473833/
16:35:19 mdbooth cfriesen: I'm not.
16:36:00 mriedem andreykurilin: i'm guessing the SecretsHelper isn't working,
16:36:13 mriedem because in my devstack, I unset OS_PASSWORD and ran 'nova list' and got a 401 response,
16:36:14 mdbooth cfriesen: I'm going to have to duck out as I have something else urgent to get done. Happy to continue this conversation tomorrow?
16:36:19 mriedem i wasn't prompted for a password
16:36:21 kashyap mdbooth: Are you saying using 'unsafe' for 'qemu-img convert' sucks?
16:36:34 cfriesen mdbooth: no worries, thanks for the chat
16:36:41 kashyap mdbooth: Just trying to ensure I'm not misparsing you. But yeah, I'll let you proceed.
16:36:59 mdbooth kashyap: No, that saturating sdb indirectly affects sda.
16:37:09 kashyap Ah, okay.
16:37:13 cfriesen kashyap: "unsafe" says that it's possible to keep it in memory and not force it to be written out. I suspect the OS will eventually write it out anyways.
16:37:37 openstackgerrit Merged openstack/python-novaclient master: Fix missing metavar in aggregate-update https://review.openstack.org/512485
16:37:43 kashyap cfriesen: Yeah, the guest has its own cache indeed.
16:37:44 cfriesen kashyap: but if you hit the system-wide dirty-page limit, it's going to affect writes for all devices on the system
16:38:27 cfriesen kashyap: remember we're talking about "qemu-img convert" here, there's no guest involved
16:39:11 kashyap cfriesen: Yes, sorry for the mixup. I was talking about the other scenario where why people tend to not use caching for guests.
16:39:30 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/520461
16:39:47 kashyap But indeed, we need to distinguish the two cases: (1) The cache used for the guest itself; (2) The cache property during 'qemu-img convert' for src and destination images.
16:42:33 kashyap cfriesen: So in your, case suckage is caused by saturating 'sdb' and hitting the 'dirty_bytes' limit
16:44:06 cfriesen kashyap: yep.
16:45:15 kashyap cfriesen: Might want to write a blog post about it, if you can. Useful for OpenStack users to be mindful of this
16:45:40 mriedem andreykurilin: any idea why our is_latest check here is not just checking for the "latest" word? https://github.com/openstack/python-novaclient/blob/master/novaclient/api_versions.py#L96
16:45:50 mriedem oh nvm https://github.com/openstack/python-novaclient/blob/master/novaclient/api_versions.py#L70
16:46:03 cfriesen kashyap: not a bad idea....things to be aware of when tuning compute nodes. :)
16:46:21 kashyap Yeah
16:46:25 andreykurilin mriedem: yup, inf simplifies calculation
16:46:30 andreykurilin and comparison
16:46:52 cfriesen kashyap: anyway, thanks for thinking about it even if it doesn't make sense for general use.
16:46:53 openstackgerrit Eric Fried proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807
16:46:54 openstackgerrit Eric Fried proposed openstack/nova master: Iterator for a provider tree https://review.openstack.org/520243
16:46:54 openstackgerrit Eric Fried proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693
16:46:55 openstackgerrit Eric Fried proposed openstack/nova master: Scheduler set_inventory_for_provider does nested https://review.openstack.org/520643
16:47:20 mriedem andreykurilin: ok, so it looks like the SecretsHelper code was essentially made useless by https://review.openstack.org/#/c/184281/
16:47:21 kashyap cfriesen: Np; thanks for opening the can^H^H^H^H bringing up the topic :P
16:47:28 cfriesen :)
16:47:48 efried jaypipes ^ Fixed Tetsuro's comments on that first one; and the next two are the start of ComputeDriver.update_inventory framework.
16:47:48 mriedem andreykurilin: because if we don't have a password or token set, we try to discover the api version here https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L796
16:47:51 mriedem which eventually requires auth
16:48:04 mriedem and the secrets helper stuff kicks in here https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L852
16:48:06 jaypipes efried: ack, nice.
16:48:10 efried jaypipes Sorry, the last two. The middle one is just a rebase.
16:48:16 jaypipes ya
16:48:28 andreykurilin mriedem: you are right
16:48:42 mriedem let me see if i can hack that in my devstack to make it prompt for a password
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Refactor the code to check for sufficient hosts https://review.openstack.org/520242
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Add Selection objects https://review.openstack.org/499239
16:57:59 openstackgerrit Ed Leafe proposed openstack/nova master: Modify select_destinations() to return objects and alts https://review.openstack.org/510159
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
17:02:02 mriedem andreykurilin: fixed!

Earlier   Later