Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-16
16:27:30 cfriesen mdbooth: and there's the system limits on how many dirty pages are allowed
16:28:23 andreykurilin mriedem: I do not have a lot of info about it as well:) It was introduced long time ago and used for prompting password, caching(?!) or something like that
16:28:43 mriedem andreykurilin: ok i'd say fix the CLI for using --os-token, and we can deal with SecretsHelper later
16:28:58 mriedem https://github.com/openstack/python-novaclient/commit/7c6f538cf6bc86ecfd9cb25f1bf7f14a6652b592
16:29:19 mriedem i wonder if that even works anymore
16:29:24 mriedem harlowja: ^?
16:29:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test alloc_cands with indirectly sharing RPs https://review.openstack.org/519601
16:29:44 kashyap mdbooth: cfriesen There's some interesting discussion w/ the Block Layer folks on this on #qemu at the moment. So the suggestion is that, Nova should be opinionated based on the knowledge we have
16:29:57 andreykurilin I suppose that part of it still work
16:30:39 mriedem andreykurilin: but i'm not sure if the original reason for it is needed anymore
16:30:46 mriedem we don't require the keyring module in novaclient
16:30:52 mriedem so no one probably knows about this,
16:31:03 mriedem i seem to remember the annoying keyring prompts when i started working on nova back in folsom times
16:31:06 gibi jaypipes, efried: I just pushed the updated https://review.openstack.org/519601 I'm signing off for today and I will not be available on Friday-Monday. Happy hacking!
16:31:17 efried gibi Thank you sir.
16:31:31 jaypipes gibi: noooooooooooooooooooooooooooooooooooooo!
16:31:45 kashyap mdbooth: cfriesen: To summarize: the "opinionated" POV for Nova could be, when using 'qemu-img': for source use 'writeback' and for destination use 'unsafe'
16:31:50 jaypipes gibi: j/k enjoy your weekend. we'll take over your patch as needed.
16:32:05 gibi jaypipes: sorry and thank you
16:32:17 mdbooth kashyap: That wouldn't solve cfriesen 's problem though
16:32:19 kashyap cfriesen: (The QCOW2 developer himself says "I think unsafe caching is the best at least for qcow2 targets")
16:32:25 andreykurilin mriedem: I suppose we can try just remove it and, possibly, no one will notice :)
16:32:29 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Stop posting to os-volumes_boot https://review.openstack.org/473833
16:32:39 cfriesen mdbooth: kashyap: aha, found my notes. we pushed sdb to 100% util, and the global number of dirty pags hit /proc/sys/vm/dirty_bytes. This cause writes to sda to be very slow (which is where the openstack stuff is writing).
16:32:42 mdbooth He's not interested in the performance of qemu-img, but its impact on other processes
16:32:57 cfriesen kashyap: looks like newer qemu-img defaults to "unsafe" for convert
16:32:58 kashyap mdbooth: Yeah, I realize that, I should've phrased better
16:33:15 andreykurilin mriedem: ok, I'll revise the context of the patch and return to it
16:33:16 kashyap cfriesen: Indeed, they're the defaults, I just learnt
16:33:27 openstackgerrit Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/520516
16:33:35 mdbooth The question is whether to compromise the performance of qemu-img to limit its system-wide impact.
16:34:41 cfriesen mdbooth: are you aware of a way to limit what percentage of system-wide dirty page limit a given process can use?
16:34:43 mdbooth cfriesen: Oof, that sucks.
16:35:17 mriedem easy novaclient cleanup thing here https://review.openstack.org/#/c/473833/
16:35:19 mdbooth cfriesen: I'm not.
16:36:00 mriedem andreykurilin: i'm guessing the SecretsHelper isn't working,
16:36:13 mriedem because in my devstack, I unset OS_PASSWORD and ran 'nova list' and got a 401 response,
16:36:14 mdbooth cfriesen: I'm going to have to duck out as I have something else urgent to get done. Happy to continue this conversation tomorrow?
16:36:19 mriedem i wasn't prompted for a password
16:36:21 kashyap mdbooth: Are you saying using 'unsafe' for 'qemu-img convert' sucks?
16:36:34 cfriesen mdbooth: no worries, thanks for the chat
16:36:41 kashyap mdbooth: Just trying to ensure I'm not misparsing you. But yeah, I'll let you proceed.
16:36:59 mdbooth kashyap: No, that saturating sdb indirectly affects sda.
16:37:09 kashyap Ah, okay.
16:37:13 cfriesen kashyap: "unsafe" says that it's possible to keep it in memory and not force it to be written out. I suspect the OS will eventually write it out anyways.
16:37:37 openstackgerrit Merged openstack/python-novaclient master: Fix missing metavar in aggregate-update https://review.openstack.org/512485
16:37:43 kashyap cfriesen: Yeah, the guest has its own cache indeed.
16:37:44 cfriesen kashyap: but if you hit the system-wide dirty-page limit, it's going to affect writes for all devices on the system
16:38:27 cfriesen kashyap: remember we're talking about "qemu-img convert" here, there's no guest involved
16:39:11 kashyap cfriesen: Yes, sorry for the mixup. I was talking about the other scenario where why people tend to not use caching for guests.
16:39:30 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/520461
16:39:47 kashyap But indeed, we need to distinguish the two cases: (1) The cache used for the guest itself; (2) The cache property during 'qemu-img convert' for src and destination images.
16:42:33 kashyap cfriesen: So in your, case suckage is caused by saturating 'sdb' and hitting the 'dirty_bytes' limit
16:44:06 cfriesen kashyap: yep.
16:45:15 kashyap cfriesen: Might want to write a blog post about it, if you can. Useful for OpenStack users to be mindful of this
16:45:40 mriedem andreykurilin: any idea why our is_latest check here is not just checking for the "latest" word? https://github.com/openstack/python-novaclient/blob/master/novaclient/api_versions.py#L96
16:45:50 mriedem oh nvm https://github.com/openstack/python-novaclient/blob/master/novaclient/api_versions.py#L70
16:46:03 cfriesen kashyap: not a bad idea....things to be aware of when tuning compute nodes. :)
16:46:21 kashyap Yeah
16:46:25 andreykurilin mriedem: yup, inf simplifies calculation
16:46:30 andreykurilin and comparison
16:46:52 cfriesen kashyap: anyway, thanks for thinking about it even if it doesn't make sense for general use.
16:46:53 openstackgerrit Eric Fried proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807
16:46:54 openstackgerrit Eric Fried proposed openstack/nova master: Iterator for a provider tree https://review.openstack.org/520243
16:46:54 openstackgerrit Eric Fried proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693
16:46:55 openstackgerrit Eric Fried proposed openstack/nova master: Scheduler set_inventory_for_provider does nested https://review.openstack.org/520643
16:47:20 mriedem andreykurilin: ok, so it looks like the SecretsHelper code was essentially made useless by https://review.openstack.org/#/c/184281/
16:47:21 kashyap cfriesen: Np; thanks for opening the can^H^H^H^H bringing up the topic :P
16:47:28 cfriesen :)
16:47:48 efried jaypipes ^ Fixed Tetsuro's comments on that first one; and the next two are the start of ComputeDriver.update_inventory framework.
16:47:48 mriedem andreykurilin: because if we don't have a password or token set, we try to discover the api version here https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L796
16:47:51 mriedem which eventually requires auth
16:48:04 mriedem and the secrets helper stuff kicks in here https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L852
16:48:06 jaypipes efried: ack, nice.
16:48:10 efried jaypipes Sorry, the last two. The middle one is just a rebase.
16:48:16 jaypipes ya
16:48:28 andreykurilin mriedem: you are right
16:48:42 mriedem let me see if i can hack that in my devstack to make it prompt for a password
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Refactor the code to check for sufficient hosts https://review.openstack.org/520242
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Add Selection objects https://review.openstack.org/499239
16:57:59 openstackgerrit Ed Leafe proposed openstack/nova master: Modify select_destinations() to return objects and alts https://review.openstack.org/510159
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
17:02:02 mriedem andreykurilin: fixed!
17:02:50 mriedem well, i'm prompted for a password now, but apparently it's not used
17:06:16 openstackgerrit Surya Seetharaman proposed openstack/nova master: instance_mapping/request_spec records exist even after the instances are archived https://review.openstack.org/515034
17:06:31 mriedem andreykurilin: so given this has been broken since liberty and no one noticed or cared, i'm thinking we could just remove it
17:06:37 mriedem there is no point in even deprecating it
17:07:50 openstack Launchpad bug 1482040 in OpenStack Compute (nova) "rebuild volume-backed instance with image does not replace image in root disk" [Undecided,Confirmed]
17:07:50 cfriesen mriedem: for https://bugs.launchpad.net/nova/+bug/1482040 would it maybe make sense to disallow rebuild of boot-from-volume instance to a different image (but allow it for the same image)?
17:08:20 openstack Launchpad bug 1732744 in python-novaclient "SecretsHelper has not worked since 2015" [Undecided,Triaged] - Assigned to Matt Riedemann (mriedem)
17:08:20 mriedem https://bugs.launchpad.net/python-novaclient/+bug/1732744
17:08:33 cfriesen mriedem: or should we completely disallow it for boot-from-volume since the image isn't replaced (but other stuff is, so it's still somewhat useful)
17:08:45 mriedem cfriesen: i think you ask me this at least once per week
17:08:54 cfriesen mriedem: last time you said to disallow it
17:09:08 cfriesen mriedem: but I'm wondering about the case where the image matches what was on the volume
17:10:40 mriedem sec
17:20:42 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Fail fast if changing image on a volume-backed server rebuild https://review.openstack.org/520660

Earlier   Later