Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-16
16:31:06 gibi jaypipes, efried: I just pushed the updated https://review.openstack.org/519601 I'm signing off for today and I will not be available on Friday-Monday. Happy hacking!
16:31:17 efried gibi Thank you sir.
16:31:31 jaypipes gibi: noooooooooooooooooooooooooooooooooooooo!
16:31:45 kashyap mdbooth: cfriesen: To summarize: the "opinionated" POV for Nova could be, when using 'qemu-img': for source use 'writeback' and for destination use 'unsafe'
16:31:50 jaypipes gibi: j/k enjoy your weekend. we'll take over your patch as needed.
16:32:05 gibi jaypipes: sorry and thank you
16:32:17 mdbooth kashyap: That wouldn't solve cfriesen 's problem though
16:32:19 kashyap cfriesen: (The QCOW2 developer himself says "I think unsafe caching is the best at least for qcow2 targets")
16:32:25 andreykurilin mriedem: I suppose we can try just remove it and, possibly, no one will notice :)
16:32:29 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Stop posting to os-volumes_boot https://review.openstack.org/473833
16:32:39 cfriesen mdbooth: kashyap: aha, found my notes. we pushed sdb to 100% util, and the global number of dirty pags hit /proc/sys/vm/dirty_bytes. This cause writes to sda to be very slow (which is where the openstack stuff is writing).
16:32:42 mdbooth He's not interested in the performance of qemu-img, but its impact on other processes
16:32:57 cfriesen kashyap: looks like newer qemu-img defaults to "unsafe" for convert
16:32:58 kashyap mdbooth: Yeah, I realize that, I should've phrased better
16:33:15 andreykurilin mriedem: ok, I'll revise the context of the patch and return to it
16:33:16 kashyap cfriesen: Indeed, they're the defaults, I just learnt
16:33:27 openstackgerrit Merged openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/520516
16:33:35 mdbooth The question is whether to compromise the performance of qemu-img to limit its system-wide impact.
16:34:41 cfriesen mdbooth: are you aware of a way to limit what percentage of system-wide dirty page limit a given process can use?
16:34:43 mdbooth cfriesen: Oof, that sucks.
16:35:17 mriedem easy novaclient cleanup thing here https://review.openstack.org/#/c/473833/
16:35:19 mdbooth cfriesen: I'm not.
16:36:00 mriedem andreykurilin: i'm guessing the SecretsHelper isn't working,
16:36:13 mriedem because in my devstack, I unset OS_PASSWORD and ran 'nova list' and got a 401 response,
16:36:14 mdbooth cfriesen: I'm going to have to duck out as I have something else urgent to get done. Happy to continue this conversation tomorrow?
16:36:19 mriedem i wasn't prompted for a password
16:36:21 kashyap mdbooth: Are you saying using 'unsafe' for 'qemu-img convert' sucks?
16:36:34 cfriesen mdbooth: no worries, thanks for the chat
16:36:41 kashyap mdbooth: Just trying to ensure I'm not misparsing you. But yeah, I'll let you proceed.
16:36:59 mdbooth kashyap: No, that saturating sdb indirectly affects sda.
16:37:09 kashyap Ah, okay.
16:37:13 cfriesen kashyap: "unsafe" says that it's possible to keep it in memory and not force it to be written out. I suspect the OS will eventually write it out anyways.
16:37:37 openstackgerrit Merged openstack/python-novaclient master: Fix missing metavar in aggregate-update https://review.openstack.org/512485
16:37:43 kashyap cfriesen: Yeah, the guest has its own cache indeed.
16:37:44 cfriesen kashyap: but if you hit the system-wide dirty-page limit, it's going to affect writes for all devices on the system
16:38:27 cfriesen kashyap: remember we're talking about "qemu-img convert" here, there's no guest involved
16:39:11 kashyap cfriesen: Yes, sorry for the mixup. I was talking about the other scenario where why people tend to not use caching for guests.
16:39:30 openstackgerrit Merged openstack/nova master: Updated from global requirements https://review.openstack.org/520461
16:39:47 kashyap But indeed, we need to distinguish the two cases: (1) The cache used for the guest itself; (2) The cache property during 'qemu-img convert' for src and destination images.
16:42:33 kashyap cfriesen: So in your, case suckage is caused by saturating 'sdb' and hitting the 'dirty_bytes' limit
16:44:06 cfriesen kashyap: yep.
16:45:15 kashyap cfriesen: Might want to write a blog post about it, if you can. Useful for OpenStack users to be mindful of this
16:45:40 mriedem andreykurilin: any idea why our is_latest check here is not just checking for the "latest" word? https://github.com/openstack/python-novaclient/blob/master/novaclient/api_versions.py#L96
16:45:50 mriedem oh nvm https://github.com/openstack/python-novaclient/blob/master/novaclient/api_versions.py#L70
16:46:03 cfriesen kashyap: not a bad idea....things to be aware of when tuning compute nodes. :)
16:46:21 kashyap Yeah
16:46:25 andreykurilin mriedem: yup, inf simplifies calculation
16:46:30 andreykurilin and comparison
16:46:52 cfriesen kashyap: anyway, thanks for thinking about it even if it doesn't make sense for general use.
16:46:53 openstackgerrit Eric Fried proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807
16:46:54 openstackgerrit Eric Fried proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693
16:46:54 openstackgerrit Eric Fried proposed openstack/nova master: Iterator for a provider tree https://review.openstack.org/520243
16:46:55 openstackgerrit Eric Fried proposed openstack/nova master: Scheduler set_inventory_for_provider does nested https://review.openstack.org/520643
16:47:20 mriedem andreykurilin: ok, so it looks like the SecretsHelper code was essentially made useless by https://review.openstack.org/#/c/184281/
16:47:21 kashyap cfriesen: Np; thanks for opening the can^H^H^H^H bringing up the topic :P
16:47:28 cfriesen :)
16:47:48 mriedem andreykurilin: because if we don't have a password or token set, we try to discover the api version here https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L796
16:47:48 efried jaypipes ^ Fixed Tetsuro's comments on that first one; and the next two are the start of ComputeDriver.update_inventory framework.
16:47:51 mriedem which eventually requires auth
16:48:04 mriedem and the secrets helper stuff kicks in here https://github.com/openstack/python-novaclient/blob/master/novaclient/shell.py#L852
16:48:06 jaypipes efried: ack, nice.
16:48:10 efried jaypipes Sorry, the last two. The middle one is just a rebase.
16:48:16 jaypipes ya
16:48:28 andreykurilin mriedem: you are right
16:48:42 mriedem let me see if i can hack that in my devstack to make it prompt for a password
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Add Selection objects https://review.openstack.org/499239
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Refactor the code to check for sufficient hosts https://review.openstack.org/520242
16:57:58 openstackgerrit Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854
16:57:59 openstackgerrit Ed Leafe proposed openstack/nova master: Modify select_destinations() to return objects and alts https://review.openstack.org/510159
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
16:58:00 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
17:02:02 mriedem andreykurilin: fixed!
17:02:50 mriedem well, i'm prompted for a password now, but apparently it's not used
17:06:16 openstackgerrit Surya Seetharaman proposed openstack/nova master: instance_mapping/request_spec records exist even after the instances are archived https://review.openstack.org/515034
17:06:31 mriedem andreykurilin: so given this has been broken since liberty and no one noticed or cared, i'm thinking we could just remove it
17:06:37 mriedem there is no point in even deprecating it
17:07:50 cfriesen mriedem: for https://bugs.launchpad.net/nova/+bug/1482040 would it maybe make sense to disallow rebuild of boot-from-volume instance to a different image (but allow it for the same image)?
17:07:50 openstack Launchpad bug 1482040 in OpenStack Compute (nova) "rebuild volume-backed instance with image does not replace image in root disk" [Undecided,Confirmed]
17:08:20 mriedem https://bugs.launchpad.net/python-novaclient/+bug/1732744
17:08:20 openstack Launchpad bug 1732744 in python-novaclient "SecretsHelper has not worked since 2015" [Undecided,Triaged] - Assigned to Matt Riedemann (mriedem)
17:08:33 cfriesen mriedem: or should we completely disallow it for boot-from-volume since the image isn't replaced (but other stuff is, so it's still somewhat useful)
17:08:45 mriedem cfriesen: i think you ask me this at least once per week
17:08:54 cfriesen mriedem: last time you said to disallow it
17:09:08 cfriesen mriedem: but I'm wondering about the case where the image matches what was on the volume
17:10:40 mriedem sec
17:20:42 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Fail fast if changing image on a volume-backed server rebuild https://review.openstack.org/520660
17:20:43 mriedem cfriesen: there you go ^ needs tests
17:23:36 cfriesen mriedem: so we want to disallow rebuild even to the same image? (I think it can fix some problems, like rebuilding cinder volume connections)
17:24:20 mriedem cfriesen: look at the patch
17:24:27 mriedem instance.image_ref != image_href
17:24:58 cfriesen mriedem: okay, but then we'll update the image ref in the instance to point to the new image (instead of having nothing like it did at boot)
17:25:19 mriedem cfriesen: huh?
17:25:31 mriedem we don't save the instance.image_ref until later, after we've already failed here
17:26:07 cfriesen mriedem: in the case of a rebuild-to-same image, before the rebuild the instance image ref is empty, after the rebuild it points to that image
17:26:41 mriedem why is the instance.image_ref empty?
17:27:19 cfriesen mriedem: dunno, it's always been that way. novaclient keys off an empty image ref to say "Attempt to boot from volume - no image supplied"
17:28:05 mriedem if the instance.image_ref is empty because it's a boot from volume, then "and compute_utils.is_volume_backed_instance" is True and we raise an exception so you can't rebuild, correct?
17:28:11 mriedem which means we don't get to the code that calls instance.save()
17:29:09 cfriesen mriedem: yeah, it actually means your proposed patch will result disallowing all rebuilds of boot-from-volume.

Earlier   Later