Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-16
15:17:58 openstackgerrit Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854
15:17:59 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
15:17:59 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707
15:18:00 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
15:21:38 mriedem yikun: https://review.openstack.org/#/c/515640/
15:22:39 jianghuaw_ johnthetubaguy, hi John. Could you help to look at this XenAPI fix? https://review.openstack.org/#/c/514902/
15:24:07 jianghuaw_ johnthetubaguy, I think that's a straightforward fix.
15:29:46 mriedem edleafe: are you going to make the changes in https://review.openstack.org/#/c/520242/ and fix the tests on the change after it? i'd like to hit those today while i'm still on the trail
15:29:52 mriedem the alternate hosts trail
15:29:53 mriedem of blood
15:33:50 edleafe mriedem: you mean the >= change?
15:34:17 edleafe if we ever select more hosts than requested, we have much bigger problems
15:35:11 mriedem ok so leave that,
15:35:15 mriedem but there are the typos and unrelated change
15:35:18 mriedem easy fixes
15:35:33 mriedem i figured you'd hit those while you cleanup the test failures in the next patch above
15:36:23 edleafe mriedem: you did see that I pushed an update to the series like 20 minutes ago?
15:37:15 mriedem orly
15:37:17 mriedem i didn't :)
15:37:51 edleafe :)
15:39:01 mriedem ok, one little thing in the test
15:42:04 johnthetubaguy jianghuaw_: been meaning to look at that, its on my list
15:43:07 openstackgerrit Ed Leafe proposed openstack/nova master: Refactor the code to check for sufficient hosts https://review.openstack.org/520242
15:43:15 edleafe mriedem: ^ added
15:44:42 mriedem thanks, lgtm
15:44:51 mriedem jaypipes: bauzas: you should review https://review.openstack.org/#/c/520242/ as well
15:44:58 edleafe mriedem: the series is blocked because you (plural) wanted to verify the whole process before beginning to commit any of the series. Is it possible to get such a review so that we can begin to merge some of these lower patches?
15:45:43 mriedem jaypipes: bauzas: review note, this change is because the next one in the series is going to be returning Selection objects from FilterScheduler._schedule to FilterScheduler.select_destinations, where that HostState.updated = None was done, which won't work with a Selection object
15:45:57 mriedem so ed moved that validation down into the _schedule part before we convert HostState to Selection objects
15:46:03 jaypipes understood.
15:46:22 mriedem edleafe: i'm working through the series
15:47:17 edleafe mriedem: I understand. I'm thinking of a more high-level view of the end result to verify the approach. Each piece can be later reviewed for specific issues
15:47:36 edleafe I'd just like to merge the lower patches to keep rebase issues to a minimum
15:47:48 edleafe Especially since I'll be out all next week
15:48:12 mriedem i can't promise anything
15:48:29 mriedem but it's good to know you'll be out next week so i don't ping you incessantly about this
15:50:03 mriedem that reminds me, need to request wednesday off...
15:51:17 cfriesen mdbooth: you around?
15:51:56 mdbooth cfriesen: I am
15:53:02 jianghuaw_ johnthetubaguy, thanks:-)
15:54:10 cfriesen mdbooth: we ran into an issue with "qemu-img convert" for a large image on a compute node consuming a lot of cache and affecting the performance of instances on that compute node. we fixed it by adding "-t none" to have it bypass the cache.
15:54:31 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: [ci] Use pseudo-random names for new resources https://review.openstack.org/479777
15:54:46 cfriesen mdbooth: do you think this is something that would be useful more generally?
15:54:51 kashyap cfriesen: "-t" == cache?
15:55:03 cfriesen kashyap: yes.
15:55:18 kashyap cfriesen: And you had 'writeback'?
15:55:18 mdbooth Hmm
15:55:42 cfriesen kashyap: no, normally nova doesn't specify anything for -t so you just get the default
15:55:54 kashyap The default is writeback :-)
15:56:10 kashyap It's the QEMU default that libvirt / Nova inherit
15:56:12 cfriesen well, okay, but we didn't explicitly specify it.
15:57:19 cfriesen _disk_qcow2_to_raw() in nova/virt/libvirt/driver.py
15:57:30 cfriesen and vice-versa
15:57:57 mdbooth kashyap: Yes, cfriesen is essentially complaining that writeback isn't what he wants.
15:58:44 mdbooth cfriesen: What impact did this have on the machine?
15:59:26 kashyap mdboth: I see. In that case, I know cfriesen sometimes posts on 'qemu-devel'; so Chris if you have the perf data then post there and Cc 'qemu-block' list...
15:59:31 efried jaypipes Got local changes to the n-r-p series? I'm doing a restack, and can hit Tetsuro's comments on https://review.openstack.org/#/c/384807/ while I'm at it.
15:59:48 kashyap But yeah, I'll let him to describe the issue here
15:59:49 jaypipes efried: nope.
15:59:52 mdbooth I'm wondering if we give instances cache=none by default
15:59:55 efried jaypipes ight
15:59:59 cfriesen kashyap: mdbooth: digging up the details
16:00:01 mdbooth I thought we did?
16:00:14 kashyap mdbooth: Easy to check, let's examine a Gate job logs...
16:00:41 kashyap I thought we let the default be 'writeback' as QEMU gives us; because there's a reason they settled on it as the default
16:02:10 mdbooth kashyap: Are you sure writeback is the default?
16:02:13 mdbooth writeback is terrible.
16:02:17 kashyap mdbooth: Yes, I just checked
16:02:31 kashyap mdbooth: Just fire up `man qemu-system-x86` and look for 'writeback'
16:03:01 kashyap mdbooth: And...Here's the evidence from the gate jobs:
16:03:04 kashyap http://logs.openstack.org/14/407514/9/check/legacy-grenade-dsvm-neutron-multinode-live-migration/8d355c4/logs/libvirt/qemu/instance-00000019.txt.gz
16:05:10 cfriesen mdbooth: kashyap: from our bug notes it looks like we were actually interfering with other host processes, not the guests. (my bad, I misremembered) So we turned off the cache and limited it to two disk-intensive operations (glance downloads, image conversions, etc.) at a time.
16:06:14 kashyap cfriesen: BTW, what are your guests?
16:06:37 kashyap Are they something super old?
16:06:58 cfriesen kashyap: libvirt/kvm...no idea what the software in the guest was, they were customer tests
16:07:07 bauzas mriedem: sure, will look at the change asap
16:08:24 mdbooth cfriesen: Do you recall what the other host processes were?
16:09:01 kashyap cfriesen: Hmm. As a reminder to self: 'writeback' caches the writes in the host page cache, and 'none' avoids it (but it also doesn't take advantage of the host RAM, especially if you have lots of it)
16:10:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test allocation candidates: multiple aggregates https://review.openstack.org/518633
16:10:45 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test helper: validate provider summaries https://review.openstack.org/518982
16:11:39 kashyap cfriesen: The QEMU folks remind me that, you have to explicitly mention cache mode for both source _and_ destination images.
16:11:44 mdbooth kashyap: cfriesen So, writeback is terrible and we shouldn't be using it for anything, with the possible exception of CI because we don't care about data there and it performs better.
16:11:49 gibi efried, jaypipes: I fixed the comments in https://review.openstack.org/#/c/518633 and in its parent
16:11:57 cfriesen mdbooth: digging in the archives...but from what I remember it filled the cache and then used up all the available dirty pages on the system, slowing down other host "management" processes
16:12:00 mdbooth I suspect, in fact, that we explicitly configure writeback in our CI
16:12:00 efried gibi Thanks!
16:12:55 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:13:06 mriedem andreykurilin: I rebased ^ but am confused about the release note you had in there
16:13:23 mriedem andreykurilin: i removed it because the same release note was already in novaclient per https://review.openstack.org/#/c/425822/
16:13:24 kashyap mdbooth: Hmm, can we make a blanket comment that it is "terrible"? Because the Ceph folks explicitly seem to prefer 'writeback', from my recollection
16:13:31 mdbooth cfriesen: Meh. Tweaking this stuff is opening a can of worms.
16:13:46 mdbooth kashyap: Yep. writeback means something else in ceph :/
16:14:34 kashyap mdbooth: Oh dear, yeah
16:14:36 mdbooth To qemu's ceph driver, writeback means only to inform the ceph server that it can use writeback mode. Data is not cached locally.
16:15:42 kashyap Hmm
16:15:57 cfriesen mdbooth: yeah, that's why mriedem wanted me to talk to you. :) Agreed that it's mostly an issue if you're trying to minimize the amount of RAM used by the host (to maximize how much the guests can get).
16:16:34 mdbooth If it was affecting instance performance it would be different, because we control both of those things.
16:16:58 mdbooth If it's just affecting random other stuff... I'm not convinced we want to go there.
16:17:04 kashyap cfriesen: Can you confirm if were specifying the cache mode for both source & destination disk images separately?
16:17:14 kashyap (When doing 'qemu-img convert', i.e.)

Earlier   Later