Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-15
15:54:00 vivsoni__ by just returning 'None' in case of empty connector ... right ?
15:54:58 yyangs we have some noisy neighbours and we are trying to use cpu_quota and cpu_period on flavor to limit the cpu cycles by those vms. It is fine in single vcpu servers. But for multi vcpus server, we are a little confused about the cpu_quota.
15:56:02 yyangs In our tests, we set cpu_period to 200ms, for 2 vcpus server, if we want to allow 50% cpu, should we set to 200ms? or 100ms?
15:57:02 leakypipes yyangs: are you referring to Nova or Kubernetes?
15:57:14 yyangs nova
15:57:42 yyangs leakypipes: we are using kvm
15:58:24 yyangs leakypipes: I though nova also relies on cgroup, right?
15:58:31 leakypipes yyangs: no.
15:59:02 leakypipes yyangs: well, maybe libvirt is configuring cgroups for the VM behind the scene, yeah
15:59:07 smcginnis vivsoni__: Sorry, distracted with some other things. But you still need to handle it in your code, even after mriedem's delicate bits.
15:59:45 vivsoni__ smcginnis: ok
16:00:05 leakypipes yyangs: I'm afraid that feature wasn't particularly complete... I'm pretty sure stuff in https://wiki.openstack.org/wiki/InstanceResourceQuota doesn't work.
16:00:21 vivsoni__ smcginnis: just wanted to confirm are you OK with mridem's fix ?
16:00:44 yyangs leakypipes: we did a couple of tests on cpu
16:01:00 smcginnis vivsoni__: I need to look again, but last update looked right. Doesn't change the need on the driver to handle valid cases where the connector is not passed in.
16:01:26 yyangs leakypipes: we have fixed cpu_period = 200ms, For single vcpus, we can see cpu_quota is actually applied properly
16:01:57 yyangs leakypipes: when cpu_quota 100ms, it only consumes half of cpu cycles
16:03:04 yyangs leakypipes: the question is on multi core, based on cgroups docs, say 2 vcpus 50% cpu cycles, seems cpu_period = 200ms, cpu_quota = 2 * 100=200ms
16:03:06 leakypipes gotta love 5-line patches that add CPU share functionality that have zero tests in them. :( https://review.openstack.org/#/c/22106/4
16:03:36 vivsoni__ smcginnis: ok
16:04:05 leakypipes yyangs: I'm trying to tell you that this "feature" is untested and probably does not work.
16:04:19 yyangs leakypipes: I see
16:05:26 leakypipes yyangs: I don't see any evidence from the patch I linked above that this "feature" worked on VMs with >1 VCPU
16:06:25 mriedem http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140
16:06:30 mriedem it's in the docs, but nothing in the nova code uses this
16:06:37 mriedem so probably an incomplete blueprint
16:06:42 leakypipes yyangs: here is what I would recommend...
16:07:23 leakypipes yyangs: spin up a VM with the number of full VCPU you wish to manage, then set cgroups within that VM manually to control smaller slices of resource (containers, processes, whatever you want).
16:08:13 yyangs leakypipes: e... ok
16:08:20 mriedem test_get_guest_config_with_cpu_quota claims to test thsi
16:08:21 mriedem *this
16:08:28 leakypipes yyangs: I know it's not the answer you were looking for, sorry :(
16:08:32 mriedem but i'm not sure where it's getting the values from in the libvirt driver
16:09:11 yyangs leakypipes: well, when we were doing 2 vcpus test, it actually did give us something. I mean it did limit the cpu cycle
16:09:31 mriedem ah here we go
16:09:35 mriedem see _update_guest_cputune in the libvirt driver
16:10:00 mriedem looks like it depends on your kernel config
16:10:18 leakypipes yyangs: you are trying to do something like 50% of 2 CPUs?
16:10:35 yyangs leakypipes: yest
16:10:42 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3938
16:10:48 mriedem ^ is the method that handles these extra specs
16:10:50 openstackgerrit Merged openstack/nova stable/newton: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/526427
16:10:54 yyangs mriedem: i think our kernel has proper config
16:10:55 openstackgerrit Merged openstack/nova stable/newton: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/526544
16:10:57 mriedem if the kernel doesn't handle it, there should be an error
16:11:01 openstackgerrit Merged openstack/nova stable/pike: propagate OSError to MigrationPreCheckError https://review.openstack.org/515330
16:11:23 mriedem does the guest domain xml have the cputune values in it?
16:11:27 yyangs mriedem: we do see the cputune blob in xml,
16:12:09 mriedem https://libvirt.org/formatdomain.html#elementsCPUTuning
16:13:07 mriedem looks like this has all been around in qemu for a long time
16:13:13 mriedem why it wouldn't be working, not sure
16:13:16 mriedem maybe ask kashyap
16:14:27 yyangs mriedem: we didnt do the cpu pinning, but we do see shares, period and quota in cputune blob
16:14:35 leakypipes mriedem, yyangs: it's not a QEMU/libvirt problem. it's a Nova problem...
16:14:55 leakypipes mriedem, yyangs: it's erroneously setting cpu_shares to 1024 * number of vCPUs:
16:14:56 leakypipes guest.cputune.shares = 1024 * guest.vcpus
16:15:39 giblet mriedem: tried to answer your questions in https://review.openstack.org/#/c/520024/ but I agree to involve leakypipes
16:15:49 leakypipes mriedem, yyangs: shouldn't that be setting CPU share for a single vCPU/qemu-kvm process?
16:16:38 yyangs leakypipes: we didn't try cpu share actually. just tweak cpu_quota with fixed cpu_period.
16:17:03 leakypipes yyangs: unless you specify cpu_shares, it will default to 1024 * VCPUs
16:17:11 leakypipes yyangs: http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3955
16:17:15 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:17:30 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528225
16:17:31 leakypipes yyangs: which would get overridden on line 3961 if you had specified cpu_shares
16:18:03 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ocata: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528226
16:18:08 yyangs leakypipes: oh yes you are right
16:19:00 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:20:54 yyangs leakypipes: mriedem Thanks for your help
16:20:57 andreykurilin_ mriedem: updated the commit message. added some details
16:21:07 leakypipes yyangs: sorry for the early noise... glad we could help.
16:21:13 yyangs leakypipes: mriedem I will do more tests, I didnt notice the cpu_shares
16:21:41 leakypipes yyangs: don't worry, we don't blame you. this area of the code is, well, poorly documented and understood.
16:21:58 leakypipes yyangs: you are free to blame us as much as you want :)
16:22:36 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406
16:22:42 yyangs leakypipes: LOL no no... But it would be better if there are examples in multi vcpus example in openstack docs.
16:23:05 leakypipes yyangs: agreed. once you do your tests, report back to us the results and we'll work with you on a patch to add that doc.
16:23:15 yyangs leakypipes: ok
16:24:56 openstackgerrit Artom Lifshitz proposed openstack/nova stable/newton: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528228
16:25:07 mriedem yyangs: if you find out how this all works and our docs are wrong or incomplete here http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140 - please report a bug so we can clarify the docs
16:25:31 lyarwood artom: erm, was there something wrong with my backports? ;)
16:25:48 artom lyarwood, besides me not knowing they exist? Absolutely nothing
16:25:51 ttsiouts Hi! Anyone interested in reviewing this? https://review.openstack.org/#/c/525110/6 I would really appreciate the help!
16:25:52 figleaf mriedem: any further thoughts on your comment here: https://review.openstack.org/#/c/511358/38/nova/conductor/manager.py@568
16:26:00 mriedem lyarwood: artom: heh, well -2
16:26:02 lyarwood artom: \o/ np
16:26:15 figleaf mriedem: I can't figure out how that's getting hit
16:26:16 mriedem figleaf: requires tracing the request id through the logs in that ironic job
16:26:21 mriedem figleaf: i just haven't had the time to do that yet
16:26:33 mriedem we should at least figure out if we are getting the alternates correctly in that CI run
16:26:50 ttsiouts mriedem: can I ask you about the --force parameter in nova quota-update?
16:26:56 figleaf maybe add some debug logging?
16:27:01 mriedem so first question is, do we get alternates at all? second is, are we using them properly? and if so, why aren't we quitting the retry loop properly?
16:27:10 lyarwood mriedem: so this is never getting into Newton or are we still going to wait to see if things break in master?
16:27:16 mriedem figleaf: sure
16:27:41 mriedem lyarwood: i think by the time we'd find out if this regresses something in master, or pike or ocata, we'll have wanted to eol newton already
16:27:57 mriedem which is why i don't want to hold newton for that change, and i think it's risky to put it in before eol
16:28:03 lyarwood ack
16:28:14 mriedem we can, and should, however, backport the schema change
16:28:15 mriedem imo
16:28:26 mriedem ttsiouts: sure

Earlier   Later