Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-15
15:51:21 mriedem yes
15:51:24 vivsoni__ ok
15:51:37 vivsoni__ so will wait for smcginnis for review then
15:51:46 mriedem there were some concerns about it potentially breaking force volume detach, but i think i've handled that in the latest revision
15:52:01 vivsoni__ yes i have seen that
15:52:25 mriedem i am, however, no expert on the delicate intricacies of the cinder volume manager
15:52:38 mriedem plus, i just wanted to say "delicate intricacies"
15:53:30 vivsoni__ in case if Sean disagree, then i need to handle that in our code
15:54:00 vivsoni__ by just returning 'None' in case of empty connector ... right ?
15:54:58 yyangs we have some noisy neighbours and we are trying to use cpu_quota and cpu_period on flavor to limit the cpu cycles by those vms. It is fine in single vcpu servers. But for multi vcpus server, we are a little confused about the cpu_quota.
15:56:02 yyangs In our tests, we set cpu_period to 200ms, for 2 vcpus server, if we want to allow 50% cpu, should we set to 200ms? or 100ms?
15:57:02 leakypipes yyangs: are you referring to Nova or Kubernetes?
15:57:14 yyangs nova
15:57:42 yyangs leakypipes: we are using kvm
15:58:24 yyangs leakypipes: I though nova also relies on cgroup, right?
15:58:31 leakypipes yyangs: no.
15:59:02 leakypipes yyangs: well, maybe libvirt is configuring cgroups for the VM behind the scene, yeah
15:59:07 smcginnis vivsoni__: Sorry, distracted with some other things. But you still need to handle it in your code, even after mriedem's delicate bits.
15:59:45 vivsoni__ smcginnis: ok
16:00:05 leakypipes yyangs: I'm afraid that feature wasn't particularly complete... I'm pretty sure stuff in https://wiki.openstack.org/wiki/InstanceResourceQuota doesn't work.
16:00:21 vivsoni__ smcginnis: just wanted to confirm are you OK with mridem's fix ?
16:00:44 yyangs leakypipes: we did a couple of tests on cpu
16:01:00 smcginnis vivsoni__: I need to look again, but last update looked right. Doesn't change the need on the driver to handle valid cases where the connector is not passed in.
16:01:26 yyangs leakypipes: we have fixed cpu_period = 200ms, For single vcpus, we can see cpu_quota is actually applied properly
16:01:57 yyangs leakypipes: when cpu_quota 100ms, it only consumes half of cpu cycles
16:03:04 yyangs leakypipes: the question is on multi core, based on cgroups docs, say 2 vcpus 50% cpu cycles, seems cpu_period = 200ms, cpu_quota = 2 * 100=200ms
16:03:06 leakypipes gotta love 5-line patches that add CPU share functionality that have zero tests in them. :( https://review.openstack.org/#/c/22106/4
16:03:36 vivsoni__ smcginnis: ok
16:04:05 leakypipes yyangs: I'm trying to tell you that this "feature" is untested and probably does not work.
16:04:19 yyangs leakypipes: I see
16:05:26 leakypipes yyangs: I don't see any evidence from the patch I linked above that this "feature" worked on VMs with >1 VCPU
16:06:25 mriedem http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140
16:06:30 mriedem it's in the docs, but nothing in the nova code uses this
16:06:37 mriedem so probably an incomplete blueprint
16:06:42 leakypipes yyangs: here is what I would recommend...
16:07:23 leakypipes yyangs: spin up a VM with the number of full VCPU you wish to manage, then set cgroups within that VM manually to control smaller slices of resource (containers, processes, whatever you want).
16:08:13 yyangs leakypipes: e... ok
16:08:20 mriedem test_get_guest_config_with_cpu_quota claims to test thsi
16:08:21 mriedem *this
16:08:28 leakypipes yyangs: I know it's not the answer you were looking for, sorry :(
16:08:32 mriedem but i'm not sure where it's getting the values from in the libvirt driver
16:09:11 yyangs leakypipes: well, when we were doing 2 vcpus test, it actually did give us something. I mean it did limit the cpu cycle
16:09:31 mriedem ah here we go
16:09:35 mriedem see _update_guest_cputune in the libvirt driver
16:10:00 mriedem looks like it depends on your kernel config
16:10:18 leakypipes yyangs: you are trying to do something like 50% of 2 CPUs?
16:10:35 yyangs leakypipes: yest
16:10:42 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3938
16:10:48 mriedem ^ is the method that handles these extra specs
16:10:50 openstackgerrit Merged openstack/nova stable/newton: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/526427
16:10:54 yyangs mriedem: i think our kernel has proper config
16:10:55 openstackgerrit Merged openstack/nova stable/newton: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/526544
16:10:57 mriedem if the kernel doesn't handle it, there should be an error
16:11:01 openstackgerrit Merged openstack/nova stable/pike: propagate OSError to MigrationPreCheckError https://review.openstack.org/515330
16:11:23 mriedem does the guest domain xml have the cputune values in it?
16:11:27 yyangs mriedem: we do see the cputune blob in xml,
16:12:09 mriedem https://libvirt.org/formatdomain.html#elementsCPUTuning
16:13:07 mriedem looks like this has all been around in qemu for a long time
16:13:13 mriedem why it wouldn't be working, not sure
16:13:16 mriedem maybe ask kashyap
16:14:27 yyangs mriedem: we didnt do the cpu pinning, but we do see shares, period and quota in cputune blob
16:14:35 leakypipes mriedem, yyangs: it's not a QEMU/libvirt problem. it's a Nova problem...
16:14:55 leakypipes mriedem, yyangs: it's erroneously setting cpu_shares to 1024 * number of vCPUs:
16:14:56 leakypipes guest.cputune.shares = 1024 * guest.vcpus
16:15:39 giblet mriedem: tried to answer your questions in https://review.openstack.org/#/c/520024/ but I agree to involve leakypipes
16:15:49 leakypipes mriedem, yyangs: shouldn't that be setting CPU share for a single vCPU/qemu-kvm process?
16:16:38 yyangs leakypipes: we didn't try cpu share actually. just tweak cpu_quota with fixed cpu_period.
16:17:03 leakypipes yyangs: unless you specify cpu_shares, it will default to 1024 * VCPUs
16:17:11 leakypipes yyangs: http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3955
16:17:15 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:17:30 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528225
16:17:31 leakypipes yyangs: which would get overridden on line 3961 if you had specified cpu_shares
16:18:03 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ocata: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528226
16:18:08 yyangs leakypipes: oh yes you are right
16:19:00 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:20:54 yyangs leakypipes: mriedem Thanks for your help
16:20:57 andreykurilin_ mriedem: updated the commit message. added some details
16:21:07 leakypipes yyangs: sorry for the early noise... glad we could help.
16:21:13 yyangs leakypipes: mriedem I will do more tests, I didnt notice the cpu_shares
16:21:41 leakypipes yyangs: don't worry, we don't blame you. this area of the code is, well, poorly documented and understood.
16:21:58 leakypipes yyangs: you are free to blame us as much as you want :)
16:22:36 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406
16:22:42 yyangs leakypipes: LOL no no... But it would be better if there are examples in multi vcpus example in openstack docs.
16:23:05 leakypipes yyangs: agreed. once you do your tests, report back to us the results and we'll work with you on a patch to add that doc.
16:23:15 yyangs leakypipes: ok
16:24:56 openstackgerrit Artom Lifshitz proposed openstack/nova stable/newton: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528228
16:25:07 mriedem yyangs: if you find out how this all works and our docs are wrong or incomplete here http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140 - please report a bug so we can clarify the docs
16:25:31 lyarwood artom: erm, was there something wrong with my backports? ;)
16:25:48 artom lyarwood, besides me not knowing they exist? Absolutely nothing
16:25:51 ttsiouts Hi! Anyone interested in reviewing this? https://review.openstack.org/#/c/525110/6 I would really appreciate the help!
16:25:52 figleaf mriedem: any further thoughts on your comment here: https://review.openstack.org/#/c/511358/38/nova/conductor/manager.py@568
16:26:00 mriedem lyarwood: artom: heh, well -2
16:26:02 lyarwood artom: \o/ np
16:26:15 figleaf mriedem: I can't figure out how that's getting hit
16:26:16 mriedem figleaf: requires tracing the request id through the logs in that ironic job
16:26:21 mriedem figleaf: i just haven't had the time to do that yet
16:26:33 mriedem we should at least figure out if we are getting the alternates correctly in that CI run
16:26:50 ttsiouts mriedem: can I ask you about the --force parameter in nova quota-update?
16:26:56 figleaf maybe add some debug logging?
16:27:01 mriedem so first question is, do we get alternates at all? second is, are we using them properly? and if so, why aren't we quitting the retry loop properly?

Earlier   Later