Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-15
15:35:23 mriedem andreykurilin_: go ahead on the hooks thing, i didn't dig into that very deep, just noticed it while doing something else
15:42:44 andreykurilin_ leakypipes: wow. cool
15:46:56 leakypipes figleaf: you can thank mriedem for that.
15:47:29 mriedem maciejjozefczyk: giblet: questions inline https://review.openstack.org/#/c/520024/
15:47:36 mriedem and you probably want leakypipes to look at ^ as well
15:48:16 vivsoni__ mriedem: Hi
15:48:49 mriedem hi
15:49:08 maciejjozefczyk mriedem: ok
15:50:14 vivsoni__ mriedem: the patch proposed by you to fix for tempest issue(connector empty to terminate_connection)
15:51:01 yyangs Hi
15:51:02 mriedem vivsoni__: yeah?
15:51:18 vivsoni__ mridem: is restored right ?
15:51:21 mriedem yes
15:51:24 vivsoni__ ok
15:51:37 vivsoni__ so will wait for smcginnis for review then
15:51:46 mriedem there were some concerns about it potentially breaking force volume detach, but i think i've handled that in the latest revision
15:52:01 vivsoni__ yes i have seen that
15:52:25 mriedem i am, however, no expert on the delicate intricacies of the cinder volume manager
15:52:38 mriedem plus, i just wanted to say "delicate intricacies"
15:53:30 vivsoni__ in case if Sean disagree, then i need to handle that in our code
15:54:00 vivsoni__ by just returning 'None' in case of empty connector ... right ?
15:54:58 yyangs we have some noisy neighbours and we are trying to use cpu_quota and cpu_period on flavor to limit the cpu cycles by those vms. It is fine in single vcpu servers. But for multi vcpus server, we are a little confused about the cpu_quota.
15:56:02 yyangs In our tests, we set cpu_period to 200ms, for 2 vcpus server, if we want to allow 50% cpu, should we set to 200ms? or 100ms?
15:57:02 leakypipes yyangs: are you referring to Nova or Kubernetes?
15:57:14 yyangs nova
15:57:42 yyangs leakypipes: we are using kvm
15:58:24 yyangs leakypipes: I though nova also relies on cgroup, right?
15:58:31 leakypipes yyangs: no.
15:59:02 leakypipes yyangs: well, maybe libvirt is configuring cgroups for the VM behind the scene, yeah
15:59:07 smcginnis vivsoni__: Sorry, distracted with some other things. But you still need to handle it in your code, even after mriedem's delicate bits.
15:59:45 vivsoni__ smcginnis: ok
16:00:05 leakypipes yyangs: I'm afraid that feature wasn't particularly complete... I'm pretty sure stuff in https://wiki.openstack.org/wiki/InstanceResourceQuota doesn't work.
16:00:21 vivsoni__ smcginnis: just wanted to confirm are you OK with mridem's fix ?
16:00:44 yyangs leakypipes: we did a couple of tests on cpu
16:01:00 smcginnis vivsoni__: I need to look again, but last update looked right. Doesn't change the need on the driver to handle valid cases where the connector is not passed in.
16:01:26 yyangs leakypipes: we have fixed cpu_period = 200ms, For single vcpus, we can see cpu_quota is actually applied properly
16:01:57 yyangs leakypipes: when cpu_quota 100ms, it only consumes half of cpu cycles
16:03:04 yyangs leakypipes: the question is on multi core, based on cgroups docs, say 2 vcpus 50% cpu cycles, seems cpu_period = 200ms, cpu_quota = 2 * 100=200ms
16:03:06 leakypipes gotta love 5-line patches that add CPU share functionality that have zero tests in them. :( https://review.openstack.org/#/c/22106/4
16:03:36 vivsoni__ smcginnis: ok
16:04:05 leakypipes yyangs: I'm trying to tell you that this "feature" is untested and probably does not work.
16:04:19 yyangs leakypipes: I see
16:05:26 leakypipes yyangs: I don't see any evidence from the patch I linked above that this "feature" worked on VMs with >1 VCPU
16:06:25 mriedem http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140
16:06:30 mriedem it's in the docs, but nothing in the nova code uses this
16:06:37 mriedem so probably an incomplete blueprint
16:06:42 leakypipes yyangs: here is what I would recommend...
16:07:23 leakypipes yyangs: spin up a VM with the number of full VCPU you wish to manage, then set cgroups within that VM manually to control smaller slices of resource (containers, processes, whatever you want).
16:08:13 yyangs leakypipes: e... ok
16:08:20 mriedem test_get_guest_config_with_cpu_quota claims to test thsi
16:08:21 mriedem *this
16:08:28 leakypipes yyangs: I know it's not the answer you were looking for, sorry :(
16:08:32 mriedem but i'm not sure where it's getting the values from in the libvirt driver
16:09:11 yyangs leakypipes: well, when we were doing 2 vcpus test, it actually did give us something. I mean it did limit the cpu cycle
16:09:31 mriedem ah here we go
16:09:35 mriedem see _update_guest_cputune in the libvirt driver
16:10:00 mriedem looks like it depends on your kernel config
16:10:18 leakypipes yyangs: you are trying to do something like 50% of 2 CPUs?
16:10:35 yyangs leakypipes: yest
16:10:42 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3938
16:10:48 mriedem ^ is the method that handles these extra specs
16:10:50 openstackgerrit Merged openstack/nova stable/newton: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/526427
16:10:54 yyangs mriedem: i think our kernel has proper config
16:10:55 openstackgerrit Merged openstack/nova stable/newton: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/526544
16:10:57 mriedem if the kernel doesn't handle it, there should be an error
16:11:01 openstackgerrit Merged openstack/nova stable/pike: propagate OSError to MigrationPreCheckError https://review.openstack.org/515330
16:11:23 mriedem does the guest domain xml have the cputune values in it?
16:11:27 yyangs mriedem: we do see the cputune blob in xml,
16:12:09 mriedem https://libvirt.org/formatdomain.html#elementsCPUTuning
16:13:07 mriedem looks like this has all been around in qemu for a long time
16:13:13 mriedem why it wouldn't be working, not sure
16:13:16 mriedem maybe ask kashyap
16:14:27 yyangs mriedem: we didnt do the cpu pinning, but we do see shares, period and quota in cputune blob
16:14:35 leakypipes mriedem, yyangs: it's not a QEMU/libvirt problem. it's a Nova problem...
16:14:55 leakypipes mriedem, yyangs: it's erroneously setting cpu_shares to 1024 * number of vCPUs:
16:14:56 leakypipes guest.cputune.shares = 1024 * guest.vcpus
16:15:39 giblet mriedem: tried to answer your questions in https://review.openstack.org/#/c/520024/ but I agree to involve leakypipes
16:15:49 leakypipes mriedem, yyangs: shouldn't that be setting CPU share for a single vCPU/qemu-kvm process?
16:16:38 yyangs leakypipes: we didn't try cpu share actually. just tweak cpu_quota with fixed cpu_period.
16:17:03 leakypipes yyangs: unless you specify cpu_shares, it will default to 1024 * VCPUs
16:17:11 leakypipes yyangs: http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3955
16:17:15 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:17:30 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528225
16:17:31 leakypipes yyangs: which would get overridden on line 3961 if you had specified cpu_shares
16:18:03 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ocata: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528226
16:18:08 yyangs leakypipes: oh yes you are right
16:19:00 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:20:54 yyangs leakypipes: mriedem Thanks for your help
16:20:57 andreykurilin_ mriedem: updated the commit message. added some details
16:21:07 leakypipes yyangs: sorry for the early noise... glad we could help.
16:21:13 yyangs leakypipes: mriedem I will do more tests, I didnt notice the cpu_shares
16:21:41 leakypipes yyangs: don't worry, we don't blame you. this area of the code is, well, poorly documented and understood.
16:21:58 leakypipes yyangs: you are free to blame us as much as you want :)
16:22:36 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406
16:22:42 yyangs leakypipes: LOL no no... But it would be better if there are examples in multi vcpus example in openstack docs.
16:23:05 leakypipes yyangs: agreed. once you do your tests, report back to us the results and we'll work with you on a patch to add that doc.
16:23:15 yyangs leakypipes: ok
16:24:56 openstackgerrit Artom Lifshitz proposed openstack/nova stable/newton: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528228
16:25:07 mriedem yyangs: if you find out how this all works and our docs are wrong or incomplete here http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140 - please report a bug so we can clarify the docs
16:25:31 lyarwood artom: erm, was there something wrong with my backports? ;)

Earlier   Later