Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-15
15:33:20 cfriesen when shelving, do we preserve the contents of the ephemeral disk(s)? the code looks like it only snapshots the main image, but ephemeral disks are supposed to last the life of the server.
15:33:46 mriedem andreykurilin_: comments / questions in https://review.openstack.org/#/c/424256/ but looks pretty good
15:34:45 mriedem cfriesen: i doubt it
15:34:49 cdent aw, leakypipes, you are a youth
15:34:54 mriedem we snapshot the image and then delete the guest
15:34:56 leakypipes heh
15:35:23 mriedem andreykurilin_: go ahead on the hooks thing, i didn't dig into that very deep, just noticed it while doing something else
15:42:44 andreykurilin_ leakypipes: wow. cool
15:46:56 leakypipes figleaf: you can thank mriedem for that.
15:47:29 mriedem maciejjozefczyk: giblet: questions inline https://review.openstack.org/#/c/520024/
15:47:36 mriedem and you probably want leakypipes to look at ^ as well
15:48:16 vivsoni__ mriedem: Hi
15:48:49 mriedem hi
15:49:08 maciejjozefczyk mriedem: ok
15:50:14 vivsoni__ mriedem: the patch proposed by you to fix for tempest issue(connector empty to terminate_connection)
15:51:01 yyangs Hi
15:51:02 mriedem vivsoni__: yeah?
15:51:18 vivsoni__ mridem: is restored right ?
15:51:21 mriedem yes
15:51:24 vivsoni__ ok
15:51:37 vivsoni__ so will wait for smcginnis for review then
15:51:46 mriedem there were some concerns about it potentially breaking force volume detach, but i think i've handled that in the latest revision
15:52:01 vivsoni__ yes i have seen that
15:52:25 mriedem i am, however, no expert on the delicate intricacies of the cinder volume manager
15:52:38 mriedem plus, i just wanted to say "delicate intricacies"
15:53:30 vivsoni__ in case if Sean disagree, then i need to handle that in our code
15:54:00 vivsoni__ by just returning 'None' in case of empty connector ... right ?
15:54:58 yyangs we have some noisy neighbours and we are trying to use cpu_quota and cpu_period on flavor to limit the cpu cycles by those vms. It is fine in single vcpu servers. But for multi vcpus server, we are a little confused about the cpu_quota.
15:56:02 yyangs In our tests, we set cpu_period to 200ms, for 2 vcpus server, if we want to allow 50% cpu, should we set to 200ms? or 100ms?
15:57:02 leakypipes yyangs: are you referring to Nova or Kubernetes?
15:57:14 yyangs nova
15:57:42 yyangs leakypipes: we are using kvm
15:58:24 yyangs leakypipes: I though nova also relies on cgroup, right?
15:58:31 leakypipes yyangs: no.
15:59:02 leakypipes yyangs: well, maybe libvirt is configuring cgroups for the VM behind the scene, yeah
15:59:07 smcginnis vivsoni__: Sorry, distracted with some other things. But you still need to handle it in your code, even after mriedem's delicate bits.
15:59:45 vivsoni__ smcginnis: ok
16:00:05 leakypipes yyangs: I'm afraid that feature wasn't particularly complete... I'm pretty sure stuff in https://wiki.openstack.org/wiki/InstanceResourceQuota doesn't work.
16:00:21 vivsoni__ smcginnis: just wanted to confirm are you OK with mridem's fix ?
16:00:44 yyangs leakypipes: we did a couple of tests on cpu
16:01:00 smcginnis vivsoni__: I need to look again, but last update looked right. Doesn't change the need on the driver to handle valid cases where the connector is not passed in.
16:01:26 yyangs leakypipes: we have fixed cpu_period = 200ms, For single vcpus, we can see cpu_quota is actually applied properly
16:01:57 yyangs leakypipes: when cpu_quota 100ms, it only consumes half of cpu cycles
16:03:04 yyangs leakypipes: the question is on multi core, based on cgroups docs, say 2 vcpus 50% cpu cycles, seems cpu_period = 200ms, cpu_quota = 2 * 100=200ms
16:03:06 leakypipes gotta love 5-line patches that add CPU share functionality that have zero tests in them. :( https://review.openstack.org/#/c/22106/4
16:03:36 vivsoni__ smcginnis: ok
16:04:05 leakypipes yyangs: I'm trying to tell you that this "feature" is untested and probably does not work.
16:04:19 yyangs leakypipes: I see
16:05:26 leakypipes yyangs: I don't see any evidence from the patch I linked above that this "feature" worked on VMs with >1 VCPU
16:06:25 mriedem http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140
16:06:30 mriedem it's in the docs, but nothing in the nova code uses this
16:06:37 mriedem so probably an incomplete blueprint
16:06:42 leakypipes yyangs: here is what I would recommend...
16:07:23 leakypipes yyangs: spin up a VM with the number of full VCPU you wish to manage, then set cgroups within that VM manually to control smaller slices of resource (containers, processes, whatever you want).
16:08:13 yyangs leakypipes: e... ok
16:08:20 mriedem test_get_guest_config_with_cpu_quota claims to test thsi
16:08:21 mriedem *this
16:08:28 leakypipes yyangs: I know it's not the answer you were looking for, sorry :(
16:08:32 mriedem but i'm not sure where it's getting the values from in the libvirt driver
16:09:11 yyangs leakypipes: well, when we were doing 2 vcpus test, it actually did give us something. I mean it did limit the cpu cycle
16:09:31 mriedem ah here we go
16:09:35 mriedem see _update_guest_cputune in the libvirt driver
16:10:00 mriedem looks like it depends on your kernel config
16:10:18 leakypipes yyangs: you are trying to do something like 50% of 2 CPUs?
16:10:35 yyangs leakypipes: yest
16:10:42 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3938
16:10:48 mriedem ^ is the method that handles these extra specs
16:10:50 openstackgerrit Merged openstack/nova stable/newton: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/526427
16:10:54 yyangs mriedem: i think our kernel has proper config
16:10:55 openstackgerrit Merged openstack/nova stable/newton: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/526544
16:10:57 mriedem if the kernel doesn't handle it, there should be an error
16:11:01 openstackgerrit Merged openstack/nova stable/pike: propagate OSError to MigrationPreCheckError https://review.openstack.org/515330
16:11:23 mriedem does the guest domain xml have the cputune values in it?
16:11:27 yyangs mriedem: we do see the cputune blob in xml,
16:12:09 mriedem https://libvirt.org/formatdomain.html#elementsCPUTuning
16:13:07 mriedem looks like this has all been around in qemu for a long time
16:13:13 mriedem why it wouldn't be working, not sure
16:13:16 mriedem maybe ask kashyap
16:14:27 yyangs mriedem: we didnt do the cpu pinning, but we do see shares, period and quota in cputune blob
16:14:35 leakypipes mriedem, yyangs: it's not a QEMU/libvirt problem. it's a Nova problem...
16:14:55 leakypipes mriedem, yyangs: it's erroneously setting cpu_shares to 1024 * number of vCPUs:
16:14:56 leakypipes guest.cputune.shares = 1024 * guest.vcpus
16:15:39 giblet mriedem: tried to answer your questions in https://review.openstack.org/#/c/520024/ but I agree to involve leakypipes
16:15:49 leakypipes mriedem, yyangs: shouldn't that be setting CPU share for a single vCPU/qemu-kvm process?
16:16:38 yyangs leakypipes: we didn't try cpu share actually. just tweak cpu_quota with fixed cpu_period.
16:17:03 leakypipes yyangs: unless you specify cpu_shares, it will default to 1024 * VCPUs
16:17:11 leakypipes yyangs: http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3955
16:17:15 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:17:30 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528225
16:17:31 leakypipes yyangs: which would get overridden on line 3961 if you had specified cpu_shares
16:18:03 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ocata: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528226
16:18:08 yyangs leakypipes: oh yes you are right
16:19:00 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:20:54 yyangs leakypipes: mriedem Thanks for your help
16:20:57 andreykurilin_ mriedem: updated the commit message. added some details
16:21:07 leakypipes yyangs: sorry for the early noise... glad we could help.
16:21:13 yyangs leakypipes: mriedem I will do more tests, I didnt notice the cpu_shares
16:21:41 leakypipes yyangs: don't worry, we don't blame you. this area of the code is, well, poorly documented and understood.
16:21:58 leakypipes yyangs: you are free to blame us as much as you want :)
16:22:36 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406

Earlier   Later