| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-15 | |||
| 16:01:57 | yyangs | leakypipes: when cpu_quota 100ms, it only consumes half of cpu cycles | |
| 16:03:04 | yyangs | leakypipes: the question is on multi core, based on cgroups docs, say 2 vcpus 50% cpu cycles, seems cpu_period = 200ms, cpu_quota = 2 * 100=200ms | |
| 16:03:06 | leakypipes | gotta love 5-line patches that add CPU share functionality that have zero tests in them. :( https://review.openstack.org/#/c/22106/4 | |
| 16:03:36 | vivsoni__ | smcginnis: ok | |
| 16:04:05 | leakypipes | yyangs: I'm trying to tell you that this "feature" is untested and probably does not work. | |
| 16:04:19 | yyangs | leakypipes: I see | |
| 16:05:26 | leakypipes | yyangs: I don't see any evidence from the patch I linked above that this "feature" worked on VMs with >1 VCPU | |
| 16:06:25 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140 | |
| 16:06:30 | mriedem | it's in the docs, but nothing in the nova code uses this | |
| 16:06:37 | mriedem | so probably an incomplete blueprint | |
| 16:06:42 | leakypipes | yyangs: here is what I would recommend... | |
| 16:07:23 | leakypipes | yyangs: spin up a VM with the number of full VCPU you wish to manage, then set cgroups within that VM manually to control smaller slices of resource (containers, processes, whatever you want). | |
| 16:08:13 | yyangs | leakypipes: e... ok | |
| 16:08:20 | mriedem | test_get_guest_config_with_cpu_quota claims to test thsi | |
| 16:08:21 | mriedem | *this | |
| 16:08:28 | leakypipes | yyangs: I know it's not the answer you were looking for, sorry :( | |
| 16:08:32 | mriedem | but i'm not sure where it's getting the values from in the libvirt driver | |
| 16:09:11 | yyangs | leakypipes: well, when we were doing 2 vcpus test, it actually did give us something. I mean it did limit the cpu cycle | |
| 16:09:31 | mriedem | ah here we go | |
| 16:09:35 | mriedem | see _update_guest_cputune in the libvirt driver | |
| 16:10:00 | mriedem | looks like it depends on your kernel config | |
| 16:10:18 | leakypipes | yyangs: you are trying to do something like 50% of 2 CPUs? | |
| 16:10:35 | yyangs | leakypipes: yest | |
| 16:10:42 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3938 | |
| 16:10:48 | mriedem | ^ is the method that handles these extra specs | |
| 16:10:50 | openstackgerrit | Merged openstack/nova stable/newton: Fix 'force' parameter in os-quota-sets PUT schema https://review.openstack.org/526427 | |
| 16:10:54 | yyangs | mriedem: i think our kernel has proper config | |
| 16:10:55 | openstackgerrit | Merged openstack/nova stable/newton: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/526544 | |
| 16:10:57 | mriedem | if the kernel doesn't handle it, there should be an error | |
| 16:11:01 | openstackgerrit | Merged openstack/nova stable/pike: propagate OSError to MigrationPreCheckError https://review.openstack.org/515330 | |
| 16:11:23 | mriedem | does the guest domain xml have the cputune values in it? | |
| 16:11:27 | yyangs | mriedem: we do see the cputune blob in xml, | |
| 16:12:09 | mriedem | https://libvirt.org/formatdomain.html#elementsCPUTuning | |
| 16:13:07 | mriedem | looks like this has all been around in qemu for a long time | |
| 16:13:13 | mriedem | why it wouldn't be working, not sure | |
| 16:13:16 | mriedem | maybe ask kashyap | |
| 16:14:27 | yyangs | mriedem: we didnt do the cpu pinning, but we do see shares, period and quota in cputune blob | |
| 16:14:35 | leakypipes | mriedem, yyangs: it's not a QEMU/libvirt problem. it's a Nova problem... | |
| 16:14:55 | leakypipes | mriedem, yyangs: it's erroneously setting cpu_shares to 1024 * number of vCPUs: | |
| 16:14:56 | leakypipes | guest.cputune.shares = 1024 * guest.vcpus | |
| 16:15:39 | giblet | mriedem: tried to answer your questions in https://review.openstack.org/#/c/520024/ but I agree to involve leakypipes | |
| 16:15:49 | leakypipes | mriedem, yyangs: shouldn't that be setting CPU share for a single vCPU/qemu-kvm process? | |
| 16:16:38 | yyangs | leakypipes: we didn't try cpu share actually. just tweak cpu_quota with fixed cpu_period. | |
| 16:17:03 | leakypipes | yyangs: unless you specify cpu_shares, it will default to 1024 * VCPUs | |
| 16:17:11 | leakypipes | yyangs: http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3955 | |
| 16:17:15 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256 | |
| 16:17:30 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528225 | |
| 16:17:31 | leakypipes | yyangs: which would get overridden on line 3961 if you had specified cpu_shares | |
| 16:18:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/ocata: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528226 | |
| 16:18:08 | yyangs | leakypipes: oh yes you are right | |
| 16:19:00 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256 | |
| 16:20:54 | yyangs | leakypipes: mriedem Thanks for your help | |
| 16:20:57 | andreykurilin_ | mriedem: updated the commit message. added some details | |
| 16:21:07 | leakypipes | yyangs: sorry for the early noise... glad we could help. | |
| 16:21:13 | yyangs | leakypipes: mriedem I will do more tests, I didnt notice the cpu_shares | |
| 16:21:41 | leakypipes | yyangs: don't worry, we don't blame you. this area of the code is, well, poorly documented and understood. | |
| 16:21:58 | leakypipes | yyangs: you are free to blame us as much as you want :) | |
| 16:22:36 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406 | |
| 16:22:42 | yyangs | leakypipes: LOL no no... But it would be better if there are examples in multi vcpus example in openstack docs. | |
| 16:23:05 | leakypipes | yyangs: agreed. once you do your tests, report back to us the results and we'll work with you on a patch to add that doc. | |
| 16:23:15 | yyangs | leakypipes: ok | |
| 16:24:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/newton: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528228 | |
| 16:25:07 | mriedem | yyangs: if you find out how this all works and our docs are wrong or incomplete here http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140 - please report a bug so we can clarify the docs | |
| 16:25:31 | lyarwood | artom: erm, was there something wrong with my backports? ;) | |
| 16:25:48 | artom | lyarwood, besides me not knowing they exist? Absolutely nothing | |
| 16:25:51 | ttsiouts | Hi! Anyone interested in reviewing this? https://review.openstack.org/#/c/525110/6 I would really appreciate the help! | |
| 16:25:52 | figleaf | mriedem: any further thoughts on your comment here: https://review.openstack.org/#/c/511358/38/nova/conductor/manager.py@568 | |
| 16:26:00 | mriedem | lyarwood: artom: heh, well -2 | |
| 16:26:02 | lyarwood | artom: \o/ np | |
| 16:26:15 | figleaf | mriedem: I can't figure out how that's getting hit | |
| 16:26:16 | mriedem | figleaf: requires tracing the request id through the logs in that ironic job | |
| 16:26:21 | mriedem | figleaf: i just haven't had the time to do that yet | |
| 16:26:33 | mriedem | we should at least figure out if we are getting the alternates correctly in that CI run | |
| 16:26:50 | ttsiouts | mriedem: can I ask you about the --force parameter in nova quota-update? | |
| 16:26:56 | figleaf | maybe add some debug logging? | |
| 16:27:01 | mriedem | so first question is, do we get alternates at all? second is, are we using them properly? and if so, why aren't we quitting the retry loop properly? | |
| 16:27:10 | lyarwood | mriedem: so this is never getting into Newton or are we still going to wait to see if things break in master? | |
| 16:27:16 | mriedem | figleaf: sure | |
| 16:27:41 | mriedem | lyarwood: i think by the time we'd find out if this regresses something in master, or pike or ocata, we'll have wanted to eol newton already | |
| 16:27:57 | mriedem | which is why i don't want to hold newton for that change, and i think it's risky to put it in before eol | |
| 16:28:03 | lyarwood | ack | |
| 16:28:14 | mriedem | we can, and should, however, backport the schema change | |
| 16:28:15 | mriedem | imo | |
| 16:28:26 | mriedem | ttsiouts: sure | |
| 16:28:36 | ttsiouts | mriedem: thanks | |
| 16:28:37 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Enable limiting GET /allocation_candidates https://review.openstack.org/513526 | |
| 16:28:51 | mriedem | figleaf: yeah maybe log the primary host and alternates we get in schedule_and_build_instances, | |
| 16:28:59 | ttsiouts | mriedem: it seems that it is missing from the openstackclient | |
| 16:29:03 | mriedem | and then log the chosen alternate in build_instances before casting to the alternate compute | |
| 16:29:06 | artom | lyarwood, right, let's actually speak to each other on this one: https://review.openstack.org/#/c/528012/ your backport or mine? | |
| 16:29:21 | mriedem | artom: you do the backports | |
| 16:29:24 | mriedem | so lee and i can approve them | |
| 16:29:28 | lyarwood | artom: yup | |
| 16:29:34 | figleaf | mriedem: I can add a patch below my series that just adds that debugging | |
| 16:29:36 | ttsiouts | mriedem: cinder and neutron have no problem setting the quota to a value less than the used.. | |
| 16:30:09 | mriedem | figleaf: ok - there is actually a patch for logging the host at the top, sec | |
| 16:30:29 | mriedem | figleaf: https://review.openstack.org/#/c/520369/ | |
| 16:30:34 | mriedem | maybe rebase on top of that? | |
| 16:31:00 | figleaf | mriedem: sounds good | |
| 16:31:02 | mriedem | figleaf: another thing this might be, | |