Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-15
16:14:55 leakypipes mriedem, yyangs: it's erroneously setting cpu_shares to 1024 * number of vCPUs:
16:14:56 leakypipes guest.cputune.shares = 1024 * guest.vcpus
16:15:39 giblet mriedem: tried to answer your questions in https://review.openstack.org/#/c/520024/ but I agree to involve leakypipes
16:15:49 leakypipes mriedem, yyangs: shouldn't that be setting CPU share for a single vCPU/qemu-kvm process?
16:16:38 yyangs leakypipes: we didn't try cpu share actually. just tweak cpu_quota with fixed cpu_period.
16:17:03 leakypipes yyangs: unless you specify cpu_shares, it will default to 1024 * VCPUs
16:17:11 leakypipes yyangs: http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n3955
16:17:15 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:17:30 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528225
16:17:31 leakypipes yyangs: which would get overridden on line 3961 if you had specified cpu_shares
16:18:03 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ocata: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528226
16:18:08 yyangs leakypipes: oh yes you are right
16:19:00 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: [CLI] Fix token auth type https://review.openstack.org/424256
16:20:54 yyangs leakypipes: mriedem Thanks for your help
16:20:57 andreykurilin_ mriedem: updated the commit message. added some details
16:21:07 leakypipes yyangs: sorry for the early noise... glad we could help.
16:21:13 yyangs leakypipes: mriedem I will do more tests, I didnt notice the cpu_shares
16:21:41 leakypipes yyangs: don't worry, we don't blame you. this area of the code is, well, poorly documented and understood.
16:21:58 leakypipes yyangs: you are free to blame us as much as you want :)
16:22:36 openstackgerrit Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406
16:22:42 yyangs leakypipes: LOL no no... But it would be better if there are examples in multi vcpus example in openstack docs.
16:23:05 leakypipes yyangs: agreed. once you do your tests, report back to us the results and we'll work with you on a patch to add that doc.
16:23:15 yyangs leakypipes: ok
16:24:56 openstackgerrit Artom Lifshitz proposed openstack/nova stable/newton: Don't persist could-be-stale InstanceGroup fields in RequestSpec https://review.openstack.org/528228
16:25:07 mriedem yyangs: if you find out how this all works and our docs are wrong or incomplete here http://git.openstack.org/cgit/openstack/nova/tree/doc/source/user/flavors.rst#n140 - please report a bug so we can clarify the docs
16:25:31 lyarwood artom: erm, was there something wrong with my backports? ;)
16:25:48 artom lyarwood, besides me not knowing they exist? Absolutely nothing
16:25:51 ttsiouts Hi! Anyone interested in reviewing this? https://review.openstack.org/#/c/525110/6 I would really appreciate the help!
16:25:52 figleaf mriedem: any further thoughts on your comment here: https://review.openstack.org/#/c/511358/38/nova/conductor/manager.py@568
16:26:00 mriedem lyarwood: artom: heh, well -2
16:26:02 lyarwood artom: \o/ np
16:26:15 figleaf mriedem: I can't figure out how that's getting hit
16:26:16 mriedem figleaf: requires tracing the request id through the logs in that ironic job
16:26:21 mriedem figleaf: i just haven't had the time to do that yet
16:26:33 mriedem we should at least figure out if we are getting the alternates correctly in that CI run
16:26:50 ttsiouts mriedem: can I ask you about the --force parameter in nova quota-update?
16:26:56 figleaf maybe add some debug logging?
16:27:01 mriedem so first question is, do we get alternates at all? second is, are we using them properly? and if so, why aren't we quitting the retry loop properly?
16:27:10 lyarwood mriedem: so this is never getting into Newton or are we still going to wait to see if things break in master?
16:27:16 mriedem figleaf: sure
16:27:41 mriedem lyarwood: i think by the time we'd find out if this regresses something in master, or pike or ocata, we'll have wanted to eol newton already
16:27:57 mriedem which is why i don't want to hold newton for that change, and i think it's risky to put it in before eol
16:28:03 lyarwood ack
16:28:14 mriedem we can, and should, however, backport the schema change
16:28:15 mriedem imo
16:28:26 mriedem ttsiouts: sure
16:28:36 ttsiouts mriedem: thanks
16:28:37 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Enable limiting GET /allocation_candidates https://review.openstack.org/513526
16:28:51 mriedem figleaf: yeah maybe log the primary host and alternates we get in schedule_and_build_instances,
16:28:59 ttsiouts mriedem: it seems that it is missing from the openstackclient
16:29:03 mriedem and then log the chosen alternate in build_instances before casting to the alternate compute
16:29:06 artom lyarwood, right, let's actually speak to each other on this one: https://review.openstack.org/#/c/528012/ your backport or mine?
16:29:21 mriedem artom: you do the backports
16:29:24 mriedem so lee and i can approve them
16:29:28 lyarwood artom: yup
16:29:34 figleaf mriedem: I can add a patch below my series that just adds that debugging
16:29:36 ttsiouts mriedem: cinder and neutron have no problem setting the quota to a value less than the used..
16:30:09 mriedem figleaf: ok - there is actually a patch for logging the host at the top, sec
16:30:29 mriedem figleaf: https://review.openstack.org/#/c/520369/
16:30:34 mriedem maybe rebase on top of that?
16:31:00 figleaf mriedem: sounds good
16:31:02 mriedem figleaf: another thing this might be,
16:31:11 mriedem not sure, but the host:node cardinality with ironic
16:31:28 mriedem i wonder if we don't have alternates because hosts < nodes
16:31:40 mriedem but, this is a 2 host job, so we should at least have one alternate i'd think?
16:31:52 figleaf 2 host or 2 node?
16:32:01 mriedem the multinode job should be 2 compute services
16:32:02 mriedem so 2 hosts
16:32:05 figleaf ok
16:32:07 mriedem i don't know how many nodes it has per host
16:32:37 mriedem but, i think this also means that the alternates stuff doesn't necessarily solve the reschedule problem for ironic, not in it's current form anyway
16:32:50 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: Remove 2 redundant methods https://review.openstack.org/528325
16:33:01 mriedem because we could have host:nodeA and host:nodeB, fail on nodeA and not try to reschedule to nodeB
16:33:15 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Verify parameters in servers.inc https://review.openstack.org/528201
16:33:16 mriedem because we think, "oh well that host failed so i'm not going to try it again"
16:33:22 figleaf mriedem: yeah, I remember having that discussion with some ironic people in Denver
16:33:28 mriedem just a guess at this point
16:33:54 mriedem i think we'd know if we have debug logs that show we try host 1, reschedule to host 2, and then need to reschedule again and are out of hosts but not out of nodes
16:34:10 andreykurilin_ mriedem: and a simple cleanup for you - https://review.openstack.org/528325 :)
16:35:02 mriedem ttsiouts: so your issue is that osc doesn't have a --force option for nova quota-update?
16:35:11 artom Wait, we don't have pike placeholders?
16:35:19 mriedem ttsiouts: if so, feel free to write a patch to openstackclient
16:35:41 artom All I see is Dan's a5e051edd9 adding the Ocata ones
16:35:56 ttsiouts mriedem: thanks
16:36:56 artom Ah, no, I was confused
16:37:14 artom The Pike placeholders will be in master, so it's b198ad349a that added them
16:37:21 artom The Ocata placeholder in Pike, etc
16:37:33 mriedem yes
16:37:39 mriedem https://review.openstack.org/#/c/427900/ was done in pike for the ocata backports
16:37:41 mriedem whew
16:40:59 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330
16:43:27 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
16:43:39 melwitt lyarwood: these stable backports would appreciate your review https://review.openstack.org/#/c/523910 https://review.openstack.org/#/c/523911
16:43:56 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ocata: Make request_spec.spec MediumText https://review.openstack.org/528332
16:43:58 figleaf mriedem: ugh, can't log the alternates in build_instances on that patch, since that method doesn't get alternates until https://review.openstack.org/#/c/511358
16:44:21 figleaf so I don't think that alternates could be causing the ironic failure
16:44:24 mriedem figleaf: i think it's fine to just update your patches to log the alternates at debug while we debug the actual issue
16:44:36 mriedem figleaf: well the ironic patch is trying to reschedule
16:44:38 figleaf since they aren't yet being passed
16:44:47 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330

Earlier   Later