| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 14:11:51 | sean-k-mooney | with out the starlinx hacks | |
| 14:12:03 | stephenfin | and because you can, we're going to have to handle that | |
| 14:12:06 | jaypipes | mriedem: cpu_allocation_ratio=1.0 has nothing to do with pinned CPUs. | |
| 14:12:16 | dansmith | well, | |
| 14:12:20 | jaypipes | (which is part of the problem) | |
| 14:12:20 | dansmith | it's all over this spec | |
| 14:12:29 | stephenfin | hence my inclination towards draining hosts and moving them to other, newly configured hosts | |
| 14:12:39 | stephenfin | *moving the instances | |
| 14:12:40 | mriedem | my point was we could have used that as indication a host can only have dedicated cpu guests | |
| 14:12:53 | mriedem | but since we didn't do that, yeah we could have mixed on the same host i guess and have to deal wit it | |
| 14:12:55 | sean-k-mooney | cpu_allocation_ratio=1.0 jsut disables oversubsciption unfrotunetly | |
| 14:12:55 | mriedem | *with | |
| 14:12:59 | dansmith | I'm -2 on making people move instances to update counting numbers in placement | |
| 14:13:31 | stephenfin | mriedem: yeah, cpu_allocation_ratio is ignored for pinned instances | |
| 14:13:36 | jaypipes | dansmith: especially when those instances are pets and pandas. all of them. | |
| 14:13:36 | sean-k-mooney | anyway im personally not to worried about fixing the allocations. | |
| 14:13:41 | dansmith | aye | |
| 14:13:42 | sean-k-mooney | i think we can do that | |
| 14:14:03 | stephenfin | so I'd imagine it's set to 16.0 for most deployments, regardless of the workload | |
| 14:14:18 | cdent | dansmith: I remain confused about why it isn't okay for those instance to remain defined/allocation in the "old way"? | |
| 14:14:28 | sean-k-mooney | stephenfin: it depense on the deployment tool | |
| 14:14:32 | stephenfin | yup | |
| 14:14:39 | dansmith | cdent: I don't think it is | |
| 14:14:39 | bauzas | sorry folks, I had to go AFK | |
| 14:14:43 | dansmith | cdent: we have to reshape | |
| 14:14:58 | cdent | dansmith: I hear you, I'm asking "why?" | |
| 14:15:08 | bauzas | dansmith: my point is that I think we only need to reshape for CONF.vcpu_pin_set | |
| 14:15:18 | bauzas | for the other options, we don't need it | |
| 14:15:30 | sean-k-mooney | vcpu_pin_set is used for host with shared cpus too | |
| 14:15:31 | bauzas | stephenfin: ^ | |
| 14:15:37 | dansmith | cdent: why not just leave them with the old accounting for five years? | |
| 14:15:44 | bauzas | sean-k-mooney: I know, that's why we need to reshape | |
| 14:15:50 | cdent | if that's how long they live, sure | |
| 14:15:55 | bauzas | but only for this option | |
| 14:16:06 | stephenfin | sean-k-mooney: Is it? I thought that was totally ignored unless you'd a NUMA topology | |
| 14:16:13 | sean-k-mooney | yes | |
| 14:16:19 | sean-k-mooney | but you can have numa without pinning | |
| 14:16:20 | cdent | dansmith: I'm not suggesting it, I'm asking why it is not okay. | |
| 14:16:27 | sean-k-mooney | and actully no | |
| 14:16:31 | stephenfin | Instead you've to use that reserved_cpus option (or whatever it's called) | |
| 14:16:37 | dansmith | cdent: because it means new instances scheduled to compute nodes that don't have proper accounting would also have to be accounted the old way? | |
| 14:16:55 | stephenfin | cdent: Yeah, that's what I was thinking to ^ | |
| 14:16:57 | stephenfin | *too | |
| 14:16:57 | mriedem | cdent: by "remain defined/allocation in the "old way"?" do you mean reporting VCPU allocations to placement rather than PCPU? | |
| 14:17:03 | sean-k-mooney | if you have no numa toplogy th enumber of enabeld cores in vcpu pinnset is still used to determin the number of cores reported to the resouce tracker and therefor to placemnt | |
| 14:17:13 | cdent | dansmith: make them end up on other nodes? | |
| 14:17:28 | stephenfin | We can't schedule new instances to that host until we know how many PCPUs are actually in use there | |
| 14:17:33 | dansmith | cdent: so I have to waste the capacity on those nodes until long-lived instances die? | |
| 14:17:46 | cdent | mriedem: yes, if that's how they were booted in the first place, how/why should they change | |
| 14:17:48 | dansmith | right, what stephenfin said | |
| 14:17:50 | cdent | dansmith: yes! | |
| 14:17:55 | dansmith | cdent: um, no | |
| 14:17:56 | sean-k-mooney | stephenfin: the advice that many gave was never to user reserved_cpus and alwauys use vcpu_pin_set instead | |
| 14:17:57 | mnaser | that's a terrible idea | |
| 14:18:04 | mnaser | tbh with my operators 2 cents: I'm not ok with moving around all my instances around to magically reshape things | |
| 14:18:09 | dansmith | this ^ | |
| 14:18:10 | mriedem | since i just went through mel's change to counting usage from placement, your quotas would be all out of whack too | |
| 14:18:18 | cdent | I'm not suggesting that people migrate their instances | |
| 14:18:22 | mnaser | and I'm not okay with my capacity sitting empty because this is $$$$ | |
| 14:18:24 | stephenfin | sean-k-mooney: Again, advice that wasn't enforced anywhere and therefore not something we can rely on :( | |
| 14:18:25 | dansmith | moving gigs and gigs of live instances so that we can adjust integers in placement is INSANE | |
| 14:18:26 | sean-k-mooney | the reason is vcpu_pin_set takes effect before teh cpu_allocation_reatio is applied and reserved_happens after | |
| 14:18:30 | cdent | meh | |
| 14:18:38 | cdent | I never suggested anybody do any migrations | |
| 14:18:41 | dansmith | reserving capacity until a five-year instance goes away so we can update integers in placement is INSANE | |
| 14:18:50 | cdent | Let stuff live out its lifecycle | |
| 14:18:56 | stephenfin | cdent: Yup, that's all me. Sorry :) | |
| 14:18:56 | mnaser | that's not possible though, that's the thing | |
| 14:19:00 | mnaser | I have no control over my environment | |
| 14:19:26 | cdent | dansmith: I'm not concerned about this from a placement standpoint: abuse placement all we want, it'll take it | |
| 14:19:41 | cdent | I'm concerned about it from a magical recognition happening on the compute node | |
| 14:19:43 | dansmith | cdent: yeah, this isn't a placement concern, it's a nova concern | |
| 14:20:05 | stephenfin | OK, so we're rewriting flavors and moving allocations around to switch everything to the new system. If that's the case, we're back to trying to think of all the edge cases that exist | |
| 14:20:11 | stephenfin | and there are many. Many many | |
| 14:20:33 | sean-k-mooney | stephenfin: we can support the new flow without requrieing flavor to be modifed | |
| 14:20:44 | dansmith | sean-k-mooney: he means the instance's flavor I think | |
| 14:20:46 | stephenfin | sean-k-mooney: via a shim, I guess? | |
| 14:20:57 | stephenfin | dansmith: I do. The embedded one | |
| 14:21:12 | sean-k-mooney | stephenfin: today we generate the palcement request via the request spec | |
| 14:21:15 | dansmith | stephenfin: you can write a nova-status check that verifies that all the instances can be fit to whatever minimally simplified scheme you support, | |
| 14:21:27 | dansmith | to warn people before they upgrade with complex instances that can't be fixed or something | |
| 14:21:28 | sean-k-mooney | we convert the VCPU element in the flavor in to a vcpu resouce request | |
| 14:22:00 | sean-k-mooney | that code can take account of the hw:cpu_policy extra spec and jsut assk for PCPU resouces instread | |
| 14:22:06 | stephenfin | dansmith: That's going to be a big check, fair warning :) | |
| 14:22:13 | dansmith | stephenfin: I'm just throwing ideas | |
| 14:22:32 | stephenfin | Yup, and good ones too | |
| 14:22:40 | dansmith | stephenfin: migrating everything, deleting everything, not upgrading until instances age out -- all not options, IMHO | |
| 14:22:55 | bauzas | dansmith: mnaser: okay, sorry, that's me who proposed migrating instances, and it was a terrible idea, I reckon | |
| 14:23:14 | bauzas | so we should stop thinking about this possibility | |
| 14:23:37 | mnaser | bauzas: all good :) ideas are good to bring up anyways | |
| 14:23:40 | bauzas | but, then, we want to just make sure that when creating a new RC, we also look at the existing capacity | |
| 14:24:12 | stephenfin | So supporting instances with just 'hw:cpu_policy=dedicated' in a deployment that has used aggregates as we suggest seems pretty easy | |
| 14:24:12 | bauzas | see the example I provided : https://review.openstack.org/#/c/555081/22/specs/train/approved/cpu-resources.rst@181 | |
| 14:24:19 | bauzas | stephenfin: ^ | |
| 14:24:41 | bauzas | if we change the capacity for VGPU, then it could be a problem | |
| 14:24:56 | dansmith | stephenfin: another thing I'd support is converting instances to allocations that are maybe overly conservative.. like if you need to reserve more resources than they really have to make the math work out, that seems like a potential compromise | |
| 14:24:59 | bauzas | "I'll try to clarify my thoughts with an upgrade example on a host with 8 physical CPUs, named CPU0 to 7:in Stein, instances are running and actively taking VCPUs that are consuming CPU0 to CPU7.in Train, operator wants to dedicate CPU0 and CPU1. Accordingly, CPU2 to 7 will be shared.Consequently, VCPU inventory for that host will be reduced by the amount of allocation_ratio * 2. In theory, we should then allocate VCPU resource | |
| 14:24:59 | bauzas | lass for instances that are taking VCPUs located on CPU2 to 7 and allocate PCPU for instances that are taking CPU0 and 1. But if ratio is 16, we could have 32 instances (asking for 1 VCPU) to be allocated against 2 PCPU with ratio=1.0." | |
| 14:25:09 | dansmith | stephenfin: and then recalculate that on migrate if they want.. depending on how that looks | |
| 14:25:32 | dansmith | I dunno what the actual complexity concern looks like, so I'm just spitballing | |