| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 14:17:03 | sean-k-mooney | if you have no numa toplogy th enumber of enabeld cores in vcpu pinnset is still used to determin the number of cores reported to the resouce tracker and therefor to placemnt | |
| 14:17:13 | cdent | dansmith: make them end up on other nodes? | |
| 14:17:28 | stephenfin | We can't schedule new instances to that host until we know how many PCPUs are actually in use there | |
| 14:17:33 | dansmith | cdent: so I have to waste the capacity on those nodes until long-lived instances die? | |
| 14:17:46 | cdent | mriedem: yes, if that's how they were booted in the first place, how/why should they change | |
| 14:17:48 | dansmith | right, what stephenfin said | |
| 14:17:50 | cdent | dansmith: yes! | |
| 14:17:55 | dansmith | cdent: um, no | |
| 14:17:56 | sean-k-mooney | stephenfin: the advice that many gave was never to user reserved_cpus and alwauys use vcpu_pin_set instead | |
| 14:17:57 | mnaser | that's a terrible idea | |
| 14:18:04 | mnaser | tbh with my operators 2 cents: I'm not ok with moving around all my instances around to magically reshape things | |
| 14:18:09 | dansmith | this ^ | |
| 14:18:10 | mriedem | since i just went through mel's change to counting usage from placement, your quotas would be all out of whack too | |
| 14:18:18 | cdent | I'm not suggesting that people migrate their instances | |
| 14:18:22 | mnaser | and I'm not okay with my capacity sitting empty because this is $$$$ | |
| 14:18:24 | stephenfin | sean-k-mooney: Again, advice that wasn't enforced anywhere and therefore not something we can rely on :( | |
| 14:18:25 | dansmith | moving gigs and gigs of live instances so that we can adjust integers in placement is INSANE | |
| 14:18:26 | sean-k-mooney | the reason is vcpu_pin_set takes effect before teh cpu_allocation_reatio is applied and reserved_happens after | |
| 14:18:30 | cdent | meh | |
| 14:18:38 | cdent | I never suggested anybody do any migrations | |
| 14:18:41 | dansmith | reserving capacity until a five-year instance goes away so we can update integers in placement is INSANE | |
| 14:18:50 | cdent | Let stuff live out its lifecycle | |
| 14:18:56 | stephenfin | cdent: Yup, that's all me. Sorry :) | |
| 14:18:56 | mnaser | that's not possible though, that's the thing | |
| 14:19:00 | mnaser | I have no control over my environment | |
| 14:19:26 | cdent | dansmith: I'm not concerned about this from a placement standpoint: abuse placement all we want, it'll take it | |
| 14:19:41 | cdent | I'm concerned about it from a magical recognition happening on the compute node | |
| 14:19:43 | dansmith | cdent: yeah, this isn't a placement concern, it's a nova concern | |
| 14:20:05 | stephenfin | OK, so we're rewriting flavors and moving allocations around to switch everything to the new system. If that's the case, we're back to trying to think of all the edge cases that exist | |
| 14:20:11 | stephenfin | and there are many. Many many | |
| 14:20:33 | sean-k-mooney | stephenfin: we can support the new flow without requrieing flavor to be modifed | |
| 14:20:44 | dansmith | sean-k-mooney: he means the instance's flavor I think | |
| 14:20:46 | stephenfin | sean-k-mooney: via a shim, I guess? | |
| 14:20:57 | stephenfin | dansmith: I do. The embedded one | |
| 14:21:12 | sean-k-mooney | stephenfin: today we generate the palcement request via the request spec | |
| 14:21:15 | dansmith | stephenfin: you can write a nova-status check that verifies that all the instances can be fit to whatever minimally simplified scheme you support, | |
| 14:21:27 | dansmith | to warn people before they upgrade with complex instances that can't be fixed or something | |
| 14:21:28 | sean-k-mooney | we convert the VCPU element in the flavor in to a vcpu resouce request | |
| 14:22:00 | sean-k-mooney | that code can take account of the hw:cpu_policy extra spec and jsut assk for PCPU resouces instread | |
| 14:22:06 | stephenfin | dansmith: That's going to be a big check, fair warning :) | |
| 14:22:13 | dansmith | stephenfin: I'm just throwing ideas | |
| 14:22:32 | stephenfin | Yup, and good ones too | |
| 14:22:40 | dansmith | stephenfin: migrating everything, deleting everything, not upgrading until instances age out -- all not options, IMHO | |
| 14:22:55 | bauzas | dansmith: mnaser: okay, sorry, that's me who proposed migrating instances, and it was a terrible idea, I reckon | |
| 14:23:14 | bauzas | so we should stop thinking about this possibility | |
| 14:23:37 | mnaser | bauzas: all good :) ideas are good to bring up anyways | |
| 14:23:40 | bauzas | but, then, we want to just make sure that when creating a new RC, we also look at the existing capacity | |
| 14:24:12 | stephenfin | So supporting instances with just 'hw:cpu_policy=dedicated' in a deployment that has used aggregates as we suggest seems pretty easy | |
| 14:24:12 | bauzas | see the example I provided : https://review.openstack.org/#/c/555081/22/specs/train/approved/cpu-resources.rst@181 | |
| 14:24:19 | bauzas | stephenfin: ^ | |
| 14:24:41 | bauzas | if we change the capacity for VGPU, then it could be a problem | |
| 14:24:56 | dansmith | stephenfin: another thing I'd support is converting instances to allocations that are maybe overly conservative.. like if you need to reserve more resources than they really have to make the math work out, that seems like a potential compromise | |
| 14:24:59 | bauzas | "I'll try to clarify my thoughts with an upgrade example on a host with 8 physical CPUs, named CPU0 to 7:in Stein, instances are running and actively taking VCPUs that are consuming CPU0 to CPU7.in Train, operator wants to dedicate CPU0 and CPU1. Accordingly, CPU2 to 7 will be shared.Consequently, VCPU inventory for that host will be reduced by the amount of allocation_ratio * 2. In theory, we should then allocate VCPU resource | |
| 14:24:59 | bauzas | lass for instances that are taking VCPUs located on CPU2 to 7 and allocate PCPU for instances that are taking CPU0 and 1. But if ratio is 16, we could have 32 instances (asking for 1 VCPU) to be allocated against 2 PCPU with ratio=1.0." | |
| 14:25:09 | dansmith | stephenfin: and then recalculate that on migrate if they want.. depending on how that looks | |
| 14:25:32 | dansmith | I dunno what the actual complexity concern looks like, so I'm just spitballing | |
| 14:25:38 | stephenfin | dansmith: That might be necessary for something like 'hw:cpu_threads_policy=isolate' | |
| 14:25:43 | dansmith | yeah | |
| 14:27:03 | stephenfin | bauzas: Yeah, I think we need a startup check to ensure NUM_VCPUS_USED_BY_INSTANCE <= NUM_VCPUS_AVAILABLE | |
| 14:27:08 | stephenfin | *INSTANCES | |
| 14:27:15 | sean-k-mooney | well we say in the spec hw:cpu_threads_policy woudl be going away | |
| 14:27:42 | bauzas | stephenfin: cool with it then | |
| 14:27:55 | bauzas | stephenfin: but then we need to call placement when restarting the compute service | |
| 14:28:01 | stephenfin | sean-k-mooney: Yup, but there has to be an intermediate step that lets us account for the fact that existing instances are using more cores than instance.vcpus | |
| 14:28:07 | bauzas | *every time* | |
| 14:28:29 | bauzas | for vGPUs, we basically only reshape once | |
| 14:28:45 | stephenfin | bauzas: Hmm, could also be a nova-status check as dansmith suggested | |
| 14:28:46 | mnaser | what's the concern in taking the current state and translating that directly into placement when the compute node goes up? | |
| 14:28:58 | dansmith | mnaser: complexity | |
| 14:29:04 | dansmith | but we have to bite that bullet I think | |
| 14:29:15 | stephenfin | mnaser: there are a lot of ways things can be inconsistent and we need to handle those | |
| 14:29:22 | sean-k-mooney | stephenfin: that only happens for pinned instacnes if the hsot has hyper treading or you have emulartor_treads=isolate | |
| 14:29:39 | sean-k-mooney | stephenfin: but yes we do | |
| 14:29:44 | stephenfin | like the way there's nothing preventing you from scheduling pinned and unpinned instances on the same host | |
| 14:30:25 | mnaser | couldn't you introspect pinned and unpinned from the libvirt definition | |
| 14:30:41 | sean-k-mooney | mnaser: we can tell form the flavor | |
| 14:30:42 | stephenfin | (so when we migrate, we could end up in a situation where an N core host could have N PCPUs and N * overallocation_ratio VCPUs in use at the same time) | |
| 14:31:14 | sean-k-mooney | its not a case of we dont know this happens we told operators that its there respociblity to ensure it does not | |
| 14:31:35 | sean-k-mooney | that is the issue we told them to do somthing but did not enforec it in code | |
| 14:31:48 | sean-k-mooney | there for we have to assuem the worst | |
| 14:32:01 | stephenfin | or the fact that when using the isolate cpu thread policy, the instance may or may not be using twice as many cores as its supposed to be using (isolate will reserve the hyperthread siblings for each core used by the instance) | |
| 14:32:25 | stephenfin | sean-k-mooney: Correct | |
| 14:32:33 | sean-k-mooney | yes alther to be faire we do account for that properly in the resocue tracker | |
| 14:32:44 | sean-k-mooney | *although | |
| 14:32:50 | stephenfin | yup | |
| 14:33:13 | sean-k-mooney | mnaser: so we have all the data to fix things if we need to | |
| 14:33:26 | bauzas | stephenfin: we *could* do it with nova-status but then operators would have to migrate (or delete some instances) :( | |
| 14:33:32 | sean-k-mooney | the issue is that the isolate pollicy is not compatible with placcement | |
| 14:33:40 | bauzas | thanks, allocation ratio | |
| 14:33:43 | mriedem | lyarwood: can you hit these to keep things rolling https://review.openstack.org/#/q/topic:bug/1669054+branch:stable/rocky | |
| 14:33:50 | sean-k-mooney | it chagnes the quantity fo resocue based in the hsot that is selected | |
| 14:34:25 | sean-k-mooney | bauzas: no i think we can fix allocation for existing instance | |
| 14:34:46 | bauzas | sean-k-mooney: how ? see my example | |
| 14:34:54 | sean-k-mooney | the thing we have to be ok with is removing cpu_thread policies | |
| 14:35:12 | sean-k-mooney | bauzas: we can over allocate RPs if we need to initally | |
| 14:36:29 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: libvirt: disconnect volume when encryption fails https://review.openstack.org/651796 | |
| 14:36:46 | sean-k-mooney | or we can say you asked for 2 cpus but you have isolate and are artully using 4 cpus and update the placement allcoation accordingly | |
| 14:37:00 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/651797 | |
| 14:37:34 | stephenfin | mriedem: If you're looking at stable stuff, think you could look at these too? https://review.openstack.org/#/c/650363/ https://review.openstack.org/#/c/650364/ | |
| 14:37:39 | sean-k-mooney | with pinned cpus there was no over subscirtion so the fact the vm is there means it can fit and we correctly do the accounting in the resouce tracker to hanel the addtional cpu usage | |