| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 14:30:42 | stephenfin | (so when we migrate, we could end up in a situation where an N core host could have N PCPUs and N * overallocation_ratio VCPUs in use at the same time) | |
| 14:31:14 | sean-k-mooney | its not a case of we dont know this happens we told operators that its there respociblity to ensure it does not | |
| 14:31:35 | sean-k-mooney | that is the issue we told them to do somthing but did not enforec it in code | |
| 14:31:48 | sean-k-mooney | there for we have to assuem the worst | |
| 14:32:01 | stephenfin | or the fact that when using the isolate cpu thread policy, the instance may or may not be using twice as many cores as its supposed to be using (isolate will reserve the hyperthread siblings for each core used by the instance) | |
| 14:32:25 | stephenfin | sean-k-mooney: Correct | |
| 14:32:33 | sean-k-mooney | yes alther to be faire we do account for that properly in the resocue tracker | |
| 14:32:44 | sean-k-mooney | *although | |
| 14:32:50 | stephenfin | yup | |
| 14:33:13 | sean-k-mooney | mnaser: so we have all the data to fix things if we need to | |
| 14:33:26 | bauzas | stephenfin: we *could* do it with nova-status but then operators would have to migrate (or delete some instances) :( | |
| 14:33:32 | sean-k-mooney | the issue is that the isolate pollicy is not compatible with placcement | |
| 14:33:40 | bauzas | thanks, allocation ratio | |
| 14:33:43 | mriedem | lyarwood: can you hit these to keep things rolling https://review.openstack.org/#/q/topic:bug/1669054+branch:stable/rocky | |
| 14:33:50 | sean-k-mooney | it chagnes the quantity fo resocue based in the hsot that is selected | |
| 14:34:25 | sean-k-mooney | bauzas: no i think we can fix allocation for existing instance | |
| 14:34:46 | bauzas | sean-k-mooney: how ? see my example | |
| 14:34:54 | sean-k-mooney | the thing we have to be ok with is removing cpu_thread policies | |
| 14:35:12 | sean-k-mooney | bauzas: we can over allocate RPs if we need to initally | |
| 14:36:29 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: libvirt: disconnect volume when encryption fails https://review.openstack.org/651796 | |
| 14:36:46 | sean-k-mooney | or we can say you asked for 2 cpus but you have isolate and are artully using 4 cpus and update the placement allcoation accordingly | |
| 14:37:00 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/651797 | |
| 14:37:34 | stephenfin | mriedem: If you're looking at stable stuff, think you could look at these too? https://review.openstack.org/#/c/650363/ https://review.openstack.org/#/c/650364/ | |
| 14:37:39 | sean-k-mooney | with pinned cpus there was no over subscirtion so the fact the vm is there means it can fit and we correctly do the accounting in the resouce tracker to hanel the addtional cpu usage | |
| 14:38:28 | mriedem | stephenfin: ok | |
| 14:38:33 | stephenfin | thanks | |
| 14:38:42 | stephenfin | bauzas: You would, but is there anyway to work around that? | |
| 14:39:01 | stephenfin | I mean, if they're in a broken state, something has to change | |
| 14:39:21 | stephenfin | bauzas: Also, wouldn't this exact same thing happen now if you messed with allocation ratios? | |
| 14:40:07 | stephenfin | i.e. If there are already instances on a host and I drop cpu_allocation_ratio from 16.0 to 2.0 and restart nova-compute, what happens? | |
| 14:40:11 | bauzas | well, I dunno what to say | |
| 14:40:20 | bauzas | stephenfin: it just works | |
| 14:40:30 | sean-k-mooney | in the placement side the ratio changes | |
| 14:40:37 | bauzas | stephenfin: but any other instance request would not go to this compute | |
| 14:40:49 | sean-k-mooney | if you are using more then is available you can nolonger allocate untill you drop below the new limit | |
| 14:40:55 | sean-k-mooney | but nothing breaks | |
| 14:40:59 | stephenfin | So it'd be the same here, right? | |
| 14:41:01 | bauzas | exactly what I said :) | |
| 14:41:07 | stephenfin | Or am I missing something? | |
| 14:41:07 | sean-k-mooney | it just prevent new instance going to the node | |
| 14:41:09 | bauzas | actually, that's a good point | |
| 14:41:28 | sean-k-mooney | yes it would be the same | |
| 14:41:30 | bauzas | if the host is oversubscribed, that's fine | |
| 14:41:40 | bauzas | it's just the options mean nothing | |
| 14:43:10 | sean-k-mooney | ok i hate myself for saying this but can we seperate hw:cpu_thread_policy into antoher spec for the removal of that option? | |
| 14:44:13 | sean-k-mooney | it can be replaced with a trait for host with SMT enabled | |
| 14:44:34 | sean-k-mooney | if we agree on that then that one less thing we need to figure out in the cpu spec | |
| 14:46:04 | sean-k-mooney | we will be loosing functionality by doing that but if are not ok with removing that option we have a blocker with the larger spec for cpus in placment anyway | |
| 14:46:09 | stephenfin | sean-k-mooney: Not _really_. I mean, 'isolate' results in extra cores being used and those have to be account for somehow | |
| 14:46:33 | sean-k-mooney | stephenfin: they are in the resouce tracker | |
| 14:46:47 | dansmith | sean-k-mooney: you can't really remove that image property | |
| 14:47:00 | dansmith | you can translate it into something more sane, but it's basically API at this point | |
| 14:47:38 | sean-k-mooney | i personally see value in it but its cause huge issues for cpu in plament | |
| 14:47:42 | dansmith | if you just start ignoring that, everyone's tooling is going to start spinning up instances they think are isolated (or whatever) but arent' and they'll find out when it's too late | |
| 14:47:54 | sean-k-mooney | ya i know | |
| 14:48:05 | sean-k-mooney | we can certely translate it | |
| 14:48:21 | stephenfin | dansmith: The migration path we'd suggested was keeping that but limiting it to hosts with "I don't have hyperthreads" trait set | |
| 14:48:23 | stephenfin | I think | |
| 14:48:25 | sean-k-mooney | to forbined:trait=COMPUTE_SMT | |
| 14:48:36 | stephenfin | Wait, yeah, that ^ | |
| 14:49:05 | dansmith | stephenfin: that's cool, it just can't like .. be removed, like sean-k-mooney was saying :) | |
| 14:49:22 | sean-k-mooney | removed was a bad phasing | |
| 14:49:26 | stephenfin | dansmith: True. We can think about deprecating it in the future though | |
| 14:49:28 | sean-k-mooney | its meaning would change | |
| 14:49:34 | dansmith | I don't | |
| 14:49:34 | stephenfin | It's that or we carry the shim forever | |
| 14:49:35 | dansmith | it's API | |
| 14:49:52 | stephenfin | We deprecate/remove other APIs though? | |
| 14:49:54 | dansmith | you can fail the boot if it's specified as anything if you want | |
| 14:50:04 | dansmith | stephenfin: this is unversioned | |
| 14:50:14 | dansmith | but I think we have to check for it basically forever | |
| 14:50:23 | dansmith | it's also API that's unversioned and spread between nova, glance and cinder | |
| 14:50:37 | stephenfin | Hmm, good point | |
| 14:50:56 | dansmith | I'm not saying you have to honor it well, with a shim forever, | |
| 14:50:59 | sean-k-mooney | ok does it help to split that bit out into another spec | |
| 14:51:02 | dansmith | but you can't ever just start ignoring it, IMHO | |
| 14:51:15 | stephenfin | ok, let's kick that can down the road | |
| 14:51:44 | stephenfin | for now, I quite like the idea of overallocating the PCPUs for existing instances with the 'isolate' policy | |
| 14:51:45 | sean-k-mooney | well its a prequisit for dedicated cpus in placmente | |
| 14:51:45 | dansmith | the only reason not to separate it is if it's a problem for your current proposal, like you can't continue to honor it as is after you make other changes | |
| 14:51:54 | dansmith | but if that's not a problem, then sure | |
| 14:52:07 | sean-k-mooney | well it is a proablem for the current proposal | |
| 14:52:12 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | |
| 14:52:16 | dansmith | then we can't separate it completely | |
| 14:52:17 | stephenfin | sean-k-mooney: Yeah, I don't think we can split it out entirely | |
| 14:52:36 | dansmith | I've been on a call for the last hour, and have another starting soon that I actually have to pay attention to, FYI | |
| 14:52:42 | stephenfin | We need to figure out what happens right now, once we have PCPUs in placement | |
| 14:52:48 | dansmith | so don't assume my pending silence on this matter is because I have shot myself in the face | |
| 14:53:34 | stephenfin | What we don't need to figure out now is what we're doing even further down the road (in terms of failing the instance if the image property is set or something else) | |
| 14:53:40 | mriedem | bauzas: you know how we reset_forced_destinations on the request spec when moving a server? | |
| 14:53:43 | stephenfin | dansmith: Ack, me too | |
| 14:53:48 | sean-k-mooney | stephenfin: well the traits thing is mentioned here https://review.openstack.org/#/c/555081/22/specs/train/approved/cpu-resources.rst@778 | |
| 14:53:51 | dansmith | stephenfin: yes, I think you can punt on that part | |
| 14:54:25 | mriedem | bauzas: if i create a server with a query scheduler hint targeted at a host or hypervisor_hostname, i can never migrate my server off the host :) same problem - but dumber | |
| 14:54:30 | stephenfin | sean-k-mooney: Oh, indeed it is. I just need to expand on that I guess | |
| 14:54:41 | stephenfin | OK, let me try and jot all this down in the spec and clean up the other issues | |
| 14:55:19 | sean-k-mooney | so its litrally already part of the spec. the only fuctionality that you loose with tah it you can nolonger use isolate to allow a host with hyperthreads to be shared between guests that want full coures and thost that can just have threads | |
| 14:56:03 | sean-k-mooney | stephenfin: the code isnt the problem with this sepc | |
| 14:56:37 | sean-k-mooney | stephenfin: the upgrade impact is and it think we can move forward with this spec but the current spec still has upgrade issues | |
| 14:57:31 | sean-k-mooney | stephenfin: i share dansmith view that we shoudl enable inplace upgrades to this new way of doing things if we can do that i will be happy with this | |