| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 14:05:31 | dansmith | maybe we need a hangout | |
| 14:05:31 | jaypipes | dansmith: yes. | |
| 14:05:32 | mriedem | b'migrate.exceptions.ScriptError: You can only have one Python script per version, but you have: /home/zuul/src/git.openstack.org/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/393_add_instances_hidden.py and /home/zuul/src/git.openstack.org/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/393_placeholder.py' | |
| 14:05:41 | stephenfin | dansmith: Initially, yeah, but at some point that entire field could go. I'm trying to find the place we discussed this in the spec previously (this is a big review) | |
| 14:05:46 | mriedem | oh nvm i need to rebase | |
| 14:05:54 | dansmith | stephenfin: I don't agree with you :) | |
| 14:06:18 | sean-k-mooney | stephenfin: i think it would be better to leave teh vcpu filed in the flavor as the total numaber of cpu and set the vcpu resouce class request to 0 instead in that instance | |
| 14:06:27 | mnaser | wait, why would we kill the cpu field | |
| 14:06:29 | stephenfin | dansmith: Then it's a good thing this isn't actually a stated goal of this particular spec | |
| 14:06:37 | sean-k-mooney | it does not break clients and achive the same goal | |
| 14:06:53 | stephenfin | Let's forget about that and move back to the original question of handling upgrades | |
| 14:07:20 | jaypipes | mnaser: because there's two different actual resource classes: dedicated (pinned) CPU and shared CPU. | |
| 14:07:31 | sean-k-mooney | i dont think haveing a paralle implemantins and inplace updates are mutally exclcive | |
| 14:07:38 | stephenfin | I've no idea how we can make resource claims for existing instances that currently don't have anything claimed | |
| 14:07:42 | jaypipes | mnaser: and the ugliness of our NUMA and pinning code has borked how we think of the CPU resources. | |
| 14:08:04 | sean-k-mooney | stephenfin: they do have clims but they are all of resocue class vcpu | |
| 14:08:26 | stephenfin | yeah, so moving those from VCPU to PCPU | |
| 14:08:29 | jaypipes | stephenfin: by looking at the flavor and image extra specs. | |
| 14:08:33 | sean-k-mooney | stephenfin: so we need to modify there exiting claimes inplace as part of the reshape | |
| 14:08:40 | stephenfin | the migration is going to be hell | |
| 14:08:44 | jaypipes | yuuup. | |
| 14:08:48 | jaypipes | always is. | |
| 14:08:56 | sean-k-mooney | will it be any worse then the vgpu reshape | |
| 14:08:59 | stephenfin | we're going to handle the stupid stuff that can happen now, like shared and dedicated instances being on the same host | |
| 14:09:01 | jaypipes | as I've mentioned before, upgrade path is about 95% of the code and effort. | |
| 14:09:39 | mriedem | i thought you couldn't have shared and dedicated on the same host today, or is that a 'recommendation' but not enforced | |
| 14:09:47 | mriedem | and we just hope people use aggregates for sanity? | |
| 14:09:49 | sean-k-mooney | stephenfin: by the way today the vms cant mix pinned and flaoting instace so the only things we have to migrate are the pinned isntance and then just need to change the resouce class | |
| 14:09:50 | stephenfin | mriedem: yeah, the latter | |
| 14:09:53 | jaypipes | mriedem: recommendation. there is no way to enforce it. | |
| 14:10:13 | stephenfin | mriedem: It's all over the docs but who reads those | |
| 14:10:40 | sean-k-mooney | windriver have some downwstream only hacks to make mixed stuff work | |
| 14:10:47 | sean-k-mooney | or i guess i should say starlingx | |
| 14:10:52 | mriedem | i know they have their crazy floating vcpu stuff in starlingx | |
| 14:10:54 | sean-k-mooney | but we shoudl not port that | |
| 14:11:09 | mriedem | i'm not sure there is no way we could've enforced it, | |
| 14:11:11 | stephenfin | sean-k-mooney: You mean you can't have pinned and floating instances on the same host? If so, you know that's not true | |
| 14:11:15 | sean-k-mooney | ya so they were only able to do that because we dont enforce that | |
| 14:11:21 | mriedem | e.g. if cpu_allocation_ratio=1.0, you have to have dedicated cpus | |
| 14:11:37 | stephenfin | Yeah, we don't so we have to handle that | |
| 14:11:42 | sean-k-mooney | stephenfin: no you can but you shouldnt | |
| 14:11:51 | sean-k-mooney | with out the starlinx hacks | |
| 14:12:03 | stephenfin | and because you can, we're going to have to handle that | |
| 14:12:06 | jaypipes | mriedem: cpu_allocation_ratio=1.0 has nothing to do with pinned CPUs. | |
| 14:12:16 | dansmith | well, | |
| 14:12:20 | dansmith | it's all over this spec | |
| 14:12:20 | jaypipes | (which is part of the problem) | |
| 14:12:29 | stephenfin | hence my inclination towards draining hosts and moving them to other, newly configured hosts | |
| 14:12:39 | stephenfin | *moving the instances | |
| 14:12:40 | mriedem | my point was we could have used that as indication a host can only have dedicated cpu guests | |
| 14:12:53 | mriedem | but since we didn't do that, yeah we could have mixed on the same host i guess and have to deal wit it | |
| 14:12:55 | mriedem | *with | |
| 14:12:55 | sean-k-mooney | cpu_allocation_ratio=1.0 jsut disables oversubsciption unfrotunetly | |
| 14:12:59 | dansmith | I'm -2 on making people move instances to update counting numbers in placement | |
| 14:13:31 | stephenfin | mriedem: yeah, cpu_allocation_ratio is ignored for pinned instances | |
| 14:13:36 | sean-k-mooney | anyway im personally not to worried about fixing the allocations. | |
| 14:13:36 | jaypipes | dansmith: especially when those instances are pets and pandas. all of them. | |
| 14:13:41 | dansmith | aye | |
| 14:13:42 | sean-k-mooney | i think we can do that | |
| 14:14:03 | stephenfin | so I'd imagine it's set to 16.0 for most deployments, regardless of the workload | |
| 14:14:18 | cdent | dansmith: I remain confused about why it isn't okay for those instance to remain defined/allocation in the "old way"? | |
| 14:14:28 | sean-k-mooney | stephenfin: it depense on the deployment tool | |
| 14:14:32 | stephenfin | yup | |
| 14:14:39 | bauzas | sorry folks, I had to go AFK | |
| 14:14:39 | dansmith | cdent: I don't think it is | |
| 14:14:43 | dansmith | cdent: we have to reshape | |
| 14:14:58 | cdent | dansmith: I hear you, I'm asking "why?" | |
| 14:15:08 | bauzas | dansmith: my point is that I think we only need to reshape for CONF.vcpu_pin_set | |
| 14:15:18 | bauzas | for the other options, we don't need it | |
| 14:15:30 | sean-k-mooney | vcpu_pin_set is used for host with shared cpus too | |
| 14:15:31 | bauzas | stephenfin: ^ | |
| 14:15:37 | dansmith | cdent: why not just leave them with the old accounting for five years? | |
| 14:15:44 | bauzas | sean-k-mooney: I know, that's why we need to reshape | |
| 14:15:50 | cdent | if that's how long they live, sure | |
| 14:15:55 | bauzas | but only for this option | |
| 14:16:06 | stephenfin | sean-k-mooney: Is it? I thought that was totally ignored unless you'd a NUMA topology | |
| 14:16:13 | sean-k-mooney | yes | |
| 14:16:19 | sean-k-mooney | but you can have numa without pinning | |
| 14:16:20 | cdent | dansmith: I'm not suggesting it, I'm asking why it is not okay. | |
| 14:16:27 | sean-k-mooney | and actully no | |
| 14:16:31 | stephenfin | Instead you've to use that reserved_cpus option (or whatever it's called) | |
| 14:16:37 | dansmith | cdent: because it means new instances scheduled to compute nodes that don't have proper accounting would also have to be accounted the old way? | |
| 14:16:55 | stephenfin | cdent: Yeah, that's what I was thinking to ^ | |
| 14:16:57 | mriedem | cdent: by "remain defined/allocation in the "old way"?" do you mean reporting VCPU allocations to placement rather than PCPU? | |
| 14:16:57 | stephenfin | *too | |
| 14:17:03 | sean-k-mooney | if you have no numa toplogy th enumber of enabeld cores in vcpu pinnset is still used to determin the number of cores reported to the resouce tracker and therefor to placemnt | |
| 14:17:13 | cdent | dansmith: make them end up on other nodes? | |
| 14:17:28 | stephenfin | We can't schedule new instances to that host until we know how many PCPUs are actually in use there | |
| 14:17:33 | dansmith | cdent: so I have to waste the capacity on those nodes until long-lived instances die? | |
| 14:17:46 | cdent | mriedem: yes, if that's how they were booted in the first place, how/why should they change | |
| 14:17:48 | dansmith | right, what stephenfin said | |
| 14:17:50 | cdent | dansmith: yes! | |
| 14:17:55 | dansmith | cdent: um, no | |
| 14:17:56 | sean-k-mooney | stephenfin: the advice that many gave was never to user reserved_cpus and alwauys use vcpu_pin_set instead | |
| 14:17:57 | mnaser | that's a terrible idea | |
| 14:18:04 | mnaser | tbh with my operators 2 cents: I'm not ok with moving around all my instances around to magically reshape things | |
| 14:18:09 | dansmith | this ^ | |
| 14:18:10 | mriedem | since i just went through mel's change to counting usage from placement, your quotas would be all out of whack too | |
| 14:18:18 | cdent | I'm not suggesting that people migrate their instances | |
| 14:18:22 | mnaser | and I'm not okay with my capacity sitting empty because this is $$$$ | |