| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-07 | |||
| 14:46:52 | sean-k-mooney | dansmith: we went with setting the flavor vlaues to 0 and just useing the custom resouce class | |
| 14:46:58 | dansmith | sean-k-mooney: no we didn't | |
| 14:47:07 | dansmith | sean-k-mooney: see that doc mriedem just posted | |
| 14:47:19 | mriedem | i think sean-k-mooney means the flavor values via the extra spec, | |
| 14:47:25 | mriedem | not the actual top-level flavor properties | |
| 14:47:30 | dansmith | I don't think he does :) | |
| 14:47:39 | stephenfin | sean-k-mooney: We don't touch the flavor properties. We just ignore them | |
| 14:47:45 | dansmith | because he described what we *are* doing as the original plan that we abandoned | |
| 14:47:55 | mriedem | i'm not sure you can even create a flavor with 0 vcpus or ram | |
| 14:48:25 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/flavor_manage.py#L33 | |
| 14:48:27 | sean-k-mooney | no https://docs.openstack.org/ironic/stein/install/configure-nova-flavors.html is exactly the use case i though this was for but i though we ended up not doing it | |
| 14:48:44 | sean-k-mooney | ok good to know that actully was done | |
| 14:48:52 | mriedem | stephenfin: you didn't realize this 6 months ago? | |
| 14:48:53 | efried | What we need is to start over with a flavorsv2 | |
| 14:49:09 | efried | How long could it possibly take to deprecate and remove cellsv1, I mean flavorsv1? | |
| 14:49:26 | dansmith | efried: not helping | |
| 14:49:38 | stephenfin | mriedem: I knew it was bad but not this bad /o\ I'd estimate a solid 80% of the code I've written has been tech debt-related | |
| 14:50:06 | mriedem | the base flavor values for vcpu/ram/disk are *everywhere* and in *everything* in nova, as far as resource accounting is concerned | |
| 14:50:12 | mriedem | so changing the meaning of one of those has huge implications | |
| 14:50:20 | dansmith | yup | |
| 14:50:23 | mriedem | the only one that was ever sort of special was disk for bfv | |
| 14:50:23 | sean-k-mooney | ya | |
| 14:50:29 | mriedem | which dansmith finally fixed in i think rocky | |
| 14:51:02 | mriedem | by fixed i mean don't claim disk usage if you're bfv | |
| 14:51:14 | mriedem | ec | |
| 14:51:14 | mriedem | and don't report to the hypervisors API that we're consuming disk for a bfv instance | |
| 14:51:16 | mriedem | *etc | |
| 14:51:41 | sean-k-mooney | i personally am not sure why we still allow bfv with root_gb !=0 i get the upgrade impact of that but it would be nice to fully lock that down | |
| 14:52:04 | sean-k-mooney | but that is a differnet hill to die on | |
| 14:52:32 | cdent | so many hills. so many deaths. | |
| 14:52:32 | sean-k-mooney | anyway for the vcpu stuff we never generate xml from the resouce:* values | |
| 14:52:44 | sean-k-mooney | we only use thos for placmenet | |
| 14:53:01 | mriedem | unless i'm mistaken, | |
| 14:53:08 | mriedem | until this pcpu hullabaloo, | |
| 14:53:18 | mriedem | the only time you'd ever override vcpu/ram/disk with extra specs is for ironic flavors | |
| 14:53:26 | mriedem | for which the driver doesn't look at those anymore anyway | |
| 14:53:42 | sean-k-mooney | no wiht the pcpu stuff we still dont plan to generate the xml form resouce: we plan to use the extra_specs | |
| 14:53:57 | mriedem | resource: *is* extra specs | |
| 14:54:09 | sean-k-mooney | but yes i think you are right the only time you would override them is ironic | |
| 14:54:09 | stephenfin | Yeah, my life has been made significantly harder by having to care about 'resources:VCPU' and 'resources:PCPU' | |
| 14:54:49 | stephenfin | I'm seriously considering a big "this isn't doing anything unless you're ironic" warning and moving on | |
| 14:54:51 | stephenfin | efried: ^ | |
| 14:54:54 | mriedem | https://review.opendev.org/#/c/637217/ | |
| 14:55:05 | stephenfin | There's edge case after edge case | |
| 14:55:13 | sean-k-mooney | mriedem: it is but i mean we have hw:cpu_policy to determin if we translate flavor.vcpu to resouce:vcpu or resouces:pcpu | |
| 14:55:26 | sean-k-mooney | stephenfin: the proably is how do you know its an ironic flavor | |
| 14:55:38 | mriedem | sean-k-mooney: if you're talking about things in the plan in the spec, i haven't read it so don't know | |
| 14:56:09 | efried | stephenfin: what specifically? resources:VCPU/PCPU in extra specs? | |
| 14:56:17 | sean-k-mooney | mriedem: we knew trying to supprot resouces: for influncing the xml generation and assignment woudl be a paid soe we are not supporting it | |
| 14:56:18 | stephenfin | mriedem: tl;dr: if you specify 'hw:cpu_policy=dedicated', you now get 'PCPU' inventory, but we need to care about 'resources:PCPU' in addition | |
| 14:56:28 | stephenfin | efried: yup | |
| 14:56:29 | sean-k-mooney | *pain | |
| 14:57:14 | sean-k-mooney | efried: i would expand it to all resocues generated form flavor.* but the problem is we have no way to know if its an ironic flavor or not | |
| 14:57:17 | mriedem | umm, couldn't you just bork if you have hw:cpu_policy=dedicated and resources:VCPU? | |
| 14:57:18 | sean-k-mooney | so we cant | |
| 14:57:51 | sean-k-mooney | we would still request resocue:PCU=flavor.vcpus in addtion to resources:VCPU | |
| 14:57:54 | mriedem | technically any ironic flavor is hw:cpu_policy=dedicated by default yeah? | |
| 14:58:13 | stephenfin | mriedem: Yup, and that's what I'm doing (ditto for explicit PCPU too). The top issue is flavor.vcpus != resources:PCPU | |
| 14:58:15 | mriedem | i mean you do'nt specify that, it's just implied b/c it's baremetal | |
| 14:58:37 | sean-k-mooney | stephenfin: there is one case where its not equal by design | |
| 14:58:38 | mriedem | you know, mogan wouldn't have this problem.... | |
| 14:58:56 | sean-k-mooney | which hw:emulator_threads_policy=isolate | |
| 14:59:19 | stephenfin | Also, 'hw:emulator_threads_policy=isolate' means "consume an extra core and put emulator threads there", so there it's (flavor.vcpus + 1) != resources:PCPU | |
| 14:59:23 | stephenfin | sean-k-mooney: jinx | |
| 14:59:40 | sean-k-mooney | mriedem: for ironic all cpus will still be reported using VCPU i think since we dont plan to update the driver | |
| 14:59:55 | sean-k-mooney | at least not unless someone asks | |
| 15:00:07 | sean-k-mooney | stephenfin: yes | |
| 15:00:22 | mriedem | mnaser: dansmith: looking at this trace again https://logs.opendev.org/33/660333/10/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/97d8bc3/logs/host/nova-compute.service.journal-23-20-40.log.txt.gz#_Aug_06_22_51_25 and where we hit it https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L422 , | |
| 15:00:36 | stephenfin | So this is all a massive PITA. TBH, I'd rather just say "don't use 'hw:cpu_policy' or any of those extra specs with 'resources:(P|V)CPU' and kick that can down the road | |
| 15:00:43 | stephenfin | efried: Could we do that for now? ^ | |
| 15:00:44 | sean-k-mooney | which if we kill that option entrily and just always map the emulator thread to the shared set in the futre will make your life simpeler | |
| 15:00:54 | mriedem | what do you think about adding a try/except for DBNotAllowed to just log something useful and re-raise, like "this service does not allow direct database access, you should run without the [api_database]/connection configured" | |
| 15:01:06 | stephenfin | sean-k-mooney: I'd be happy to do that but I need a cycle to do it | |
| 15:01:30 | stephenfin | Because we break everything otherwise, assuming the operator doesn't have cpu_shared_set configured everywhere | |
| 15:01:31 | sean-k-mooney | ya | |
| 15:01:50 | mriedem | sean-k-mooney: reported....where? | |
| 15:02:11 | mriedem | sean-k-mooney: for ironic the only inventory on the node should be the custom resource class reported by the driver's update_provider_tree method, which gets the resource class off the ironic node resource | |
| 15:02:14 | sean-k-mooney | mriedem: in placement | |
| 15:02:20 | sean-k-mooney | oh right | |
| 15:02:22 | mriedem | and the flavor should only consume, in placement, those custom resource class allocatoins | |
| 15:02:24 | sean-k-mooney | ya i forgot | |
| 15:02:31 | mriedem | that's why the vcpu/ram/disk are all zero'ed out | |
| 15:02:44 | mriedem | the values for those in the flavor are just cosmetic | |
| 15:02:57 | sean-k-mooney | so no change for ironic and hw:cpu_policy will have no effect | |
| 15:03:06 | sean-k-mooney | it should not be used but its meaning less | |
| 15:03:21 | mriedem | mnaser: if you have a bug for that osa issue i could tack on a nova logging patch to maybe make that more clear if someone hits that misconfig | |
| 15:04:37 | melwitt | mriedem: will do | |
| 15:05:00 | efried | stephenfin: I would be fine limiting the use of resources:(P|V)CPU as strictly as possible without breaking backward compat. | |
| 15:05:00 | mnaser | mriedem: i think noonedeadpunk will likely look into fixing it so might be good if he created a bug to track it ^ | |
| 15:05:18 | efried | but resources$S:* will be a different story. | |
| 15:05:44 | stephenfin | efried: What would the reason for 'resources$S:PCPU' be? | |
| 15:05:57 | stephenfin | I mean, without NUMA at least | |
| 15:06:07 | stephenfin | *NUMA in placement | |
| 15:06:58 | dansmith | mriedem: sure | |
| 15:07:21 | efried | stephenfin: Exactly NUMA in placement. | |
| 15:07:38 | efried | "will be" future tense intended | |
| 15:07:58 | sean-k-mooney | efried: why not block resources$S:* for pcus and vcpus | |
| 15:08:09 | efried | for now, yes, we certainly can ^ | |
| 15:08:15 | stephenfin | Why not forever? | |
| 15:08:17 | sean-k-mooney | we shoudl only support using the hw:numa_cpu.0 values | |