| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-09 | |||
| 14:17:43 | belmorei_ | hi. I'm having an issue with Ironic allocations in placement | |
| 14:18:58 | belmorei_ | What I see is that allocations are only done for the resource class, and not vcpus, ram, disk. I imagine this intentional? | |
| 14:18:58 | belmorei_ | We define the Ironic flavors with resources:VCPU=0, ... as documented in "https://docs.openstack.org/ironic/queens/install/configure-nova-flavors.html". | |
| 14:20:10 | efried | what do you mean, only done for the resource class? | |
| 14:20:43 | belmorei_ | the allocation for the resource_provider is only the ironic resource_class | |
| 14:20:54 | dansmith | belmorei_: yes, expected | |
| 14:21:22 | belmorei_ | great. Let me describe the issue | |
| 14:24:13 | belmorei_ | The problem is that we enable the ironic flavors per project (these projects are only for baremetal) (projects are mapped to a cell that only has the baremetal nodes) | |
| 14:24:59 | belmorei_ | However, users also have access to the "default" flavors that are for VMs in these projects (we can't remove public flavors) | |
| 14:26:37 | belmorei_ | If a user makes the mistake to use a default flavor in these projects (flavor for VMs) placement can return already in use baremetal nodes because they have cpu, ram, ... | |
| 14:27:30 | dansmith | belmorei_: the baremetal nodes should be exposing no cpu,ram,etc inventory | |
| 14:27:40 | dansmith | belmorei_: they should expose one inventory item of the baremetal resource class and nothing else | |
| 14:28:43 | belmorei_ | dansmith: Good to know I would expect that, but it's not happening. Maybe a conf issue in my side | |
| 14:29:10 | dansmith | belmorei_: yeah, I'm not sure how that could be happening anymore.. jroll dtantsur ? | |
| 14:29:34 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/virt/ironic/driver.py#L790 | |
| 14:29:47 | mriedem | we still reported cpu/ram/disk inventory for ironic nodes in queens | |
| 14:30:26 | mriedem | https://github.com/openstack/nova/commit/a985e34cdeef777fe7ff943e363a5f1be6d991b7#diff-1e4547e2c3b36b8f836d8f851f85fde7 removed that in stein | |
| 14:31:33 | dansmith | mriedem: um, we should have had a cutover so we're not exposing both right? | |
| 14:31:39 | dansmith | I thought that was like pike | |
| 14:31:57 | dansmith | comment there says zero in pike | |
| 14:31:58 | jroll | dansmith: we didn't do the cutover so people could migrate their flavor | |
| 14:32:08 | dansmith | jroll: right, but in queens? | |
| 14:32:19 | jroll | this is the first I've heard of this, fwiw, though it makes sense | |
| 14:32:24 | jroll | dansmith: people were scared to remove it | |
| 14:32:50 | dansmith | hmm | |
| 14:33:01 | jroll | a quick workaround would likely be to set 0 for cpu/ram in ironic node.properties | |
| 14:33:14 | dansmith | I thought this was long since sorted | |
| 14:33:14 | jroll | and then it will report 0 | |
| 14:33:24 | jroll | yeah, I thought it just worked as well | |
| 14:33:39 | dansmith | what was the plan during the overlap to avoid exposing the old values once everything was migrated? | |
| 14:33:47 | dansmith | override properties in ironic? | |
| 14:33:53 | dansmith | s/everything/flavors/ | |
| 14:33:57 | jroll | I don't remember, sorry | |
| 14:34:15 | belmorei_ | jroll: that means updating all ironic nodes... can't do it. | |
| 14:34:43 | belmorei_ | How about if I remove the "resources:VCPU=0", ... from the flavors? | |
| 14:34:57 | mriedem | your bm flavors aren't the problem | |
| 14:35:11 | mriedem | the problem is the bm flavors are getting scheduled to the ironic nodes right? | |
| 14:35:13 | mriedem | *vm flavors | |
| 14:35:15 | dansmith | right | |
| 14:35:24 | jroll | that would prevent VMs from landing on an active ironic node, but not an inactive one | |
| 14:35:28 | mriedem | and that's because the bm nodes are reporting ram/cpu/disk inventory | |
| 14:35:47 | belmorei_ | jroll: true | |
| 14:39:22 | belmorei_ | ok, so for me the best is to backport the commit mentioned by mriedem | |
| 14:39:26 | mriedem | belmorei_: you might be able to just patch https://github.com/openstack/nova/blob/stable/queens/nova/virt/ironic/driver.py#L797 to be 0 | |
| 14:39:37 | belmorei_ | yeah, thanks | |
| 14:39:38 | mriedem | belmorei_: i'm guessing that's not going to backport cleanly to queens | |
| 14:39:47 | bauzas | gibi: thanks for the email recap | |
| 14:39:54 | belmorei_ | any plan to still include this in rocky? | |
| 14:40:10 | bauzas | gibi: I now understand the problem :) | |
| 14:40:12 | belmorei_ | because for me is a bug | |
| 14:40:24 | gibi | bauzas: :) | |
| 14:40:44 | mriedem | belmorei_: i'm not sure how others would feel about a stable/queens / rocky only config option to disable reporting vcpu/ram/disk inventory for ironic | |
| 14:40:52 | mriedem | a workarounds option | |
| 14:41:01 | gibi | mriedem, efried, edleafe: force flag means I want to move the server so desperately that I don't care about any safeties | |
| 14:41:10 | gibi | at least for me it means that | |
| 14:41:11 | mriedem | default to disabled, but if enabled, report 0 total vcpu/ram/disk inventory for ironic nodes | |
| 14:41:37 | mriedem | dansmith: ^ how do you feel about a config option backdoor for belmorei_'s case? | |
| 14:42:20 | mriedem | we're not going to backport https://github.com/openstack/nova/commit/a985e34cdeef777fe7ff943e363a5f1be6d991b7 | |
| 14:42:29 | dansmith | mriedem: I'm for it because I'm not sure how we're not effing people over with this right now | |
| 14:42:31 | belmorei_ | mriedem: this is easy for me to patch. Having a conf backdoor option doesn't seem good | |
| 14:42:39 | dansmith | I'm guessing tripleo people don't care because undercloud/overcloud | |
| 14:42:58 | mriedem | belmorei_: a workarounds config option provides a generic solution for *everyone* with this problem | |
| 14:43:22 | belmorei_ | mriedem: true | |
| 14:43:25 | mriedem | essentially it means enabling it says you've done your ironic instance flavor migration and you're good to go | |
| 14:43:35 | dansmith | right | |
| 14:43:37 | mriedem | and we have a nova-status check for that as well | |
| 14:43:57 | mriedem | belmorei_: how about you report a bug to start and we can go from there? | |
| 14:44:15 | mriedem | jroll: btw i do remember something breaking after we removed that code in stein, but i can't remember what off the top of my head | |
| 14:44:34 | mriedem | which is why i wanted to hold off on removing it right before the rocky GA | |
| 14:46:09 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't emit warning when ironic properties are zero https://review.openstack.org/608573 | |
| 14:46:11 | openstack | Launchpad bug 1787509 in OpenStack Compute (nova) "Baremetal filters and default filters cannot be used simultaneously in the same nova" [Undecided,Won't fix] | |
| 14:46:11 | jroll | mriedem: this? https://bugs.launchpad.net/nova/+bug/1787509 | |
| 14:46:40 | jroll | or maybe https://bugs.launchpad.net/tripleo/+bug/1787910/ | |
| 14:46:41 | openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | |
| 14:47:00 | belmorei_ | mriedem I will create the bug report | |
| 14:47:01 | mriedem | maybe | |
| 14:47:07 | belmorei_ | mriedem dansmith jroll thanks for the help | |
| 14:47:13 | mriedem | jroll: around the time we deprecated the core/ram/disk filters | |
| 14:47:51 | jroll | mriedem: yeah I don't remember what either, just going by irc logs | |
| 14:47:52 | openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | |
| 14:47:52 | mriedem | yeah https://bugs.launchpad.net/tripleo/+bug/1787910 | |
| 14:52:16 | dansmith | mriedem: so they still had ram required in those flavors and failed when ironic stopped reporting ram inventory | |
| 14:52:17 | dansmith | yeah? | |
| 14:52:44 | dansmith | we have to cut over at some point and I thought we already had.. workaround config flag to let them get over the hump seems like the best thing at this point | |
| 14:54:27 | mriedem | dansmith: they being tripleo in that bug? | |
| 14:54:36 | dansmith | well, ovb but yeah | |
| 14:55:05 | mriedem | yeah looks like it based on https://review.openstack.org/#/c/596093/ | |
| 15:05:57 | dtantsur | belmorei_: a workaround may be to remove memory_mb and vcpus from ironic nodes properties | |
| 15:06:27 | dtantsur | with something like $ openstack baremetal node unset <node> --property memory_mb | |
| 15:07:20 | dtantsur | mriedem: this may be a bit easier than hacking nova ^^ | |
| 15:08:02 | belmorei_ | dtantsur: thanks, but the problem is the number of baremetal nodes that we have. Also, we would need to change the commission procedure to include that. | |
| 15:08:38 | belmorei_ | for now I will just patch this in nova | |
| 15:08:56 | dtantsur | belmorei_: do you use something like inspection to populate these properties? | |
| 15:09:36 | dtantsur | also before resource classes I used to use host aggregates to more or less separate bm and vm nodes on the same nova | |
| 15:11:35 | belmorei_ | dtantsur: yes, inspection populate them | |
| 15:13:47 | mriedem | belmorei_: out of curiosity, before cells v2, could your vm flavors get scheduled to bm cells/ | |
| 15:13:48 | mriedem | ? | |
| 15:14:05 | mriedem | or were the flavors segregated at the top cell layer? | |
| 15:17:22 | mriedem | belmorei_: also fyi, you can't set total=0 for inventory on the resource class as i said above, placement api will reject that since total must be >=1 | |
| 15:17:36 | mriedem | so need to just omit posting those non-custom-resource-class inventories | |