| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-09 | |||
| 14:33:53 | dansmith | s/everything/flavors/ | |
| 14:33:57 | jroll | I don't remember, sorry | |
| 14:34:15 | belmorei_ | jroll: that means updating all ironic nodes... can't do it. | |
| 14:34:43 | belmorei_ | How about if I remove the "resources:VCPU=0", ... from the flavors? | |
| 14:34:57 | mriedem | your bm flavors aren't the problem | |
| 14:35:11 | mriedem | the problem is the bm flavors are getting scheduled to the ironic nodes right? | |
| 14:35:13 | mriedem | *vm flavors | |
| 14:35:15 | dansmith | right | |
| 14:35:24 | jroll | that would prevent VMs from landing on an active ironic node, but not an inactive one | |
| 14:35:28 | mriedem | and that's because the bm nodes are reporting ram/cpu/disk inventory | |
| 14:35:47 | belmorei_ | jroll: true | |
| 14:39:22 | belmorei_ | ok, so for me the best is to backport the commit mentioned by mriedem | |
| 14:39:26 | mriedem | belmorei_: you might be able to just patch https://github.com/openstack/nova/blob/stable/queens/nova/virt/ironic/driver.py#L797 to be 0 | |
| 14:39:37 | belmorei_ | yeah, thanks | |
| 14:39:38 | mriedem | belmorei_: i'm guessing that's not going to backport cleanly to queens | |
| 14:39:47 | bauzas | gibi: thanks for the email recap | |
| 14:39:54 | belmorei_ | any plan to still include this in rocky? | |
| 14:40:10 | bauzas | gibi: I now understand the problem :) | |
| 14:40:12 | belmorei_ | because for me is a bug | |
| 14:40:24 | gibi | bauzas: :) | |
| 14:40:44 | mriedem | belmorei_: i'm not sure how others would feel about a stable/queens / rocky only config option to disable reporting vcpu/ram/disk inventory for ironic | |
| 14:40:52 | mriedem | a workarounds option | |
| 14:41:01 | gibi | mriedem, efried, edleafe: force flag means I want to move the server so desperately that I don't care about any safeties | |
| 14:41:10 | gibi | at least for me it means that | |
| 14:41:11 | mriedem | default to disabled, but if enabled, report 0 total vcpu/ram/disk inventory for ironic nodes | |
| 14:41:37 | mriedem | dansmith: ^ how do you feel about a config option backdoor for belmorei_'s case? | |
| 14:42:20 | mriedem | we're not going to backport https://github.com/openstack/nova/commit/a985e34cdeef777fe7ff943e363a5f1be6d991b7 | |
| 14:42:29 | dansmith | mriedem: I'm for it because I'm not sure how we're not effing people over with this right now | |
| 14:42:31 | belmorei_ | mriedem: this is easy for me to patch. Having a conf backdoor option doesn't seem good | |
| 14:42:39 | dansmith | I'm guessing tripleo people don't care because undercloud/overcloud | |
| 14:42:58 | mriedem | belmorei_: a workarounds config option provides a generic solution for *everyone* with this problem | |
| 14:43:22 | belmorei_ | mriedem: true | |
| 14:43:25 | mriedem | essentially it means enabling it says you've done your ironic instance flavor migration and you're good to go | |
| 14:43:35 | dansmith | right | |
| 14:43:37 | mriedem | and we have a nova-status check for that as well | |
| 14:43:57 | mriedem | belmorei_: how about you report a bug to start and we can go from there? | |
| 14:44:15 | mriedem | jroll: btw i do remember something breaking after we removed that code in stein, but i can't remember what off the top of my head | |
| 14:44:34 | mriedem | which is why i wanted to hold off on removing it right before the rocky GA | |
| 14:46:09 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't emit warning when ironic properties are zero https://review.openstack.org/608573 | |
| 14:46:11 | openstack | Launchpad bug 1787509 in OpenStack Compute (nova) "Baremetal filters and default filters cannot be used simultaneously in the same nova" [Undecided,Won't fix] | |
| 14:46:11 | jroll | mriedem: this? https://bugs.launchpad.net/nova/+bug/1787509 | |
| 14:46:40 | jroll | or maybe https://bugs.launchpad.net/tripleo/+bug/1787910/ | |
| 14:46:41 | openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | |
| 14:47:00 | belmorei_ | mriedem I will create the bug report | |
| 14:47:01 | mriedem | maybe | |
| 14:47:07 | belmorei_ | mriedem dansmith jroll thanks for the help | |
| 14:47:13 | mriedem | jroll: around the time we deprecated the core/ram/disk filters | |
| 14:47:51 | jroll | mriedem: yeah I don't remember what either, just going by irc logs | |
| 14:47:52 | openstack | Launchpad bug 1787910 in OpenStack Compute (nova) rocky "OVB overcloud deploy fails on nova placement errors" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | |
| 14:47:52 | mriedem | yeah https://bugs.launchpad.net/tripleo/+bug/1787910 | |
| 14:52:16 | dansmith | mriedem: so they still had ram required in those flavors and failed when ironic stopped reporting ram inventory | |
| 14:52:17 | dansmith | yeah? | |
| 14:52:44 | dansmith | we have to cut over at some point and I thought we already had.. workaround config flag to let them get over the hump seems like the best thing at this point | |
| 14:54:27 | mriedem | dansmith: they being tripleo in that bug? | |
| 14:54:36 | dansmith | well, ovb but yeah | |
| 14:55:05 | mriedem | yeah looks like it based on https://review.openstack.org/#/c/596093/ | |
| 15:05:57 | dtantsur | belmorei_: a workaround may be to remove memory_mb and vcpus from ironic nodes properties | |
| 15:06:27 | dtantsur | with something like $ openstack baremetal node unset <node> --property memory_mb | |
| 15:07:20 | dtantsur | mriedem: this may be a bit easier than hacking nova ^^ | |
| 15:08:02 | belmorei_ | dtantsur: thanks, but the problem is the number of baremetal nodes that we have. Also, we would need to change the commission procedure to include that. | |
| 15:08:38 | belmorei_ | for now I will just patch this in nova | |
| 15:08:56 | dtantsur | belmorei_: do you use something like inspection to populate these properties? | |
| 15:09:36 | dtantsur | also before resource classes I used to use host aggregates to more or less separate bm and vm nodes on the same nova | |
| 15:11:35 | belmorei_ | dtantsur: yes, inspection populate them | |
| 15:13:47 | mriedem | belmorei_: out of curiosity, before cells v2, could your vm flavors get scheduled to bm cells/ | |
| 15:13:48 | mriedem | ? | |
| 15:14:05 | mriedem | or were the flavors segregated at the top cell layer? | |
| 15:17:22 | mriedem | belmorei_: also fyi, you can't set total=0 for inventory on the resource class as i said above, placement api will reject that since total must be >=1 | |
| 15:17:36 | mriedem | so need to just omit posting those non-custom-resource-class inventories | |
| 15:20:42 | belmorei_ | mriedem: with cellsV1 we were using the baremetal filters, so they will not be schedule to an already deployed node. But yes, if a user used a vm flavor a think it would be the same (it would use the physical node to create the vm flavor instance) | |
| 15:21:34 | belmorei_ | mriedem: thanks for the heads up for the patch | |
| 15:32:40 | openstackgerrit | Martin Midolesov proposed openstack/nova master: vmware:PropertyCollector for caching instance properties https://review.openstack.org/608278 | |
| 15:33:35 | mriedem | dansmith: belmorei_: fyi i'm working on a rocky patch with the workaround option | |
| 15:37:34 | dansmith | mriedem: cool | |
| 15:39:22 | belmorei_ | mriedem: thanks | |
| 15:39:25 | belmorei_ | mriedem: https://bugs.launchpad.net/nova/+bug/1796920 | |
| 15:39:27 | openstack | Launchpad bug 1796920 in OpenStack Compute (nova) "Baremetal nodes should not be exposing non-custom-resource-class (vcpu, ram, disk)" [Undecided,New] | |
| 15:58:26 | mriedem | dansmith: looks like zuulv3 status something or other changed and now openstack-gerrit-dashboard is getting NoneType errors - you see the same? | |
| 15:59:08 | dansmith | mriedem: I noticed it was failing this morning but didn't go to look if zuul was down. usually that's the reason | |
| 15:59:32 | mriedem | i'm guessing API change http://zuul.openstack.org/status | |
| 15:59:43 | mriedem | not sure, but the dashboard is different | |
| 15:59:57 | dansmith | ah yeah | |
| 16:03:06 | imacdonn | dansmith: could you take a peek at this, please? https://review.openstack.org/608091 | |
| 16:08:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: [stable-only] Add report_ironic_standard_resource_class_inventory option https://review.openstack.org/609043 | |
| 16:08:50 | mriedem | dansmith: jroll: dtantsur: ^ belmiro took off....would be nice if he can confirm that fixes his problem | |
| 16:09:34 | jroll | thanks | |
| 16:10:44 | imacdonn | that's one long option name :) | |
| 16:10:56 | mriedem | suggestions welcome | |
| 16:11:16 | mriedem | i figured do_the_dew wouldn't be helpful | |
| 16:11:18 | dansmith | imacdonn: done | |
| 16:11:29 | dansmith | imacdonn: mriedem should look at that too | |
| 16:11:33 | dansmith | or rather | |
| 16:11:39 | mriedem | i did once.. | |
| 16:11:40 | dansmith | mriedem should look at and agree with me on that too | |
| 16:13:51 | imacdonn | I do see your point | |
| 16:14:26 | imacdonn | not sure if anyone is actually doing the "keep hammering on it until it concedes" approach, but yeah | |
| 16:14:37 | edmondsw | and that notification having the message is also important for PowerVC, since it has means to present errors from notifications in the PowerVC GUI | |
| 16:14:53 | dansmith | imacdonn: I expect everyone is | |
| 16:15:08 | edmondsw | oops, ignore ^, somehow jumped channels | |
| 16:15:39 | imacdonn | my suspicion is that some people are running it once, and missing the fact that there are failures, and maybe others are not running it at all | |