| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-09 | |||
| 14:02:08 | mriedem | i can field this one... | |
| 14:02:21 | mriedem | efried: before the force flag, specifying a host at all bypassed the scheduler, | |
| 14:02:49 | mriedem | then a microversion was added which made passing a host go through the scheduler for validation, but apparently at least one person though we should preserve the ability to bypass the scheduler, so the force flag was added to do that backdoor | |
| 14:03:11 | efried | What was the motivation to "bypass the scheduler"? Because it was inefficient? | |
| 14:03:53 | mriedem | idk, i'm assuming to oversubscribe a host just to move things around | |
| 14:03:58 | mriedem | at least temporarily | |
| 14:04:10 | efried | is oversubscribe even possible at this point? | |
| 14:04:23 | efried | (Outside of allocation ratio, which doesn't count) | |
| 14:04:37 | mriedem | i don't think so, at least not for vcpu/ram/dis | |
| 14:04:38 | mriedem | *disk | |
| 14:04:52 | mriedem | as i noted in gibi's patch - we broke that in pike when force still goes through claiming resource allocations in conductor | |
| 14:04:54 | sean-k-mooney | efried: if it bypassed the scduer it proably bypassed the placement claim too but i have never check that | |
| 14:05:05 | mriedem | sean-k-mooney: incorrect | |
| 14:05:28 | efried | Yeah, that's the point. Since we started claiming from placement, you can't oversubscribe. | |
| 14:05:35 | mriedem | efried: in the old days, before placement, if you bypass the scheduler, conductor would not send limits down to the RT so it wouldn't fail the limits check on the resource claim | |
| 14:05:46 | sean-k-mooney | mriedem: was that unintentional however as you jsut said we "broke" that in pike | |
| 14:05:59 | efried | has anyone screamed about that breakage? | |
| 14:06:13 | efried | Or do we still not have enough serious operators on pike yet? :P | |
| 14:06:17 | mriedem | there are like 3 people i now on >= pike but no... | |
| 14:06:47 | mriedem | *know | |
| 14:06:51 | edleafe | efried: one reason for the force option was that admins wanted to be able to say "I know what I'm doing, dammit!". It wasn't about code efficiency | |
| 14:07:17 | efried | edleafe: But IIUC, the destination host is observed regardless. | |
| 14:07:32 | mriedem | observed? | |
| 14:07:41 | efried | meaning you either get on the suggested host or you die | |
| 14:07:50 | mriedem | correct | |
| 14:07:51 | efried | I'm not sure how we would "fix" the oversubscribe thing at this point, without adding a placement feature to allow it. | |
| 14:08:22 | mriedem | i don't think adding features to support shooting yourself is something we want to do at this point | |
| 14:08:22 | efried | which I doubt we want to do | |
| 14:08:40 | efried | "Hey placement, you know that one thing you're supposed to be designed to do? Yeah, don't do that." | |
| 14:08:42 | edleafe | efried: it isn't just oversubscription that would cause the scheduler to reject it. Things like affinity, etc., would also cause it to fail | |
| 14:08:52 | mriedem | note that even with forced live migration, conductor still runs some checks that could cause us to reject the host | |
| 14:09:05 | mriedem | edleafe: nope | |
| 14:09:12 | efried | you mean besides the allocation claim? | |
| 14:09:12 | mriedem | we don't do any affinity checks outside of the scheduler for live migration | |
| 14:09:30 | edleafe | mriedem: without the force option, we do | |
| 14:09:36 | mriedem | in the scheduler | |
| 14:10:01 | mriedem | but that's more to my point - force is bad | |
| 14:10:15 | efried | I think what I'm getting at is, force isn't so much bad as... obsolete? | |
| 14:10:20 | mriedem | you could screw up AZs too | |
| 14:10:22 | efried | Like, it doesn't do anything useful anymore. | |
| 14:10:24 | edleafe | well, yeah, but admins wanted to be able to override | |
| 14:10:28 | mriedem | efried: agree | |
| 14:10:35 | edleafe | I'm not saying it's right | |
| 14:10:38 | mriedem | efried: it made a bit more sense pre-placement | |
| 14:10:42 | edleafe | Just that that was the pushback at the time | |
| 14:10:46 | efried | because oversubscribe | |
| 14:11:11 | efried | Okay, thanks, this helps validate my response to gibi's thread. | |
| 14:13:47 | openstackgerrit | Merged openstack/os-vif master: Remove IPTools deprecated implementation https://review.openstack.org/605422 | |
| 14:15:51 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type https://review.openstack.org/572081 | |
| 14:17:43 | belmorei_ | hi. I'm having an issue with Ironic allocations in placement | |
| 14:18:58 | belmorei_ | We define the Ironic flavors with resources:VCPU=0, ... as documented in "https://docs.openstack.org/ironic/queens/install/configure-nova-flavors.html". | |
| 14:18:58 | belmorei_ | What I see is that allocations are only done for the resource class, and not vcpus, ram, disk. I imagine this intentional? | |
| 14:20:10 | efried | what do you mean, only done for the resource class? | |
| 14:20:43 | belmorei_ | the allocation for the resource_provider is only the ironic resource_class | |
| 14:20:54 | dansmith | belmorei_: yes, expected | |
| 14:21:22 | belmorei_ | great. Let me describe the issue | |
| 14:24:13 | belmorei_ | The problem is that we enable the ironic flavors per project (these projects are only for baremetal) (projects are mapped to a cell that only has the baremetal nodes) | |
| 14:24:59 | belmorei_ | However, users also have access to the "default" flavors that are for VMs in these projects (we can't remove public flavors) | |
| 14:26:37 | belmorei_ | If a user makes the mistake to use a default flavor in these projects (flavor for VMs) placement can return already in use baremetal nodes because they have cpu, ram, ... | |
| 14:27:30 | dansmith | belmorei_: the baremetal nodes should be exposing no cpu,ram,etc inventory | |
| 14:27:40 | dansmith | belmorei_: they should expose one inventory item of the baremetal resource class and nothing else | |
| 14:28:43 | belmorei_ | dansmith: Good to know I would expect that, but it's not happening. Maybe a conf issue in my side | |
| 14:29:10 | dansmith | belmorei_: yeah, I'm not sure how that could be happening anymore.. jroll dtantsur ? | |
| 14:29:34 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/virt/ironic/driver.py#L790 | |
| 14:29:47 | mriedem | we still reported cpu/ram/disk inventory for ironic nodes in queens | |
| 14:30:26 | mriedem | https://github.com/openstack/nova/commit/a985e34cdeef777fe7ff943e363a5f1be6d991b7#diff-1e4547e2c3b36b8f836d8f851f85fde7 removed that in stein | |
| 14:31:33 | dansmith | mriedem: um, we should have had a cutover so we're not exposing both right? | |
| 14:31:39 | dansmith | I thought that was like pike | |
| 14:31:57 | dansmith | comment there says zero in pike | |
| 14:31:58 | jroll | dansmith: we didn't do the cutover so people could migrate their flavor | |
| 14:32:08 | dansmith | jroll: right, but in queens? | |
| 14:32:19 | jroll | this is the first I've heard of this, fwiw, though it makes sense | |
| 14:32:24 | jroll | dansmith: people were scared to remove it | |
| 14:32:50 | dansmith | hmm | |
| 14:33:01 | jroll | a quick workaround would likely be to set 0 for cpu/ram in ironic node.properties | |
| 14:33:14 | jroll | and then it will report 0 | |
| 14:33:14 | dansmith | I thought this was long since sorted | |
| 14:33:24 | jroll | yeah, I thought it just worked as well | |
| 14:33:39 | dansmith | what was the plan during the overlap to avoid exposing the old values once everything was migrated? | |
| 14:33:47 | dansmith | override properties in ironic? | |
| 14:33:53 | dansmith | s/everything/flavors/ | |
| 14:33:57 | jroll | I don't remember, sorry | |
| 14:34:15 | belmorei_ | jroll: that means updating all ironic nodes... can't do it. | |
| 14:34:43 | belmorei_ | How about if I remove the "resources:VCPU=0", ... from the flavors? | |
| 14:34:57 | mriedem | your bm flavors aren't the problem | |
| 14:35:11 | mriedem | the problem is the bm flavors are getting scheduled to the ironic nodes right? | |
| 14:35:13 | mriedem | *vm flavors | |
| 14:35:15 | dansmith | right | |
| 14:35:24 | jroll | that would prevent VMs from landing on an active ironic node, but not an inactive one | |
| 14:35:28 | mriedem | and that's because the bm nodes are reporting ram/cpu/disk inventory | |
| 14:35:47 | belmorei_ | jroll: true | |
| 14:39:22 | belmorei_ | ok, so for me the best is to backport the commit mentioned by mriedem | |
| 14:39:26 | mriedem | belmorei_: you might be able to just patch https://github.com/openstack/nova/blob/stable/queens/nova/virt/ironic/driver.py#L797 to be 0 | |
| 14:39:37 | belmorei_ | yeah, thanks | |
| 14:39:38 | mriedem | belmorei_: i'm guessing that's not going to backport cleanly to queens | |
| 14:39:47 | bauzas | gibi: thanks for the email recap | |
| 14:39:54 | belmorei_ | any plan to still include this in rocky? | |
| 14:40:10 | bauzas | gibi: I now understand the problem :) | |
| 14:40:12 | belmorei_ | because for me is a bug | |
| 14:40:24 | gibi | bauzas: :) | |