| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-18 | |||
| 15:41:29 | edleafe | because we don't create the allocation for nodes with that custom RC | |
| 15:42:23 | dtantsur | wait, wasn't that periodic task in the ironic virt driver written to update the allocations? | |
| 15:43:11 | dansmith | dtantsur: no, the flavors | |
| 15:43:29 | dansmith | dtantsur: when it was written the compute node would unceremoniously heal allocations | |
| 15:43:32 | edleafe | dtantsur: that was supposed to be part of it, but only the flavor migration was added | |
| 15:43:38 | dansmith | dtantsur: right over top of the scheduler, which is why we made it stop, | |
| 15:43:44 | dtantsur | oooooh | |
| 15:43:51 | dansmith | dtantsur: but that periodic in the ironic driver was depending on that | |
| 15:44:09 | dtantsur | was the update bit removed in Pike or Queens or both? | |
| 15:44:29 | dansmith | dtantsur: pike once all compute nodes have been updated | |
| 15:44:29 | johnthetubaguy | Pike | |
| 15:44:44 | dansmith | dtantsur: it would keep doing it as long as ocata nodes were in the cluster | |
| 15:46:30 | dtantsur | so, is our only option to update the periodic task to fix up allocations as well? | |
| 15:46:38 | dtantsur | I don't know nova well enough to suggest anything else :) | |
| 15:48:08 | dansmith | dtantsur: we're going to let ironic have a vote on the "should we keep healing" decision | |
| 15:50:05 | efried | cdent I wrote that test case, but I can't get it to fail properly. I suspect it may be due to that bug, though. | |
| 15:50:20 | efried | cdent Mebbe I post it and see if you can find holes. | |
| 15:50:34 | cdent | efried: sounds like a grand plan | |
| 15:52:22 | dtantsur | dansmith: I just don't see other options. I don't think we should ask operators to go into Placement and fix up stuff themselves. or should we? | |
| 15:52:39 | dansmith | dtantsur: nobody is suggesting that | |
| 15:52:40 | dtantsur | johnthetubaguy: btw you may also hit https://bugs.launchpad.net/nova/+bug/1723423 in your testing | |
| 15:52:41 | openstack | Launchpad bug 1723423 in OpenStack Compute (nova) "Ironic node cannot be used if it does not report VCPU" [Undecided,In progress] - Assigned to Dmitry Tantsur (divius) | |
| 15:53:48 | johnthetubaguy | dtantsur: so ironic should eventually return 0 for all the resources that are no longer being claimed? | |
| 15:54:46 | dtantsur | johnthetubaguy: sorry, I don't quite get it. Which of the bugs are you referring to? | |
| 15:55:07 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for resource classes https://review.openstack.org/511182 | |
| 15:55:08 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: [WIP] RP list: member_of and resources parameters https://review.openstack.org/511183 | |
| 15:55:20 | johnthetubaguy | dtantsur: the one you just linked too, that might be a bit of the picture I was missing | |
| 15:55:41 | johnthetubaguy | dtantsur: once the resource class is updated, we stop reporting the vcpu mem and disk from ironic? | |
| 15:56:46 | dtantsur | johnthetubaguy: we don't, no. only if you remove 'cpus' key from node.properties | |
| 15:56:51 | efried | cdent Actually the problem is that AllocationCandidates refuses to find anything in the shared RP as long as the compute node also has inventory in the same RC (DISK_GB). | |
| 15:56:52 | dtantsur | which is what I tried doing in my testing | |
| 15:57:01 | efried | alex_xu yt? | |
| 15:57:47 | cdent | efried: are you based on alex’s stuff? that may be a semi-intentional holdover from the early policy of “we don’t do that” | |
| 15:57:59 | johnthetubaguy | dtantsur: OK, I didn't know that was a thing, makes sense now, thanks. | |
| 15:58:00 | cdent | in which case you’ve found a version (there are many) of the bug | |
| 15:58:11 | efried | cdent Yes, I'm based on his in-flight patch to implement traits in allocation candidates. | |
| 15:59:17 | efried | cdent Got a handy dashboard of placement bugs? This one is worth filing separately if we don't already have it. | |
| 15:59:19 | dtantsur | so, back to the first bug. are there any options except for fixing the periodic task? I really don't know | |
| 15:59:48 | cdent | efried: just make a nova one tagged placement: https://bugs.launchpad.net/nova/+bugs?field.tag=placement | |
| 16:00:20 | johnthetubaguy | dtantsur: we went through some, they all seem a lot worse, like calling into placement to correct the allocations after we migrate the instance flavor | |
| 16:00:21 | efried | cdent Cool, but don't want to dup if there's already one out there. | |
| 16:00:43 | efried | cdent Oh. That's the dashboard. Got it :) | |
| 16:01:22 | openstackgerrit | Merged openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 16:05:00 | dtantsur | johnthetubaguy: okay.. wanna try to make a patch? at least you have a reproducer already | |
| 16:05:18 | efried | cdent What do we call the "main" RP to distinguish it from the shared RPs? Today it's always the "compute node RP" - but do we have a generic name for it? | |
| 16:06:22 | cdent | efried: we haven’t really settled on anything. alex_xu tried to use “root” in his stuff but that overlaps with nested | |
| 16:06:30 | efried | cdent Right | |
| 16:06:58 | efried | I'll use "compute node" for the sake of the bug, but eventually we should have a word. | |
| 16:07:01 | cdent | the other one is the “sharing” rp, so maybe the main one is the “non-sharing” (that’s a bit weak though) | |
| 16:09:01 | cfriesen | cdent: we could have multiple non-shared RPs per compute node, no? I think I remember hearing that some Intel folks were talking about creating a separate non-shared RP to provide L3 cache. | |
| 16:09:44 | openstackgerrit | John Garbutt proposed openstack/nova master: Keep updating allocations for Ironic https://review.openstack.org/513085 | |
| 16:09:44 | cfriesen | if so then it's really the "nova compute node RP" or something like that. :) | |
| 16:09:54 | johnthetubaguy | dtantsur: that is my first stab at it ^ | |
| 16:11:28 | cdent | cfriesen: non-sharing-root-provider is more complete then | |
| 16:11:41 | cdent | was trying to leave that complexity out for mow | |
| 16:12:37 | cfriesen | cdent: although in the case of L3 cache it would be properly modelled as a per-NUMA-node resource and would need nested providers. :) | |
| 16:13:17 | cdent | let’s hope so | |
| 16:15:08 | edleafe | "greedy rp"? (i.e., not sharing) | |
| 16:16:00 | mriedem | efried: dansmith: i went through https://review.openstack.org/#/c/510244/ for the granular allocation candidates request syntax, | |
| 16:16:17 | mriedem | only sticking point right now is this means you can only specify one set of resource classes/traits which are spread across providers, right? | |
| 16:16:25 | mriedem | because all numbered groups are applied to the same RP | |
| 16:16:30 | efried | mriedem Correct. | |
| 16:16:48 | efried | mriedem Why would you need more than... oh. | |
| 16:16:54 | mriedem | i'm trying to think if that screws us later | |
| 16:17:09 | efried | mriedem Actually, no, I think that's aight. | |
| 16:17:23 | mriedem | i'm mostly worried about the shared storage scenario, | |
| 16:17:27 | mriedem | which i know we're punting, but | |
| 16:17:34 | dansmith | wait, what? | |
| 16:17:42 | dansmith | the point of this is you specifying traits/classes that go together | |
| 16:17:45 | efried | mriedem Well, the spec deliberately leaves out shared RP semantics. | |
| 16:18:10 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Network bandwidth resource provider https://review.openstack.org/502306 | |
| 16:18:20 | efried | mriedem But we do have a couple of pretty big holes even without this, when it comes to traits + aggregates. | |
| 16:18:24 | efried | I'm writing one up now :) | |
| 16:19:05 | dansmith | ah, I see | |
| 16:19:06 | mriedem | so with shared storage, i think we wanted the scheduler to ask for vcpu/memory_mb and disk_gb, and we could get 2 providers back, | |
| 16:19:18 | mriedem | one compute node RP for the vcpu/memory_mb, and one shared storage provider for the disk_gb | |
| 16:19:22 | mriedem | scheduler would claim on both | |
| 16:19:37 | dansmith | with this, | |
| 16:19:38 | mriedem | maybe that still works ok with this new thing | |
| 16:19:47 | mriedem | you request those in an unnumbered resources group | |
| 16:19:52 | mriedem | so the resources can be spread across providers | |
| 16:19:53 | dansmith | you could ask for resources2:DISK=1,required2=SHARED | |
| 16:20:06 | dansmith | to make sure you get a shared disk, right? | |
| 16:20:37 | mriedem | in previous thinking about shared storage i didn't think the flavor had to be specific as to the type of storage | |
| 16:20:57 | dansmith | it doesn't have to be | |
| 16:21:21 | mriedem | so like i said, i think we don't lose the ability to do what we can do today with an unnumbered request, | |
| 16:21:30 | mriedem | you just can't have more than one unnumbered request | |
| 16:21:38 | mriedem | but i can't think of any reason why that might be bad | |
| 16:21:52 | efried | Correct. Except "what we can do today with an unnumbered request" is poorly defined and broken. | |
| 16:22:08 | dansmith | um | |
| 16:22:25 | dansmith | is there some special behavior assigned to resources= that isn't there for resourcesN=? | |
| 16:22:31 | dansmith | I must have missed that if so | |
| 16:22:35 | mriedem | also, now that i'm thinking about this, what is preventing placementing from returning 2 compute node providers, one that satisfies the vcpu requirement and one that satisfies the memory_mb requirement, scheduler claiming on both and the build failing? | |
| 16:22:39 | efried | Sorry, by "today" I actually mean "not today at all". I mean when both traits and aggregates are considered. | |
| 16:23:03 | mriedem | dansmith: my comment here https://review.openstack.org/#/c/510244/6/specs/queens/approved/granular-resource-requests.rst@230 | |
| 16:23:51 | dansmith | mriedem: ah, I see, I had read that first bullet differently I guess | |
| 16:24:03 | dansmith | I read this as "if you only have one grouping" | |
| 16:24:54 | efried | "same tree" is the key there, mriedem. That's how you don't get VCPU and MEMORY_MB from separate computes. | |
| 16:25:40 | mriedem | ok so compute node today is a tree of 1 node | |
| 16:25:45 | mriedem | the alpha and omega | |