| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-28 | |||
| 15:41:38 | openstack | Launchpad bug 1714248 in OpenStack Compute (nova) "Compute node HA for ironic doesn't work due to the name duplication of Resource Provider " [High,Confirmed] | |
| 15:41:41 | jaypipes | cdent: you mean we're *not* unique? | |
| 15:42:47 | efried | but no skin in the game. Reading bugh... | |
| 15:42:57 | cdent | jaypipes: we enforce uniqueness on the name column | |
| 15:43:15 | jaypipes | cdent: ah, great. that's good then. | |
| 15:43:25 | cdent | jaypipes: which people are saying is bad | |
| 15:43:41 | cdent | because they want to blue green a compute node or something | |
| 15:43:55 | cdent | and names get in the way | |
| 15:45:09 | efried | cdent Skimming this bug, it seems to me like the problem is that the compute service that's taking over should *not* be attempting to create a new RP. | |
| 15:45:19 | efried | It should be looking up and using the old one. | |
| 15:45:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Ensure instance can migrate when launched concurrently https://review.openstack.org/506093 | |
| 15:45:35 | mriedem | bauzas: ^ is all cleaned up now | |
| 15:45:45 | cdent | efried: that would be one way to solve that particular problem | |
| 15:45:47 | efried | Is the RP somehow associated with the compute process rather than the node(s) it's managing? | |
| 15:46:00 | cfriesen | efried: do we ensure hypervisor host name uniqueness? | |
| 15:46:34 | cfriesen | efried: actually, I guess this is for ironic, so it's not even hypervisor, just host | |
| 15:46:59 | cfriesen | or is there some other way to uniquely identify a host in ironic (I know nothing) | |
| 15:47:43 | efried | Yeah, I don't know from ironic. Is it like, compute runs on the hypervisor and that represents a single possible instance? | |
| 15:47:58 | efried | (johnthetubaguy this conversation may interest you) | |
| 15:48:05 | cfriesen | efried: I believe nova-compute represents a bunch of baremetal machines | |
| 15:49:01 | efried | And that nova-compute could run on any one of several possible "hypervisors" that all "manage" that bunch of baremetal machines? | |
| 15:49:02 | cfriesen | efried: so then the node where nova-compute is running crashes, and they want a different nova-compute process to take over managing those baremetal machines. | |
| 15:49:43 | efried | Cool. So my question (cdent, johnthetubaguy): Why is there a RP associated with the nova-compute at all?? Seems like there should just be a RP per baremetal machine. | |
| 15:50:34 | cdent | efried: you’d think so, but no | |
| 15:50:44 | cdent | a node has inventory of classes of baremetal | |
| 15:51:01 | cdent | (last I recall, it’s hard to remember/keep straight) | |
| 15:51:03 | cfriesen | efried: I think the idea is that each baremetal machine is a resource | |
| 15:51:27 | cfriesen | or is several types of resource | |
| 15:51:45 | cfriesen | since you always claim a whole machine at a time | |
| 15:52:40 | efried | But but but... that would mean that every baremetal node in that nova-compute's purview is "identical". | |
| 15:53:21 | mriedem | johnthetubaguy: while you're awake, this is holding up the live migration new style volume attach change, and is simple https://review.openstack.org/#/c/506805/ | |
| 15:56:56 | cdent | efried: ironic’s virtdriver’s get_inventory may explain things a bit: https://github.com/openstack/nova/blob/ae4b5d0147cb3e345bf57034221e9c8fedf3cad2/nova/virt/ironic/driver.py#L751 | |
| 15:57:14 | efried | cdent Was just looking at that. | |
| 15:57:43 | efried | And was looking for the change jaypipes was going TODO. | |
| 15:57:48 | cdent | i’m not sure, but I think you can different classes of baremental node, resprsented by multiple custom resource classes | |
| 15:57:58 | efried | And was going to look for where the RP is set up, but not sure where to start looking for that. | |
| 15:58:01 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Return Alternate Hosts https://review.openstack.org/504275 | |
| 15:58:17 | edleafe | mriedem: johnthetubaguy: ^^ | |
| 15:58:43 | efried | cdent Yeah, sounds like that's where they're going, but haven't yet. | |
| 15:59:07 | cdent | efried: edleafe did some further work on that chunk of code | |
| 15:59:48 | johnthetubaguy | I am keen to help in the ironic scheduling side of things btw, trying to write up all the traits discussions | |
| 16:00:29 | cfriesen | cdent: why wouldn't you follow efried's suggestion and have an RP per baremetal machine that reports resources for memory/disk/cpu? Then when you want to allocate a baremetal node you look for the smallest machine that can provide what you're looking for. | |
| 16:02:31 | cfriesen | I suspect this has been discussed already, so if anyone has links to the discussion.... | |
| 16:02:47 | efried | johnthetubaguy Yeah, I was just poring over your https://review.openstack.org/#/c/507052/2/specs/queens/approved/ironic-traits.rst | |
| 16:03:13 | efried | which is actually what led me down the rabbit hole and ultimately prompted this discussion :) | |
| 16:03:37 | efried | Though the name conflict bug thing is a tangent-to-a-tangent... | |
| 16:03:52 | johnthetubaguy | so for ironic, resource class matching, and requesting no VCPU,MEM, etc, is the current way forward I thought? | |
| 16:04:18 | cfriesen | johnthetubaguy: this spec? https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/node-resource-class.html | |
| 16:05:21 | cfriesen | or maybe this one is more accurate: https://blueprints.launchpad.net/nova/+spec/custom-resource-classes-pike | |
| 16:05:40 | johnthetubaguy | the later might be closer, but yeah, that support has all merged | |
| 16:05:53 | cdent | cfriesen: sorry, in yet another meeting, so lost track of the discussion, will come back soon | |
| 16:06:48 | cfriesen | cdent: no worries, just idly curious | |
| 16:06:55 | johnthetubaguy | cfriesen: both actually are done I think | |
| 16:07:10 | dansmith | cdent: either your new job comes with lots of extra meetings, or you enjoy and announce them more often | |
| 16:08:13 | cdent | dansmith: these are all upstream meetings, I guess I’m just conscious lately of the extent to which they are distracting me from chatting with you | |
| 16:08:23 | dansmith | cdent: ah :) | |
| 16:08:28 | efried | johnthetubaguy iiuc, you want to get rid of tracking CPU, memory, etc.; tag each of your nodes with some identifier; and then have your flavor just use that identifier? | |
| 16:09:01 | johnthetubaguy | efried: so I thought the CPU memory, etc tracking was all going in queens, regardless? | |
| 16:09:18 | efried | "going into queens" or "going away in queens" ? | |
| 16:09:25 | johnthetubaguy | going away in queens | |
| 16:09:37 | johnthetubaguy | oops, missed the important word there | |
| 16:09:47 | cfriesen | efried: there's an interesting short email chain here: https://openstack.nimeyo.com/91493/openstack-dev-ironic-nova-indivisible-resource-providers | |
| 16:10:26 | efried | johnthetubaguy I assume that's just an ironic statement | |
| 16:10:45 | efried | I honestly know nothing about ironic, learning as I go in the context of this chat. | |
| 16:10:52 | johnthetubaguy | that's what the comment in the Nova code was saying, based on plan A for Nova I believe? | |
| 16:11:25 | johnthetubaguy | https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L758 | |
| 16:11:50 | johnthetubaguy | ironic resources are indivisable | |
| 16:12:01 | johnthetubaguy | custom resource classes model that nicely | |
| 16:12:46 | efried | Right, so the point will be to have a custom resource class representing a "kind" of node, such that any node associated with that resource class is effectively "identical" to any other. | |
| 16:13:14 | johnthetubaguy | well, "identical" from the point of view of the flavor | |
| 16:13:30 | johnthetubaguy | you might have several generations mapped to a single resource class, if you want | |
| 16:13:32 | efried | Right. And you're modeling the compute process as the resource *provider* of the resources of the various classes it can see. | |
| 16:13:50 | efried | (What do we call the thing the compute process runs on? The "hypervisor"?) | |
| 16:14:05 | johnthetubaguy | no, the resource provider is the ironic node uuid in Ironic | |
| 16:14:36 | efried | Where "node" isn't the baremetal machine, it's the thing that manages 'em, right? | |
| 16:14:42 | johnthetubaguy | nova-compute process manages multiple ironic nodes (that list varies depending on the hash ring, hence the bug) | |
| 16:14:52 | johnthetubaguy | so the host is nova-compute | |
| 16:14:55 | efried | ayee | |
| 16:15:03 | efried | Terminology fail. | |
| 16:15:03 | johnthetubaguy | the node is the ironic node | |
| 16:15:21 | johnthetubaguy | the node is the resource provider | |
| 16:15:32 | cdent | nova-compute hosts multiple ironic nodes, each of which have an inventory of customr resource class with count of 1 | |
| 16:15:36 | cdent | (my memory is starting to refresh) | |
| 16:15:43 | johnthetubaguy | +1 cdent | |
| 16:16:37 | efried | But some of those ironic nodes can be using the same resource class, if they're functionally "identical" | |
| 16:16:38 | efried | ? | |
| 16:16:47 | cdent | yes | |
| 16:17:04 | efried | So it would seem like, as part of the failover process, the host that's taking over ought to delete the RP for the failed host. | |
| 16:17:13 | johnthetubaguy | yeah, each is offering 1: CUSTOM_GOLD | |
| 16:17:26 | efried | Seems like that ought to happen regardless, otherwise the scheduler will think it's still there and may try to schedule stuff to it. | |
| 16:17:35 | johnthetubaguy | so for that bug | |
| 16:17:40 | efried | yeah, for that bug | |
| 16:17:46 | johnthetubaguy | the dead thing would normally be responsible for the delete of it | |
| 16:17:59 | efried | That... doesn't make sense. It's dead. It can't call the placement API. | |
| 16:18:10 | johnthetubaguy | right, that's the problem | |
| 16:18:25 | efried | At some level, the new guy must know he's taking over for the old guy. | |
| 16:18:39 | efried | So why can't he delete the old guy's RP? | |
| 16:18:49 | johnthetubaguy | not sure if it sees that edge | |
| 16:18:59 | johnthetubaguy | it just sees a new node right now | |
| 16:19:09 | johnthetubaguy | but not 100% sure on that | |