Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-19
17:14:24 jroll efried: agree, and I'm not sure we want to special case it. but if we did: host = random(get_ironic_compute_hosts())
17:14:34 mriedem mordred: doesn't look like it http://paste.openstack.org/show/677614/
17:15:06 jroll efried: I realize now I don't know how that works with placement. in the old world, both the host and node are in the compute_nodes record
17:15:07 mriedem mordred: well, i guess if there is only 1 'available' zone then you know what the default is :)
17:15:10 efried jroll: Oh, so all ironic nodes under an n-cond can be managed by any n-cpu under that same n-cond?
17:15:37 jroll efried: in theory, yes, we don't do that in reality today
17:16:15 jroll efried: but we do currently shuffle them between ironic n-cpu hosts that are up, as the hosts go up and down
17:18:52 efried jroll: But don't ops also always deploy specific ironic nodes?
17:19:23 jroll efried: as in "nova boot this-specific-machine"?
17:19:29 efried Yeah; maybe I'm misremembering that.
17:19:34 jroll nope, not at all
17:19:36 efried k
17:19:41 jroll some do - and they use things like scheduler hints
17:20:01 efried Rightright, that's what I was thinking - that it's a requirement to be able to do so.
17:20:03 jroll other installations are just another cloud, with the hypervisors missing
17:20:32 jroll efried: the "requirement" part of that depends who you ask :)
17:20:40 jroll I'm not sure it works upstream today
17:20:45 efried So yeah, I have absolutely no idea how we're planning to model ironic in Placement-land. But I suspect there will have to be some additional affordance outside of what we're doing in u_p_t.
17:21:27 efried Because right now, if you tried to model each ironic node as its own root/tree, it wouldn't work.
17:21:47 efried (by "right now" I mean "with NRP and u_p_t implemented as specced")
17:22:42 jroll yeah, I'm not up to speed on the nested things
17:23:20 jroll I suspect jay has ideas here, dunno
17:27:30 openstackgerrit Eric Fried proposed openstack/nova-specs master: Update Provider Tree https://review.openstack.org/540111
17:27:36 efried mriedem: See how that grabs ya ^
17:27:47 efried jaypipes, edleafe, cdent ^
17:32:50 edleafe efried: stop grabbing me!
17:32:59 efried mriedem: Updated that diagram a little bit too https://awwapp.com/b/u8ghqr9rr/
17:33:09 efried edleafe: Dangit, there goes my career 40 years from now.
17:34:11 mriedem efried: then i think you definitely need a diagram in the spec to give an example
17:34:14 jroll efried: I see nodename everywhere in this spec, that signals to me that one tree per ironic node is expected
17:34:20 mriedem of what's in vs what's out when the driver gets the provider tree object
17:34:55 efried jroll: That's probably just a lack of precision on my part more than anything.
17:35:14 efried jroll: I still don't understand the difference between nodename, hypervisor_hostname, and host.name.
17:35:54 efried jroll: But I wrote the code first, and 'nodename' was the var that made sense to pass around in there, so that's probably how it ended up in the spec.
17:36:02 jroll efried: nodename ~= hypervisor_hostname, I'm not sure what host.name refers to
17:36:46 efried nodename == hypervisor_hostname, except for ironic where they're never equal; but hypervisor_hostname == host.name always ??? Or something ???
17:37:28 jroll in the ironic world, nodename == hypervisor_hostname, for sure. compute_node.host is the different one (and the one that actually looks like a hostname)
17:37:37 efried jroll: When you say ~= is that "regular expression match" à la Perl, or "not equal" à la other things?
17:37:47 efried oh, okay.
17:38:19 efried So that makes zero sense to me, that ironic node name should be the same as the hypervisor host's name. So the latter is actually a total misnomer.
17:38:22 jroll efried: equal, but I'm not 100% sure on weird drivers, so I'm going with approximately equal :)
17:38:28 efried gotcha
17:39:24 jroll sure, it's a poorly named thing. originally it was for things like xen, where compute.host (how you reach n-cpu) might be separate from the actual hypervisor (how you reach xenapi)
17:39:40 jroll s/hypervisor/hypervisor_hostname/
17:40:31 mordred mriedem: :) ... and yah - the lack of info about default in config in the az list is, I think, the thing
17:41:04 mordred mriedem: came up in some discussions around nodepool and azs - thanks for confirming
17:41:21 mriedem mordred: from what i remember, there are actually >1 nova config options related to a 'default' az
17:41:49 mordred mriedem: awesome. and of course there are :)
17:41:55 openstackgerrit sean mooney proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257
17:41:56 openstackgerrit sean mooney proposed openstack/nova master: Add Neutron port capabilities to devspec in request https://review.openstack.org/451777
17:41:57 openstackgerrit sean mooney proposed openstack/nova master: Read Neutron port 'binding_profile' during boot https://review.openstack.org/507481
17:41:57 openstackgerrit sean mooney proposed openstack/nova master: Format NIC features using os-traits definitions https://review.openstack.org/466051
17:42:05 mriedem default_availability_zone and default_schedule_zone
17:42:38 jroll efried: as far as the one vs many trees for ironic... _update_available_resource is called once per ironic node, not once per nova-compute host. (and thus the same for get_inventory and such). in old-nova terms, it's called once per compute_node record, of which many may exist per compute service. does that make sense?
17:42:46 mordred mriedem: at root is that currently nodepool gets a list of azs and the balances across them by default - which has led to unexpected behavior before (citycloud had an az specifically for a type of flavor and were confused why were launching nodes there)
17:42:56 jroll efried: I'll do a full review of the spec and drop some comments there, if that works better for you
17:43:32 mordred mriedem: but on clouds with more than one az where they're all the same, if we DON'T balance across them we wind up with usage only in the default zone
17:43:46 jroll efried: I guess I'm mostly curious if the "one tree" constraint you're thinking of is "one root per compute service" or "one root per resource provider"?
17:43:51 efried jroll: That would be great, though I don't know that we're going to get much satisfaction out of that.
17:44:23 mriedem mordred: ok, and in the citycloud case, as a user, you don't have access to see the flavor linked to the AZ via host aggregate
17:44:28 mriedem because users can't see host aggregates
17:44:31 efried jroll: I explicitly state in the spec that update_provider_tree is being called once per ironic node, BUT that doesn't really affect how the result is used.
17:44:53 mriedem mordred: in other words, if using this flavor, don't specify AZ
17:45:18 mordred mriedem: yup
17:45:47 mriedem in queens they can at least now put a description on the flavor to say that, but your tooling likely doesn't care about that until you've already tried debugging what's going wrong
17:46:01 mordred mriedem: so there's a few potentially missing pieces of metadata .. however, clarkb has been advocating that we stop trying to balance across azs by default and instead require someone to configure az use explicitly
17:46:04 mriedem and then you have to put in special logic for that flavor on that cloud
17:46:33 mordred mriedem: exactly. when if there was some metadata, I could even validate the flavor/az combo in shade/openstacksdk
17:46:49 mordred before we even bother making an API call
17:46:54 mordred "dude, this combo won't work"
17:46:57 jroll efried: hm, having trouble finding that note, but also seeing now that this code is mostly done, so maybe I need to back up and look at the bigger picture
17:47:20 mriedem mordred: so do you end up getting a novalidhost in the citycloud case with that flavor and some other AZ?
17:47:23 efried jroll: I was not trying to (actually "trying not to") predict how ironic is going to model. What I'm trying to say is, if ironic wants to model with a separate tree/root per ironic node, this spec is insufficient to handle it.
17:48:14 mordred mriedem: I don't think so - no - I think what happened was we got scheduled on hardware that was intended for something else (they weren't expecting people to request anything in that az unless someone told them to) so we got nodes that had messed up networking or something else
17:48:28 jroll efried: sure, I'm trying to figure out what about this spec precludes doing such a thing
17:48:29 efried jroll: https://review.openstack.org/#/c/540111/5/specs/rocky/approved/update-provider-tree.rst L57-61
17:48:48 efried jroll: It's not anything about the spec. It's the way the rest of Nova is shaped currently.
17:50:00 jroll efried: ah, that block. I guess that reads to me as "one root per compute node", nothing that compute service != compute node. I guess I need to read the code.
17:50:23 efried jroll: Actually, the freshly-updated text in the note on L69-76 *does* preclude the multiple-trees-that-aren't-sharing thing.
17:50:52 jroll hmm
17:51:17 jroll ok, I will dig, I need to learn more about NRP before I can say much else. thanks, efried
17:52:04 efried jroll: Let me know if you need pointers. Much of the code for this is not yet merged.
17:52:24 efried jroll: Pending code is in series starting at https://review.openstack.org/#/c/537648/
17:52:30 jroll efried: yep, I'm there
17:53:25 openstackgerrit Merged openstack/nova stable/ocata: Add 'delete_host' command in 'nova-manage cell_v2' https://review.openstack.org/513721
17:53:53 openstackgerrit Merged openstack/nova stable/ocata: Fix test_instance_get_all_by_host https://review.openstack.org/516486
17:54:45 lyarwood mriedem: back online, yeah I'm around this week, I'll take a look this evening if there's anything left.
17:59:09 efried jaypipes: Do you have a couple minutes to help me understand the RT flow for ironic?
18:00:39 jroll efried: looking through some of this, I'm too far behind on how some of this works to fully discuss why this is or isn't fundamentally broken for ironic today, but I hope to be able to later this week, or worst case in person next week
18:00:54 jroll I can try to help you understand the current flow, if you have questions in mind
18:01:30 efried jroll: Yeah, if you don't mind.
18:01:48 efried I'm trying to understand how the current get_inventory is called.
18:02:11 efried In the ironic case, there's actually multiple ComputeNode s ?
18:02:29 jroll correct - there is a ComputeNode per ironic node
18:02:36 efried And there's a loop over those, and RT does the update_compute_node stuff for each?
18:03:01 jroll actually, there's an instance of the RT class created for each ComputeNode, IIRC
18:03:13 efried ah, that would explain why I wasn't finding said loop.
18:03:35 jroll that may have changed somewhere
18:03:35 efried But ultimately what it means is that we are indeed creating a provider in placement for each ironic node, separately.
18:03:40 jroll correct

Earlier   Later