| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-10 | |||
| 13:59:09 | efried | cdent Which would be fine, if that's what we're doing, AND the multiple control points understand the semantics of the provider having multiple aggs associated or not. | |
| 13:59:36 | cdent | sean-k-mooney: I will keep you in beverages of your choice for the duration of dublin if you promise me to never bring up multi-parents again :) | |
| 14:00:02 | sean-k-mooney | cdent: efried so in the case of vhost-vfio interfaces i was considering them to be owned by neutron not nova yes i want to model numa relationship too | |
| 14:00:09 | efried | mgoddard Which is what report client should be doing, but isn't yet, but I'm working on it. | |
| 14:00:09 | kashyap | mdbooth: I see, stopped the audit for a bit; will get back. | |
| 14:00:51 | kashyap | mdbooth: https://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg00845.html -- [PATCH 0/2] qemu-img: Let "info" warn and go ahead without -U | |
| 14:00:59 | sean-k-mooney | cdent:hehe yes i know we dont want to have multiple parent so im exploing other options. | |
| 14:01:17 | kashyap | (The "-U" == '--force-share') | |
| 14:01:27 | efried | mgoddard It also sends the generation - whatever was the last value it got when it retrieved the RP record - whenever it updates the RP or associations (traits, aggs*, etc.). And then the placement API bounces with a 409 if that's not the same as the generation it thinks that RP should have. | |
| 14:01:58 | efried | mgoddard The client can/should respond to that by re-GETting the provider/associations, redriving the change, and re-PUTting with the new generation. | |
| 14:02:07 | sean-k-mooney | cdent: basically i want to model the resouce is a sub resouce of a numa node(eventully) and have an easy way to associate them with the neutron agent that manages them | |
| 14:02:56 | sean-k-mooney | cdent: without the numa aspect i had originally considerd modeling the neutron agent as a resouce providers and making it the parent of the sub resouce | |
| 14:02:59 | cdent | that sounds a bit like the model of shared provider where the neutron agent is a resource provider | |
| 14:03:10 | efried | sean-k-mooney Placement won't stop you from doing that, but it would be highly irregular for neutron to own (create/modify/control) a provider in the middle of a tree that's otherwise owned by compute. | |
| 14:03:52 | cdent | I think we've already talked in the past about considering being able to parent neutron rps into a compute generated tree, as leafs | |
| 14:03:59 | sean-k-mooney | efried: well the resoce for this design is i dont think nova long term should be respocible for tracking networking resouces in placement | |
| 14:04:01 | efried | sean-k-mooney It would be preferable if you found a way to model it such that the provider(s) owned by neutron were sharing providers associated by aggregate with ones in the compute's tree. | |
| 14:04:10 | cdent | there was some of that in denver. not that we _will_ do it, but might be able to do it | |
| 14:05:49 | cdent | I have some vague concerns that the provider tree model in compute is going to make it harder to manage rps from multiple places, which was an original goal | |
| 14:06:09 | efried | I mean, I guess, as long as compute and the neutron agent both agree on how that dance is done. It worries me, though, because different vendors' drivers have to talk to the same neutron agent sometimes. | |
| 14:06:17 | cdent | I need to unvague those concerns, but at the moment I can't even get devstack to do the right things for my simple experiments, so... | |
| 14:07:00 | sean-k-mooney | cdent: well the issue is i dont think nova should own the root node of the tree. i think we should considerd the root node to be something earch service can create subtree from | |
| 14:08:06 | efried | sean-k-mooney What do you call that root node? | |
| 14:08:09 | efried | Is it a host? | |
| 14:08:21 | efried | a "cloud"? | |
| 14:08:50 | efried | some nebulous as-yet-unnamed entity that exists solely as an anchor point for this multi-owner provider tree? | |
| 14:09:06 | efried | ...but doesn't correspond to anything in the real world? | |
| 14:09:06 | cdent | sean-k-mooney: That's kind of related to why I disputed that root provider be a thing in the data struture or representation. I think we should be able to access a tree of providers anywhere in whatever trees people like, and declare subtrees to be whole if that's what suits them. That is, if we're gonna have trees, let's have _trees_. | |
| 14:09:15 | cdent | (I'd rather just not have trees, but that ship sailed) | |
| 14:11:16 | sean-k-mooney | efried: in the case of a nova created tree i was assuming a compute node. e.g. a host yes | |
| 14:12:10 | efried | sean-k-mooney Right, I was referring to what you said earlier about "i dont think nova should own the root node of the tree" and "the root node [is] something each service can create subtree from" | |
| 14:12:45 | sean-k-mooney | yep i think we are violently agreeing on that point :) | |
| 14:12:46 | mgoddard | efried: so with your change, will a generation change in the provider tree cause inventory, traits, and aggregates to be updated, and checked that all those responses contain the same generation? | |
| 14:12:49 | efried | In that picture, who owns the root node, and what does it represent? | |
| 14:14:01 | sean-k-mooney | efried: everyone owns it. i think we just need to agree on how the root node is create for example by saying that the node is idenfied by the host_id which default to the hostname | |
| 14:14:29 | efried | sean-k-mooney But if the root node represents a compute host, doesn't it make sense for compute to own/create it? | |
| 14:14:46 | sean-k-mooney | efried: not in a converged deployment | |
| 14:15:25 | sean-k-mooney | for example if you have cinder running on the same physical server then its also a storage node not just a compute node | |
| 14:16:42 | sean-k-mooney | if cinder and nova agree on how to create/select that root node then can boot create subtrees for that phyical server | |
| 14:16:49 | efried | mgoddard At the report client level, the plan for set_traits_for_provider is: If PUT /rp/{uuid}/traits succeeds (200), I'll update_traits on the ProviderTree, setting the generation based on what's in the PUT response. (The generation is an attribute of the provider, so updating it via update_traits updates it for anything else associated with that provider.) | |
| 14:17:33 | efried | mgoddard If the PUT fails 409, set_traits_for_provider will raise (a subclass of) this new conflict exception. | |
| 14:17:34 | sean-k-mooney | s/ then can boot/ then both can/ | |
| 14:18:39 | efried | mgoddard The report client consumer (i.e. resource tracker) should then redrive the overarching operation, which should entail first asking report client to refresh its cached representation of that provider in the ProviderTree. | |
| 14:19:11 | efried | mgoddard Which means reGETting the provider and all its associated bits (traits, aggs, etc.) and calling the appropriate ProviderTree methods to update them all. | |
| 14:19:53 | efried | mgoddard Then the consumer (resource tracker) would make whatever change again (in this example setting the traits). Rinse, repeat. | |
| 14:20:14 | efried | mgoddard Not sure if I'm explaining this particularly well. | |
| 14:20:35 | sean-k-mooney | efried: cdent anyway that when a little off topic but i think one design would be for neutorn to create its resouces under the numa nodes create by nova and add the resouces it creates to an aggreate using the neutron agent uuid as the aggreate uuid. | |
| 14:21:00 | efried | sean-k-mooney Why do you need the aggregate in that picture? | |
| 14:21:11 | efried | sean-k-mooney Are the neutron resources common to multiple hosts? | |
| 14:21:41 | efried | sean-k-mooney If so, they should *not* be created as providers in the tree under the numa nodes. | |
| 14:22:00 | sean-k-mooney | efried: no they are specific to that compute but i want a simple way to look up all the resocurce providers created by the neutron agent on that host | |
| 14:22:48 | efried | sean-k-mooney Ahhh, that makes sense. | |
| 14:23:16 | efried | sean-k-mooney So the aggregate is really just saying "neutron owns these". It's not associating the providers with any shared providers. | |
| 14:23:42 | efried | sean-k-mooney Which is a crucial distinction, because you wouldn't want an allocation candidate request to bleed across compute trees. | |
| 14:24:08 | efried | sean-k-mooney So in this scenario, the providers are associated with an aggregate, but nobody in that aggregate has the MISC_SHARES_VIA_AGGREGATE trait. | |
| 14:25:06 | sean-k-mooney | efried: yes exactly | |
| 14:26:10 | efried | mgoddard But note that report client is the pinch point for everything. You can't directly change the report client's cached ProviderTree from outside the report client. Conversely and more importantly, you can't effect a change to placement by changing that cached ProviderTree. You can only do that by calling report client methods like set_*_on_provider etc. | |
| 14:27:13 | efried | mgoddard Where it gets confusing is that ComputeDriver.update_provider_tree *sounds* like it's doing just that (effecting a change to placement by changing the cached ProviderTree). | |
| 14:27:22 | efried | mgoddard Which is almost but not quite the case. | |
| 14:28:26 | efried | mgoddard RT will ask RC for the provider tree; RC will return a *copy*. RT will then ask ComputeDriver to update_provider_tree. RT will then ask RC to diff and flush any changes back to placement, which will update RC's cached ProviderTree in the process. | |
| 14:30:01 | efried | sean-k-mooney Is there any reason that agg ID has to be the same across all the computes for which that neutron instance is handling the network providers? | |
| 14:30:29 | efried | sean-k-mooney I guess I can see the benefits either way. | |
| 14:30:58 | sean-k-mooney | efried: in this case i was assuming the aggregate would not span multiple computes | |
| 14:31:16 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add support to scheduler client for setting traits https://review.openstack.org/532539 | |
| 14:31:32 | sean-k-mooney | i am considering if aggregates can be used to group resouce providres in a singel compute that are releated in some way | |
| 14:32:42 | sean-k-mooney | efried: do you rememebr https://review.openstack.org/#/c/502306/14/specs/queens/approved/bandwidth-resource-provider.rst effectivly im wondering how best to remove the agent resouce providers so that we can also model numa affinity of networking resources | |
| 14:32:54 | mgoddard | efried: Ah, I think I see now. We can only get a 200 back if no other changes have occurred other than those we just PUT, therefore our provider tree must be up to date. Thanks for explaining | |
| 14:33:05 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Collocate encryptor and volume driver calls https://review.openstack.org/460243 | |
| 14:33:08 | lyarwood | mdbooth / stephenfin ; ^ one final respin with the comment and nits taken care of | |
| 14:36:00 | efried | mgoddard Just so. Given that report client is the pinch point for placement in nova, and there's only one report client, and it's running on one compute node, under only one thread, we shouldn't actually see these conflicts if nova is the only thingy managing all the providers. But as you can see from the conversation above, there are plans to potentially have multiple sources of ownership for a given provider. | |
| 14:36:56 | sean-k-mooney | efried: https://review.openstack.org/#/c/502306/14/specs/queens/approved/bandwidth-resource-provider.rst was assumign numa would either be a trait or we would have multiple parents. i like haveing numa nodes be resouce provider but that means neutron agent cant be if tehre sub resouce have numa affinity. | |
| 14:37:23 | efried | mgoddard BTW, I put a * above: today aggregates aren't included in the generation thing. cdent is working on fixing that. It's unclear whether that's going to get deferred to Rocky. IMO sean-k-mooney just described a non-sharing-RP use case where we need it. | |
| 14:38:00 | efried | sean-k-mooney Multiple parents, eh? How so? | |
| 14:38:22 | efried | sean-k-mooney I *think* the best way to model NUMA affinity is going to be using a numbered group with granular syntax. | |
| 14:38:52 | mriedem | yikun: if you wanted to start looking at something, this part of the spec hasn't been started https://specs.openstack.org/openstack/nova-specs/specs/queens/approved/neutron-new-port-binding-api.html | |
| 14:38:57 | mriedem | yikun: "Prior to the RPC call from live_migration on the source host to pre_live_migration on the dest host, start a wait thread for the vif-plugged event from Neutron, similar to during initial spawn." | |
| 14:39:43 | sean-k-mooney | efried: if resouce are allowed to have 1+ parents tehn one parent is the numa node and the other the neutron agent but that makes the trees a graph which we dont whant in general | |
| 14:41:44 | ameeda | anyone can help me to write functional regression test for this review.openstack.org/#/c/526900/ ? | |
| 14:42:34 | efried | sean-k-mooney Agree. The tree should be a tree (to quote cdent), the neutron agent can control the providers (at least the inventories thereof, if not the RP records themselves), and we can use aggregates to "tag" those providers as being controlled thusly. | |
| 14:42:46 | efried | sean-k-mooney An alternative approach for such tagging would be to use a trait. | |
| 14:43:16 | sean-k-mooney | efried: ya that was the other thing i was going to explore | |
| 14:43:31 | efried | sean-k-mooney Which might engender less confusion wrt sharing providers (which these ain't) as well as remove the requirement for the agg generation thing cdent is working. | |
| 14:43:36 | sean-k-mooney | e.g. CUSTOM_AGENT_<uuid goes here> | |
| 14:43:42 | cdent | efried: you're welcome to quote me efried, but you're taking me out of context. the implication i was trying to make there is that the trees should be big and anything can be a root | |
| 14:44:07 | efried | cdent Sorry, I take it back. Just didn't want to plagiarize :) | |
| 14:44:49 | efried | sean-k-mooney Or just CUSTOM_MANAGED_BY_NEUTRON. Why would you need a UUID in the trait? | |
| 14:44:51 | cdent | my utterances here are cca | |
| 14:45:16 | efried | I'll quote you on that. | |
| 14:45:23 | efried | (but not give you credit) | |
| 14:45:30 | sean-k-mooney | efried: again so i have a singel api call i can make to retrive all resouce provders that are managed by that agent | |
| 14:46:12 | sean-k-mooney | both aggreates via memberof and traits with the uuid give me that within the existing model | |
| 14:46:23 | efried | sean-k-mooney Isn't the scope of one neutron in this scenario the same as the scope of one placement service? | |
| 14:47:21 | sean-k-mooney | one placement service? is someone suggesting there would be more then one placement service? | |
| 14:47:42 | cdent | efried: sean's assertion was neutron agent, not neutron | |
| 14:48:01 | sean-k-mooney | oh yes what cdent said | |
| 14:48:03 | cdent | sean-k-mooney: in the dark corners of the universe there has been talk of nested placement services | |
| 14:48:18 | efried | (cdent that's not what I was talking about) | |
| 14:48:32 | efried | cdent sean-k-mooney In that case, there's one neutron agent per compute host, yah? | |
| 14:48:39 | sean-k-mooney | cdent: beyond one per cell that sound like i need something stronger then the coffee im drinking | |
| 14:49:00 | cdent | sean-k-mooney: yeah, it's the sort of thing where you should keep _me_ in drinks in dublin to not raise again | |