| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-06-04 | |||
| 14:37:49 | shaohe_feng | then user still apply one CUSTOM_FPGA_XILINX, what's will go on? | |
| 14:38:04 | edleafe | But besides the issues noted with moving from non-nested to nested, Cyborg will also have to re-do a lot of the naming of custom resource classes | |
| 14:38:11 | Sundar_ | shaohe: traits go with resource providers (RPs), not resource classes (RCs) | |
| 14:39:55 | shaohe_feng | Sundar_, yes, that's issue. | |
| 14:40:41 | shaohe_feng | Sundar_, so the user still can apply a FPGA, but not XILINX that he expect. | |
| 14:44:16 | shaohe_feng | edleafe, any suggestion on this issue? | |
| 14:44:48 | Sundar_ | Hi | |
| 14:45:02 | openstackgerrit | wangzhh proposed openstack/cyborg master: Fix Deployable get_by_host https://review.openstack.org/572080 | |
| 14:45:16 | Sundar_ | Can you see my typing? | |
| 14:45:20 | shaohe_feng | Sundar_, welcome to come back | |
| 14:45:33 | shaohe_feng | Sundar_, just see "Hi" | |
| 14:45:37 | Sundar_ | I got blocked for some reason -- whatever I typed did not show up | |
| 14:45:49 | shaohe_feng | edleafe, Sundar_ , only support one kind of FPGA to avoid this issue? | |
| 14:46:13 | Sundar_ | Yes, omultiple devices ok, but all of the same type | |
| 14:46:30 | Sundar_ | We cannot have a GPU and a FPGA on the same host, or 2 kinds of FPGAs | |
| 14:46:39 | Sundar_ | That should be ok for Rocky | |
| 14:48:05 | edleafe | shaohe_feng: sorry, had to step away for a moment | |
| 14:48:39 | edleafe | shaohe_feng: FPGA is a resource *class*. GPU is a resource *class* | |
| 14:48:49 | edleafe | They should be modeled that way from the start. | |
| 14:49:01 | edleafe | specific types should be distinguished with traits | |
| 14:49:27 | Sundar_ | edleafe: That is exactly what we are doing | |
| 14:49:54 | edleafe | Sundar_: good. I really would hate to see things like CUSTOM_FPGA_XILINX | |
| 14:50:14 | Sundar_ | It is just that, without nRPs, we will apply the traits to the compute node, for Rocky cycle alone. That means, all devices on the same host must have the same traits. | |
| 14:50:37 | Sundar_ | edleafe: CUSTOM_FPGA_XILINX would be a trait on the RP, not a RC | |
| 14:50:39 | shaohe_feng | CUSTOM_FPGA_XILINX is resource or traits? | |
| 14:50:55 | edleafe | We are still hoping to have NRP complete in Rocky | |
| 14:51:02 | Sundar_ | shaohe: trait, not RC | |
| 14:51:25 | edleafe | ah, that wasn't clear. In context I thought you were using it as an RC | |
| 14:51:39 | Sundar_ | edleafe: I understand, but with 2 months left to go, I think we are risking Cyborg's Rocky goals by waiting further | |
| 14:52:42 | edleafe | Sure, that's understandable. I just want to make sure you know that that will make moving to an NRP design later harder | |
| 14:53:08 | Sundar_ | What makes the switch to nRPs hard? | |
| 14:53:54 | Sundar_ | The set of RCs and traits will stay the same. But we will apply the traits to individual device RPs later | |
| 14:54:50 | edleafe | You will have inventory of devices for the compute node. When you upgrade, somehow these must be converted to a nested design, and any existing instances that are using those resources will have to have their allocations moved. That is the current upgrade discussion going on. | |
| 14:55:42 | edleafe | Remember, allocations are for an instance against the RP. When you move to nested, you now have a new child RP, and the allocations should be against it | |
| 14:56:28 | edleafe | But they will be against the compute node for any existing instances at the time of upgrade. How to reconcile all of this correctly is what we are trying to work out now | |
| 14:58:31 | shaohe_feng | OK | |
| 14:58:48 | Sundar_ | OK. There are 2 options: (a) Do not support expectations of upgrades for Cyborg in rocky (IMHO, Rocky addresses the basic Cyborg use cases and lays a strong foundation for further development) (b) Support upgrades by providing some mechanism to delete traits on compute node at a safe time (would appreciate your input here) | |
| 15:00:23 | Sundar_ | edleafe: What do you think? | |
| 15:00:24 | shaohe_feng | Sundar_, can you summary it, and we can get conclusion on Wednesday's meeting | |
| 15:00:56 | shaohe_feng | ^ edleafe, any suggestion on it? | |
| 15:01:00 | Sundar_ | shaohe: Above is the summary :). What additional info do you want? | |
| 15:01:14 | edleafe | You would probably have to write some custom upgrade script to iterate over all machines to find old-style traits and allocations, convert them to the new style, and then delete the old ones | |
| 15:01:55 | shaohe_feng | Sundar_, does these 2 options are mutually exclusive? | |
| 15:02:20 | Sundar_ | edleafe: On an upgrade, the new agent/driver(s) will automatically create nested RPs and apply traits there, while the old traits on the compute node still exist | |
| 15:03:21 | Sundar_ | Can we then delete the traits on the compute node, while instances are still running? | |
| 15:04:14 | Sundar_ | If so, we can provide a script that the operator must run after upgrade, which deletes Cyborg traits on compute nodes | |
| 15:05:14 | Sundar_ | shaohe: they are exclusive. | |
| 15:06:29 | edleafe | Sundar_: Traits aren't the issue. Allocations / Inventory is what is important to update | |
| 15:07:15 | edleafe | Otherwise, a compute node will have, say, inventory of 2 FPGAs, and will now have two child RPs with an FPGA inventory | |
| 15:07:38 | edleafe | In that example, Placement will now see 4 FPGAs on the one compute node. | |
| 15:08:06 | Sundar_ | edleafe: True. May be the upgrade script can set the reserved to total for the compute node RP? | |
| 15:09:07 | edleafe | That's one way. The other would be to simply delete that inventory, since it really isn't the compute node's anymore | |
| 15:09:44 | Sundar_ | Can we do that while the instances are still using that inventory? | |
| 15:09:44 | edleafe | Allocations will also have to be adjusted, because having a double allocation has an impact on things like measuring quotas | |
| 15:10:09 | edleafe | Sundar_: :) Now you | |
| 15:10:12 | edleafe | oops | |
| 15:10:33 | edleafe | Now you're seeing why I don't like to go the non-nested to nested path | |
| 15:10:43 | Sundar_ | Can we do that while the instances are still using that inventory? | |
| 15:11:03 | Sundar_ | i.e. delete the inventory on compute node RPs | |
| 15:11:05 | edleafe | There are a lot of issues, and we're tyring to come up with a generic solution that will work for Cyborg as well as things like NUMA | |
| 15:11:40 | shaohe_feng | edleafe, does provider support NUMA at present? | |
| 15:12:11 | shaohe_feng | edleafe, how do we organizate | |
| 15:12:21 | edleafe | shaohe_feng: yes, but in a non-nested way | |
| 15:12:59 | edleafe | We're trying to figure out how to do that upgrade, and it doesn't look easy | |
| 15:13:58 | shaohe_feng | one in numa node0, another in numa node1 | |
| 15:14:38 | edleafe | The way NUMA has been modeled in Nova is a horrible hack that was done before Placement even existed | |
| 15:14:40 | Sundar_ | May be it is simplest to go with option a: for upgrades with Cyborg, first stop all instances using accelerators, run a script that cleans up, then upgrade to new Cyborg. For other subsystems, I understand the issue. But, Cyborg is a new and we can set expectations | |
| 15:15:14 | edleafe | Sundar_: yes, that is a luxury that a generic solution wouldn't have | |
| 15:16:39 | shaohe_feng | edleafe, ^ any suggestion on the ultima solution for cyborg accelerator numa topo ? | |
| 15:18:22 | edleafe | shaohe_feng: I would think it would look something like: compute_node -> NUMA node -> FPGA device -> FPGA regions - > FPGA function | |
| 15:18:52 | edleafe | But from what I know of NUMA, you can configure it multiple ways. | |
| 15:20:43 | shaohe_feng | compute_node is provider, NUMA node is provider, FPGA device , FPGA regions, FPGA function are all provider? | |
| 15:20:49 | shaohe_feng | ^ edleafe, | |
| 15:21:15 | shaohe_feng | Sundar_, have consider numa topo for FPGA | |
| 15:21:23 | edleafe | shaohe_feng: yes. The only inventory is the function, which is what the user wants | |
| 15:21:23 | shaohe_feng | ? | |
| 15:21:51 | shaohe_feng | edleafe, ok, got it. 4 level provider. | |
| 15:21:53 | shaohe_feng | thanks | |
| 15:22:30 | Sundar_ | shaohe: kinda. But, my suggestion is to focus on the basics for Rocky. If we try to throw in everything, we will not deliver anything | |
| 15:23:01 | edleafe | shaohe_feng: of course, it doesn't *have to* be that way, but it is one possibility | |
| 15:24:31 | shaohe_feng | edleafe, thanks again. | |
| 15:24:50 | shaohe_feng | Sundar_, let's go ahead? | |
| 15:25:00 | shaohe_feng | for next task? | |
| 15:25:29 | Sundar_ | Yes, I will send an email to openstack-dev, proposing this (lack of) upgrade plan | |
| 15:25:55 | shaohe_feng | Sundar_, OK, please, thanks. | |
| 15:26:06 | shaohe_feng | next is Intel FPGA driver | |
| 15:26:40 | shaohe_feng | I see the owner is rorojo | |
| 15:26:58 | shaohe_feng | Sundar_, rorojo is not on line | |
| 15:27:09 | shaohe_feng | can you help to sync with him? | |
| 15:27:31 | Sundar_ | Yes. Discussions are ongoing about the implementation | |
| 15:27:40 | shaohe_feng | great. | |
| 15:27:46 | shaohe_feng | any update there? | |
| 15:28:35 | Sundar_ | Nothing significant. I helped Rodrigo to get started, with code browsing etc | |
| 15:28:57 | Sundar_ | I had issues with devstack myself :), but I got that fixed, so I have a working deployment on MCP now | |
| 15:29:20 | Sundar_ | I am now working on agent-driver API update | |
| 15:29:59 | shaohe_feng | Sundar_, Oh, we have improve the devstack doc | |
| 15:30:08 | shaohe_feng | let me find it for you. | |
| 15:31:02 | shaohe_feng | #link https://docs.openstack.org/cyborg/latest/contributor/devstack_setup.html#devstack-quick-start | |
| 15:31:06 | Sundar_ | The issue was not with Cyborg plugin. I was hitting version conflicts on various components in oslo etc. | |
| 15:31:38 | shaohe_feng | OK, there's a bug on oslo components? | |