| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-06-04 | |||
| 14:19:12 | shaohe_feng | ^ wangzhh, update the status. | |
| 14:19:23 | shaohe_feng | OK, next. | |
| 14:19:29 | wangzhh | shaohe_feng: OK | |
| 14:19:46 | shaohe_feng | SPDK, Helloway are you on line? | |
| 14:20:38 | Helloway | yes | |
| 14:21:15 | shaohe_feng | so any process on the SPDK? | |
| 14:22:37 | Helloway | Temporarily no | |
| 14:24:38 | shaohe_feng | Helloway, can you help to update the SPDK status on the #link https://etherpad.openstack.org/p/cyborg-driver-tasks ？ | |
| 14:24:58 | shaohe_feng | OK, next provider report. | |
| 14:25:52 | shaohe_feng | Sundar_, Do we support multi-resource class and nest provider in this release | |
| 14:26:09 | Sundar_ | Shaohe: IMHO, it is becoming risky | |
| 14:26:31 | shaohe_feng | Sundar_, what's the risky? | |
| 14:26:38 | Sundar_ | We have been waiting too long. Even in today's discussion in Nova scheduler meeting, it is not clear that it is going to come soon | |
| 14:27:05 | shaohe_feng | how should we do? | |
| 14:27:08 | Sundar_ | IMO, we should switch immediately to my originally proposed plan to use compute node as RP, till we get nRP | |
| 14:27:56 | Sundar_ | This means we can have multiple devices on the same host but they must all have the same types/traits | |
| 14:28:02 | shaohe_feng | OK. Can you have a discuss with jaypipes about cyborg's resource provider? | |
| 14:28:11 | shaohe_feng | during the summit? | |
| 14:28:30 | Sundar_ | E.g. 2 GPUs, 2 FPGAs both from Xilinx with same device family, 2 FPGAs both from Intel (say A10) etc | |
| 14:29:22 | Sundar_ | I discussed with edleafe etc., primarily on the comments that we should not have vendor/product/device names in traits. | |
| 14:29:40 | Sundar_ | That got resolved, because we do need vendor/device names, but not product names. The spec has been updated | |
| 14:29:57 | Sundar_ | On nRP, it is a larger discussion, that is still going on in the Nova community | |
| 14:30:09 | shaohe_feng | if we have a 1 Xilinx and 1 intel's FPGA on one host, what's the resource name? and what's the traits name? | |
| 14:30:33 | Sundar_ | We cannot have that unless we get nRPs. I'll explain why | |
| 14:30:47 | shaohe_feng | OK. please | |
| 14:31:04 | Sundar_ | For each device (GPU or FPGA), we will apply a trait like CUSTOM_GPU_AMD, CUSTOM_FPGA_INTEL | |
| 14:31:30 | Sundar_ | In your example, it will be CUSTOM_FPGA_INTEL and CUSTOM_FPGA_XILINX | |
| 14:31:55 | Sundar_ | We will also publish the resource class as CUSTOM_ACCELERATOR_FPGA | |
| 14:32:14 | edleafe | I would prefer 2 nested children, each with an inventory of 1 CUSTOM_FPGA. Then use traits to distinguish them | |
| 14:32:20 | Sundar_ | However, without nRP, it will get applied on the compute node, not on the individual devices | |
| 14:32:47 | Sundar_ | shaohe: that requires nRP. We may not get that soon enough to meet our Rocky goals. | |
| 14:33:26 | edleafe | There is currently discussion in placement on the potential upgrade problems when moving from non-nested to nested. It would be better to only plan on implementing on the nested model if possible | |
| 14:34:00 | Sundar_ | Without nRP, it will get applied on the compute node. So, the cpmpute node will advertise 2 units of CUSTOM_ACCELERATOR_FPGA, with 2 traits: CUSTOM_FPGA_INTEL and CUSTOM_FPGA_XILINX. But, Placement wil see that as 2 RCs each with 2 traits | |
| 14:34:23 | Sundar_ | There is no way to say that one unit of the RC has one trait, and the other has another trait | |
| 14:34:59 | Sundar_ | Does that make sense? | |
| 14:35:07 | edleafe | Sundar_: precisely. That's the main reason for waiting for nested | |
| 14:35:58 | shaohe_feng | edleafe, Sundar_ : there's issue in this way. | |
| 14:36:24 | Sundar_ | edleafe: Thanks for joining. My concern is, we are already in June and even today's Nova sched discussion indicates concerns with rolling upgrades and nRPs | |
| 14:36:46 | Sundar_ | It would be better to get something done with caveats, than nothing | |
| 14:37:23 | shaohe_feng | for example, user apply one CUSTOM_FPGA_XILINX, then inventory will remain one intel's CUSTOM_ACCELERATOR_FPGA | |
| 14:37:49 | shaohe_feng | then user still apply one CUSTOM_FPGA_XILINX, what's will go on? | |
| 14:38:04 | edleafe | But besides the issues noted with moving from non-nested to nested, Cyborg will also have to re-do a lot of the naming of custom resource classes | |
| 14:38:11 | Sundar_ | shaohe: traits go with resource providers (RPs), not resource classes (RCs) | |
| 14:39:55 | shaohe_feng | Sundar_, yes, that's issue. | |
| 14:40:41 | shaohe_feng | Sundar_, so the user still can apply a FPGA, but not XILINX that he expect. | |
| 14:44:16 | shaohe_feng | edleafe, any suggestion on this issue? | |
| 14:44:48 | Sundar_ | Hi | |
| 14:45:02 | openstackgerrit | wangzhh proposed openstack/cyborg master: Fix Deployable get_by_host https://review.openstack.org/572080 | |
| 14:45:16 | Sundar_ | Can you see my typing? | |
| 14:45:20 | shaohe_feng | Sundar_, welcome to come back | |
| 14:45:33 | shaohe_feng | Sundar_, just see "Hi" | |
| 14:45:37 | Sundar_ | I got blocked for some reason -- whatever I typed did not show up | |
| 14:45:49 | shaohe_feng | edleafe, Sundar_ , only support one kind of FPGA to avoid this issue? | |
| 14:46:13 | Sundar_ | Yes, omultiple devices ok, but all of the same type | |
| 14:46:30 | Sundar_ | We cannot have a GPU and a FPGA on the same host, or 2 kinds of FPGAs | |
| 14:46:39 | Sundar_ | That should be ok for Rocky | |
| 14:48:05 | edleafe | shaohe_feng: sorry, had to step away for a moment | |
| 14:48:39 | edleafe | shaohe_feng: FPGA is a resource *class*. GPU is a resource *class* | |
| 14:48:49 | edleafe | They should be modeled that way from the start. | |
| 14:49:01 | edleafe | specific types should be distinguished with traits | |
| 14:49:27 | Sundar_ | edleafe: That is exactly what we are doing | |
| 14:49:54 | edleafe | Sundar_: good. I really would hate to see things like CUSTOM_FPGA_XILINX | |
| 14:50:14 | Sundar_ | It is just that, without nRPs, we will apply the traits to the compute node, for Rocky cycle alone. That means, all devices on the same host must have the same traits. | |
| 14:50:37 | Sundar_ | edleafe: CUSTOM_FPGA_XILINX would be a trait on the RP, not a RC | |
| 14:50:39 | shaohe_feng | CUSTOM_FPGA_XILINX is resource or traits? | |
| 14:50:55 | edleafe | We are still hoping to have NRP complete in Rocky | |
| 14:51:02 | Sundar_ | shaohe: trait, not RC | |
| 14:51:25 | edleafe | ah, that wasn't clear. In context I thought you were using it as an RC | |
| 14:51:39 | Sundar_ | edleafe: I understand, but with 2 months left to go, I think we are risking Cyborg's Rocky goals by waiting further | |
| 14:52:42 | edleafe | Sure, that's understandable. I just want to make sure you know that that will make moving to an NRP design later harder | |
| 14:53:08 | Sundar_ | What makes the switch to nRPs hard? | |
| 14:53:54 | Sundar_ | The set of RCs and traits will stay the same. But we will apply the traits to individual device RPs later | |
| 14:54:50 | edleafe | You will have inventory of devices for the compute node. When you upgrade, somehow these must be converted to a nested design, and any existing instances that are using those resources will have to have their allocations moved. That is the current upgrade discussion going on. | |
| 14:55:42 | edleafe | Remember, allocations are for an instance against the RP. When you move to nested, you now have a new child RP, and the allocations should be against it | |
| 14:56:28 | edleafe | But they will be against the compute node for any existing instances at the time of upgrade. How to reconcile all of this correctly is what we are trying to work out now | |
| 14:58:31 | shaohe_feng | OK | |
| 14:58:48 | Sundar_ | OK. There are 2 options: (a) Do not support expectations of upgrades for Cyborg in rocky (IMHO, Rocky addresses the basic Cyborg use cases and lays a strong foundation for further development) (b) Support upgrades by providing some mechanism to delete traits on compute node at a safe time (would appreciate your input here) | |
| 15:00:23 | Sundar_ | edleafe: What do you think? | |
| 15:00:24 | shaohe_feng | Sundar_, can you summary it, and we can get conclusion on Wednesday's meeting | |
| 15:00:56 | shaohe_feng | ^ edleafe, any suggestion on it? | |
| 15:01:00 | Sundar_ | shaohe: Above is the summary :). What additional info do you want? | |
| 15:01:14 | edleafe | You would probably have to write some custom upgrade script to iterate over all machines to find old-style traits and allocations, convert them to the new style, and then delete the old ones | |
| 15:01:55 | shaohe_feng | Sundar_, does these 2 options are mutually exclusive? | |
| 15:02:20 | Sundar_ | edleafe: On an upgrade, the new agent/driver(s) will automatically create nested RPs and apply traits there, while the old traits on the compute node still exist | |
| 15:03:21 | Sundar_ | Can we then delete the traits on the compute node, while instances are still running? | |
| 15:04:14 | Sundar_ | If so, we can provide a script that the operator must run after upgrade, which deletes Cyborg traits on compute nodes | |
| 15:05:14 | Sundar_ | shaohe: they are exclusive. | |
| 15:06:29 | edleafe | Sundar_: Traits aren't the issue. Allocations / Inventory is what is important to update | |
| 15:07:15 | edleafe | Otherwise, a compute node will have, say, inventory of 2 FPGAs, and will now have two child RPs with an FPGA inventory | |
| 15:07:38 | edleafe | In that example, Placement will now see 4 FPGAs on the one compute node. | |
| 15:08:06 | Sundar_ | edleafe: True. May be the upgrade script can set the reserved to total for the compute node RP? | |
| 15:09:07 | edleafe | That's one way. The other would be to simply delete that inventory, since it really isn't the compute node's anymore | |
| 15:09:44 | edleafe | Allocations will also have to be adjusted, because having a double allocation has an impact on things like measuring quotas | |
| 15:09:44 | Sundar_ | Can we do that while the instances are still using that inventory? | |
| 15:10:09 | edleafe | Sundar_: :) Now you | |
| 15:10:12 | edleafe | oops | |
| 15:10:33 | edleafe | Now you're seeing why I don't like to go the non-nested to nested path | |
| 15:10:43 | Sundar_ | Can we do that while the instances are still using that inventory? | |
| 15:11:03 | Sundar_ | i.e. delete the inventory on compute node RPs | |
| 15:11:05 | edleafe | There are a lot of issues, and we're tyring to come up with a generic solution that will work for Cyborg as well as things like NUMA | |