| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2019-09-19 | |||
| 03:06:07 | chenke | Great | |
| 03:06:24 | wangzhh | Cool | |
| 03:06:33 | Sundar | Not so good news: Our Nova patches did not enough reviews from Nova developers, and so did not make the cut. | |
| 03:07:24 | Sundar | Part of the problem is that, Cyborg patches were open for a long time, so Nova developers did not see it as ready, though we could put up a VM with Cyborg + Nova patches | |
| 03:08:16 | Sundar | Also, there was a longstanding request to show tempest CI working. That completed exactly in the milestone week. That was too late to get sustained reviews. | |
| 03:08:23 | shaohe_feng | We know intigration is a big effort | |
| 03:08:45 | shaohe_feng | Sundar: you d a lot of effort. Thanks | |
| 03:09:42 | chenke | It is understandable the patch in nova be merged slowly. | |
| 03:09:51 | Sundar | NP, thanks Shaohe. I am optimistic about U because I think we are close. and I have re-proposed the Nova spec. This time, tempest and most things are merged. Things that attratc croos-project attention, like tempest, privsep, sdk_adapter stuff, etc. are all done or making good progress | |
| 03:10:36 | Sundar | Hope to get the Nova patches in the runway very early in the cycle. The more we wait, the more things get bogged down among the tons of other reviews. | |
| 03:11:28 | Sundar | That said, we have a few more things to wrap up in Train :) | |
| 03:12:12 | Sundar | First, remove the hardcoding of 'dvstack-admin'. Thanks, chenker and all for addressing that :) | |
| 03:12:47 | Sundar | Second, v1 API is deprecated but still supported in Train. But it is not working because we removed all v1 from devstack. I should re-enable it, I think | |
| 03:13:28 | xinranwang | #info xinranwang | |
| 03:13:31 | Sundar | SHaohe's async bind, privsep, rbac are important | |
| 03:13:36 | xinranwang | Hi all | |
| 03:14:05 | Sundar | I think all the pep8/flake fixes from chenker/zhurong are looking good and will probably merge this week | |
| 03:14:33 | Sundar | Can you all think of anything else? | |
| 03:14:44 | Yumeng | Sundar: and please don't forget update device_profile db by conductor:https://review.opendev.org/#/c/679406/ | |
| 03:14:48 | Yumeng | just updated | |
| 03:15:06 | shaohe_feng | Sundar remain some slot for me to introduce the async jobs, so other's can easily to review it. | |
| 03:15:17 | shaohe_feng | Thanks | |
| 03:15:25 | Yumeng | and this gpu fix :https://review.opendev.org/#/c/675059/ I tested in my devstack env, it works | |
| 03:15:57 | Sundar | Ah yes, that too, Yumeng :) There are quite a few patches up there, including https://review.opendev.org/680953. | |
| 03:16:26 | Sundar | Sure, let's knock off as much as we can. Was just listing the ones critical to complete in Train | |
| 03:16:35 | openstackgerrit | Merged openstack/cyborg master: P5: Fix pep8 error in cyborg/accelerator https://review.opendev.org/679175 | |
| 03:16:54 | Sundar | shaohe_feng: Sure | |
| 03:17:15 | Sundar | Folks, anything else before we dive into Shaohe's async bind? | |
| 03:18:01 | s_shogo | I'm starting test&validation task, with real machine , begin with common functions, independet from specific accelerators. | |
| 03:18:11 | s_shogo | If extracted some bugs or erros, report that or post patches till the Train release. | |
| 03:19:36 | Sundar | Sure, s_shogo. I think the client effort can be aimed early in U release, since the Train release milestone for clients is past | |
| 03:19:41 | Sundar | I have some questions on RBAC: https://review.opendev.org/#/c/678177/ . In https://review.opendev.org/#/c/678177/3/cyborg/common/policy.py@83, should it be allow rule? ANybody can create an ARQ and thereby bind that ARQ, and so program an FPGA? | |
| 03:21:11 | s_shogo | Sundar: OK, I'll do the client&sdk task continuously, to the U release. | |
| 03:22:26 | Sundar | wangzhh: What do you think? | |
| 03:22:33 | xinranwang | should we complete v2 API in T? | |
| 03:23:09 | wangzhh | Sundar, it should be allowed and recheck it in the method if it is a program action or not. | |
| 03:23:29 | Sundar | wangzhh: ok | |
| 03:23:51 | Sundar | xinranwang: Only devices API remains. We are supposed to merge only bug fixes, I think. So, it will probably go to U. Is anything else remaining? | |
| 03:25:30 | Sundar | OK, 35 min remaining. Let's move to async bind. | |
| 03:25:41 | Sundar | #topic Async bind | |
| 03:25:54 | Sundar | Shaohe, take it away! | |
| 03:26:05 | shaohe_feng | Now let's we start to introduce async bind. Any questions can fafter the introduction. | |
| 03:26:12 | shaohe_feng | Briefly put, bind is to find a suitable device(maybe PCI, or MDEV) on the right host for a server instance to use. | |
| 03:26:18 | shaohe_feng | So what's the suitable device, we need a spec to describe it. | |
| 03:26:25 | shaohe_feng | On v1 we discribe the device directly on nova flavor extra spec, and cyborg parser the spec, Xinran implement this work. | |
| 03:26:32 | shaohe_feng | On v2, after the PTG discussion, we define it in cyborgs owen Device Pofile. And Sundar implement it. | |
| 03:26:43 | shaohe_feng | I have no chance to attend PTG for discussion, More details please talk with Sundar. | |
| 03:26:50 | shaohe_feng | Thans Xinran and Sundar's effor. | |
| 03:26:59 | shaohe_feng | Before we introduce async bind, let's know some implement(rules) in the current code firstly. | |
| 03:27:08 | shaohe_feng | profile group in order to get the expected devices. | |
| 03:27:08 | shaohe_feng | 1. The AtachHandler in ExtARQ is not a list, so only one AtachHandler(one devcie for ARQ) | |
| 03:27:19 | shaohe_feng | Now Our cyborg ARQ API bind API is sync, be we define it as async, so need to improve. | |
| 03:27:28 | shaohe_feng | So what we changed: | |
| 03:27:43 | shaohe_feng | 1. Use a thread pool to start the async job. | |
| 03:27:49 | shaohe_feng | In cyborg spec, sundar suggests use concurrent, yes it is a python stand lib. See python office link: | |
| 03:27:57 | shaohe_feng | https://docs.python.org/3/library/concurrent.futures.html | |
| 03:28:05 | shaohe_feng | Also we can greening it by greenlet. patched it by eventlet. | |
| 03:28:11 | shaohe_feng | utures = eventlet.import_patched('concurrent.futures') # 'greening' futures, | |
| 03:28:13 | openstackgerrit | Merged openstack/cyborg master: P6: Fix pep8 error in cyborg/agent and cyborg/db https://review.opendev.org/679193 | |
| 03:28:27 | shaohe_feng | easily to greening | |
| 03:28:37 | shaohe_feng | See python mail list discussion. | |
| 03:28:52 | shaohe_feng | I have simply test it, it can work, but I did not test it performance, do not enable greening in the patch. | |
| 03:29:00 | shaohe_feng | 2. I move out the bind logical from ExtARQ object. | |
| 03:29:13 | shaohe_feng | Let the ExtARQ maintain's its base function, such as its attribution's CRUD. | |
| 03:29:20 | shaohe_feng | Move it to cyborg/accelerator/common/handler.py (not sure this is a good place, this is a OPEN) | |
| 03:29:30 | shaohe_feng | Add a basic and general bind handle class named Accelerators. (not sure this is a good name, this is a OPEN) | |
| 03:29:37 | shaohe_feng | It support the base _bind | |
| 03:29:44 | shaohe_feng | https://review.opendev.org/#/c/681005/16/cyborg/accelerator/common/handler.py | |
| 03:29:54 | shaohe_feng | If a new acclerators need extra opeation, can derived it and extend it if needed, such as FPGA | |
| 03:30:02 | shaohe_feng | line 386 at | |
| 03:30:47 | shaohe_feng | For FPGA it need to get image metadata, download image, program image and update the placement. | |
| 03:31:18 | shaohe_feng | If _bind is time consume, use "wrap_job_tb" to wraper it. | |
| 03:31:30 | shaohe_feng | In this wraper I add it with "is_job" and can catch every Exception/traceback during bind process, then log it. | |
| 03:31:31 | openstackgerrit | Merged openstack/cyborg master: P7: Fix pep8 error in cyborg/objects and cyborg/image https://review.opendev.org/679526 | |
| 03:31:32 | openstackgerrit | Merged openstack/cyborg master: P8: Fix pep8 error in cyborg/tests and add post_mortem_debug.py https://review.opendev.org/679538 | |
| 03:31:38 | shaohe_feng | I also add a bind in the general class to start the jobs tagged with "is_job". | |
| 03:31:46 | shaohe_feng | I also add a master to monitor the jobs(as sundar suggestted) | |
| 03:31:52 | shaohe_feng | https://review.opendev.org/#/c/681005/16/cyborg/accelerator/common/handler.py | |
| 03:32:00 | shaohe_feng | It checks the jobs status and also will get the job Exception/traceback. | |
| 03:32:28 | shaohe_feng | please add a SUPPORT_RESOURCES in | |
| 03:32:41 | shaohe_feng | 4. I add ARQ_STATES_TRANSFORM_MATRIX to sync the status. | |
| 03:32:49 | shaohe_feng | Talked with sundar and xinran, we add extra status: ARQ_DELETING and ARQ_BIND_STARTED | |
| 03:32:57 | shaohe_feng | line at 29 | |
| 03:33:11 | shaohe_feng | I just refacor Sundar's effort. Do not change his logical, at present. So did not change any API define exposed to user. Thanks for Sundar's effort. | |
| 03:33:17 | shaohe_feng | I did not test multi/batch AQRs, for example, a request for 2 FPGAs, or 1 GPU and 1 FPGA. | |
| 03:33:21 | shaohe_feng | Have no really env. | |
| 03:33:50 | shaohe_feng | So I think we need to merge the patch, and let more developers test it. | |
| 03:34:00 | shaohe_feng | That's the different with VM management. Ironic or Cyborg sometimes need hardware, so it is difficult to manage. | |
| 03:34:39 | shaohe_feng | the commit message show you how to test this patch and | |
| 03:35:17 | shaohe_feng | analyze the process by log: https://review.opendev.org/#/c/681005/16//COMMIT_MSG | |
| 03:35:39 | shaohe_feng | Also there's still lot of works on it. Need to improve it continuously. Let it works firstly, then improvement. | |
| 03:36:31 | shaohe_feng | sorry | |
| 03:37:01 | shaohe_feng | any questions? | |
| 03:37:30 | Sundar | shaohe_feng: Thanks for all the time and hard work | |
| 03:38:19 | Sundar | For testing, hope people can use the fake driver. It supports FPGA resource class. Can we get it to take the programming patch but treat it as a no-op? | |
| 03:38:36 | Sundar | *programming code path | |
| 03:39:19 | shaohe_feng | Do you means make some mock do not really programming? | |
| 03:39:23 | Sundar | Yes | |
| 03:39:36 | shaohe_feng | Hardware support is really than VM | |
| 03:39:51 | Yumeng | shaohe_feng: that's really a comprehensive and deep research and very helpful introduction. | |