| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2018-06-04 | |||
| 15:09:07 | edleafe | That's one way. The other would be to simply delete that inventory, since it really isn't the compute node's anymore | |
| 15:09:44 | Sundar_ | Can we do that while the instances are still using that inventory? | |
| 15:09:44 | edleafe | Allocations will also have to be adjusted, because having a double allocation has an impact on things like measuring quotas | |
| 15:10:09 | edleafe | Sundar_: :) Now you | |
| 15:10:12 | edleafe | oops | |
| 15:10:33 | edleafe | Now you're seeing why I don't like to go the non-nested to nested path | |
| 15:10:43 | Sundar_ | Can we do that while the instances are still using that inventory? | |
| 15:11:03 | Sundar_ | i.e. delete the inventory on compute node RPs | |
| 15:11:05 | edleafe | There are a lot of issues, and we're tyring to come up with a generic solution that will work for Cyborg as well as things like NUMA | |
| 15:11:40 | shaohe_feng | edleafe, does provider support NUMA at present? | |
| 15:12:11 | shaohe_feng | edleafe, how do we organizate | |
| 15:12:21 | edleafe | shaohe_feng: yes, but in a non-nested way | |
| 15:12:59 | edleafe | We're trying to figure out how to do that upgrade, and it doesn't look easy | |
| 15:13:58 | shaohe_feng | one in numa node0, another in numa node1 | |
| 15:14:38 | edleafe | The way NUMA has been modeled in Nova is a horrible hack that was done before Placement even existed | |
| 15:14:40 | Sundar_ | May be it is simplest to go with option a: for upgrades with Cyborg, first stop all instances using accelerators, run a script that cleans up, then upgrade to new Cyborg. For other subsystems, I understand the issue. But, Cyborg is a new and we can set expectations | |
| 15:15:14 | edleafe | Sundar_: yes, that is a luxury that a generic solution wouldn't have | |
| 15:16:39 | shaohe_feng | edleafe, ^ any suggestion on the ultima solution for cyborg accelerator numa topo ? | |
| 15:18:22 | edleafe | shaohe_feng: I would think it would look something like: compute_node -> NUMA node -> FPGA device -> FPGA regions - > FPGA function | |
| 15:18:52 | edleafe | But from what I know of NUMA, you can configure it multiple ways. | |
| 15:20:43 | shaohe_feng | compute_node is provider, NUMA node is provider, FPGA device , FPGA regions, FPGA function are all provider? | |
| 15:20:49 | shaohe_feng | ^ edleafe, | |
| 15:21:15 | shaohe_feng | Sundar_, have consider numa topo for FPGA | |
| 15:21:23 | edleafe | shaohe_feng: yes. The only inventory is the function, which is what the user wants | |
| 15:21:23 | shaohe_feng | ? | |
| 15:21:51 | shaohe_feng | edleafe, ok, got it. 4 level provider. | |
| 15:21:53 | shaohe_feng | thanks | |
| 15:22:30 | Sundar_ | shaohe: kinda. But, my suggestion is to focus on the basics for Rocky. If we try to throw in everything, we will not deliver anything | |
| 15:23:01 | edleafe | shaohe_feng: of course, it doesn't *have to* be that way, but it is one possibility | |
| 15:24:31 | shaohe_feng | edleafe, thanks again. | |
| 15:24:50 | shaohe_feng | Sundar_, let's go ahead? | |
| 15:25:00 | shaohe_feng | for next task? | |
| 15:25:29 | Sundar_ | Yes, I will send an email to openstack-dev, proposing this (lack of) upgrade plan | |
| 15:25:55 | shaohe_feng | Sundar_, OK, please, thanks. | |
| 15:26:06 | shaohe_feng | next is Intel FPGA driver | |
| 15:26:40 | shaohe_feng | I see the owner is rorojo | |
| 15:26:58 | shaohe_feng | Sundar_, rorojo is not on line | |
| 15:27:09 | shaohe_feng | can you help to sync with him? | |
| 15:27:31 | Sundar_ | Yes. Discussions are ongoing about the implementation | |
| 15:27:40 | shaohe_feng | great. | |
| 15:27:46 | shaohe_feng | any update there? | |
| 15:28:35 | Sundar_ | Nothing significant. I helped Rodrigo to get started, with code browsing etc | |
| 15:28:57 | Sundar_ | I had issues with devstack myself :), but I got that fixed, so I have a working deployment on MCP now | |
| 15:29:20 | Sundar_ | I am now working on agent-driver API update | |
| 15:29:59 | shaohe_feng | Sundar_, Oh, we have improve the devstack doc | |
| 15:30:08 | shaohe_feng | let me find it for you. | |
| 15:31:02 | shaohe_feng | #link https://docs.openstack.org/cyborg/latest/contributor/devstack_setup.html#devstack-quick-start | |
| 15:31:06 | Sundar_ | The issue was not with Cyborg plugin. I was hitting version conflicts on various components in oslo etc. | |
| 15:31:38 | shaohe_feng | OK, there's a bug on oslo components? | |
| 15:32:13 | Sundar_ | I had to do a 'pip install --upgrade' on many components, because they were lower version than the minimum | |
| 15:32:37 | Sundar_ | Then, I took eveything in /opt/stack/requirements/lower_constraints.text and tried to do a mass upgrade | |
| 15:33:06 | shaohe_feng | should we submit a patch to upgrade the cyborg requirement? | |
| 15:33:09 | Sundar_ | That failed because some components need Python 3. So, I excluded some things manually, and otherwise hacked, till it worked | |
| 15:34:02 | Sundar_ | The issue is, devstack does not seem to upgrade the components to their minimum versions automatically. | |
| 15:34:08 | openstackgerrit | wangzhh proposed openstack/cyborg master: Fix Deployable get_by_host https://review.openstack.org/572080 | |
| 15:34:43 | Sundar_ | This is not Cyborg-specific | |
| 15:35:31 | shaohe_feng | FPGA programing, Li_liu is not on line. let's skip it. | |
| 15:35:54 | shaohe_feng | HPTS, yumeng is not online, skip it. | |
| 15:36:12 | shaohe_feng | Cyborg/Nova interaction, Sundar_ | |
| 15:36:33 | Sundar_ | You may have seen the spec updates | |
| 15:36:46 | Sundar_ | Waiting for approval | |
| 15:36:50 | wangzhh | Sundar_: Maybe you can try PIP_UPGRADE=True in local.conf | |
| 15:37:22 | Sundar_ | wangzhh: Thanks, will try it next time | |
| 15:37:40 | shaohe_feng | wangzhh, thanks. can you submit a patch for our developer guider? | |
| 15:38:13 | Guest4480 | Hi Sundar, there are one problem we faced in our environment: suppose there are two GPU in one host, when attaching one gpu to vm, if this attachment is failed, will nova try to attach the second gpu in this host to vm? | |
| 15:38:23 | shaohe_feng | wangzhh, also with other doc bugs. #link https://docs.openstack.org/cyborg/latest/contributor/devstack_setup.html#devstack-quick-start | |
| 15:38:37 | wangzhh | It's an common config. Do we need to maintain it? | |
| 15:39:04 | shaohe_feng | wangzhh, maybe we can add some note for developer. | |
| 15:39:05 | Guest4480 | sorry to interrupt. | |
| 15:39:18 | shaohe_feng | Guest4480, it does not matter. | |
| 15:39:18 | wangzhh | OK. | |
| 15:39:33 | shaohe_feng | Guest4480, go ahead. | |
| 15:39:55 | shaohe_feng | Sundar_, Guest4480 ask help from you. | |
| 15:40:12 | Sundar_ | Guest4480: since they are independent resources, failure to attach one should not affect the other one. | |
| 15:41:49 | shaohe_feng | next task. | |
| 15:42:26 | shaohe_feng | Cyborg/Nova interaction(nova side), Sundar_ any plan for it. Should we move it in next release? | |
| 15:42:58 | Sundar_ | What does that mean? | |
| 15:43:15 | shaohe_feng | Sundar_, I think you will talk with jaypipes, edleafe and other nova developer's about it. | |
| 15:43:24 | Sundar_ | Do you mean, nova compute calling into os-acc? | |
| 15:43:55 | Sundar_ | Yes, we need to follow up. We have an os-acc spec, which is awaiting approval | |
| 15:44:36 | Sundar_ | I will update the specs once more by this week | |
| 15:44:53 | shaohe_feng | zhipengh[m], zhuli_ will works on os-acc. | |
| 15:45:04 | edleafe | Sundar_: ping me in -nova when they are ready | |
| 15:45:30 | Sundar_ | edleafe: Sure. Thanks | |
| 15:45:43 | shaohe_feng | Sundar_, but we still need volunteers working on nova side. | |
| 15:45:50 | Guest4480 | OK, we will follow the spec to see where to add our work(code) which has be done. | |
| 15:46:12 | edleafe | shaohe_feng: I can help with the nova side | |
| 15:46:55 | Sundar_ | Excellent, edleafe. Nova compute needs to call into os-acc for specific instance events, as documented in os-acc spec | |
| 15:47:11 | Sundar_ | #link https://review.openstack.org/#/c/566798/ | |
| 15:47:18 | shaohe_feng | edleafe, thanks. | |
| 15:48:22 | shaohe_feng | Sundar_, you can have more details with edleafe. | |
| 15:48:30 | Sundar_ | Yes | |
| 15:48:36 | shaohe_feng | thanks. | |
| 15:48:43 | shaohe_feng | next task. | |
| 15:48:45 | shaohe_feng | Quota for accelerator | |
| 15:48:51 | shaohe_feng | xinran__, are you on line? | |
| 15:49:14 | xinran__ | Yes | |
| 15:49:15 | Sundar_ | I need to drop off in 5 minutes | |
| 15:49:20 | xinran__ | I’m here | |
| 15:49:25 | shaohe_feng | OK. | |