| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2020-04-16 | |||
| 03:49:30 | shaohe_feng | do Placement update first and update Cyborg db only if that succeeds is the easiest way to keep consistent | |
| 03:49:46 | shaohe_feng | we have discuss it before. | |
| 03:50:26 | Sundar | Ok, I'll support your decision. Has it been tested fully in various scenarios, including Placement service down for some time and then comes up? | |
| 03:50:39 | shaohe_feng | bet another things: if placement DB successfully, but cyborg DB failed | |
| 03:50:51 | shaohe_feng | then what should cyborg do? | |
| 03:51:28 | xinranwang | Sundar: please see chenke's patch here: https://review.opendev.org/#/c/711912/2 | |
| 03:51:32 | songwenping | Sundar:no problem | |
| 03:51:41 | brinzhang | shaohe_feng: that will be raise > 500, I rembered chenke done this | |
| 03:52:06 | songwenping | i have test this scenario | |
| 03:52:10 | xinranwang | The commit message has explain the root reason of data imconsistency | |
| 03:53:02 | shaohe_feng | so I prefer make sure cyborg DB successfully first. | |
| 03:53:48 | Yumeng | agree we do the revert to handle the exception. and Let's add reconsidering placement report as a V release goal. whether we need to decouple placement report or not. | |
| 03:55:27 | xinranwang | If placement succeed and cyborg DB failed. The next time cyborg conductor do diff, it finds a diff, writes db and update placement with same info. I think the placment will return like "resourve provider already exists" something like this, But it will not crash the service. What do you think? | |
| 03:55:40 | shaohe_feng | sundar's comment: Personally, I'd prefer to update Cyborg db always, and mark the objects that failed to sync with Placement. That way, operators will know that Cyborg did its job correctly and Placement failed. | |
| 03:56:03 | brinzhang | this way just reduce the date inconsistence's scenario, that cannot cover all scenarios. | |
| 03:56:39 | chenke | actually. no one can cover all scenriaos. | |
| 03:56:43 | s_shogo | I also discussed the Placement-Cyborg interaction in enable/disable API , with xinranwang. | |
| 03:57:03 | shaohe_feng | we have issues(same to sudar's comments) this in Dan's patch, let us check, why not we give this? | |
| 03:57:08 | shaohe_feng | let me check it. | |
| 03:57:51 | shaohe_feng | another things: | |
| 03:58:31 | xinranwang | chenke's patch has fix several secenarios already. But as we said, it is difficult to cover all sceanrios. | |
| 03:58:32 | shaohe_feng | we need More accelerators drivers support | |
| 03:59:03 | xinranwang | I think we can revert like this patch does, and discuss the placement and db update decoupling in V. | |
| 03:59:20 | chenke | agree. | |
| 03:59:21 | brinzhang | if we add a flag to mark the data, I think we also need to add a period task to check this flag, and then do resource consistency | |
| 03:59:49 | Sundar | Our philosophy should be that Cyborg db is the source of knowledge about accelerators in any OpenStack cloud. If we let Placement take precedence, that reduces the value of Cyborg. That's just my opinion. As brinzhang said, it may be complex to make those changes. | |
| 04:00:10 | Sundar | brinzhang: we already have a periodc task from the agent | |
| 04:01:35 | shaohe_feng | we discuss it in: https://review.opendev.org/#/c/708726/ | |
| 04:02:02 | shaohe_feng | Sundar agree. | |
| 04:02:07 | shaohe_feng | cyborg db should firstly. | |
| 04:03:18 | shaohe_feng | but I see brinzhang agree Dan comments. | |
| 04:03:47 | brinzhang | yeah, I agree this way, but now we should do? | |
| 04:03:59 | Sundar | shaohe_feng: Dan Smith's suggestion may not apply well for Cyborg. For example, when an instance terminates, Placement resources get released. ARQs are unbound and deleted, but the unbinding may take some time if we do any device cleanup in the future. There is a period of time when the device is still not free (which Cyborg knows) but Placement | |
| 04:04:00 | Sundar | doesn't know. | |
| 04:04:22 | brinzhang | maybe we can make this as a plan in V release | |
| 04:04:37 | shaohe_feng | yes. This is a known and controversial issue | |
| 04:04:43 | shaohe_feng | I agree with you. | |
| 04:04:47 | brinzhang | songwenping's just make this as a bug fix | |
| 04:04:52 | Sundar | brinzhang and all: first, is it an important issue to fix in Ussuri? | |
| 04:04:57 | shaohe_feng | and I have discuss with xinranwang for it. | |
| 04:05:28 | shaohe_feng | we are all same options with you. | |
| 04:06:24 | shaohe_feng | make cyborg db correctly firstly? | |
| 04:06:40 | shaohe_feng | brinzhang agree? | |
| 04:07:29 | shaohe_feng | or you can list some pros and cons to make the conclusion | |
| 04:07:49 | xinranwang | The root reason is that now we coupling placement and db update, and we can not avoid this dependency. IMHO, we should discuss the decoupling in V. Now, we just make sure that there is less risk to have data inconsistency. | |
| 04:08:31 | chenke | +1 | |
| 04:08:32 | brinzhang | I am not sure whether is good for me, when would like to fix this issue? shaohe_feng | |
| 04:09:28 | brinzhang | xinranwang: agree +1 | |
| 04:09:47 | shaohe_feng | OK， let's xinranwang to make a good design firstly | |
| 04:09:59 | songwenping | xinranwang: agree +1 | |
| 04:10:06 | Sundar | xinranwang: are you voting to merge the patch now, and revisit it later? | |
| 04:10:34 | brinzhang | xinranwang's mean is to agree to merge this patch, then in V we should do Sundar suggestion, right? | |
| 04:10:40 | shaohe_feng | maybe a new spec for it firstly, and we can discuss it base on the new spec. | |
| 04:11:26 | xinranwang | I think we can merge it to let U have less risk of data inconsistency | |
| 04:12:13 | Sundar | all ok with that? | |
| 04:12:32 | Yumeng | agree +1 | |
| 04:12:53 | Sundar | shaohe_feng, s_shogo ^ | |
| 04:13:03 | s_shogo | agree, +1 | |
| 04:13:15 | chenke | some advice inline . | |
| 04:13:25 | chenke | https://review.opendev.org/#/c/718584/9/cyborg/conductor/manager.py | |
| 04:13:26 | brinzhang | Yumeng, xinranwang: please add this to V release plan | |
| 04:13:27 | xinranwang | I have thought about the decoupling a bit, there's lots of things to discuss,several gaps as I mentioned in last meeting. We should think more and discusss more to have a solution of decoupling. | |
| 04:13:32 | shaohe_feng | I give up vote. | |
| 04:13:55 | shaohe_feng | stay neutral | |
| 04:14:05 | Sundar | Ok, great. Please mention this in commit message of this patch. | |
| 04:14:36 | shaohe_feng | if we merge it, please leave details note in the patch | |
| 04:14:47 | brinzhang | xinranwang: can you leave comments in this patch? that songwenping can update that | |
| 04:14:57 | shaohe_feng | such as FIXME or NOTE in it. | |
| 04:15:14 | brinzhang | That TODO(), I think | |
| 04:15:17 | Sundar | Anything else, folks? We are over the time. | |
| 04:15:19 | xinranwang | shaohe_feng: yes, understand, it is a hard decision, we have a long way to go lol... | |
| 04:15:30 | xinranwang | brinzhang: sure | |
| 04:15:47 | brinzhang | xinranwang: thanks | |
| 04:16:07 | brinzhang | it's time to launch, end meetting? | |
| 04:16:13 | shaohe_feng | OK. | |
| 04:16:28 | shaohe_feng | more drivers are welcome this new release right? | |
| 04:16:36 | Sundar | Great. Thanks a lot, everybody! Great progress this cycle. We have another week to make even more contributions. Take care and stay safe! | |
| 04:16:39 | Sundar | #endmeeting | |
| 04:16:41 | openstack | Meeting ended Thu Apr 16 04:16:39 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 04:16:43 | openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2020/openstack_cyborg.2020-04-16-03.02.html | |
| 04:16:44 | openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_cyborg/2020/openstack_cyborg.2020-04-16-03.02.txt | |
| 04:16:45 | openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2020/openstack_cyborg.2020-04-16-03.02.log.html | |
| 04:16:47 | shaohe_feng | ^ Sundar | |
| 04:17:27 | Sundar | shaohe_feng: Yes, newer drivers are always welcome, hopefully with CI support so that they can be maintained. | |
| 06:29:37 | openstackgerrit | YumengBao proposed openstack/cyborg master: Refactor v2 arq api https://review.opendev.org/696089 | |
| 06:57:32 | openstackgerrit | Shogo Saito proposed openstack/cyborg master: Programming support (v2 Deployable API) https://review.opendev.org/698190 | |
| 09:07:47 | openstackgerrit | Merged openstack/cyborg master: Refactor v2 arq api https://review.opendev.org/696089 | |
| 10:35:20 | openstackgerrit | Wenping Song proposed openstack/cyborg master: revert device and deployable when resource provider create fail https://review.opendev.org/718584 | |
| 11:54:38 | openstackgerrit | Wenping Song proposed openstack/cyborg master: revert device and deployable when resource provider create fail https://review.opendev.org/718584 | |
| 12:38:16 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: Ascend driver:[B602:subprocess_popen_with_shell_equals_true] https://review.opendev.org/720456 | |
| 12:42:59 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: Ascend driver:[B602:subprocess_popen_with_shell_equals_true] https://review.opendev.org/720456 | |
| 12:43:56 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: [B104:hardcoded_bind_all_interfaces] https://review.opendev.org/720149 | |
| 13:25:24 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: SPDK driver:[B602:subprocess_popen_with_shell_equals_true] https://review.opendev.org/720475 | |
| 13:48:51 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: [B108:hardcoded_tmp_directory] https://review.opendev.org/720143 | |
| 13:48:51 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: [B104:hardcoded_bind_all_interfaces] https://review.opendev.org/720149 | |
| 13:48:52 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: Ascend driver:[B602:subprocess_popen_with_shell_equals_true] https://review.opendev.org/720456 | |
| 13:48:52 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: SPDK driver:[B602:subprocess_popen_with_shell_equals_true] https://review.opendev.org/720475 | |
| 13:48:53 | openstackgerrit | YumengBao proposed openstack/cyborg master: Change bandit job from non-voting to voting https://review.opendev.org/720479 | |
| #openstack-cyborg - 2020-04-17 | |||
| 00:10:44 | openstackgerrit | Brin Zhang proposed openstack/cyborg master: revert device and deployable when resource provider create fail https://review.opendev.org/718584 | |
| 03:14:28 | openstackgerrit | YumengBao proposed openstack/cyborg master: Fix bandit error: SPDK driver:[B602:subprocess_popen_with_shell_equals_true] https://review.opendev.org/720475 | |