| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2020-04-16 | |||
| 03:33:51 | Sundar | brinzhang: The HTTP verbs are PUT, POST and PATCH. When you say update, which one do you mean? | |
| 03:34:22 | brinzhang | shaohe_feng, Sundar: ... PUT, soory | |
| 03:34:33 | Sundar | IMO, PATCH is the one that is closest in semantics to the programming operation. We are also using it for programming during ARQ binding. | |
| 03:35:06 | shaohe_feng | brinzhang you can read feilding's paper | |
| 03:35:42 | shaohe_feng | try to find which is more accurate | |
| 03:36:33 | brinzhang | I just think it's not very popular, if you all think that ok, I will be not concern this ^ | |
| 03:36:49 | songwenping | https://review.opendev.org/#/c/718584/ | |
| 03:36:55 | Sundar | PUT should be idempotent. PATCH need not be. The programming operation is not idempotent -- it can fail once and succeed the next time | |
| 03:37:26 | shaohe_feng | but the patch seems is newer method, I'm not sure feilding have discuss the patch in his paper. | |
| 03:38:09 | Sundar | brinzhang: I think you pointed me to some Nova oprrations. But Nova developers themselves criticized my patches when I followed their precedent. :) | |
| 03:38:20 | Sundar | *operations with PUT | |
| 03:39:01 | brinzhang | Sundar: I see that, and I have not looked into | |
| 03:39:11 | brinzhang | pls see https://review.opendev.org/#/c/718584/ | |
| 03:40:46 | brinzhang | and https://review.opendev.org/#/c/718584, that revert device and deployable when resource provider create fail | |
| 03:41:25 | Sundar | Does everybody think that the right approach is to delete/revert Cyborg objects when a Placement update fails? | |
| 03:41:30 | brinzhang | Sundar: did you review this patch? | |
| 03:41:37 | shaohe_feng | brinzhang why do you prefer PUT? | |
| 03:42:18 | songwenping | i am sorry my network is not good | |
| 03:42:45 | brinzhang | songwenping: Sundar said Does everybody think that the right approach is to delete/revert Cyborg objects when a Placement update fails? | |
| 03:43:36 | Sundar | xinranwang, Yumeng, s_shogo, shaohe_feng, all: Re. https://review.opendev.org/#/c/718584, Does everybody think that the right approach is to delete/revert Cyborg objects when a Placement update fails? | |
| 03:44:43 | xinranwang | IMO, It is reasonable. | |
| 03:45:16 | xinranwang | Otherwise, the conductor will not notice the diff and will now report to placement correctly. | |
| 03:45:20 | shaohe_feng | Let me think about it | |
| 03:45:46 | xinranwang | which will cause inconsistency | |
| 03:46:03 | Sundar | Ok. if you all decide that the best approach is to revert Cyborg db updates, then why not do Placement update first and update Cyborg db only if that succeeds? | |
| 03:46:18 | songwenping | xinranwang:right | |
| 03:47:02 | shaohe_feng | yes， there will be disaccord between placement and cyborg DB | |
| 03:47:48 | Sundar | Personally, I'd prefer to update Cyborg db always, and mark the objects that failed to sync with Placement. That way, operators will know that Cyborg did its job correctly and Placement failed. | |
| 03:48:03 | shaohe_feng | Sundar our design is do placement update first. | |
| 03:48:28 | shaohe_feng | we have discuss this issue in Dan smith's patch. | |
| 03:49:00 | brinzhang | Sundar: I saw you comment, I think your suggestion will be compex, but it is also an alternative way | |
| 03:49:20 | brinzhang | s/compex/complex | |
| 03:49:30 | shaohe_feng | do Placement update first and update Cyborg db only if that succeeds is the easiest way to keep consistent | |
| 03:49:46 | shaohe_feng | we have discuss it before. | |
| 03:50:26 | Sundar | Ok, I'll support your decision. Has it been tested fully in various scenarios, including Placement service down for some time and then comes up? | |
| 03:50:39 | shaohe_feng | bet another things: if placement DB successfully, but cyborg DB failed | |
| 03:50:51 | shaohe_feng | then what should cyborg do? | |
| 03:51:28 | xinranwang | Sundar: please see chenke's patch here: https://review.opendev.org/#/c/711912/2 | |
| 03:51:32 | songwenping | Sundar:no problem | |
| 03:51:41 | brinzhang | shaohe_feng: that will be raise > 500, I rembered chenke done this | |
| 03:52:06 | songwenping | i have test this scenario | |
| 03:52:10 | xinranwang | The commit message has explain the root reason of data imconsistency | |
| 03:53:02 | shaohe_feng | so I prefer make sure cyborg DB successfully first. | |
| 03:53:48 | Yumeng | agree we do the revert to handle the exception. and Let's add reconsidering placement report as a V release goal. whether we need to decouple placement report or not. | |
| 03:55:27 | xinranwang | If placement succeed and cyborg DB failed. The next time cyborg conductor do diff, it finds a diff, writes db and update placement with same info. I think the placment will return like "resourve provider already exists" something like this, But it will not crash the service. What do you think? | |
| 03:55:40 | shaohe_feng | sundar's comment: Personally, I'd prefer to update Cyborg db always, and mark the objects that failed to sync with Placement. That way, operators will know that Cyborg did its job correctly and Placement failed. | |
| 03:56:03 | brinzhang | this way just reduce the date inconsistence's scenario, that cannot cover all scenarios. | |
| 03:56:39 | chenke | actually. no one can cover all scenriaos. | |
| 03:56:43 | s_shogo | I also discussed the Placement-Cyborg interaction in enable/disable API , with xinranwang. | |
| 03:57:03 | shaohe_feng | we have issues(same to sudar's comments) this in Dan's patch, let us check, why not we give this? | |
| 03:57:08 | shaohe_feng | let me check it. | |
| 03:57:51 | shaohe_feng | another things: | |
| 03:58:31 | xinranwang | chenke's patch has fix several secenarios already. But as we said, it is difficult to cover all sceanrios. | |
| 03:58:32 | shaohe_feng | we need More accelerators drivers support | |
| 03:59:03 | xinranwang | I think we can revert like this patch does, and discuss the placement and db update decoupling in V. | |
| 03:59:20 | chenke | agree. | |
| 03:59:21 | brinzhang | if we add a flag to mark the data, I think we also need to add a period task to check this flag, and then do resource consistency | |
| 03:59:49 | Sundar | Our philosophy should be that Cyborg db is the source of knowledge about accelerators in any OpenStack cloud. If we let Placement take precedence, that reduces the value of Cyborg. That's just my opinion. As brinzhang said, it may be complex to make those changes. | |
| 04:00:10 | Sundar | brinzhang: we already have a periodc task from the agent | |
| 04:01:35 | shaohe_feng | we discuss it in: https://review.opendev.org/#/c/708726/ | |
| 04:02:02 | shaohe_feng | Sundar agree. | |
| 04:02:07 | shaohe_feng | cyborg db should firstly. | |
| 04:03:18 | shaohe_feng | but I see brinzhang agree Dan comments. | |
| 04:03:47 | brinzhang | yeah, I agree this way, but now we should do? | |
| 04:03:59 | Sundar | shaohe_feng: Dan Smith's suggestion may not apply well for Cyborg. For example, when an instance terminates, Placement resources get released. ARQs are unbound and deleted, but the unbinding may take some time if we do any device cleanup in the future. There is a period of time when the device is still not free (which Cyborg knows) but Placement | |
| 04:04:00 | Sundar | doesn't know. | |
| 04:04:22 | brinzhang | maybe we can make this as a plan in V release | |
| 04:04:37 | shaohe_feng | yes. This is a known and controversial issue | |
| 04:04:43 | shaohe_feng | I agree with you. | |
| 04:04:47 | brinzhang | songwenping's just make this as a bug fix | |
| 04:04:52 | Sundar | brinzhang and all: first, is it an important issue to fix in Ussuri? | |
| 04:04:57 | shaohe_feng | and I have discuss with xinranwang for it. | |
| 04:05:28 | shaohe_feng | we are all same options with you. | |
| 04:06:24 | shaohe_feng | make cyborg db correctly firstly? | |
| 04:06:40 | shaohe_feng | brinzhang agree? | |
| 04:07:29 | shaohe_feng | or you can list some pros and cons to make the conclusion | |
| 04:07:49 | xinranwang | The root reason is that now we coupling placement and db update, and we can not avoid this dependency. IMHO, we should discuss the decoupling in V. Now, we just make sure that there is less risk to have data inconsistency. | |
| 04:08:31 | chenke | +1 | |
| 04:08:32 | brinzhang | I am not sure whether is good for me, when would like to fix this issue? shaohe_feng | |
| 04:09:28 | brinzhang | xinranwang: agree +1 | |
| 04:09:47 | shaohe_feng | OK， let's xinranwang to make a good design firstly | |
| 04:09:59 | songwenping | xinranwang: agree +1 | |
| 04:10:06 | Sundar | xinranwang: are you voting to merge the patch now, and revisit it later? | |
| 04:10:34 | brinzhang | xinranwang's mean is to agree to merge this patch, then in V we should do Sundar suggestion, right? | |
| 04:10:40 | shaohe_feng | maybe a new spec for it firstly, and we can discuss it base on the new spec. | |
| 04:11:26 | xinranwang | I think we can merge it to let U have less risk of data inconsistency | |
| 04:12:13 | Sundar | all ok with that? | |
| 04:12:32 | Yumeng | agree +1 | |
| 04:12:53 | Sundar | shaohe_feng, s_shogo ^ | |
| 04:13:03 | s_shogo | agree, +1 | |
| 04:13:15 | chenke | some advice inline . | |
| 04:13:25 | chenke | https://review.opendev.org/#/c/718584/9/cyborg/conductor/manager.py | |
| 04:13:26 | brinzhang | Yumeng, xinranwang: please add this to V release plan | |
| 04:13:27 | xinranwang | I have thought about the decoupling a bit, there's lots of things to discuss,several gaps as I mentioned in last meeting. We should think more and discusss more to have a solution of decoupling. | |
| 04:13:32 | shaohe_feng | I give up vote. | |
| 04:13:55 | shaohe_feng | stay neutral | |
| 04:14:05 | Sundar | Ok, great. Please mention this in commit message of this patch. | |
| 04:14:36 | shaohe_feng | if we merge it, please leave details note in the patch | |
| 04:14:47 | brinzhang | xinranwang: can you leave comments in this patch? that songwenping can update that | |
| 04:14:57 | shaohe_feng | such as FIXME or NOTE in it. | |