| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2017-03-29 | |||
| 15:08:26 | zhipeng | if we have any outstanding issues, please feel free to shout out | |
| 15:11:16 | jkilpatr | do we want to consolidate all DB interaction to the api end point (presumably on the controllers) or have the agent on the computes update the database themselves. | |
| 15:11:49 | jkilpatr | essentially that's a question of where to handle communication and parallelism. | |
| 15:13:08 | zhipeng | if we choose the latter one, does it mean the controller side will be sometimes out of sync with the agents ? | |
| 15:14:11 | jkilpatr | it would make it a concern, it would be possible to prevent if we tried. | |
| 15:14:32 | jkilpatr | on the other hand if the agent's don't store anything themselves then we can lose accelerator state if anything disrupts the agent. | |
| 15:17:21 | zhipeng | how could we prevent the out-of-sync problem ? using heartbeat ? | |
| 15:17:44 | crushil | I would prefer the latter option> And I believe heartbeat can be an option | |
| 15:18:02 | jkilpatr | just make sure both sides refresh info from the DB when it might have changed. So proper cache invalidation. | |
| 15:18:07 | jkilpatr | and probably more DB load | |
| 15:18:33 | crushil | But the agent should definitely keep its database updated | |
| 15:19:06 | zhipeng | but in real deployment that always hard to manage, I always hear complaints from our product team about the heartbeat | |
| 15:19:21 | zhipeng | cache invalidation is prong to go wrong | |
| 15:19:58 | zhipeng | is there a way for us to simplify so that we could avoid the common shortcomings | |
| 15:20:25 | jkilpatr | so agent <-> rabbit <-> api end point <-> db | |
| 15:20:54 | jkilpatr | that way we don't need invalidation because the end point is the only one that interacts with the DB | |
| 15:22:18 | crushil | Isn't that overkill though? Going through hoops to update db? | |
| 15:22:35 | crushil | Can | |
| 15:23:05 | crushil | Can't we have agent maintain a local copy of the db and agent keeps updating it? | |
| 15:23:52 | zhipeng | but if say we have 3 compute node that all have FPGA based iNIC on it | |
| 15:24:01 | jkilpatr | why does the agent need info about all accelerators? it only needs to manage one machine (there are many agents on many compute nodes, but each one only needs to care about it's own scope) I guess they could have a mini db each but what good is that. | |
| 15:24:07 | zhipeng | if the DB is updated only via local copy | |
| 15:24:52 | zhipeng | i'm terminating my thoughts here~~ | |
| 15:25:04 | crushil | Well, I feel that the agent needs to have a more efficient way of updating the db | |
| 15:26:24 | jkilpatr | maybe we just have the agent update the db for things that are longer term? Like a new accelerator? | |
| 15:26:52 | zhipeng | how about the config changes | |
| 15:27:03 | zhipeng | that would be part of the life cycle mgmt usually | |
| 15:28:09 | zhipeng | or we have two seperate DBs (sorta), the main one resides with the control node | |
| 15:28:13 | jkilpatr | I guess agents need to read from the db for config updates, we don't want to obligate the operator to ssh into every node to change a config value. | |
| 15:28:28 | zhipeng | like jkilpatr it only get updated on major events | |
| 15:28:38 | zhipeng | and agent locally manages a cache | |
| 15:28:49 | zhipeng | for the more constant changes ? | |
| 15:29:40 | crushil | That seems fine, although how would you define major events? | |
| 15:29:49 | zhipeng | the info in the locan cache should be of no concern of either other compute nodes or the control node | |
| 15:30:23 | zhipeng | like when accelerator attached, created, deleted | |
| 15:30:47 | zhipeng | and for small changes | |
| 15:30:57 | zhipeng | agent could advertise to the api-endpoint | |
| 15:31:16 | zhipeng | as a bulk of its local cache info, if it deemed important by the control node | |
| 15:31:44 | zhipeng | so what I'm think is that user could explicitly define events that they care | |
| 15:31:52 | zhipeng | in the cyborg.conf for example | |
| 15:32:13 | zhipeng | and those events will be aggregated and reported by the agents to the api-end-point | |
| 15:32:21 | crushil | So, you're suggesting maintaining a central db and a local cached db. The central db would be updated on all events and cached db will be updated only on select events? | |
| 15:32:26 | zhipeng | alongside the usual big event | |
| 15:32:47 | ttk2[m] | Jkilpatr here had to go mobile. | |
| 15:33:22 | zhipeng | central/main DB will get updated on all the major events (attach/detach/delete/create) and smaller events that user specified | |
| 15:33:46 | zhipeng | the local copy is just cache for all the local updates, refreshed constantly | |
| 15:33:55 | zhipeng | that is my thinking | |
| 15:34:08 | _gryf | zhipeng, +1 for the cache instead of full blown db | |
| 15:34:37 | ttk2[m] | Sounds like a workable compromise. | |
| 15:34:56 | crushil | that sounds ok to me | |
| 15:35:36 | zhipeng | great :) then the DB design would remain the same, and there will be an additional design on the local cache for the agent | |
| 15:35:57 | zhipeng | is this ttk2[m] or crushil's ? | |
| 15:36:44 | ttk2[m] | I'll do the agent cache. | |
| 15:37:14 | zhipeng | fantastic :) | |
| 15:37:15 | crushil | cool | |
| 15:37:30 | zhipeng | #action Justin to update with the agent cache design | |
| 15:37:46 | ttk2[m] | Should I put that in the same blueprint? | |
| 15:37:59 | zhipeng | i think you could do that | |
| 15:38:12 | zhipeng | it should not be a big deal | |
| 15:40:33 | zhipeng | okey any other topics ? | |
| 15:40:50 | zhipeng | any questions on Roman's BP re the interaction between Nova and Cyborg | |
| 15:43:19 | crushil | nothing from me | |
| 15:44:02 | zhipeng | well let's keep review the spec anyways :P | |
| 15:44:09 | zhipeng | #topic AoB | |
| 15:44:17 | zhipeng | any other buisness ? | |
| 15:44:36 | ttk2[m] | Do we want to set a blueprint finish goal? | |
| 15:44:52 | zhipeng | ttk[m] what do you mean ? for a deadline date ? | |
| 15:44:58 | _gryf | ttk2[m], like a deadline or smth? | |
| 15:45:14 | ttk2[m] | Yes. Just something to keep things moving. | |
| 15:45:42 | _gryf | in other projects that depends on how big is the core reviewers pipe | |
| 15:46:10 | zhipeng | i would love to set a deadline lol | |
| 15:46:28 | _gryf | in nova there are deadlines, so they can be sure, that core reviewers spend quality time on desired and accepted features | |
| 15:46:35 | zhipeng | what we called in China is the lazy cancer kicked in all the time :P | |
| 15:47:03 | _gryf | otoh, I've seen projects, (smaller ones), that they obey only openstack release schedule | |
| 15:47:45 | zhipeng | I think setting a deadline/milestone would be a good idea | |
| 15:47:54 | _gryf | that fine. | |
| 15:47:56 | zhipeng | so that everyone on the same page and pace | |
| 15:48:05 | zhipeng | others ? | |
| 15:48:20 | ttk2[m] | As long as it's not too tight. | |
| 15:48:22 | crushil | So, for current specs, should we say EOR should be the deadline? | |
| 15:48:28 | zhipeng | (if we have a milestone then maybe we need a sprint someday ...) | |
| 15:48:41 | zhipeng | crushil for specs that would be too relax | |
| 15:48:49 | zhipeng | we need to get code in for Pike | |
| 15:48:59 | crushil | I meant implementation of the specs | |
| 15:49:07 | zhipeng | yes that'd agre | |
| 15:49:18 | zhipeng | I would agree | |
| 15:49:23 | ttk2[m] | Let's just put a deadline on specs for now. | |
| 15:49:29 | zhipeng | Apr 15th ? | |
| 15:49:46 | ttk2[m] | That's what 3 meetings? | |
| 15:49:57 | crushil | lol | |
| 15:50:02 | zhipeng | sounds about right | |
| 15:50:10 | zhipeng | too relax or too tight ? lol | |
| 15:50:40 | crushil | I think for most of the specs, the owners know what needs to be added | |
| 15:50:57 | crushil | So, it is slightly on the more relaxed side | |
| 15:51:01 | zhipeng | it is the reviews that takes time | |
| 15:51:25 | zhipeng | _gryf we are not that famous project so we could move faster XD | |
| 15:51:25 | crushil | Fair | |
| 15:51:31 | _gryf | which obviously means slipping to the next release :> | |
| 15:51:40 | crushil | Let's do April 15th then | |
| 15:51:52 | ttk2[m] | +1 | |