| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2017-03-22 | |||
| 15:42:13 | zhipeng | so Boston Summit will have Forum sessions | |
| 15:42:34 | zhipeng | suppose to be a substitute for the good old design summit | |
| 15:43:02 | zhipeng | I don't think we need a dedicated Cyborg session, we could piggyback on many of the available sessions | |
| 15:43:17 | zhipeng | one is, again, from SWG | |
| 15:43:30 | zhipeng | #link https://etherpad.openstack.org/p/BOS-UC-brainstorming-scientific-wg | |
| 15:44:24 | zhipeng | or do we need a cross-project session with Nova ? | |
| 15:46:39 | jkilpatr | zhipeng, I think we should have a firm grasp of what we need to do with nova and what we want and who's going to do it before we do a cross team session | |
| 15:46:58 | zhipeng | jkilpatr good suggestion | |
| 15:46:59 | jkilpatr | so maybe a couple of weeks from now? when these blueprints are mature? | |
| 15:47:08 | zhipeng | I agree | |
| 15:47:30 | crushil | zhipeng: I agree with jkilpatr. We need to have things streamlined on our end before we propose such sessions | |
| 15:48:08 | zhipeng | Nova will have a placement session anyway, so we could use that occasion even if we don't propose our session | |
| 15:48:16 | zhipeng | crushil agree :) | |
| 15:48:43 | zhipeng | #topic AoB | |
| 15:49:09 | zhipeng | any other buisnees before I could set the miserable bot loose ? lol | |
| 15:52:35 | zhipeng | okey I guess we could conclude the meeting early today | |
| 15:52:44 | zhipeng | thx everyone for participation ! | |
| 15:52:51 | zhipeng | #endmeeting | |
| 15:52:53 | openstack | Meeting ended Wed Mar 22 15:52:51 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 15:52:54 | openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2017/openstack_cyborg.2017-03-22-15.02.html | |
| 15:52:56 | openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_cyborg/2017/openstack_cyborg.2017-03-22-15.02.txt | |
| 15:52:57 | openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2017/openstack_cyborg.2017-03-22-15.02.log.html | |
| 15:53:07 | NokMikeR | thanks all | |
| 15:53:28 | zhipeng | got the full log, not bad :P | |
| 15:53:34 | NokMikeR | "its alive" | |
| 15:53:43 | NokMikeR | good its working. | |
| 15:54:11 | zhipeng | it is just running in da background :P | |
| #openstack-cyborg - 2017-03-25 | |||
| 00:31:52 | natra | hi | |
| 01:23:45 | zhipengh[m] | Hey | |
| #openstack-cyborg - 2017-03-29 | |||
| 14:53:37 | jkilpatr | anyone alive in here? | |
| 14:54:10 | jkilpatr | oh man I should check on my commits more often. | |
| 14:54:21 | NokMikeR | barely | |
| 14:54:52 | zhipengh[m] | lol | |
| 15:00:18 | zhipeng | #startmeeting openstack-cyborg | |
| 15:00:19 | openstack | Meeting started Wed Mar 29 15:00:18 2017 UTC and is due to finish in 60 minutes. The chair is zhipeng. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 15:00:20 | openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 15:00:22 | openstack | The meeting name has been set to 'openstack_cyborg' | |
| 15:00:38 | zhipeng | #topic Roll Call | |
| 15:00:49 | crushil | \o | |
| 15:01:09 | zhipeng | \o | |
| 15:01:22 | jkilpatr | o/ | |
| 15:02:20 | zhipeng | waiting for more folks if we have | |
| 15:03:35 | skelso | o/ | |
| 15:03:47 | mpaolino | \o | |
| 15:06:51 | zhipeng | okey let's proceed | |
| 15:07:02 | zhipeng | #topic BP review | |
| 15:07:12 | zhipeng | #link https://review.openstack.org/#/q/project:openstack/cyborg | |
| 15:07:44 | zhipeng | i must appologize i was buried with kubecon work this week so haven't got the time for the reviews | |
| 15:08:02 | zhipeng | i see _gryf posted a lot of good comments | |
| 15:08:10 | zhipeng | as well as rushil and others | |
| 15:08:26 | zhipeng | if we have any outstanding issues, please feel free to shout out | |
| 15:11:16 | jkilpatr | do we want to consolidate all DB interaction to the api end point (presumably on the controllers) or have the agent on the computes update the database themselves. | |
| 15:11:49 | jkilpatr | essentially that's a question of where to handle communication and parallelism. | |
| 15:13:08 | zhipeng | if we choose the latter one, does it mean the controller side will be sometimes out of sync with the agents ? | |
| 15:14:11 | jkilpatr | it would make it a concern, it would be possible to prevent if we tried. | |
| 15:14:32 | jkilpatr | on the other hand if the agent's don't store anything themselves then we can lose accelerator state if anything disrupts the agent. | |
| 15:17:21 | zhipeng | how could we prevent the out-of-sync problem ? using heartbeat ? | |
| 15:17:44 | crushil | I would prefer the latter option> And I believe heartbeat can be an option | |
| 15:18:02 | jkilpatr | just make sure both sides refresh info from the DB when it might have changed. So proper cache invalidation. | |
| 15:18:07 | jkilpatr | and probably more DB load | |
| 15:18:33 | crushil | But the agent should definitely keep its database updated | |
| 15:19:06 | zhipeng | but in real deployment that always hard to manage, I always hear complaints from our product team about the heartbeat | |
| 15:19:21 | zhipeng | cache invalidation is prong to go wrong | |
| 15:19:58 | zhipeng | is there a way for us to simplify so that we could avoid the common shortcomings | |
| 15:20:25 | jkilpatr | so agent <-> rabbit <-> api end point <-> db | |
| 15:20:54 | jkilpatr | that way we don't need invalidation because the end point is the only one that interacts with the DB | |
| 15:22:18 | crushil | Isn't that overkill though? Going through hoops to update db? | |
| 15:22:35 | crushil | Can | |
| 15:23:05 | crushil | Can't we have agent maintain a local copy of the db and agent keeps updating it? | |
| 15:23:52 | zhipeng | but if say we have 3 compute node that all have FPGA based iNIC on it | |
| 15:24:01 | jkilpatr | why does the agent need info about all accelerators? it only needs to manage one machine (there are many agents on many compute nodes, but each one only needs to care about it's own scope) I guess they could have a mini db each but what good is that. | |
| 15:24:07 | zhipeng | if the DB is updated only via local copy | |
| 15:24:52 | zhipeng | i'm terminating my thoughts here~~ | |
| 15:25:04 | crushil | Well, I feel that the agent needs to have a more efficient way of updating the db | |
| 15:26:24 | jkilpatr | maybe we just have the agent update the db for things that are longer term? Like a new accelerator? | |
| 15:26:52 | zhipeng | how about the config changes | |
| 15:27:03 | zhipeng | that would be part of the life cycle mgmt usually | |
| 15:28:09 | zhipeng | or we have two seperate DBs (sorta), the main one resides with the control node | |
| 15:28:13 | jkilpatr | I guess agents need to read from the db for config updates, we don't want to obligate the operator to ssh into every node to change a config value. | |
| 15:28:28 | zhipeng | like jkilpatr it only get updated on major events | |
| 15:28:38 | zhipeng | and agent locally manages a cache | |
| 15:28:49 | zhipeng | for the more constant changes ? | |
| 15:29:40 | crushil | That seems fine, although how would you define major events? | |
| 15:29:49 | zhipeng | the info in the locan cache should be of no concern of either other compute nodes or the control node | |
| 15:30:23 | zhipeng | like when accelerator attached, created, deleted | |
| 15:30:47 | zhipeng | and for small changes | |
| 15:30:57 | zhipeng | agent could advertise to the api-endpoint | |
| 15:31:16 | zhipeng | as a bulk of its local cache info, if it deemed important by the control node | |
| 15:31:44 | zhipeng | so what I'm think is that user could explicitly define events that they care | |
| 15:31:52 | zhipeng | in the cyborg.conf for example | |
| 15:32:13 | zhipeng | and those events will be aggregated and reported by the agents to the api-end-point | |
| 15:32:21 | crushil | So, you're suggesting maintaining a central db and a local cached db. The central db would be updated on all events and cached db will be updated only on select events? | |
| 15:32:26 | zhipeng | alongside the usual big event | |
| 15:32:47 | ttk2[m] | Jkilpatr here had to go mobile. | |
| 15:33:22 | zhipeng | central/main DB will get updated on all the major events (attach/detach/delete/create) and smaller events that user specified | |
| 15:33:46 | zhipeng | the local copy is just cache for all the local updates, refreshed constantly | |
| 15:33:55 | zhipeng | that is my thinking | |
| 15:34:08 | _gryf | zhipeng, +1 for the cache instead of full blown db | |
| 15:34:37 | ttk2[m] | Sounds like a workable compromise. | |
| 15:34:56 | crushil | that sounds ok to me | |