| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2017-05-12 | |||
| 17:16:19 | gryf | there has to be at least one of the accelerators available for test | |
| 17:16:25 | gryf | so basically - yeah | |
| 17:16:30 | gryf | bare metal | |
| 17:16:56 | jkilpatr | I have baremetal Ci running now, but it's a pricey endevor. I guess we will have to have unit tests and mostly trust them. | |
| 17:19:37 | gryf | in neutorn or cinder they shift burden of the drivers testing to vendors | |
| 17:19:40 | gryf | afairc | |
| 17:20:37 | gryf | so that it is possible to jump with your own accelerator and driver, but you have to provide appropriate test env | |
| 17:29:26 | jkilpatr | gryf, might be most cost feasible to get a blade server with pci slots in the blades and just get hardware donations of accelerators. But that's far far in the future, implementation now | |
| 17:29:43 | jkilpatr | zhipengh[m], you're doing the api right? thoughts on starting that? I'll start the conductor at the same time so we can colab | |
| 17:29:58 | jkilpatr | who's got the dummy driver again? | |
| 17:31:30 | gryf | jkilpatr, well, sure. | |
| #openstack-cyborg - 2017-05-13 | |||
| 09:52:39 | zhipengh[m] | driver is on rushil | |
| 09:52:45 | zhipengh[m] | And yes I will do the api | |
| #openstack-cyborg - 2017-05-16 | |||
| 14:47:48 | jkilpatr | so questions I need to remember to ask at tomorrows meeting | |
| 14:48:21 | jkilpatr | 1. the current API doesn't have an "attach" or anything like that, so how do we spawn accelerator using instances? | |
| 14:48:38 | jkilpatr | 2. For Nova, can we do live PCI attachment? | |
| 14:49:18 | jkilpatr | 3. Depending on 1 what is the exact attachment workflow we want like instance on machine then we reboot it and attach it, or do we emulate the current scripts by having conductor hit the nova api often based on agents | |
| 14:49:52 | jkilpatr | 4. Database selection 5. implementation start this month? | |
| #openstack-cyborg - 2017-05-17 | |||
| 14:56:47 | jkilpatr | morning everyone | |
| 14:57:15 | crushil | Morning jkilpatr | |
| 15:00:11 | crushil | \o | |
| 15:00:26 | crushil | Meeting? | |
| 15:03:08 | jkilpatr | meeting, you know zhiping | |
| 15:03:21 | jkilpatr | his internet probably decided not to wake up this morning. | |
| 15:03:31 | zhipeng | #startmeeting openstack-cyborg | |
| 15:03:31 | openstack | Meeting started Wed May 17 15:03:31 2017 UTC and is due to finish in 60 minutes. The chair is zhipeng. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 15:03:32 | openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 15:03:34 | openstack | The meeting name has been set to 'openstack_cyborg' | |
| 15:03:38 | jkilpatr | ah speak the devil. | |
| 15:03:41 | zhipeng | just got connected | |
| 15:03:43 | zhipeng | :P | |
| 15:03:59 | jkilpatr | ok then I had a list of topics to go over | |
| 15:04:08 | jkilpatr | unless someone wants to bring other things up before I steamroll head? | |
| 15:04:11 | jkilpatr | ahead* | |
| 15:04:29 | zhipeng | #topic BP discussion | |
| 15:04:38 | zhipeng | jkilpatrt go ahead | |
| 15:05:55 | jkilpatr | zhipeng, your outlined Cyborg api doesn't actually have an attach api call. What did you plan to do catch some property of a booting instance? | |
| 15:07:03 | zhipeng | as far as I could tell I did include the attach api ? | |
| 15:07:04 | jkilpatr | nova *can* do pci hotplug, supposedly. which would make live attachment *possible* but I have no grasp on the gap between "it should work" and "it actually will" | |
| 15:08:34 | jkilpatr | so get have get, post, put ,delete but they all seem to be for managing the database of accelerators, you can put to update a accelerator spec but what if I have an instance I need to attach it to | |
| 15:09:35 | zhipeng | Okey for VM instance, I still think we would have to look for Nova for the actual attachment operation | |
| 15:09:59 | zhipeng | actually Jay and I discussed about this today | |
| 15:10:29 | zhipeng | unless we have identify a set of properties for the accelerator connection to the host | |
| 15:10:48 | zhipeng | meaning that we have a os-brick like library | |
| 15:11:16 | zhipeng | unless we have that, then we could just assume it is a regular PCI-e attachment | |
| 15:12:06 | zhipeng | I think for most of the use cases we have at the moment, PCIe would be the most usual case and Nova currently support that | |
| 15:13:08 | jkilpatr | I think we need to come to a workflow conclusion sooner rather than later, how does the user ask for a vm with a specific accelerator, do they post to us and then we talk to nova, do they post to nova and we watch for some tag etc | |
| 15:13:34 | jkilpatr | should there be like a 'user workflow' spec? I guess it goes into the api | |
| 15:14:01 | zhipeng | workflow is kinda an end-to-end thing | |
| 15:14:46 | zhipeng | I personally would suggest for the moment, we take the Cinder approach, and expand later base upon this | |
| 15:15:02 | jkilpatr | that meaning live attachment? | |
| 15:15:13 | zhipeng | because this might be the only way that we don't create a major impact on the existing implementations | |
| 15:15:42 | zhipeng | not necessary live-attachment | |
| 15:15:51 | zhipeng | but just attach.detach ops in general | |
| 15:16:00 | zhipeng | for VM instances | |
| 15:16:15 | zhipeng | since you will need the instance id and host id anyway | |
| 15:16:22 | zhipeng | and nova got all these | |
| 15:16:26 | jkilpatr | ok then, we have a booted vm we need to do the attachment, we can setup nova to do the passthrough and then reboot the instance, not sure how nova feels about it but if it doesn't work I don't think they would be opposed to us making that work | |
| 15:16:37 | jkilpatr | but also nova does support live pci attachment, so we can/should just use that | |
| 15:17:07 | zhipeng | yes if Nova indeed could support that | |
| 15:17:14 | jkilpatr | https://wiki.openstack.org/wiki/Nova/pci_hotplug | |
| 15:17:25 | jkilpatr | this is a stub right now, so some support | |
| 15:17:28 | zhipeng | we just don't differentiate in Cyborg | |
| 15:18:14 | jkilpatr | the driver actually handles this of course, we just run attach on a live instance that was spawned with a flavor or some other indicator that says "cyborg: TeslaP100" | |
| 15:18:59 | zhipeng | yes | |
| 15:19:21 | jkilpatr | because we have to make sure Nova gets it to the right spot firs, so we need to have the placement api fed live knowlege and then the instance needs to call the resource name we fed the placement api | |
| 15:19:40 | jkilpatr | maybe we can have a command in cyborg that will help you make flavors, but that's for later. | |
| 15:19:51 | zhipeng | hotplug could be the trait when we use placement api | |
| 15:20:33 | zhipeng | so when we schedule it, it know it needs a compute node with hotplug feature | |
| 15:20:46 | jkilpatr | wouldn't we want a bunch of traits | |
| 15:20:58 | zhipeng | yep I guess so :) | |
| 15:21:03 | jkilpatr | like gpu, with cuda support, with hotplug support .... so on and so forth | |
| 15:21:10 | zhipeng | we could start with really basic and simple ones | |
| 15:21:11 | jkilpatr | this is why we need a flavor creation wizard in cyborg but once again, later. | |
| 15:21:25 | zhipeng | yes | |
| 15:21:34 | jkilpatr | ok anyone have comments on these ideas? things that they might want that this won't cover? | |
| 15:22:15 | jkilpatr | zhipeng, I'm going to put a comment on your api patch as a reminder to add cinder like attach/detach to the spec sound good? | |
| 15:23:00 | zhipeng | which I thought is already done in the current patch ? | |
| 15:23:27 | cdent | it would be great to see, at some point, a narration of the expected end to end flow from a user's standpoint, if it doesn't already exist. Including how various services will be touched. | |
| 15:24:17 | zhipeng | cdent we got a flow chart in our BOS presentation, but rather rudimentary at the moment | |
| 15:24:20 | jkilpatr | cdent, we have a decent idea of how we want it to work but I expect some things will change as we get into the nitty gritty of placement problems | |
| 15:24:42 | cdent | sure, change is the nature of this stuff :) | |
| 15:24:47 | jkilpatr | zhipeng, ok so if I want to attach an accelerator to an instance what do I do? Do I put to update an accelerator spec with a new instance ID to attach to? | |
| 15:24:48 | zhipeng | :) | |
| 15:25:18 | jkilpatr | cdent, I think I'll put up a user workflow spec later today, just so that we keep track of all of this better. | |
| 15:25:30 | cdent | \o/ | |
| 15:25:48 | cdent | can you add me as a review on that when it is up, so I get some email to remind me to look? | |
| 15:26:23 | zhipeng | just as we drew for our presentation, after the user using Cyborg service to complete the discovery phase and Cyborg finishing interaction with placement to advertise the accelerator inventory | |
| 15:26:24 | jkilpatr | will do, whats your email? | |
| 15:26:44 | cdent | cdent@anticdent.org is me on gerrit | |
| 15:27:01 | zhipeng | then user just request to create an instance on a compute node with the corresponding accelerator trait | |
| 15:27:32 | zhipeng | if trait include hotplug, then maybe it will be a live attachment | |
| 15:27:45 | jkilpatr | I really don't think we're going to get away with one trait per accelerator, users will probably bundle them into flavors, but instead of being tied to a list of whitelisted pci devices these flavors can be much mroe general. | |
| 15:27:50 | zhipeng | which means user could attach the accelerator after VM creation | |
| 15:28:35 | zhipeng | I was told by jay today that trait are per resource provider | |
| 15:29:02 | zhipeng | so it would mostly be one trait per compute node | |
| 15:29:08 | jkilpatr | I'll have to look at it in detail, I was watching the summit presentation again today. | |
| 15:29:17 | zhipeng | or we got vGPUs or FPGA virtual functions | |
| 15:29:28 | zhipeng | then it would be nested resource provider | |
| 15:29:38 | zhipeng | and we could have trait on the virtual functions | |