| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-29 | |||
| 13:18:01 | efried | alex_xu_: Hold on, let me send this note, it summarizes my position. | |
| 13:18:27 | edleafe | If an FPGA can never change its functions, then the resource class is the function. The device is not inventory, it's an RP | |
| 13:18:33 | efried | edleafe: Right, decide up front how you're going to allow it to be configured. Which is limiting, but much simplifying. | |
| 13:18:38 | alex_xu_ | edleafe: right | |
| 13:18:51 | edleafe | Same as with NUMA - limiting but simplifying | |
| 13:19:12 | edleafe | If we don't set limits, we'll be back in the "nova will take care of everything for you" business | |
| 13:19:40 | edleafe | We'll spend 90% of our effort on the 5% use cases | |
| 13:19:58 | alex_xu_ | I totally agree with that. actually I'm saying another race which is due to Cyborg will do second-level scheduling... | |
| 13:20:29 | alex_xu_ | oops, maybe I didn't say clear | |
| 13:21:23 | alex_xu_ | efried: actually I want to attach trait as function type is for re-use previously burn function | |
| 13:21:47 | alex_xu_ | and for the device only support single function | |
| 13:22:11 | edleafe | alex_xu_: you mean trait like CUSTOM_CAN_RUN_FUNCTION_X ? | |
| 13:22:27 | alex_xu_ | edleafe: yes | |
| 13:22:38 | alex_xu_ | edleafe: sorry, no | |
| 13:23:16 | alex_xu_ | just CUSTOM_FUCNTION_X, not can run, attach trait after cyborg burn the function x into the device | |
| 13:24:17 | edleafe | alex_xu_: so a request for a VM with that function comes in. How will find that device if the trait isn't present initially? | |
| 13:24:45 | alex_xu_ | edleafe: a fpga can burn any function | |
| 13:25:26 | alex_xu_ | edleafe: we only tell cyborg we need function x, please burn that into the any available device. | |
| 13:25:41 | edleafe | alex_xu_: sure, so why do we need to add the trait? | |
| 13:26:10 | alex_xu_ | edleafe: after the VM is destroy, there is use-case, we want to reuse that device which already burn function x, so that is why I want to attach that trait. | |
| 13:26:41 | edleafe | alex_xu_: Cyborg is supposed to keep track of that - not placement | |
| 13:26:54 | edleafe | that's what they want to do with a weigher | |
| 13:27:13 | edleafe | prefer hosts that already have that function available | |
| 13:27:26 | alex_xu_ | edleafe: cyborg is responsible for attach trait to the fpga RP | |
| 13:27:55 | alex_xu_ | edleafe: and that need 'preferred trait' which we don't have today... | |
| 13:28:03 | edleafe | ...which is a weigher | |
| 13:28:17 | alex_xu_ | yes | |
| 13:29:01 | dansmith | efried: you needn't corral all of my future reactions into "have a cow" territory just because there have been a few significant disagreements lately | |
| 13:29:09 | edleafe | it would be much cleaner to not have them mess with these traits, and just have the weigher make a call to Cyborg to prefer those hosts that have the function available | |
| 13:29:22 | dansmith | I don't birth bovine for pleasure | |
| 13:29:25 | efried | dansmith: It's not that, it's about previously-had cows on this exact topic. | |
| 13:30:33 | alex_xu_ | jaypipes said he is ok to have a 'preferred trait' in flavor extra, and has a nova scheduler weigher to weigh the traits. not a weigher inside placement | |
| 13:30:46 | efried | alex_xu_, edleafe: Right, but jaypipes and dansmith have had a pretty firm stance from the start that we can't do traits like THIS_FPGA_HAS_FUNCTION_A_ON_IT_RIGHT_NOW. | |
| 13:31:06 | edleafe | efried: ++ | |
| 13:31:06 | efried | Even when we're talking about "preferred" traits. | |
| 13:31:21 | alex_xu_ | efried, why? | |
| 13:31:21 | edleafe | that's why all that logic needs to be in Cyborg | |
| 13:31:42 | alex_xu_ | sorry for asking people repeat the thing people already said in PTG... | |
| 13:31:53 | efried | alex_xu_: The statement is that "traits are capability, not state". | |
| 13:32:02 | efried | alex_xu_: TBH, I've never understood the "why" of that. | |
| 13:32:26 | dansmith | efried: I'm not sure I've taken a hard stance against that, but let me say: | |
| 13:32:30 | alex_xu_ | dansmith: ^ :) | |
| 13:32:50 | dansmith | flipping a trait at high speed to indicate something like "cpu load appears low right now" | |
| 13:32:55 | dansmith | seems like a really bad idea to me | |
| 13:32:56 | dansmith | however, | |
| 13:33:16 | dansmith | if cyborg is managing resources, and it decides when to remove a function from a device, like a cache manager kind of thing, | |
| 13:33:37 | dansmith | I can see it committing a resource (so it knows it's claimed/committed) and then leaving a trait on it, | |
| 13:34:03 | dansmith | and then deciding six hours after the last instance is gone that "okay, this should go back to generic empty" or something like that | |
| 13:34:15 | dansmith | it's not great, and it could certainly be abused past my tolerance level, | |
| 13:34:26 | dansmith | but I'm not sure it's the worst thing | |
| 13:34:42 | dansmith | now, I don't think jaypipes agrees with me there, but.. | |
| 13:35:04 | jaypipes | dansmith: I do. | |
| 13:35:08 | cdent | people are going to "abuse" traits no matter what anyone says, so there's a part of me that thinks we should let it happen | |
| 13:35:15 | efried | this ^ | |
| 13:35:17 | alex_xu_ | I agree with that. | |
| 13:35:28 | jaypipes | disagree. | |
| 13:35:39 | efried | Well, I'm not sure how we can stop 'em. | |
| 13:35:43 | edleafe | dansmith: cyborg can set all the traits they want. The problem is expecting Nova to act on them outside of the normal flow of placement/scheduler | |
| 13:35:53 | jaypipes | efried: by not encouraging it. | |
| 13:35:53 | alex_xu_ | I only think about burn that image, and don't change it anymore... | |
| 13:36:16 | edleafe | jaypipes: and not supporting it | |
| 13:36:17 | dansmith | edleafe: right, we control how much of nova tries to obsess over the traits being set | |
| 13:36:29 | jaypipes | traits != status changes | |
| 13:36:57 | efried | jaypipes: Would you mind expostulating a bit on the reasoning behind that in the ML thread? | |
| 13:37:18 | efried | jaypipes: "[nova] [cyborg] Race condition in the Cyborg/Nova flow" | |
| 13:37:46 | efried | jaypipes: Though it seems like this conversation happens at least once a quarter, so maybe a permadoc somewhere would be expedient. | |
| 13:37:56 | jaypipes | alex_xu_: I'm fine with a trait that means "this FPGA is capable of doing X". What I don't want to see is a precedence set for traits that mean "this foo is on at this moment or off at this moment" and have that trait be used by virt drivers or external agents as a poor-man's servicegroup API. | |
| 13:38:40 | alex_xu_ | jaypipes: I see that, I think we can use trait for the pre-programmed fpga device | |
| 13:38:41 | jaypipes | efried: yes, I will comment on the ML thread. not sure about expostulating, though. | |
| 13:38:51 | edleafe | efried: another problem is we spent a lot of time in Dublin discussing these things and agreeing on boundaries. The current discussion seems to ignore all of that | |
| 13:38:57 | alex_xu_ | and pre-programmed fpga device doesn't allow to re-program | |
| 13:39:10 | alex_xu_ | jaypipes: in that case, does make sense? | |
| 13:39:11 | dansmith | alex_xu_: jaypipes: let me restate what I think is a legit use, again using cyborg | |
| 13:39:16 | edleafe | alex_xu_: pre-programmed functions are resource classes | |
| 13:39:17 | efried | edleafe: How ignoring? I've been saying it ain't gonna fly. | |
| 13:39:23 | edleafe | efried: not you | |
| 13:39:32 | edleafe | sundar primarily | |
| 13:39:37 | alex_xu_ | edleafe: I also think about that... | |
| 13:39:57 | alex_xu_ | pre-programmed is normal accelerator device... | |
| 13:40:01 | dansmith | I think it's legit for blazar to pre-provision images into FPGAs via cyborg according to upcoming schedule, set traits on them for the image, and de-provision/trait them at the end of the 24-hour reservation window | |
| 13:40:14 | edleafe | if it's programmed for X and can't be changed, then it has an inventory on that function. That's a resource class | |
| 13:40:35 | jaypipes | hold up, let dansmith finish his synopsis for a sec | |
| 13:40:59 | dansmith | that's it.. just another example of pre-provisioned images getting traits so we can schedule to them by function | |
| 13:41:10 | jaypipes | ok | |
| 13:41:22 | edleafe | dansmith: that sounds like using traits as resource class proxies | |
| 13:41:25 | alex_xu_ | dansmith: what's different with cyborg cache manager... | |
| 13:41:33 | jaypipes | and frankly, that's what I *thought* the agreement was in Dublin, but granted, I only made the tail end of the discussion. | |
| 13:41:34 | dansmith | I don't think that programming in a dynamic device needs to be resource class | |
| 13:41:46 | dansmith | jaypipes: same | |
| 13:42:04 | dansmith | alex_xu_: it's not, as I said above in the earlier example | |
| 13:42:58 | jaypipes | dansmith: it *could* be done via a custom resource class, but then again, I find a nice simplicity in just having a standard resource class called FPGA_CONTEXT (or something like that) and decorating those pre-programmed slots (providers) with particular traits | |
| 13:43:10 | dansmith | sure it could, I said "need" | |
| 13:43:11 | efried | edleafe: The RC-vs-trait discussion is a tangent. The issue is whether/when it's okay to modify RPs "on the fly". And how on-the-fly is "on the fly"? | |
| 13:43:23 | dansmith | jaypipes: it needs to be class per type of device, but yeah | |
| 13:43:49 | edleafe | efried: I don't agree it's a tangent. If you model your resources correctly, the whole discussion goes away | |
| 13:44:05 | dansmith | efried: if you have a preferred trait of HAS_SSL, you get one without it, you claim it in scheduling, it's cool IMHO for cyborg to trait it up when it does the programming | |
| 13:44:26 | alex_xu_ | dansmith: I must misunderstand this words "it's not great, and it could certainly be abused past my tolerance level," It means yes... or no.. | |
| 13:44:51 | efried | dansmith: That would be cool. jaypipes are you on board with that? | |
| 13:44:51 | dansmith | alex_xu_: that was referring to people doing other things, not this | |
| 13:45:39 | jaypipes | I'm cool with what dansmith just said, yes. | |