| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-29 | |||
| 13:02:14 | alex_xu_ | efried: bauzas another problem is cyborg want to own some inventory info in his db and has filter to second level scheduling | |
| 13:02:59 | alex_xu_ | in cyborg db, the usage of functions and the fuction is record | |
| 13:03:07 | alex_xu_ | s/fuction/function type/ | |
| 13:03:29 | alex_xu_ | for the usage of functions, that can be resolved by function as resource class | |
| 13:03:37 | alex_xu_ | for the function type we can use trait | |
| 13:04:11 | alex_xu_ | and I'm thinking tag the traits after cyborg burn bitstream to the fpga | |
| 13:04:26 | alex_xu_ | that is only support for single function device | |
| 13:08:54 | efried | alex_xu_: If we're doing that via a "preconfigure" step, I can see it being acceptable. What I'm worried about jaypipes and dansmith having a cow about is if we do it e.g. based on the first time an allocation is made against the device. | |
| 13:10:06 | alex_xu_ | efried: for single function device, we will have RP for the device, the inventory is always 1 | |
| 13:10:54 | alex_xu_ | that is different with GPU case | |
| 13:12:39 | cdent | just a heads up to anyone listening; I'm pretty sure that the move of the placement related db tests into the placement hierarchy has tweaked something with the management of trait syncing, such that sometimes (only sometimes) a databsase reset is happening but the reset of the trait sync flag is not happening | |
| 13:12:46 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: PowerVM Driver: Network interface attach/detach https://review.openstack.org/546813 | |
| 13:12:49 | cdent | I'm fixing it (and will make bug etc) | |
| 13:13:03 | mriedem | cdent: thanks | |
| 13:15:19 | efried | edmondsw: Thanks for fixing that up. | |
| 13:15:21 | alex_xu_ | efried: are you saying the race case for the GPU tweak trait in the fly, just as you said in the ML. I see that case. But we won't have problem for the RP always has only 1 inventory | |
| 13:15:38 | edmondsw | efried np | |
| 13:15:43 | efried | mriedem: Sorry about that, I was in a hurry. Haste makes waist, or something. | |
| 13:16:18 | efried | alex_xu_: I don't care about the race. We can't avoid that (I'm composing an answer to Sundar on that topic). | |
| 13:16:36 | efried | alex_xu_: I don't think splitting into one RP per virtual function is going to help us. | |
| 13:16:51 | mriedem | efried: what about your size 32 waist? | |
| 13:17:17 | efried | mriedem: I haven't been a 32 since college. Probably because of haste. | |
| 13:17:26 | edleafe | alex_xu_: efried: I think that this is similar to the discussion about NUMA configuration | |
| 13:17:29 | alex_xu_ | efried: you mean you don't care the reason of two level scheduling. or you mean you don't care the race for tweak inventory/traits in the fly? | |
| 13:17:32 | efried | edleafe: Yes it is. | |
| 13:17:46 | alex_xu_ | I think sundar mix that two race in single email | |
| 13:17:49 | edleafe | IOW, decide what it will be, and set inventory appropriately | |
| 13:18:01 | efried | alex_xu_: Hold on, let me send this note, it summarizes my position. | |
| 13:18:27 | edleafe | If an FPGA can never change its functions, then the resource class is the function. The device is not inventory, it's an RP | |
| 13:18:33 | efried | edleafe: Right, decide up front how you're going to allow it to be configured. Which is limiting, but much simplifying. | |
| 13:18:38 | alex_xu_ | edleafe: right | |
| 13:18:51 | edleafe | Same as with NUMA - limiting but simplifying | |
| 13:19:12 | edleafe | If we don't set limits, we'll be back in the "nova will take care of everything for you" business | |
| 13:19:40 | edleafe | We'll spend 90% of our effort on the 5% use cases | |
| 13:19:58 | alex_xu_ | I totally agree with that. actually I'm saying another race which is due to Cyborg will do second-level scheduling... | |
| 13:20:29 | alex_xu_ | oops, maybe I didn't say clear | |
| 13:21:23 | alex_xu_ | efried: actually I want to attach trait as function type is for re-use previously burn function | |
| 13:21:47 | alex_xu_ | and for the device only support single function | |
| 13:22:11 | edleafe | alex_xu_: you mean trait like CUSTOM_CAN_RUN_FUNCTION_X ? | |
| 13:22:27 | alex_xu_ | edleafe: yes | |
| 13:22:38 | alex_xu_ | edleafe: sorry, no | |
| 13:23:16 | alex_xu_ | just CUSTOM_FUCNTION_X, not can run, attach trait after cyborg burn the function x into the device | |
| 13:24:17 | edleafe | alex_xu_: so a request for a VM with that function comes in. How will find that device if the trait isn't present initially? | |
| 13:24:45 | alex_xu_ | edleafe: a fpga can burn any function | |
| 13:25:26 | alex_xu_ | edleafe: we only tell cyborg we need function x, please burn that into the any available device. | |
| 13:25:41 | edleafe | alex_xu_: sure, so why do we need to add the trait? | |
| 13:26:10 | alex_xu_ | edleafe: after the VM is destroy, there is use-case, we want to reuse that device which already burn function x, so that is why I want to attach that trait. | |
| 13:26:41 | edleafe | alex_xu_: Cyborg is supposed to keep track of that - not placement | |
| 13:26:54 | edleafe | that's what they want to do with a weigher | |
| 13:27:13 | edleafe | prefer hosts that already have that function available | |
| 13:27:26 | alex_xu_ | edleafe: cyborg is responsible for attach trait to the fpga RP | |
| 13:27:55 | alex_xu_ | edleafe: and that need 'preferred trait' which we don't have today... | |
| 13:28:03 | edleafe | ...which is a weigher | |
| 13:28:17 | alex_xu_ | yes | |
| 13:29:01 | dansmith | efried: you needn't corral all of my future reactions into "have a cow" territory just because there have been a few significant disagreements lately | |
| 13:29:09 | edleafe | it would be much cleaner to not have them mess with these traits, and just have the weigher make a call to Cyborg to prefer those hosts that have the function available | |
| 13:29:22 | dansmith | I don't birth bovine for pleasure | |
| 13:29:25 | efried | dansmith: It's not that, it's about previously-had cows on this exact topic. | |
| 13:30:33 | alex_xu_ | jaypipes said he is ok to have a 'preferred trait' in flavor extra, and has a nova scheduler weigher to weigh the traits. not a weigher inside placement | |
| 13:30:46 | efried | alex_xu_, edleafe: Right, but jaypipes and dansmith have had a pretty firm stance from the start that we can't do traits like THIS_FPGA_HAS_FUNCTION_A_ON_IT_RIGHT_NOW. | |
| 13:31:06 | efried | Even when we're talking about "preferred" traits. | |
| 13:31:06 | edleafe | efried: ++ | |
| 13:31:21 | edleafe | that's why all that logic needs to be in Cyborg | |
| 13:31:21 | alex_xu_ | efried, why? | |
| 13:31:42 | alex_xu_ | sorry for asking people repeat the thing people already said in PTG... | |
| 13:31:53 | efried | alex_xu_: The statement is that "traits are capability, not state". | |
| 13:32:02 | efried | alex_xu_: TBH, I've never understood the "why" of that. | |
| 13:32:26 | dansmith | efried: I'm not sure I've taken a hard stance against that, but let me say: | |
| 13:32:30 | alex_xu_ | dansmith: ^ :) | |
| 13:32:50 | dansmith | flipping a trait at high speed to indicate something like "cpu load appears low right now" | |
| 13:32:55 | dansmith | seems like a really bad idea to me | |
| 13:32:56 | dansmith | however, | |
| 13:33:16 | dansmith | if cyborg is managing resources, and it decides when to remove a function from a device, like a cache manager kind of thing, | |
| 13:33:37 | dansmith | I can see it committing a resource (so it knows it's claimed/committed) and then leaving a trait on it, | |
| 13:34:03 | dansmith | and then deciding six hours after the last instance is gone that "okay, this should go back to generic empty" or something like that | |
| 13:34:15 | dansmith | it's not great, and it could certainly be abused past my tolerance level, | |
| 13:34:26 | dansmith | but I'm not sure it's the worst thing | |
| 13:34:42 | dansmith | now, I don't think jaypipes agrees with me there, but.. | |
| 13:35:04 | jaypipes | dansmith: I do. | |
| 13:35:08 | cdent | people are going to "abuse" traits no matter what anyone says, so there's a part of me that thinks we should let it happen | |
| 13:35:15 | efried | this ^ | |
| 13:35:17 | alex_xu_ | I agree with that. | |
| 13:35:28 | jaypipes | disagree. | |
| 13:35:39 | efried | Well, I'm not sure how we can stop 'em. | |
| 13:35:43 | edleafe | dansmith: cyborg can set all the traits they want. The problem is expecting Nova to act on them outside of the normal flow of placement/scheduler | |
| 13:35:53 | alex_xu_ | I only think about burn that image, and don't change it anymore... | |
| 13:35:53 | jaypipes | efried: by not encouraging it. | |
| 13:36:16 | edleafe | jaypipes: and not supporting it | |
| 13:36:17 | dansmith | edleafe: right, we control how much of nova tries to obsess over the traits being set | |
| 13:36:29 | jaypipes | traits != status changes | |
| 13:36:57 | efried | jaypipes: Would you mind expostulating a bit on the reasoning behind that in the ML thread? | |
| 13:37:18 | efried | jaypipes: "[nova] [cyborg] Race condition in the Cyborg/Nova flow" | |
| 13:37:46 | efried | jaypipes: Though it seems like this conversation happens at least once a quarter, so maybe a permadoc somewhere would be expedient. | |
| 13:37:56 | jaypipes | alex_xu_: I'm fine with a trait that means "this FPGA is capable of doing X". What I don't want to see is a precedence set for traits that mean "this foo is on at this moment or off at this moment" and have that trait be used by virt drivers or external agents as a poor-man's servicegroup API. | |
| 13:38:40 | alex_xu_ | jaypipes: I see that, I think we can use trait for the pre-programmed fpga device | |
| 13:38:41 | jaypipes | efried: yes, I will comment on the ML thread. not sure about expostulating, though. | |
| 13:38:51 | edleafe | efried: another problem is we spent a lot of time in Dublin discussing these things and agreeing on boundaries. The current discussion seems to ignore all of that | |
| 13:38:57 | alex_xu_ | and pre-programmed fpga device doesn't allow to re-program | |
| 13:39:10 | alex_xu_ | jaypipes: in that case, does make sense? | |
| 13:39:11 | dansmith | alex_xu_: jaypipes: let me restate what I think is a legit use, again using cyborg | |