| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-29 | |||
| 12:54:16 | alex_xu_ | does make sense | |
| 12:56:06 | efried | alex_xu_: Yes, that is my understanding of how it will need to work. However, as you've seen from the dev ML, it can be more complicated than that. | |
| 12:57:01 | efried | alex_xu_: For example, there seem to be FPGAs where, once you decide they're going to have a particular function on them, they can no longer have a different one. bauzas is running into the same thing for his VGPUs. | |
| 12:57:58 | bauzas | right | |
| 12:59:23 | alex_xu_ | efried: bauzas yes, that is why i also think about we only support pre-programmed mode for the multi functions device | |
| 13:00:04 | alex_xu_ | and same type in the all the functions in a device | |
| 13:00:21 | alex_xu_ | if we want to support different function, we can create multiple RP for different type | |
| 13:00:53 | efried | alex_xu_: Based on a config file or a user command. You're essentially talking about preconfiguring the physical function and locking down what virtual functions it's capable of. | |
| 13:01:11 | efried | Which I think is probably an acceptable interim solution. | |
| 13:01:21 | alex_xu_ | efried: yea | |
| 13:01:22 | efried | Though ultimately it would be nice to be dynamic | |
| 13:01:36 | alex_xu_ | yes, at least for now, that is what we can have | |
| 13:02:14 | alex_xu_ | efried: bauzas another problem is cyborg want to own some inventory info in his db and has filter to second level scheduling | |
| 13:02:59 | alex_xu_ | in cyborg db, the usage of functions and the fuction is record | |
| 13:03:07 | alex_xu_ | s/fuction/function type/ | |
| 13:03:29 | alex_xu_ | for the usage of functions, that can be resolved by function as resource class | |
| 13:03:37 | alex_xu_ | for the function type we can use trait | |
| 13:04:11 | alex_xu_ | and I'm thinking tag the traits after cyborg burn bitstream to the fpga | |
| 13:04:26 | alex_xu_ | that is only support for single function device | |
| 13:08:54 | efried | alex_xu_: If we're doing that via a "preconfigure" step, I can see it being acceptable. What I'm worried about jaypipes and dansmith having a cow about is if we do it e.g. based on the first time an allocation is made against the device. | |
| 13:10:06 | alex_xu_ | efried: for single function device, we will have RP for the device, the inventory is always 1 | |
| 13:10:54 | alex_xu_ | that is different with GPU case | |
| 13:12:39 | cdent | just a heads up to anyone listening; I'm pretty sure that the move of the placement related db tests into the placement hierarchy has tweaked something with the management of trait syncing, such that sometimes (only sometimes) a databsase reset is happening but the reset of the trait sync flag is not happening | |
| 13:12:46 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: PowerVM Driver: Network interface attach/detach https://review.openstack.org/546813 | |
| 13:12:49 | cdent | I'm fixing it (and will make bug etc) | |
| 13:13:03 | mriedem | cdent: thanks | |
| 13:15:19 | efried | edmondsw: Thanks for fixing that up. | |
| 13:15:21 | alex_xu_ | efried: are you saying the race case for the GPU tweak trait in the fly, just as you said in the ML. I see that case. But we won't have problem for the RP always has only 1 inventory | |
| 13:15:38 | edmondsw | efried np | |
| 13:15:43 | efried | mriedem: Sorry about that, I was in a hurry. Haste makes waist, or something. | |
| 13:16:18 | efried | alex_xu_: I don't care about the race. We can't avoid that (I'm composing an answer to Sundar on that topic). | |
| 13:16:36 | efried | alex_xu_: I don't think splitting into one RP per virtual function is going to help us. | |
| 13:16:51 | mriedem | efried: what about your size 32 waist? | |
| 13:17:17 | efried | mriedem: I haven't been a 32 since college. Probably because of haste. | |
| 13:17:26 | edleafe | alex_xu_: efried: I think that this is similar to the discussion about NUMA configuration | |
| 13:17:29 | alex_xu_ | efried: you mean you don't care the reason of two level scheduling. or you mean you don't care the race for tweak inventory/traits in the fly? | |
| 13:17:32 | efried | edleafe: Yes it is. | |
| 13:17:46 | alex_xu_ | I think sundar mix that two race in single email | |
| 13:17:49 | edleafe | IOW, decide what it will be, and set inventory appropriately | |
| 13:18:01 | efried | alex_xu_: Hold on, let me send this note, it summarizes my position. | |
| 13:18:27 | edleafe | If an FPGA can never change its functions, then the resource class is the function. The device is not inventory, it's an RP | |
| 13:18:33 | efried | edleafe: Right, decide up front how you're going to allow it to be configured. Which is limiting, but much simplifying. | |
| 13:18:38 | alex_xu_ | edleafe: right | |
| 13:18:51 | edleafe | Same as with NUMA - limiting but simplifying | |
| 13:19:12 | edleafe | If we don't set limits, we'll be back in the "nova will take care of everything for you" business | |
| 13:19:40 | edleafe | We'll spend 90% of our effort on the 5% use cases | |
| 13:19:58 | alex_xu_ | I totally agree with that. actually I'm saying another race which is due to Cyborg will do second-level scheduling... | |
| 13:20:29 | alex_xu_ | oops, maybe I didn't say clear | |
| 13:21:23 | alex_xu_ | efried: actually I want to attach trait as function type is for re-use previously burn function | |
| 13:21:47 | alex_xu_ | and for the device only support single function | |
| 13:22:11 | edleafe | alex_xu_: you mean trait like CUSTOM_CAN_RUN_FUNCTION_X ? | |
| 13:22:27 | alex_xu_ | edleafe: yes | |
| 13:22:38 | alex_xu_ | edleafe: sorry, no | |
| 13:23:16 | alex_xu_ | just CUSTOM_FUCNTION_X, not can run, attach trait after cyborg burn the function x into the device | |
| 13:24:17 | edleafe | alex_xu_: so a request for a VM with that function comes in. How will find that device if the trait isn't present initially? | |
| 13:24:45 | alex_xu_ | edleafe: a fpga can burn any function | |
| 13:25:26 | alex_xu_ | edleafe: we only tell cyborg we need function x, please burn that into the any available device. | |
| 13:25:41 | edleafe | alex_xu_: sure, so why do we need to add the trait? | |
| 13:26:10 | alex_xu_ | edleafe: after the VM is destroy, there is use-case, we want to reuse that device which already burn function x, so that is why I want to attach that trait. | |
| 13:26:41 | edleafe | alex_xu_: Cyborg is supposed to keep track of that - not placement | |
| 13:26:54 | edleafe | that's what they want to do with a weigher | |
| 13:27:13 | edleafe | prefer hosts that already have that function available | |
| 13:27:26 | alex_xu_ | edleafe: cyborg is responsible for attach trait to the fpga RP | |
| 13:27:55 | alex_xu_ | edleafe: and that need 'preferred trait' which we don't have today... | |
| 13:28:03 | edleafe | ...which is a weigher | |
| 13:28:17 | alex_xu_ | yes | |
| 13:29:01 | dansmith | efried: you needn't corral all of my future reactions into "have a cow" territory just because there have been a few significant disagreements lately | |
| 13:29:09 | edleafe | it would be much cleaner to not have them mess with these traits, and just have the weigher make a call to Cyborg to prefer those hosts that have the function available | |
| 13:29:22 | dansmith | I don't birth bovine for pleasure | |
| 13:29:25 | efried | dansmith: It's not that, it's about previously-had cows on this exact topic. | |
| 13:30:33 | alex_xu_ | jaypipes said he is ok to have a 'preferred trait' in flavor extra, and has a nova scheduler weigher to weigh the traits. not a weigher inside placement | |
| 13:30:46 | efried | alex_xu_, edleafe: Right, but jaypipes and dansmith have had a pretty firm stance from the start that we can't do traits like THIS_FPGA_HAS_FUNCTION_A_ON_IT_RIGHT_NOW. | |
| 13:31:06 | efried | Even when we're talking about "preferred" traits. | |
| 13:31:06 | edleafe | efried: ++ | |
| 13:31:21 | edleafe | that's why all that logic needs to be in Cyborg | |
| 13:31:21 | alex_xu_ | efried, why? | |
| 13:31:42 | alex_xu_ | sorry for asking people repeat the thing people already said in PTG... | |
| 13:31:53 | efried | alex_xu_: The statement is that "traits are capability, not state". | |
| 13:32:02 | efried | alex_xu_: TBH, I've never understood the "why" of that. | |
| 13:32:26 | dansmith | efried: I'm not sure I've taken a hard stance against that, but let me say: | |
| 13:32:30 | alex_xu_ | dansmith: ^ :) | |
| 13:32:50 | dansmith | flipping a trait at high speed to indicate something like "cpu load appears low right now" | |
| 13:32:55 | dansmith | seems like a really bad idea to me | |
| 13:32:56 | dansmith | however, | |
| 13:33:16 | dansmith | if cyborg is managing resources, and it decides when to remove a function from a device, like a cache manager kind of thing, | |
| 13:33:37 | dansmith | I can see it committing a resource (so it knows it's claimed/committed) and then leaving a trait on it, | |
| 13:34:03 | dansmith | and then deciding six hours after the last instance is gone that "okay, this should go back to generic empty" or something like that | |
| 13:34:15 | dansmith | it's not great, and it could certainly be abused past my tolerance level, | |
| 13:34:26 | dansmith | but I'm not sure it's the worst thing | |
| 13:34:42 | dansmith | now, I don't think jaypipes agrees with me there, but.. | |
| 13:35:04 | jaypipes | dansmith: I do. | |
| 13:35:08 | cdent | people are going to "abuse" traits no matter what anyone says, so there's a part of me that thinks we should let it happen | |
| 13:35:15 | efried | this ^ | |
| 13:35:17 | alex_xu_ | I agree with that. | |
| 13:35:28 | jaypipes | disagree. | |
| 13:35:39 | efried | Well, I'm not sure how we can stop 'em. | |
| 13:35:43 | edleafe | dansmith: cyborg can set all the traits they want. The problem is expecting Nova to act on them outside of the normal flow of placement/scheduler | |
| 13:35:53 | alex_xu_ | I only think about burn that image, and don't change it anymore... | |
| 13:35:53 | jaypipes | efried: by not encouraging it. | |
| 13:36:16 | edleafe | jaypipes: and not supporting it | |