| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-29 | |||
| 13:32:02 | efried | alex_xu_: TBH, I've never understood the "why" of that. | |
| 13:32:26 | dansmith | efried: I'm not sure I've taken a hard stance against that, but let me say: | |
| 13:32:30 | alex_xu_ | dansmith: ^ :) | |
| 13:32:50 | dansmith | flipping a trait at high speed to indicate something like "cpu load appears low right now" | |
| 13:32:55 | dansmith | seems like a really bad idea to me | |
| 13:32:56 | dansmith | however, | |
| 13:33:16 | dansmith | if cyborg is managing resources, and it decides when to remove a function from a device, like a cache manager kind of thing, | |
| 13:33:37 | dansmith | I can see it committing a resource (so it knows it's claimed/committed) and then leaving a trait on it, | |
| 13:34:03 | dansmith | and then deciding six hours after the last instance is gone that "okay, this should go back to generic empty" or something like that | |
| 13:34:15 | dansmith | it's not great, and it could certainly be abused past my tolerance level, | |
| 13:34:26 | dansmith | but I'm not sure it's the worst thing | |
| 13:34:42 | dansmith | now, I don't think jaypipes agrees with me there, but.. | |
| 13:35:04 | jaypipes | dansmith: I do. | |
| 13:35:08 | cdent | people are going to "abuse" traits no matter what anyone says, so there's a part of me that thinks we should let it happen | |
| 13:35:15 | efried | this ^ | |
| 13:35:17 | alex_xu_ | I agree with that. | |
| 13:35:28 | jaypipes | disagree. | |
| 13:35:39 | efried | Well, I'm not sure how we can stop 'em. | |
| 13:35:43 | edleafe | dansmith: cyborg can set all the traits they want. The problem is expecting Nova to act on them outside of the normal flow of placement/scheduler | |
| 13:35:53 | alex_xu_ | I only think about burn that image, and don't change it anymore... | |
| 13:35:53 | jaypipes | efried: by not encouraging it. | |
| 13:36:16 | edleafe | jaypipes: and not supporting it | |
| 13:36:17 | dansmith | edleafe: right, we control how much of nova tries to obsess over the traits being set | |
| 13:36:29 | jaypipes | traits != status changes | |
| 13:36:57 | efried | jaypipes: Would you mind expostulating a bit on the reasoning behind that in the ML thread? | |
| 13:37:18 | efried | jaypipes: "[nova] [cyborg] Race condition in the Cyborg/Nova flow" | |
| 13:37:46 | efried | jaypipes: Though it seems like this conversation happens at least once a quarter, so maybe a permadoc somewhere would be expedient. | |
| 13:37:56 | jaypipes | alex_xu_: I'm fine with a trait that means "this FPGA is capable of doing X". What I don't want to see is a precedence set for traits that mean "this foo is on at this moment or off at this moment" and have that trait be used by virt drivers or external agents as a poor-man's servicegroup API. | |
| 13:38:40 | alex_xu_ | jaypipes: I see that, I think we can use trait for the pre-programmed fpga device | |
| 13:38:41 | jaypipes | efried: yes, I will comment on the ML thread. not sure about expostulating, though. | |
| 13:38:51 | edleafe | efried: another problem is we spent a lot of time in Dublin discussing these things and agreeing on boundaries. The current discussion seems to ignore all of that | |
| 13:38:57 | alex_xu_ | and pre-programmed fpga device doesn't allow to re-program | |
| 13:39:10 | alex_xu_ | jaypipes: in that case, does make sense? | |
| 13:39:11 | dansmith | alex_xu_: jaypipes: let me restate what I think is a legit use, again using cyborg | |
| 13:39:16 | edleafe | alex_xu_: pre-programmed functions are resource classes | |
| 13:39:17 | efried | edleafe: How ignoring? I've been saying it ain't gonna fly. | |
| 13:39:23 | edleafe | efried: not you | |
| 13:39:32 | edleafe | sundar primarily | |
| 13:39:37 | alex_xu_ | edleafe: I also think about that... | |
| 13:39:57 | alex_xu_ | pre-programmed is normal accelerator device... | |
| 13:40:01 | dansmith | I think it's legit for blazar to pre-provision images into FPGAs via cyborg according to upcoming schedule, set traits on them for the image, and de-provision/trait them at the end of the 24-hour reservation window | |
| 13:40:14 | edleafe | if it's programmed for X and can't be changed, then it has an inventory on that function. That's a resource class | |
| 13:40:35 | jaypipes | hold up, let dansmith finish his synopsis for a sec | |
| 13:40:59 | dansmith | that's it.. just another example of pre-provisioned images getting traits so we can schedule to them by function | |
| 13:41:10 | jaypipes | ok | |
| 13:41:22 | edleafe | dansmith: that sounds like using traits as resource class proxies | |
| 13:41:25 | alex_xu_ | dansmith: what's different with cyborg cache manager... | |
| 13:41:33 | jaypipes | and frankly, that's what I *thought* the agreement was in Dublin, but granted, I only made the tail end of the discussion. | |
| 13:41:34 | dansmith | I don't think that programming in a dynamic device needs to be resource class | |
| 13:41:46 | dansmith | jaypipes: same | |
| 13:42:04 | dansmith | alex_xu_: it's not, as I said above in the earlier example | |
| 13:42:58 | jaypipes | dansmith: it *could* be done via a custom resource class, but then again, I find a nice simplicity in just having a standard resource class called FPGA_CONTEXT (or something like that) and decorating those pre-programmed slots (providers) with particular traits | |
| 13:43:10 | dansmith | sure it could, I said "need" | |
| 13:43:11 | efried | edleafe: The RC-vs-trait discussion is a tangent. The issue is whether/when it's okay to modify RPs "on the fly". And how on-the-fly is "on the fly"? | |
| 13:43:23 | dansmith | jaypipes: it needs to be class per type of device, but yeah | |
| 13:43:49 | edleafe | efried: I don't agree it's a tangent. If you model your resources correctly, the whole discussion goes away | |
| 13:44:05 | dansmith | efried: if you have a preferred trait of HAS_SSL, you get one without it, you claim it in scheduling, it's cool IMHO for cyborg to trait it up when it does the programming | |
| 13:44:26 | alex_xu_ | dansmith: I must misunderstand this words "it's not great, and it could certainly be abused past my tolerance level," It means yes... or no.. | |
| 13:44:51 | dansmith | alex_xu_: that was referring to people doing other things, not this | |
| 13:44:51 | efried | dansmith: That would be cool. jaypipes are you on board with that? | |
| 13:45:39 | jaypipes | I'm cool with what dansmith just said, yes. | |
| 13:45:41 | edleafe | dansmith: that's a totally legit use case | |
| 13:45:48 | jaypipes | here are the things I do *not* want to see: | |
| 13:46:04 | jaypipes | 1) agents, including the virt driver, using the traits API as a servicegroup/status API. | |
| 13:46:05 | dansmith | alex_xu_: sorry I see why that was confusing I mixed that into other thoughts when I re-read | |
| 13:46:21 | jaypipes | 2) *Consumer requests* causing traits to be set against a provider. | |
| 13:47:04 | dansmith | jaypipes: 2 meaning something like nova setting traits on providers of has-instances or has-10-instances, or something like that yeah? | |
| 13:47:26 | dansmith | or has-some-volumes-attached-to-host-right-now-yo | |
| 13:47:55 | mriedem | bottom 6 patches in the placement-req-filter series have a +2 https://review.openstack.org/#/c/544730/ | |
| 13:47:57 | jaypipes | It is that latter point that I am afraid of, since it means we essentially throw out all the scheduling heuristics that are now made in placement (since we would no longer be able to filter on providers having a trait but would need to add some mechanism to detect providers that *could have* a trait on them, if certain conditions apply) | |
| 13:48:08 | efried | jaypipes: #2 is a bit sticky, though, innit? The consumer requests an instance with FPGA with function X, cyborg finds a previously-unfettered device suitable for use and flashes it with function X, then sets a trait saying it's got function X on it now. Is that "consumer requests causing traits to be set"? | |
| 13:48:08 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes Multiple GPU types https://review.openstack.org/557065 | |
| 13:48:12 | dansmith | jaypipes: yup | |
| 13:48:29 | dansmith | efried: it has given that device capability | |
| 13:48:48 | bauzas | jaypipes: dansmith: very simple spec about providing a config option for defining a list of pGPUs a specific type would be supported https://review.openstack.org/557065 | |
| 13:48:51 | dansmith | it has changed the device in some meaningful way that means it can do things | |
| 13:48:58 | efried | dansmith: tbc, *I* am fine with that scenario. I'm asking if it violates jaypipes' #2 | |
| 13:49:12 | jaypipes | efried: it would, yes. | |
| 13:49:16 | bauzas | dansmith: jaypipes: if you can review it quickly, that would unblock me from using n-r-p for libvirt vGPUs | |
| 13:49:27 | dansmith | jaypipes: o.O | |
| 13:50:02 | bauzas | mriedem: dansmith: I'll sponsor https://review.openstack.org/#/c/544730/ and above | |
| 13:50:14 | dansmith | jaypipes: now I'm confused :) | |
| 13:50:16 | jaypipes | efried: if cyborg configures its inventory/traits dynamically *at the time and due to a specific request from a consumer*, I have an issue with that | |
| 13:50:35 | dansmith | jaypipes: heh, that's what you just said you were okay with | |
| 13:50:45 | mriedem | alex_xu_: interesting idea to deprecate the backup API in https://review.openstack.org/#/c/511825/ - you should probably start a ML thread on that if you want wider input | |
| 13:50:46 | jaypipes | dansmith: it's a difference in timing. | |
| 13:51:06 | dansmith | jaypipes: no, I said above "sets the trait when programming" | |
| 13:51:19 | alex_xu_ | mriedem: will do that later | |
| 13:51:22 | jaypipes | dansmith: when is the programming done. | |
| 13:51:44 | jaypipes | dansmith: that was a question, sorry.. when is the programming of the slots done? | |
| 13:52:03 | jaypipes | dansmith: if it's done after GET /allocation_candidates, then that won't work. | |
| 13:52:13 | dansmith | jaypipes: sure it will | |
| 13:52:23 | jaypipes | dansmith: since clearly the trait won't be useful in filtering out destination providers. | |
| 13:52:29 | alex_xu_ | jaypipes: dansmith note, for programmable fpga, it inventory always 1 | |
| 13:52:32 | dansmith | jaypipes: it would be preferred | |
| 13:52:56 | alex_xu_ | there is no race like we tweak traits in the fly for gpu | |
| 13:53:07 | dansmith | jaypipes: hangout? | |
| 13:53:11 | jaypipes | dansmith: sdure | |
| 13:53:12 | jaypipes | sure | |
| 13:53:20 | jaypipes | set it up. I will coffee up. | |