Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-20
20:22:42 dansmith so I guess that means I have to keep plugging at this
20:22:52 dansmith jaypipes: yea, feel free, but that's separate from my other work here
20:23:08 jaypipes k. is there anything I can help you with on your work here?
20:23:21 dansmith jaypipes: you just did
20:23:24 dansmith thanks
20:23:25 jaypipes :)
20:23:40 jaypipes well, at least that gives me something to smile about today.
20:23:42 jaypipes thanks.
20:25:19 jaypipes FTR, on the cyborg thing, I *also* believe that cyborg should be modeling should be inventories, not RPs with tons of traits masquerading as resource classes.
20:25:38 jaypipes dansmith: ^
20:25:47 dansmith jaypipes: yay.
20:25:59 jaypipes indeed.
20:26:04 mriedem dansmith: on top of the improved join on what you're doing, i think it's a 2-part change in that the metadata api doesn't need to be pre-loading on system_metadata - at least not anymore
20:26:42 dansmith mriedem: yeah, so that will address the acute issue right?
20:26:45 mriedem last i looked the only thing in meta-api that would use sysmeta is a vendor data provider if configured
20:26:53 dansmith maybe I should punt this until we have a better reason to do this work
20:27:02 mriedem i believe so, and i think that's what the workday ops guy said he did in the ML
20:27:06 mriedem heh
20:27:18 mriedem see, i started trying to do what you said and sparks flew immediately
20:27:19 mriedem and i gave up
20:27:34 dansmith oh did you
20:27:35 dansmith ?
20:27:40 mriedem locally
20:27:45 dansmith maybe that should be my impetus to fix it
20:27:45 mriedem never pushed it up b/c tests failed horribly
20:27:52 mriedem to show me up?
20:27:55 dansmith yeah
20:27:59 mriedem by all means
20:28:06 dansmith nah, sounds hard.
20:28:12 mriedem next you can fix the nova/cinder cross az attach mess
20:28:29 mriedem which i have a fix for, but it's fugly as all get out
20:29:09 dansmith I like it already
20:40:34 efried dansmith, jaypipes: Modeling accelerators via specific resource classes, so like CUSTOM_FPGA_GZIP rather than rc=FPGA + traits=[GZIP] ?
20:41:05 dansmith GZIP isn't a trait, IMHO
20:41:18 dansmith like, I don't ask for SOME_SILICON=1024, trait=RAM
20:41:19 efried um. The FPGA is capable of processing gzips?
20:42:01 jaypipes the resource class is a context to a GZIP program flashed to a device.
20:42:04 dansmith right, but there's a difference between asking for an FPGA and asking for GZIP offload to me
20:42:18 jaypipes dansmith++
20:42:26 dansmith if I want an FPGA that I can program myself, I want an FPGA=1.. if I want a GZIP handler, I want GZIP=1,
20:42:32 dansmith which might be an FPGA in the back end
20:42:37 dansmith or it might be an ASIC
20:42:39 dansmith or whatever
20:44:01 efried But then a resource provider representing a blank (as-yet-unprogrammed) FPGA would have to show inventories of multiple resource classes, and then when one of those is consumed, we would have to nix the other resource classes (or do the reserved=total trick for them).
20:44:17 efried which is racy, as well as being ew.
20:44:19 dansmith same for the trait right?
20:44:32 dansmith you say FPGA=1, trats=GZIP,TLS,BITCOIN
20:44:51 efried no. GZIP_CAPABLE stays. Not sure we have to retrait every time we reprogram.
20:45:32 efried but if we do, the GZIP_IS_ON_THIS_THING_AT_THE_MOMENT trait would be separate, and have separate meaning, than the GZIP_CAPABLE.
20:45:57 dansmith you're just providing no abstraction there
20:46:02 efried the former would be used only as an optimization, if/when we have "preferred traits", to avoid reprogramming if there's one that's already set up.
20:46:30 jaypipes premature optimization...
20:47:03 efried I'd be happy if we skipped that whole bit for the first pass and just used the *_CAPABLE traits.
20:47:14 efried Programming gets done after the claim, if and as necessary.
20:47:23 jaypipes I'd be happy if we just skipped everything other than just using custom resource classes.
20:47:46 efried jaypipes: So preprogram everything?
20:48:05 dansmith pretty sure we said the first step was assuming everything was static, no?
20:48:22 dansmith except for the "user will program it themselves" case of course
20:48:44 jaypipes dansmith++ again.
20:49:21 dansmith when we talked about this in (denver I think?) I think the overwhelming majority of cases where this really applies is the pre-programmed case,
20:49:39 dansmith because it provides for locality in certain FGPAs that have one code region and multiple execution contexts,
20:50:11 dansmith such that if you co-locate a GZIP and a TLS, they both can't use the same FPGA, but if you get two GZIP tenants on the same box, they can
20:50:41 dansmith and I thought we agreed to avoid boiling the ocean with "everything is completely dynamic all the time forever" until we could do, you know, fucking anything :)
20:50:56 efried I can buy it for a first pass. Long-term, that seems like not very cloudy. Though I suppose if the "pre"programming is done by a higher orchestrator, it could fly.
20:50:58 dansmith maybe that's just me missing something, but..
20:51:14 efried okay, thanks for the fresher.
20:51:15 dansmith efried: well, if you do it with inventories, you can actually count usage of those things,
20:51:33 dansmith and then your pre-programming workflow can ensure that X% of GZIP is available based on current demand
20:51:41 dansmith but if you do it with traits that seems a lot messier
20:52:04 dansmith usage and capacity I mean
20:52:07 slaweq mriedem: thx for taking care of this issue
20:52:37 dansmith efried: the orchestrator that maintains a certain amount of available inventory of stuff, I mean
20:52:40 jaypipes I was under the impression Cyborg was gonna contain that "pre-orchestrator/pre-programming" thing...
20:53:05 dansmith jaypipes: yeah I dunno, could be, or could be a cron job you run once an hour that makes sure the capacity is within limits
20:53:17 dansmith meaning I dunno if that level of cron is something cyborg was going to do or not,
20:53:22 efried "load balancing" your accelerators
20:53:51 efried guess that implies moving them around, which isn't what we're talking about
20:53:54 efried but yeah, I get the idea.
21:46:53 jaypipes sean-k-mooney: questions for you on https://review.openstack.org/#/c/602384/ please
22:09:15 openstackgerrit Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941
22:09:43 openstackgerrit Chris Dent proposed openstack/nova master: WIP: Delete the placement code https://review.openstack.org/618215
22:13:53 mriedem efried: turns out https://review.openstack.org/#/c/619061/ does fix the pg thing
22:14:03 mriedem also, wee lots of red http://logs.openstack.org/05/613305/7/check/tempest-full/999ec9f/controller/logs/screen-n-api.txt.gz?level=ERROR
22:14:08 mriedem unrelated regression
22:14:37 efried ack x2
22:15:35 efried mriedem: What were the func test failures? Actual differences in results?
22:15:56 mriedem dansmith: looks like an unintended side effect of using the scatter_gather_single_cell for nova show is the scatter thing logs errors from the query^ even for things we expect
22:16:07 mriedem efried: yeah, got 5 rows when 1 expected
22:16:12 mriedem efried: pull it down and try it out
22:16:17 efried okay, must be aggregate functions
22:16:27 efried nah, higher priorities.
22:16:28 dansmith mriedem: oh yeah, I think I called that out initially and then totally forgot :(
22:16:34 dansmith it's spewing errors to the logs right?
22:16:36 mriedem yes
22:16:39 mriedem i'll open a bug
22:16:40 efried If it fixes the problem, let's roll with it.
22:19:09 mriedem https://bugs.launchpad.net/nova/+bug/1804325
22:19:09 openstack Launchpad bug 1804325 in OpenStack Compute (nova) "InstanceNotFound traceback errors in n-api logs while polling for server delete" [High,Triaged]
22:20:16 mriedem so, i think we just remove that exception line since the caller can get the actual exception type now
22:20:20 mriedem and decide if it needs to log

Earlier   Later