| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-20 | |||
| 20:22:52 | dansmith | jaypipes: yea, feel free, but that's separate from my other work here | |
| 20:23:08 | jaypipes | k. is there anything I can help you with on your work here? | |
| 20:23:21 | dansmith | jaypipes: you just did | |
| 20:23:24 | dansmith | thanks | |
| 20:23:25 | jaypipes | :) | |
| 20:23:40 | jaypipes | well, at least that gives me something to smile about today. | |
| 20:23:42 | jaypipes | thanks. | |
| 20:25:19 | jaypipes | FTR, on the cyborg thing, I *also* believe that cyborg should be modeling should be inventories, not RPs with tons of traits masquerading as resource classes. | |
| 20:25:38 | jaypipes | dansmith: ^ | |
| 20:25:47 | dansmith | jaypipes: yay. | |
| 20:25:59 | jaypipes | indeed. | |
| 20:26:04 | mriedem | dansmith: on top of the improved join on what you're doing, i think it's a 2-part change in that the metadata api doesn't need to be pre-loading on system_metadata - at least not anymore | |
| 20:26:42 | dansmith | mriedem: yeah, so that will address the acute issue right? | |
| 20:26:45 | mriedem | last i looked the only thing in meta-api that would use sysmeta is a vendor data provider if configured | |
| 20:26:53 | dansmith | maybe I should punt this until we have a better reason to do this work | |
| 20:27:02 | mriedem | i believe so, and i think that's what the workday ops guy said he did in the ML | |
| 20:27:06 | mriedem | heh | |
| 20:27:18 | mriedem | see, i started trying to do what you said and sparks flew immediately | |
| 20:27:19 | mriedem | and i gave up | |
| 20:27:34 | dansmith | oh did you | |
| 20:27:35 | dansmith | ? | |
| 20:27:40 | mriedem | locally | |
| 20:27:45 | dansmith | maybe that should be my impetus to fix it | |
| 20:27:45 | mriedem | never pushed it up b/c tests failed horribly | |
| 20:27:52 | mriedem | to show me up? | |
| 20:27:55 | dansmith | yeah | |
| 20:27:59 | mriedem | by all means | |
| 20:28:06 | dansmith | nah, sounds hard. | |
| 20:28:12 | mriedem | next you can fix the nova/cinder cross az attach mess | |
| 20:28:29 | mriedem | which i have a fix for, but it's fugly as all get out | |
| 20:29:09 | dansmith | I like it already | |
| 20:40:34 | efried | dansmith, jaypipes: Modeling accelerators via specific resource classes, so like CUSTOM_FPGA_GZIP rather than rc=FPGA + traits=[GZIP] ? | |
| 20:41:05 | dansmith | GZIP isn't a trait, IMHO | |
| 20:41:18 | dansmith | like, I don't ask for SOME_SILICON=1024, trait=RAM | |
| 20:41:19 | efried | um. The FPGA is capable of processing gzips? | |
| 20:42:01 | jaypipes | the resource class is a context to a GZIP program flashed to a device. | |
| 20:42:04 | dansmith | right, but there's a difference between asking for an FPGA and asking for GZIP offload to me | |
| 20:42:18 | jaypipes | dansmith++ | |
| 20:42:26 | dansmith | if I want an FPGA that I can program myself, I want an FPGA=1.. if I want a GZIP handler, I want GZIP=1, | |
| 20:42:32 | dansmith | which might be an FPGA in the back end | |
| 20:42:37 | dansmith | or it might be an ASIC | |
| 20:42:39 | dansmith | or whatever | |
| 20:44:01 | efried | But then a resource provider representing a blank (as-yet-unprogrammed) FPGA would have to show inventories of multiple resource classes, and then when one of those is consumed, we would have to nix the other resource classes (or do the reserved=total trick for them). | |
| 20:44:17 | efried | which is racy, as well as being ew. | |
| 20:44:19 | dansmith | same for the trait right? | |
| 20:44:32 | dansmith | you say FPGA=1, trats=GZIP,TLS,BITCOIN | |
| 20:44:51 | efried | no. GZIP_CAPABLE stays. Not sure we have to retrait every time we reprogram. | |
| 20:45:32 | efried | but if we do, the GZIP_IS_ON_THIS_THING_AT_THE_MOMENT trait would be separate, and have separate meaning, than the GZIP_CAPABLE. | |
| 20:45:57 | dansmith | you're just providing no abstraction there | |
| 20:46:02 | efried | the former would be used only as an optimization, if/when we have "preferred traits", to avoid reprogramming if there's one that's already set up. | |
| 20:46:30 | jaypipes | premature optimization... | |
| 20:47:03 | efried | I'd be happy if we skipped that whole bit for the first pass and just used the *_CAPABLE traits. | |
| 20:47:14 | efried | Programming gets done after the claim, if and as necessary. | |
| 20:47:23 | jaypipes | I'd be happy if we just skipped everything other than just using custom resource classes. | |
| 20:47:46 | efried | jaypipes: So preprogram everything? | |
| 20:48:05 | dansmith | pretty sure we said the first step was assuming everything was static, no? | |
| 20:48:22 | dansmith | except for the "user will program it themselves" case of course | |
| 20:48:44 | jaypipes | dansmith++ again. | |
| 20:49:21 | dansmith | when we talked about this in (denver I think?) I think the overwhelming majority of cases where this really applies is the pre-programmed case, | |
| 20:49:39 | dansmith | because it provides for locality in certain FGPAs that have one code region and multiple execution contexts, | |
| 20:50:11 | dansmith | such that if you co-locate a GZIP and a TLS, they both can't use the same FPGA, but if you get two GZIP tenants on the same box, they can | |
| 20:50:41 | dansmith | and I thought we agreed to avoid boiling the ocean with "everything is completely dynamic all the time forever" until we could do, you know, fucking anything :) | |
| 20:50:56 | efried | I can buy it for a first pass. Long-term, that seems like not very cloudy. Though I suppose if the "pre"programming is done by a higher orchestrator, it could fly. | |
| 20:50:58 | dansmith | maybe that's just me missing something, but.. | |
| 20:51:14 | efried | okay, thanks for the fresher. | |
| 20:51:15 | dansmith | efried: well, if you do it with inventories, you can actually count usage of those things, | |
| 20:51:33 | dansmith | and then your pre-programming workflow can ensure that X% of GZIP is available based on current demand | |
| 20:51:41 | dansmith | but if you do it with traits that seems a lot messier | |
| 20:52:04 | dansmith | usage and capacity I mean | |
| 20:52:07 | slaweq | mriedem: thx for taking care of this issue | |
| 20:52:37 | dansmith | efried: the orchestrator that maintains a certain amount of available inventory of stuff, I mean | |
| 20:52:40 | jaypipes | I was under the impression Cyborg was gonna contain that "pre-orchestrator/pre-programming" thing... | |
| 20:53:05 | dansmith | jaypipes: yeah I dunno, could be, or could be a cron job you run once an hour that makes sure the capacity is within limits | |
| 20:53:17 | dansmith | meaning I dunno if that level of cron is something cyborg was going to do or not, | |
| 20:53:22 | efried | "load balancing" your accelerators | |
| 20:53:51 | efried | guess that implies moving them around, which isn't what we're talking about | |
| 20:53:54 | efried | but yeah, I get the idea. | |
| 21:46:53 | jaypipes | sean-k-mooney: questions for you on https://review.openstack.org/#/c/602384/ please | |
| 22:09:15 | openstackgerrit | Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941 | |
| 22:09:43 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Delete the placement code https://review.openstack.org/618215 | |
| 22:13:53 | mriedem | efried: turns out https://review.openstack.org/#/c/619061/ does fix the pg thing | |
| 22:14:03 | mriedem | also, wee lots of red http://logs.openstack.org/05/613305/7/check/tempest-full/999ec9f/controller/logs/screen-n-api.txt.gz?level=ERROR | |
| 22:14:08 | mriedem | unrelated regression | |
| 22:14:37 | efried | ack x2 | |
| 22:15:35 | efried | mriedem: What were the func test failures? Actual differences in results? | |
| 22:15:56 | mriedem | dansmith: looks like an unintended side effect of using the scatter_gather_single_cell for nova show is the scatter thing logs errors from the query^ even for things we expect | |
| 22:16:07 | mriedem | efried: yeah, got 5 rows when 1 expected | |
| 22:16:12 | mriedem | efried: pull it down and try it out | |
| 22:16:17 | efried | okay, must be aggregate functions | |
| 22:16:27 | efried | nah, higher priorities. | |
| 22:16:28 | dansmith | mriedem: oh yeah, I think I called that out initially and then totally forgot :( | |
| 22:16:34 | dansmith | it's spewing errors to the logs right? | |
| 22:16:36 | mriedem | yes | |
| 22:16:39 | mriedem | i'll open a bug | |
| 22:16:40 | efried | If it fixes the problem, let's roll with it. | |
| 22:19:09 | mriedem | https://bugs.launchpad.net/nova/+bug/1804325 | |
| 22:19:09 | openstack | Launchpad bug 1804325 in OpenStack Compute (nova) "InstanceNotFound traceback errors in n-api logs while polling for server delete" [High,Triaged] | |
| 22:20:16 | mriedem | so, i think we just remove that exception line since the caller can get the actual exception type now | |
| 22:20:20 | mriedem | and decide if it needs to log | |
| 22:20:33 | mriedem | dansmith: you want it or shall i? | |