Posted	Nick	Remark
#openstack-nova - 2018-11-20
20:14:26	dansmith	yep
20:14:45	dansmith	it's grotesque yet oddly satisfying
20:14:56	jaypipes	ewww.
20:15:17	jaypipes	dansmith: are you referring to the eagerload thing?
20:15:26	dansmith	no
20:15:35	dansmith	jaypipes: in the oldentimes,
20:15:47	dansmith	we would load an instance, joined with metadata, system_metadata, etc
20:16:11	dansmith	which would end up returning X*Y*Z rows for X instances, with Y rows of metadata and Z rows of sysmeta
20:16:17	jaypipes	right.
20:16:29	dansmith	which was the reason RAX failed to deploy icehouse after we moved flavor data to sysmeta
20:16:40	dansmith	now we query metadata separately from the actual instance load
20:17:05	dansmith	but I thought a convo with you a long time ago yielded you saying that we could do an inner-outer-blue-unicorn join to avoid that somehow
20:17:54	jaypipes	yes, we can do a single query to get all instance metadata (and sysmeta) for all selected instances.
20:18:21	dansmith	to be clear,
20:18:24	jaypipes	a query that would just yield (instance_uuid, key, value) tuples.
20:18:33	dansmith	no, that's not what I'm asking
20:19:18	dansmith	the old query was returning the instance data itself, and the metadata key,value and the sysmeta key,value
20:19:31	dansmith	so the instance data was repeated for every row in meta, sysmeta
20:19:48	dansmith	that was a single query for the instance itself, and the metadata(s)
20:20:10	dansmith	I know we can query the instance, and then query for the metadatas efficiently
20:20:12	dansmith	but it's two queries
20:20:27	dansmith	that's basically what we do now
20:20:50	jaypipes	ok. well, that's the most efficient way to solve this particular problem.
20:20:56	dansmith	two queries?
20:20:59	jaypipes	yup.
20:21:03	dansmith	okay
20:21:19	dansmith	I'm pretty sure you called me a stupid ignoramus for doing that back in the icehouse days,
20:21:21	jaypipes	I thought we were doing >1 query for grabbing instance metadata and system metadata
20:21:33	dansmith	well, we are, but only for convenience
20:21:34	jaypipes	no, I never called you anything.
20:21:38	dansmith	YOU DID
20:22:10	dansmith	we're doing three queries now, instance, meta, sysmeta
20:22:11	jaypipes	if I ever said anything about performance it would have been because we were doing a query on instances, then *for each instance* issuing a query to get some instance metadata.
20:22:21	dansmith	we're not doing that
20:22:25	jaypipes	ok, coools.
20:22:37	jaypipes	I can reduce the meta + sysmeta to a single query.
20:22:42	dansmith	so I guess that means I have to keep plugging at this
20:22:52	dansmith	jaypipes: yea, feel free, but that's separate from my other work here
20:23:08	jaypipes	k. is there anything I can help you with on your work here?
20:23:21	dansmith	jaypipes: you just did
20:23:24	dansmith	thanks
20:23:25	jaypipes	:)
20:23:40	jaypipes	well, at least that gives me something to smile about today.
20:23:42	jaypipes	thanks.
20:25:19	jaypipes	FTR, on the cyborg thing, I *also* believe that cyborg should be modeling should be inventories, not RPs with tons of traits masquerading as resource classes.
20:25:38	jaypipes	dansmith: ^
20:25:47	dansmith	jaypipes: yay.
20:25:59	jaypipes	indeed.
20:26:04	mriedem	dansmith: on top of the improved join on what you're doing, i think it's a 2-part change in that the metadata api doesn't need to be pre-loading on system_metadata - at least not anymore
20:26:42	dansmith	mriedem: yeah, so that will address the acute issue right?
20:26:45	mriedem	last i looked the only thing in meta-api that would use sysmeta is a vendor data provider if configured
20:26:53	dansmith	maybe I should punt this until we have a better reason to do this work
20:27:02	mriedem	i believe so, and i think that's what the workday ops guy said he did in the ML
20:27:06	mriedem	heh
20:27:18	mriedem	see, i started trying to do what you said and sparks flew immediately
20:27:19	mriedem	and i gave up
20:27:34	dansmith	oh did you
20:27:35	dansmith	?
20:27:40	mriedem	locally
20:27:45	dansmith	maybe that should be my impetus to fix it
20:27:45	mriedem	never pushed it up b/c tests failed horribly
20:27:52	mriedem	to show me up?
20:27:55	dansmith	yeah
20:27:59	mriedem	by all means
20:28:06	dansmith	nah, sounds hard.
20:28:12	mriedem	next you can fix the nova/cinder cross az attach mess
20:28:29	mriedem	which i have a fix for, but it's fugly as all get out
20:29:09	dansmith	I like it already
20:40:34	efried	dansmith, jaypipes: Modeling accelerators via specific resource classes, so like CUSTOM_FPGA_GZIP rather than rc=FPGA + traits=[GZIP] ?
20:41:05	dansmith	GZIP isn't a trait, IMHO
20:41:18	dansmith	like, I don't ask for SOME_SILICON=1024, trait=RAM
20:41:19	efried	um. The FPGA is capable of processing gzips?
20:42:01	jaypipes	the resource class is a context to a GZIP program flashed to a device.
20:42:04	dansmith	right, but there's a difference between asking for an FPGA and asking for GZIP offload to me
20:42:18	jaypipes	dansmith++
20:42:26	dansmith	if I want an FPGA that I can program myself, I want an FPGA=1.. if I want a GZIP handler, I want GZIP=1,
20:42:32	dansmith	which might be an FPGA in the back end
20:42:37	dansmith	or it might be an ASIC
20:42:39	dansmith	or whatever
20:44:01	efried	But then a resource provider representing a blank (as-yet-unprogrammed) FPGA would have to show inventories of multiple resource classes, and then when one of those is consumed, we would have to nix the other resource classes (or do the reserved=total trick for them).
20:44:17	efried	which is racy, as well as being ew.
20:44:19	dansmith	same for the trait right?
20:44:32	dansmith	you say FPGA=1, trats=GZIP,TLS,BITCOIN
20:44:51	efried	no. GZIP_CAPABLE stays. Not sure we have to retrait every time we reprogram.
20:45:32	efried	but if we do, the GZIP_IS_ON_THIS_THING_AT_THE_MOMENT trait would be separate, and have separate meaning, than the GZIP_CAPABLE.
20:45:57	dansmith	you're just providing no abstraction there
20:46:02	efried	the former would be used only as an optimization, if/when we have "preferred traits", to avoid reprogramming if there's one that's already set up.
20:46:30	jaypipes	premature optimization...
20:47:03	efried	I'd be happy if we skipped that whole bit for the first pass and just used the *_CAPABLE traits.
20:47:14	efried	Programming gets done after the claim, if and as necessary.
20:47:23	jaypipes	I'd be happy if we just skipped everything other than just using custom resource classes.
20:47:46	efried	jaypipes: So preprogram everything?
20:48:05	dansmith	pretty sure we said the first step was assuming everything was static, no?
20:48:22	dansmith	except for the "user will program it themselves" case of course
20:48:44	jaypipes	dansmith++ again.
20:49:21	dansmith	when we talked about this in (denver I think?) I think the overwhelming majority of cases where this really applies is the pre-programmed case,
20:49:39	dansmith	because it provides for locality in certain FGPAs that have one code region and multiple execution contexts,
20:50:11	dansmith	such that if you co-locate a GZIP and a TLS, they both can't use the same FPGA, but if you get two GZIP tenants on the same box, they can
20:50:41	dansmith	and I thought we agreed to avoid boiling the ocean with "everything is completely dynamic all the time forever" until we could do, you know, fucking anything :)
20:50:56	efried	I can buy it for a first pass. Long-term, that seems like not very cloudy. Though I suppose if the "pre"programming is done by a higher orchestrator, it could fly.

Earlier   Later