Posted	Nick	Remark
#openstack-nova - 2020-01-29
15:13:30	stephenfin	I'm almost certain someone will go tweak the YAML file we provide though
15:13:39	bauzas	yuuuuup
15:13:46	dansmith	stephenfin: I would argue that we don't/shouldn't encode our baseline validations into yaml and distribute them that way, as it's just asking for that problem
15:13:49	dansmith	right
15:13:52	sean-k-mooney	dansmith: so i was suggesting add a &validate query arge to turn off validation in the new microversion
15:13:53	efried	If we do in-tree stuff with yaml, we put the nova-owned stuff within the nova package so nobody fs with it.
15:14:06	dansmith	efried: riiiight
15:14:07	stephenfin	sean-k-mooney: hold up, let's figure the YAML thing out first
15:14:38	efried	riiiight they'll still f with it? Let 'em. That's breaking the seal, voiding the warranty, not different than if they go patch the code.
15:14:46	sean-k-mooney	efried: distos will modify it every time we backport stuff
15:14:51	gibi	we somebody want to tweak what is written in the python DSL for in-tree validation he can do it, it is almos like a yaml file
15:14:52	dansmith	surely the yaml to validate something complex like numa whatever is going to be nasty or naive right?
15:15:12	sean-k-mooney	it will be an interesting regex
15:15:23	gibi	but nothing more than a regex
15:15:26	stephenfin	gibi: I don't think it's a DSL, really. It's just an object. I could use dicts if that would be preferable
15:15:28	dansmith	an "interesting" regex or some relatively simple python right?
15:15:42	bauzas	:)
15:15:43	sean-k-mooney	actully its still a regex in stephns code
15:15:49	sean-k-mooney	well poc
15:15:56	stephenfin	not entirely
15:15:57	sean-k-mooney	but it could be simpler
15:16:11	stephenfin	I let you specify enums etc.
15:16:19	sean-k-mooney	yes
15:16:37	stephenfin	granted, all that gets converted into a regex but there's no reason I have to do that for all cases. It's a PoC after all
15:16:43	sean-k-mooney	although you convert the enum int an bool filed back to regex form
15:17:25	dansmith	so I think my feeling on this is: (a) I'm not sure we need an interface for people to be able to specify their own validators, but (b) if we do, that seems like an obvious use case for entrypoints and a dead simple plugin interface with a super-controlled input and output behavior set
15:17:51	efried	gibi: it won't be a lot of duplication, really. The parser will construct the same kind of python objects we have in tree and then feed all of them into the validation engine.
15:18:04	stephenfin	gibi: yeah, what efried said
15:18:19	gibi	OK, I rest my case then
15:18:21	stephenfin	it's just a slightly different representation of the exact same data
15:18:30	stephenfin	dansmith: So kill the YAML idea entirely?
15:19:39	stephenfin	If so, the main issue I'd see with that is that they'd have to write and install a Python package for the custom stuff
15:19:49	dansmith	stephenfin: I don't think our validations should be in yaml, and I don't really see why custom ones should be either.. they'd be a lot harder to test than a single python function you can run with a string you would put in your flavor
15:20:07	bauzas	stephenfin: I don't see a problem here, we asked for that since a while
15:20:25	efried	If we do that, we can't have strict validation. Writing python code and figuring out stevedore for snowflake extra specs is too high a barrier to entry for that to be the only option.
15:20:31	dansmith	stephenfin: I think that would be a pretty simple example thing, they have to do that for scheduler filters, and I expect anyone that feels they need this (honestly I don't see it being that important) will be fine with that standard
15:20:36	bauzas	(like, Blazar was having a specific nova-ish package for nova plugins he needed)
15:21:06	stephenfin	I was about to ask what the story with extra scheduler filters was. Those are a package too so
15:21:06	bauzas	yeah, it's very similar to the scheduler filters case
15:21:13	bauzas	yup
15:21:22	bauzas	or delivered by other means
15:21:27	stephenfin	efried: Are snowflake extra specs a real thing people use?
15:21:27	dansmith	efried: stevedore is on the loader side, they don't have to do anything like that.. entry points are a stock python thing
15:21:50	stephenfin	Far as I knew, extra specs were for scheduler filters and virt drivers
15:21:53	sean-k-mooney	they use custom one for filter
15:21:57	efried	f man, even if you managed to miss one of the stock ones
15:22:00	dansmith	stephenfin: you can in conjunction with the json filter or yeah a custom scheduler filter
15:22:19	dansmith	stephenfin: if you're writing a custom scheduler flter, you're doing a package, so throwing a validator in there makes plenty of sense
15:22:29	stephenfin	If it's a custom filter, we're saying they're using a package already so no big deal
15:22:31	stephenfin	dansmith: yeah
15:22:37	gibi	my employer's downstream product has at least two snowflake extra spec
15:22:48	gibi	but sure I can implement a python filter for that
15:22:49	sean-k-mooney	same with a thrid party virt dirver
15:22:50	dansmith	stephenfin: imagine the fame and fortune if you wrote up a nice blog post on how to do this super cleanly :D
15:23:00	dansmith	sean-k-mooney: shut yo' mouth! :D
15:23:11	sean-k-mooney	:)
15:23:31	stephenfin	If I can do one thing well, it's write lots of docs and bug people continuously to review them
15:23:39	bauzas	scheduler filters are the exact reasons why we never constrained extra specs
15:23:42	stephenfin	no issues there
15:24:06	bauzas	so I guess 3rd-parties would *love* to consider filters and validation code to be considered equal
15:24:10	sean-k-mooney	so that bring up stict vs permissive vs off
15:24:17	bauzas	so they could package this at once
15:24:23	stephenfin	efried: I don't think that's as big a deal as you think. Realistically, how often do people create new flavors
15:24:30	sean-k-mooney	we could skip all non namespaced extra space or ones in a namespace we dont know about
15:24:37	sean-k-mooney	in permisive mode at least
15:24:52	stephenfin	They try create one, we fail because we don't recognize the spec, they report and bug and we add/backport the missing validator
15:24:59	stephenfin	It's just a bug
15:25:21	stephenfin	*report a bug
15:25:47	dansmith	are you going to make admins able to create flavors that fail validation
15:25:48	dansmith	?
15:25:58	stephenfin	dansmith: that's the question
15:26:11	stephenfin	I want a microversion that starts failing extra specs that don't pass validation
15:26:19	dansmith	if there is a &force=yes then that gives them a way past the validation if there's a bug
15:26:57	sean-k-mooney	dansmith: that is what is was kind of suggestin with &validate=strict|permissive|off
15:27:00	stephenfin	and I'm arguing that ^ is unnecessary since we should be able to audit the extra specs we have in-tree
15:27:14	dansmith	sean-k-mooney: oh per-create, I thought you meant global config
15:27:26	sean-k-mooney	no per request
15:27:40	stephenfin	efried suggested global config. I said no cos config-driven API behavior is bad
15:27:42	dansmith	stephenfin: yeah, I dunno.. I think some of them are complex enough that you might not be able to do that as well as you think
15:27:48	dansmith	stephenfin: agree
15:28:31	dansmith	stephenfin: but I think it's legit to have a validation=no per-request, but that can return 403 either because of policy or config if it's documented when you add it, no? so later we could turn that off or default it off
15:29:13	stephenfin	hmm, that wouldn't be so bad
15:29:27	dansmith	either way, I just think you might not want to assume you can enumerate every crazy way people have abused those extra_specs and having a bunch of angry people calling your home :D
15:29:45	sean-k-mooney	the reason we would wont that is if we add this in 2.68 and in 2.69 we add another field to the flavor but we still want to disable validtion it would be nice to be able to turn it off
15:30:18	bauzas	honestly, leave them time to change their flavors
15:30:21	stephenfin	sean-k-mooney: yeah, I've been suggesting using the older microversion purely as a stop gap to allow us iron out the kinks
15:30:31	dansmith	IMHO, and AFAICT, validation is *just* to help make the admin's life easier so they don't have to keep booting instances to test that they completely broke an extra_spec right?
15:30:42	bauzas	also, are we proposing to fix existing embedded flavors ?
15:30:45	sean-k-mooney	but that would not work if the thing i wanted to set was added in 2.69
15:30:45	dansmith	so being able to turn it off in an emergency doesn't seem like a bad thing to me
15:30:54	stephenfin	This also won't break existing flavors. It'll only prevent them creating new broken flavors
15:30:55	dansmith	you're not going to periodically fsck your flavors so, what does it matter?
15:30:58	bauzas	stephenfin: I mean, instance's embedded flavors
15:31:30	sean-k-mooney	stephenfin: creating or updating
15:31:40	stephenfin	bauzas: Nope. Out of scope for this. This focuses purely on setting new flavor extra specs
15:31:47	sean-k-mooney	you want to be able to do the valiation optionaly on flaovr updates too
15:32:03	dansmith	sean-k-mooney: that's true, if you're updating a flavor that currently doesn't pass validation
15:32:12	stephenfin	We could look into doing that with a nova-manage command (or nova-audit) command in the future but I'm not even thinking about that at the moment
15:32:38	stephenfin	sean-k-mooney: Well, setting or updating a new extra spec
15:32:42	sean-k-mooney	bauzas: ya for not if you need to update embeded flavor resize

Earlier   Later