Posted	Nick	Remark
#openstack-nova - 2020-06-17
16:19:44	dansmith	I dunno if there would be any ironic parallels here, but changing the image from inside the guest means that anything we do after that based on the image metadata will be wrong
16:20:06	sean-k-mooney	i basically just said thats not supported sorry, it just a leaky abstraction
16:20:18	dansmith	like attaching a device and using the correct device model, or migrating somewhere that looks at image metadata for OS properties, etc
16:20:26	sean-k-mooney	dansmith: well so i have heard of people upgradeing form like 16.04 to 18.04
16:20:37	sean-k-mooney	but then you get into the pets vs cattle debate
16:20:55	dansmith	which is wholly uncloudy, but is a lot less bad than going from 16.04 to Windows 10
16:20:59	dansmith	yup
16:21:46	sean-k-mooney	ya updating the same disto is not really a problem different openrating systemsn entierely like linux to windows or esxi is a tolltal differnt matter
16:25:01	sean-k-mooney	TheJulia: honestly it sound like they should update there kickstart playbook to delegate the os imagiang and provisioning to openstack/ironic and then just deal with boot straping the deployed hosts
16:26:13	TheJulia	the case that is coming up at CERN is apparently they are deploying nodes with configurations that are impossible to articulate in images, so they are making the necessary major changes to get to their required end state after the fact
16:27:04	sean-k-mooney	TheJulia: sure that makes sense but i dont know wny they would not just orcestrat the software payload via ansible and have a seperate role to deploy the orginal server
16:27:19	sean-k-mooney	TheJulia: kind of like kaobe
16:27:27	TheJulia	Really, ideally they should jsut be talking with ironic in some of these cases, but I guess they are trying to force their users into cloudy mindsets/patterns/behaviors
16:28:22	dansmith	yup
16:28:33	TheJulia	It is a mixed vm/baremetal environment, fwiw
16:29:14	dansmith	what are the "node configurations that are impossible to articulate in images" ?
16:29:17	TheJulia	And high level consistency seems to be the key they need, it is almost like power users just need that extra next step
16:29:22	sean-k-mooney	ya that is not really an issue johnthetubaguy could proably tell you more about how they orcestrate the deployment of the Square Kilometer Array (SKA)
16:29:34	TheJulia	multi-disk/raid/lvm/etc
16:29:51	TheJulia	dansmith: this came out of the whole disk partitioning discussion getting resurrected
16:30:18	dansmith	ack, well, addressing that makes more sense to me than just saying "short-circuit the bottom half of nova because it's easier"
16:30:54	TheJulia	except, then get into the whole pets/cattle argument again and everyone tableflips and walks away
16:31:29	sean-k-mooney	cern have a blog post on that but it seam sto be down the stack hpc one is up however https://www.stackhpc.com/software-raid-in-ironic.html
16:31:38	dansmith	that's not a convincing argument to do the short-circuit :)
16:32:54	TheJulia	dansmith: it is a solid reason people have downstream patches and are why they are not upgrading though, because their business processes need something along those lines. Anyway, if we enable them to short circuit in ironci, I think that is fine. Nova will believe it is happy. they can do whatever special config assertion as long as we populate leasee which we should be doing anyway.
16:33:53	sean-k-mooney	TheJulia: there are ways to do per instance raid config today howere right
16:34:01	TheJulia	which _should_ make the world a happier place. Dogs and cats could get married and all
16:34:51	dansmith	whatever, it's just avoiding an abstraction because it's hard
16:34:52	TheJulia	sean-k-mooney: it is not just raid sadly, the convergence seemed to be that there are teams with whole kickstart configs and to try and articulate that with traits will just make flavor/trait explosion even worse.
16:35:03	dansmith	even though our only purpose in life is to be an abstraction
16:37:01	TheJulia	in a perfect end state, I'd prefer them to get everything in glance and then tie it all together on an image basis, but smaller building blocks
16:37:28	sean-k-mooney	TheJulia: well im not sure i agree that this should all live in glance or in the flavor
16:37:36	sean-k-mooney	some of it should be user-data
16:38:00	dansmith	it's not a smaller building block, it's a hammer to break the existing blocks
16:38:01	dansmith	it doesn't build us toward a solution, it's an exit plan
16:38:43	dansmith	instead of ever building new blocks, it's a forever solution of "get it to 40% complete and then do the rest custom out of band"
16:39:46	sean-k-mooney	TheJulia: it sound like a way to pass info on a per instance basis to the ironic python agent would partly solve there issue. e.g. if the agent had a defien api of actions such as bios and raid config that could be pass on a per instance basis it would help with the image/flavor explosion issue
16:39:58	TheJulia	woudl it help to have a larger discussion regarding this again, because where this always stalled out was the "how do we end up having something that nova will agree to" in the past in terms of data. and I think the last agreement of a possible solution involved templates stored in glance associated via traits or something which I think made it complex and cumbersome to interact with
16:40:19	sean-k-mooney	the problem is that that for that to make sense it has to be somewhat abstract as we will not know the precise host ahead of time by design
16:40:55	dansmith	we did have a long conversation about this in dublin,
16:41:03	TheJulia	sean-k-mooney: we're actually have it, just turns out people want to do even more complex things their own way
16:41:10	dansmith	which involved nova just knowing about a hardware template that gets passed to ironic to define these things
16:41:27	dansmith	there was no tableflipping
16:41:54	sean-k-mooney	dansmith: to do that we would want to do it in a way that did not invovle specifying the template in the flavor or image right
16:41:59	TheJulia	maybe I was thinking of the time before
16:42:30	sean-k-mooney	such as a swift url or glance artifcact uuid in server metadata or something else instance based
16:42:31	dansmith	sean-k-mooney: I dunno, in the image seems fine to me
16:42:44	sean-k-mooney	dansmith: in the image would work only to a point
16:42:55	openstackgerrit	Alexandre Arents proposed openstack/nova stable/stein: Add admin doc information about image cache resource accounting https://review.opendev.org/732542
16:43:07	sean-k-mooney	if you assume the raid config is an atribute of that image then sure
16:43:31	sean-k-mooney	but if the workload in that image has different performace configurtion it can be deployed in for different slas then you need multiple images
16:43:58	dansmith	sean-k-mooney: how the user gets to select it (if at all) is really a smaller concern than all the components supporting that thing regardless of how it's presented
16:44:19	sean-k-mooney	true
16:44:57	TheJulia	I suspect your thinking that fitting the needs is a conundrum, but in these sorts of cases, the user knows exactly what the hardware is because they have a special flavor to use to match it
16:45:04	dansmith	I tend to think that flavors and images are our templating mechanism and optimizing for a one-off deployment of one thing with weird config is not the direction we should go
16:45:29	dansmith	TheJulia: right, which is why I think that in the flavor is reasonable in a lot of cases
16:46:00	dansmith	they're already selecting an ironic machine class via flavor anyway
16:46:27	TheJulia	From the flip side of that, another operator on the call we had today basically stressed that if they ahd to do that their flavors would explode even more and I think they already have tens of thousands
16:47:14	sean-k-mooney	other then the fact openstack flavor list will be ... difficut to use did they have a partical problem with that
16:47:19	TheJulia	(approximately 350,000 physical machines in their fleet is what they say)
16:47:33	dansmith	I suspect that means they're doing it wrong, but regardless, if you literally have thousands of bespoke configurations, you're going to end up with thousands of incompatible objects somewhere in the system
16:47:53	sean-k-mooney	TheJulia: but presumable that is grouped into a manageable set of SKUs
16:48:26	dansmith	sean-k-mooney: that's why I said I expect they're doing it wrong.. they obviously have less than 350k unique configurations of machines, they probably just want to think of each machine as a snowflake
16:48:30	sean-k-mooney	TheJulia: do you know how many unique ironic resouce classes they have
16:49:03	TheJulia	sean-k-mooney: I'll ask, I don't know off the top of my head
16:49:26	sean-k-mooney	dansmith: right they should have 1 flavor per ironic resouce class + there vm flavors
16:50:15	dansmith	right, and not an ironic resource class for every unique serial number :)
16:50:17	TheJulia	that is not how anyone is implementing it because they are also needing to access control on the back end who has the ability to access specific resource classes
16:50:36	sean-k-mooney	TheJulia: well that is where they are doing it wrong
16:50:53	sean-k-mooney	nova has ways to do access contol on flaovr or via aggreates based on tenatns
16:51:18	TheJulia	arne_wiebalck: ^^^ fyi, you might find this interesting or super frustrating
16:51:52	sean-k-mooney	nova has the idea of private flavors so you can create 1 flaovr an choose what tenats can see it
16:52:31	TheJulia	asked the super large operator, I think cern is operating with something like like ~130-140 distinct SKUs across their fleet, some of it is private skus for hyper specific needs/purposes, others are general from what I understand. I believe they've mentioned some crazy number of flavors as well.
16:53:56	TheJulia	ugh, apparently the super large operator has lots of downstream patches wrapped around flavors/classes, so their huge numbers are not exactly valid for this discussion since we don't understand the entire story/composition there.
16:54:44	sean-k-mooney	i can see them doing thing like adding a customer specific resouce class so they can dedicate specifc pools to them
16:54:55	sean-k-mooney	but that is not how its ment to be done
16:55:07	TheJulia	sean-k-mooney: I _think_ cern has mentioned them doing private flavors before, but that it adds up for admins to dump the list out
16:55:46	mnaser	fwiw openstack cli shows public flavors only by default, even for admins
16:55:52	mnaser	you have to explicitly give it --private to get those
16:56:10	dansmith	sean-k-mooney: exactly
16:56:26	sean-k-mooney	well it can but when you create private flavor you dont create the same flavor multiple times for different tenatnts. you create it once and add the teant 1:1 to the existing flavor
16:57:09	sean-k-mooney	mnaser: that is true today but i actully want to change how that work. i want ti to be private if you add a tenat too the flavor acess list and public if not
16:57:19	dansmith	and again, if you want thousands of different configs, you'll either end up with thousands of flavors, or thousands of instances with whatever bespoke thing you've asked for, it will just be far less discoverable
16:57:32	sean-k-mooney	mnaser: at the moemnt you cant change the private public status and that causes pain
16:57:54	dansmith	so people having "all the same flavor" which have been individually hand-managed to be different is just going to end up with that madness being hidden
16:58:06	mnaser	sean-k-mooney: yeah the move private => public is a bit harsh but im not sure why/how/when it's been the case. i would have never touched the db and manually changed that field before. ever.
16:58:16	dansmith	which is kinda the point of what we're doing here -- trying to manage N things as a unit, where N is too large for any human to do manually
16:59:07	mnaser	also, i agree about the comments wrt instance type shouldn't mean different things, it sucks to manage it at scale, but, there's tons of ways you can (relatively easily) automate something like this
16:59:10	mnaser	say, this ansible
16:59:10	sean-k-mooney	mnaser: its useful to go from public to private too for decomisioning old flavors
16:59:30	mnaser	sean-k-mooney: yes, that's true too. i'm sure someone out there knows the real reason behind it
16:59:58	mnaser	i have risked moving private to public, but not public to private as that seemed a little more intense in terms of the things that can break
17:00:55	sean-k-mooney	mnaser: yep i discussed this with gmann a while back and its on the list of things to "clean up" in a future microverions but we have not put up a blueprint for it yet
17:01:36	sean-k-mooney	mnaser: when private flavors where added the ability to move between them was just never adressed so in that regard its a half implemnted feature
17:01:58	sean-k-mooney	but its one i think we can improve.
17:02:36	sean-k-mooney	dansmith: i guess the problem is that right now they are managing the N things with kickstart
17:03:02	sean-k-mooney	and not tacking advantage of the abstractions provided by openstack
17:03:08	dansmith	sean-k-mooney: I thought the tenants were doing that, but yeah
17:04:35	TheJulia	sean-k-mooney: no, they are only using kickstart files to rebuild machines
17:04:48	TheJulia	because they simply need a way more complex running configuration

Earlier   Later