Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-06-17
16:33:53 sean-k-mooney TheJulia: there are ways to do per instance raid config today howere right
16:34:01 TheJulia which _should_ make the world a happier place. Dogs and cats could get married and all
16:34:51 dansmith whatever, it's just avoiding an abstraction because it's hard
16:34:52 TheJulia sean-k-mooney: it is not just raid sadly, the convergence seemed to be that there are teams with whole kickstart configs and to try and articulate that with traits will just make flavor/trait explosion even worse.
16:35:03 dansmith even though our only purpose in life is to be an abstraction
16:37:01 TheJulia in a perfect end state, I'd prefer them to get everything in glance and then tie it all together on an image basis, but smaller building blocks
16:37:28 sean-k-mooney TheJulia: well im not sure i agree that this should all live in glance or in the flavor
16:37:36 sean-k-mooney some of it should be user-data
16:38:00 dansmith it's not a smaller building block, it's a hammer to break the existing blocks
16:38:01 dansmith it doesn't build us toward a solution, it's an exit plan
16:38:43 dansmith instead of ever building new blocks, it's a forever solution of "get it to 40% complete and then do the rest custom out of band"
16:39:46 sean-k-mooney TheJulia: it sound like a way to pass info on a per instance basis to the ironic python agent would partly solve there issue. e.g. if the agent had a defien api of actions such as bios and raid config that could be pass on a per instance basis it would help with the image/flavor explosion issue
16:39:58 TheJulia woudl it help to have a larger discussion regarding this again, because where this always stalled out was the "how do we end up having something that nova will agree to" in the past in terms of data. and I think the last agreement of a possible solution involved templates stored in glance associated via traits or something which I think made it complex and cumbersome to interact with
16:40:19 sean-k-mooney the problem is that that for that to make sense it has to be somewhat abstract as we will not know the precise host ahead of time by design
16:40:55 dansmith we did have a long conversation about this in dublin,
16:41:03 TheJulia sean-k-mooney: we're actually have it, just turns out people want to do even more complex things their own way
16:41:10 dansmith which involved nova just knowing about a hardware template that gets passed to ironic to define these things
16:41:27 dansmith there was no tableflipping
16:41:54 sean-k-mooney dansmith: to do that we would want to do it in a way that did not invovle specifying the template in the flavor or image right
16:41:59 TheJulia maybe I was thinking of the time before
16:42:30 sean-k-mooney such as a swift url or glance artifcact uuid in server metadata or something else instance based
16:42:31 dansmith sean-k-mooney: I dunno, in the image seems fine to me
16:42:44 sean-k-mooney dansmith: in the image would work only to a point
16:42:55 openstackgerrit Alexandre Arents proposed openstack/nova stable/stein: Add admin doc information about image cache resource accounting https://review.opendev.org/732542
16:43:07 sean-k-mooney if you assume the raid config is an atribute of that image then sure
16:43:31 sean-k-mooney but if the workload in that image has different performace configurtion it can be deployed in for different slas then you need multiple images
16:43:58 dansmith sean-k-mooney: how the user gets to select it (if at all) is really a smaller concern than all the components supporting that thing regardless of how it's presented
16:44:19 sean-k-mooney true
16:44:57 TheJulia I suspect your thinking that fitting the needs is a conundrum, but in these sorts of cases, the user knows exactly what the hardware is because they have a special flavor to use to match it
16:45:04 dansmith I tend to think that flavors and images are our templating mechanism and optimizing for a one-off deployment of one thing with weird config is not the direction we should go
16:45:29 dansmith TheJulia: right, which is why I think that in the flavor is reasonable in a lot of cases
16:46:00 dansmith they're already selecting an ironic machine class via flavor anyway
16:46:27 TheJulia From the flip side of that, another operator on the call we had today basically stressed that if they ahd to do that their flavors would explode even more and I think they already have tens of thousands
16:47:14 sean-k-mooney other then the fact openstack flavor list will be ... difficut to use did they have a partical problem with that
16:47:19 TheJulia (approximately 350,000 physical machines in their fleet is what they say)
16:47:33 dansmith I suspect that means they're doing it wrong, but regardless, if you literally have thousands of bespoke configurations, you're going to end up with thousands of incompatible objects somewhere in the system
16:47:53 sean-k-mooney TheJulia: but presumable that is grouped into a manageable set of SKUs
16:48:26 dansmith sean-k-mooney: that's why I said I expect they're doing it wrong.. they obviously have less than 350k unique configurations of machines, they probably just want to think of each machine as a snowflake
16:48:30 sean-k-mooney TheJulia: do you know how many unique ironic resouce classes they have
16:49:03 TheJulia sean-k-mooney: I'll ask, I don't know off the top of my head
16:49:26 sean-k-mooney dansmith: right they should have 1 flavor per ironic resouce class + there vm flavors
16:50:15 dansmith right, and not an ironic resource class for every unique serial number :)
16:50:17 TheJulia that is not how anyone is implementing it because they are also needing to access control on the back end who has the ability to access specific resource classes
16:50:36 sean-k-mooney TheJulia: well that is where they are doing it wrong
16:50:53 sean-k-mooney nova has ways to do access contol on flaovr or via aggreates based on tenatns
16:51:18 TheJulia arne_wiebalck: ^^^ fyi, you might find this interesting or super frustrating
16:51:52 sean-k-mooney nova has the idea of private flavors so you can create 1 flaovr an choose what tenats can see it
16:52:31 TheJulia asked the super large operator, I think cern is operating with something like like ~130-140 distinct SKUs across their fleet, some of it is private skus for hyper specific needs/purposes, others are general from what I understand. I believe they've mentioned some crazy number of flavors as well.
16:53:56 TheJulia ugh, apparently the super large operator has lots of downstream patches wrapped around flavors/classes, so their huge numbers are not exactly valid for this discussion since we don't understand the entire story/composition there.
16:54:44 sean-k-mooney i can see them doing thing like adding a customer specific resouce class so they can dedicate specifc pools to them
16:54:55 sean-k-mooney but that is not how its ment to be done
16:55:07 TheJulia sean-k-mooney: I _think_ cern has mentioned them doing private flavors before, but that it adds up for admins to dump the list out
16:55:46 mnaser fwiw openstack cli shows public flavors only by default, even for admins
16:55:52 mnaser you have to explicitly give it --private to get those
16:56:10 dansmith sean-k-mooney: exactly
16:56:26 sean-k-mooney well it can but when you create private flavor you dont create the same flavor multiple times for different tenatnts. you create it once and add the teant 1:1 to the existing flavor
16:57:09 sean-k-mooney mnaser: that is true today but i actully want to change how that work. i want ti to be private if you add a tenat too the flavor acess list and public if not
16:57:19 dansmith and again, if you want thousands of different configs, you'll either end up with thousands of flavors, or thousands of instances with whatever bespoke thing you've asked for, it will just be far less discoverable
16:57:32 sean-k-mooney mnaser: at the moemnt you cant change the private public status and that causes pain
16:57:54 dansmith so people having "all the same flavor" which have been individually hand-managed to be different is just going to end up with that madness being hidden
16:58:06 mnaser sean-k-mooney: yeah the move private => public is a bit harsh but im not sure why/how/when it's been the case. i would have never touched the db and manually changed that field before. ever.
16:58:16 dansmith which is kinda the point of what we're doing here -- trying to manage N things as a unit, where N is too large for any human to do manually
16:59:07 mnaser also, i agree about the comments wrt instance type shouldn't mean different things, it sucks to manage it at scale, but, there's tons of ways you can (relatively easily) automate something like this
16:59:10 mnaser say, this ansible
16:59:10 sean-k-mooney mnaser: its useful to go from public to private too for decomisioning old flavors
16:59:30 mnaser sean-k-mooney: yes, that's true too. i'm sure someone out there knows the real reason behind it
16:59:58 mnaser i have risked moving private to public, but not public to private as that seemed a little more intense in terms of the things that can break
17:00:55 sean-k-mooney mnaser: yep i discussed this with gmann a while back and its on the list of things to "clean up" in a future microverions but we have not put up a blueprint for it yet
17:01:36 sean-k-mooney mnaser: when private flavors where added the ability to move between them was just never adressed so in that regard its a half implemnted feature
17:01:58 sean-k-mooney but its one i think we can improve.
17:02:36 sean-k-mooney dansmith: i guess the problem is that right now they are managing the N things with kickstart
17:03:02 sean-k-mooney and not tacking advantage of the abstractions provided by openstack
17:03:08 dansmith sean-k-mooney: I thought the tenants were doing that, but yeah
17:04:35 TheJulia sean-k-mooney: no, they are only using kickstart files to rebuild machines
17:04:48 TheJulia because they simply need a way more complex running configuration
17:05:27 sean-k-mooney TheJulia: right but if openstack coudl provide that config would they use it or whould they keep doing it the way they are now so that they can support non openstack envs
17:07:05 sean-k-mooney by the way i do not use ironic that often so im not sure what capablity you can use via ironic direct vs ironic via nova but even though ironic direct can leverage ironic to do raid and bios config today they are not using it
17:07:50 TheJulia sean-k-mooney: in a perfect world, they would want to be able to re-use the same config or formatting instead of having to go through a bunch of additional steps or abstractions to just get the job done for the day. The point here is they want everything through openstack because of the single pane of glass benefit.
17:08:42 sean-k-mooney TheJulia: right which is why im confused that they would reimage the machine since that is the opisiste of doing everythin vis openstack
17:09:35 dansmith sean-k-mooney: they want to put a break statement anywhere they want in the workflow to use as much or little openstack as they want
17:09:46 TheJulia sean-k-mooney: because the constraints in which openstack puts them with are insufficent to meet their needs of their business for their 10, 100, 1000 machines they may ask for on a given day
17:10:29 gmann mnaser: sean-k-mooney: on flavor things, its in list to cleanup - item 10 in https://etherpad.opendev.org/p/nova-api-cleanup
17:10:59 gmann which is basically get rid of is_public thing and every flavor is private and accessible to allowed tenants only
17:11:01 sean-k-mooney gmann: yep i knew you added it to the list but i did not have it bookmarked
17:11:31 sean-k-mooney gmann: that is not what i had in mind
17:12:01 gmann and that bug adding public flavor things is since liberty - https://bugs.launchpad.net/nova/+bug/1361476
17:12:01 openstack Launchpad bug 1361476 in OpenStack Compute (nova) "flavor access create should check public/private first" [Low,Fix released] - Assigned to Sergey Nikitin (snikitin)
17:12:03 sean-k-mooney gmann: i was thinking every flavor would be public unless you addd a tenatn via flavor access in which case it would only be visable to those that were in the list
17:12:45 gmann i sex, what is is_public default?
17:12:52 gmann sec
17:13:06 sean-k-mooney it defaults to public/true
17:13:13 sean-k-mooney so i wanted to keep that behavior
17:13:40 sean-k-mooney but instead of seting it to public or private explictly base it on the flavor acess info
17:14:19 gmann yeah https://github.com/openstack/nova/blob/6ed556f4adfd8a98fae2e91204f1ce0484f5dd9d/nova/api/openstack/compute/flavor_manage.py#L69
17:14:59 sean-k-mooney e.g. is_public = "select count(*) form flavor_access where flavor_id = $whatever" == 0
17:15:06 gmann right, keeping the same behavior but allowing it to add tenant access and so does list
17:15:43 openstackgerrit Sean McGinnis proposed openstack/nova master: Use unittest.mock instead of third party mock https://review.opendev.org/714676
17:16:21 sean-k-mooney right so you start with a public flaovr. then if you add a tenant to the acess list it "private" and only that tenatn and an admin can see it
17:16:30 sean-k-mooney if you remove them it avaiable to all again
17:17:29 sean-k-mooney nova flavor-access-list --flavor <my public flavor> should have eitehr retruned None

Earlier   Later