Posted	Nick	Remark
#openstack-nova - 2020-06-17
15:55:13	melwitt	gibi: oh maybe, thanks. I missed that, was looking for "gate" in ML
15:55:19	gmann	melwitt: that is same issue grenade multinode job facing
15:55:29	gmann	and need fix in devstack-gate
15:56:04	melwitt	oh, the error message was different so I didn't realize it was related
16:11:09	TheJulia	dansmith: the goal is more, have the nova-compute driver ask ironic to hold the node, set appropriate values to represent the user has possesion of it and allow the user to do the needful.
16:11:41	TheJulia	sean-k-mooney: the conundrum is the users expect to use nova, and currently get resources via nova today, and what they are doing is basically going and making internal invasive chagnes inside the deployed instance to re-install the instance using a kickstart
16:11:58	dansmith	TheJulia: yeah that's the same as what I said, just with nova-compute being slightly involved in the enablement of the out-of-band activities
16:12:26	TheJulia	Well, it may be logical to enable setting the "leasee" field in ironic's api anyway
16:12:39	TheJulia	which would allow operators to optionally grant direct api rights
16:13:05	TheJulia	I guess there could always be a trait that populates itself that just signals "don't actually deploy this machine, just allocate it"
16:13:15	dansmith	that's akin to nova setting up a libvirt VM and then tolerating the user or admin mucking with the underlying VM XML
16:13:18	TheJulia	and we could do that under the hood in ironic fairly easily.
16:13:21	dansmith	which today we declare as "not okay"
16:15:12	TheJulia	Well, we should still eventually store who asked for the thing and if we have community consensus behind a "delayed deployment" trigger, then that is totally up to the operators and their running policy to decide if they want to permit or not to meet their use cases
16:17:23	sean-k-mooney	dansmith: for what its worth we had someone asking about pxe booting nova instance a few weeks ago
16:18:01	sean-k-mooney	they were confused by the fact that if you enable the qemu boot menu via the image porperty you can select pxeboot as an option in the boot menu via the console
16:18:24	sean-k-mooney	but could not figure out how to configure it which makes sense since we dont support you pxe booting the vms booted by nova
16:18:27	dansmith	sean-k-mooney: you mean asking again
16:19:04	sean-k-mooney	it came up on irc like a month ago i assuem it has come up before
16:19:09	dansmith	many times
16:19:44	dansmith	I dunno if there would be any ironic parallels here, but changing the image from inside the guest means that anything we do after that based on the image metadata will be wrong
16:20:06	sean-k-mooney	i basically just said thats not supported sorry, it just a leaky abstraction
16:20:18	dansmith	like attaching a device and using the correct device model, or migrating somewhere that looks at image metadata for OS properties, etc
16:20:26	sean-k-mooney	dansmith: well so i have heard of people upgradeing form like 16.04 to 18.04
16:20:37	sean-k-mooney	but then you get into the pets vs cattle debate
16:20:55	dansmith	which is wholly uncloudy, but is a lot less bad than going from 16.04 to Windows 10
16:20:59	dansmith	yup
16:21:46	sean-k-mooney	ya updating the same disto is not really a problem different openrating systemsn entierely like linux to windows or esxi is a tolltal differnt matter
16:25:01	sean-k-mooney	TheJulia: honestly it sound like they should update there kickstart playbook to delegate the os imagiang and provisioning to openstack/ironic and then just deal with boot straping the deployed hosts
16:26:13	TheJulia	the case that is coming up at CERN is apparently they are deploying nodes with configurations that are impossible to articulate in images, so they are making the necessary major changes to get to their required end state after the fact
16:27:04	sean-k-mooney	TheJulia: sure that makes sense but i dont know wny they would not just orcestrat the software payload via ansible and have a seperate role to deploy the orginal server
16:27:19	sean-k-mooney	TheJulia: kind of like kaobe
16:27:27	TheJulia	Really, ideally they should jsut be talking with ironic in some of these cases, but I guess they are trying to force their users into cloudy mindsets/patterns/behaviors
16:28:22	dansmith	yup
16:28:33	TheJulia	It is a mixed vm/baremetal environment, fwiw
16:29:14	dansmith	what are the "node configurations that are impossible to articulate in images" ?
16:29:17	TheJulia	And high level consistency seems to be the key they need, it is almost like power users just need that extra next step
16:29:22	sean-k-mooney	ya that is not really an issue johnthetubaguy could proably tell you more about how they orcestrate the deployment of the Square Kilometer Array (SKA)
16:29:34	TheJulia	multi-disk/raid/lvm/etc
16:29:51	TheJulia	dansmith: this came out of the whole disk partitioning discussion getting resurrected
16:30:18	dansmith	ack, well, addressing that makes more sense to me than just saying "short-circuit the bottom half of nova because it's easier"
16:30:54	TheJulia	except, then get into the whole pets/cattle argument again and everyone tableflips and walks away
16:31:29	sean-k-mooney	cern have a blog post on that but it seam sto be down the stack hpc one is up however https://www.stackhpc.com/software-raid-in-ironic.html
16:31:38	dansmith	that's not a convincing argument to do the short-circuit :)
16:32:54	TheJulia	dansmith: it is a solid reason people have downstream patches and are why they are not upgrading though, because their business processes need something along those lines. Anyway, if we enable them to short circuit in ironci, I think that is fine. Nova will believe it is happy. they can do whatever special config assertion as long as we populate leasee which we should be doing anyway.
16:33:53	sean-k-mooney	TheJulia: there are ways to do per instance raid config today howere right
16:34:01	TheJulia	which _should_ make the world a happier place. Dogs and cats could get married and all
16:34:51	dansmith	whatever, it's just avoiding an abstraction because it's hard
16:34:52	TheJulia	sean-k-mooney: it is not just raid sadly, the convergence seemed to be that there are teams with whole kickstart configs and to try and articulate that with traits will just make flavor/trait explosion even worse.
16:35:03	dansmith	even though our only purpose in life is to be an abstraction
16:37:01	TheJulia	in a perfect end state, I'd prefer them to get everything in glance and then tie it all together on an image basis, but smaller building blocks
16:37:28	sean-k-mooney	TheJulia: well im not sure i agree that this should all live in glance or in the flavor
16:37:36	sean-k-mooney	some of it should be user-data
16:38:00	dansmith	it's not a smaller building block, it's a hammer to break the existing blocks
16:38:01	dansmith	it doesn't build us toward a solution, it's an exit plan
16:38:43	dansmith	instead of ever building new blocks, it's a forever solution of "get it to 40% complete and then do the rest custom out of band"
16:39:46	sean-k-mooney	TheJulia: it sound like a way to pass info on a per instance basis to the ironic python agent would partly solve there issue. e.g. if the agent had a defien api of actions such as bios and raid config that could be pass on a per instance basis it would help with the image/flavor explosion issue
16:39:58	TheJulia	woudl it help to have a larger discussion regarding this again, because where this always stalled out was the "how do we end up having something that nova will agree to" in the past in terms of data. and I think the last agreement of a possible solution involved templates stored in glance associated via traits or something which I think made it complex and cumbersome to interact with
16:40:19	sean-k-mooney	the problem is that that for that to make sense it has to be somewhat abstract as we will not know the precise host ahead of time by design
16:40:55	dansmith	we did have a long conversation about this in dublin,
16:41:03	TheJulia	sean-k-mooney: we're actually have it, just turns out people want to do even more complex things their own way
16:41:10	dansmith	which involved nova just knowing about a hardware template that gets passed to ironic to define these things
16:41:27	dansmith	there was no tableflipping
16:41:54	sean-k-mooney	dansmith: to do that we would want to do it in a way that did not invovle specifying the template in the flavor or image right
16:41:59	TheJulia	maybe I was thinking of the time before
16:42:30	sean-k-mooney	such as a swift url or glance artifcact uuid in server metadata or something else instance based
16:42:31	dansmith	sean-k-mooney: I dunno, in the image seems fine to me
16:42:44	sean-k-mooney	dansmith: in the image would work only to a point
16:42:55	openstackgerrit	Alexandre Arents proposed openstack/nova stable/stein: Add admin doc information about image cache resource accounting https://review.opendev.org/732542
16:43:07	sean-k-mooney	if you assume the raid config is an atribute of that image then sure
16:43:31	sean-k-mooney	but if the workload in that image has different performace configurtion it can be deployed in for different slas then you need multiple images
16:43:58	dansmith	sean-k-mooney: how the user gets to select it (if at all) is really a smaller concern than all the components supporting that thing regardless of how it's presented
16:44:19	sean-k-mooney	true
16:44:57	TheJulia	I suspect your thinking that fitting the needs is a conundrum, but in these sorts of cases, the user knows exactly what the hardware is because they have a special flavor to use to match it
16:45:04	dansmith	I tend to think that flavors and images are our templating mechanism and optimizing for a one-off deployment of one thing with weird config is not the direction we should go
16:45:29	dansmith	TheJulia: right, which is why I think that in the flavor is reasonable in a lot of cases
16:46:00	dansmith	they're already selecting an ironic machine class via flavor anyway
16:46:27	TheJulia	From the flip side of that, another operator on the call we had today basically stressed that if they ahd to do that their flavors would explode even more and I think they already have tens of thousands
16:47:14	sean-k-mooney	other then the fact openstack flavor list will be ... difficut to use did they have a partical problem with that
16:47:19	TheJulia	(approximately 350,000 physical machines in their fleet is what they say)
16:47:33	dansmith	I suspect that means they're doing it wrong, but regardless, if you literally have thousands of bespoke configurations, you're going to end up with thousands of incompatible objects somewhere in the system
16:47:53	sean-k-mooney	TheJulia: but presumable that is grouped into a manageable set of SKUs
16:48:26	dansmith	sean-k-mooney: that's why I said I expect they're doing it wrong.. they obviously have less than 350k unique configurations of machines, they probably just want to think of each machine as a snowflake
16:48:30	sean-k-mooney	TheJulia: do you know how many unique ironic resouce classes they have
16:49:03	TheJulia	sean-k-mooney: I'll ask, I don't know off the top of my head
16:49:26	sean-k-mooney	dansmith: right they should have 1 flavor per ironic resouce class + there vm flavors
16:50:15	dansmith	right, and not an ironic resource class for every unique serial number :)
16:50:17	TheJulia	that is not how anyone is implementing it because they are also needing to access control on the back end who has the ability to access specific resource classes
16:50:36	sean-k-mooney	TheJulia: well that is where they are doing it wrong
16:50:53	sean-k-mooney	nova has ways to do access contol on flaovr or via aggreates based on tenatns
16:51:18	TheJulia	arne_wiebalck: ^^^ fyi, you might find this interesting or super frustrating
16:51:52	sean-k-mooney	nova has the idea of private flavors so you can create 1 flaovr an choose what tenats can see it
16:52:31	TheJulia	asked the super large operator, I think cern is operating with something like like ~130-140 distinct SKUs across their fleet, some of it is private skus for hyper specific needs/purposes, others are general from what I understand. I believe they've mentioned some crazy number of flavors as well.
16:53:56	TheJulia	ugh, apparently the super large operator has lots of downstream patches wrapped around flavors/classes, so their huge numbers are not exactly valid for this discussion since we don't understand the entire story/composition there.
16:54:44	sean-k-mooney	i can see them doing thing like adding a customer specific resouce class so they can dedicate specifc pools to them
16:54:55	sean-k-mooney	but that is not how its ment to be done
16:55:07	TheJulia	sean-k-mooney: I _think_ cern has mentioned them doing private flavors before, but that it adds up for admins to dump the list out
16:55:46	mnaser	fwiw openstack cli shows public flavors only by default, even for admins
16:55:52	mnaser	you have to explicitly give it --private to get those
16:56:10	dansmith	sean-k-mooney: exactly

Earlier   Later