Posted	Nick	Remark
#openstack-nova - 2020-05-11
14:24:18	sean-k-mooney	bug: #1831723
14:24:18	openstack	bug 1831723 in OpenStack Compute (nova) "The flavor hide_hypervisor_id value can be overridden by the image img_hide_hypervisor_id" [Undecided,In progress] https://launchpad.net/bugs/1831723 - Assigned to Stephen Finucane (stephenfinucane)
14:24:50	sean-k-mooney	downstream i think we would want to backport that as im sure customer will hit it at some point
14:25:10	sean-k-mooney	but it might be nice to backport upstream too but im not sure its allowed
14:25:24	sean-k-mooney	lyarwood: ^ any toughts on the topic
14:26:07	sean-k-mooney	we are just adding a namespaced version of an existing unnamsepaced extra_spec
14:26:29	sean-k-mooney	downstream that is not conisderd an api change since extra_specs are not part of the api
14:26:35	lyarwood	sean-k-mooney: reading
14:27:34	openstackgerrit	James Page proposed openstack/nova stable/queens: hardware: fix memory check usage for small/large pages https://review.opendev.org/726867
14:28:58	sean-k-mooney	lyarwood: actully that is not the patch i ment to link
14:29:34	lyarwood	sean-k-mooney: right you had me slightly confused tbh
14:29:50	sean-k-mooney	that one we might also want to backport but one sec
14:29:58	lyarwood	yeah that's looks valid to backport
14:30:01	lyarwood	that*
14:30:04	sean-k-mooney	https://review.opendev.org/#/c/722187/
14:30:08	sean-k-mooney	that is the one i ment
14:30:27	sean-k-mooney	for https://bugs.launchpad.net/nova/+bug/1841932
14:30:27	openstack	Launchpad bug 1841932 in OpenStack Compute (nova) "hide_hypervisor_id extra_specs in nova flavor cannot pass AggregateInstanceExtraSpecsFilter" [Low,In progress] - Assigned to Stephen Finucane (stephenfinucane)
14:33:03	lyarwood	sean-k-mooney: so we deprecate hide_hypervisor_id in that change but still provide backward compatability so at first glance I think we can backport this upstream?
14:33:36	sean-k-mooney	lyarwood: correct deprecated but still supported
14:33:49	sean-k-mooney	we would have to drop the validation changes
14:33:53	sean-k-mooney	form the patch
14:34:07	sean-k-mooney	but the rest of it i think would be fine
14:34:47	lyarwood	ah right that only just landed
14:34:50	openstackgerrit	James Page proposed openstack/nova stable/queens: hardware: fix memory check usage for small/large pages https://review.opendev.org/726867
14:34:51	openstackgerrit	James Page proposed openstack/nova stable/queens: Fix overcommit for NUMA-based instances https://review.opendev.org/726868
14:35:27	sean-k-mooney	lyarwood: yep but that should be fine and easy to call out on the intiall backport patch
14:35:32	sean-k-mooney	its pretty self contained
15:01:03	jsuchome	dansmith: thanks, I'll address the changes in spec
15:12:29	gibi	lyarwood: reported a bug for the image cache issue https://bugs.launchpad.net/nova/+bug/1878024
15:12:29	openstack	Launchpad bug 1878024 in OpenStack Compute (nova) "disk usage of the nova image cache is not counted as used disk space" [Undecided,New]
15:12:58	gibi	lyarwood: could you please check if it make sense what I wrote there?
15:13:30	gibi	dansmith: you worked with the image cache recently so you might be interested ^^
15:13:37	dansmith	yeah, reading now
15:13:52	gibi	thanks
15:13:57	dansmith	if the op has a separate mount for the cache, this wouldn't be a problem
15:14:09	dansmith	so it's not going to affect everyone the sae
15:14:10	dansmith	*same
15:14:30	gibi	yeah, what the downstream customer has is a simple disk partition for the nova instances_path
15:14:41	gibi	and the cache is under the instances_path
15:14:43	dansmith	right, which I'm sure is common
15:14:57	lyarwood	gibi: ack thanks just on calls for a while but will look once I'm off
15:15:04	gibi	lyarwood: thanks
15:15:39	dansmith	the problem with something like A is that when you're scheduling,
15:15:58	dansmith	you don't know whether or not the image is already on the remote system, so you don't know whether to look for hosts with 2*$size disk space or not
15:16:39	gibi	dansmith: true
15:16:43	dansmith	B doesn't really work either because you can't assume you can purge your way out of the cache limit
15:17:02	dansmith	if you boot a hundred instances from different images, you can't prevent the image cache from going over the desired size,
15:17:19	dansmith	unless you refuse to boot instances there, which people will complain about because there is plenty of disk space and not understand
15:18:11	gibi	could nova-compute periodically update a separate DISK_GB allocation in placement based on the actual size of the image cache
15:18:17	gibi	?
15:18:52	dansmith	not periodically, but synchronously with the decision to cache an image (either during boot or otherwise)
15:18:57	dansmith	otherwise you have a race
15:20:02	dansmith	I'd have to think about that, but we'd need to only do that if the images and cache are on the same filesystem, otherwise we'd count against the wrong total
15:20:13	gibi	it will be racey anyhow due to what you said about the problem of requesting allocation for the cache during scheduling
15:21:07	dansmith	I mean racing for disk space, which could go badly if you lose, not just racing for image boot
15:21:30	dansmith	but yes, the scheduler is never going to know whether or not to pick a host based on cache availability, so you always have that
15:22:24	dansmith	we could go totally crazy and create an allocation for each image, by image uuid and after selecting a host, the scheduler could check to see if there was an allocation for that image against the host's provider to decide if it thinks it will fit :)
15:22:43	dansmith	that has some nice benefits, but it's a little crazy and there's still plenty of room for racing of course,
15:23:03	dansmith	and plenty of room for exhausting all the candidates in a small query set, leading to non-ideal looping of retries
15:23:39	dansmith	plus we'd have to have a separate provider for the cache disk if they're separate
15:27:07	gibi	complicated indeed
15:27:41	gibi	I will pass the workaround of having the cache on a different partition to downstream
15:27:56	gibi	at least that is something that the downstream project can do
15:28:24	aarents	gibi: dansmith we have this issue, painfull one
15:28:48	dansmith	gibi: ack
15:29:02	aarents	gibi yep in some case we put cahe in another file system to get ride of this
15:29:22	gibi	dansmith: the old DiskFilter had the disk_available_least info to prevent overallocation but we removed the DiskFilter
15:30:27	dansmith	yeah, fair point
15:30:56	gibi	we might want to re-introduce something like diks_available_least as a filter? or a pre-filter with placement support?
15:31:40	dansmith	I think diskfilter had plenty of other problems, like the other way where the filter behavior conflicted with the hypervisors listing, which definitely causes support cases
15:31:50	dansmith	not that they agree now, but..
15:32:18	dansmith	gibi: I don't think we'd want that to be a pre-filter because you'd have to provide either an inclusion or exclusion list of all hosts to placement each time
15:32:40	dansmith	not like a trait or aggregate, but "any one of these hosts: [... array of 5000 ...]
15:34:00	gibi	yeah I don't want to bring back the whole DiskFilter, just bring back the extra information to the scheduler / placement that how much actual disk space is free under the instance_path
15:34:30	gibi	as an idea
15:34:59	dansmith	gibi: all I'm saying is doing it as a pre-filter is the wrong place
15:35:06	gibi	ack
15:35:28	dansmith	we might still be reporting that value such that a filter can check it
15:38:17	dansmith	one other option is that the cache_images() thing that I added is setup to work as a call, returning information about presence,
15:38:44	dansmith	so if we were to pass it a "don't download just check" flag, or a "download in background" flag, then we would get back an indication if it's present or not,
15:39:04	dansmith	which would let the scheduler know whether or not to filter out hosts with 1x or 2x the disk space
15:40:00	dansmith	that's pretty heavy, and would likely need to be done from conductor,
15:40:21	dansmith	but it's a little less racy than checking some minutes-old disk free amount and assuming it's going to work
15:40:54	dansmith	another cloudy way to look at this is to say we should just have people weigh hosts on free space,
15:41:07	dansmith	in line with our "we don't schedule at capacity" project goal
15:42:35	gibi	dansmith: so if the rpc call reports that the image is not cached, then we would add 2x disk space the allocation candidate query, but only allocate 1x disk space on the selected host for the instance, then on the compute side the image cache code would allocate the other 1x disk space in placement for the cache
15:42:42	dansmith	no
15:42:48	dansmith	we've already done the a-c query at that point
15:43:02	dansmith	we'd just use that to advise us which of the a-c are valid
15:43:11	gibi	ahh yeah, you have to now which host you send the rpc call
15:43:14	dansmith	right
15:46:14	gibi	the weigher thing is good for big deployments but will fall short for edge. As far as I understand my downstream report is from a really small edge site close to capacity. :/
15:47:40	dansmith	indeed, although I think I'd argue that for highly constrained situations the separate filesystem is the right approach there anyway, given the (a) usually constrain-able image sets for edge and (b) the need to avoid the race that we'll have in some form anyway
15:47:46	dansmith	but yep, it's not a great answer for someone that just wants it to work ideally
15:48:29	gibi	I will definitely suggest the separate partition for now as I feel whathever solution we come up with (if any) it will not be backportable
15:48:38	dansmith	for sure
15:49:53	dansmith	fwiw, making a pre-call to cache_images(background=True) would theoretically give us some lower time-to-boot performance in other cases
15:50:10	dansmith	I'm really not sure whether that's a terrible idea or not, but it's an interesting thought
15:50:30	dansmith	similar to the cyborg case of starting the programming at bind time from the conductor in parallel to the rest of the instance bringup

Earlier   Later