Posted	Nick	Remark
#openstack-nova - 2020-05-11
14:33:53	sean-k-mooney	form the patch
14:34:07	sean-k-mooney	but the rest of it i think would be fine
14:34:47	lyarwood	ah right that only just landed
14:34:50	openstackgerrit	James Page proposed openstack/nova stable/queens: hardware: fix memory check usage for small/large pages https://review.opendev.org/726867
14:34:51	openstackgerrit	James Page proposed openstack/nova stable/queens: Fix overcommit for NUMA-based instances https://review.opendev.org/726868
14:35:27	sean-k-mooney	lyarwood: yep but that should be fine and easy to call out on the intiall backport patch
14:35:32	sean-k-mooney	its pretty self contained
15:01:03	jsuchome	dansmith: thanks, I'll address the changes in spec
15:12:29	gibi	lyarwood: reported a bug for the image cache issue https://bugs.launchpad.net/nova/+bug/1878024
15:12:29	openstack	Launchpad bug 1878024 in OpenStack Compute (nova) "disk usage of the nova image cache is not counted as used disk space" [Undecided,New]
15:12:58	gibi	lyarwood: could you please check if it make sense what I wrote there?
15:13:30	gibi	dansmith: you worked with the image cache recently so you might be interested ^^
15:13:37	dansmith	yeah, reading now
15:13:52	gibi	thanks
15:13:57	dansmith	if the op has a separate mount for the cache, this wouldn't be a problem
15:14:09	dansmith	so it's not going to affect everyone the sae
15:14:10	dansmith	*same
15:14:30	gibi	yeah, what the downstream customer has is a simple disk partition for the nova instances_path
15:14:41	gibi	and the cache is under the instances_path
15:14:43	dansmith	right, which I'm sure is common
15:14:57	lyarwood	gibi: ack thanks just on calls for a while but will look once I'm off
15:15:04	gibi	lyarwood: thanks
15:15:39	dansmith	the problem with something like A is that when you're scheduling,
15:15:58	dansmith	you don't know whether or not the image is already on the remote system, so you don't know whether to look for hosts with 2*$size disk space or not
15:16:39	gibi	dansmith: true
15:16:43	dansmith	B doesn't really work either because you can't assume you can purge your way out of the cache limit
15:17:02	dansmith	if you boot a hundred instances from different images, you can't prevent the image cache from going over the desired size,
15:17:19	dansmith	unless you refuse to boot instances there, which people will complain about because there is plenty of disk space and not understand
15:18:11	gibi	could nova-compute periodically update a separate DISK_GB allocation in placement based on the actual size of the image cache
15:18:17	gibi	?
15:18:52	dansmith	not periodically, but synchronously with the decision to cache an image (either during boot or otherwise)
15:18:57	dansmith	otherwise you have a race
15:20:02	dansmith	I'd have to think about that, but we'd need to only do that if the images and cache are on the same filesystem, otherwise we'd count against the wrong total
15:20:13	gibi	it will be racey anyhow due to what you said about the problem of requesting allocation for the cache during scheduling
15:21:07	dansmith	I mean racing for disk space, which could go badly if you lose, not just racing for image boot
15:21:30	dansmith	but yes, the scheduler is never going to know whether or not to pick a host based on cache availability, so you always have that
15:22:24	dansmith	we could go totally crazy and create an allocation for each image, by image uuid and after selecting a host, the scheduler could check to see if there was an allocation for that image against the host's provider to decide if it thinks it will fit :)
15:22:43	dansmith	that has some nice benefits, but it's a little crazy and there's still plenty of room for racing of course,
15:23:03	dansmith	and plenty of room for exhausting all the candidates in a small query set, leading to non-ideal looping of retries
15:23:39	dansmith	plus we'd have to have a separate provider for the cache disk if they're separate
15:27:07	gibi	complicated indeed
15:27:41	gibi	I will pass the workaround of having the cache on a different partition to downstream
15:27:56	gibi	at least that is something that the downstream project can do
15:28:24	aarents	gibi: dansmith we have this issue, painfull one
15:28:48	dansmith	gibi: ack
15:29:02	aarents	gibi yep in some case we put cahe in another file system to get ride of this
15:29:22	gibi	dansmith: the old DiskFilter had the disk_available_least info to prevent overallocation but we removed the DiskFilter
15:30:27	dansmith	yeah, fair point
15:30:56	gibi	we might want to re-introduce something like diks_available_least as a filter? or a pre-filter with placement support?
15:31:40	dansmith	I think diskfilter had plenty of other problems, like the other way where the filter behavior conflicted with the hypervisors listing, which definitely causes support cases
15:31:50	dansmith	not that they agree now, but..
15:32:18	dansmith	gibi: I don't think we'd want that to be a pre-filter because you'd have to provide either an inclusion or exclusion list of all hosts to placement each time
15:32:40	dansmith	not like a trait or aggregate, but "any one of these hosts: [... array of 5000 ...]
15:34:00	gibi	yeah I don't want to bring back the whole DiskFilter, just bring back the extra information to the scheduler / placement that how much actual disk space is free under the instance_path
15:34:30	gibi	as an idea
15:34:59	dansmith	gibi: all I'm saying is doing it as a pre-filter is the wrong place
15:35:06	gibi	ack
15:35:28	dansmith	we might still be reporting that value such that a filter can check it
15:38:17	dansmith	one other option is that the cache_images() thing that I added is setup to work as a call, returning information about presence,
15:38:44	dansmith	so if we were to pass it a "don't download just check" flag, or a "download in background" flag, then we would get back an indication if it's present or not,
15:39:04	dansmith	which would let the scheduler know whether or not to filter out hosts with 1x or 2x the disk space
15:40:00	dansmith	that's pretty heavy, and would likely need to be done from conductor,
15:40:21	dansmith	but it's a little less racy than checking some minutes-old disk free amount and assuming it's going to work
15:40:54	dansmith	another cloudy way to look at this is to say we should just have people weigh hosts on free space,
15:41:07	dansmith	in line with our "we don't schedule at capacity" project goal
15:42:35	gibi	dansmith: so if the rpc call reports that the image is not cached, then we would add 2x disk space the allocation candidate query, but only allocate 1x disk space on the selected host for the instance, then on the compute side the image cache code would allocate the other 1x disk space in placement for the cache
15:42:42	dansmith	no
15:42:48	dansmith	we've already done the a-c query at that point
15:43:02	dansmith	we'd just use that to advise us which of the a-c are valid
15:43:11	gibi	ahh yeah, you have to now which host you send the rpc call
15:43:14	dansmith	right
15:46:14	gibi	the weigher thing is good for big deployments but will fall short for edge. As far as I understand my downstream report is from a really small edge site close to capacity. :/
15:47:40	dansmith	indeed, although I think I'd argue that for highly constrained situations the separate filesystem is the right approach there anyway, given the (a) usually constrain-able image sets for edge and (b) the need to avoid the race that we'll have in some form anyway
15:47:46	dansmith	but yep, it's not a great answer for someone that just wants it to work ideally
15:48:29	gibi	I will definitely suggest the separate partition for now as I feel whathever solution we come up with (if any) it will not be backportable
15:48:38	dansmith	for sure
15:49:53	dansmith	fwiw, making a pre-call to cache_images(background=True) would theoretically give us some lower time-to-boot performance in other cases
15:50:10	dansmith	I'm really not sure whether that's a terrible idea or not, but it's an interesting thought
15:50:30	dansmith	similar to the cyborg case of starting the programming at bind time from the conductor in parallel to the rest of the instance bringup
15:50:36	gibi	it would have a side effect to cache image to a compute that otherwise will not be selected
15:51:18	gibi	hm or not
15:51:31	dansmith	in the tight case you mean right? We'd start caching an image on a host that the scheduler was going to exclude anyway, which is true
15:52:11	gibi	yeah for the thight case when the image would fit into the cache but the instance root disk would not any more
15:52:16	dansmith	yep, for sure
15:52:24	gibi	but that really tight
15:53:54	gibi	what if for thight cases we allow disabling the cache entirely? it is thight so no space for cache
15:54:30	dansmith	doing that would require a substantial resdesign of the whole image backend for libvirt I think
15:55:03	gibi	is it because we assume that there is a backing file for the root fs image which happen to be the cached image?
15:56:41	dansmith	I'm actually not sure what happens if you configure qcow2 and "flatten_images" actually, I'd have to look
15:56:50	dansmith	that might have the same effect, I'm not sure
15:59:21	gibi	do you mean force_raw_images conf option?
16:00:26	gibi	or use_cow_images = False? or a linear combination of the two :)
16:00:44	markguz_	Hi nova folks. i have an instance that failed that i migrated (not live) to another compute host, but upon restarting it errors with
16:00:45	markguz_	Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed'
16:00:58	dansmith	gibi: yeah there's some combination that results in full flattening, but I'm not sure what they are
16:01:10	dansmith	gibi: not sure if tat actually results in the cache image going away, or getting copied or what
16:01:24	dansmith	actually, as to be the latter I think since we have to expand the size of it
16:01:26	markguz_	i've tried the various solutions found via google but non work
16:01:46	dansmith	so you probably end up with 2x the space initially at least, and then you'd need to immediately purge the original or something
16:02:22	gibi	dansmith: ack

Earlier   Later