Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-11
14:26:29 sean-k-mooney downstream that is not conisderd an api change since extra_specs are not part of the api
14:26:35 lyarwood sean-k-mooney: reading
14:27:34 openstackgerrit James Page proposed openstack/nova stable/queens: hardware: fix memory check usage for small/large pages https://review.opendev.org/726867
14:28:58 sean-k-mooney lyarwood: actully that is not the patch i ment to link
14:29:34 lyarwood sean-k-mooney: right you had me slightly confused tbh
14:29:50 sean-k-mooney that one we might also want to backport but one sec
14:29:58 lyarwood yeah that's looks valid to backport
14:30:01 lyarwood that*
14:30:04 sean-k-mooney https://review.opendev.org/#/c/722187/
14:30:08 sean-k-mooney that is the one i ment
14:30:27 sean-k-mooney for https://bugs.launchpad.net/nova/+bug/1841932
14:30:27 openstack Launchpad bug 1841932 in OpenStack Compute (nova) "hide_hypervisor_id extra_specs in nova flavor cannot pass AggregateInstanceExtraSpecsFilter" [Low,In progress] - Assigned to Stephen Finucane (stephenfinucane)
14:33:03 lyarwood sean-k-mooney: so we deprecate hide_hypervisor_id in that change but still provide backward compatability so at first glance I think we can backport this upstream?
14:33:36 sean-k-mooney lyarwood: correct deprecated but still supported
14:33:49 sean-k-mooney we would have to drop the validation changes
14:33:53 sean-k-mooney form the patch
14:34:07 sean-k-mooney but the rest of it i think would be fine
14:34:47 lyarwood ah right that only just landed
14:34:50 openstackgerrit James Page proposed openstack/nova stable/queens: hardware: fix memory check usage for small/large pages https://review.opendev.org/726867
14:34:51 openstackgerrit James Page proposed openstack/nova stable/queens: Fix overcommit for NUMA-based instances https://review.opendev.org/726868
14:35:27 sean-k-mooney lyarwood: yep but that should be fine and easy to call out on the intiall backport patch
14:35:32 sean-k-mooney its pretty self contained
15:01:03 jsuchome dansmith: thanks, I'll address the changes in spec
15:12:29 gibi lyarwood: reported a bug for the image cache issue https://bugs.launchpad.net/nova/+bug/1878024
15:12:29 openstack Launchpad bug 1878024 in OpenStack Compute (nova) "disk usage of the nova image cache is not counted as used disk space" [Undecided,New]
15:12:58 gibi lyarwood: could you please check if it make sense what I wrote there?
15:13:30 gibi dansmith: you worked with the image cache recently so you might be interested ^^
15:13:37 dansmith yeah, reading now
15:13:52 gibi thanks
15:13:57 dansmith if the op has a separate mount for the cache, this wouldn't be a problem
15:14:09 dansmith so it's not going to affect everyone the sae
15:14:10 dansmith *same
15:14:30 gibi yeah, what the downstream customer has is a simple disk partition for the nova instances_path
15:14:41 gibi and the cache is under the instances_path
15:14:43 dansmith right, which I'm sure is common
15:14:57 lyarwood gibi: ack thanks just on calls for a while but will look once I'm off
15:15:04 gibi lyarwood: thanks
15:15:39 dansmith the problem with something like A is that when you're scheduling,
15:15:58 dansmith you don't know whether or not the image is already on the remote system, so you don't know whether to look for hosts with 2*$size disk space or not
15:16:39 gibi dansmith: true
15:16:43 dansmith B doesn't really work either because you can't assume you can purge your way out of the cache limit
15:17:02 dansmith if you boot a hundred instances from different images, you can't prevent the image cache from going over the desired size,
15:17:19 dansmith unless you refuse to boot instances there, which people will complain about because there is plenty of disk space and not understand
15:18:11 gibi could nova-compute periodically update a separate DISK_GB allocation in placement based on the actual size of the image cache
15:18:17 gibi ?
15:18:52 dansmith not periodically, but synchronously with the decision to cache an image (either during boot or otherwise)
15:18:57 dansmith otherwise you have a race
15:20:02 dansmith I'd have to think about that, but we'd need to only do that if the images and cache are on the same filesystem, otherwise we'd count against the wrong total
15:20:13 gibi it will be racey anyhow due to what you said about the problem of requesting allocation for the cache during scheduling
15:21:07 dansmith I mean racing for disk space, which could go badly if you lose, not just racing for image boot
15:21:30 dansmith but yes, the scheduler is never going to know whether or not to pick a host based on cache availability, so you always have that
15:22:24 dansmith we could go totally crazy and create an allocation for each image, by image uuid and after selecting a host, the scheduler could check to see if there was an allocation for that image against the host's provider to decide if it thinks it will fit :)
15:22:43 dansmith that has some nice benefits, but it's a little crazy and there's still plenty of room for racing of course,
15:23:03 dansmith and plenty of room for exhausting all the candidates in a small query set, leading to non-ideal looping of retries
15:23:39 dansmith plus we'd have to have a separate provider for the cache disk if they're separate
15:27:07 gibi complicated indeed
15:27:41 gibi I will pass the workaround of having the cache on a different partition to downstream
15:27:56 gibi at least that is something that the downstream project can do
15:28:24 aarents gibi: dansmith we have this issue, painfull one
15:28:48 dansmith gibi: ack
15:29:02 aarents gibi yep in some case we put cahe in another file system to get ride of this
15:29:22 gibi dansmith: the old DiskFilter had the disk_available_least info to prevent overallocation but we removed the DiskFilter
15:30:27 dansmith yeah, fair point
15:30:56 gibi we might want to re-introduce something like diks_available_least as a filter? or a pre-filter with placement support?
15:31:40 dansmith I think diskfilter had plenty of other problems, like the other way where the filter behavior conflicted with the hypervisors listing, which definitely causes support cases
15:31:50 dansmith not that they agree now, but..
15:32:18 dansmith gibi: I don't think we'd want that to be a pre-filter because you'd have to provide either an inclusion or exclusion list of all hosts to placement each time
15:32:40 dansmith not like a trait or aggregate, but "any one of these hosts: [... array of 5000 ...]
15:34:00 gibi yeah I don't want to bring back the whole DiskFilter, just bring back the extra information to the scheduler / placement that how much actual disk space is free under the instance_path
15:34:30 gibi as an idea
15:34:59 dansmith gibi: all I'm saying is doing it as a pre-filter is the wrong place
15:35:06 gibi ack
15:35:28 dansmith we might still be reporting that value such that a filter can check it
15:38:17 dansmith one other option is that the cache_images() thing that I added is setup to work as a call, returning information about presence,
15:38:44 dansmith so if we were to pass it a "don't download just check" flag, or a "download in background" flag, then we would get back an indication if it's present or not,
15:39:04 dansmith which would let the scheduler know whether or not to filter out hosts with 1x or 2x the disk space
15:40:00 dansmith that's pretty heavy, and would likely need to be done from conductor,
15:40:21 dansmith but it's a little less racy than checking some minutes-old disk free amount and assuming it's going to work
15:40:54 dansmith another cloudy way to look at this is to say we should just have people weigh hosts on free space,
15:41:07 dansmith in line with our "we don't schedule at capacity" project goal
15:42:35 gibi dansmith: so if the rpc call reports that the image is not cached, then we would add 2x disk space the allocation candidate query, but only allocate 1x disk space on the selected host for the instance, then on the compute side the image cache code would allocate the other 1x disk space in placement for the cache
15:42:42 dansmith no
15:42:48 dansmith we've already done the a-c query at that point
15:43:02 dansmith we'd just use that to advise us which of the a-c are valid
15:43:11 gibi ahh yeah, you have to now which host you send the rpc call
15:43:14 dansmith right
15:46:14 gibi the weigher thing is good for big deployments but will fall short for edge. As far as I understand my downstream report is from a really small edge site close to capacity. :/
15:47:40 dansmith indeed, although I think I'd argue that for highly constrained situations the separate filesystem is the right approach there anyway, given the (a) usually constrain-able image sets for edge and (b) the need to avoid the race that we'll have in some form anyway
15:47:46 dansmith but yep, it's not a great answer for someone that just wants it to work ideally
15:48:29 gibi I will definitely suggest the separate partition for now as I feel whathever solution we come up with (if any) it will not be backportable
15:48:38 dansmith for sure
15:49:53 dansmith fwiw, making a pre-call to cache_images(background=True) would theoretically give us some lower time-to-boot performance in other cases
15:50:10 dansmith I'm really not sure whether that's a terrible idea or not, but it's an interesting thought
15:50:30 dansmith similar to the cyborg case of starting the programming at bind time from the conductor in parallel to the rest of the instance bringup
15:50:36 gibi it would have a side effect to cache image to a compute that otherwise will not be selected
15:51:18 gibi hm or not
15:51:31 dansmith in the tight case you mean right? We'd start caching an image on a host that the scheduler was going to exclude anyway, which is true
15:52:11 gibi yeah for the thight case when the image would fit into the cache but the instance root disk would not any more
15:52:16 dansmith yep, for sure
15:52:24 gibi but that really tight

Earlier   Later