| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 14:03:13 | mriedem | we calculate the overhead for the vm | |
| 14:04:05 | mriedem | and the claim.memory_mb is based on the instance.flavor.memory_mb + that overhead value from the compute driver | |
| 14:04:11 | bauzas | right | |
| 14:05:13 | mriedem | what i'm trying to figure out, | |
| 14:05:30 | mriedem | is if this overhead estimate from the virt driver factors into scheduling decisions in the scheduler anywhere | |
| 14:05:49 | mriedem | or is it something we can only determine once we're on a host, find out that with the overhead we are over the limit, and fail the claim test | |
| 14:05:51 | mriedem | and trigger a retry | |
| 14:06:25 | bauzas | mriedem: the RamFilter doesn't get that for a specific virt driver, so I'd assume no | |
| 14:06:53 | mriedem | right. but do we record it in the allocation record? | |
| 14:06:55 | mriedem | in placement | |
| 14:07:05 | bauzas | the overhead ? | |
| 14:07:20 | mriedem | e.g. if the full allocation for MEMORY_MB for a given instance is instance.flavor.memory_mb + overhead from virt driver, do we put that into the allocation? | |
| 14:07:26 | mriedem | or just instance.flavor.memory_mb? | |
| 14:07:49 | mriedem | the overhead value that comes out of virt.estimate_instance_overhead | |
| 14:07:54 | bauzas | well, all happens in _update_usage_from_instance so looking | |
| 14:08:34 | bauzas | mriedem: looks like it's not | |
| 14:08:40 | jaypipes | mriedem: yes, it's the inventory for MEMORY_MB :) | |
| 14:09:24 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:09:41 | mriedem | aha | |
| 14:09:47 | mriedem | then yeah that seems like a bug | |
| 14:09:59 | jaypipes | mriedem: what? the overhead? | |
| 14:10:03 | mriedem | yes | |
| 14:10:11 | mriedem | because https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L106 | |
| 14:10:35 | mriedem | which is tested in the claim here https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L168 | |
| 14:10:37 | mriedem | right? | |
| 14:10:41 | bauzas | jaypipes: the overhead is only calculated when calling the claim AFAICS | |
| 14:11:09 | mriedem | requested = self.memory_mb == instance.flavor.memory_mb + overhead['memory_mb'] | |
| 14:11:46 | jaypipes | mriedem: meh, maybe. it's true there's a little bit of possible inconsistency. I was hoping to resolve that with claims in scheduler. but we need to know the virt driver of the target host *before* we actually allocate :( which is tarded. | |
| 14:11:58 | bauzas | mriedem: well, seems like it's self-healed actually | |
| 14:12:02 | mriedem | jaypipes: right, so we have this whole chicken and egg | |
| 14:12:23 | bauzas | mriedem: because RT._update_usage() does calculate the overhead past the allocation | |
| 14:13:00 | bauzas | mriedem: and given we call _update() after _update_usage() with the resources that are taking in account that overhead, it auto-heals | |
| 14:13:01 | jaypipes | mriedem: instead of doing the chicken egg thing, I think it's better to instead use the inventory record's reserved value. that may be a coarser-grained solution, but it's certainly cleaner. | |
| 14:13:02 | johnthetubaguy | mriedem: probably not a shock, but alot of that memory crazy is Xen related, or at least very related to when you have "hard" memory limits | |
| 14:13:10 | cdent | shouldn't overhead (as a sum of potentials) be recorded within reserved and _not_ be in the consumer's allocations? | |
| 14:13:38 | mriedem | bauzas: i'm not following the full call chain yet | |
| 14:13:40 | jaypipes | cdent: it's overhead *per instance*, whereas reserved is per provider/resource cllass. | |
| 14:13:51 | mriedem | bauzas: but you agree we can hit ComputeResourcesUnavailable due to the overhead pushing us over right? | |
| 14:13:51 | bauzas | what jaypipes said | |
| 14:13:52 | jaypipes | cdent: thus reserved is "more coarse-grained" | |
| 14:13:52 | cdent | jaypipes: I know, thus the "sum of potentials" | |
| 14:13:59 | jaypipes | cdent: right. | |
| 14:13:59 | cdent | and thus the "jinx" | |
| 14:14:03 | jaypipes | :) | |
| 14:14:11 | bauzas | mriedem: sec, explaining | |
| 14:14:14 | johnthetubaguy | jaypipes: cdent: so the memory overhead is VM specific in some cases (ducks) but I would argue we might want to ignore that | |
| 14:14:26 | mriedem | johnthetubaguy: exactly | |
| 14:14:30 | jaypipes | johnthetubaguy: ++ | |
| 14:14:34 | mriedem | the overhead calculation is totally based on the flavor used | |
| 14:14:36 | bauzas | mriedem: so, the RamFilter is calculating against the free ram based on what the RT reports, right? | |
| 14:15:07 | bauzas | mriedem: what I'm saying is that I just saw that we do calculate the overhead when updating the usage in the RT | |
| 14:15:12 | mriedem | so do you then make a generic reserved value for the RT on that provider for what you expect to land on that host flavor-wise? | |
| 14:15:32 | mriedem | bauzas: i thought we wanted to deprecate/remove the RamFilter? | |
| 14:15:34 | mriedem | because we use placement now | |
| 14:15:38 | johnthetubaguy | mriedem: yeah, I got the impression that its party host CPU model specific, but I never got a straight answer about that | |
| 14:16:42 | bauzas | mriedem: well, I'm trying to figure out the situation in a pre-placement world | |
| 14:16:43 | mriedem | ok, so the reason i'm even looking at this, is because someone internal is looking at this for the libvirt driver and with an example vm they calculated ~226MB of overhead | |
| 14:16:50 | mriedem | which isn't being taken into account for scheduling | |
| 14:16:50 | jaypipes | frankly, mriedem and johnthetubaguy, we're going to end up with a similar issue with the emulator threads. | |
| 14:17:02 | johnthetubaguy | jaypipes: oh, true | |
| 14:17:06 | jaypipes | mriedem, johnthetubaguy: because the reserved amount of those is specific to a virt driver./ | |
| 14:17:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 14:17:22 | jaypipes | course, we need nested providers first, but still, thinking ahead... | |
| 14:17:39 | mriedem | can we agree that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should be taking into account the overhead? | |
| 14:18:19 | dansmith | I haven't been paying attention, | |
| 14:18:26 | bauzas | I'm still looking at the 100+ calls involved in the decision-making :) | |
| 14:18:28 | dansmith | but the overhead should be included in the inventory reserved amount no? | |
| 14:18:34 | jaypipes | mriedem: perhaps, though that may lead to a failure to allocate. | |
| 14:18:38 | bauzas | dansmith: it's per instance | |
| 14:18:48 | jaypipes | dansmith: yeah, per instance... | |
| 14:18:52 | johnthetubaguy | so there are a few overheads going around here | |
| 14:19:01 | dansmith | hmm, I must be missing that part | |
| 14:19:03 | bauzas | the reserved bit is for a global hypervisor overhead | |
| 14:19:22 | bauzas | like a memory heap size you wanna not touch | |
| 14:19:35 | mriedem | jaypipes: hmm, but what we claim would actually be more than what we report for the actual allocation, agreed? | |
| 14:19:36 | dansmith | having allocations include per-instance overhead seems confusing to me, unless it's called out specifically | |
| 14:19:37 | mriedem | or could be | |
| 14:19:51 | dansmith | otherwise it will not be super helpful for use by other tools | |
| 14:19:52 | jaypipes | mriedem: right, which is partly why we didn't include it in that code. | |
| 14:20:14 | mriedem | jaypipes: but shouldn't be as accurate as possible in the allocation since that's going to be used by the scheduler? | |
| 14:20:37 | jaypipes | mriedem: I guess, but the scheduler doesn't ask for that amount :) | |
| 14:21:03 | dansmith | ah, estimate_instance_overhead | |
| 14:21:06 | mriedem | i'm going to need a refresher on what the scheduler asks for in placement, is it just based on inventory? | |
| 14:21:11 | dansmith | jaypipes: nor can it, right? it comes from the virt driver | |
| 14:21:11 | bauzas | yep, that's what I was trying to explain | |
| 14:21:18 | jaypipes | mriedem: in other words, the scheduler doesn't say "I want X MEMORY_MB plus some overhead if the instance is on a Xen hypervisor" | |
| 14:21:22 | mriedem | dansmith: right claims in the scheduler can't use this | |
| 14:21:22 | bauzas | there are two things to consider | |
| 14:21:24 | jaypipes | dansmith: right. | |
| 14:21:31 | bauzas | 1/ what you request and 2/ what you have as free | |
| 14:21:41 | mriedem | jaypipes: i realize, that's not what i'm saying though | |
| 14:21:50 | bauzas | in the claim, we add more to request with the overhead but we don't touch 2/ | |
| 14:22:07 | mriedem | do we or don't we look at allocations when figuring out which providers can serve a request in the scheduler? | |
| 14:22:10 | jaypipes | mriedem: I think the long-term solution to this is the following: | |
| 14:22:10 | bauzas | while we do later in the self-heal process, do decrease 2/ | |
| 14:22:35 | bauzas | so, when it comes to placement, the free space is already decremented by the sum of overheads | |
| 14:22:39 | jaypipes | mriedem: add an overheads table to placement API, containing resource_provider_id, resource_class_id, and amount. | |
| 14:22:47 | bauzas | (or when it comes to RAMfilter) | |
| 14:22:57 | jaypipes | mriedem: each record in that table would contain the overhead for that resource class on that provider for each consumer of that resource class. | |
| 14:23:22 | dansmith | jaypipes: ooh, yeah I like that | |