| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 14:06:53 | mriedem | right. but do we record it in the allocation record? | |
| 14:06:55 | mriedem | in placement | |
| 14:07:05 | bauzas | the overhead ? | |
| 14:07:20 | mriedem | e.g. if the full allocation for MEMORY_MB for a given instance is instance.flavor.memory_mb + overhead from virt driver, do we put that into the allocation? | |
| 14:07:26 | mriedem | or just instance.flavor.memory_mb? | |
| 14:07:49 | mriedem | the overhead value that comes out of virt.estimate_instance_overhead | |
| 14:07:54 | bauzas | well, all happens in _update_usage_from_instance so looking | |
| 14:08:34 | bauzas | mriedem: looks like it's not | |
| 14:08:40 | jaypipes | mriedem: yes, it's the inventory for MEMORY_MB :) | |
| 14:09:24 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:09:41 | mriedem | aha | |
| 14:09:47 | mriedem | then yeah that seems like a bug | |
| 14:09:59 | jaypipes | mriedem: what? the overhead? | |
| 14:10:03 | mriedem | yes | |
| 14:10:11 | mriedem | because https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L106 | |
| 14:10:35 | mriedem | which is tested in the claim here https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L168 | |
| 14:10:37 | mriedem | right? | |
| 14:10:41 | bauzas | jaypipes: the overhead is only calculated when calling the claim AFAICS | |
| 14:11:09 | mriedem | requested = self.memory_mb == instance.flavor.memory_mb + overhead['memory_mb'] | |
| 14:11:46 | jaypipes | mriedem: meh, maybe. it's true there's a little bit of possible inconsistency. I was hoping to resolve that with claims in scheduler. but we need to know the virt driver of the target host *before* we actually allocate :( which is tarded. | |
| 14:11:58 | bauzas | mriedem: well, seems like it's self-healed actually | |
| 14:12:02 | mriedem | jaypipes: right, so we have this whole chicken and egg | |
| 14:12:23 | bauzas | mriedem: because RT._update_usage() does calculate the overhead past the allocation | |
| 14:13:00 | bauzas | mriedem: and given we call _update() after _update_usage() with the resources that are taking in account that overhead, it auto-heals | |
| 14:13:01 | jaypipes | mriedem: instead of doing the chicken egg thing, I think it's better to instead use the inventory record's reserved value. that may be a coarser-grained solution, but it's certainly cleaner. | |
| 14:13:02 | johnthetubaguy | mriedem: probably not a shock, but alot of that memory crazy is Xen related, or at least very related to when you have "hard" memory limits | |
| 14:13:10 | cdent | shouldn't overhead (as a sum of potentials) be recorded within reserved and _not_ be in the consumer's allocations? | |
| 14:13:38 | mriedem | bauzas: i'm not following the full call chain yet | |
| 14:13:40 | jaypipes | cdent: it's overhead *per instance*, whereas reserved is per provider/resource cllass. | |
| 14:13:51 | mriedem | bauzas: but you agree we can hit ComputeResourcesUnavailable due to the overhead pushing us over right? | |
| 14:13:51 | bauzas | what jaypipes said | |
| 14:13:52 | jaypipes | cdent: thus reserved is "more coarse-grained" | |
| 14:13:52 | cdent | jaypipes: I know, thus the "sum of potentials" | |
| 14:13:59 | jaypipes | cdent: right. | |
| 14:13:59 | cdent | and thus the "jinx" | |
| 14:14:03 | jaypipes | :) | |
| 14:14:11 | bauzas | mriedem: sec, explaining | |
| 14:14:14 | johnthetubaguy | jaypipes: cdent: so the memory overhead is VM specific in some cases (ducks) but I would argue we might want to ignore that | |
| 14:14:26 | mriedem | johnthetubaguy: exactly | |
| 14:14:30 | jaypipes | johnthetubaguy: ++ | |
| 14:14:34 | mriedem | the overhead calculation is totally based on the flavor used | |
| 14:14:36 | bauzas | mriedem: so, the RamFilter is calculating against the free ram based on what the RT reports, right? | |
| 14:15:07 | bauzas | mriedem: what I'm saying is that I just saw that we do calculate the overhead when updating the usage in the RT | |
| 14:15:12 | mriedem | so do you then make a generic reserved value for the RT on that provider for what you expect to land on that host flavor-wise? | |
| 14:15:32 | mriedem | bauzas: i thought we wanted to deprecate/remove the RamFilter? | |
| 14:15:34 | mriedem | because we use placement now | |
| 14:15:38 | johnthetubaguy | mriedem: yeah, I got the impression that its party host CPU model specific, but I never got a straight answer about that | |
| 14:16:42 | bauzas | mriedem: well, I'm trying to figure out the situation in a pre-placement world | |
| 14:16:43 | mriedem | ok, so the reason i'm even looking at this, is because someone internal is looking at this for the libvirt driver and with an example vm they calculated ~226MB of overhead | |
| 14:16:50 | mriedem | which isn't being taken into account for scheduling | |
| 14:16:50 | jaypipes | frankly, mriedem and johnthetubaguy, we're going to end up with a similar issue with the emulator threads. | |
| 14:17:02 | johnthetubaguy | jaypipes: oh, true | |
| 14:17:06 | jaypipes | mriedem, johnthetubaguy: because the reserved amount of those is specific to a virt driver./ | |
| 14:17:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 14:17:22 | jaypipes | course, we need nested providers first, but still, thinking ahead... | |
| 14:17:39 | mriedem | can we agree that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should be taking into account the overhead? | |
| 14:18:19 | dansmith | I haven't been paying attention, | |
| 14:18:26 | bauzas | I'm still looking at the 100+ calls involved in the decision-making :) | |
| 14:18:28 | dansmith | but the overhead should be included in the inventory reserved amount no? | |
| 14:18:34 | jaypipes | mriedem: perhaps, though that may lead to a failure to allocate. | |
| 14:18:38 | bauzas | dansmith: it's per instance | |
| 14:18:48 | jaypipes | dansmith: yeah, per instance... | |
| 14:18:52 | johnthetubaguy | so there are a few overheads going around here | |
| 14:19:01 | dansmith | hmm, I must be missing that part | |
| 14:19:03 | bauzas | the reserved bit is for a global hypervisor overhead | |
| 14:19:22 | bauzas | like a memory heap size you wanna not touch | |
| 14:19:35 | mriedem | jaypipes: hmm, but what we claim would actually be more than what we report for the actual allocation, agreed? | |
| 14:19:36 | dansmith | having allocations include per-instance overhead seems confusing to me, unless it's called out specifically | |
| 14:19:37 | mriedem | or could be | |
| 14:19:51 | dansmith | otherwise it will not be super helpful for use by other tools | |
| 14:19:52 | jaypipes | mriedem: right, which is partly why we didn't include it in that code. | |
| 14:20:14 | mriedem | jaypipes: but shouldn't be as accurate as possible in the allocation since that's going to be used by the scheduler? | |
| 14:20:37 | jaypipes | mriedem: I guess, but the scheduler doesn't ask for that amount :) | |
| 14:21:03 | dansmith | ah, estimate_instance_overhead | |
| 14:21:06 | mriedem | i'm going to need a refresher on what the scheduler asks for in placement, is it just based on inventory? | |
| 14:21:11 | dansmith | jaypipes: nor can it, right? it comes from the virt driver | |
| 14:21:11 | bauzas | yep, that's what I was trying to explain | |
| 14:21:18 | jaypipes | mriedem: in other words, the scheduler doesn't say "I want X MEMORY_MB plus some overhead if the instance is on a Xen hypervisor" | |
| 14:21:22 | mriedem | dansmith: right claims in the scheduler can't use this | |
| 14:21:22 | bauzas | there are two things to consider | |
| 14:21:24 | jaypipes | dansmith: right. | |
| 14:21:31 | bauzas | 1/ what you request and 2/ what you have as free | |
| 14:21:41 | mriedem | jaypipes: i realize, that's not what i'm saying though | |
| 14:21:50 | bauzas | in the claim, we add more to request with the overhead but we don't touch 2/ | |
| 14:22:07 | mriedem | do we or don't we look at allocations when figuring out which providers can serve a request in the scheduler? | |
| 14:22:10 | jaypipes | mriedem: I think the long-term solution to this is the following: | |
| 14:22:10 | bauzas | while we do later in the self-heal process, do decrease 2/ | |
| 14:22:35 | bauzas | so, when it comes to placement, the free space is already decremented by the sum of overheads | |
| 14:22:39 | jaypipes | mriedem: add an overheads table to placement API, containing resource_provider_id, resource_class_id, and amount. | |
| 14:22:47 | bauzas | (or when it comes to RAMfilter) | |
| 14:22:57 | jaypipes | mriedem: each record in that table would contain the overhead for that resource class on that provider for each consumer of that resource class. | |
| 14:23:22 | dansmith | jaypipes: ooh, yeah I like that | |
| 14:23:24 | jaypipes | mriedem: that way, we can properly model both the inventory and allocation request in the placement API. | |
| 14:23:30 | dansmith | jaypipes: make the virt driver delcare it ahead of time | |
| 14:23:41 | jaypipes | dansmith: right, in the RT's init_compute_node(), etc | |
| 14:23:42 | mriedem | but, | |
| 14:23:53 | BobBall | jaypipes: My understanding is the overhead on Xen for RAM is smallish. Yes, there will be some instances where we are running very tight against total RAM that need rescheduling, but I'm not sure it's worth the code changes to fix it? | |
| 14:23:53 | mriedem | the overhead is a factor of the flavor | |
| 14:23:53 | dansmith | jaypipes: which kinda pushes us more to a model where we speak about everything in terms of resourceclasses | |
| 14:24:08 | dansmith | mriedem: right, but make it declare how it calculates it in terms of resource classes | |