| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 14:13:40 | jaypipes | cdent: it's overhead *per instance*, whereas reserved is per provider/resource cllass. | |
| 14:13:51 | mriedem | bauzas: but you agree we can hit ComputeResourcesUnavailable due to the overhead pushing us over right? | |
| 14:13:51 | bauzas | what jaypipes said | |
| 14:13:52 | jaypipes | cdent: thus reserved is "more coarse-grained" | |
| 14:13:52 | cdent | jaypipes: I know, thus the "sum of potentials" | |
| 14:13:59 | jaypipes | cdent: right. | |
| 14:13:59 | cdent | and thus the "jinx" | |
| 14:14:03 | jaypipes | :) | |
| 14:14:11 | bauzas | mriedem: sec, explaining | |
| 14:14:14 | johnthetubaguy | jaypipes: cdent: so the memory overhead is VM specific in some cases (ducks) but I would argue we might want to ignore that | |
| 14:14:26 | mriedem | johnthetubaguy: exactly | |
| 14:14:30 | jaypipes | johnthetubaguy: ++ | |
| 14:14:34 | mriedem | the overhead calculation is totally based on the flavor used | |
| 14:14:36 | bauzas | mriedem: so, the RamFilter is calculating against the free ram based on what the RT reports, right? | |
| 14:15:07 | bauzas | mriedem: what I'm saying is that I just saw that we do calculate the overhead when updating the usage in the RT | |
| 14:15:12 | mriedem | so do you then make a generic reserved value for the RT on that provider for what you expect to land on that host flavor-wise? | |
| 14:15:32 | mriedem | bauzas: i thought we wanted to deprecate/remove the RamFilter? | |
| 14:15:34 | mriedem | because we use placement now | |
| 14:15:38 | johnthetubaguy | mriedem: yeah, I got the impression that its party host CPU model specific, but I never got a straight answer about that | |
| 14:16:42 | bauzas | mriedem: well, I'm trying to figure out the situation in a pre-placement world | |
| 14:16:43 | mriedem | ok, so the reason i'm even looking at this, is because someone internal is looking at this for the libvirt driver and with an example vm they calculated ~226MB of overhead | |
| 14:16:50 | mriedem | which isn't being taken into account for scheduling | |
| 14:16:50 | jaypipes | frankly, mriedem and johnthetubaguy, we're going to end up with a similar issue with the emulator threads. | |
| 14:17:02 | johnthetubaguy | jaypipes: oh, true | |
| 14:17:06 | jaypipes | mriedem, johnthetubaguy: because the reserved amount of those is specific to a virt driver./ | |
| 14:17:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 14:17:22 | jaypipes | course, we need nested providers first, but still, thinking ahead... | |
| 14:17:39 | mriedem | can we agree that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should be taking into account the overhead? | |
| 14:18:19 | dansmith | I haven't been paying attention, | |
| 14:18:26 | bauzas | I'm still looking at the 100+ calls involved in the decision-making :) | |
| 14:18:28 | dansmith | but the overhead should be included in the inventory reserved amount no? | |
| 14:18:34 | jaypipes | mriedem: perhaps, though that may lead to a failure to allocate. | |
| 14:18:38 | bauzas | dansmith: it's per instance | |
| 14:18:48 | jaypipes | dansmith: yeah, per instance... | |
| 14:18:52 | johnthetubaguy | so there are a few overheads going around here | |
| 14:19:01 | dansmith | hmm, I must be missing that part | |
| 14:19:03 | bauzas | the reserved bit is for a global hypervisor overhead | |
| 14:19:22 | bauzas | like a memory heap size you wanna not touch | |
| 14:19:35 | mriedem | jaypipes: hmm, but what we claim would actually be more than what we report for the actual allocation, agreed? | |
| 14:19:36 | dansmith | having allocations include per-instance overhead seems confusing to me, unless it's called out specifically | |
| 14:19:37 | mriedem | or could be | |
| 14:19:51 | dansmith | otherwise it will not be super helpful for use by other tools | |
| 14:19:52 | jaypipes | mriedem: right, which is partly why we didn't include it in that code. | |
| 14:20:14 | mriedem | jaypipes: but shouldn't be as accurate as possible in the allocation since that's going to be used by the scheduler? | |
| 14:20:37 | jaypipes | mriedem: I guess, but the scheduler doesn't ask for that amount :) | |
| 14:21:03 | dansmith | ah, estimate_instance_overhead | |
| 14:21:06 | mriedem | i'm going to need a refresher on what the scheduler asks for in placement, is it just based on inventory? | |
| 14:21:11 | dansmith | jaypipes: nor can it, right? it comes from the virt driver | |
| 14:21:11 | bauzas | yep, that's what I was trying to explain | |
| 14:21:18 | jaypipes | mriedem: in other words, the scheduler doesn't say "I want X MEMORY_MB plus some overhead if the instance is on a Xen hypervisor" | |
| 14:21:22 | mriedem | dansmith: right claims in the scheduler can't use this | |
| 14:21:22 | bauzas | there are two things to consider | |
| 14:21:24 | jaypipes | dansmith: right. | |
| 14:21:31 | bauzas | 1/ what you request and 2/ what you have as free | |
| 14:21:41 | mriedem | jaypipes: i realize, that's not what i'm saying though | |
| 14:21:50 | bauzas | in the claim, we add more to request with the overhead but we don't touch 2/ | |
| 14:22:07 | mriedem | do we or don't we look at allocations when figuring out which providers can serve a request in the scheduler? | |
| 14:22:10 | jaypipes | mriedem: I think the long-term solution to this is the following: | |
| 14:22:10 | bauzas | while we do later in the self-heal process, do decrease 2/ | |
| 14:22:35 | bauzas | so, when it comes to placement, the free space is already decremented by the sum of overheads | |
| 14:22:39 | jaypipes | mriedem: add an overheads table to placement API, containing resource_provider_id, resource_class_id, and amount. | |
| 14:22:47 | bauzas | (or when it comes to RAMfilter) | |
| 14:22:57 | jaypipes | mriedem: each record in that table would contain the overhead for that resource class on that provider for each consumer of that resource class. | |
| 14:23:22 | dansmith | jaypipes: ooh, yeah I like that | |
| 14:23:24 | jaypipes | mriedem: that way, we can properly model both the inventory and allocation request in the placement API. | |
| 14:23:30 | dansmith | jaypipes: make the virt driver delcare it ahead of time | |
| 14:23:41 | jaypipes | dansmith: right, in the RT's init_compute_node(), etc | |
| 14:23:42 | mriedem | but, | |
| 14:23:53 | BobBall | jaypipes: My understanding is the overhead on Xen for RAM is smallish. Yes, there will be some instances where we are running very tight against total RAM that need rescheduling, but I'm not sure it's worth the code changes to fix it? | |
| 14:23:53 | mriedem | the overhead is a factor of the flavor | |
| 14:23:53 | dansmith | jaypipes: which kinda pushes us more to a model where we speak about everything in terms of resourceclasses | |
| 14:24:08 | dansmith | mriedem: right, but make it declare how it calculates it in terms of resource classes | |
| 14:24:10 | jaypipes | mriedem: no, it's a factor of the resource class. | |
| 14:24:23 | jaypipes | mriedem: each virt driver has a separate overhead for cpu, disk and memory. | |
| 14:24:24 | dansmith | mriedem: because eventually we get rid of weirdo extra specs and it's all resource classes | |
| 14:24:29 | jaypipes | mriedem: and soon, emulator_threads :) | |
| 14:25:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/hyperv/vmops.py#L124 is a simple case | |
| 14:25:10 | jaypipes | BobBall: I think having an overheads table in the placement API containing this information is the best long-term solution. it would make the system data-driven instead of hard-coded in the virt driver. | |
| 14:25:35 | mriedem | so the overheads table contains the multiplier? | |
| 14:25:36 | jaypipes | mriedem: right, that's not per-flavor. it's per reesource class.. | |
| 14:25:41 | mriedem | for a given resource class on that provider? | |
| 14:25:46 | jaypipes | yep | |
| 14:25:53 | dansmith | jaypipes: well, that's crossing the line.. because it's disk = memory + N | |
| 14:25:55 | BobBall | jaypipes: OK - but it's going to be a best guess at any rate. AIUI the actual overhead will depend on the guest type as well as various other factors which we really won't want to encode in any table... | |
| 14:26:01 | dansmith | jaypipes: that one is ... hard | |
| 14:26:14 | mriedem | dansmith: store a lambda in the table?! :P | |
| 14:26:25 | dansmith | mriedem: genius! | |
| 14:27:02 | cdent | Is there some way we can do this algorithmically based on the inventory of the node when it starts up, do a calculation, and simply add that to the reserved? Because this all sounds fragile and constantly in need of adding yet more stuff. | |
| 14:27:31 | cdent | If we know that an average use scenaior of an inventory that looks like X needs a multiplier on some scale... | |
| 14:27:32 | bauzas | tbh, looks very dynamic to me, since it's provided by the virt driver | |
| 14:27:45 | mriedem | right, i don't know how we encode this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L713 | |
| 14:27:47 | bauzas | I mean, are we sure the overhead can mathematically be given ? | |
| 14:29:18 | dansmith | mriedem: the libvirt one is not that bad I think, but the hyperv one is | |
| 14:29:49 | jaypipes | mriedem: I think for that hyperv one, it may just be that the DISK_GB inventory record on hyperv compute nodes needs a pre-set reserved value :( | |
| 14:29:59 | dansmith | although the libvirt one would need to be able to specify a trait that increases the cpu overhead by one, I guess | |
| 14:30:16 | cdent | /o\ | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791 | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: Update resource tracker to PUT custom resource classes https://review.openstack.org/456915 | |
| 14:30:59 | mriedem | bauzas: going back to your auto-heal, are you referring to https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 which eventually calls _update_usage? | |
| 14:31:12 | mriedem | and then later we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L648 | |