| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 13:48:20 | mriedem | https://review.openstack.org/#/c/385364/ | |
| 13:48:27 | mriedem | wait link to what? | |
| 13:48:41 | mriedem | efried: claims in the scheduler spec https://review.openstack.org/#/c/437424/ | |
| 13:48:56 | efried | Yeah, that. Thanks. | |
| 13:49:26 | efried | mriedem FYI: https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L249 | |
| 13:51:26 | mriedem | btw, the fact we can pass in 3 different things to this... | |
| 13:51:33 | efried | Yeah... awkward | |
| 13:51:35 | mriedem | terrible | |
| 13:52:00 | mriedem | if it should just always be a flavor, we should just do that | |
| 13:52:13 | mriedem | i like how the powervm driver is turning it into a flavor at the start | |
| 13:53:25 | efried | mriedem If the caller(s) just changed to always use a flavor, it remains compatible, and consumers can clean out dead code at their leisure. | |
| 13:55:03 | efried | worthy of a (possibly specless) blueprint? | |
| 13:57:17 | mriedem | probably don't need a blueprint. we could also use a new method and deprecate the old one. or just as you say always pass a flavor and update the docstrings and let virt drivers get updated | |
| 14:00:39 | bauzas | mriedem: sorry, my IRC client crashed silently | |
| 14:00:44 | bauzas | mriedem: what do you mean ? | |
| 14:00:54 | mriedem | jaypipes: does memory_mb_used on the compute node find it's way into placement anywhere? | |
| 14:01:04 | bauzas | yup, some virt drivers reserve some overhead, like Xen AFAIK | |
| 14:01:13 | mriedem | i only see cn.memory_mb used for inventory on the MEMORY_MB resource class for the compute RT | |
| 14:01:15 | mriedem | *RP | |
| 14:01:23 | mriedem | bauzas: xen, hyperv and now libvirt | |
| 14:01:37 | mriedem | which impacts how cn.memory_mb_used is reported in the RT | |
| 14:01:44 | mriedem | and cn.free_ram_mb | |
| 14:01:55 | bauzas | mriedem: memory_mb_used is just summing the flavor.memory for all the running instances AFAIK | |
| 14:02:28 | bauzas | mriedem: compared to the total size which is returned by the hypervisor | |
| 14:02:40 | mriedem | via get_available_resource right? | |
| 14:02:42 | mriedem | or get_inventory | |
| 14:02:43 | bauzas | yup | |
| 14:03:00 | bauzas | formerly get_avail_resource IIRC | |
| 14:03:08 | mriedem | right so when we make a claim for a new instance, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L144 | |
| 14:03:10 | bauzas | with ironic, get_inventory | |
| 14:03:13 | mriedem | we calculate the overhead for the vm | |
| 14:04:05 | mriedem | and the claim.memory_mb is based on the instance.flavor.memory_mb + that overhead value from the compute driver | |
| 14:04:11 | bauzas | right | |
| 14:05:13 | mriedem | what i'm trying to figure out, | |
| 14:05:30 | mriedem | is if this overhead estimate from the virt driver factors into scheduling decisions in the scheduler anywhere | |
| 14:05:49 | mriedem | or is it something we can only determine once we're on a host, find out that with the overhead we are over the limit, and fail the claim test | |
| 14:05:51 | mriedem | and trigger a retry | |
| 14:06:25 | bauzas | mriedem: the RamFilter doesn't get that for a specific virt driver, so I'd assume no | |
| 14:06:53 | mriedem | right. but do we record it in the allocation record? | |
| 14:06:55 | mriedem | in placement | |
| 14:07:05 | bauzas | the overhead ? | |
| 14:07:20 | mriedem | e.g. if the full allocation for MEMORY_MB for a given instance is instance.flavor.memory_mb + overhead from virt driver, do we put that into the allocation? | |
| 14:07:26 | mriedem | or just instance.flavor.memory_mb? | |
| 14:07:49 | mriedem | the overhead value that comes out of virt.estimate_instance_overhead | |
| 14:07:54 | bauzas | well, all happens in _update_usage_from_instance so looking | |
| 14:08:34 | bauzas | mriedem: looks like it's not | |
| 14:08:40 | jaypipes | mriedem: yes, it's the inventory for MEMORY_MB :) | |
| 14:09:24 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:09:41 | mriedem | aha | |
| 14:09:47 | mriedem | then yeah that seems like a bug | |
| 14:09:59 | jaypipes | mriedem: what? the overhead? | |
| 14:10:03 | mriedem | yes | |
| 14:10:11 | mriedem | because https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L106 | |
| 14:10:35 | mriedem | which is tested in the claim here https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L168 | |
| 14:10:37 | mriedem | right? | |
| 14:10:41 | bauzas | jaypipes: the overhead is only calculated when calling the claim AFAICS | |
| 14:11:09 | mriedem | requested = self.memory_mb == instance.flavor.memory_mb + overhead['memory_mb'] | |
| 14:11:46 | jaypipes | mriedem: meh, maybe. it's true there's a little bit of possible inconsistency. I was hoping to resolve that with claims in scheduler. but we need to know the virt driver of the target host *before* we actually allocate :( which is tarded. | |
| 14:11:58 | bauzas | mriedem: well, seems like it's self-healed actually | |
| 14:12:02 | mriedem | jaypipes: right, so we have this whole chicken and egg | |
| 14:12:23 | bauzas | mriedem: because RT._update_usage() does calculate the overhead past the allocation | |
| 14:13:00 | bauzas | mriedem: and given we call _update() after _update_usage() with the resources that are taking in account that overhead, it auto-heals | |
| 14:13:01 | jaypipes | mriedem: instead of doing the chicken egg thing, I think it's better to instead use the inventory record's reserved value. that may be a coarser-grained solution, but it's certainly cleaner. | |
| 14:13:02 | johnthetubaguy | mriedem: probably not a shock, but alot of that memory crazy is Xen related, or at least very related to when you have "hard" memory limits | |
| 14:13:10 | cdent | shouldn't overhead (as a sum of potentials) be recorded within reserved and _not_ be in the consumer's allocations? | |
| 14:13:38 | mriedem | bauzas: i'm not following the full call chain yet | |
| 14:13:40 | jaypipes | cdent: it's overhead *per instance*, whereas reserved is per provider/resource cllass. | |
| 14:13:51 | mriedem | bauzas: but you agree we can hit ComputeResourcesUnavailable due to the overhead pushing us over right? | |
| 14:13:51 | bauzas | what jaypipes said | |
| 14:13:52 | jaypipes | cdent: thus reserved is "more coarse-grained" | |
| 14:13:52 | cdent | jaypipes: I know, thus the "sum of potentials" | |
| 14:13:59 | jaypipes | cdent: right. | |
| 14:13:59 | cdent | and thus the "jinx" | |
| 14:14:03 | jaypipes | :) | |
| 14:14:11 | bauzas | mriedem: sec, explaining | |
| 14:14:14 | johnthetubaguy | jaypipes: cdent: so the memory overhead is VM specific in some cases (ducks) but I would argue we might want to ignore that | |
| 14:14:26 | mriedem | johnthetubaguy: exactly | |
| 14:14:30 | jaypipes | johnthetubaguy: ++ | |
| 14:14:34 | mriedem | the overhead calculation is totally based on the flavor used | |
| 14:14:36 | bauzas | mriedem: so, the RamFilter is calculating against the free ram based on what the RT reports, right? | |
| 14:15:07 | bauzas | mriedem: what I'm saying is that I just saw that we do calculate the overhead when updating the usage in the RT | |
| 14:15:12 | mriedem | so do you then make a generic reserved value for the RT on that provider for what you expect to land on that host flavor-wise? | |
| 14:15:32 | mriedem | bauzas: i thought we wanted to deprecate/remove the RamFilter? | |
| 14:15:34 | mriedem | because we use placement now | |
| 14:15:38 | johnthetubaguy | mriedem: yeah, I got the impression that its party host CPU model specific, but I never got a straight answer about that | |
| 14:16:42 | bauzas | mriedem: well, I'm trying to figure out the situation in a pre-placement world | |
| 14:16:43 | mriedem | ok, so the reason i'm even looking at this, is because someone internal is looking at this for the libvirt driver and with an example vm they calculated ~226MB of overhead | |
| 14:16:50 | mriedem | which isn't being taken into account for scheduling | |
| 14:16:50 | jaypipes | frankly, mriedem and johnthetubaguy, we're going to end up with a similar issue with the emulator threads. | |
| 14:17:02 | johnthetubaguy | jaypipes: oh, true | |
| 14:17:06 | jaypipes | mriedem, johnthetubaguy: because the reserved amount of those is specific to a virt driver./ | |
| 14:17:10 | openstackgerrit | Dan Smith proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 14:17:22 | jaypipes | course, we need nested providers first, but still, thinking ahead... | |
| 14:17:39 | mriedem | can we agree that https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should be taking into account the overhead? | |
| 14:18:19 | dansmith | I haven't been paying attention, | |
| 14:18:26 | bauzas | I'm still looking at the 100+ calls involved in the decision-making :) | |
| 14:18:28 | dansmith | but the overhead should be included in the inventory reserved amount no? | |
| 14:18:34 | jaypipes | mriedem: perhaps, though that may lead to a failure to allocate. | |
| 14:18:38 | bauzas | dansmith: it's per instance | |
| 14:18:48 | jaypipes | dansmith: yeah, per instance... | |