| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 13:30:12 | andreykurilin | hi folks! Could someone look at small (+9, -5) patch to novaclient https://review.openstack.org/#/c/450763/ ? | |
| 13:30:24 | bauzas | tbc, the conductor only get a subset of hosts returned by the scheduler, not all the accepted nodes | |
| 13:30:37 | bauzas | where the default subset is 1 | |
| 13:31:31 | bauzas | mriedem: the problem wasn't really with the POST logic, rather the DELETE allocation logic when rescheduling or moving the instance | |
| 13:31:32 | mriedem | right, i didn't realize that until yesterday | |
| 13:31:50 | mriedem | bauzas: i think it's fine to do the delete in conductor on reschedules | |
| 13:31:53 | bauzas | mriedem: given the scheduler doesn't know whether it's a boot request, or a move | |
| 13:31:58 | mriedem | like we do for cleaning up bdms and ports | |
| 13:32:05 | bauzas | okay | |
| 13:32:08 | bauzas | fine by me then | |
| 13:32:16 | bauzas | that's 2 distinct situations | |
| 13:32:25 | mriedem | ok, think you can have the spec updated today? | |
| 13:32:45 | bauzas | we could POST the allocation in the current synchronous method we have | |
| 13:32:50 | mriedem | andreykurilin: why does that really need to change? does it just bug you? or is something building on that later? | |
| 13:33:04 | bauzas | and then DELETE allocations for reschedules/moves in the conductor | |
| 13:33:12 | bauzas | mriedem: yup, was planning to | |
| 13:33:26 | mriedem | bauzas: post somewhere in here? https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L124 | |
| 13:33:32 | bauzas | nope | |
| 13:34:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L80 ? | |
| 13:34:23 | bauzas | mriedem: within consume_from_request https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L131 | |
| 13:34:33 | bauzas | because it's a synchronous method | |
| 13:34:54 | bauzas | I mean there is a semaphore | |
| 13:35:02 | bauzas | synchronous section of code | |
| 13:35:27 | bauzas | dammit, 3.5 days out of OpenStack and my English is still on holiday :/ | |
| 13:35:29 | mriedem | so https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L271 | |
| 13:35:50 | bauzas | probablement oui | |
| 13:36:10 | bauzas | but that's an implementation detail IMHO | |
| 13:36:17 | andreykurilin | mriedem: I just saw an usage of that method in some script and it was not clear what does it accept without searching at docs.o.o/nova (usually, looking at novaclient's docstring is enough) | |
| 13:41:13 | mriedem | do we have some sort of memory overhead estimate code in the virt drivers that impacts scheduling via the resource tracker? | |
| 13:42:08 | mriedem | ah estimate_instance_overhead | |
| 13:43:55 | mriedem | wow that's a fun method, it can get 1 of 3 different input parameter types | |
| 13:45:02 | mriedem | bauzas: hmm, so that's going to be something interesting that will have to be taken into account to do claims in the scheduler | |
| 13:45:13 | mriedem | totally coincidental but someone internal was asking about this for kvm | |
| 13:45:22 | efried | mriedem I know the PowerVM code for estimate_instance_overhead is... interesting. | |
| 13:45:32 | mriedem | but i see RT.instance_claim calls estimate_instance_overhead on the virt driver | |
| 13:46:06 | mriedem | so how are we going to get this information from the virt driver when we're doing it in the scheduler | |
| 13:47:17 | mriedem | oh it looks like sahid just implemented this for the libvirt driver | |
| 13:47:53 | efried | mriedem Link? | |
| 13:48:07 | efried | Is this something all drivers (including OOT) will need to make changes for? | |
| 13:48:20 | mriedem | https://review.openstack.org/#/c/385364/ | |
| 13:48:27 | mriedem | wait link to what? | |
| 13:48:41 | mriedem | efried: claims in the scheduler spec https://review.openstack.org/#/c/437424/ | |
| 13:48:56 | efried | Yeah, that. Thanks. | |
| 13:49:26 | efried | mriedem FYI: https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L249 | |
| 13:51:26 | mriedem | btw, the fact we can pass in 3 different things to this... | |
| 13:51:33 | efried | Yeah... awkward | |
| 13:51:35 | mriedem | terrible | |
| 13:52:00 | mriedem | if it should just always be a flavor, we should just do that | |
| 13:52:13 | mriedem | i like how the powervm driver is turning it into a flavor at the start | |
| 13:53:25 | efried | mriedem If the caller(s) just changed to always use a flavor, it remains compatible, and consumers can clean out dead code at their leisure. | |
| 13:55:03 | efried | worthy of a (possibly specless) blueprint? | |
| 13:57:17 | mriedem | probably don't need a blueprint. we could also use a new method and deprecate the old one. or just as you say always pass a flavor and update the docstrings and let virt drivers get updated | |
| 14:00:39 | bauzas | mriedem: sorry, my IRC client crashed silently | |
| 14:00:44 | bauzas | mriedem: what do you mean ? | |
| 14:00:54 | mriedem | jaypipes: does memory_mb_used on the compute node find it's way into placement anywhere? | |
| 14:01:04 | bauzas | yup, some virt drivers reserve some overhead, like Xen AFAIK | |
| 14:01:13 | mriedem | i only see cn.memory_mb used for inventory on the MEMORY_MB resource class for the compute RT | |
| 14:01:15 | mriedem | *RP | |
| 14:01:23 | mriedem | bauzas: xen, hyperv and now libvirt | |
| 14:01:37 | mriedem | which impacts how cn.memory_mb_used is reported in the RT | |
| 14:01:44 | mriedem | and cn.free_ram_mb | |
| 14:01:55 | bauzas | mriedem: memory_mb_used is just summing the flavor.memory for all the running instances AFAIK | |
| 14:02:28 | bauzas | mriedem: compared to the total size which is returned by the hypervisor | |
| 14:02:40 | mriedem | via get_available_resource right? | |
| 14:02:42 | mriedem | or get_inventory | |
| 14:02:43 | bauzas | yup | |
| 14:03:00 | bauzas | formerly get_avail_resource IIRC | |
| 14:03:08 | mriedem | right so when we make a claim for a new instance, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L144 | |
| 14:03:10 | bauzas | with ironic, get_inventory | |
| 14:03:13 | mriedem | we calculate the overhead for the vm | |
| 14:04:05 | mriedem | and the claim.memory_mb is based on the instance.flavor.memory_mb + that overhead value from the compute driver | |
| 14:04:11 | bauzas | right | |
| 14:05:13 | mriedem | what i'm trying to figure out, | |
| 14:05:30 | mriedem | is if this overhead estimate from the virt driver factors into scheduling decisions in the scheduler anywhere | |
| 14:05:49 | mriedem | or is it something we can only determine once we're on a host, find out that with the overhead we are over the limit, and fail the claim test | |
| 14:05:51 | mriedem | and trigger a retry | |
| 14:06:25 | bauzas | mriedem: the RamFilter doesn't get that for a specific virt driver, so I'd assume no | |
| 14:06:53 | mriedem | right. but do we record it in the allocation record? | |
| 14:06:55 | mriedem | in placement | |
| 14:07:05 | bauzas | the overhead ? | |
| 14:07:20 | mriedem | e.g. if the full allocation for MEMORY_MB for a given instance is instance.flavor.memory_mb + overhead from virt driver, do we put that into the allocation? | |
| 14:07:26 | mriedem | or just instance.flavor.memory_mb? | |
| 14:07:49 | mriedem | the overhead value that comes out of virt.estimate_instance_overhead | |
| 14:07:54 | bauzas | well, all happens in _update_usage_from_instance so looking | |
| 14:08:34 | bauzas | mriedem: looks like it's not | |
| 14:08:40 | jaypipes | mriedem: yes, it's the inventory for MEMORY_MB :) | |
| 14:09:24 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:09:41 | mriedem | aha | |
| 14:09:47 | mriedem | then yeah that seems like a bug | |
| 14:09:59 | jaypipes | mriedem: what? the overhead? | |
| 14:10:03 | mriedem | yes | |
| 14:10:11 | mriedem | because https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L106 | |
| 14:10:35 | mriedem | which is tested in the claim here https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L168 | |
| 14:10:37 | mriedem | right? | |
| 14:10:41 | bauzas | jaypipes: the overhead is only calculated when calling the claim AFAICS | |
| 14:11:09 | mriedem | requested = self.memory_mb == instance.flavor.memory_mb + overhead['memory_mb'] | |
| 14:11:46 | jaypipes | mriedem: meh, maybe. it's true there's a little bit of possible inconsistency. I was hoping to resolve that with claims in scheduler. but we need to know the virt driver of the target host *before* we actually allocate :( which is tarded. | |
| 14:11:58 | bauzas | mriedem: well, seems like it's self-healed actually | |
| 14:12:02 | mriedem | jaypipes: right, so we have this whole chicken and egg | |
| 14:12:23 | bauzas | mriedem: because RT._update_usage() does calculate the overhead past the allocation | |