| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 13:27:26 | bauzas | mriedem: now, the compute node should HTTP get the list of allocations instead of having those passed thru RPC, correct? | |
| 13:28:13 | bauzas | mriedem: also, seems like there was kind of a confusion where the POST allocations should be called, ie. in the conductor vs. scheduler | |
| 13:28:22 | openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: tests: add a hook for functional testing in the gate https://review.openstack.org/452122 | |
| 13:28:48 | bauzas | FWIW, I thought we all agreed to have the conductor doing that | |
| 13:28:58 | bauzas | (during the hangout) | |
| 13:29:16 | mriedem | bauzas: i thought that when i thought, incorrectly, that the conductor got the list of filtered and weighed hosts | |
| 13:29:42 | bauzas | yup | |
| 13:29:48 | mriedem | we could do it in either, but it seems to me it would be better to retry by just hitting the next host in the list of already filtered hosts, in the scheduler, | |
| 13:30:03 | mriedem | rather than having conductor ask the scheduler to pull a fresh new set all over again | |
| 13:30:12 | andreykurilin | hi folks! Could someone look at small (+9, -5) patch to novaclient https://review.openstack.org/#/c/450763/ ? | |
| 13:30:24 | bauzas | tbc, the conductor only get a subset of hosts returned by the scheduler, not all the accepted nodes | |
| 13:30:37 | bauzas | where the default subset is 1 | |
| 13:31:31 | bauzas | mriedem: the problem wasn't really with the POST logic, rather the DELETE allocation logic when rescheduling or moving the instance | |
| 13:31:32 | mriedem | right, i didn't realize that until yesterday | |
| 13:31:50 | mriedem | bauzas: i think it's fine to do the delete in conductor on reschedules | |
| 13:31:53 | bauzas | mriedem: given the scheduler doesn't know whether it's a boot request, or a move | |
| 13:31:58 | mriedem | like we do for cleaning up bdms and ports | |
| 13:32:05 | bauzas | okay | |
| 13:32:08 | bauzas | fine by me then | |
| 13:32:16 | bauzas | that's 2 distinct situations | |
| 13:32:25 | mriedem | ok, think you can have the spec updated today? | |
| 13:32:45 | bauzas | we could POST the allocation in the current synchronous method we have | |
| 13:32:50 | mriedem | andreykurilin: why does that really need to change? does it just bug you? or is something building on that later? | |
| 13:33:04 | bauzas | and then DELETE allocations for reschedules/moves in the conductor | |
| 13:33:12 | bauzas | mriedem: yup, was planning to | |
| 13:33:26 | mriedem | bauzas: post somewhere in here? https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L124 | |
| 13:33:32 | bauzas | nope | |
| 13:34:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L80 ? | |
| 13:34:23 | bauzas | mriedem: within consume_from_request https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L131 | |
| 13:34:33 | bauzas | because it's a synchronous method | |
| 13:34:54 | bauzas | I mean there is a semaphore | |
| 13:35:02 | bauzas | synchronous section of code | |
| 13:35:27 | bauzas | dammit, 3.5 days out of OpenStack and my English is still on holiday :/ | |
| 13:35:29 | mriedem | so https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L271 | |
| 13:35:50 | bauzas | probablement oui | |
| 13:36:10 | bauzas | but that's an implementation detail IMHO | |
| 13:36:17 | andreykurilin | mriedem: I just saw an usage of that method in some script and it was not clear what does it accept without searching at docs.o.o/nova (usually, looking at novaclient's docstring is enough) | |
| 13:41:13 | mriedem | do we have some sort of memory overhead estimate code in the virt drivers that impacts scheduling via the resource tracker? | |
| 13:42:08 | mriedem | ah estimate_instance_overhead | |
| 13:43:55 | mriedem | wow that's a fun method, it can get 1 of 3 different input parameter types | |
| 13:45:02 | mriedem | bauzas: hmm, so that's going to be something interesting that will have to be taken into account to do claims in the scheduler | |
| 13:45:13 | mriedem | totally coincidental but someone internal was asking about this for kvm | |
| 13:45:22 | efried | mriedem I know the PowerVM code for estimate_instance_overhead is... interesting. | |
| 13:45:32 | mriedem | but i see RT.instance_claim calls estimate_instance_overhead on the virt driver | |
| 13:46:06 | mriedem | so how are we going to get this information from the virt driver when we're doing it in the scheduler | |
| 13:47:17 | mriedem | oh it looks like sahid just implemented this for the libvirt driver | |
| 13:47:53 | efried | mriedem Link? | |
| 13:48:07 | efried | Is this something all drivers (including OOT) will need to make changes for? | |
| 13:48:20 | mriedem | https://review.openstack.org/#/c/385364/ | |
| 13:48:27 | mriedem | wait link to what? | |
| 13:48:41 | mriedem | efried: claims in the scheduler spec https://review.openstack.org/#/c/437424/ | |
| 13:48:56 | efried | Yeah, that. Thanks. | |
| 13:49:26 | efried | mriedem FYI: https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L249 | |
| 13:51:26 | mriedem | btw, the fact we can pass in 3 different things to this... | |
| 13:51:33 | efried | Yeah... awkward | |
| 13:51:35 | mriedem | terrible | |
| 13:52:00 | mriedem | if it should just always be a flavor, we should just do that | |
| 13:52:13 | mriedem | i like how the powervm driver is turning it into a flavor at the start | |
| 13:53:25 | efried | mriedem If the caller(s) just changed to always use a flavor, it remains compatible, and consumers can clean out dead code at their leisure. | |
| 13:55:03 | efried | worthy of a (possibly specless) blueprint? | |
| 13:57:17 | mriedem | probably don't need a blueprint. we could also use a new method and deprecate the old one. or just as you say always pass a flavor and update the docstrings and let virt drivers get updated | |
| 14:00:39 | bauzas | mriedem: sorry, my IRC client crashed silently | |
| 14:00:44 | bauzas | mriedem: what do you mean ? | |
| 14:00:54 | mriedem | jaypipes: does memory_mb_used on the compute node find it's way into placement anywhere? | |
| 14:01:04 | bauzas | yup, some virt drivers reserve some overhead, like Xen AFAIK | |
| 14:01:13 | mriedem | i only see cn.memory_mb used for inventory on the MEMORY_MB resource class for the compute RT | |
| 14:01:15 | mriedem | *RP | |
| 14:01:23 | mriedem | bauzas: xen, hyperv and now libvirt | |
| 14:01:37 | mriedem | which impacts how cn.memory_mb_used is reported in the RT | |
| 14:01:44 | mriedem | and cn.free_ram_mb | |
| 14:01:55 | bauzas | mriedem: memory_mb_used is just summing the flavor.memory for all the running instances AFAIK | |
| 14:02:28 | bauzas | mriedem: compared to the total size which is returned by the hypervisor | |
| 14:02:40 | mriedem | via get_available_resource right? | |
| 14:02:42 | mriedem | or get_inventory | |
| 14:02:43 | bauzas | yup | |
| 14:03:00 | bauzas | formerly get_avail_resource IIRC | |
| 14:03:08 | mriedem | right so when we make a claim for a new instance, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L144 | |
| 14:03:10 | bauzas | with ironic, get_inventory | |
| 14:03:13 | mriedem | we calculate the overhead for the vm | |
| 14:04:05 | mriedem | and the claim.memory_mb is based on the instance.flavor.memory_mb + that overhead value from the compute driver | |
| 14:04:11 | bauzas | right | |
| 14:05:13 | mriedem | what i'm trying to figure out, | |
| 14:05:30 | mriedem | is if this overhead estimate from the virt driver factors into scheduling decisions in the scheduler anywhere | |
| 14:05:49 | mriedem | or is it something we can only determine once we're on a host, find out that with the overhead we are over the limit, and fail the claim test | |
| 14:05:51 | mriedem | and trigger a retry | |
| 14:06:25 | bauzas | mriedem: the RamFilter doesn't get that for a specific virt driver, so I'd assume no | |
| 14:06:53 | mriedem | right. but do we record it in the allocation record? | |
| 14:06:55 | mriedem | in placement | |
| 14:07:05 | bauzas | the overhead ? | |
| 14:07:20 | mriedem | e.g. if the full allocation for MEMORY_MB for a given instance is instance.flavor.memory_mb + overhead from virt driver, do we put that into the allocation? | |
| 14:07:26 | mriedem | or just instance.flavor.memory_mb? | |
| 14:07:49 | mriedem | the overhead value that comes out of virt.estimate_instance_overhead | |
| 14:07:54 | bauzas | well, all happens in _update_usage_from_instance so looking | |
| 14:08:34 | bauzas | mriedem: looks like it's not | |
| 14:08:40 | jaypipes | mriedem: yes, it's the inventory for MEMORY_MB :) | |
| 14:09:24 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:09:41 | mriedem | aha | |
| 14:09:47 | mriedem | then yeah that seems like a bug | |
| 14:09:59 | jaypipes | mriedem: what? the overhead? | |
| 14:10:03 | mriedem | yes | |