Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-18
13:29:42 bauzas yup
13:29:48 mriedem we could do it in either, but it seems to me it would be better to retry by just hitting the next host in the list of already filtered hosts, in the scheduler,
13:30:03 mriedem rather than having conductor ask the scheduler to pull a fresh new set all over again
13:30:12 andreykurilin hi folks! Could someone look at small (+9, -5) patch to novaclient https://review.openstack.org/#/c/450763/ ?
13:30:24 bauzas tbc, the conductor only get a subset of hosts returned by the scheduler, not all the accepted nodes
13:30:37 bauzas where the default subset is 1
13:31:31 bauzas mriedem: the problem wasn't really with the POST logic, rather the DELETE allocation logic when rescheduling or moving the instance
13:31:32 mriedem right, i didn't realize that until yesterday
13:31:50 mriedem bauzas: i think it's fine to do the delete in conductor on reschedules
13:31:53 bauzas mriedem: given the scheduler doesn't know whether it's a boot request, or a move
13:31:58 mriedem like we do for cleaning up bdms and ports
13:32:05 bauzas okay
13:32:08 bauzas fine by me then
13:32:16 bauzas that's 2 distinct situations
13:32:25 mriedem ok, think you can have the spec updated today?
13:32:45 bauzas we could POST the allocation in the current synchronous method we have
13:32:50 mriedem andreykurilin: why does that really need to change? does it just bug you? or is something building on that later?
13:33:04 bauzas and then DELETE allocations for reschedules/moves in the conductor
13:33:12 bauzas mriedem: yup, was planning to
13:33:26 mriedem bauzas: post somewhere in here? https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L124
13:33:32 bauzas nope
13:34:20 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L80 ?
13:34:23 bauzas mriedem: within consume_from_request https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L131
13:34:33 bauzas because it's a synchronous method
13:34:54 bauzas I mean there is a semaphore
13:35:02 bauzas synchronous section of code
13:35:27 bauzas dammit, 3.5 days out of OpenStack and my English is still on holiday :/
13:35:29 mriedem so https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L271
13:35:50 bauzas probablement oui
13:36:10 bauzas but that's an implementation detail IMHO
13:36:17 andreykurilin mriedem: I just saw an usage of that method in some script and it was not clear what does it accept without searching at docs.o.o/nova (usually, looking at novaclient's docstring is enough)
13:41:13 mriedem do we have some sort of memory overhead estimate code in the virt drivers that impacts scheduling via the resource tracker?
13:42:08 mriedem ah estimate_instance_overhead
13:43:55 mriedem wow that's a fun method, it can get 1 of 3 different input parameter types
13:45:02 mriedem bauzas: hmm, so that's going to be something interesting that will have to be taken into account to do claims in the scheduler
13:45:13 mriedem totally coincidental but someone internal was asking about this for kvm
13:45:22 efried mriedem I know the PowerVM code for estimate_instance_overhead is... interesting.
13:45:32 mriedem but i see RT.instance_claim calls estimate_instance_overhead on the virt driver
13:46:06 mriedem so how are we going to get this information from the virt driver when we're doing it in the scheduler
13:47:17 mriedem oh it looks like sahid just implemented this for the libvirt driver
13:47:53 efried mriedem Link?
13:48:07 efried Is this something all drivers (including OOT) will need to make changes for?
13:48:20 mriedem https://review.openstack.org/#/c/385364/
13:48:27 mriedem wait link to what?
13:48:41 mriedem efried: claims in the scheduler spec https://review.openstack.org/#/c/437424/
13:48:56 efried Yeah, that. Thanks.
13:49:26 efried mriedem FYI: https://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L249
13:51:26 mriedem btw, the fact we can pass in 3 different things to this...
13:51:33 efried Yeah... awkward
13:51:35 mriedem terrible
13:52:00 mriedem if it should just always be a flavor, we should just do that
13:52:13 mriedem i like how the powervm driver is turning it into a flavor at the start
13:53:25 efried mriedem If the caller(s) just changed to always use a flavor, it remains compatible, and consumers can clean out dead code at their leisure.
13:55:03 efried worthy of a (possibly specless) blueprint?
13:57:17 mriedem probably don't need a blueprint. we could also use a new method and deprecate the old one. or just as you say always pass a flavor and update the docstrings and let virt drivers get updated
14:00:39 bauzas mriedem: sorry, my IRC client crashed silently
14:00:44 bauzas mriedem: what do you mean ?
14:00:54 mriedem jaypipes: does memory_mb_used on the compute node find it's way into placement anywhere?
14:01:04 bauzas yup, some virt drivers reserve some overhead, like Xen AFAIK
14:01:13 mriedem i only see cn.memory_mb used for inventory on the MEMORY_MB resource class for the compute RT
14:01:15 mriedem *RP
14:01:23 mriedem bauzas: xen, hyperv and now libvirt
14:01:37 mriedem which impacts how cn.memory_mb_used is reported in the RT
14:01:44 mriedem and cn.free_ram_mb
14:01:55 bauzas mriedem: memory_mb_used is just summing the flavor.memory for all the running instances AFAIK
14:02:28 bauzas mriedem: compared to the total size which is returned by the hypervisor
14:02:40 mriedem via get_available_resource right?
14:02:42 mriedem or get_inventory
14:02:43 bauzas yup
14:03:00 bauzas formerly get_avail_resource IIRC
14:03:08 mriedem right so when we make a claim for a new instance, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L144
14:03:10 bauzas with ironic, get_inventory
14:03:13 mriedem we calculate the overhead for the vm
14:04:05 mriedem and the claim.memory_mb is based on the instance.flavor.memory_mb + that overhead value from the compute driver
14:04:11 bauzas right
14:05:13 mriedem what i'm trying to figure out,
14:05:30 mriedem is if this overhead estimate from the virt driver factors into scheduling decisions in the scheduler anywhere
14:05:49 mriedem or is it something we can only determine once we're on a host, find out that with the overhead we are over the limit, and fail the claim test
14:05:51 mriedem and trigger a retry
14:06:25 bauzas mriedem: the RamFilter doesn't get that for a specific virt driver, so I'd assume no
14:06:53 mriedem right. but do we record it in the allocation record?
14:06:55 mriedem in placement
14:07:05 bauzas the overhead ?
14:07:20 mriedem e.g. if the full allocation for MEMORY_MB for a given instance is instance.flavor.memory_mb + overhead from virt driver, do we put that into the allocation?
14:07:26 mriedem or just instance.flavor.memory_mb?
14:07:49 mriedem the overhead value that comes out of virt.estimate_instance_overhead
14:07:54 bauzas well, all happens in _update_usage_from_instance so looking
14:08:34 bauzas mriedem: looks like it's not
14:08:40 jaypipes mriedem: yes, it's the inventory for MEMORY_MB :)
14:09:24 bauzas mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156
14:09:41 mriedem aha
14:09:47 mriedem then yeah that seems like a bug
14:09:59 jaypipes mriedem: what? the overhead?
14:10:03 mriedem yes
14:10:11 mriedem because https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L106
14:10:35 mriedem which is tested in the claim here https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L168
14:10:37 mriedem right?
14:10:41 bauzas jaypipes: the overhead is only calculated when calling the claim AFAICS
14:11:09 mriedem requested = self.memory_mb == instance.flavor.memory_mb + overhead['memory_mb']
14:11:46 jaypipes mriedem: meh, maybe. it's true there's a little bit of possible inconsistency. I was hoping to resolve that with claims in scheduler. but we need to know the virt driver of the target host *before* we actually allocate :( which is tarded.

Earlier   Later