| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 14:20:14 | mriedem | jaypipes: but shouldn't be as accurate as possible in the allocation since that's going to be used by the scheduler? | |
| 14:20:37 | jaypipes | mriedem: I guess, but the scheduler doesn't ask for that amount :) | |
| 14:21:03 | dansmith | ah, estimate_instance_overhead | |
| 14:21:06 | mriedem | i'm going to need a refresher on what the scheduler asks for in placement, is it just based on inventory? | |
| 14:21:11 | bauzas | yep, that's what I was trying to explain | |
| 14:21:11 | dansmith | jaypipes: nor can it, right? it comes from the virt driver | |
| 14:21:18 | jaypipes | mriedem: in other words, the scheduler doesn't say "I want X MEMORY_MB plus some overhead if the instance is on a Xen hypervisor" | |
| 14:21:22 | bauzas | there are two things to consider | |
| 14:21:22 | mriedem | dansmith: right claims in the scheduler can't use this | |
| 14:21:24 | jaypipes | dansmith: right. | |
| 14:21:31 | bauzas | 1/ what you request and 2/ what you have as free | |
| 14:21:41 | mriedem | jaypipes: i realize, that's not what i'm saying though | |
| 14:21:50 | bauzas | in the claim, we add more to request with the overhead but we don't touch 2/ | |
| 14:22:07 | mriedem | do we or don't we look at allocations when figuring out which providers can serve a request in the scheduler? | |
| 14:22:10 | bauzas | while we do later in the self-heal process, do decrease 2/ | |
| 14:22:10 | jaypipes | mriedem: I think the long-term solution to this is the following: | |
| 14:22:35 | bauzas | so, when it comes to placement, the free space is already decremented by the sum of overheads | |
| 14:22:39 | jaypipes | mriedem: add an overheads table to placement API, containing resource_provider_id, resource_class_id, and amount. | |
| 14:22:47 | bauzas | (or when it comes to RAMfilter) | |
| 14:22:57 | jaypipes | mriedem: each record in that table would contain the overhead for that resource class on that provider for each consumer of that resource class. | |
| 14:23:22 | dansmith | jaypipes: ooh, yeah I like that | |
| 14:23:24 | jaypipes | mriedem: that way, we can properly model both the inventory and allocation request in the placement API. | |
| 14:23:30 | dansmith | jaypipes: make the virt driver delcare it ahead of time | |
| 14:23:41 | jaypipes | dansmith: right, in the RT's init_compute_node(), etc | |
| 14:23:42 | mriedem | but, | |
| 14:23:53 | dansmith | jaypipes: which kinda pushes us more to a model where we speak about everything in terms of resourceclasses | |
| 14:23:53 | mriedem | the overhead is a factor of the flavor | |
| 14:23:53 | BobBall | jaypipes: My understanding is the overhead on Xen for RAM is smallish. Yes, there will be some instances where we are running very tight against total RAM that need rescheduling, but I'm not sure it's worth the code changes to fix it? | |
| 14:24:08 | dansmith | mriedem: right, but make it declare how it calculates it in terms of resource classes | |
| 14:24:10 | jaypipes | mriedem: no, it's a factor of the resource class. | |
| 14:24:23 | jaypipes | mriedem: each virt driver has a separate overhead for cpu, disk and memory. | |
| 14:24:24 | dansmith | mriedem: because eventually we get rid of weirdo extra specs and it's all resource classes | |
| 14:24:29 | jaypipes | mriedem: and soon, emulator_threads :) | |
| 14:25:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/hyperv/vmops.py#L124 is a simple case | |
| 14:25:10 | jaypipes | BobBall: I think having an overheads table in the placement API containing this information is the best long-term solution. it would make the system data-driven instead of hard-coded in the virt driver. | |
| 14:25:35 | mriedem | so the overheads table contains the multiplier? | |
| 14:25:36 | jaypipes | mriedem: right, that's not per-flavor. it's per reesource class.. | |
| 14:25:41 | mriedem | for a given resource class on that provider? | |
| 14:25:46 | jaypipes | yep | |
| 14:25:53 | dansmith | jaypipes: well, that's crossing the line.. because it's disk = memory + N | |
| 14:25:55 | BobBall | jaypipes: OK - but it's going to be a best guess at any rate. AIUI the actual overhead will depend on the guest type as well as various other factors which we really won't want to encode in any table... | |
| 14:26:01 | dansmith | jaypipes: that one is ... hard | |
| 14:26:14 | mriedem | dansmith: store a lambda in the table?! :P | |
| 14:26:25 | dansmith | mriedem: genius! | |
| 14:27:02 | cdent | Is there some way we can do this algorithmically based on the inventory of the node when it starts up, do a calculation, and simply add that to the reserved? Because this all sounds fragile and constantly in need of adding yet more stuff. | |
| 14:27:31 | cdent | If we know that an average use scenaior of an inventory that looks like X needs a multiplier on some scale... | |
| 14:27:32 | bauzas | tbh, looks very dynamic to me, since it's provided by the virt driver | |
| 14:27:45 | mriedem | right, i don't know how we encode this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L713 | |
| 14:27:47 | bauzas | I mean, are we sure the overhead can mathematically be given ? | |
| 14:29:18 | dansmith | mriedem: the libvirt one is not that bad I think, but the hyperv one is | |
| 14:29:49 | jaypipes | mriedem: I think for that hyperv one, it may just be that the DISK_GB inventory record on hyperv compute nodes needs a pre-set reserved value :( | |
| 14:29:59 | dansmith | although the libvirt one would need to be able to specify a trait that increases the cpu overhead by one, I guess | |
| 14:30:16 | cdent | /o\ | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: Update resource tracker to PUT custom resource classes https://review.openstack.org/456915 | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791 | |
| 14:30:59 | mriedem | bauzas: going back to your auto-heal, are you referring to https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 which eventually calls _update_usage? | |
| 14:31:12 | mriedem | and then later we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L648 | |
| 14:31:36 | jaypipes | mriedem: for the emulator threads, I think we can handle that differently... | |
| 14:31:54 | bauzas | mriedem: I'm saying we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L774 | |
| 14:32:15 | bauzas | because indeed we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 | |
| 14:32:27 | jaypipes | mriedem: for libvirt compute nodes, we'd have a fixed number of EMULATOR_THREADS inventory, with no overcommit. For other drivers, we'd have a huge overcommit of EMULATOR_THREADS inventory. | |
| 14:32:30 | mriedem | bauzas: right _update_usage_from_instance calls _update_usage | |
| 14:33:02 | mriedem | bauzas: but _update just changes the inventory record in placement | |
| 14:33:02 | bauzas | mriedem: so, _update_usage_from_instance calls placement by not adding the overhead | |
| 14:33:06 | mriedem | and the total doesn't change does it? | |
| 14:33:30 | bauzas | mriedem: but _update updates the inventory with the values it was updated by _update_usage() | |
| 14:34:13 | mriedem | yes, but, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L779-L795 | |
| 14:34:18 | mriedem | only sets the used/free values on the cn | |
| 14:34:25 | mriedem | which aren't used when setting the inventory in placement for that provider | |
| 14:34:34 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L117 | |
| 14:34:56 | mriedem | the inventory value comes from driver.get_available_resource/get_inventory, right? | |
| 14:35:01 | bauzas | mriedem: oh fuck, you're right | |
| 14:35:02 | jaypipes | yes | |
| 14:35:05 | mriedem | YES | |
| 14:35:09 | mriedem | ha, ok | |
| 14:35:10 | mriedem | SO... | |
| 14:35:20 | bauzas | it's just updating memory_used | |
| 14:35:24 | mriedem | going back to https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:35:25 | bauzas | so, not called to placement | |
| 14:35:26 | mriedem | ^ is wrong | |
| 14:35:31 | bauzas | mriedem: yup | |
| 14:35:47 | bauzas | mriedem: in a perfect world, we should add that to placement | |
| 14:35:53 | mriedem | let me look up the voodoo that placement does when scheduler asks for providers for a given reqspec | |
| 14:35:55 | bauzas | mriedem: because RamFilter does that | |
| 14:36:06 | bauzas | and we want placement to behave like the filter | |
| 14:38:15 | jaypipes | bauzas: I don't care if placement doesn't behave exactly like the RamFilter. I just want placement to be clear, clean, and accurate without leaking implementation details (of the virt driver) out of the API. | |
| 14:39:15 | bauzas | jaypipes: I couldn't agree more, but at the moment, something could trigger a rescheduler while it could be avoided if we would ask for more | |
| 14:39:23 | mriedem | so when the filter scheduler does the GET /resource_providers?resources=VCPUS:2,MEMORY_MB=4096,DISK_GB=60, it looks like it eventually joins and takes into account allocations on each provider, right? | |
| 14:39:25 | mriedem | i'm lost in the sql | |
| 14:39:34 | cdent | mriedem: that is correct | |
| 14:39:35 | bauzas | mriedem: you're correct | |
| 14:39:38 | mriedem | ok | |
| 14:39:40 | mriedem | thanks, | |
| 14:39:55 | jaypipes | mriedem: yes, of course. | |
| 14:40:01 | mriedem | so, it seems https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should then be reporting the full allocation that we actually claimed on the compute node, right? | |
| 14:40:09 | bauzas | so we underestimate the already taken memory | |
| 14:40:14 | mriedem | correct | |
| 14:40:30 | mriedem | and we could pike a RP in the scheduler that fails the claim test on the host | |
| 14:40:40 | mriedem | because maybe we shaved off enough overheads in our allocations reporting that we screwed ourselves | |
| 14:40:46 | mriedem | worst case that just triggers a retry | |