| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 14:24:08 | dansmith | mriedem: right, but make it declare how it calculates it in terms of resource classes | |
| 14:24:10 | jaypipes | mriedem: no, it's a factor of the resource class. | |
| 14:24:23 | jaypipes | mriedem: each virt driver has a separate overhead for cpu, disk and memory. | |
| 14:24:24 | dansmith | mriedem: because eventually we get rid of weirdo extra specs and it's all resource classes | |
| 14:24:29 | jaypipes | mriedem: and soon, emulator_threads :) | |
| 14:25:05 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/hyperv/vmops.py#L124 is a simple case | |
| 14:25:10 | jaypipes | BobBall: I think having an overheads table in the placement API containing this information is the best long-term solution. it would make the system data-driven instead of hard-coded in the virt driver. | |
| 14:25:35 | mriedem | so the overheads table contains the multiplier? | |
| 14:25:36 | jaypipes | mriedem: right, that's not per-flavor. it's per reesource class.. | |
| 14:25:41 | mriedem | for a given resource class on that provider? | |
| 14:25:46 | jaypipes | yep | |
| 14:25:53 | dansmith | jaypipes: well, that's crossing the line.. because it's disk = memory + N | |
| 14:25:55 | BobBall | jaypipes: OK - but it's going to be a best guess at any rate. AIUI the actual overhead will depend on the guest type as well as various other factors which we really won't want to encode in any table... | |
| 14:26:01 | dansmith | jaypipes: that one is ... hard | |
| 14:26:14 | mriedem | dansmith: store a lambda in the table?! :P | |
| 14:26:25 | dansmith | mriedem: genius! | |
| 14:27:02 | cdent | Is there some way we can do this algorithmically based on the inventory of the node when it starts up, do a calculation, and simply add that to the reserved? Because this all sounds fragile and constantly in need of adding yet more stuff. | |
| 14:27:31 | cdent | If we know that an average use scenaior of an inventory that looks like X needs a multiplier on some scale... | |
| 14:27:32 | bauzas | tbh, looks very dynamic to me, since it's provided by the virt driver | |
| 14:27:45 | mriedem | right, i don't know how we encode this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L713 | |
| 14:27:47 | bauzas | I mean, are we sure the overhead can mathematically be given ? | |
| 14:29:18 | dansmith | mriedem: the libvirt one is not that bad I think, but the hyperv one is | |
| 14:29:49 | jaypipes | mriedem: I think for that hyperv one, it may just be that the DISK_GB inventory record on hyperv compute nodes needs a pre-set reserved value :( | |
| 14:29:59 | dansmith | although the libvirt one would need to be able to specify a trait that increases the cpu overhead by one, I guess | |
| 14:30:16 | cdent | /o\ | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791 | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: Update resource tracker to PUT custom resource classes https://review.openstack.org/456915 | |
| 14:30:59 | mriedem | bauzas: going back to your auto-heal, are you referring to https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 which eventually calls _update_usage? | |
| 14:31:12 | mriedem | and then later we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L648 | |
| 14:31:36 | jaypipes | mriedem: for the emulator threads, I think we can handle that differently... | |
| 14:31:54 | bauzas | mriedem: I'm saying we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L774 | |
| 14:32:15 | bauzas | because indeed we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 | |
| 14:32:27 | jaypipes | mriedem: for libvirt compute nodes, we'd have a fixed number of EMULATOR_THREADS inventory, with no overcommit. For other drivers, we'd have a huge overcommit of EMULATOR_THREADS inventory. | |
| 14:32:30 | mriedem | bauzas: right _update_usage_from_instance calls _update_usage | |
| 14:33:02 | bauzas | mriedem: so, _update_usage_from_instance calls placement by not adding the overhead | |
| 14:33:02 | mriedem | bauzas: but _update just changes the inventory record in placement | |
| 14:33:06 | mriedem | and the total doesn't change does it? | |
| 14:33:30 | bauzas | mriedem: but _update updates the inventory with the values it was updated by _update_usage() | |
| 14:34:13 | mriedem | yes, but, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L779-L795 | |
| 14:34:18 | mriedem | only sets the used/free values on the cn | |
| 14:34:25 | mriedem | which aren't used when setting the inventory in placement for that provider | |
| 14:34:34 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L117 | |
| 14:34:56 | mriedem | the inventory value comes from driver.get_available_resource/get_inventory, right? | |
| 14:35:01 | bauzas | mriedem: oh fuck, you're right | |
| 14:35:02 | jaypipes | yes | |
| 14:35:05 | mriedem | YES | |
| 14:35:09 | mriedem | ha, ok | |
| 14:35:10 | mriedem | SO... | |
| 14:35:20 | bauzas | it's just updating memory_used | |
| 14:35:24 | mriedem | going back to https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:35:25 | bauzas | so, not called to placement | |
| 14:35:26 | mriedem | ^ is wrong | |
| 14:35:31 | bauzas | mriedem: yup | |
| 14:35:47 | bauzas | mriedem: in a perfect world, we should add that to placement | |
| 14:35:53 | mriedem | let me look up the voodoo that placement does when scheduler asks for providers for a given reqspec | |
| 14:35:55 | bauzas | mriedem: because RamFilter does that | |
| 14:36:06 | bauzas | and we want placement to behave like the filter | |
| 14:38:15 | jaypipes | bauzas: I don't care if placement doesn't behave exactly like the RamFilter. I just want placement to be clear, clean, and accurate without leaking implementation details (of the virt driver) out of the API. | |
| 14:39:15 | bauzas | jaypipes: I couldn't agree more, but at the moment, something could trigger a rescheduler while it could be avoided if we would ask for more | |
| 14:39:23 | mriedem | so when the filter scheduler does the GET /resource_providers?resources=VCPUS:2,MEMORY_MB=4096,DISK_GB=60, it looks like it eventually joins and takes into account allocations on each provider, right? | |
| 14:39:25 | mriedem | i'm lost in the sql | |
| 14:39:34 | cdent | mriedem: that is correct | |
| 14:39:35 | bauzas | mriedem: you're correct | |
| 14:39:38 | mriedem | ok | |
| 14:39:40 | mriedem | thanks, | |
| 14:39:55 | jaypipes | mriedem: yes, of course. | |
| 14:40:01 | mriedem | so, it seems https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should then be reporting the full allocation that we actually claimed on the compute node, right? | |
| 14:40:09 | bauzas | so we underestimate the already taken memory | |
| 14:40:14 | mriedem | correct | |
| 14:40:30 | mriedem | and we could pike a RP in the scheduler that fails the claim test on the host | |
| 14:40:40 | mriedem | because maybe we shaved off enough overheads in our allocations reporting that we screwed ourselves | |
| 14:40:46 | mriedem | worst case that just triggers a retry | |
| 14:40:50 | mriedem | UNTIL | |
| 14:40:53 | mriedem | we don't have retries anymore | |
| 14:40:54 | bauzas | yup, retry | |
| 14:40:56 | mriedem | see where i'm going :) | |
| 14:41:01 | bauzas | I know | |
| 14:41:14 | mriedem | jaypipes: bauzas: cdent: can we at least agree this is a bug? | |
| 14:41:24 | jaypipes | mriedem: sure :) | |
| 14:41:28 | mriedem | ok | |
| 14:41:28 | bauzas | I think we can reasonably assume it is a nice one | |
| 14:41:30 | mriedem | good | |
| 14:41:38 | cdent | mriedem: yes, there is a mismatch between reality and what's recorded | |
| 14:41:51 | mriedem | because i started this morning reading this email from a perf guy in china, not knowing anything about this overhead stuff, to realizing there is a bug, | |
| 14:41:57 | jaypipes | cdent: that is the plotline of my entire life. | |
| 14:42:03 | bauzas | meiwelcome in the club | |
| 14:42:07 | bauzas | mriedem: welcome in the club | |
| 14:42:07 | mriedem | so i'd like to take that back to him instead of just saying, 'i have no f'ing clue what you're talking about' | |
| 14:42:08 | openstackgerrit | Gergely Csatari proposed openstack/nova master: Checking the parameters of servers-actions.inc https://review.openstack.org/327112 | |
| 14:42:18 | cdent | jaypipes: I think there's a good song in there somewhere | |
| 14:42:28 | sfinucan | jaypipes: I see https://review.openstack.org/#/c/361140/ didn't get in before the spec freeze. Think it's a fair candidate for the exception process? | |
| 14:42:32 | mriedem | cdent: every country western song ever | |
| 14:42:45 | mriedem | sfinucan: not imo | |
| 14:43:01 | cdent | mriedem: need to add some tears in my ears from crying on my pillow over you | |
| 14:43:04 | jaypipes | sfinucan: not up to me. but I will review today regardless. | |
| 14:43:08 | sfinucan | mriedem: no? | |
| 14:43:17 | bauzas | oh, sfinucan reminded me one | |
| 14:43:18 | sfinucan | any particular reason? | |
| 14:43:48 | mriedem | sfinucan: because at this point we're overcommitted and the only exceptions i'm interested in are for priorities or things that impact all of nova, not narrow use cases | |
| 14:44:02 | bauzas | mriedem: https://review.openstack.org/#/c/450122/18 is getting 2 +2s from jaypipes and me but I refrained my +W, so I leave you use your axe | |