| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 14:27:31 | cdent | If we know that an average use scenaior of an inventory that looks like X needs a multiplier on some scale... | |
| 14:27:32 | bauzas | tbh, looks very dynamic to me, since it's provided by the virt driver | |
| 14:27:45 | mriedem | right, i don't know how we encode this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L713 | |
| 14:27:47 | bauzas | I mean, are we sure the overhead can mathematically be given ? | |
| 14:29:18 | dansmith | mriedem: the libvirt one is not that bad I think, but the hyperv one is | |
| 14:29:49 | jaypipes | mriedem: I think for that hyperv one, it may just be that the DISK_GB inventory record on hyperv compute nodes needs a pre-set reserved value :( | |
| 14:29:59 | dansmith | although the libvirt one would need to be able to specify a trait that increases the cpu overhead by one, I guess | |
| 14:30:16 | cdent | /o\ | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791 | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: Update resource tracker to PUT custom resource classes https://review.openstack.org/456915 | |
| 14:30:59 | mriedem | bauzas: going back to your auto-heal, are you referring to https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 which eventually calls _update_usage? | |
| 14:31:12 | mriedem | and then later we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L648 | |
| 14:31:36 | jaypipes | mriedem: for the emulator threads, I think we can handle that differently... | |
| 14:31:54 | bauzas | mriedem: I'm saying we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L774 | |
| 14:32:15 | bauzas | because indeed we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 | |
| 14:32:27 | jaypipes | mriedem: for libvirt compute nodes, we'd have a fixed number of EMULATOR_THREADS inventory, with no overcommit. For other drivers, we'd have a huge overcommit of EMULATOR_THREADS inventory. | |
| 14:32:30 | mriedem | bauzas: right _update_usage_from_instance calls _update_usage | |
| 14:33:02 | bauzas | mriedem: so, _update_usage_from_instance calls placement by not adding the overhead | |
| 14:33:02 | mriedem | bauzas: but _update just changes the inventory record in placement | |
| 14:33:06 | mriedem | and the total doesn't change does it? | |
| 14:33:30 | bauzas | mriedem: but _update updates the inventory with the values it was updated by _update_usage() | |
| 14:34:13 | mriedem | yes, but, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L779-L795 | |
| 14:34:18 | mriedem | only sets the used/free values on the cn | |
| 14:34:25 | mriedem | which aren't used when setting the inventory in placement for that provider | |
| 14:34:34 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L117 | |
| 14:34:56 | mriedem | the inventory value comes from driver.get_available_resource/get_inventory, right? | |
| 14:35:01 | bauzas | mriedem: oh fuck, you're right | |
| 14:35:02 | jaypipes | yes | |
| 14:35:05 | mriedem | YES | |
| 14:35:09 | mriedem | ha, ok | |
| 14:35:10 | mriedem | SO... | |
| 14:35:20 | bauzas | it's just updating memory_used | |
| 14:35:24 | mriedem | going back to https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:35:25 | bauzas | so, not called to placement | |
| 14:35:26 | mriedem | ^ is wrong | |
| 14:35:31 | bauzas | mriedem: yup | |
| 14:35:47 | bauzas | mriedem: in a perfect world, we should add that to placement | |
| 14:35:53 | mriedem | let me look up the voodoo that placement does when scheduler asks for providers for a given reqspec | |
| 14:35:55 | bauzas | mriedem: because RamFilter does that | |
| 14:36:06 | bauzas | and we want placement to behave like the filter | |
| 14:38:15 | jaypipes | bauzas: I don't care if placement doesn't behave exactly like the RamFilter. I just want placement to be clear, clean, and accurate without leaking implementation details (of the virt driver) out of the API. | |
| 14:39:15 | bauzas | jaypipes: I couldn't agree more, but at the moment, something could trigger a rescheduler while it could be avoided if we would ask for more | |
| 14:39:23 | mriedem | so when the filter scheduler does the GET /resource_providers?resources=VCPUS:2,MEMORY_MB=4096,DISK_GB=60, it looks like it eventually joins and takes into account allocations on each provider, right? | |
| 14:39:25 | mriedem | i'm lost in the sql | |
| 14:39:34 | cdent | mriedem: that is correct | |
| 14:39:35 | bauzas | mriedem: you're correct | |
| 14:39:38 | mriedem | ok | |
| 14:39:40 | mriedem | thanks, | |
| 14:39:55 | jaypipes | mriedem: yes, of course. | |
| 14:40:01 | mriedem | so, it seems https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should then be reporting the full allocation that we actually claimed on the compute node, right? | |
| 14:40:09 | bauzas | so we underestimate the already taken memory | |
| 14:40:14 | mriedem | correct | |
| 14:40:30 | mriedem | and we could pike a RP in the scheduler that fails the claim test on the host | |
| 14:40:40 | mriedem | because maybe we shaved off enough overheads in our allocations reporting that we screwed ourselves | |
| 14:40:46 | mriedem | worst case that just triggers a retry | |
| 14:40:50 | mriedem | UNTIL | |
| 14:40:53 | mriedem | we don't have retries anymore | |
| 14:40:54 | bauzas | yup, retry | |
| 14:40:56 | mriedem | see where i'm going :) | |
| 14:41:01 | bauzas | I know | |
| 14:41:14 | mriedem | jaypipes: bauzas: cdent: can we at least agree this is a bug? | |
| 14:41:24 | jaypipes | mriedem: sure :) | |
| 14:41:28 | mriedem | ok | |
| 14:41:28 | bauzas | I think we can reasonably assume it is a nice one | |
| 14:41:30 | mriedem | good | |
| 14:41:38 | cdent | mriedem: yes, there is a mismatch between reality and what's recorded | |
| 14:41:51 | mriedem | because i started this morning reading this email from a perf guy in china, not knowing anything about this overhead stuff, to realizing there is a bug, | |
| 14:41:57 | jaypipes | cdent: that is the plotline of my entire life. | |
| 14:42:03 | bauzas | meiwelcome in the club | |
| 14:42:07 | bauzas | mriedem: welcome in the club | |
| 14:42:07 | mriedem | so i'd like to take that back to him instead of just saying, 'i have no f'ing clue what you're talking about' | |
| 14:42:08 | openstackgerrit | Gergely Csatari proposed openstack/nova master: Checking the parameters of servers-actions.inc https://review.openstack.org/327112 | |
| 14:42:18 | cdent | jaypipes: I think there's a good song in there somewhere | |
| 14:42:28 | sfinucan | jaypipes: I see https://review.openstack.org/#/c/361140/ didn't get in before the spec freeze. Think it's a fair candidate for the exception process? | |
| 14:42:32 | mriedem | cdent: every country western song ever | |
| 14:42:45 | mriedem | sfinucan: not imo | |
| 14:43:01 | cdent | mriedem: need to add some tears in my ears from crying on my pillow over you | |
| 14:43:04 | jaypipes | sfinucan: not up to me. but I will review today regardless. | |
| 14:43:08 | sfinucan | mriedem: no? | |
| 14:43:17 | bauzas | oh, sfinucan reminded me one | |
| 14:43:18 | sfinucan | any particular reason? | |
| 14:43:48 | mriedem | sfinucan: because at this point we're overcommitted and the only exceptions i'm interested in are for priorities or things that impact all of nova, not narrow use cases | |
| 14:44:02 | bauzas | mriedem: https://review.openstack.org/#/c/450122/18 is getting 2 +2s from jaypipes and me but I refrained my +W, so I leave you use your axe | |
| 14:47:42 | mriedem | i'm -2 on the vgpus one also | |
| 14:47:44 | mriedem | same reason | |
| 14:47:48 | sfinucan | mriedem: Those use cases are the primary use cases for a lot of our customers | |
| 14:48:04 | sfinucan | the VGPU less so. That needs more exploration, IMO | |
| 14:48:08 | mriedem | sfinucan: then tell me which of the other red hat priority blueprints you'd like to defer in it's place | |
| 14:48:19 | mriedem | emulator threads? | |
| 14:48:23 | mriedem | secure console proxy? | |
| 14:48:28 | mriedem | your other pci scheduler one? | |
| 14:48:38 | sfinucan | emulator threads is done | |
| 14:49:06 | sfinucan | the other PCI scheduler is a two patch change, ~100 line change | |
| 14:49:54 | sfinucan | and, if Gerrit's logs are anything to go by, I'm going to be very fortunate to get the secure console proxy stuff in. No one wants to review it :( | |
| 14:50:19 | sfinucan | the PCI use cases are my primary focus this cycle. There's no lack of reviewer bandwidth on my end | |
| 14:56:50 | openstackgerrit | Peter Hamilton proposed openstack/nova master: Add configuration options for certificate validation https://review.openstack.org/457678 | |
| 14:56:51 | mriedem | sfinucan: you can work with artom on the tagged attach/detach stuff then | |
| 14:57:03 | mriedem | sfinucan: honestly look at what we've already approved https://blueprints.launchpad.net/nova/pike | |
| 14:57:47 | voelzmo | @johnthetubaguy I've updated https://review.openstack.org/#/c/416997 with some comments, not sure how we should continue the discussion | |
| 14:59:13 | johnthetubaguy | voelzmo: here is fine, for me it was a case of consistency | |