| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-18 | |||
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791 | |
| 14:30:33 | openstackgerrit | Chris Dent proposed openstack/nova master: Update resource tracker to PUT custom resource classes https://review.openstack.org/456915 | |
| 14:30:59 | mriedem | bauzas: going back to your auto-heal, are you referring to https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 which eventually calls _update_usage? | |
| 14:31:12 | mriedem | and then later we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L648 | |
| 14:31:36 | jaypipes | mriedem: for the emulator threads, I think we can handle that differently... | |
| 14:31:54 | bauzas | mriedem: I'm saying we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L774 | |
| 14:32:15 | bauzas | because indeed we call https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L616 | |
| 14:32:27 | jaypipes | mriedem: for libvirt compute nodes, we'd have a fixed number of EMULATOR_THREADS inventory, with no overcommit. For other drivers, we'd have a huge overcommit of EMULATOR_THREADS inventory. | |
| 14:32:30 | mriedem | bauzas: right _update_usage_from_instance calls _update_usage | |
| 14:33:02 | bauzas | mriedem: so, _update_usage_from_instance calls placement by not adding the overhead | |
| 14:33:02 | mriedem | bauzas: but _update just changes the inventory record in placement | |
| 14:33:06 | mriedem | and the total doesn't change does it? | |
| 14:33:30 | bauzas | mriedem: but _update updates the inventory with the values it was updated by _update_usage() | |
| 14:34:13 | mriedem | yes, but, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L779-L795 | |
| 14:34:18 | mriedem | only sets the used/free values on the cn | |
| 14:34:25 | mriedem | which aren't used when setting the inventory in placement for that provider | |
| 14:34:34 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L117 | |
| 14:34:56 | mriedem | the inventory value comes from driver.get_available_resource/get_inventory, right? | |
| 14:35:01 | bauzas | mriedem: oh fuck, you're right | |
| 14:35:02 | jaypipes | yes | |
| 14:35:05 | mriedem | YES | |
| 14:35:09 | mriedem | ha, ok | |
| 14:35:10 | mriedem | SO... | |
| 14:35:20 | bauzas | it's just updating memory_used | |
| 14:35:24 | mriedem | going back to https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 | |
| 14:35:25 | bauzas | so, not called to placement | |
| 14:35:26 | mriedem | ^ is wrong | |
| 14:35:31 | bauzas | mriedem: yup | |
| 14:35:47 | bauzas | mriedem: in a perfect world, we should add that to placement | |
| 14:35:53 | mriedem | let me look up the voodoo that placement does when scheduler asks for providers for a given reqspec | |
| 14:35:55 | bauzas | mriedem: because RamFilter does that | |
| 14:36:06 | bauzas | and we want placement to behave like the filter | |
| 14:38:15 | jaypipes | bauzas: I don't care if placement doesn't behave exactly like the RamFilter. I just want placement to be clear, clean, and accurate without leaking implementation details (of the virt driver) out of the API. | |
| 14:39:15 | bauzas | jaypipes: I couldn't agree more, but at the moment, something could trigger a rescheduler while it could be avoided if we would ask for more | |
| 14:39:23 | mriedem | so when the filter scheduler does the GET /resource_providers?resources=VCPUS:2,MEMORY_MB=4096,DISK_GB=60, it looks like it eventually joins and takes into account allocations on each provider, right? | |
| 14:39:25 | mriedem | i'm lost in the sql | |
| 14:39:34 | cdent | mriedem: that is correct | |
| 14:39:35 | bauzas | mriedem: you're correct | |
| 14:39:38 | mriedem | ok | |
| 14:39:40 | mriedem | thanks, | |
| 14:39:55 | jaypipes | mriedem: yes, of course. | |
| 14:40:01 | mriedem | so, it seems https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L156 should then be reporting the full allocation that we actually claimed on the compute node, right? | |
| 14:40:09 | bauzas | so we underestimate the already taken memory | |
| 14:40:14 | mriedem | correct | |
| 14:40:30 | mriedem | and we could pike a RP in the scheduler that fails the claim test on the host | |
| 14:40:40 | mriedem | because maybe we shaved off enough overheads in our allocations reporting that we screwed ourselves | |
| 14:40:46 | mriedem | worst case that just triggers a retry | |
| 14:40:50 | mriedem | UNTIL | |
| 14:40:53 | mriedem | we don't have retries anymore | |
| 14:40:54 | bauzas | yup, retry | |
| 14:40:56 | mriedem | see where i'm going :) | |
| 14:41:01 | bauzas | I know | |
| 14:41:14 | mriedem | jaypipes: bauzas: cdent: can we at least agree this is a bug? | |
| 14:41:24 | jaypipes | mriedem: sure :) | |
| 14:41:28 | mriedem | ok | |
| 14:41:28 | bauzas | I think we can reasonably assume it is a nice one | |
| 14:41:30 | mriedem | good | |
| 14:41:38 | cdent | mriedem: yes, there is a mismatch between reality and what's recorded | |
| 14:41:51 | mriedem | because i started this morning reading this email from a perf guy in china, not knowing anything about this overhead stuff, to realizing there is a bug, | |
| 14:41:57 | jaypipes | cdent: that is the plotline of my entire life. | |
| 14:42:03 | bauzas | meiwelcome in the club | |
| 14:42:07 | bauzas | mriedem: welcome in the club | |
| 14:42:07 | mriedem | so i'd like to take that back to him instead of just saying, 'i have no f'ing clue what you're talking about' | |
| 14:42:08 | openstackgerrit | Gergely Csatari proposed openstack/nova master: Checking the parameters of servers-actions.inc https://review.openstack.org/327112 | |
| 14:42:18 | cdent | jaypipes: I think there's a good song in there somewhere | |
| 14:42:28 | sfinucan | jaypipes: I see https://review.openstack.org/#/c/361140/ didn't get in before the spec freeze. Think it's a fair candidate for the exception process? | |
| 14:42:32 | mriedem | cdent: every country western song ever | |
| 14:42:45 | mriedem | sfinucan: not imo | |
| 14:43:01 | cdent | mriedem: need to add some tears in my ears from crying on my pillow over you | |
| 14:43:04 | jaypipes | sfinucan: not up to me. but I will review today regardless. | |
| 14:43:08 | sfinucan | mriedem: no? | |
| 14:43:17 | bauzas | oh, sfinucan reminded me one | |
| 14:43:18 | sfinucan | any particular reason? | |
| 14:43:48 | mriedem | sfinucan: because at this point we're overcommitted and the only exceptions i'm interested in are for priorities or things that impact all of nova, not narrow use cases | |
| 14:44:02 | bauzas | mriedem: https://review.openstack.org/#/c/450122/18 is getting 2 +2s from jaypipes and me but I refrained my +W, so I leave you use your axe | |
| 14:47:42 | mriedem | i'm -2 on the vgpus one also | |
| 14:47:44 | mriedem | same reason | |
| 14:47:48 | sfinucan | mriedem: Those use cases are the primary use cases for a lot of our customers | |
| 14:48:04 | sfinucan | the VGPU less so. That needs more exploration, IMO | |
| 14:48:08 | mriedem | sfinucan: then tell me which of the other red hat priority blueprints you'd like to defer in it's place | |
| 14:48:19 | mriedem | emulator threads? | |
| 14:48:23 | mriedem | secure console proxy? | |
| 14:48:28 | mriedem | your other pci scheduler one? | |
| 14:48:38 | sfinucan | emulator threads is done | |
| 14:49:06 | sfinucan | the other PCI scheduler is a two patch change, ~100 line change | |
| 14:49:54 | sfinucan | and, if Gerrit's logs are anything to go by, I'm going to be very fortunate to get the secure console proxy stuff in. No one wants to review it :( | |
| 14:50:19 | sfinucan | the PCI use cases are my primary focus this cycle. There's no lack of reviewer bandwidth on my end | |
| 14:56:50 | openstackgerrit | Peter Hamilton proposed openstack/nova master: Add configuration options for certificate validation https://review.openstack.org/457678 | |
| 14:56:51 | mriedem | sfinucan: you can work with artom on the tagged attach/detach stuff then | |
| 14:57:03 | mriedem | sfinucan: honestly look at what we've already approved https://blueprints.launchpad.net/nova/pike | |
| 14:57:47 | voelzmo | @johnthetubaguy I've updated https://review.openstack.org/#/c/416997 with some comments, not sure how we should continue the discussion | |
| 14:59:13 | johnthetubaguy | voelzmo: here is fine, for me it was a case of consistency | |
| 14:59:32 | sfinucan | mriedem: Yeah, I've been horse trading with him for reviews on the secure console proxy stuff, heh :) | |
| 14:59:55 | johnthetubaguy | voelzmo: its a strange API to only support this for half of the nic types, thats really hard API docs and validation, etc | |
| 15:00:09 | johnthetubaguy | voelzmo: but maybe I am seeing it all wrong | |
| 15:00:53 | mriedem | speaking of deleting ports, | |
| 15:00:57 | mriedem | let's fix latent bugs https://review.openstack.org/#/c/452577/ | |
| 15:01:13 | voelzmo | @johnthetubaguy I understand the consistency argument, I just fear that it adds scope that I didn't mean to include. Not sure how special the documentation and code really will get. From what I've seen it is already pretty special in the sense that it sets the 'remove port' flag based on whether nova creates it or the user passed it in | |
| 15:01:21 | sfinucan | mriedem: Also get your point about being oversubscribed, but these are the things I'm paid to care about. If push came to shove, could we drop the secure console proxy work in favour of that | |
| 15:02:13 | johnthetubaguy | voelzmo: but saying you can customise only one of those cases seems strange. | |