| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-27 | |||
| 13:27:27 | bauzas | I'm not seeing the reserved bit to be that dynamicv | |
| 13:27:31 | mriedem | the operator can adjust reserved space dynamically | |
| 13:27:35 | johnthetubaguy | cdent: I separate allocations are easier to update than a single reserved value | |
| 13:27:42 | efried | I've heard tell (possibly from jay) that it's a no-no to update the *total* amounts. | |
| 13:27:44 | mriedem | we talked about this in boston | |
| 13:27:56 | bauzas | but yeah, the operator can just set or raise the reserved bit when they want | |
| 13:27:58 | mriedem | about how we don't account for overhead, and operators would have to handle that by toggling reserved values | |
| 13:28:23 | cdent | mriedem: yes, but the new ingredient here is the dynamism | |
| 13:28:25 | cdent | maybe | |
| 13:28:27 | bauzas | mriedem: do you think we should clearly state that in https://docs.openstack.org/nova/latest/contributor/project-scope.html ? | |
| 13:28:40 | efried | ...unless, like, they hot-plug a whole new disk, or jack up the disk size on the SAN or whatever. In that case, we should be able to bump the totals, yes? | |
| 13:28:43 | bauzas | the fact that Nova doesn't support direct hypervisor VMs, but can leave room for them | |
| 13:28:49 | cdent | efried: yes | |
| 13:28:56 | efried | Same actually applies to CPUs | |
| 13:29:30 | efried | Not sure if more primitive hypervisors have this, but there's a dynamic entitlement thing where you can unlock previously-unavailable CPUs on the fly. | |
| 13:29:35 | johnthetubaguy | bhagyashris: there was a -1 review on that patch with no answer when I last looked, so I skipped looking any deeper, did you respond to their questions yet? | |
| 13:30:28 | johnthetubaguy | mriedem: it totally feels like the toggling reserved values ticks the 80% case, with very few surpizes | |
| 13:31:02 | cdent | efried, mriedem, bauzas, johnthetubaguy, rgerganov: it feels like we probably have the tooling to make something workable for this, but as we figure it out it would be good to document some kind of (hate this phrase) best practice, | |
| 13:31:32 | efried | Me, I'm concerned about the penalty of getting the full list of instances every time get_inventory is run. | |
| 13:32:35 | johnthetubaguy | its feels like making the 80% case work really well should be step 1? And some rough best practices for the other bits for now, that could work. | |
| 13:33:01 | cdent | efried: instances from the hypervisor’s point of view, nova’s point of view or placement’s point of view? or some combination? That is, which one worries you most? | |
| 13:33:18 | johnthetubaguy | honestly doing both of those worry me | |
| 13:33:46 | efried | Yeah, I have to do both. | |
| 13:33:47 | johnthetubaguy | I like the system doing nothing when its idle, in an ideal world of course. | |
| 13:33:58 | bauzas | we shouldn't really create a way where people would directly call their VMs if they wish, because that would mean we would silently say we support that | |
| 13:34:20 | efried | Cause the alternative is maintaining some kind of cache and trying to keep it up to date every time a VM gets spawned or deleted (I can get notifications for the OOB ones, and update the in-band ones from spawn/destroy). | |
| 13:34:38 | bhagyashris | johnthetubaguy: Actually I have not respond yet, but Andrey Volkov is given -1 and he suggested that we can fix this issue at schema level (same thing I have proposed in the patch set 3) but after some discussion we decided to fix this in logic and at schema level. | |
| 13:34:44 | bauzas | if the operator doesn't know the dedicated amount their wanna reserve, then either they should disable some compute from the pool, or just set a high value | |
| 13:35:18 | johnthetubaguy | but isn't the easy case much easier here? deployer sets the reserved values, no period updates, worry about special cases later? | |
| 13:35:19 | bauzas | providing a way to dynamically adjust resources in Nova would just mean we create things to support | |
| 13:35:23 | cdent | johnthetubaguy, bauzas: perhaps sad, but true, there are in tree hypervisors where oob vms are fairly common. nova in itself may not need to support that, but the hypervisors will likely continue doing it | |
| 13:35:25 | bhagyashris | s/and at schema level/and not at schema level | |
| 13:35:50 | bauzas | johnthetubaguy: that's my point | |
| 13:36:08 | dansmith | mriedem: have you seen the stuff I added to the etherpad last night? | |
| 13:36:22 | bauzas | johnthetubaguy: if operators really want a space for out-of-band VMs, then they have the reserved bit they can set and try to modify when they feel necessary | |
| 13:36:43 | bhagyashris | johnthetubaguy: i will respond to that question. | |
| 13:36:54 | bauzas | but trying to provide some mechanism for having a dynamic reserved bit seems to me something we shouldn't do | |
| 13:36:54 | johnthetubaguy | cdent: but even those drivers, there are folks who run dedicated clusters just for OpenStack | |
| 13:37:41 | bauzas | and honestly, wouldn't it be better to just disable a host and play with it, if that's really needed ? | |
| 13:37:44 | cdent | johnthetubaguy: yes, totally | |
| 13:37:47 | johnthetubaguy | cdent: that was me arguing for making the sync optional, for when you know Nova owns the world | |
| 13:37:58 | rgerganov | johnthetubaguy, most users runs dedicated clusters but also used shared storage | |
| 13:38:07 | dansmith | johnthetubaguy: are you talking about the periodic that will try to delete unknown vms? | |
| 13:38:09 | mriedem | dansmith: sure did | |
| 13:38:14 | johnthetubaguy | bhagyashris: thanks, responding in the review will help wehn I get back to that | |
| 13:38:21 | mriedem | dansmith: nope, talking about update_available_resource | |
| 13:38:41 | johnthetubaguy | dansmith: well, I was thinking about that early, with all the logging by default, but there I was meaning the resource tracker updater | |
| 13:38:41 | mriedem | and using get_inventory to report reserved space for other things running on the hypervisor outside of nova's control | |
| 13:39:10 | dansmith | mriedem: eff that. there's a reserved count for that kind of stuff. | |
| 13:39:17 | bhagyashris | johnthetubaguy: Thank you :) | |
| 13:39:34 | mriedem | dansmith: meaning the config option? | |
| 13:39:48 | dansmith | meaning counting other things and subtracting it ourselves | |
| 13:40:02 | mriedem | i think the config option is what's used to report reserved inventory in the RT today | |
| 13:40:13 | dansmith | yes | |
| 13:40:33 | dansmith | well, actually, I dunno that it is today, but it should be, and that should be your outlet to carve out space | |
| 13:41:12 | dansmith | it is | |
| 13:41:17 | dansmith | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L102-L125 | |
| 13:41:39 | dansmith | make those hot-reloadable if you want, but that's the path, IMHO | |
| 13:42:58 | efried | dansmith That guy still gets run on a periodic basis, right? I.e. dynamic tweaking of reserved amounts is supported (at least by the driver - not asking if those conf opts are dynamic) | |
| 13:43:31 | johnthetubaguy | we should make SIG_HUP trigger a resource refresh? | |
| 13:43:46 | dansmith | johnthetubaguy: HUP already triggers a config re-read | |
| 13:43:54 | johnthetubaguy | I was meaning a resource tracker update | |
| 13:44:03 | dansmith | johnthetubaguy: so if they were reloadable you'd get a new value the next time inventory runs | |
| 13:44:07 | johnthetubaguy | so we don't have to keep the periodic thing, just do it on events | |
| 13:44:30 | dansmith | I'm not sure there's any reason not to do it perioically | |
| 13:44:53 | johnthetubaguy | system load, all that lists of instances, etc | |
| 13:45:28 | dansmith | it doesn't need to anymore | |
| 13:46:00 | dansmith | in pike, once there aren't any ocata computes around we're not doing healing for instances other than deleted ones | |
| 13:46:31 | johnthetubaguy | I guess I am confused | |
| 13:47:24 | johnthetubaguy | I thought we were left with no periodic updates in queens | |
| 13:47:31 | johnthetubaguy | to inventory, etc | |
| 13:47:44 | johnthetubaguy | although... ironic needs those | |
| 13:47:52 | johnthetubaguy | dang it | |
| 13:48:19 | dansmith | I'm saying, we can remove a bunch of the RT stuff for healing instances now I think, | |
| 13:48:34 | johnthetubaguy | ah, OK, so maybe I am agreeing with you | |
| 13:48:35 | dansmith | which will make the periodic mostly an inventory mechanism, which doesn't need to be listing instances and such | |
| 13:49:32 | efried | The periodic will use driver.get_inventory() to update inventories? | |
| 13:49:57 | efried | (still) | |
| 13:50:04 | dansmith | or get_available_resource() if it doesn't have that | |
| 13:50:09 | efried | Cool. | |
| 13:50:10 | dansmith | or whatever that method is called | |
| 13:50:49 | efried | So I think the point is that, in order to do dynamic adjustment of 'reserved' amounts properly, get_inventory itself may need to be listing instances and such. | |
| 13:50:59 | dansmith | no | |
| 13:51:07 | dansmith | inventory has nothing to do with instances | |
| 13:51:42 | rgerganov | dansmith, there could be no way to calculate reserve without listing instances | |
| 13:52:06 | dansmith | reserved is a conf value | |
| 13:52:44 | bauzas | that's my whole point | |
| 13:52:46 | dansmith | you all were talking about other vms not owned by nova on a hypervisor right? and something external adding that value to reserved so nova doesn't try to use it, right? | |
| 13:52:53 | bauzas | reserved is set by the operator | |
| 13:53:08 | bauzas | so, if the operator wants to raise that value, fair enough | |
| 13:53:18 | dansmith | in that case, some other thing could be updating CONF.reserved_whatever and HUPing nova so that the next time the periodic runs, it just stabs the new scalar value into inventory.. no counting required. | |
| 13:53:19 | bauzas | but I don't see why Nova should care of thaty | |
| 13:53:24 | dansmith | bauzas: most definitely | |
| 13:53:26 | bauzas | dansmith: +1 | |
| 13:55:10 | mriedem | bauzas: in case you didn't see this https://bugs.launchpad.net/nova/+bug/1719730 | |
| 13:55:12 | openstack | Launchpad bug 1719730 in OpenStack Compute (nova) pike "Reschedule after the late affinity check fails with "'NoneType' object is not iterable"" [High,Confirmed] | |
| 13:55:19 | mriedem | bauzas: melwitt might be working on a fix, i'm not sure | |
| 13:55:19 | efried | As user experiences go, "Extend SAN disk, edit conf file, HUP n-cpu" ain't as friendly as "Extend SAN disk". | |
| 13:55:24 | mriedem | bauzas: but it's a regression in pike | |
| 13:55:32 | bauzas | mriedem: yup, I just provided my thoughts litterally 5 mins ago :p | |