| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 14:01:44 | mriedem | and there is no scheduler local to the cell, | |
| 14:01:52 | mriedem | so the local conductor can't ask the scheduler for a new list of hosts to retry | |
| 14:02:21 | mriedem | it would have to upcall to the top-level scheduler (the only scheduler), and i think we're avoid upcalls, because that's like cells v1 | |
| 14:02:35 | dansmith | it's not just that we want to avoid them (we do), | |
| 14:02:39 | openstackgerrit | Andy McCrae proposed openstack/nova master: Allow CONTENT_LENGTH to be present but empty https://review.openstack.org/455710 | |
| 14:02:42 | dansmith | but we have no connection information for the upcall | |
| 14:03:45 | mriedem | dansmith: does the cell conductor nova.conf have api_database set? | |
| 14:03:52 | dansmith | no | |
| 14:04:01 | mriedem | ok, just [database]connection = cell db | |
| 14:04:04 | edleafe | So the cell conductor talk to the super conductor? | |
| 14:04:15 | mriedem | edleafe: no | |
| 14:04:24 | dansmith | edleafe: they're on different busses, and the lower layer has no information about how to connect to the top layer | |
| 14:05:17 | mriedem | jaypipes: so for the claims in scheduler spec and the overhead stuff, couldn't we take the overhead values from the virt driver and adjust the allocations in the resource tracker? | |
| 14:05:33 | mriedem | if the overhead pushes us over what's available for inventory on that provider, we get a 409 i think, and then we fail the build | |
| 14:06:30 | dansmith | ugh | |
| 14:06:34 | mriedem | i would think if you had reserved some space for ram and disk for the provider itself, then the chances of hitting overhead 409s once we pick a host might be slim | |
| 14:06:36 | dansmith | I really hate that | |
| 14:06:43 | mriedem | dansmith: see my other options in the spec :) | |
| 14:06:47 | mriedem | there are worse options | |
| 14:07:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add online data migration for populating services.uuid https://review.openstack.org/454899 | |
| 14:10:12 | jaypipes | mriedem: yes, we could possibly do that | |
| 14:10:40 | jaypipes | mriedem: I don't like it, same as dansmith, but not sure a better solution | |
| 14:10:59 | dansmith | so the point being, | |
| 14:11:04 | mriedem | i figured that would be better than conductor doing an rpc call into the compute to get the overhead values before posting allocations | |
| 14:11:18 | dansmith | we tell people to make sure they account for the slack in the reserved amounts on each host to keep the chances of a failure down? | |
| 14:11:25 | mriedem | dansmith: yeah | |
| 14:11:27 | mriedem | it sucks | |
| 14:11:28 | mriedem | i know | |
| 14:11:42 | dansmith | the conductor->compute call isn't so terrible, | |
| 14:11:52 | dansmith | it's just that we can't make it until we have a compute | |
| 14:11:55 | mriedem | right | |
| 14:11:55 | dansmith | which lengthens our window | |
| 14:12:05 | cdent | if we wiggle the reserved then we can't add the overhead to the allocations, right? | |
| 14:12:17 | cdent | it's one or the other | |
| 14:12:21 | mriedem | so conductor asks scheduler for a host, conductor gets the host mapping, rpc calls to the compute with the flavor to get the overhead, and uses that to post the allocations to placement, | |
| 14:12:28 | mriedem | if that fails, we have to go back through the scheduler again for a new host | |
| 14:12:36 | dansmith | cdent: hmm, yeah, placement won't let you eat the reserved amount I guess right? | |
| 14:12:42 | cdent | dansmith: correct | |
| 14:12:45 | dansmith | yeah | |
| 14:13:37 | cdent | I left a big comment there about quota too, not sure if/how that fits in | |
| 14:13:41 | mriedem | i liked the idea of posting the allocations from the scheduler where if we needed to retry we already have the list of filtered hosts, but we know that's a toss up perf wise | |
| 14:14:01 | mriedem | if we have to rpc call into the compute to get overhead, i think we then have to move everything into conductor | |
| 14:14:08 | dansmith | mriedem: well, and scheduler calling to compute is worse than conductor | |
| 14:14:09 | dansmith | much worse | |
| 14:14:10 | dansmith | yeah | |
| 14:15:06 | dansmith | so, another way to look at this: | |
| 14:15:24 | dansmith | we're talking about having resource overrides in the flavor | |
| 14:15:53 | dansmith | we could also just say that the drivers can't calculate different values for the resources, and have to build the vm from the things they're given, | |
| 14:16:00 | bauzas | sorry, was afk | |
| 14:16:08 | dansmith | which in the case of hyperv means you get $mem fewer MBs of disk, | |
| 14:16:13 | dansmith | and in the case of libvirt, you get one fewer vcpu | |
| 14:16:34 | dansmith | and if you don't want that as an operator, then you override the actual values for those with room for the slack, leaving the display values as what people will really get | |
| 14:16:56 | dansmith | for people that really use the same flavor for two hypervisor types (if that really happens) then things get ugly | |
| 14:17:18 | dansmith | but I kinda feel like the cloudy approach here is to not be as snowflake-y on the virt side | |
| 14:17:26 | bauzas | could someone tell which problem you folks are discussing ? #1 reschedules in a cellsv2 world or #2 overhead values for instance given by virt drivers ? | |
| 14:17:31 | mriedem | that also pushes a lot of complexity on the operator | |
| 14:17:37 | dansmith | it does, and that sucks | |
| 14:17:39 | mriedem | bauzas: #2 | |
| 14:18:15 | dansmith | mriedem: so, we should probably go the extra-call-to-compute route for now and if that ends up being too expensive, then punt later | |
| 14:18:31 | dansmith | we already do that for things like getting predicted-but-never-correct bdm device names right? | |
| 14:18:37 | mriedem | yeah, we do that from the api | |
| 14:18:42 | bauzas | so, my take on that is that overheads, since are only virt-related are things which wouldn't be exposed to users | |
| 14:18:43 | mriedem | when attaching a volume | |
| 14:18:54 | dansmith | so here's another potential optimization: | |
| 14:18:57 | bauzas | users ask for memory sizes they know | |
| 14:19:10 | bauzas | they don't know how much the hypervisor has | |
| 14:19:14 | bauzas | so | |
| 14:19:25 | dansmith | we could do that call once, and record the value we got for the flavor id, hypervisor type and version, and only do that once per boot if we have to try several hosts in a single cycle | |
| 14:19:53 | mriedem | oh btw, this is nice in the py35 tests: "Fatal Python error: Cannot recover from stack overflow." | |
| 14:19:54 | dansmith | mriedem: like, if we have to try four libvirt hosts in a single boot call, we only have to call to one of the computes to get the overhead | |
| 14:19:56 | mriedem | sdague: ^ | |
| 14:20:00 | bauzas | when the scheduler is choosing a destination, he has to pick it based on the user request, but related to the total amount of memory is left | |
| 14:20:21 | mriedem | dansmith: yeah that's not a bad idea | |
| 14:20:27 | mriedem | cache the results in conductor | |
| 14:20:36 | mriedem | take the rpc call hit once | |
| 14:20:38 | dansmith | we need more information than just the hostname coming back from scheduler though, else we have to hit the db ourselves to look up the host info again | |
| 14:20:43 | dansmith | yeah | |
| 14:21:02 | dansmith | either once per call or once per conductor worker | |
| 14:21:06 | dansmith | or memcache or something | |
| 14:21:15 | mriedem | i assumed once per worker | |
| 14:21:22 | dansmith | well, that could get big, | |
| 14:21:33 | dansmith | as there are tons of workers, and potentially lots of flavors and stuff | |
| 14:21:33 | mriedem | sure, if we use the cache utils stuff, | |
| 14:21:35 | mriedem | it's configurable | |
| 14:21:41 | dansmith | yeah, that's the way to go I think | |
| 14:21:43 | mriedem | default is per worker in memory cache | |
| 14:21:48 | mriedem | if you want memcache, configure it | |
| 14:21:50 | dansmith | yeah | |
| 14:22:02 | bauzas | wait | |
| 14:22:30 | mriedem | gcb: i'm seeing "Fatal Python error: Cannot recover from stack overflow." in the py35 job for nova unit tests randomly | |
| 14:22:30 | bauzas | why placement couldn't compare the user flavor vs. the size of what's left, ie. inventory - instances ? | |
| 14:22:35 | mriedem | gcb: http://logs.openstack.org/97/456397/3/check/gate-nova-python35/05322ec/console.html#_2017-04-19_12_45_10_724578 | |
| 14:22:47 | bauzas | instances being "instance.flavor + instance.overhead" ? | |
| 14:22:51 | mriedem | bauzas: placement doesn't have the overhead info | |
| 14:23:05 | bauzas | mriedem: placement knows inventory I know | |
| 14:23:16 | mriedem | bauzas: we're saying we need to get the overhead from the compute before we post the allocations | |
| 14:23:20 | mriedem | to take into account the overhead | |
| 14:23:33 | mriedem | then placement tells us if that works or not for the inventory on the provider | |
| 14:23:44 | mriedem | if POST /allocations/instance fails, then we retry through the scheduler | |
| 14:24:06 | mriedem | dan was just saying that we can cache the rpc call results in conductor - the results of the call to the compute to get the overhead | |