| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 14:13:41 | mriedem | i liked the idea of posting the allocations from the scheduler where if we needed to retry we already have the list of filtered hosts, but we know that's a toss up perf wise | |
| 14:14:01 | mriedem | if we have to rpc call into the compute to get overhead, i think we then have to move everything into conductor | |
| 14:14:08 | dansmith | mriedem: well, and scheduler calling to compute is worse than conductor | |
| 14:14:09 | dansmith | much worse | |
| 14:14:10 | dansmith | yeah | |
| 14:15:06 | dansmith | so, another way to look at this: | |
| 14:15:24 | dansmith | we're talking about having resource overrides in the flavor | |
| 14:15:53 | dansmith | we could also just say that the drivers can't calculate different values for the resources, and have to build the vm from the things they're given, | |
| 14:16:00 | bauzas | sorry, was afk | |
| 14:16:08 | dansmith | which in the case of hyperv means you get $mem fewer MBs of disk, | |
| 14:16:13 | dansmith | and in the case of libvirt, you get one fewer vcpu | |
| 14:16:34 | dansmith | and if you don't want that as an operator, then you override the actual values for those with room for the slack, leaving the display values as what people will really get | |
| 14:16:56 | dansmith | for people that really use the same flavor for two hypervisor types (if that really happens) then things get ugly | |
| 14:17:18 | dansmith | but I kinda feel like the cloudy approach here is to not be as snowflake-y on the virt side | |
| 14:17:26 | bauzas | could someone tell which problem you folks are discussing ? #1 reschedules in a cellsv2 world or #2 overhead values for instance given by virt drivers ? | |
| 14:17:31 | mriedem | that also pushes a lot of complexity on the operator | |
| 14:17:37 | dansmith | it does, and that sucks | |
| 14:17:39 | mriedem | bauzas: #2 | |
| 14:18:15 | dansmith | mriedem: so, we should probably go the extra-call-to-compute route for now and if that ends up being too expensive, then punt later | |
| 14:18:31 | dansmith | we already do that for things like getting predicted-but-never-correct bdm device names right? | |
| 14:18:37 | mriedem | yeah, we do that from the api | |
| 14:18:42 | bauzas | so, my take on that is that overheads, since are only virt-related are things which wouldn't be exposed to users | |
| 14:18:43 | mriedem | when attaching a volume | |
| 14:18:54 | dansmith | so here's another potential optimization: | |
| 14:18:57 | bauzas | users ask for memory sizes they know | |
| 14:19:10 | bauzas | they don't know how much the hypervisor has | |
| 14:19:14 | bauzas | so | |
| 14:19:25 | dansmith | we could do that call once, and record the value we got for the flavor id, hypervisor type and version, and only do that once per boot if we have to try several hosts in a single cycle | |
| 14:19:53 | mriedem | oh btw, this is nice in the py35 tests: "Fatal Python error: Cannot recover from stack overflow." | |
| 14:19:54 | dansmith | mriedem: like, if we have to try four libvirt hosts in a single boot call, we only have to call to one of the computes to get the overhead | |
| 14:19:56 | mriedem | sdague: ^ | |
| 14:20:00 | bauzas | when the scheduler is choosing a destination, he has to pick it based on the user request, but related to the total amount of memory is left | |
| 14:20:21 | mriedem | dansmith: yeah that's not a bad idea | |
| 14:20:27 | mriedem | cache the results in conductor | |
| 14:20:36 | mriedem | take the rpc call hit once | |
| 14:20:38 | dansmith | we need more information than just the hostname coming back from scheduler though, else we have to hit the db ourselves to look up the host info again | |
| 14:20:43 | dansmith | yeah | |
| 14:21:02 | dansmith | either once per call or once per conductor worker | |
| 14:21:06 | dansmith | or memcache or something | |
| 14:21:15 | mriedem | i assumed once per worker | |
| 14:21:22 | dansmith | well, that could get big, | |
| 14:21:33 | dansmith | as there are tons of workers, and potentially lots of flavors and stuff | |
| 14:21:33 | mriedem | sure, if we use the cache utils stuff, | |
| 14:21:35 | mriedem | it's configurable | |
| 14:21:41 | dansmith | yeah, that's the way to go I think | |
| 14:21:43 | mriedem | default is per worker in memory cache | |
| 14:21:48 | mriedem | if you want memcache, configure it | |
| 14:21:50 | dansmith | yeah | |
| 14:22:02 | bauzas | wait | |
| 14:22:30 | mriedem | gcb: i'm seeing "Fatal Python error: Cannot recover from stack overflow." in the py35 job for nova unit tests randomly | |
| 14:22:30 | bauzas | why placement couldn't compare the user flavor vs. the size of what's left, ie. inventory - instances ? | |
| 14:22:35 | mriedem | gcb: http://logs.openstack.org/97/456397/3/check/gate-nova-python35/05322ec/console.html#_2017-04-19_12_45_10_724578 | |
| 14:22:47 | bauzas | instances being "instance.flavor + instance.overhead" ? | |
| 14:22:51 | mriedem | bauzas: placement doesn't have the overhead info | |
| 14:23:05 | bauzas | mriedem: placement knows inventory I know | |
| 14:23:16 | mriedem | bauzas: we're saying we need to get the overhead from the compute before we post the allocations | |
| 14:23:20 | mriedem | to take into account the overhead | |
| 14:23:33 | mriedem | then placement tells us if that works or not for the inventory on the provider | |
| 14:23:44 | mriedem | if POST /allocations/instance fails, then we retry through the scheduler | |
| 14:24:06 | mriedem | dan was just saying that we can cache the rpc call results in conductor - the results of the call to the compute to get the overhead | |
| 14:24:23 | mriedem | per flavorid/hypervisortype/version | |
| 14:24:25 | bauzas | mriedem: that would assume the overhead is not calculated per instance | |
| 14:24:32 | mriedem | huh? | |
| 14:24:36 | gcb | mriedem: I saw the email , trying to figure out the root cause and fix it | |
| 14:24:39 | mriedem | the overhead is calculated per instance | |
| 14:24:46 | mriedem | gcb: this is a different error :) | |
| 14:25:01 | bauzas | mriedem: yup, so we would query the compute every time a boot request comes in to know the overhead, right? | |
| 14:25:06 | mriedem | bauzas: yes | |
| 14:25:09 | mriedem | and cache the results | |
| 14:25:25 | bauzas | mriedem: cache what ? given we don't know which compute | |
| 14:25:34 | gcb | mriedem, wow, let me check the details | |
| 14:25:39 | mriedem | bauzas: we'd do this in conductor after we get a host from the scheduler | |
| 14:25:41 | bauzas | we would have to cache overheads per flavor for all computes | |
| 14:25:47 | dansmith | no | |
| 14:25:47 | mriedem | bauzas: right | |
| 14:25:48 | bauzas | mriedem: oh, after ? | |
| 14:25:57 | bauzas | erm, not sure I like that | |
| 14:26:07 | dansmith | per hypervisor type/version | |
| 14:26:10 | bauzas | I'd tend to prefer considering that the flavor is what the user requests | |
| 14:26:24 | mriedem | bauzas: that's not how it works today | |
| 14:26:36 | bauzas | and placement should verify against the inventory which would be decremented by the sum of overheads | |
| 14:26:52 | mriedem | i request 8GB of RAM for my vm, but xen says i really need 8GB + 256MB or something | |
| 14:27:02 | bauzas | at the moment, placement matches against the inventory minus the sum of allocations | |
| 14:27:04 | lyarwood | mriedem: ack sure looking now | |
| 14:27:26 | mriedem | bauzas: "and placement should verify against the inventory which would be decremented by the sum of overheads" - yes, but we have to get the overheads to placement | |
| 14:27:32 | bauzas | I'm proposing to match against the inventory minus the allocations+overhead | |
| 14:27:33 | mriedem | we do that via the allocations | |
| 14:27:42 | mriedem | bauzas: yes, we all agree on that | |
| 14:27:51 | mriedem | that's not new | |
| 14:27:54 | dansmith | the fact that we have larger allocations than the flavor is not ideal, but at the moment we have no choice | |
| 14:28:15 | bauzas | but you would not just compare with the flavor for the instance, but flavor+cached_overhead ? | |
| 14:28:25 | bauzas | well, huge thing | |
| 14:28:40 | mriedem | the allocations are made up of the flavor today | |
| 14:28:42 | bauzas | dansmith: mriedem: okay, I think why you'd like to cache that | |
| 14:29:46 | mriedem | should i go back and update the spec with a summary? | |
| 14:30:16 | mriedem | bauzas: unfortunately for you, this means redoing a large part of the spec where you changed it from posting allocations in conductor to scheduler, now back to conductor :( | |
| 14:31:07 | cdent | mriedem: what does that mean with regard to the reasons it moved back to the scheduler in the first place? | |
| 14:31:53 | cdent | conductor ->(reasons)->sched->(new reasons)->conductor | |
| 14:32:53 | bauzas | mriedem: this is fine https://www.nytimes.com/2016/08/06/arts/this-is-fine-meme-dog-fire.html | |
| 14:33:13 | mriedem | cdent: we moved it to scheduler because that's where the filtered host list is, so retries there might be theoretically faster, | |