| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 14:19:25 | dansmith | we could do that call once, and record the value we got for the flavor id, hypervisor type and version, and only do that once per boot if we have to try several hosts in a single cycle | |
| 14:19:53 | mriedem | oh btw, this is nice in the py35 tests: "Fatal Python error: Cannot recover from stack overflow." | |
| 14:19:54 | dansmith | mriedem: like, if we have to try four libvirt hosts in a single boot call, we only have to call to one of the computes to get the overhead | |
| 14:19:56 | mriedem | sdague: ^ | |
| 14:20:00 | bauzas | when the scheduler is choosing a destination, he has to pick it based on the user request, but related to the total amount of memory is left | |
| 14:20:21 | mriedem | dansmith: yeah that's not a bad idea | |
| 14:20:27 | mriedem | cache the results in conductor | |
| 14:20:36 | mriedem | take the rpc call hit once | |
| 14:20:38 | dansmith | we need more information than just the hostname coming back from scheduler though, else we have to hit the db ourselves to look up the host info again | |
| 14:20:43 | dansmith | yeah | |
| 14:21:02 | dansmith | either once per call or once per conductor worker | |
| 14:21:06 | dansmith | or memcache or something | |
| 14:21:15 | mriedem | i assumed once per worker | |
| 14:21:22 | dansmith | well, that could get big, | |
| 14:21:33 | mriedem | sure, if we use the cache utils stuff, | |
| 14:21:33 | dansmith | as there are tons of workers, and potentially lots of flavors and stuff | |
| 14:21:35 | mriedem | it's configurable | |
| 14:21:41 | dansmith | yeah, that's the way to go I think | |
| 14:21:43 | mriedem | default is per worker in memory cache | |
| 14:21:48 | mriedem | if you want memcache, configure it | |
| 14:21:50 | dansmith | yeah | |
| 14:22:02 | bauzas | wait | |
| 14:22:30 | bauzas | why placement couldn't compare the user flavor vs. the size of what's left, ie. inventory - instances ? | |
| 14:22:30 | mriedem | gcb: i'm seeing "Fatal Python error: Cannot recover from stack overflow." in the py35 job for nova unit tests randomly | |
| 14:22:35 | mriedem | gcb: http://logs.openstack.org/97/456397/3/check/gate-nova-python35/05322ec/console.html#_2017-04-19_12_45_10_724578 | |
| 14:22:47 | bauzas | instances being "instance.flavor + instance.overhead" ? | |
| 14:22:51 | mriedem | bauzas: placement doesn't have the overhead info | |
| 14:23:05 | bauzas | mriedem: placement knows inventory I know | |
| 14:23:16 | mriedem | bauzas: we're saying we need to get the overhead from the compute before we post the allocations | |
| 14:23:20 | mriedem | to take into account the overhead | |
| 14:23:33 | mriedem | then placement tells us if that works or not for the inventory on the provider | |
| 14:23:44 | mriedem | if POST /allocations/instance fails, then we retry through the scheduler | |
| 14:24:06 | mriedem | dan was just saying that we can cache the rpc call results in conductor - the results of the call to the compute to get the overhead | |
| 14:24:23 | mriedem | per flavorid/hypervisortype/version | |
| 14:24:25 | bauzas | mriedem: that would assume the overhead is not calculated per instance | |
| 14:24:32 | mriedem | huh? | |
| 14:24:36 | gcb | mriedem: I saw the email , trying to figure out the root cause and fix it | |
| 14:24:39 | mriedem | the overhead is calculated per instance | |
| 14:24:46 | mriedem | gcb: this is a different error :) | |
| 14:25:01 | bauzas | mriedem: yup, so we would query the compute every time a boot request comes in to know the overhead, right? | |
| 14:25:06 | mriedem | bauzas: yes | |
| 14:25:09 | mriedem | and cache the results | |
| 14:25:25 | bauzas | mriedem: cache what ? given we don't know which compute | |
| 14:25:34 | gcb | mriedem, wow, let me check the details | |
| 14:25:39 | mriedem | bauzas: we'd do this in conductor after we get a host from the scheduler | |
| 14:25:41 | bauzas | we would have to cache overheads per flavor for all computes | |
| 14:25:47 | mriedem | bauzas: right | |
| 14:25:47 | dansmith | no | |
| 14:25:48 | bauzas | mriedem: oh, after ? | |
| 14:25:57 | bauzas | erm, not sure I like that | |
| 14:26:07 | dansmith | per hypervisor type/version | |
| 14:26:10 | bauzas | I'd tend to prefer considering that the flavor is what the user requests | |
| 14:26:24 | mriedem | bauzas: that's not how it works today | |
| 14:26:36 | bauzas | and placement should verify against the inventory which would be decremented by the sum of overheads | |
| 14:26:52 | mriedem | i request 8GB of RAM for my vm, but xen says i really need 8GB + 256MB or something | |
| 14:27:02 | bauzas | at the moment, placement matches against the inventory minus the sum of allocations | |
| 14:27:04 | lyarwood | mriedem: ack sure looking now | |
| 14:27:26 | mriedem | bauzas: "and placement should verify against the inventory which would be decremented by the sum of overheads" - yes, but we have to get the overheads to placement | |
| 14:27:32 | bauzas | I'm proposing to match against the inventory minus the allocations+overhead | |
| 14:27:33 | mriedem | we do that via the allocations | |
| 14:27:42 | mriedem | bauzas: yes, we all agree on that | |
| 14:27:51 | mriedem | that's not new | |
| 14:27:54 | dansmith | the fact that we have larger allocations than the flavor is not ideal, but at the moment we have no choice | |
| 14:28:15 | bauzas | but you would not just compare with the flavor for the instance, but flavor+cached_overhead ? | |
| 14:28:25 | bauzas | well, huge thing | |
| 14:28:40 | mriedem | the allocations are made up of the flavor today | |
| 14:28:42 | bauzas | dansmith: mriedem: okay, I think why you'd like to cache that | |
| 14:29:46 | mriedem | should i go back and update the spec with a summary? | |
| 14:30:16 | mriedem | bauzas: unfortunately for you, this means redoing a large part of the spec where you changed it from posting allocations in conductor to scheduler, now back to conductor :( | |
| 14:31:07 | cdent | mriedem: what does that mean with regard to the reasons it moved back to the scheduler in the first place? | |
| 14:31:53 | cdent | conductor ->(reasons)->sched->(new reasons)->conductor | |
| 14:32:53 | bauzas | mriedem: this is fine https://www.nytimes.com/2016/08/06/arts/this-is-fine-meme-dog-fire.html | |
| 14:33:13 | mriedem | cdent: we moved it to scheduler because that's where the filtered host list is, so retries there might be theoretically faster, | |
| 14:33:33 | cdent | so it's just a performance thing? I guess that's okay then | |
| 14:33:35 | mriedem | cdent: we're moving to conductor now because conductor is what's going to need to do the rpc call into the compute to get the overhead values, | |
| 14:33:49 | bauzas | mriedem: dansmith: wait, we expose flavors to scheduler, right? | |
| 14:34:04 | bauzas | oh crap, nvm | |
| 14:34:28 | mriedem | so conductor -> scheduler (select_destinations) -> conductor (rpc call to compute to get overhead for the selected host) -> post allocations (flavor + overhead) to placement -> (1) success, build on compute, or (2) fail, retry through scheduler for a new host | |
| 14:34:35 | mriedem | cdent: ^ make sense? | |
| 14:34:41 | bauzas | I'm trying to consider how we could inform placement of the overhead for the specific instance | |
| 14:34:51 | mriedem | bauzas: via the allocations that we post | |
| 14:34:53 | mriedem | see ^ | |
| 14:35:12 | cdent | mriedem: yeah, makes sense, but is frustrating | |
| 14:35:20 | mriedem | cdent: agree, | |
| 14:35:36 | bauzas | mriedem: I'd love to have posted when we do GET /RPs=memory=X | |
| 14:35:37 | mriedem | especially since i for one didn't even know this overhead thing existed until yesterday | |
| 14:35:47 | mriedem | bauzas: we don't know the host then | |
| 14:35:50 | mriedem | it's a chicken and egg | |
| 14:35:52 | bauzas | mriedem: where X would be flavor+overhead | |
| 14:36:01 | bauzas | fuuuuuuuuuu | |
| 14:36:03 | mriedem | bauzas: we have to have the host to get the overhead | |
| 14:36:09 | mriedem | because the virt driver gives the overhead values | |
| 14:36:47 | bauzas | right, so the only problem is with the new POST allocation | |
| 14:36:52 | bauzas | sorry, slow brain here | |
| 14:37:12 | bauzas | my brain is fried by that shit ton of political discussions I'm eating | |
| 14:37:59 | mriedem | i don't read the news anymore | |
| 14:38:08 | mriedem | i heard we bombed afghanistan with a large bomb | |
| 14:38:27 | mriedem | to show how cool we are | |
| 14:38:39 | edleafe | Is the overhead specific to each hypervisor, but the same for all instances on that hypervisor, regardless of instance size? | |
| 14:39:07 | mriedem | edleafe: no | |