| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 14:19:53 | mriedem | oh btw, this is nice in the py35 tests: "Fatal Python error: Cannot recover from stack overflow." | |
| 14:19:54 | dansmith | mriedem: like, if we have to try four libvirt hosts in a single boot call, we only have to call to one of the computes to get the overhead | |
| 14:19:56 | mriedem | sdague: ^ | |
| 14:20:00 | bauzas | when the scheduler is choosing a destination, he has to pick it based on the user request, but related to the total amount of memory is left | |
| 14:20:21 | mriedem | dansmith: yeah that's not a bad idea | |
| 14:20:27 | mriedem | cache the results in conductor | |
| 14:20:36 | mriedem | take the rpc call hit once | |
| 14:20:38 | dansmith | we need more information than just the hostname coming back from scheduler though, else we have to hit the db ourselves to look up the host info again | |
| 14:20:43 | dansmith | yeah | |
| 14:21:02 | dansmith | either once per call or once per conductor worker | |
| 14:21:06 | dansmith | or memcache or something | |
| 14:21:15 | mriedem | i assumed once per worker | |
| 14:21:22 | dansmith | well, that could get big, | |
| 14:21:33 | mriedem | sure, if we use the cache utils stuff, | |
| 14:21:33 | dansmith | as there are tons of workers, and potentially lots of flavors and stuff | |
| 14:21:35 | mriedem | it's configurable | |
| 14:21:41 | dansmith | yeah, that's the way to go I think | |
| 14:21:43 | mriedem | default is per worker in memory cache | |
| 14:21:48 | mriedem | if you want memcache, configure it | |
| 14:21:50 | dansmith | yeah | |
| 14:22:02 | bauzas | wait | |
| 14:22:30 | bauzas | why placement couldn't compare the user flavor vs. the size of what's left, ie. inventory - instances ? | |
| 14:22:30 | mriedem | gcb: i'm seeing "Fatal Python error: Cannot recover from stack overflow." in the py35 job for nova unit tests randomly | |
| 14:22:35 | mriedem | gcb: http://logs.openstack.org/97/456397/3/check/gate-nova-python35/05322ec/console.html#_2017-04-19_12_45_10_724578 | |
| 14:22:47 | bauzas | instances being "instance.flavor + instance.overhead" ? | |
| 14:22:51 | mriedem | bauzas: placement doesn't have the overhead info | |
| 14:23:05 | bauzas | mriedem: placement knows inventory I know | |
| 14:23:16 | mriedem | bauzas: we're saying we need to get the overhead from the compute before we post the allocations | |
| 14:23:20 | mriedem | to take into account the overhead | |
| 14:23:33 | mriedem | then placement tells us if that works or not for the inventory on the provider | |
| 14:23:44 | mriedem | if POST /allocations/instance fails, then we retry through the scheduler | |
| 14:24:06 | mriedem | dan was just saying that we can cache the rpc call results in conductor - the results of the call to the compute to get the overhead | |
| 14:24:23 | mriedem | per flavorid/hypervisortype/version | |
| 14:24:25 | bauzas | mriedem: that would assume the overhead is not calculated per instance | |
| 14:24:32 | mriedem | huh? | |
| 14:24:36 | gcb | mriedem: I saw the email , trying to figure out the root cause and fix it | |
| 14:24:39 | mriedem | the overhead is calculated per instance | |
| 14:24:46 | mriedem | gcb: this is a different error :) | |
| 14:25:01 | bauzas | mriedem: yup, so we would query the compute every time a boot request comes in to know the overhead, right? | |
| 14:25:06 | mriedem | bauzas: yes | |
| 14:25:09 | mriedem | and cache the results | |
| 14:25:25 | bauzas | mriedem: cache what ? given we don't know which compute | |
| 14:25:34 | gcb | mriedem, wow, let me check the details | |
| 14:25:39 | mriedem | bauzas: we'd do this in conductor after we get a host from the scheduler | |
| 14:25:41 | bauzas | we would have to cache overheads per flavor for all computes | |
| 14:25:47 | mriedem | bauzas: right | |
| 14:25:47 | dansmith | no | |
| 14:25:48 | bauzas | mriedem: oh, after ? | |
| 14:25:57 | bauzas | erm, not sure I like that | |
| 14:26:07 | dansmith | per hypervisor type/version | |
| 14:26:10 | bauzas | I'd tend to prefer considering that the flavor is what the user requests | |
| 14:26:24 | mriedem | bauzas: that's not how it works today | |
| 14:26:36 | bauzas | and placement should verify against the inventory which would be decremented by the sum of overheads | |
| 14:26:52 | mriedem | i request 8GB of RAM for my vm, but xen says i really need 8GB + 256MB or something | |
| 14:27:02 | bauzas | at the moment, placement matches against the inventory minus the sum of allocations | |
| 14:27:04 | lyarwood | mriedem: ack sure looking now | |
| 14:27:26 | mriedem | bauzas: "and placement should verify against the inventory which would be decremented by the sum of overheads" - yes, but we have to get the overheads to placement | |
| 14:27:32 | bauzas | I'm proposing to match against the inventory minus the allocations+overhead | |
| 14:27:33 | mriedem | we do that via the allocations | |
| 14:27:42 | mriedem | bauzas: yes, we all agree on that | |
| 14:27:51 | mriedem | that's not new | |
| 14:27:54 | dansmith | the fact that we have larger allocations than the flavor is not ideal, but at the moment we have no choice | |
| 14:28:15 | bauzas | but you would not just compare with the flavor for the instance, but flavor+cached_overhead ? | |
| 14:28:25 | bauzas | well, huge thing | |
| 14:28:40 | mriedem | the allocations are made up of the flavor today | |
| 14:28:42 | bauzas | dansmith: mriedem: okay, I think why you'd like to cache that | |
| 14:29:46 | mriedem | should i go back and update the spec with a summary? | |
| 14:30:16 | mriedem | bauzas: unfortunately for you, this means redoing a large part of the spec where you changed it from posting allocations in conductor to scheduler, now back to conductor :( | |
| 14:31:07 | cdent | mriedem: what does that mean with regard to the reasons it moved back to the scheduler in the first place? | |
| 14:31:53 | cdent | conductor ->(reasons)->sched->(new reasons)->conductor | |
| 14:32:53 | bauzas | mriedem: this is fine https://www.nytimes.com/2016/08/06/arts/this-is-fine-meme-dog-fire.html | |
| 14:33:13 | mriedem | cdent: we moved it to scheduler because that's where the filtered host list is, so retries there might be theoretically faster, | |
| 14:33:33 | cdent | so it's just a performance thing? I guess that's okay then | |
| 14:33:35 | mriedem | cdent: we're moving to conductor now because conductor is what's going to need to do the rpc call into the compute to get the overhead values, | |
| 14:33:49 | bauzas | mriedem: dansmith: wait, we expose flavors to scheduler, right? | |
| 14:34:04 | bauzas | oh crap, nvm | |
| 14:34:28 | mriedem | so conductor -> scheduler (select_destinations) -> conductor (rpc call to compute to get overhead for the selected host) -> post allocations (flavor + overhead) to placement -> (1) success, build on compute, or (2) fail, retry through scheduler for a new host | |
| 14:34:35 | mriedem | cdent: ^ make sense? | |
| 14:34:41 | bauzas | I'm trying to consider how we could inform placement of the overhead for the specific instance | |
| 14:34:51 | mriedem | bauzas: via the allocations that we post | |
| 14:34:53 | mriedem | see ^ | |
| 14:35:12 | cdent | mriedem: yeah, makes sense, but is frustrating | |
| 14:35:20 | mriedem | cdent: agree, | |
| 14:35:36 | bauzas | mriedem: I'd love to have posted when we do GET /RPs=memory=X | |
| 14:35:37 | mriedem | especially since i for one didn't even know this overhead thing existed until yesterday | |
| 14:35:47 | mriedem | bauzas: we don't know the host then | |
| 14:35:50 | mriedem | it's a chicken and egg | |
| 14:35:52 | bauzas | mriedem: where X would be flavor+overhead | |
| 14:36:01 | bauzas | fuuuuuuuuuu | |
| 14:36:03 | mriedem | bauzas: we have to have the host to get the overhead | |
| 14:36:09 | mriedem | because the virt driver gives the overhead values | |
| 14:36:47 | bauzas | right, so the only problem is with the new POST allocation | |
| 14:36:52 | bauzas | sorry, slow brain here | |
| 14:37:12 | bauzas | my brain is fried by that shit ton of political discussions I'm eating | |
| 14:37:59 | mriedem | i don't read the news anymore | |
| 14:38:08 | mriedem | i heard we bombed afghanistan with a large bomb | |
| 14:38:27 | mriedem | to show how cool we are | |
| 14:38:39 | edleafe | Is the overhead specific to each hypervisor, but the same for all instances on that hypervisor, regardless of instance size? | |
| 14:39:07 | mriedem | edleafe: no | |
| 14:39:15 | edleafe | ugh | |