| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-06 | |||
| 19:11:45 | cfriesen | ronlund: I think we check quota fairly early, no? | |
| 19:11:50 | ronlund | this isn't quota | |
| 19:12:07 | ronlund | although, sure you wouldn't hit this for real if you had default quota of 10 | |
| 19:12:11 | sean-k-mooney | jgwentworth: https://github.com/openstack-infra/project-config/blob/dbdef981de7cb56e9cd44514a41102270bfc9bac/jenkins/scripts/run-tox.sh#L32-L48 this is why its failing we only grab the last result dir | |
| 19:12:12 | ronlund | i've disabled quota | |
| 19:12:24 | ronlund | if you did allow a tenant 1000 instances to burst at once, | |
| 19:12:33 | ronlund | then you are going to have to deal with big ass rpc timeouts | |
| 19:13:03 | leakypipes | ronlund: there's a crapload of timeouts on the MQ in that log file... | |
| 19:14:05 | ronlund | yeah i know | |
| 19:14:14 | ronlund | and by default oslo.messaging retries twice | |
| 19:14:51 | superdan | so it just adds to the load? | |
| 19:15:03 | superdan | because it's starting extra scheduling runs for the same set of stuff? | |
| 19:15:14 | superdan | seems like we'd be breaking hard on that already anyway, pre-placement | |
| 19:17:02 | ronlund | yeah this is definitely user error on my part :) | |
| 19:17:21 | ronlund | maybe it's a decent simulator of sorts, but probably not | |
| 19:17:42 | ronlund | "how to dos your devstack" | |
| 19:18:00 | superdan | user error why? | |
| 19:18:08 | superdan | just because all the limits are removed? | |
| 19:18:12 | ronlund | yeah | |
| 19:18:35 | ronlund | we could definitely hammer scheduler/placement with concurrent requests, | |
| 19:18:38 | cdent | in a perfect world it would at least fail gracefully rather than confusedly | |
| 19:18:45 | ronlund | but within a single tenant, default quota is 10 so that'd be your max | |
| 19:18:56 | superdan | well, we probably really should never retry a call to the scheduler like this after a timeout, I'm thinking | |
| 19:19:00 | ronlund | well, the good news is it totally does the allocation cleanup properly | |
| 19:19:07 | ronlund | and everything is put into error state and shoved in cell0 | |
| 19:19:24 | superdan | because you could hit that timeout for other reasons | |
| 19:19:48 | superdan | well, I guess it has already created the instance records, so maybe not a huge deal I guess, | |
| 19:20:16 | superdan | I expect there is a case where you could do a boot, fail to hear from scheduler, never send boots to compute, | |
| 19:20:21 | superdan | but the scheduler made allocations | |
| 19:20:48 | superdan | I guess we just live with that and assume they're cleaned up at delete, but technically it's holding space for those dead ones | |
| 19:21:06 | superdan | actually, maybe we wouldn't clean up allocations on delete in that case since instance.host=None? | |
| 19:23:19 | ronlund | superdan: well i think if instance.host == None we assume the allocations are already gone | |
| 19:23:24 | ronlund | either it failed to schedule, | |
| 19:23:27 | ronlund | or it was shelved offloaded | |
| 19:23:36 | ronlund | and we remove allocations when shelve offloading | |
| 19:23:36 | superdan | right, my point | |
| 19:23:41 | superdan | we call to scheduler, timeout, | |
| 19:23:44 | superdan | scheduler has made allocations | |
| 19:23:55 | superdan | we just delete from db because it never scheduled | |
| 19:24:02 | ronlund | instance goes to error | |
| 19:24:04 | ronlund | but has allocations | |
| 19:24:13 | superdan | yes | |
| 19:24:17 | ronlund | heh yeah | |
| 19:26:05 | ronlund | checking for something like that on every delete kind of sucks if it's a super edge case | |
| 19:26:26 | superdan | but no healing, so.. leaking capacity will anger people and rightly so :) | |
| 19:26:38 | ronlund | right | |
| 19:26:54 | ronlund | plus a delete request for allocations that never existing should be fast | |
| 19:26:58 | ronlund | *existed | |
| 19:27:02 | superdan | yes | |
| 19:27:47 | ronlund | i know huawei customers love nfv, i need to see what their instance quota limit is quick... :) | |
| 19:29:48 | ronlund | ha | |
| 19:29:49 | ronlund | @utils.retry_select_destinations | |
| 19:29:53 | ronlund | that's what's causing the retry | |
| 19:29:54 | ronlund | it's by design | |
| 19:29:59 | ronlund | it's not oslo.messaging, it's nova | |
| 19:31:02 | ronlund | https://github.com/openstack/nova/blob/353db2d1932965b6502e002b8be510440ff529c0/nova/scheduler/utils.py#L599 | |
| 19:31:38 | superdan | yeah | |
| 19:31:49 | ronlund | so yeah, now that we're doing claims in the scheduler, that seems like a bad idea... | |
| 19:32:14 | ronlund | it does it up to max_attempts-1, so by default 2 retries | |
| 19:32:27 | superdan | that doesn't fix the allocation leak, mind you, | |
| 19:32:43 | superdan | but yeah, seems like if you fail talking to it, you're just going to hurt things by adding to the load with a retry | |
| 19:33:11 | ronlund | i wonder if we double up the 2nd allocation request for the same consumer | |
| 19:33:25 | ronlund | maybe not if there is only 1 rp uuid in the request | |
| 19:33:40 | ronlund | note the 2nd time through the scheduler on the retry, we could likely target a completely different host :) | |
| 19:33:49 | ronlund | thus totally fucking up things for everything | |
| 19:33:52 | superdan | yeah | |
| 19:34:08 | ronlund | huh, well this is fun | |
| 19:50:57 | leakypipes | fried_rice: putting: "blueprint: XXXX" does the same thing. | |
| 19:51:27 | fried_rice | leakypipes Coolio. Is there a Source Of Truth for these taggy things? | |
| 19:54:31 | leakypipes | fried_rice: meh, https://wiki.openstack.org/wiki/GitCommitMessages | |
| 19:54:45 | leakypipes | fried_rice: but it only mentions using Implements: blueprint XXX | |
| 19:54:50 | fried_rice | mm | |
| 19:55:23 | leakypipes | fried_rice: that's not necessary though. the word "blueprint" followed by a tag-like thing is all that's needed to link the patch with the blueprint on Launchpad. | |
| 19:55:42 | fried_rice | leakypipes Including having whatever bot add the URL to the whiteboard on the bp? | |
| 19:55:58 | fried_rice | Cause that seems to be a thing. | |
| 19:56:06 | leakypipes | fried_rice: correct, that's what I mean. | |
| 19:56:22 | fried_rice | k, thought you were just talking about gerrit turning it into a nice hyperlink to the LP page. | |
| 19:56:41 | fried_rice | Anyway, I dig it. | |
| 19:58:07 | ronlund | bp also works i think | |
| 19:58:12 | ronlund | maybe not | |
| 19:59:56 | mtreinish | there is a lot of backscroll, but I couldn't see a link to what I should be looking at | |
| 19:59:56 | mtreinish | ronlund, jgwentworth: do you have a link to the thing you're seeing? | |
| 20:00:11 | jgwentworth | sec | |
| 20:00:43 | jgwentworth | mtreinish: this is happening on stable/ocata and stable/pike only http://logs.openstack.org/39/509439/1/check/gate-nova-python27-ubuntu-xenial/e456c8f/testr_results.html.gz | |
| 20:01:05 | jgwentworth | I think it's just a display issue, showing the os profiler result instead of the unit tests result. in the console you can see that both ran | |
| 20:01:23 | mtreinish | jgwentworth: ok, yeah that's because you have 2 test runs in the tox command | |
| 20:01:43 | mtreinish | for the post processing to generate that we run testr last --subunit and pipe that into subunit2html to generate that page | |
| 20:01:46 | jgwentworth | it shows the right thing on master and stable/newton for some reason even though we have 2 runs | |
| 20:01:59 | mtreinish | but testr doesn't let you combine the results | |
| 20:02:10 | mtreinish | so it's just showing the results from the second one | |
| 20:02:12 | jgwentworth | yeah, that's what sean-k-mooney was saying | |
| 20:02:37 | jgwentworth | well, I think testr is showing the first one. os profiler always runs last | |
| 20:02:39 | mtreinish | it works on master because stestr has a --combine flag that treats the 2 commands as a single run | |
| 20:03:23 | ronlund | gah 2017-10-06 04:30:20.654 | /opt/stack/new/devstack/inc/meta-config: line 209: /opt/stack/new/devstack/.localrc.auto: Permission denied | |
| 20:03:41 | jgwentworth | mtreinish: oh, on stable/newton we're not running the os profiler thing | |
| 20:04:00 | jgwentworth | that's why that one shows up correctly | |
| 20:04:28 | mtreinish | yep | |
| 20:04:53 | mtreinish | this was the same thing I was seeing on openstack-health, which is why I pushed https://review.openstack.org/#/c/501842/ before doing the stestr migration | |
| 20:04:59 | jgwentworth | weird. it seems like this would always have been happening before stestr but I could have sworn I had seen full lists of the unit tests on a pass run prior to stestr | |
| 20:05:32 | jgwentworth | maybe I dreamed it | |