| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-06 | |||
| 18:55:03 | ronlund | if we log ^ but no error, it just means the claim was successful | |
| 18:55:06 | cdent | ronlund: I’m not sure that’s the case? | |
| 18:55:18 | ronlund | we don't log anything for a successful claim | |
| 18:55:59 | cdent | so in that case we have an attempt for instance X, then a fail for instance Y, where is the attempt for Y? | |
| 18:56:08 | ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-placement-api.txt.gz#_Oct_04_18_32_08_825897 is that other instance | |
| 18:56:48 | cdent | so why are we getting the warn about the failure after the debug about the other attempt | |
| 18:56:53 | cdent | they _are_ out of order | |
| 18:57:04 | cdent | which is what we would expect in a monkey patched eventlet scene | |
| 18:59:20 | cdent | bbs, gotta do the dishes | |
| 18:59:38 | ronlund | i'm seeing something f'ed in the scheduler code, | |
| 18:59:43 | ronlund | it's doing the same scheduling request 3 times | |
| 18:59:45 | ronlund | "Starting to schedule for" | |
| 19:00:22 | ronlund | 3 seconds apart | |
| 19:00:29 | ronlund | well, 1 second apart, 3 times | |
| 19:00:46 | ronlund | all with the same request id | |
| 19:01:22 | ronlund | leakypipes: well that answers that i think ^ | |
| 19:01:30 | ronlund | the 409 is probably because we already posted allocations for the same consumer | |
| 19:01:52 | Jose____ | Hi! i found this in nova placement api log: "Placement API returning an error response: JSON does not validate: 0 is less than the minimum of 1", can this be affecting with the image creation service? | |
| 19:02:02 | ronlund | Jose____: no | |
| 19:02:04 | leakypipes | ronlund: hmm, that's odd indeed. | |
| 19:03:11 | cdent | ronlund: is that request id the local one or the one passed up from nova-api | |
| 19:04:13 | ronlund | cdent: the api http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-api.txt.gz#_Oct_04_18_30_21_221501 | |
| 19:04:20 | ronlund | {"server": {"name": "test-alloc-conflict", "imageRef": "947823a9-8c15-420a-95d8-f35cd2f024b9", "flavorRef": "1", "max_count": 1000, "min_count": 1000, "networks": "none"}} | |
| 19:04:50 | ronlund | i see a successful allocation for the failed instance 2 seconds after the final 409 retry failure | |
| 19:04:51 | ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-placement-api.txt.gz#_Oct_04_18_34_13_854776 | |
| 19:05:40 | ronlund | wth, why would we have 3 greenthreads? | |
| 19:08:16 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: de-ORM ResourceProvider.get_by_uuid() https://review.openstack.org/509025 | |
| 19:08:17 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: Remove RP.get_traits() method https://review.openstack.org/509027 | |
| 19:08:17 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: Move RP._get|set_aggregates() to module scope https://review.openstack.org/509026 | |
| 19:08:18 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove CRUD operations on Inventory class https://review.openstack.org/509029 | |
| 19:08:18 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: move RP._set_traits() to module scope https://review.openstack.org/509028 | |
| 19:08:19 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove dead code in Allocation._create_in_db() https://review.openstack.org/509031 | |
| 19:08:19 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: streamline InventoryList.get_all_by_rp_uuid() https://review.openstack.org/509030 | |
| 19:08:20 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove ability to delete 1 allocation record https://review.openstack.org/509032 | |
| 19:08:21 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove _HasAResourceProvider mixin https://review.openstack.org/509036 | |
| 19:08:21 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: rework AllocList.get_all_by_consumer_id() https://review.openstack.org/509035 | |
| 19:08:21 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: fix up AllocList.get_by_resource_provider_uuid https://review.openstack.org/509033 | |
| 19:08:22 | openstackgerrit | Jay Pipes proposed openstack/nova master: rp: break functions out of _set_traits() https://review.openstack.org/509908 | |
| 19:09:02 | ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-super-cond.txt.gz#_Oct_04_18_32_07_052398 | |
| 19:09:02 | ronlund | aha | |
| 19:09:21 | ronlund | select_destinations took too long, so oslo.messaging retried | |
| 19:09:53 | ronlund | well that explains that i think | |
| 19:09:58 | ronlund | i'm going to bump up the rpc timeout | |
| 19:10:18 | cdent | that doesn’t sound safe | |
| 19:10:47 | cdent | (dishwasher not done yet) | |
| 19:11:14 | ronlund | huh, given we don't rate limit min_count at all, | |
| 19:11:23 | ronlund | you could totally bork some stuff up with making a huge request | |
| 19:11:35 | ronlund | if the server's rpc timeout isn't configured high enough to handle it | |
| 19:11:45 | cfriesen | ronlund: I think we check quota fairly early, no? | |
| 19:11:50 | ronlund | this isn't quota | |
| 19:12:07 | ronlund | although, sure you wouldn't hit this for real if you had default quota of 10 | |
| 19:12:11 | sean-k-mooney | jgwentworth: https://github.com/openstack-infra/project-config/blob/dbdef981de7cb56e9cd44514a41102270bfc9bac/jenkins/scripts/run-tox.sh#L32-L48 this is why its failing we only grab the last result dir | |
| 19:12:12 | ronlund | i've disabled quota | |
| 19:12:24 | ronlund | if you did allow a tenant 1000 instances to burst at once, | |
| 19:12:33 | ronlund | then you are going to have to deal with big ass rpc timeouts | |
| 19:13:03 | leakypipes | ronlund: there's a crapload of timeouts on the MQ in that log file... | |
| 19:14:05 | ronlund | yeah i know | |
| 19:14:14 | ronlund | and by default oslo.messaging retries twice | |
| 19:14:51 | superdan | so it just adds to the load? | |
| 19:15:03 | superdan | because it's starting extra scheduling runs for the same set of stuff? | |
| 19:15:14 | superdan | seems like we'd be breaking hard on that already anyway, pre-placement | |
| 19:17:02 | ronlund | yeah this is definitely user error on my part :) | |
| 19:17:21 | ronlund | maybe it's a decent simulator of sorts, but probably not | |
| 19:17:42 | ronlund | "how to dos your devstack" | |
| 19:18:00 | superdan | user error why? | |
| 19:18:08 | superdan | just because all the limits are removed? | |
| 19:18:12 | ronlund | yeah | |
| 19:18:35 | ronlund | we could definitely hammer scheduler/placement with concurrent requests, | |
| 19:18:38 | cdent | in a perfect world it would at least fail gracefully rather than confusedly | |
| 19:18:45 | ronlund | but within a single tenant, default quota is 10 so that'd be your max | |
| 19:18:56 | superdan | well, we probably really should never retry a call to the scheduler like this after a timeout, I'm thinking | |
| 19:19:00 | ronlund | well, the good news is it totally does the allocation cleanup properly | |
| 19:19:07 | ronlund | and everything is put into error state and shoved in cell0 | |
| 19:19:24 | superdan | because you could hit that timeout for other reasons | |
| 19:19:48 | superdan | well, I guess it has already created the instance records, so maybe not a huge deal I guess, | |
| 19:20:16 | superdan | I expect there is a case where you could do a boot, fail to hear from scheduler, never send boots to compute, | |
| 19:20:21 | superdan | but the scheduler made allocations | |
| 19:20:48 | superdan | I guess we just live with that and assume they're cleaned up at delete, but technically it's holding space for those dead ones | |
| 19:21:06 | superdan | actually, maybe we wouldn't clean up allocations on delete in that case since instance.host=None? | |
| 19:23:19 | ronlund | superdan: well i think if instance.host == None we assume the allocations are already gone | |
| 19:23:24 | ronlund | either it failed to schedule, | |
| 19:23:27 | ronlund | or it was shelved offloaded | |
| 19:23:36 | ronlund | and we remove allocations when shelve offloading | |
| 19:23:36 | superdan | right, my point | |
| 19:23:41 | superdan | we call to scheduler, timeout, | |
| 19:23:44 | superdan | scheduler has made allocations | |
| 19:23:55 | superdan | we just delete from db because it never scheduled | |
| 19:24:02 | ronlund | instance goes to error | |
| 19:24:04 | ronlund | but has allocations | |
| 19:24:13 | superdan | yes | |
| 19:24:17 | ronlund | heh yeah | |
| 19:26:05 | ronlund | checking for something like that on every delete kind of sucks if it's a super edge case | |
| 19:26:26 | superdan | but no healing, so.. leaking capacity will anger people and rightly so :) | |
| 19:26:38 | ronlund | right | |
| 19:26:54 | ronlund | plus a delete request for allocations that never existing should be fast | |
| 19:26:58 | ronlund | *existed | |
| 19:27:02 | superdan | yes | |
| 19:27:47 | ronlund | i know huawei customers love nfv, i need to see what their instance quota limit is quick... :) | |
| 19:29:48 | ronlund | ha | |
| 19:29:49 | ronlund | @utils.retry_select_destinations | |