| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-26 | |||
| 23:00:56 | dansmith | you can still fail for other reasons | |
| 23:01:10 | mriedem | filters i suppose yeah | |
| 23:01:27 | dansmith | but yeah, people really hate this behavior that we tell them they can boot 500 things and later fail 20% of them because we suck | |
| 23:01:32 | dansmith | this is much better | |
| 23:02:25 | dansmith | so I guess my thinking about why these are changing was because of concurrent requests, but if you're just doing one boot, I'm not sure | |
| 23:02:32 | melwitt | yeah, that's what I was trying to think in real life the retries would have a much better chance of succeeding bc not so many concentrated on one host, right | |
| 23:02:53 | dansmith | maybe the compute is tickling the inventory in some way, such that during 500 of them it gets changed | |
| 23:03:06 | melwitt | because the claim would detect the host full and then it would move on to another host | |
| 23:03:07 | dansmith | melwitt: yeah, I'm pretty confident it's related to the having of one compute and 500 instances | |
| 23:04:11 | melwitt | the thing I'm wondering is if the host list order is consistent, could this happen in real life because they'll all try the first host in the list first. but as things are claimed, that host will fill up and then no longer be considered | |
| 23:04:42 | mriedem | melwitt: i'm wondering the same | |
| 23:04:48 | mriedem | if we're not shuffling the hosts a bit | |
| 23:04:49 | melwitt | so won't get bombarded that badly? | |
| 23:05:56 | dansmith | mriedem: I want to know what the instance uuid is for each of those lines | |
| 23:06:11 | dansmith | mriedem: like maybe it retried once for a few instances, then three times for one and bailed the whole process | |
| 23:06:33 | dansmith | because any one of them gets to three retries and it should fail and stop the num_instances loop I think | |
| 23:06:58 | mriedem | 5f9f4f7d-8a2f-4fb8-b30a-024ed2e8e49d | |
| 23:07:21 | mriedem | oh...http://paste.openstack.org/show/621996/ | |
| 23:07:28 | mriedem | we don't have the instance uuid in that message | |
| 23:07:43 | dansmith | ...that's what I'm asking for yeah | |
| 23:07:47 | mriedem | ok, adding | |
| 23:07:56 | mriedem | that explains the 6 log messages | |
| 23:08:57 | dansmith | while you're re-testing, I'd like to suggest that we not hold up the instance list stuff on this if it takes too much longer | |
| 23:09:29 | dansmith | we can revert it if there's a real performance regression pretty easily, and we're delaying soak-time for any non-perf-related bugs we might be able to resolve just from our own infra workload | |
| 23:09:36 | mriedem | well, i wasted most of my day on this 500 novalidhost thing, | |
| 23:09:43 | mriedem | now i know i can create in 100 chunks | |
| 23:09:47 | dansmith | and we know we're not regressing the performance of the infra jobs that have run against this | |
| 23:09:54 | mriedem | so i'm at the point that i've got 500 in cell0 and 500 in cell1, and going to get numbers on those | |
| 23:12:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log instance uuid when retrying claims in the scheduler https://review.openstack.org/507705 | |
| 23:16:59 | openstackgerrit | Moshe Levi proposed openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/505613 | |
| 23:24:42 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Return Selection Objects https://review.openstack.org/498830 | |
| 23:25:48 | dansmith | I'm actually not sure why we'd be hitting concurrent updates during an allocation event, | |
| 23:25:54 | dansmith | given that we're not passing rp_generation | |
| 23:26:08 | dansmith | either the allocation fits or doesn't | |
| 23:27:39 | melwitt | hm, yeah | |
| 23:27:43 | dansmith | it must be that placement doesn't hide transaction commits from us | |
| 23:27:46 | dansmith | based on the commit that added it | |
| 23:28:12 | dansmith | which would just be "tons of churn on one provider" as the reason | |
| 23:29:57 | melwitt | what does that mean? placement not hiding transaction commits | |
| 23:31:14 | dansmith | an allocation will increment the generation of the provider on the placement side | |
| 23:31:23 | dansmith | and a transaction will abort if we try to do it at the same time as something else | |
| 23:31:26 | melwitt | ah, right. that's what I was thinking of | |
| 23:31:52 | dansmith | placement should really retry those for us server-side I would think, for an allocation type request | |
| 23:31:58 | melwitt | I couldn't remember where the RP generation was related there | |
| 23:32:22 | dansmith | doesn't matter, but I would think it would be better, especially given the advice for that error is "try exactly the same thing again" | |
| 23:32:24 | melwitt | yeah, I was thinking the same | |
| 23:32:29 | mriedem | this was the error from the server | |
| 23:32:30 | mriedem | There was a conflict when trying to complete your request.\n\n Inventory changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update | |
| 23:32:38 | dansmith | right | |
| 23:32:41 | mriedem | i'm not sure why inventory would change | |
| 23:32:45 | mriedem | that's static in this case | |
| 23:33:01 | dansmith | any allocation will change the generation | |
| 23:33:04 | mriedem | the update_available_resource periodic will post inventory, but only if it changes | |
| 23:33:19 | dansmith | so any two allocations can conflict | |
| 23:33:21 | melwitt | I'm +1 on the idea of retrying server-side | |
| 23:34:16 | melwitt | I'm a little worried how many conflicts can we get in real life by ppl trying to create 500 servers and the scheduler is doing a "pack" pattern | |
| 23:35:10 | melwitt | as far as how to tune how many retries | |
| 23:35:23 | melwitt | to allow | |
| 23:35:33 | dansmith | this is also a highly synthetic scenario with a "virt driver" that doesn't get looked at much.. it could be doing something to re-stab inventory for no reason or something | |
| 23:36:00 | mriedem | for each instance, we go through the filters | |
| 23:36:17 | mriedem | and doesn't the scheduler have some kind of tracking on the HostState objects themselves for chosen hosts? | |
| 23:36:22 | edleafe | mriedem: that error message is poorly worded. It should be something like "available inventory has changed" | |
| 23:36:25 | melwitt | oh, the "inventory changed" yeah, I don't know anything about that. so that means it wasn't an allocation writing conflict? | |
| 23:36:41 | dansmith | melwitt: it's al related | |
| 23:36:45 | dansmith | *all | |
| 23:37:07 | dansmith | edleafe: that's not really accurate, AFACT, since allocations will cause the generation to increase | |
| 23:37:16 | dansmith | edleafe: and thus it could be nothing changed with inventory to cause that | |
| 23:37:34 | edleafe | dansmith: uh, that's why I added "available". | |
| 23:37:46 | edleafe | dansmith: IOW, some inventory has been allocated | |
| 23:37:47 | mriedem | this is basically the inventory reporting for the fake driver https://github.com/openstack/nova/blob/master/nova/virt/fake.py#L111 | |
| 23:37:51 | edleafe | and changed the generation | |
| 23:38:01 | dansmith | edleafe: okay, I wouldn't word it that way for clarity, but okay :) | |
| 23:38:47 | edleafe | I wouldn't word it that way either, but I was guessing the author's intent | |
| 23:39:07 | edleafe | *cough* cdent *cough* | |
| 23:39:08 | dansmith | mriedem: that's the info that we use to generate it, yes | |
| 23:39:33 | melwitt | yeah, if it can happen without inventory (total possible capacity) changing, then that error message is confusing to me | |
| 23:40:05 | edleafe | melwitt: yeah, it's worded very poorly | |
| 23:40:10 | dansmith | mriedem: I'd look to see if the compute is hitting placement /inventory ever after the first go, and maybe check the nothing-changed short-circuit to make sure we're never going through it | |
| 23:40:32 | mriedem | the inventory nothing changed? | |
| 23:40:37 | melwitt | I dunno, I think I know just enough for it to be confusing. for an end user, it might not be confusing | |
| 23:40:47 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L571-L572 | |
| 23:40:56 | mriedem | melwitt: end user won't see it, they'll see NoValidHost on 500 instances | |
| 23:41:01 | mriedem | the operator will see it | |
| 23:41:04 | melwitt | good point | |
| 23:41:47 | dansmith | mriedem: or just look at placement logs to see if inventory is hit any time after compute startup | |
| 23:46:59 | mriedem | gdi, how do i regex search with grep | |
| 23:47:06 | mriedem | sudo journalctl -a -u devstack@placement-api.service | grep '.*PUT.*\/inventories.*' | |
| 23:48:12 | dansmith | PUT.*invent should be all you need | |
| 23:49:29 | mriedem | doesn't work | |
| 23:51:13 | mriedem | ah, well, | |
| 23:51:16 | mriedem | PUT.*alloc works | |
| 23:51:21 | mriedem | so it probably just wrapped | |
| 23:51:28 | mriedem | and it's not updating inventory, as it shouldn't | |
| 23:52:16 | mriedem | got my 1000 instances now, so will do the test stuff once i'm done with dinner | |
| 23:53:00 | dansmith | so I'd also check to make sure compute isn't doing the ocata healing during boot or something like that | |
| 23:57:18 | takashin | Spec cores, could you review https://review.openstack.org/#/c/489029/ ? It got one +2. | |
| #openstack-nova - 2017-09-27 | |||
| 00:13:05 | mriedem | dansmith: interesting, listing without details, 500 error (cell0) and 500 active (cell1) is a lot faster than the 1000 active, | |
| 00:13:13 | mriedem | i suppose because we don't have as much to join | |
| 00:13:27 | dansmith | mriedem: with my patch or before? | |
| 00:13:52 | mriedem | oh shit, nvm - copy paste error | |