| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-04 | |||
| 14:44:02 | dansmith | sure, I need to get back to that, sorry | |
| 14:44:07 | mriedem | np | |
| 14:46:37 | mriedem | huh, maybe another case for long_rpc_timeout - for reserve_block_device_name call to compute when attaching a volume https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4 | |
| 14:46:42 | mriedem | we must have some really slow nodes in the gate | |
| 14:54:50 | mriedem | inap and ovh again | |
| 14:54:56 | mriedem | just like in the 'state of the gate' email | |
| 14:57:14 | efried | mriedem: is https://review.opendev.org/#/c/692550/ (long_rpc_timeout) going to manifest in the gate anywhere in a way it can be seen easily? I would think not; we would have to get lucky and see an operation that takes longer than the old timeout. | |
| 14:57:26 | efried | just wondering if we're waiting to see anything in particular before approving ^ | |
| 14:58:03 | mriedem | not likely, the timeout is 30 minutes by default | |
| 14:58:11 | mriedem | tempest would timeout way before that | |
| 14:58:42 | efried | I don't mean hitting the long one, I mean seeing an operation that takes somewhere *between* the old timeout and the new one (i.e. would have timed out before this change but doesn't now) | |
| 15:00:13 | efried | anyway, I'm getting ready to +2 that one, unless you or dansmith tells me there's a reason not to. | |
| 15:00:39 | mriedem | i noticed it because of a gate failure http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010494.html | |
| 15:00:41 | mriedem | if that's what you're asking | |
| 15:01:09 | dansmith | efried: I can look after I'm done with a review | |
| 15:01:13 | mriedem | api response timed out waiting for POST /allocations to finish which took 3 minutes b/c we had no i/o on the node | |
| 15:01:31 | mriedem | ^ is not realistic though, and i'm not sure why that would happen in the gate | |
| 15:01:37 | efried | right, my point is, in order to see it "working", we would have to find that same situation again, but where it *didn't* timeout, but took longer than the original timeout. | |
| 15:01:41 | mriedem | as cdent said in the ML, what do we do if this happens? | |
| 15:02:04 | mriedem | efried: that is probably not very possible without adding some logging that logs if an operation took over a given threshold, | |
| 15:02:05 | efried | yeah, seems like we've ascertained the I/O thing is just overloading. | |
| 15:02:24 | mriedem | i'm not sure if there is logging in oslo.messaging that goes off if an operation is over the given 60 second heartbeat, but there probably is | |
| 15:02:54 | efried | It's not worth it, I was just asking whether there would be any other way to tell. Doesn't sound like it. But that's not a reason not to merge the thing. | |
| 15:03:14 | efried | The harm would be, if something is wrong in such a way that it's never going to come back, you've just wasted time. | |
| 15:03:19 | efried | but such is the eternal dilemma of timeouts. | |
| 15:04:07 | efried | Re the I/O thing, perhaps it would benefit us to have fewer zuul nodes if it means a lower rate of spurious failure. | |
| 15:04:33 | mriedem | i see this in oslo.messaging but it's debug level which we don't index https://github.com/openstack/oslo.messaging/blob/6bca848f5b272149bb32353a62f9d37108fcbe15/oslo_messaging/_drivers/amqpdriver.py#L533 | |
| 15:04:49 | mriedem | what do you mean by fewer zuul nodes? | |
| 15:05:36 | efried | or, if what cdent suggested would work, if making the CI nodes less CPU-powerful would reduce their ability to burden the I/O. Same result, really: overall fewer runs occur in a given amount of time, but if the success rate increases, it's a win. | |
| 15:06:00 | efried | mriedem: I mean, if the I/O is overloaded, try to make it less overloaded by reducing the number of things hitting it at a time. | |
| 15:06:01 | mriedem | reducing cpu is likely going to cause failures in other ways i'd think | |
| 15:06:26 | mriedem | historically cpu/ram has only gotten higher in devstack nodes in the gate over time | |
| 15:06:48 | mriedem | when i started in openstack a full dsvm-tempest job took 45 minutes and could use 4vcpu/gb ram | |
| 15:07:12 | mriedem | now we're basically topped out at 8cpu/ram with 2 api workers per control service | |
| 15:07:18 | dansmith | efried: oh yeah that migrate_server one, mriedem and I discussed last week, so probably good for a +2 | |
| 15:07:31 | efried | dansmith: that's done, waiting for your +A | |
| 15:07:38 | mriedem | we could try dropping API_WORKERS=1 in devstack if that would help with load | |
| 15:07:38 | dansmith | yup yup | |
| 15:07:56 | mriedem | but we might hit more timeouts with that, idk | |
| 15:08:08 | efried | mriedem: except that would slow down everywhere | |
| 15:08:19 | efried | It seems like we're hitting this problem on specific providers, yes? | |
| 15:08:19 | dansmith | mriedem: might have to lower tempest parallelism too | |
| 15:08:28 | mriedem | efried: yes, inap and ovh | |
| 15:08:35 | mriedem | dansmith: i thought about that last week as well, | |
| 15:08:36 | efried | So I'm looking for an answer that would affect those providers without impacting others. | |
| 15:08:43 | mriedem | currently we're 4 tests at a time | |
| 15:08:48 | dansmith | mriedem: I just mean if you reduce the api workers | |
| 15:09:21 | mriedem | yeah...so if node_provider in inap/ovh, API_WORKERS=1, tempest concurrency=2, build_timeout * 2 | |
| 15:09:42 | mriedem | or just get them to stop throttling disk io so much | |
| 15:09:44 | dansmith | I dunno how large those pools are, but could we just run unit, functional, doc, etc jobs on those providers? | |
| 15:09:54 | dansmith | I think we've done that kind of job-provider affinity before | |
| 15:09:59 | mriedem | that's not a bad idea, | |
| 15:10:05 | mriedem | and yeah i think provider job affinity has been done for baremetal jobs | |
| 15:10:09 | dansmith | yeah | |
| 15:10:21 | dansmith | if they ran *all* those kinds of jobs, the others could focus on the devstack jobs | |
| 15:11:09 | efried | I want to say the infra guys have poo-poohed that idea in the past | |
| 15:11:56 | mriedem | i don't know how easy it would be | |
| 15:12:03 | mriedem | it's likely not easy | |
| 15:12:38 | efried | it was this thread I was thinking of most recently, but I don't see the response http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009595.html | |
| 15:12:48 | mriedem | i'm assuming if we had a kind of node provider label for that kind of thing, then projects like nova could opt into which pool of providers each job runs in | |
| 15:12:50 | dansmith | well, if we have some affinity today, I would think it'd be doable | |
| 15:13:11 | mriedem | because some functional jobs do use devstack in some projects | |
| 15:13:23 | dansmith | ah | |
| 15:13:23 | mriedem | so i'd think this kind of thing has to be up to each project since they know the jobs they run | |
| 15:14:06 | efried | http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009592.html | |
| 15:14:26 | efried | also no response, hm | |
| 15:14:48 | dansmith | ah yeah, that's exactly it | |
| 15:19:09 | mriedem | likely need to tag with infra or ask -infra people directly to comment | |
| 15:19:18 | mriedem | and they are probably all in shanghai this week | |
| 15:19:48 | efried | yeah | |
| 15:19:59 | efried | fwiw it looks like ovn has ~200 nodes | |
| 15:20:07 | dansmith | ovh? | |
| 15:20:44 | efried | isn't that the provider that's been choking? | |
| 15:20:49 | efried | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 | |
| 15:20:51 | dansmith | you said ovn | |
| 15:20:55 | efried | oh, sorry | |
| 15:20:58 | efried | yes, ovh | |
| 15:21:01 | dansmith | which is a network technology, so just making sure :) | |
| 15:21:09 | mriedem | ez bake ovn | |
| 15:21:25 | efried | taking mriedem back to his childhood | |
| 15:21:39 | mriedem | dude i'm making some muffins in the thing right now | |
| 15:22:01 | efried | Takes *way* longer with these stupid CFLs | |
| 15:22:14 | dansmith | pretty sure mriedem never left childhood | |
| 15:23:34 | mriedem | i will say, since maya likes legos i do find myself tempted around xmas to splurge on a $150 lego set just so i can help build it | |
| 15:23:52 | dansmith | nice | |
| 15:24:08 | efried | donnyd: can you think of an easy way $nodepool_provider could switch up configuration to reduce the chance of choking I/O? | |
| 15:24:51 | donnyd | On FN? | |
| 15:24:58 | mriedem | "but dad i only like the friends and disney princess sets" "you're getting medieval castle gdi" | |
| 15:25:05 | efried | specifically ovh seems to be the problem at the moment. | |
| 15:25:06 | dansmith | mriedem: lol | |
| 15:25:47 | donnyd | Oh, well I know I avoid that issue by using local storage on FN. Not sure what ovh has on the backend of their instances | |
| 15:26:10 | efried | not asking for action on your part donnyd, just advice. If a patch has 10 zuul jobs an any one of them lands on ovh and chokes on I/O and times out, the whole patch has to be retried, which is a royal PITA. Been happening a really lot over the last week or two. | |
| 15:26:43 | efried | Yeah, I guess they would need to look into where the actual bottleneck is. | |
| 15:26:53 | efried | not sure if there's any way to tell from here via grafana... | |
| 15:26:55 | donnyd | The reality is we should try and label jobs by what they are bound by | |
| 15:27:09 | efried | heh, we were just talking about your email from Sept. | |
| 15:27:25 | donnyd | If there is a cpu bound job, scheduling on FN would be less optimal | |
| 15:27:44 | efried | I couldn't see anywhere infra had responded to that idea, but I thought I remembered them shutting it down hard for some reason. | |
| 15:27:50 | cdent | efried, mriedem : what I meant by less cpu, was less cpu for all nodes, not per node | |
| 15:27:51 | donnyd | But IO bound jobs will go like stink on FN | |