| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-04 | |||
| 13:55:56 | mriedem | i'm thinking the latter since otherwise it means we're just silently failing and your scheduling results could be screwed up, and it would be confusing to figure that out later | |
| 14:16:32 | mriedem | nevermind, going with the latter. it's a bigger diff but it's more correct i think | |
| 14:26:19 | dansmith | mriedem: okay can read that stuff in a sec | |
| 14:26:28 | dansmith | also, zomg my aggregates patch merged | |
| 14:26:35 | dansmith | er, notificatons I mean | |
| 14:29:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852 | |
| 14:29:57 | mriedem | yeah a few things did over the weekend | |
| 14:30:00 | mriedem | it's a miracle | |
| 14:30:17 | cdent | praise be | |
| 14:31:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852 | |
| 14:31:42 | mriedem | cdent: i finally got that cleaned up ^ what i had sitting didn't ... sit well with me | |
| 14:31:55 | mriedem | because it was like, "oh something failed, wah, nice try retrying it sucker" | |
| 14:33:54 | cdent | a good lesson for entitled whippersnappers, I say | |
| 14:34:03 | cdent | learn em up about life | |
| 14:34:25 | mriedem | back in my day we didn't have idempotent apis | |
| 14:34:33 | mriedem | we had to fight to delete things | |
| 14:34:42 | cdent | exactly so | |
| 14:35:05 | cdent | What has idempotency got us? Social media addiction! | |
| 14:36:12 | mriedem | that's a bit of a stretch, even for me | |
| 14:41:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log reason for remove_host action failing https://review.opendev.org/692833 | |
| 14:43:37 | mriedem | dansmith: i know it's a bit big and not very fun to review, but the bottom few patches of the cross-cell-resize series are ready for review and +2ed by gibi if you get some time to sift through one or two of those | |
| 14:44:02 | dansmith | sure, I need to get back to that, sorry | |
| 14:44:07 | mriedem | np | |
| 14:46:37 | mriedem | huh, maybe another case for long_rpc_timeout - for reserve_block_device_name call to compute when attaching a volume https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4 | |
| 14:46:42 | mriedem | we must have some really slow nodes in the gate | |
| 14:54:50 | mriedem | inap and ovh again | |
| 14:54:56 | mriedem | just like in the 'state of the gate' email | |
| 14:57:14 | efried | mriedem: is https://review.opendev.org/#/c/692550/ (long_rpc_timeout) going to manifest in the gate anywhere in a way it can be seen easily? I would think not; we would have to get lucky and see an operation that takes longer than the old timeout. | |
| 14:57:26 | efried | just wondering if we're waiting to see anything in particular before approving ^ | |
| 14:58:03 | mriedem | not likely, the timeout is 30 minutes by default | |
| 14:58:11 | mriedem | tempest would timeout way before that | |
| 14:58:42 | efried | I don't mean hitting the long one, I mean seeing an operation that takes somewhere *between* the old timeout and the new one (i.e. would have timed out before this change but doesn't now) | |
| 15:00:13 | efried | anyway, I'm getting ready to +2 that one, unless you or dansmith tells me there's a reason not to. | |
| 15:00:39 | mriedem | i noticed it because of a gate failure http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010494.html | |
| 15:00:41 | mriedem | if that's what you're asking | |
| 15:01:09 | dansmith | efried: I can look after I'm done with a review | |
| 15:01:13 | mriedem | api response timed out waiting for POST /allocations to finish which took 3 minutes b/c we had no i/o on the node | |
| 15:01:31 | mriedem | ^ is not realistic though, and i'm not sure why that would happen in the gate | |
| 15:01:37 | efried | right, my point is, in order to see it "working", we would have to find that same situation again, but where it *didn't* timeout, but took longer than the original timeout. | |
| 15:01:41 | mriedem | as cdent said in the ML, what do we do if this happens? | |
| 15:02:04 | mriedem | efried: that is probably not very possible without adding some logging that logs if an operation took over a given threshold, | |
| 15:02:05 | efried | yeah, seems like we've ascertained the I/O thing is just overloading. | |
| 15:02:24 | mriedem | i'm not sure if there is logging in oslo.messaging that goes off if an operation is over the given 60 second heartbeat, but there probably is | |
| 15:02:54 | efried | It's not worth it, I was just asking whether there would be any other way to tell. Doesn't sound like it. But that's not a reason not to merge the thing. | |
| 15:03:14 | efried | The harm would be, if something is wrong in such a way that it's never going to come back, you've just wasted time. | |
| 15:03:19 | efried | but such is the eternal dilemma of timeouts. | |
| 15:04:07 | efried | Re the I/O thing, perhaps it would benefit us to have fewer zuul nodes if it means a lower rate of spurious failure. | |
| 15:04:33 | mriedem | i see this in oslo.messaging but it's debug level which we don't index https://github.com/openstack/oslo.messaging/blob/6bca848f5b272149bb32353a62f9d37108fcbe15/oslo_messaging/_drivers/amqpdriver.py#L533 | |
| 15:04:49 | mriedem | what do you mean by fewer zuul nodes? | |
| 15:05:36 | efried | or, if what cdent suggested would work, if making the CI nodes less CPU-powerful would reduce their ability to burden the I/O. Same result, really: overall fewer runs occur in a given amount of time, but if the success rate increases, it's a win. | |
| 15:06:00 | efried | mriedem: I mean, if the I/O is overloaded, try to make it less overloaded by reducing the number of things hitting it at a time. | |
| 15:06:01 | mriedem | reducing cpu is likely going to cause failures in other ways i'd think | |
| 15:06:26 | mriedem | historically cpu/ram has only gotten higher in devstack nodes in the gate over time | |
| 15:06:48 | mriedem | when i started in openstack a full dsvm-tempest job took 45 minutes and could use 4vcpu/gb ram | |
| 15:07:12 | mriedem | now we're basically topped out at 8cpu/ram with 2 api workers per control service | |
| 15:07:18 | dansmith | efried: oh yeah that migrate_server one, mriedem and I discussed last week, so probably good for a +2 | |
| 15:07:31 | efried | dansmith: that's done, waiting for your +A | |
| 15:07:38 | dansmith | yup yup | |
| 15:07:38 | mriedem | we could try dropping API_WORKERS=1 in devstack if that would help with load | |
| 15:07:56 | mriedem | but we might hit more timeouts with that, idk | |
| 15:08:08 | efried | mriedem: except that would slow down everywhere | |
| 15:08:19 | dansmith | mriedem: might have to lower tempest parallelism too | |
| 15:08:19 | efried | It seems like we're hitting this problem on specific providers, yes? | |
| 15:08:28 | mriedem | efried: yes, inap and ovh | |
| 15:08:35 | mriedem | dansmith: i thought about that last week as well, | |
| 15:08:36 | efried | So I'm looking for an answer that would affect those providers without impacting others. | |
| 15:08:43 | mriedem | currently we're 4 tests at a time | |
| 15:08:48 | dansmith | mriedem: I just mean if you reduce the api workers | |
| 15:09:21 | mriedem | yeah...so if node_provider in inap/ovh, API_WORKERS=1, tempest concurrency=2, build_timeout * 2 | |
| 15:09:42 | mriedem | or just get them to stop throttling disk io so much | |
| 15:09:44 | dansmith | I dunno how large those pools are, but could we just run unit, functional, doc, etc jobs on those providers? | |
| 15:09:54 | dansmith | I think we've done that kind of job-provider affinity before | |
| 15:09:59 | mriedem | that's not a bad idea, | |
| 15:10:05 | mriedem | and yeah i think provider job affinity has been done for baremetal jobs | |
| 15:10:09 | dansmith | yeah | |
| 15:10:21 | dansmith | if they ran *all* those kinds of jobs, the others could focus on the devstack jobs | |
| 15:11:09 | efried | I want to say the infra guys have poo-poohed that idea in the past | |
| 15:11:56 | mriedem | i don't know how easy it would be | |
| 15:12:03 | mriedem | it's likely not easy | |
| 15:12:38 | efried | it was this thread I was thinking of most recently, but I don't see the response http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009595.html | |
| 15:12:48 | mriedem | i'm assuming if we had a kind of node provider label for that kind of thing, then projects like nova could opt into which pool of providers each job runs in | |
| 15:12:50 | dansmith | well, if we have some affinity today, I would think it'd be doable | |
| 15:13:11 | mriedem | because some functional jobs do use devstack in some projects | |
| 15:13:23 | mriedem | so i'd think this kind of thing has to be up to each project since they know the jobs they run | |
| 15:13:23 | dansmith | ah | |
| 15:14:06 | efried | http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009592.html | |
| 15:14:26 | efried | also no response, hm | |
| 15:14:48 | dansmith | ah yeah, that's exactly it | |
| 15:19:09 | mriedem | likely need to tag with infra or ask -infra people directly to comment | |
| 15:19:18 | mriedem | and they are probably all in shanghai this week | |
| 15:19:48 | efried | yeah | |
| 15:19:59 | efried | fwiw it looks like ovn has ~200 nodes | |
| 15:20:07 | dansmith | ovh? | |
| 15:20:44 | efried | isn't that the provider that's been choking? | |
| 15:20:49 | efried | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 | |
| 15:20:51 | dansmith | you said ovn | |
| 15:20:55 | efried | oh, sorry | |
| 15:20:58 | efried | yes, ovh | |
| 15:21:01 | dansmith | which is a network technology, so just making sure :) | |
| 15:21:09 | mriedem | ez bake ovn | |