Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-04
14:34:42 cdent exactly so
14:35:05 cdent What has idempotency got us? Social media addiction!
14:36:12 mriedem that's a bit of a stretch, even for me
14:41:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Log reason for remove_host action failing https://review.opendev.org/692833
14:43:37 mriedem dansmith: i know it's a bit big and not very fun to review, but the bottom few patches of the cross-cell-resize series are ready for review and +2ed by gibi if you get some time to sift through one or two of those
14:44:02 dansmith sure, I need to get back to that, sorry
14:44:07 mriedem np
14:46:37 mriedem huh, maybe another case for long_rpc_timeout - for reserve_block_device_name call to compute when attaching a volume https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4
14:46:42 mriedem we must have some really slow nodes in the gate
14:54:50 mriedem inap and ovh again
14:54:56 mriedem just like in the 'state of the gate' email
14:57:14 efried mriedem: is https://review.opendev.org/#/c/692550/ (long_rpc_timeout) going to manifest in the gate anywhere in a way it can be seen easily? I would think not; we would have to get lucky and see an operation that takes longer than the old timeout.
14:57:26 efried just wondering if we're waiting to see anything in particular before approving ^
14:58:03 mriedem not likely, the timeout is 30 minutes by default
14:58:11 mriedem tempest would timeout way before that
14:58:42 efried I don't mean hitting the long one, I mean seeing an operation that takes somewhere *between* the old timeout and the new one (i.e. would have timed out before this change but doesn't now)
15:00:13 efried anyway, I'm getting ready to +2 that one, unless you or dansmith tells me there's a reason not to.
15:00:39 mriedem i noticed it because of a gate failure http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010494.html
15:00:41 mriedem if that's what you're asking
15:01:09 dansmith efried: I can look after I'm done with a review
15:01:13 mriedem api response timed out waiting for POST /allocations to finish which took 3 minutes b/c we had no i/o on the node
15:01:31 mriedem ^ is not realistic though, and i'm not sure why that would happen in the gate
15:01:37 efried right, my point is, in order to see it "working", we would have to find that same situation again, but where it *didn't* timeout, but took longer than the original timeout.
15:01:41 mriedem as cdent said in the ML, what do we do if this happens?
15:02:04 mriedem efried: that is probably not very possible without adding some logging that logs if an operation took over a given threshold,
15:02:05 efried yeah, seems like we've ascertained the I/O thing is just overloading.
15:02:24 mriedem i'm not sure if there is logging in oslo.messaging that goes off if an operation is over the given 60 second heartbeat, but there probably is
15:02:54 efried It's not worth it, I was just asking whether there would be any other way to tell. Doesn't sound like it. But that's not a reason not to merge the thing.
15:03:14 efried The harm would be, if something is wrong in such a way that it's never going to come back, you've just wasted time.
15:03:19 efried but such is the eternal dilemma of timeouts.
15:04:07 efried Re the I/O thing, perhaps it would benefit us to have fewer zuul nodes if it means a lower rate of spurious failure.
15:04:33 mriedem i see this in oslo.messaging but it's debug level which we don't index https://github.com/openstack/oslo.messaging/blob/6bca848f5b272149bb32353a62f9d37108fcbe15/oslo_messaging/_drivers/amqpdriver.py#L533
15:04:49 mriedem what do you mean by fewer zuul nodes?
15:05:36 efried or, if what cdent suggested would work, if making the CI nodes less CPU-powerful would reduce their ability to burden the I/O. Same result, really: overall fewer runs occur in a given amount of time, but if the success rate increases, it's a win.
15:06:00 efried mriedem: I mean, if the I/O is overloaded, try to make it less overloaded by reducing the number of things hitting it at a time.
15:06:01 mriedem reducing cpu is likely going to cause failures in other ways i'd think
15:06:26 mriedem historically cpu/ram has only gotten higher in devstack nodes in the gate over time
15:06:48 mriedem when i started in openstack a full dsvm-tempest job took 45 minutes and could use 4vcpu/gb ram
15:07:12 mriedem now we're basically topped out at 8cpu/ram with 2 api workers per control service
15:07:18 dansmith efried: oh yeah that migrate_server one, mriedem and I discussed last week, so probably good for a +2
15:07:31 efried dansmith: that's done, waiting for your +A
15:07:38 mriedem we could try dropping API_WORKERS=1 in devstack if that would help with load
15:07:38 dansmith yup yup
15:07:56 mriedem but we might hit more timeouts with that, idk
15:08:08 efried mriedem: except that would slow down everywhere
15:08:19 efried It seems like we're hitting this problem on specific providers, yes?
15:08:19 dansmith mriedem: might have to lower tempest parallelism too
15:08:28 mriedem efried: yes, inap and ovh
15:08:35 mriedem dansmith: i thought about that last week as well,
15:08:36 efried So I'm looking for an answer that would affect those providers without impacting others.
15:08:43 mriedem currently we're 4 tests at a time
15:08:48 dansmith mriedem: I just mean if you reduce the api workers
15:09:21 mriedem yeah...so if node_provider in inap/ovh, API_WORKERS=1, tempest concurrency=2, build_timeout * 2
15:09:42 mriedem or just get them to stop throttling disk io so much
15:09:44 dansmith I dunno how large those pools are, but could we just run unit, functional, doc, etc jobs on those providers?
15:09:54 dansmith I think we've done that kind of job-provider affinity before
15:09:59 mriedem that's not a bad idea,
15:10:05 mriedem and yeah i think provider job affinity has been done for baremetal jobs
15:10:09 dansmith yeah
15:10:21 dansmith if they ran *all* those kinds of jobs, the others could focus on the devstack jobs
15:11:09 efried I want to say the infra guys have poo-poohed that idea in the past
15:11:56 mriedem i don't know how easy it would be
15:12:03 mriedem it's likely not easy
15:12:38 efried it was this thread I was thinking of most recently, but I don't see the response http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009595.html
15:12:48 mriedem i'm assuming if we had a kind of node provider label for that kind of thing, then projects like nova could opt into which pool of providers each job runs in
15:12:50 dansmith well, if we have some affinity today, I would think it'd be doable
15:13:11 mriedem because some functional jobs do use devstack in some projects
15:13:23 dansmith ah
15:13:23 mriedem so i'd think this kind of thing has to be up to each project since they know the jobs they run
15:14:06 efried http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009592.html
15:14:26 efried also no response, hm
15:14:48 dansmith ah yeah, that's exactly it
15:19:09 mriedem likely need to tag with infra or ask -infra people directly to comment
15:19:18 mriedem and they are probably all in shanghai this week
15:19:48 efried yeah
15:19:59 efried fwiw it looks like ovn has ~200 nodes
15:20:07 dansmith ovh?
15:20:44 efried isn't that the provider that's been choking?
15:20:49 efried http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1
15:20:51 dansmith you said ovn
15:20:55 efried oh, sorry
15:20:58 efried yes, ovh
15:21:01 dansmith which is a network technology, so just making sure :)
15:21:09 mriedem ez bake ovn
15:21:25 efried taking mriedem back to his childhood
15:21:39 mriedem dude i'm making some muffins in the thing right now
15:22:01 efried Takes *way* longer with these stupid CFLs
15:22:14 dansmith pretty sure mriedem never left childhood
15:23:34 mriedem i will say, since maya likes legos i do find myself tempted around xmas to splurge on a $150 lego set just so i can help build it
15:23:52 dansmith nice
15:24:08 efried donnyd: can you think of an easy way $nodepool_provider could switch up configuration to reduce the chance of choking I/O?
15:24:51 donnyd On FN?
15:24:58 mriedem "but dad i only like the friends and disney princess sets" "you're getting medieval castle gdi"
15:25:05 efried specifically ovh seems to be the problem at the moment.
15:25:06 dansmith mriedem: lol
15:25:47 donnyd Oh, well I know I avoid that issue by using local storage on FN. Not sure what ovh has on the backend of their instances
15:26:10 efried not asking for action on your part donnyd, just advice. If a patch has 10 zuul jobs an any one of them lands on ovh and chokes on I/O and times out, the whole patch has to be retried, which is a royal PITA. Been happening a really lot over the last week or two.
15:26:43 efried Yeah, I guess they would need to look into where the actual bottleneck is.
15:26:53 efried not sure if there's any way to tell from here via grafana...
15:26:55 donnyd The reality is we should try and label jobs by what they are bound by

Earlier   Later