Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-03
15:30:16 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Add flavor group https://review.opendev.org/663563
15:56:14 openstackgerrit Lenny Verkhovsky proposed openstack/os-vif master: WIP: Testing CI https://review.opendev.org/686937
17:15:37 openstackgerrit Merged openstack/nova stable/train: Imported Translations from Zanata https://review.opendev.org/692710
17:16:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.opendev.org/614353
17:16:48 openstackgerrit Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.opendev.org/651650
17:16:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478
17:16:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional test for anti-affinity cross-cell migration https://review.opendev.org/661859
17:16:50 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269
17:16:50 openstackgerrit Matt Riedemann proposed openstack/nova master: libvirt: flatten rbd image during cross-cell move spawn at dest https://review.opendev.org/691991
17:16:51 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add negative test to delete server during cross-cell resize claim https://review.opendev.org/688832
17:16:51 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656
#openstack-nova - 2019-11-04
01:25:58 gibi_ptg ~>
01:32:03 gibi_ptg stephenfin: hi! it would be nice to sync up about the nova project update today after 17:30
05:38:23 gibi_cn alex_xu: talked to stephenfin we will meet up at 14:00 in front of the Marketplace entrance
09:06:28 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Add flavor group https://review.opendev.org/663563
11:52:52 openstackgerrit Merged openstack/nova stable/train: Avoid error 500 on shelve task_state race https://review.opendev.org/692628
13:54:26 mriedem dansmith: so back in rocky the aggregate mirroring stuff was added with a todo to remove some placement failures in the api if you didn't have the api configured for placement (to ease upgrades),
13:54:40 mriedem i'm removing that handling now but there is one thing that is bugging me on the remove case,
13:54:41 mriedem https://review.opendev.org/#/c/660852/1/nova/compute/api.py@5615
13:54:58 mriedem which is the sync_aggregates command can't be used to sync and remove hosts from aggregates, only add
13:55:22 mriedem so i'm trying to decide if placement errors in the remove case should just be caught and logged or actually result in the compute API operation failing
13:55:56 mriedem i'm thinking the latter since otherwise it means we're just silently failing and your scheduling results could be screwed up, and it would be confusing to figure that out later
14:16:32 mriedem nevermind, going with the latter. it's a bigger diff but it's more correct i think
14:26:19 dansmith mriedem: okay can read that stuff in a sec
14:26:28 dansmith also, zomg my aggregates patch merged
14:26:35 dansmith er, notificatons I mean
14:29:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852
14:29:57 mriedem yeah a few things did over the weekend
14:30:00 mriedem it's a miracle
14:30:17 cdent praise be
14:31:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852
14:31:42 mriedem cdent: i finally got that cleaned up ^ what i had sitting didn't ... sit well with me
14:31:55 mriedem because it was like, "oh something failed, wah, nice try retrying it sucker"
14:33:54 cdent a good lesson for entitled whippersnappers, I say
14:34:03 cdent learn em up about life
14:34:25 mriedem back in my day we didn't have idempotent apis
14:34:33 mriedem we had to fight to delete things
14:34:42 cdent exactly so
14:35:05 cdent What has idempotency got us? Social media addiction!
14:36:12 mriedem that's a bit of a stretch, even for me
14:41:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Log reason for remove_host action failing https://review.opendev.org/692833
14:43:37 mriedem dansmith: i know it's a bit big and not very fun to review, but the bottom few patches of the cross-cell-resize series are ready for review and +2ed by gibi if you get some time to sift through one or two of those
14:44:02 dansmith sure, I need to get back to that, sorry
14:44:07 mriedem np
14:46:37 mriedem huh, maybe another case for long_rpc_timeout - for reserve_block_device_name call to compute when attaching a volume https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4
14:46:42 mriedem we must have some really slow nodes in the gate
14:54:50 mriedem inap and ovh again
14:54:56 mriedem just like in the 'state of the gate' email
14:57:14 efried mriedem: is https://review.opendev.org/#/c/692550/ (long_rpc_timeout) going to manifest in the gate anywhere in a way it can be seen easily? I would think not; we would have to get lucky and see an operation that takes longer than the old timeout.
14:57:26 efried just wondering if we're waiting to see anything in particular before approving ^
14:58:03 mriedem not likely, the timeout is 30 minutes by default
14:58:11 mriedem tempest would timeout way before that
14:58:42 efried I don't mean hitting the long one, I mean seeing an operation that takes somewhere *between* the old timeout and the new one (i.e. would have timed out before this change but doesn't now)
15:00:13 efried anyway, I'm getting ready to +2 that one, unless you or dansmith tells me there's a reason not to.
15:00:39 mriedem i noticed it because of a gate failure http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010494.html
15:00:41 mriedem if that's what you're asking
15:01:09 dansmith efried: I can look after I'm done with a review
15:01:13 mriedem api response timed out waiting for POST /allocations to finish which took 3 minutes b/c we had no i/o on the node
15:01:31 mriedem ^ is not realistic though, and i'm not sure why that would happen in the gate
15:01:37 efried right, my point is, in order to see it "working", we would have to find that same situation again, but where it *didn't* timeout, but took longer than the original timeout.
15:01:41 mriedem as cdent said in the ML, what do we do if this happens?
15:02:04 mriedem efried: that is probably not very possible without adding some logging that logs if an operation took over a given threshold,
15:02:05 efried yeah, seems like we've ascertained the I/O thing is just overloading.
15:02:24 mriedem i'm not sure if there is logging in oslo.messaging that goes off if an operation is over the given 60 second heartbeat, but there probably is
15:02:54 efried It's not worth it, I was just asking whether there would be any other way to tell. Doesn't sound like it. But that's not a reason not to merge the thing.
15:03:14 efried The harm would be, if something is wrong in such a way that it's never going to come back, you've just wasted time.
15:03:19 efried but such is the eternal dilemma of timeouts.
15:04:07 efried Re the I/O thing, perhaps it would benefit us to have fewer zuul nodes if it means a lower rate of spurious failure.
15:04:33 mriedem i see this in oslo.messaging but it's debug level which we don't index https://github.com/openstack/oslo.messaging/blob/6bca848f5b272149bb32353a62f9d37108fcbe15/oslo_messaging/_drivers/amqpdriver.py#L533
15:04:49 mriedem what do you mean by fewer zuul nodes?
15:05:36 efried or, if what cdent suggested would work, if making the CI nodes less CPU-powerful would reduce their ability to burden the I/O. Same result, really: overall fewer runs occur in a given amount of time, but if the success rate increases, it's a win.
15:06:00 efried mriedem: I mean, if the I/O is overloaded, try to make it less overloaded by reducing the number of things hitting it at a time.
15:06:01 mriedem reducing cpu is likely going to cause failures in other ways i'd think
15:06:26 mriedem historically cpu/ram has only gotten higher in devstack nodes in the gate over time
15:06:48 mriedem when i started in openstack a full dsvm-tempest job took 45 minutes and could use 4vcpu/gb ram
15:07:12 mriedem now we're basically topped out at 8cpu/ram with 2 api workers per control service
15:07:18 dansmith efried: oh yeah that migrate_server one, mriedem and I discussed last week, so probably good for a +2
15:07:31 efried dansmith: that's done, waiting for your +A
15:07:38 dansmith yup yup
15:07:38 mriedem we could try dropping API_WORKERS=1 in devstack if that would help with load
15:07:56 mriedem but we might hit more timeouts with that, idk
15:08:08 efried mriedem: except that would slow down everywhere
15:08:19 dansmith mriedem: might have to lower tempest parallelism too
15:08:19 efried It seems like we're hitting this problem on specific providers, yes?
15:08:28 mriedem efried: yes, inap and ovh
15:08:35 mriedem dansmith: i thought about that last week as well,
15:08:36 efried So I'm looking for an answer that would affect those providers without impacting others.
15:08:43 mriedem currently we're 4 tests at a time
15:08:48 dansmith mriedem: I just mean if you reduce the api workers
15:09:21 mriedem yeah...so if node_provider in inap/ovh, API_WORKERS=1, tempest concurrency=2, build_timeout * 2
15:09:42 mriedem or just get them to stop throttling disk io so much
15:09:44 dansmith I dunno how large those pools are, but could we just run unit, functional, doc, etc jobs on those providers?
15:09:54 dansmith I think we've done that kind of job-provider affinity before
15:09:59 mriedem that's not a bad idea,
15:10:05 mriedem and yeah i think provider job affinity has been done for baremetal jobs
15:10:09 dansmith yeah
15:10:21 dansmith if they ran *all* those kinds of jobs, the others could focus on the devstack jobs
15:11:09 efried I want to say the infra guys have poo-poohed that idea in the past
15:11:56 mriedem i don't know how easy it would be
15:12:03 mriedem it's likely not easy

Earlier   Later