Posted	Nick	Remark
#openstack-nova - 2019-11-02
23:57:43	openstackgerrit	Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592
23:57:44	openstackgerrit	Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012
23:57:44	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451
23:57:45	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058
23:57:45	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013
23:57:46	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075
23:57:46	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add ConfirmResizeTask https://review.opendev.org/637070
23:57:47	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630
23:57:47	openstackgerrit	Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316
23:57:48	openstackgerrit	Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453
#openstack-nova - 2019-11-03
06:27:05	openstackgerrit	OpenStack Proposal Bot proposed openstack/nova stable/stein: Imported Translations from Zanata https://review.opendev.org/692701
06:48:32	openstackgerrit	Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243
06:48:32	openstackgerrit	Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242
06:48:33	openstackgerrit	Sundar Nadathur proposed openstack/nova master: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.opendev.org/631245
06:48:33	openstackgerrit	Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
06:48:34	openstackgerrit	Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
06:48:34	openstackgerrit	Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
06:48:35	openstackgerrit	Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707
06:51:50	openstackgerrit	OpenStack Proposal Bot proposed openstack/nova stable/train: Imported Translations from Zanata https://review.opendev.org/692710
14:08:27	openstackgerrit	Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070
15:30:16	openstackgerrit	Brin Zhang proposed openstack/nova-specs master: Add flavor group https://review.opendev.org/663563
15:56:14	openstackgerrit	Lenny Verkhovsky proposed openstack/os-vif master: WIP: Testing CI https://review.opendev.org/686937
17:15:37	openstackgerrit	Merged openstack/nova stable/train: Imported Translations from Zanata https://review.opendev.org/692710
17:16:48	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.opendev.org/614353
17:16:48	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.opendev.org/651650
17:16:49	openstackgerrit	Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478
17:16:49	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add functional test for anti-affinity cross-cell migration https://review.opendev.org/661859
17:16:50	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269
17:16:50	openstackgerrit	Matt Riedemann proposed openstack/nova master: libvirt: flatten rbd image during cross-cell move spawn at dest https://review.opendev.org/691991
17:16:51	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Add negative test to delete server during cross-cell resize claim https://review.opendev.org/688832
17:16:51	openstackgerrit	Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656
#openstack-nova - 2019-11-04
01:25:58	gibi_ptg	~>
01:32:03	gibi_ptg	stephenfin: hi! it would be nice to sync up about the nova project update today after 17:30
05:38:23	gibi_cn	alex_xu: talked to stephenfin we will meet up at 14:00 in front of the Marketplace entrance
09:06:28	openstackgerrit	Brin Zhang proposed openstack/nova-specs master: Add flavor group https://review.opendev.org/663563
11:52:52	openstackgerrit	Merged openstack/nova stable/train: Avoid error 500 on shelve task_state race https://review.opendev.org/692628
13:54:26	mriedem	dansmith: so back in rocky the aggregate mirroring stuff was added with a todo to remove some placement failures in the api if you didn't have the api configured for placement (to ease upgrades),
13:54:40	mriedem	i'm removing that handling now but there is one thing that is bugging me on the remove case,
13:54:41	mriedem	https://review.opendev.org/#/c/660852/1/nova/compute/api.py@5615
13:54:58	mriedem	which is the sync_aggregates command can't be used to sync and remove hosts from aggregates, only add
13:55:22	mriedem	so i'm trying to decide if placement errors in the remove case should just be caught and logged or actually result in the compute API operation failing
13:55:56	mriedem	i'm thinking the latter since otherwise it means we're just silently failing and your scheduling results could be screwed up, and it would be confusing to figure that out later
14:16:32	mriedem	nevermind, going with the latter. it's a bigger diff but it's more correct i think
14:26:19	dansmith	mriedem: okay can read that stuff in a sec
14:26:28	dansmith	also, zomg my aggregates patch merged
14:26:35	dansmith	er, notificatons I mean
14:29:54	openstackgerrit	Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852
14:29:57	mriedem	yeah a few things did over the weekend
14:30:00	mriedem	it's a miracle
14:30:17	cdent	praise be
14:31:23	openstackgerrit	Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852
14:31:42	mriedem	cdent: i finally got that cleaned up ^ what i had sitting didn't ... sit well with me
14:31:55	mriedem	because it was like, "oh something failed, wah, nice try retrying it sucker"
14:33:54	cdent	a good lesson for entitled whippersnappers, I say
14:34:03	cdent	learn em up about life
14:34:25	mriedem	back in my day we didn't have idempotent apis
14:34:33	mriedem	we had to fight to delete things
14:34:42	cdent	exactly so
14:35:05	cdent	What has idempotency got us? Social media addiction!
14:36:12	mriedem	that's a bit of a stretch, even for me
14:41:56	openstackgerrit	Matt Riedemann proposed openstack/nova master: Log reason for remove_host action failing https://review.opendev.org/692833
14:43:37	mriedem	dansmith: i know it's a bit big and not very fun to review, but the bottom few patches of the cross-cell-resize series are ready for review and +2ed by gibi if you get some time to sift through one or two of those
14:44:02	dansmith	sure, I need to get back to that, sorry
14:44:07	mriedem	np
14:46:37	mriedem	huh, maybe another case for long_rpc_timeout - for reserve_block_device_name call to compute when attaching a volume https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4
14:46:42	mriedem	we must have some really slow nodes in the gate
14:54:50	mriedem	inap and ovh again
14:54:56	mriedem	just like in the 'state of the gate' email
14:57:14	efried	mriedem: is https://review.opendev.org/#/c/692550/ (long_rpc_timeout) going to manifest in the gate anywhere in a way it can be seen easily? I would think not; we would have to get lucky and see an operation that takes longer than the old timeout.
14:57:26	efried	just wondering if we're waiting to see anything in particular before approving ^
14:58:03	mriedem	not likely, the timeout is 30 minutes by default
14:58:11	mriedem	tempest would timeout way before that
14:58:42	efried	I don't mean hitting the long one, I mean seeing an operation that takes somewhere *between* the old timeout and the new one (i.e. would have timed out before this change but doesn't now)
15:00:13	efried	anyway, I'm getting ready to +2 that one, unless you or dansmith tells me there's a reason not to.
15:00:39	mriedem	i noticed it because of a gate failure http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010494.html
15:00:41	mriedem	if that's what you're asking
15:01:09	dansmith	efried: I can look after I'm done with a review
15:01:13	mriedem	api response timed out waiting for POST /allocations to finish which took 3 minutes b/c we had no i/o on the node
15:01:31	mriedem	^ is not realistic though, and i'm not sure why that would happen in the gate
15:01:37	efried	right, my point is, in order to see it "working", we would have to find that same situation again, but where it *didn't* timeout, but took longer than the original timeout.
15:01:41	mriedem	as cdent said in the ML, what do we do if this happens?
15:02:04	mriedem	efried: that is probably not very possible without adding some logging that logs if an operation took over a given threshold,
15:02:05	efried	yeah, seems like we've ascertained the I/O thing is just overloading.
15:02:24	mriedem	i'm not sure if there is logging in oslo.messaging that goes off if an operation is over the given 60 second heartbeat, but there probably is
15:02:54	efried	It's not worth it, I was just asking whether there would be any other way to tell. Doesn't sound like it. But that's not a reason not to merge the thing.
15:03:14	efried	The harm would be, if something is wrong in such a way that it's never going to come back, you've just wasted time.
15:03:19	efried	but such is the eternal dilemma of timeouts.
15:04:07	efried	Re the I/O thing, perhaps it would benefit us to have fewer zuul nodes if it means a lower rate of spurious failure.
15:04:33	mriedem	i see this in oslo.messaging but it's debug level which we don't index https://github.com/openstack/oslo.messaging/blob/6bca848f5b272149bb32353a62f9d37108fcbe15/oslo_messaging/_drivers/amqpdriver.py#L533
15:04:49	mriedem	what do you mean by fewer zuul nodes?
15:05:36	efried	or, if what cdent suggested would work, if making the CI nodes less CPU-powerful would reduce their ability to burden the I/O. Same result, really: overall fewer runs occur in a given amount of time, but if the success rate increases, it's a win.
15:06:00	efried	mriedem: I mean, if the I/O is overloaded, try to make it less overloaded by reducing the number of things hitting it at a time.
15:06:01	mriedem	reducing cpu is likely going to cause failures in other ways i'd think
15:06:26	mriedem	historically cpu/ram has only gotten higher in devstack nodes in the gate over time
15:06:48	mriedem	when i started in openstack a full dsvm-tempest job took 45 minutes and could use 4vcpu/gb ram
15:07:12	mriedem	now we're basically topped out at 8cpu/ram with 2 api workers per control service
15:07:18	dansmith	efried: oh yeah that migrate_server one, mriedem and I discussed last week, so probably good for a +2
15:07:31	efried	dansmith: that's done, waiting for your +A
15:07:38	dansmith	yup yup
15:07:38	mriedem	we could try dropping API_WORKERS=1 in devstack if that would help with load

Earlier   Later