Posted	Nick	Remark
#openstack-nova - 2019-10-02
15:56:29	kashyap	(But "in the works" is boring to hear, I know.)
16:04:38	openstackgerrit	Merged openstack/nova master: Do not print default dicts during heal_allocations https://review.opendev.org/686001
16:05:05	stephenfin	lyarwood: Care to look at https://review.opendev.org/#/c/685980/ so I can proceed with the other backports? https://review.opendev.org/#/c/685980/
16:09:22	lyarwood	stephenfin: sure looking
16:20:50	openstackgerrit	melanie witt proposed openstack/nova stable/stein: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686063
16:21:44	openstackgerrit	Eric Fried proposed openstack/nova master: WIP: Another swing at `spy` https://review.opendev.org/686207
16:21:44	openstackgerrit	Eric Fried proposed openstack/nova master: Replace mock with spy in test https://review.opendev.org/686208
16:21:47	efried	gibi, dansmith: I think I have something that will make the caller happy. I have to run for a bit, so I haven't done up the actual tests yet, but... ^
16:24:02	openstackgerrit	melanie witt proposed openstack/nova stable/rocky: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686067
16:26:54	mriedem	i'd like to get a stable/stein release out soon but i'd like it to have https://review.opendev.org/#/c/686063/ in it as well since it's a regression, but i'm not sure how that works in relation to train, i.e. can we release that in stein once it's in train rc2? i think that's probably fine
16:27:09	mriedem	we fix stuff in master all the time, backport it and release in stable before it's released in master i guess
16:29:47	openstackgerrit	Stephen Finucane proposed openstack/nova master: docs: Remove a whole load of unused images https://review.opendev.org/686211
16:32:14	melwitt	mriedem: did you mean can we release it in stein before we release train rc2?
16:33:19	lyarwood	I'm okay with doing that given train hasn't GA'd yet
16:36:46	mriedem	melwitt: yes, we obviously released the regression on stable before train
16:36:52	mriedem	so i think it's ok
16:36:58	openstackgerrit	Stephen Finucane proposed openstack/nova master: docs: Remove a whole load of unused images, most remainder https://review.opendev.org/686211
16:37:03	mriedem	i think i'm just overthinking the train stuff
16:37:10	gibi	efried: ack, looks promising
16:37:27	melwitt	ok
16:51:26	openstackgerrit	Stephen Finucane proposed openstack/nova master: Add AggregateMultiTenancyIsolation filter tests https://review.opendev.org/464002
16:54:21	openstackgerrit	Stephen Finucane proposed openstack/nova master: Update the file for IPv4-only or IPv6-only network https://review.opendev.org/465891
17:02:22	openstackgerrit	Stephen Finucane proposed openstack/nova master: Add warning message for cache_utils connection failure https://review.opendev.org/509060
17:02:27	openstackgerrit	Stephen Finucane proposed openstack/nova master: Add warning message for cache_utils connection failure https://review.opendev.org/509060
17:03:29	stephenfin	efried: This looks related to what we were talking about yesterday https://review.opendev.org/#/c/518119/
17:03:38	stephenfin	Or not. I just see references to the same thing
17:03:47	stephenfin	I'm looking through super old bugs
17:03:49	stephenfin	*patches
17:05:39	mriedem	stephenfin: i think we already have a test for this now https://review.opendev.org/#/c/464002/
17:06:37	openstackgerrit	Stephen Finucane proposed openstack/nova master: Don't create instance_extra entry for deleted instance https://review.opendev.org/412771
17:09:48	stephenfin	mriedem: Good spot. I guess I can rebase/rework that
17:09:57	stephenfin	Tomorrow though
17:10:05	mriedem	well, if it's duplicate we don't need it
17:10:33	stephenfin	Not sure if there's stuff missing or not
17:10:36	stephenfin	I need to diff
17:10:56	stephenfin	or maybe what we have is good enough. Eh
17:11:06	mriedem	up to you, o/
17:23:12	openstackgerrit	Merged openstack/python-novaclient stable/train: Stop silently ignoring invalid 'nova boot --hint' options https://review.opendev.org/685980
17:44:30	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017
17:44:30	openstack	bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem)
17:44:30	openstackgerrit	Matt Riedemann proposed openstack/nova master: Add Selection.availability_zone field https://review.opendev.org/685807
17:44:30	openstackgerrit	Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047
17:44:30	openstackgerrit	Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during migrate reschedule https://review.opendev.org/686050
17:44:31	openstackgerrit	Matt Riedemann proposed openstack/nova master: Update cells v2 up-call caveats doc https://review.opendev.org/686053
17:44:49	mriedem	gibi: i had to remove the mock.patch context manager usage in the migrate functional test in that series ^ because it wasn't working properly
17:48:39	openstackgerrit	Matt Riedemann proposed openstack/nova stable/train: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686226
17:54:16	openstackgerrit	Merged openstack/nova master: Handle get_host_availability_zone error during reschedule https://review.opendev.org/685997
17:54:25	openstackgerrit	Merged openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095
18:12:52	openstackgerrit	Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
18:17:58	openstackgerrit	Merged openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020
18:40:00	openstackgerrit	Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
18:52:52	melwitt	mriedem or efried: do you know if this is a known gate bug? 409 from placement "consumer generation conflict - expected null but got 1" https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dc8/684118/1/check/nova-multi-cell/dc8f056/testr_results.html.gz
18:54:49	mriedem	dagnabbit dansmith i was nearly done with PS1 on your spec
18:55:25	dansmith	mriedem: sorry, it was just a couple formatting things
18:56:03	mriedem	ok comments inline on PS1
18:56:24	dansmith	thanks
18:56:32	mriedem	artom asked some of the same
18:58:23	mriedem	melwitt: i want to say not necessarily known, but i know i've fixed a race like that in scheduling before
18:59:54	melwitt	ack. this one is new to me
19:00:13	dansmith	....artom reviewed something of mine?
19:00:17	mriedem	melwitt: this is the gate issue that's like that which i fixed before https://review.opendev.org/#/c/623596/
19:00:32	artom	dansmith, payback time
19:00:36	mriedem	i thought artom reserved his reviews specifically just for -1ing my stuff
19:00:57	melwitt	mriedem: thanks
19:09:46	mriedem	melwitt: given the name of the test, test_delete_server_while_in_building_state, i'm not surprised that there might be a race somehow
19:10:20	dansmith	artom: mriedem thanks for all that, and in record turnaround time :D
19:10:32	mriedem	http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22consumer%20generation%20conflict%20-%20expected%20null%20but%20got%201%5C%22&from=7d
19:11:02	mriedem	neither change that hit it look related to causing the issue though
19:12:12	mriedem	Oct 01 22:17:03.897751 ubuntu-bionic-rax-iad-0012085716 devstack@n-api.service[17786]: INFO nova.api.openstack.wsgi [None req-c9e65a63-ea1f-4144-aace-c148ba386c8e tempest-DeleteServersTestJSON-641512281 tempest-DeleteServersTestJSON-641512281] HTTP exception thrown: Failed to delete allocations for consumer e7339a26-880b-4c27-972c-265d66eb5d82. Error: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a con
19:12:12	mriedem	t when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "code": "placement.concurrent_update", "request_id": "req-374479cc-0a6d-47b2-afbd-17c6e2f5413d"}]}
19:13:18	mriedem	so in this case the server probably got through scheduling and allocated resources on a provider after the delete request started but before we tried to delete the allocations
19:13:27	mriedem	in the local delete path in the api since the server doesn't have a host set yet
19:13:57	mriedem	melwitt: might as well report a bug for tracking
19:17:39	mriedem	melwitt: looks like this changed in stein to return the 409 https://review.opendev.org/#/c/591597/
19:17:43	mriedem	before that we'd log but not fail the delete request
19:18:02	mriedem	when i wrote this in rocky anyway https://review.opendev.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d
19:20:17	mriedem	if delete_allocation_for_instance were smarter and returned a generation conflict exception, the api code could retry the call
19:21:00	mriedem	we could look for this in the error from placement: "code": "placement.concurrent_update"
19:21:08	mriedem	we do that elsewhere to detect races
19:21:26	mriedem	see SchedulerReportClient.claim_resources
19:26:38	openstackgerrit	Merged openstack/nova master: conf: Remove deprecated 'project_id_regex' opt https://review.opendev.org/501057
19:26:44	openstackgerrit	Merged openstack/nova master: Replace 'fake' with a real project ID https://review.opendev.org/685645
19:31:09	openstackgerrit	Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
19:32:15	efried	melwitt: did you get your answer on that 409?
19:33:15	efried	stephenfin: yes, you're right, that appears to be hitting some of the same stuff I was looking at, will dig, thanks for the pointer.
19:42:27	mriedem	efried: i think so - see above
19:42:40	mriedem	efried: just need a bug and a retry decorator on delete_allocations_for_instances i think
19:43:01	openstackgerrit	Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
19:43:31	efried	mriedem: Yeah, that sounds familiar, I want to say I had that somewhere on my to-do list ages ago but haven't gotten back to it.
19:44:35	openstackgerrit	Matt Riedemann proposed openstack/nova stable/stein: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686264
19:51:16	melwitt	mriedem: ok, I'll file a bug. thanks
19:55:12	mriedem	dansmith: i'm not sure that "sufficiently-endowed" should ever be in a spec
19:55:35	dansmith	mriedem: not with you around, that's for sure
19:55:43	mriedem	:P
19:55:44	artom	dansmith, "sufficiently-endowed", seriously?
19:55:53	artom	mriedem, dammit
19:55:53	mriedem	now don't go removing it yet, i'm commenting
19:56:05	dansmith	you two being the only two reviewers, I had to bait you good
19:56:56	artom	Careful how you spell bait
19:57:44	dansmith	you guys are too easy

Earlier   Later