Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-02
19:00:13 dansmith ....artom reviewed something of mine?
19:00:17 mriedem melwitt: this is the gate issue that's like that which i fixed before https://review.opendev.org/#/c/623596/
19:00:32 artom dansmith, payback time
19:00:36 mriedem i thought artom reserved his reviews specifically just for -1ing my stuff
19:00:57 melwitt mriedem: thanks
19:09:46 mriedem melwitt: given the name of the test, test_delete_server_while_in_building_state, i'm not surprised that there might be a race somehow
19:10:20 dansmith artom: mriedem thanks for all that, and in record turnaround time :D
19:10:32 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22consumer%20generation%20conflict%20-%20expected%20null%20but%20got%201%5C%22&from=7d
19:11:02 mriedem neither change that hit it look related to causing the issue though
19:12:12 mriedem t when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "code": "placement.concurrent_update", "request_id": "req-374479cc-0a6d-47b2-afbd-17c6e2f5413d"}]}
19:12:12 mriedem Oct 01 22:17:03.897751 ubuntu-bionic-rax-iad-0012085716 devstack@n-api.service[17786]: INFO nova.api.openstack.wsgi [None req-c9e65a63-ea1f-4144-aace-c148ba386c8e tempest-DeleteServersTestJSON-641512281 tempest-DeleteServersTestJSON-641512281] HTTP exception thrown: Failed to delete allocations for consumer e7339a26-880b-4c27-972c-265d66eb5d82. Error: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a con
19:13:18 mriedem so in this case the server probably got through scheduling and allocated resources on a provider after the delete request started but before we tried to delete the allocations
19:13:27 mriedem in the local delete path in the api since the server doesn't have a host set yet
19:13:57 mriedem melwitt: might as well report a bug for tracking
19:17:39 mriedem melwitt: looks like this changed in stein to return the 409 https://review.opendev.org/#/c/591597/
19:17:43 mriedem before that we'd log but not fail the delete request
19:18:02 mriedem when i wrote this in rocky anyway https://review.opendev.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d
19:20:17 mriedem if delete_allocation_for_instance were smarter and returned a generation conflict exception, the api code could retry the call
19:21:00 mriedem we could look for this in the error from placement: "code": "placement.concurrent_update"
19:21:08 mriedem we do that elsewhere to detect races
19:21:26 mriedem see SchedulerReportClient.claim_resources
19:26:38 openstackgerrit Merged openstack/nova master: conf: Remove deprecated 'project_id_regex' opt https://review.opendev.org/501057
19:26:44 openstackgerrit Merged openstack/nova master: Replace 'fake' with a real project ID https://review.opendev.org/685645
19:31:09 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
19:32:15 efried melwitt: did you get your answer on that 409?
19:33:15 efried stephenfin: yes, you're right, that appears to be hitting some of the same stuff I was looking at, will dig, thanks for the pointer.
19:42:27 mriedem efried: i think so - see above
19:42:40 mriedem efried: just need a bug and a retry decorator on delete_allocations_for_instances i think
19:43:01 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
19:43:31 efried mriedem: Yeah, that sounds familiar, I want to say I had that somewhere on my to-do list ages ago but haven't gotten back to it.
19:44:35 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686264
19:51:16 melwitt mriedem: ok, I'll file a bug. thanks
19:55:12 mriedem dansmith: i'm not sure that "sufficiently-endowed" should ever be in a spec
19:55:35 dansmith mriedem: not with you around, that's for sure
19:55:43 mriedem :P
19:55:44 artom dansmith, "sufficiently-endowed", seriously?
19:55:53 mriedem now don't go removing it yet, i'm commenting
19:55:53 artom mriedem, dammit
19:56:05 dansmith you two being the only two reviewers, I had to bait you good
19:56:56 artom Careful how you spell bait
19:57:44 dansmith you guys are too easy
19:57:56 artom It's more fun that way.
19:58:10 dansmith endowed with permission is a totally legit phrasing
19:59:06 sean-k-mooney dansmith: you have a lot of references in that spec. i added it to my list of things to revew
19:59:32 dansmith sean-k-mooney: I listed a bunch of them so that reviewers would be afraid to not approve it
20:00:15 sean-k-mooney it looks relitivly short for a spec but im to tired to review stuff a tthe momemnt so ill go through it tomorrow.
20:00:53 sean-k-mooney hehe i see you are going for the over load them with example of why this is need so you cant question it approch
20:01:18 sean-k-mooney it fairly effective
20:02:30 artom dansmith, I don't suppose setting up glance -> computes multicast is part of your MVP spec ;)
20:03:04 artom But it'd be a way to get both the ease of fire-and-forget RPC cast while not killing the glance server
20:03:11 artom Not that it's, you know, realistic in any way
20:03:46 dansmith artom: the right solution, IMHO, is bittorrent
20:03:52 dansmith which was proposed and implemented in RAX at one point
20:04:11 dansmith basically, glance becomes a torrent seeder and only has to dish out the image once
20:04:12 artom I genuinely thought you were kidding at first
20:04:25 dansmith or however many workers you have, that's how many seeds you have
20:04:44 artom But yeah, that's actually smart
20:04:53 dansmith imagine how good that would be for an edge site where you have four computes.. right now they all pull the image over the WAN... but that makes no sense, they should share once one of them has it
20:04:54 dansmith right
20:05:38 dansmith and if one got it yesterday and still has it, the peer compute can pull it from it instead of over the way
20:05:39 dansmith *wan
20:06:31 dansmith also imagine how in the future iteration where we need to report which computes have that image cached, if we could just ask the torrent magnet which computes are now seeds... :D
20:07:42 artom *stunned Pikachu face*
20:08:04 openstackgerrit melanie witt proposed openstack/nova master: Restore console proxy deployment info to cells v2 layout doc https://review.opendev.org/686271
20:08:29 sean-k-mooney we talkabout using bittorrent like 2 years ago
20:08:42 dansmith we talked about it in 2012
20:08:47 melwitt efried, mriedem: added ^ to https://etherpad.openstack.org/p/nova-train-release-todo as a possible thing for RC2
20:08:49 sean-k-mooney was it that long
20:08:49 dansmith but probably two years ago as well
20:09:14 sean-k-mooney anyway im glad its now a thing
20:09:24 sean-k-mooney because ya for edge it toptally makes sense
20:10:17 sean-k-mooney its resilent, spreads the load across multiple peers and works over low bandwitd links
20:10:24 dansmith I'm not really proposing that in the short term, but it's always been something I wanted us to do
20:11:18 sean-k-mooney well part of why i like it is its also not us e.g. openstack comping up with our own solution for a proablem that has and existing soltion already
20:11:33 dansmith yup
20:14:05 mriedem more comments
20:14:50 artom dansmith, brainstorming here - but does oslo.concurrency give us a distributed semaphore?
20:15:07 dansmith artom: not without some external service
20:15:11 artom I'm thinking - cast to the computes, but have the grab the semaphore when downloading
20:15:51 bnemec Sounds like a use case for tooz.
20:15:58 mriedem i was just going to make an etcd joke
20:16:08 bnemec oslo.concurrency only works within a single machine.
20:16:24 artom That way the concurrency becomes configurable (just set the semaphore's... err, cardinality?) and you don't ddos the glance server
20:16:26 dansmith artom: yeah, it's just not really necessary when we can threadpool it from the conductor
20:16:49 artom dansmith, for an RPC call though, right?
20:17:07 dansmith artom: yup
20:17:23 artom Yeah, I may be overthinking this
20:17:26 mriedem melwitt: ack i'm cool with that, docs only and regression in train
20:20:26 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Noop CantStartEngineError in targets_cell if API DB not configured https://review.opendev.org/686276
20:26:02 dansmith mriedem: yawna read my response to the parallelism thing and if you're cool with it I can squash those todos in my next rev: https://review.opendev.org/#/c/686232/4/specs/ussuri/approved/image-precache-support.rst@123
20:26:02 openstackgerrit Merged openstack/nova master: Use os-brick locking for volume attach and detach https://review.opendev.org/614190
20:26:12 melwitt mriedem: coolio
20:28:36 mriedem dansmith: replied
20:29:04 dansmith mriedem: thanks
20:30:52 openstackgerrit melanie witt proposed openstack/nova master: Restore console proxy deployment info to cells v2 layout doc https://review.opendev.org/686271
20:35:06 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
20:36:50 efried melwitt, mriedem: +2, good catch
20:38:58 efried aspiers: how's that SEV fixup looking? Would be nice to produce RC2 tomorrow (and not need RC3). Your fix is the last one I know of that's not well on the way to merging.
20:42:12 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686292
20:44:11 mriedem efried: deadline for RCs is oct 10 right?

Earlier   Later