| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-02 | |||
| 17:44:30 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem) | |
| 17:44:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017 | |
| 17:44:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update cells v2 up-call caveats doc https://review.opendev.org/686053 | |
| 17:44:49 | mriedem | gibi: i had to remove the mock.patch context manager usage in the migrate functional test in that series ^ because it wasn't working properly | |
| 17:48:39 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/train: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686226 | |
| 17:54:16 | openstackgerrit | Merged openstack/nova master: Handle get_host_availability_zone error during reschedule https://review.opendev.org/685997 | |
| 17:54:25 | openstackgerrit | Merged openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095 | |
| 18:12:52 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232 | |
| 18:17:58 | openstackgerrit | Merged openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020 | |
| 18:40:00 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232 | |
| 18:52:52 | melwitt | mriedem or efried: do you know if this is a known gate bug? 409 from placement "consumer generation conflict - expected null but got 1" https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dc8/684118/1/check/nova-multi-cell/dc8f056/testr_results.html.gz | |
| 18:54:49 | mriedem | dagnabbit dansmith i was nearly done with PS1 on your spec | |
| 18:55:25 | dansmith | mriedem: sorry, it was just a couple formatting things | |
| 18:56:03 | mriedem | ok comments inline on PS1 | |
| 18:56:24 | dansmith | thanks | |
| 18:56:32 | mriedem | artom asked some of the same | |
| 18:58:23 | mriedem | melwitt: i want to say not necessarily known, but i know i've fixed a race like that in scheduling before | |
| 18:59:54 | melwitt | ack. this one is new to me | |
| 19:00:13 | dansmith | ....artom reviewed something of mine? | |
| 19:00:17 | mriedem | melwitt: this is the gate issue that's like that which i fixed before https://review.opendev.org/#/c/623596/ | |
| 19:00:32 | artom | dansmith, payback time | |
| 19:00:36 | mriedem | i thought artom reserved his reviews specifically just for -1ing my stuff | |
| 19:00:57 | melwitt | mriedem: thanks | |
| 19:09:46 | mriedem | melwitt: given the name of the test, test_delete_server_while_in_building_state, i'm not surprised that there might be a race somehow | |
| 19:10:20 | dansmith | artom: mriedem thanks for all that, and in record turnaround time :D | |
| 19:10:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22consumer%20generation%20conflict%20-%20expected%20null%20but%20got%201%5C%22&from=7d | |
| 19:11:02 | mriedem | neither change that hit it look related to causing the issue though | |
| 19:12:12 | mriedem | t when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "code": "placement.concurrent_update", "request_id": "req-374479cc-0a6d-47b2-afbd-17c6e2f5413d"}]} | |
| 19:12:12 | mriedem | Oct 01 22:17:03.897751 ubuntu-bionic-rax-iad-0012085716 devstack@n-api.service[17786]: INFO nova.api.openstack.wsgi [None req-c9e65a63-ea1f-4144-aace-c148ba386c8e tempest-DeleteServersTestJSON-641512281 tempest-DeleteServersTestJSON-641512281] HTTP exception thrown: Failed to delete allocations for consumer e7339a26-880b-4c27-972c-265d66eb5d82. Error: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a con | |
| 19:13:18 | mriedem | so in this case the server probably got through scheduling and allocated resources on a provider after the delete request started but before we tried to delete the allocations | |
| 19:13:27 | mriedem | in the local delete path in the api since the server doesn't have a host set yet | |
| 19:13:57 | mriedem | melwitt: might as well report a bug for tracking | |
| 19:17:39 | mriedem | melwitt: looks like this changed in stein to return the 409 https://review.opendev.org/#/c/591597/ | |
| 19:17:43 | mriedem | before that we'd log but not fail the delete request | |
| 19:18:02 | mriedem | when i wrote this in rocky anyway https://review.opendev.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d | |
| 19:20:17 | mriedem | if delete_allocation_for_instance were smarter and returned a generation conflict exception, the api code could retry the call | |
| 19:21:00 | mriedem | we could look for this in the error from placement: "code": "placement.concurrent_update" | |
| 19:21:08 | mriedem | we do that elsewhere to detect races | |
| 19:21:26 | mriedem | see SchedulerReportClient.claim_resources | |
| 19:26:38 | openstackgerrit | Merged openstack/nova master: conf: Remove deprecated 'project_id_regex' opt https://review.opendev.org/501057 | |
| 19:26:44 | openstackgerrit | Merged openstack/nova master: Replace 'fake' with a real project ID https://review.opendev.org/685645 | |
| 19:31:09 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232 | |
| 19:32:15 | efried | melwitt: did you get your answer on that 409? | |
| 19:33:15 | efried | stephenfin: yes, you're right, that appears to be hitting some of the same stuff I was looking at, will dig, thanks for the pointer. | |
| 19:42:27 | mriedem | efried: i think so - see above | |
| 19:42:40 | mriedem | efried: just need a bug and a retry decorator on delete_allocations_for_instances i think | |
| 19:43:01 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232 | |
| 19:43:31 | efried | mriedem: Yeah, that sounds familiar, I want to say I had that somewhere on my to-do list ages ago but haven't gotten back to it. | |
| 19:44:35 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686264 | |
| 19:51:16 | melwitt | mriedem: ok, I'll file a bug. thanks | |
| 19:55:12 | mriedem | dansmith: i'm not sure that "sufficiently-endowed" should ever be in a spec | |
| 19:55:35 | dansmith | mriedem: not with you around, that's for sure | |
| 19:55:43 | mriedem | :P | |
| 19:55:44 | artom | dansmith, "sufficiently-endowed", seriously? | |
| 19:55:53 | mriedem | now don't go removing it yet, i'm commenting | |
| 19:55:53 | artom | mriedem, dammit | |
| 19:56:05 | dansmith | you two being the only two reviewers, I had to bait you good | |
| 19:56:56 | artom | Careful how you spell bait | |
| 19:57:44 | dansmith | you guys are too easy | |
| 19:57:56 | artom | It's more fun that way. | |
| 19:58:10 | dansmith | endowed with permission is a totally legit phrasing | |
| 19:59:06 | sean-k-mooney | dansmith: you have a lot of references in that spec. i added it to my list of things to revew | |
| 19:59:32 | dansmith | sean-k-mooney: I listed a bunch of them so that reviewers would be afraid to not approve it | |
| 20:00:15 | sean-k-mooney | it looks relitivly short for a spec but im to tired to review stuff a tthe momemnt so ill go through it tomorrow. | |
| 20:00:53 | sean-k-mooney | hehe i see you are going for the over load them with example of why this is need so you cant question it approch | |
| 20:01:18 | sean-k-mooney | it fairly effective | |
| 20:02:30 | artom | dansmith, I don't suppose setting up glance -> computes multicast is part of your MVP spec ;) | |
| 20:03:04 | artom | But it'd be a way to get both the ease of fire-and-forget RPC cast while not killing the glance server | |
| 20:03:11 | artom | Not that it's, you know, realistic in any way | |
| 20:03:46 | dansmith | artom: the right solution, IMHO, is bittorrent | |
| 20:03:52 | dansmith | which was proposed and implemented in RAX at one point | |
| 20:04:11 | dansmith | basically, glance becomes a torrent seeder and only has to dish out the image once | |
| 20:04:12 | artom | I genuinely thought you were kidding at first | |
| 20:04:25 | dansmith | or however many workers you have, that's how many seeds you have | |
| 20:04:44 | artom | But yeah, that's actually smart | |
| 20:04:53 | dansmith | imagine how good that would be for an edge site where you have four computes.. right now they all pull the image over the WAN... but that makes no sense, they should share once one of them has it | |
| 20:04:54 | dansmith | right | |
| 20:05:38 | dansmith | and if one got it yesterday and still has it, the peer compute can pull it from it instead of over the way | |
| 20:05:39 | dansmith | *wan | |
| 20:06:31 | dansmith | also imagine how in the future iteration where we need to report which computes have that image cached, if we could just ask the torrent magnet which computes are now seeds... :D | |
| 20:07:42 | artom | *stunned Pikachu face* | |
| 20:08:04 | openstackgerrit | melanie witt proposed openstack/nova master: Restore console proxy deployment info to cells v2 layout doc https://review.opendev.org/686271 | |
| 20:08:29 | sean-k-mooney | we talkabout using bittorrent like 2 years ago | |
| 20:08:42 | dansmith | we talked about it in 2012 | |
| 20:08:47 | melwitt | efried, mriedem: added ^ to https://etherpad.openstack.org/p/nova-train-release-todo as a possible thing for RC2 | |
| 20:08:49 | sean-k-mooney | was it that long | |
| 20:08:49 | dansmith | but probably two years ago as well | |
| 20:09:14 | sean-k-mooney | anyway im glad its now a thing | |
| 20:09:24 | sean-k-mooney | because ya for edge it toptally makes sense | |
| 20:10:17 | sean-k-mooney | its resilent, spreads the load across multiple peers and works over low bandwitd links | |
| 20:10:24 | dansmith | I'm not really proposing that in the short term, but it's always been something I wanted us to do | |
| 20:11:18 | sean-k-mooney | well part of why i like it is its also not us e.g. openstack comping up with our own solution for a proablem that has and existing soltion already | |
| 20:11:33 | dansmith | yup | |
| 20:14:05 | mriedem | more comments | |
| 20:14:50 | artom | dansmith, brainstorming here - but does oslo.concurrency give us a distributed semaphore? | |
| 20:15:07 | dansmith | artom: not without some external service | |
| 20:15:11 | artom | I'm thinking - cast to the computes, but have the grab the semaphore when downloading | |
| 20:15:51 | bnemec | Sounds like a use case for tooz. | |
| 20:15:58 | mriedem | i was just going to make an etcd joke | |
| 20:16:08 | bnemec | oslo.concurrency only works within a single machine. | |