| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-02 | |||
| 15:36:18 | sean-k-mooney | lyarwood: could you take a look at this reno backport https://review.opendev.org/#/c/662095/ for the rabbitmq eventlet behavior | |
| 15:37:17 | sean-k-mooney | oh its not a backprot but ya we proably want to land that before the next stable/stein release whenever that is | |
| 15:42:05 | lyarwood | sean-k-mooney: LGTM, I missed that Mel had updated this recently thanks. | |
| 15:42:38 | sean-k-mooney | so did i i thought i had actully merged a week or two ago but went looking for it after our downstream call | |
| 15:50:38 | mnaser | kashyap: oh i see -- i will read around a bit more | |
| 15:51:08 | kashyap | mnaser: We (I?) should document this somewhere. As many people will come to wonder. | |
| 15:51:31 | mnaser | kashyap: yeah, i'm still wondering if this is a "im running really new hardware" issue or "its normal and i dont get it hat well" heh | |
| 15:52:26 | kashyap | (Maybe I'll send a patch to the upstream libvirt docs itself; and then to Nova.) | |
| 15:53:00 | kashyap | mnaser: This is a new hardware with some CPU features that are not yet supported by KVM/QEMU. | |
| 15:53:32 | mnaser | kashyap: i really wanted to run el8 but rdo and friends are not there yet unfortunately | |
| 15:54:23 | kashyap | mnaser: What's the issue? Stein not yet tested on CentOS8? | |
| 15:54:35 | mnaser | kashyap: yeah there's no openstack packages for el8 afaik | |
| 15:54:56 | mnaser | there's f28 but not el8 | |
| 15:55:31 | kashyap | mnaser: Yeah, I think the issue was CentOS-8 was released like last week. (I _guess_ it was delayed partly due to the "CentOS Stream", the parallel distro, announcement) | |
| 15:55:46 | mnaser | yes, i think that's the theory that i heard too :) | |
| 15:56:08 | kashyap | mnaser: Also, from my chat w/ Alan Pevec (from RDO) the other day, EL-8 packages are in the works. | |
| 15:56:29 | kashyap | (But "in the works" is boring to hear, I know.) | |
| 16:04:38 | openstackgerrit | Merged openstack/nova master: Do not print default dicts during heal_allocations https://review.opendev.org/686001 | |
| 16:05:05 | stephenfin | lyarwood: Care to look at https://review.opendev.org/#/c/685980/ so I can proceed with the other backports? https://review.opendev.org/#/c/685980/ | |
| 16:09:22 | lyarwood | stephenfin: sure looking | |
| 16:20:50 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686063 | |
| 16:21:44 | openstackgerrit | Eric Fried proposed openstack/nova master: Replace mock with spy in test https://review.opendev.org/686208 | |
| 16:21:44 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Another swing at `spy` https://review.opendev.org/686207 | |
| 16:21:47 | efried | gibi, dansmith: I think I have something that will make the caller happy. I have to run for a bit, so I haven't done up the actual tests yet, but... ^ | |
| 16:24:02 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686067 | |
| 16:26:54 | mriedem | i'd like to get a stable/stein release out soon but i'd like it to have https://review.opendev.org/#/c/686063/ in it as well since it's a regression, but i'm not sure how that works in relation to train, i.e. can we release that in stein once it's in train rc2? i think that's probably fine | |
| 16:27:09 | mriedem | we fix stuff in master all the time, backport it and release in stable before it's released in master i guess | |
| 16:29:47 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Remove a whole load of unused images https://review.opendev.org/686211 | |
| 16:32:14 | melwitt | mriedem: did you mean can we release it in stein before we release train rc2? | |
| 16:33:19 | lyarwood | I'm okay with doing that given train hasn't GA'd yet | |
| 16:36:46 | mriedem | melwitt: yes, we obviously released the regression on stable before train | |
| 16:36:52 | mriedem | so i think it's ok | |
| 16:36:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Remove a whole load of unused images, most remainder https://review.opendev.org/686211 | |
| 16:37:03 | mriedem | i think i'm just overthinking the train stuff | |
| 16:37:10 | gibi | efried: ack, looks promising | |
| 16:37:27 | melwitt | ok | |
| 16:51:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add AggregateMultiTenancyIsolation filter tests https://review.opendev.org/464002 | |
| 16:54:21 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Update the file for IPv4-only or IPv6-only network https://review.opendev.org/465891 | |
| 17:02:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add warning message for cache_utils connection failure https://review.opendev.org/509060 | |
| 17:02:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add warning message for cache_utils connection failure https://review.opendev.org/509060 | |
| 17:03:29 | stephenfin | efried: This looks related to what we were talking about yesterday https://review.opendev.org/#/c/518119/ | |
| 17:03:38 | stephenfin | Or not. I just see references to the same thing | |
| 17:03:47 | stephenfin | I'm looking through super old bugs | |
| 17:03:49 | stephenfin | *patches | |
| 17:05:39 | mriedem | stephenfin: i think we already have a test for this now https://review.opendev.org/#/c/464002/ | |
| 17:06:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't create instance_extra entry for deleted instance https://review.opendev.org/412771 | |
| 17:09:48 | stephenfin | mriedem: Good spot. I guess I can rebase/rework that | |
| 17:09:57 | stephenfin | Tomorrow though | |
| 17:10:05 | mriedem | well, if it's duplicate we don't need it | |
| 17:10:33 | stephenfin | Not sure if there's stuff missing or not | |
| 17:10:36 | stephenfin | I need to diff | |
| 17:10:56 | stephenfin | or maybe what we have is good enough. Eh | |
| 17:11:06 | mriedem | up to you, o/ | |
| 17:23:12 | openstackgerrit | Merged openstack/python-novaclient stable/train: Stop silently ignoring invalid 'nova boot --hint' options https://review.opendev.org/685980 | |
| 17:44:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during migrate reschedule https://review.opendev.org/686050 | |
| 17:44:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047 | |
| 17:44:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Selection.availability_zone field https://review.opendev.org/685807 | |
| 17:44:30 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem) | |
| 17:44:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017 | |
| 17:44:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update cells v2 up-call caveats doc https://review.opendev.org/686053 | |
| 17:44:49 | mriedem | gibi: i had to remove the mock.patch context manager usage in the migrate functional test in that series ^ because it wasn't working properly | |
| 17:48:39 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/train: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686226 | |
| 17:54:16 | openstackgerrit | Merged openstack/nova master: Handle get_host_availability_zone error during reschedule https://review.opendev.org/685997 | |
| 17:54:25 | openstackgerrit | Merged openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095 | |
| 18:12:52 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232 | |
| 18:17:58 | openstackgerrit | Merged openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020 | |
| 18:40:00 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232 | |
| 18:52:52 | melwitt | mriedem or efried: do you know if this is a known gate bug? 409 from placement "consumer generation conflict - expected null but got 1" https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dc8/684118/1/check/nova-multi-cell/dc8f056/testr_results.html.gz | |
| 18:54:49 | mriedem | dagnabbit dansmith i was nearly done with PS1 on your spec | |
| 18:55:25 | dansmith | mriedem: sorry, it was just a couple formatting things | |
| 18:56:03 | mriedem | ok comments inline on PS1 | |
| 18:56:24 | dansmith | thanks | |
| 18:56:32 | mriedem | artom asked some of the same | |
| 18:58:23 | mriedem | melwitt: i want to say not necessarily known, but i know i've fixed a race like that in scheduling before | |
| 18:59:54 | melwitt | ack. this one is new to me | |
| 19:00:13 | dansmith | ....artom reviewed something of mine? | |
| 19:00:17 | mriedem | melwitt: this is the gate issue that's like that which i fixed before https://review.opendev.org/#/c/623596/ | |
| 19:00:32 | artom | dansmith, payback time | |
| 19:00:36 | mriedem | i thought artom reserved his reviews specifically just for -1ing my stuff | |
| 19:00:57 | melwitt | mriedem: thanks | |
| 19:09:46 | mriedem | melwitt: given the name of the test, test_delete_server_while_in_building_state, i'm not surprised that there might be a race somehow | |
| 19:10:20 | dansmith | artom: mriedem thanks for all that, and in record turnaround time :D | |
| 19:10:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22consumer%20generation%20conflict%20-%20expected%20null%20but%20got%201%5C%22&from=7d | |
| 19:11:02 | mriedem | neither change that hit it look related to causing the issue though | |
| 19:12:12 | mriedem | t when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "code": "placement.concurrent_update", "request_id": "req-374479cc-0a6d-47b2-afbd-17c6e2f5413d"}]} | |
| 19:12:12 | mriedem | Oct 01 22:17:03.897751 ubuntu-bionic-rax-iad-0012085716 devstack@n-api.service[17786]: INFO nova.api.openstack.wsgi [None req-c9e65a63-ea1f-4144-aace-c148ba386c8e tempest-DeleteServersTestJSON-641512281 tempest-DeleteServersTestJSON-641512281] HTTP exception thrown: Failed to delete allocations for consumer e7339a26-880b-4c27-972c-265d66eb5d82. Error: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a con | |
| 19:13:18 | mriedem | so in this case the server probably got through scheduling and allocated resources on a provider after the delete request started but before we tried to delete the allocations | |
| 19:13:27 | mriedem | in the local delete path in the api since the server doesn't have a host set yet | |
| 19:13:57 | mriedem | melwitt: might as well report a bug for tracking | |
| 19:17:39 | mriedem | melwitt: looks like this changed in stein to return the 409 https://review.opendev.org/#/c/591597/ | |
| 19:17:43 | mriedem | before that we'd log but not fail the delete request | |
| 19:18:02 | mriedem | when i wrote this in rocky anyway https://review.opendev.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d | |
| 19:20:17 | mriedem | if delete_allocation_for_instance were smarter and returned a generation conflict exception, the api code could retry the call | |
| 19:21:00 | mriedem | we could look for this in the error from placement: "code": "placement.concurrent_update" | |
| 19:21:08 | mriedem | we do that elsewhere to detect races | |
| 19:21:26 | mriedem | see SchedulerReportClient.claim_resources | |
| 19:26:38 | openstackgerrit | Merged openstack/nova master: conf: Remove deprecated 'project_id_regex' opt https://review.opendev.org/501057 | |
| 19:26:44 | openstackgerrit | Merged openstack/nova master: Replace 'fake' with a real project ID https://review.opendev.org/685645 | |
| 19:31:09 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232 | |
| 19:32:15 | efried | melwitt: did you get your answer on that 409? | |