Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-02
15:29:23 efried gibi, dansmith: There's my crack at it --^ (done in a separate series)
15:30:01 gibi efried: ack
15:30:02 efried I have another idea that, if it works, will be an uglier helper but more intuitive calling syntax.
15:36:18 sean-k-mooney lyarwood: could you take a look at this reno backport https://review.opendev.org/#/c/662095/ for the rabbitmq eventlet behavior
15:37:17 sean-k-mooney oh its not a backprot but ya we proably want to land that before the next stable/stein release whenever that is
15:42:05 lyarwood sean-k-mooney: LGTM, I missed that Mel had updated this recently thanks.
15:42:38 sean-k-mooney so did i i thought i had actully merged a week or two ago but went looking for it after our downstream call
15:50:38 mnaser kashyap: oh i see -- i will read around a bit more
15:51:08 kashyap mnaser: We (I?) should document this somewhere. As many people will come to wonder.
15:51:31 mnaser kashyap: yeah, i'm still wondering if this is a "im running really new hardware" issue or "its normal and i dont get it hat well" heh
15:52:26 kashyap (Maybe I'll send a patch to the upstream libvirt docs itself; and then to Nova.)
15:53:00 kashyap mnaser: This is a new hardware with some CPU features that are not yet supported by KVM/QEMU.
15:53:32 mnaser kashyap: i really wanted to run el8 but rdo and friends are not there yet unfortunately
15:54:23 kashyap mnaser: What's the issue? Stein not yet tested on CentOS8?
15:54:35 mnaser kashyap: yeah there's no openstack packages for el8 afaik
15:54:56 mnaser there's f28 but not el8
15:55:31 kashyap mnaser: Yeah, I think the issue was CentOS-8 was released like last week. (I _guess_ it was delayed partly due to the "CentOS Stream", the parallel distro, announcement)
15:55:46 mnaser yes, i think that's the theory that i heard too :)
15:56:08 kashyap mnaser: Also, from my chat w/ Alan Pevec (from RDO) the other day, EL-8 packages are in the works.
15:56:29 kashyap (But "in the works" is boring to hear, I know.)
16:04:38 openstackgerrit Merged openstack/nova master: Do not print default dicts during heal_allocations https://review.opendev.org/686001
16:05:05 stephenfin lyarwood: Care to look at https://review.opendev.org/#/c/685980/ so I can proceed with the other backports? https://review.opendev.org/#/c/685980/
16:09:22 lyarwood stephenfin: sure looking
16:20:50 openstackgerrit melanie witt proposed openstack/nova stable/stein: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686063
16:21:44 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Another swing at `spy` https://review.opendev.org/686207
16:21:44 openstackgerrit Eric Fried proposed openstack/nova master: Replace mock with spy in test https://review.opendev.org/686208
16:21:47 efried gibi, dansmith: I think I have something that will make the caller happy. I have to run for a bit, so I haven't done up the actual tests yet, but... ^
16:24:02 openstackgerrit melanie witt proposed openstack/nova stable/rocky: Reduce scope of 'path' query parameter to noVNC consoles https://review.opendev.org/686067
16:26:54 mriedem i'd like to get a stable/stein release out soon but i'd like it to have https://review.opendev.org/#/c/686063/ in it as well since it's a regression, but i'm not sure how that works in relation to train, i.e. can we release that in stein once it's in train rc2? i think that's probably fine
16:27:09 mriedem we fix stuff in master all the time, backport it and release in stable before it's released in master i guess
16:29:47 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Remove a whole load of unused images https://review.opendev.org/686211
16:32:14 melwitt mriedem: did you mean can we release it in stein before we release train rc2?
16:33:19 lyarwood I'm okay with doing that given train hasn't GA'd yet
16:36:46 mriedem melwitt: yes, we obviously released the regression on stable before train
16:36:52 mriedem so i think it's ok
16:36:58 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Remove a whole load of unused images, most remainder https://review.opendev.org/686211
16:37:03 mriedem i think i'm just overthinking the train stuff
16:37:10 gibi efried: ack, looks promising
16:37:27 melwitt ok
16:51:26 openstackgerrit Stephen Finucane proposed openstack/nova master: Add AggregateMultiTenancyIsolation filter tests https://review.opendev.org/464002
16:54:21 openstackgerrit Stephen Finucane proposed openstack/nova master: Update the file for IPv4-only or IPv6-only network https://review.opendev.org/465891
17:02:22 openstackgerrit Stephen Finucane proposed openstack/nova master: Add warning message for cache_utils connection failure https://review.opendev.org/509060
17:02:27 openstackgerrit Stephen Finucane proposed openstack/nova master: Add warning message for cache_utils connection failure https://review.opendev.org/509060
17:03:29 stephenfin efried: This looks related to what we were talking about yesterday https://review.opendev.org/#/c/518119/
17:03:38 stephenfin Or not. I just see references to the same thing
17:03:47 stephenfin I'm looking through super old bugs
17:03:49 stephenfin *patches
17:05:39 mriedem stephenfin: i think we already have a test for this now https://review.opendev.org/#/c/464002/
17:06:37 openstackgerrit Stephen Finucane proposed openstack/nova master: Don't create instance_extra entry for deleted instance https://review.opendev.org/412771
17:09:48 stephenfin mriedem: Good spot. I guess I can rebase/rework that
17:09:57 stephenfin Tomorrow though
17:10:05 mriedem well, if it's duplicate we don't need it
17:10:33 stephenfin Not sure if there's stuff missing or not
17:10:36 stephenfin I need to diff
17:10:56 stephenfin or maybe what we have is good enough. Eh
17:11:06 mriedem up to you, o/
17:23:12 openstackgerrit Merged openstack/python-novaclient stable/train: Stop silently ignoring invalid 'nova boot --hint' options https://review.opendev.org/685980
17:44:30 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017
17:44:30 openstack bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem)
17:44:30 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Selection.availability_zone field https://review.opendev.org/685807
17:44:30 openstackgerrit Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during build reschedule https://review.opendev.org/686047
17:44:30 openstackgerrit Matt Riedemann proposed openstack/nova master: Set Instance AZ from Selection AZ during migrate reschedule https://review.opendev.org/686050
17:44:31 openstackgerrit Matt Riedemann proposed openstack/nova master: Update cells v2 up-call caveats doc https://review.opendev.org/686053
17:44:49 mriedem gibi: i had to remove the mock.patch context manager usage in the migrate functional test in that series ^ because it wasn't working properly
17:48:39 openstackgerrit Matt Riedemann proposed openstack/nova stable/train: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686226
17:54:16 openstackgerrit Merged openstack/nova master: Handle get_host_availability_zone error during reschedule https://review.opendev.org/685997
17:54:25 openstackgerrit Merged openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095
18:12:52 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
18:17:58 openstackgerrit Merged openstack/nova master: Fix incorrect usages of fake moref in VMware tests https://review.opendev.org/685020
18:40:00 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add image-precache-support spec https://review.opendev.org/686232
18:52:52 melwitt mriedem or efried: do you know if this is a known gate bug? 409 from placement "consumer generation conflict - expected null but got 1" https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_dc8/684118/1/check/nova-multi-cell/dc8f056/testr_results.html.gz
18:54:49 mriedem dagnabbit dansmith i was nearly done with PS1 on your spec
18:55:25 dansmith mriedem: sorry, it was just a couple formatting things
18:56:03 mriedem ok comments inline on PS1
18:56:24 dansmith thanks
18:56:32 mriedem artom asked some of the same
18:58:23 mriedem melwitt: i want to say not necessarily known, but i know i've fixed a race like that in scheduling before
18:59:54 melwitt ack. this one is new to me
19:00:13 dansmith ....artom reviewed something of mine?
19:00:17 mriedem melwitt: this is the gate issue that's like that which i fixed before https://review.opendev.org/#/c/623596/
19:00:32 artom dansmith, payback time
19:00:36 mriedem i thought artom reserved his reviews specifically just for -1ing my stuff
19:00:57 melwitt mriedem: thanks
19:09:46 mriedem melwitt: given the name of the test, test_delete_server_while_in_building_state, i'm not surprised that there might be a race somehow
19:10:20 dansmith artom: mriedem thanks for all that, and in record turnaround time :D
19:10:32 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22consumer%20generation%20conflict%20-%20expected%20null%20but%20got%201%5C%22&from=7d
19:11:02 mriedem neither change that hit it look related to causing the issue though
19:12:12 mriedem Oct 01 22:17:03.897751 ubuntu-bionic-rax-iad-0012085716 devstack@n-api.service[17786]: INFO nova.api.openstack.wsgi [None req-c9e65a63-ea1f-4144-aace-c148ba386c8e tempest-DeleteServersTestJSON-641512281 tempest-DeleteServersTestJSON-641512281] HTTP exception thrown: Failed to delete allocations for consumer e7339a26-880b-4c27-972c-265d66eb5d82. Error: {"errors": [{"status": 409, "title": "Conflict", "detail": "There was a con
19:12:12 mriedem t when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "code": "placement.concurrent_update", "request_id": "req-374479cc-0a6d-47b2-afbd-17c6e2f5413d"}]}
19:13:18 mriedem so in this case the server probably got through scheduling and allocated resources on a provider after the delete request started but before we tried to delete the allocations
19:13:27 mriedem in the local delete path in the api since the server doesn't have a host set yet
19:13:57 mriedem melwitt: might as well report a bug for tracking
19:17:39 mriedem melwitt: looks like this changed in stein to return the 409 https://review.opendev.org/#/c/591597/
19:17:43 mriedem before that we'd log but not fail the delete request
19:18:02 mriedem when i wrote this in rocky anyway https://review.opendev.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d
19:20:17 mriedem if delete_allocation_for_instance were smarter and returned a generation conflict exception, the api code could retry the call
19:21:00 mriedem we could look for this in the error from placement: "code": "placement.concurrent_update"
19:21:08 mriedem we do that elsewhere to detect races
19:21:26 mriedem see SchedulerReportClient.claim_resources
19:26:38 openstackgerrit Merged openstack/nova master: conf: Remove deprecated 'project_id_regex' opt https://review.opendev.org/501057

Earlier   Later