Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-04
21:27:50 mriedem well i guess not, whatever
21:28:04 efried mriedem: Yeah, then it would have differed from its cherry-pick-ees.
21:28:04 mriedem efried: maybe should ask fungi to promote that?
21:28:19 dansmith mriedem: why the upgrade levels thing?
21:28:56 mriedem dansmith: because of https://bugs.launchpad.net/nova/+bug/1815697
21:28:57 openstack Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed]
21:29:11 mriedem "Also note the default database max_attempts and retry_interval are 10 which means for each API object created that hits this, it's going to take 100 seconds to timeout per route handler per API worker. I count 31 route handlers that create an API object, so that's by default 3100 seconds or about ~52 minutes per worker on startup."
21:29:21 dansmith oh because auto doesn't support the timeouts?
21:29:23 melwitt mriedem, efried: reminder we're also waiting on https://review.openstack.org/649630 it's #1 in the gate but if it fails...
21:29:34 dansmith I think I remember that now
21:29:38 efried fungi: TLDR https://review.openstack.org/#/c/650043/ represents a security regression we're holding nova-stein-rc2 for
21:29:49 mriedem dansmith: not really about the timeouts, it's just we hit that query on each API controller startup x number of API workers
21:29:51 efried melwitt: ack, succeeding so far.
21:30:02 efried though it hasn't finished the freakin grenade-lm job yet
21:30:18 mriedem heh we could have rebased https://review.openstack.org/#/c/650043/ on top of https://review.openstack.org/649630
21:30:25 melwitt yeah. wanted to point that out in case of gate promotions, we have two that are in flight
21:32:01 mriedem dansmith: https://review.openstack.org/#/c/649197/ was an attempt at something semi-related
21:33:02 dansmith mriedem: oh, is this because we actually start them in threads early enough (because wsgi) that we haven't populated the cache yet?
21:33:34 mriedem well, in my devstack recreate, cell1 is down and cell1 is the only thing that has compute services,
21:33:43 mriedem so we get 0 back for the min compute service version which does not get cached
21:34:05 mriedem https://review.openstack.org/#/c/649197/1/nova/compute/rpcapi.py@424
21:34:17 dansmith oh okay I meant the perf thing, but that's probably legit too
21:34:50 mriedem also related https://bugs.launchpad.net/nova/+bug/1815692
21:34:51 openstack Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged]
21:35:24 dansmith ah
21:36:13 mriedem that would not affect the other bug on startup though because we don't call that method which uses the cache https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/objects/service.py#L507
21:36:47 mriedem but as i recall objects.Service.enable_min_version_cache() is meant for perf gains in the api lookups
21:36:49 mriedem after startup
21:36:51 dansmith I think your singleton patch looks okay
21:37:08 dansmith so,
21:37:11 mriedem i haven't looked into the functional test failures
21:37:13 dansmith with the sighup thing not working,
21:37:18 dansmith you might not want to fix the caching thing :P
21:37:41 dansmith that might be why it hasn't been as much of a thing.. potentially.. if people aren't sighupping but still getting the pin recalculated
21:37:57 dansmith although the perf impact is probably worse than just having to bump all your services fully
21:38:18 mriedem the sighup for nova-compute is particularly busted b/c of the privsep usage and neutron event callback handler stuff,
21:38:21 dansmith what a tangled mess of shit
21:38:35 dansmith right, but it's also going to cause anything else to do a full restart
21:38:42 mriedem but i wasn't sure if hup'ing the controller services was still ok because we have the cells cache for connections and disabled cells and all that
21:38:44 dansmith not as impactful for apis I guess
21:39:01 dansmith but for conductor it's probably not great either
21:39:13 mriedem ok so i think you're saying huping the controller services will do what you want - reload caches, but not in the graceful way we intended
21:39:30 dansmith I dunno about conductor, probably not great there,
21:39:30 mriedem b/c it's a restart
21:39:43 mriedem scheduler has the hup for the disabled cells i think
21:39:47 dansmith as I think anything currently waiting on an rpc call, or polling for state or whatever will die
21:39:49 mriedem and cells cache in the request context
21:40:18 mriedem tangled mess of shit yes
21:40:24 dansmith for serious
21:40:25 mriedem who's being depressing NOW!
21:40:33 dansmith o/
21:40:42 mriedem maybe my depression rubs off on those that speak with me
21:40:46 dansmith and it's thursday, closer to friday
21:40:57 mriedem efried: melwitt: how are you feeling right now?
21:41:03 dansmith you're also the arsehole that is finding this stuff
21:41:10 dansmith we could just ignore it if it wasn't for you
21:43:20 melwitt yeah, seriously
21:43:35 melwitt :)
21:44:51 mriedem it is also cold and rainy here right now (well most of this week)
21:44:54 mriedem so it's just a gloom
21:46:27 fungi efried: cool, yeah i was advancing/monitoring the master branch version of 650043 earlier so that one could be approved sooner, can also shuffle that one up now that it's been approved (sorry for the delay, was just out for an afternoon walk)
21:46:49 efried thanks fungi
21:59:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Log notifications if assertion in _test_live_migration_force_complete fails https://review.openstack.org/650155
22:19:48 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Re-propose cross-cell-resize spec for Train https://review.openstack.org/642807
22:28:36 melwitt efried, fungi: 650043,2 just failed lower-constraints :( for what looks to be https://bugs.launchpad.net/nova/+bug/1793364
22:28:37 openstack Launchpad bug 1793364 in Manila "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [Undecided,In progress] - Assigned to Goutham Pacha Ravi (gouthamr)
22:30:57 fungi melwitt: thanks for the heads up, i've manually reset the gate so it will restart jobs on that change now
22:31:09 melwitt thanks!
22:58:48 efried melwitt, mriedem: fungi: I gotta run. You got this?
22:59:09 melwitt efried: yeah, I'll watch the patches tonight
22:59:13 efried Thanks melwitt
23:00:29 fungi i'll be around as well
23:00:34 fungi feel free to ping me for anything
23:00:58 fungi eager to see this resolved so it doesn't become a bigger issue after release
23:01:17 melwitt ok, thank you
23:01:20 mriedem no VMT bias there :)
23:01:20 fungi (since that usually means more work for all of us, and we already have more than enough of that to go around)
23:01:27 fungi ;)
23:02:31 fungi yeah, juggling plenty of security issues which have cropped up in releases we can't go back in time to fix, so it's nice when we catch one *before* releasing
#openstack-nova - 2019-04-05
00:26:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Add testing guide for down cells https://review.openstack.org/650167
00:28:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Add testing guide for down cells https://review.openstack.org/650167
01:01:04 openstackgerrit Merged openstack/nova stable/stein: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/650043
01:01:14 openstackgerrit Merged openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630
01:01:34 openstackgerrit Merged openstack/nova master: Delete allocations even if _confirm_resize raises https://review.openstack.org/647566
01:01:41 openstackgerrit Merged openstack/nova master: api-ref: add more details to confirmResize troubleshooting https://review.openstack.org/647741
01:06:00 ileixe efried: Thanks for the answer. Yes I already saw the dansmith 'oschown' repository and actually we have an third API handle the request (It's very much looks like his solution). I felt hard to maintain the custom API since there's too much exceptional cases that I could not imagine.
01:07:28 ileixe efried: So I was expecting the blueprint, but I wonder why the community decided to give up the blueprint.
01:12:47 melwitt ileixe: some of the folks in the "transfer ownership" forum session didn't like the 'oschown' idea as much and preferred something more like API plugins. but if you are in support of 'oschown', you are encouraged to tell your opinion in the session or add your opinion on the etherpad for the session with your IRC nick
01:14:46 melwitt it looks like there isn't a new etherpad yet though
01:14:56 ileixe melwitt: Thanks for the procedure. It seems to not be abandonded completely yet then. I'm with plugin like API plugins, so I willing to make comments.
01:16:30 melwitt ileixe: I don't think a solid decision has been made yet, discussion hasn't reached a way forward yet. sounds good. I'm trying to find other threads about it, one sec
01:21:23 melwitt ileixe: this is the thread I was thinking about http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000280.html
01:34:11 openstackgerrit melanie witt proposed openstack/nova master: Fix SynchronousThreadPoolExecutorFixture mock spec https://review.openstack.org/650171
01:45:05 ileixe @melwitt: Many thanks. I was thirsty about the background of decision making.
01:46:48 melwitt ileixe: np. are you going to be at the summit? if so, try to make it to the session https://www.openstack.org/summit/denver-2019/summit-schedule/events/23675/change-ownership-of-resources-followup
01:47:51 ileixe @melwitt: Unfortunately, I do not participate.
01:48:06 ileixe Thanks anyway :)
01:48:37 melwitt ileixe: ok. I'll ping you when the etherpad for the session appears, so you can write your thoughts on it and it will be part of the discussion
01:49:26 ileixe @melwitt: Cool. See you then.
01:49:36 melwitt :)

Earlier   Later