Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-04
21:22:45 mriedem cdent: i definitely suspect eventlet weirdness with http://status.openstack.org/elastic-recheck/#1793364
21:22:50 mriedem cdent: but yeah that
21:22:54 mriedem timing doesn't line up with the failures
21:23:06 mriedem i do think we hit some thread switch and then the test just times out
21:23:17 mriedem older notes are in https://bugs.launchpad.net/nova/+bug/1793364
21:23:18 openstack Launchpad bug 1793364 in Manila "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [Undecided,In progress] - Assigned to Goutham Pacha Ravi (gouthamr)
21:25:07 efried mriedem: https://review.openstack.org/#/c/650043/ is failing in grenade again. Ima rebase it so we don't have to wait for the rest to finish, if you can re+W it?
21:25:22 mriedem sure
21:25:55 openstackgerrit Eric Fried proposed openstack/nova stable/stein: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/650043
21:26:11 efried mriedem: needed trivial edit ^ since no newer master commit to rebase onto
21:26:31 melwitt mriedem: looks comprehensive to me, I can't think of anything missing other than linking to the docs about what down cell means and such when you propose the patch
21:26:42 mriedem melwitt: yup lots of links to be made
21:26:50 mriedem thanks for looking
21:26:52 cdent if it turns out to be sdist we should decode why. in at least some of our tests runs when sdist is true we'll install requirements as discovered by python setup.py sdist and then install requirements _again_ so there's a possibility that pip is changing it's mind about some things when upper constraints is more involved. hard to say, especially at my late hour
21:27:07 melwitt mriedem: yeah, looks good. good stuff to have in the docs
21:27:25 mriedem efried: you could have chosen something besides the cherry-picked line :)
21:27:50 mriedem well i guess not, whatever
21:28:04 mriedem efried: maybe should ask fungi to promote that?
21:28:04 efried mriedem: Yeah, then it would have differed from its cherry-pick-ees.
21:28:19 dansmith mriedem: why the upgrade levels thing?
21:28:56 mriedem dansmith: because of https://bugs.launchpad.net/nova/+bug/1815697
21:28:57 openstack Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed]
21:29:11 mriedem "Also note the default database max_attempts and retry_interval are 10 which means for each API object created that hits this, it's going to take 100 seconds to timeout per route handler per API worker. I count 31 route handlers that create an API object, so that's by default 3100 seconds or about ~52 minutes per worker on startup."
21:29:21 dansmith oh because auto doesn't support the timeouts?
21:29:23 melwitt mriedem, efried: reminder we're also waiting on https://review.openstack.org/649630 it's #1 in the gate but if it fails...
21:29:34 dansmith I think I remember that now
21:29:38 efried fungi: TLDR https://review.openstack.org/#/c/650043/ represents a security regression we're holding nova-stein-rc2 for
21:29:49 mriedem dansmith: not really about the timeouts, it's just we hit that query on each API controller startup x number of API workers
21:29:51 efried melwitt: ack, succeeding so far.
21:30:02 efried though it hasn't finished the freakin grenade-lm job yet
21:30:18 mriedem heh we could have rebased https://review.openstack.org/#/c/650043/ on top of https://review.openstack.org/649630
21:30:25 melwitt yeah. wanted to point that out in case of gate promotions, we have two that are in flight
21:32:01 mriedem dansmith: https://review.openstack.org/#/c/649197/ was an attempt at something semi-related
21:33:02 dansmith mriedem: oh, is this because we actually start them in threads early enough (because wsgi) that we haven't populated the cache yet?
21:33:34 mriedem well, in my devstack recreate, cell1 is down and cell1 is the only thing that has compute services,
21:33:43 mriedem so we get 0 back for the min compute service version which does not get cached
21:34:05 mriedem https://review.openstack.org/#/c/649197/1/nova/compute/rpcapi.py@424
21:34:17 dansmith oh okay I meant the perf thing, but that's probably legit too
21:34:50 mriedem also related https://bugs.launchpad.net/nova/+bug/1815692
21:34:51 openstack Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged]
21:35:24 dansmith ah
21:36:13 mriedem that would not affect the other bug on startup though because we don't call that method which uses the cache https://github.com/openstack/nova/blob/0bed18ffbb46c4f2d0ec87e64a39188c165398eb/nova/objects/service.py#L507
21:36:47 mriedem but as i recall objects.Service.enable_min_version_cache() is meant for perf gains in the api lookups
21:36:49 mriedem after startup
21:36:51 dansmith I think your singleton patch looks okay
21:37:08 dansmith so,
21:37:11 mriedem i haven't looked into the functional test failures
21:37:13 dansmith with the sighup thing not working,
21:37:18 dansmith you might not want to fix the caching thing :P
21:37:41 dansmith that might be why it hasn't been as much of a thing.. potentially.. if people aren't sighupping but still getting the pin recalculated
21:37:57 dansmith although the perf impact is probably worse than just having to bump all your services fully
21:38:18 mriedem the sighup for nova-compute is particularly busted b/c of the privsep usage and neutron event callback handler stuff,
21:38:21 dansmith what a tangled mess of shit
21:38:35 dansmith right, but it's also going to cause anything else to do a full restart
21:38:42 mriedem but i wasn't sure if hup'ing the controller services was still ok because we have the cells cache for connections and disabled cells and all that
21:38:44 dansmith not as impactful for apis I guess
21:39:01 dansmith but for conductor it's probably not great either
21:39:13 mriedem ok so i think you're saying huping the controller services will do what you want - reload caches, but not in the graceful way we intended
21:39:30 mriedem b/c it's a restart
21:39:30 dansmith I dunno about conductor, probably not great there,
21:39:43 mriedem scheduler has the hup for the disabled cells i think
21:39:47 dansmith as I think anything currently waiting on an rpc call, or polling for state or whatever will die
21:39:49 mriedem and cells cache in the request context
21:40:18 mriedem tangled mess of shit yes
21:40:24 dansmith for serious
21:40:25 mriedem who's being depressing NOW!
21:40:33 dansmith o/
21:40:42 mriedem maybe my depression rubs off on those that speak with me
21:40:46 dansmith and it's thursday, closer to friday
21:40:57 mriedem efried: melwitt: how are you feeling right now?
21:41:03 dansmith you're also the arsehole that is finding this stuff
21:41:10 dansmith we could just ignore it if it wasn't for you
21:43:20 melwitt yeah, seriously
21:43:35 melwitt :)
21:44:51 mriedem it is also cold and rainy here right now (well most of this week)
21:44:54 mriedem so it's just a gloom
21:46:27 fungi efried: cool, yeah i was advancing/monitoring the master branch version of 650043 earlier so that one could be approved sooner, can also shuffle that one up now that it's been approved (sorry for the delay, was just out for an afternoon walk)
21:46:49 efried thanks fungi
21:59:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Log notifications if assertion in _test_live_migration_force_complete fails https://review.openstack.org/650155
22:19:48 openstackgerrit Matt Riedemann proposed openstack/nova-specs master: Re-propose cross-cell-resize spec for Train https://review.openstack.org/642807
22:28:36 melwitt efried, fungi: 650043,2 just failed lower-constraints :( for what looks to be https://bugs.launchpad.net/nova/+bug/1793364
22:28:37 openstack Launchpad bug 1793364 in Manila "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [Undecided,In progress] - Assigned to Goutham Pacha Ravi (gouthamr)
22:30:57 fungi melwitt: thanks for the heads up, i've manually reset the gate so it will restart jobs on that change now
22:31:09 melwitt thanks!
22:58:48 efried melwitt, mriedem: fungi: I gotta run. You got this?
22:59:09 melwitt efried: yeah, I'll watch the patches tonight
22:59:13 efried Thanks melwitt
23:00:29 fungi i'll be around as well
23:00:34 fungi feel free to ping me for anything
23:00:58 fungi eager to see this resolved so it doesn't become a bigger issue after release
23:01:17 melwitt ok, thank you
23:01:20 fungi (since that usually means more work for all of us, and we already have more than enough of that to go around)
23:01:20 mriedem no VMT bias there :)
23:01:27 fungi ;)
23:02:31 fungi yeah, juggling plenty of security issues which have cropped up in releases we can't go back in time to fix, so it's nice when we catch one *before* releasing
#openstack-nova - 2019-04-05
00:26:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Add testing guide for down cells https://review.openstack.org/650167
00:28:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Add testing guide for down cells https://review.openstack.org/650167
01:01:04 openstackgerrit Merged openstack/nova stable/stein: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/650043
01:01:14 openstackgerrit Merged openstack/nova stable/stein: Libvirt: gracefully handle non-nic VFs https://review.openstack.org/649630
01:01:34 openstackgerrit Merged openstack/nova master: Delete allocations even if _confirm_resize raises https://review.openstack.org/647566

Earlier   Later