Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-04
20:05:11 mriedem https://bugs.launchpad.net/nova/+bug/1823251
20:05:12 openstack Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019" [Medium,Confirmed]
20:05:28 mriedem fungi: logan-: ok i guess we'll just tinker with seeing if adding skipsdist=True back fixes the issue
20:05:31 mriedem and i'll get an e-r query up
20:06:06 efried mriedem: So is https://review.openstack.org/#/c/650043/ not safe to recheck?
20:06:21 efried we're holding RC2 for it, nah?
20:06:23 mriedem efried: melwitt already approved it
20:06:32 efried yes but it's sitting failed
20:06:32 mriedem but i just looked at the grenade failure, it's a known gate failure
20:06:45 mriedem don't need to recheck it if it's approved now
20:06:46 efried right, I'm asking if it's a 100% failure or if we can recheck?
20:06:49 melwitt it's in the zuul check queue
20:06:54 efried oh, okay.
20:13:30 mriedem for this pymysql InterfaceError, it's showing up in more than just nova,
20:13:44 mriedem so did something change in upper-constraints or something more global on 4/1?
20:14:56 mriedem https://github.com/openstack/requirements/commit/27d0b42e23f8d964464862cb5590dea870ae9fd7#diff-0bdd949ed8a7fdd4f95240bd951779c8 merged on 4/1
20:15:48 mriedem i don't see anything obvious though
20:19:56 mriedem logan-: are you able to tell if anything changed around 4/1 wrt limestone-regionone bionic node packages?
20:20:10 mriedem specifically related to mysql/sqlalchemy/pymysql
20:20:17 mriedem although the latter 2 should be coming from pypi in our CI runs
20:20:41 logan- the region came back online after a p->q->r upgrade on 3/27, nothing has changed since then though.
20:21:08 logan- since we have been chatting, I just found a config issue that is being resolved: https://review.openstack.org/#/c/650135/
20:22:07 mriedem heh what a coincidence https://review.openstack.org/#/c/641981/
20:23:23 logan- hah
20:39:06 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Re-add skipsdist=True, except for lower-constraints https://review.openstack.org/650123
20:40:29 efried kashyap, aspiers: FYI, I did not seed the review-day etherpad with the specs you mentioned, because it sounded like they weren't quite ready for that yet.
20:44:58 openstackgerrit Michael Still proposed openstack/nova master: Hacking N362: Don't abbrev/alias privsep import https://review.openstack.org/649190
20:44:59 openstackgerrit Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs. https://review.openstack.org/648602
20:44:59 openstackgerrit Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.path. https://review.openstack.org/648601
20:45:00 openstackgerrit Michael Still proposed openstack/nova master: Add test coverage for nova.privsep.libvirt. https://review.openstack.org/648616
20:45:00 openstackgerrit Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs, continued. https://review.openstack.org/648603
20:45:01 openstackgerrit Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431
20:45:01 openstackgerrit Michael Still proposed openstack/nova master: Add test coverage for nova.privsep.qemu. https://review.openstack.org/649191
20:45:02 openstackgerrit Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435
20:45:02 openstackgerrit Michael Still proposed openstack/nova master: Remove unused FP device creation and deletion methods. https://review.openstack.org/635433
20:45:03 openstackgerrit Michael Still proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.openstack.org/636508
20:45:03 openstackgerrit Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436
20:45:04 openstackgerrit Michael Still proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.openstack.org/639281
20:45:04 openstackgerrit Michael Still proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.openstack.org/639280
20:45:05 openstackgerrit Michael Still proposed openstack/nova master: Move setting of device trust to privsep. https://review.openstack.org/639283
20:45:05 openstackgerrit Michael Still proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.openstack.org/639282
20:45:06 openstackgerrit Michael Still proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.openstack.org/639581
20:45:06 openstackgerrit Michael Still proposed openstack/nova master: Move final bridge commands to privsep. https://review.openstack.org/639580
20:45:07 openstackgerrit Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438
20:45:46 efried mdbooth_: anything on this morning's weird db race topic?
20:45:53 efried o, it's like midnight for you.
21:09:54 efried http://logs.openstack.org/01/648601/5/check/openstack-tox-py36/0e12826/testr_results.html.gz
21:09:54 efried http://logs.openstack.org/66/649566/1/check/openstack-tox-py37/e989684/testr_results.html.gz
21:09:54 efried http://logs.openstack.org/53/639453/9/check/openstack-tox-py36/2b6b01f/testr_results.html.gz
21:09:54 efried http://logs.openstack.org/13/621313/21/check/openstack-tox-py36/ba2c848/testr_results.html.gz
21:09:54 efried http://logs.openstack.org/92/627892/21/check/openstack-tox-py37/66b8eab/testr_results.html.gz
21:09:54 efried http://logs.openstack.org/57/649757/1/check/openstack-tox-py37/a965c6b/testr_results.html.gz
21:09:54 efried mriedem: I'm dyin here. Can't figure out how to get logstash to return these. Here are representative examples:
21:11:20 mriedem think i already got it https://review.openstack.org/650136
21:12:15 efried sigh
21:12:57 efried I was trying to include other parts of the "message". How do you do that? Whenever I use AND it comes back with nothing.
21:13:25 efried fixtures._fixtures.timeout.TimeoutException seems relevant. Because I would have thought that InterfaceException could be caused by any number of things.
21:13:32 efried InterfaceError, rather
21:15:13 mriedem console log entries are indexed as single lines
21:15:23 mriedem so you unfortunately can't do an AND on messages from different lines
21:15:32 mriedem the only multi-line indexing we have is tracebacks in screen logs
21:15:36 mriedem e.g. nova-compute
21:16:09 mriedem which is frustrating with tests that fail like this https://bugs.launchpad.net/nova/+bug/1823215
21:16:10 openstack Launchpad bug 1823215 in OpenStack Compute (nova) "TestInstanceNotificationSampleWithMultipleComputeOldAttachFlow._test_live_migration_force_complete intermittent fails with MismatchError: 6 != 7" [Medium,Confirmed]
21:16:21 mriedem means we have to do a better job of logging at the point of failure
21:19:37 mriedem melwitt: dansmith: i was going to write up something in our contributor testing reference docs about testing with a down cell in devstack, my notes are here http://paste.openstack.org/show/748904/ - do you see anything in there that i could be missing which might be interesting/useful?
21:19:56 mriedem somewhere linked from here https://docs.openstack.org/nova/latest/contributor/#testing
21:21:07 cdent did any of mdbooth's eventlet clean ups merge?
21:22:07 mriedem cdent: https://review.openstack.org/#/q/topic:bug/1808975+(status:open+OR+status:merged)
21:22:15 cdent because it seems like what the pymysql interface errors are showing is that something has disrupted some delicate timing and a bit of unitidiness in the opportunistic tests is not getting the leeway desired
21:22:44 cdent hmmm, that's further back than would make sense for this
21:22:45 mriedem cdent: i definitely suspect eventlet weirdness with http://status.openstack.org/elastic-recheck/#1793364
21:22:50 mriedem cdent: but yeah that
21:22:54 mriedem timing doesn't line up with the failures
21:23:06 mriedem i do think we hit some thread switch and then the test just times out
21:23:17 mriedem older notes are in https://bugs.launchpad.net/nova/+bug/1793364
21:23:18 openstack Launchpad bug 1793364 in Manila "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [Undecided,In progress] - Assigned to Goutham Pacha Ravi (gouthamr)
21:25:07 efried mriedem: https://review.openstack.org/#/c/650043/ is failing in grenade again. Ima rebase it so we don't have to wait for the rest to finish, if you can re+W it?
21:25:22 mriedem sure
21:25:55 openstackgerrit Eric Fried proposed openstack/nova stable/stein: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/650043
21:26:11 efried mriedem: needed trivial edit ^ since no newer master commit to rebase onto
21:26:31 melwitt mriedem: looks comprehensive to me, I can't think of anything missing other than linking to the docs about what down cell means and such when you propose the patch
21:26:42 mriedem melwitt: yup lots of links to be made
21:26:50 mriedem thanks for looking
21:26:52 cdent if it turns out to be sdist we should decode why. in at least some of our tests runs when sdist is true we'll install requirements as discovered by python setup.py sdist and then install requirements _again_ so there's a possibility that pip is changing it's mind about some things when upper constraints is more involved. hard to say, especially at my late hour
21:27:07 melwitt mriedem: yeah, looks good. good stuff to have in the docs
21:27:25 mriedem efried: you could have chosen something besides the cherry-picked line :)
21:27:50 mriedem well i guess not, whatever
21:28:04 mriedem efried: maybe should ask fungi to promote that?
21:28:04 efried mriedem: Yeah, then it would have differed from its cherry-pick-ees.
21:28:19 dansmith mriedem: why the upgrade levels thing?
21:28:56 mriedem dansmith: because of https://bugs.launchpad.net/nova/+bug/1815697
21:28:57 openstack Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed]
21:29:11 mriedem "Also note the default database max_attempts and retry_interval are 10 which means for each API object created that hits this, it's going to take 100 seconds to timeout per route handler per API worker. I count 31 route handlers that create an API object, so that's by default 3100 seconds or about ~52 minutes per worker on startup."
21:29:21 dansmith oh because auto doesn't support the timeouts?
21:29:23 melwitt mriedem, efried: reminder we're also waiting on https://review.openstack.org/649630 it's #1 in the gate but if it fails...
21:29:34 dansmith I think I remember that now
21:29:38 efried fungi: TLDR https://review.openstack.org/#/c/650043/ represents a security regression we're holding nova-stein-rc2 for
21:29:49 mriedem dansmith: not really about the timeouts, it's just we hit that query on each API controller startup x number of API workers
21:29:51 efried melwitt: ack, succeeding so far.

Earlier   Later