| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-04 | |||
| 19:08:26 | mriedem | melwitt: have you seen this? intermittent unit test failure http://logs.openstack.org/06/641806/7/check/openstack-tox-lower-constraints/1f1fce6/job-output.txt.gz#_2019-04-03_22_36_18_975096 | |
| 19:12:27 | mriedem | looks like this unit test starts up a conductor service using the ServiceFixture which calls self.addCleanup(self.service.kill) | |
| 19:13:56 | melwitt | no, haven't seen that before. I've seen the error before, I remember helping to fix a bug for that way back when | |
| 19:14:40 | melwitt | this one https://review.openstack.org/555093 | |
| 19:15:26 | melwitt | and in the fail you linked, we're getting it during service.destroy() this time | |
| 19:15:55 | melwitt | which must mean in that flow somewhere we have a reader transaction context nested outside a writer transaction context | |
| 19:18:08 | mriedem | yeah not sure, also doesn't really show up much in logstash | |
| 19:18:13 | mriedem | so i guess i'll recheck and assume it's a fluke | |
| 19:18:27 | melwitt | looking at the code, so far I don't see how the nesting could be happening | |
| 19:33:55 | mriedem | efried_mtg: these failures http://logs.openstack.org/13/621313/21/check/openstack-tox-py36/ba2c848/testr_results.html.gz definitely started spiking after 4/1 with https://github.com/openstack/nova/commit/e5269f2fbb773953266fccbfd9df3f75e351eeb9 | |
| 19:34:06 | mriedem | i wonder if removing kipsdist = True in tox.ini is contributing | |
| 19:34:17 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22sqlalchemy.exc.InterfaceError%3A%20(pymysql.err.InterfaceError)%20(0%2C%20'')%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 19:34:23 | mriedem | *skipsdist = True | |
| 19:34:58 | mriedem | and it's always on node provider: limestone-regionone | |
| 19:45:47 | mriedem | tox on the limestone node is 3.8.4 and on an ovh node that passes tox is 3.8.1 | |
| 19:45:52 | mriedem | pymysql is the same between them | |
| 19:47:00 | efried | mriedem: What can I do to help? | |
| 19:47:13 | mriedem | hold me | |
| 19:47:36 | mriedem | but not in a tap out kind of way | |
| 19:47:46 | mriedem | i'm reporting a new bug and will push an e-r query | |
| 19:48:20 | efried | should I push a change adding skipdist back and see what happens? | |
| 19:49:02 | mriedem | idk, i wasn't involved in that lower-constraints fix so i don't know if that will screw up the lower-constraints job, but i guess you'd find out | |
| 19:49:24 | efried | according to the commit message, cdent thought skipdist was "otherwise superfluous" | |
| 19:49:35 | efried | so hopefully that means it at least won't hurt anything. | |
| 19:49:46 | mriedem | if lower-constraints needs usedevelop=false and skipsdist=false, it could just specify that explicitly | |
| 19:49:53 | efried | yuh | |
| 19:50:07 | mriedem | usedevelop defaults to false also, so i'm not sure why the lower-constraints job is calling it out | |
| 19:50:39 | mriedem | i'm currently being defeated by launchpad timeouts | |
| 19:53:21 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Re-add skipsdist=True https://review.openstack.org/650123 | |
| 19:53:34 | efried | mriedem: ^ any way to get that to run on the offending node? | |
| 19:53:50 | mriedem | not that i know of | |
| 19:54:08 | mriedem | unless there is a way to specify that in the zuul job definition | |
| 19:54:10 | mriedem | fungi: ^ | |
| 19:54:34 | mriedem | question is can we restrict a change to only run on limestone-regionone nodes | |
| 19:56:28 | mriedem | btw this would explain the unit test job timeouts creeping up again | |
| 19:56:45 | mriedem | b/c when this fails, i think there is some thread switch or something and the fixture just goes away and we timeout | |
| 19:57:15 | logan- | i can provide a project to run it on in limestone-regionone | |
| 19:58:20 | logan- | aside from setting up custom nodepool labels I don't think there is a way to set affinity in the job | |
| 19:59:34 | openstackgerrit | Merged openstack/nova master: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/649775 | |
| 19:59:41 | openstackgerrit | Merged openstack/nova master: Add docs on what not to include in notifications https://review.openstack.org/650020 | |
| 20:04:52 | fungi | mriedem: efried: logan- is correct. we'd need special node labels or nodesets | |
| 20:05:11 | mriedem | https://bugs.launchpad.net/nova/+bug/1823251 | |
| 20:05:12 | openstack | Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019" [Medium,Confirmed] | |
| 20:05:28 | mriedem | fungi: logan-: ok i guess we'll just tinker with seeing if adding skipsdist=True back fixes the issue | |
| 20:05:31 | mriedem | and i'll get an e-r query up | |
| 20:06:06 | efried | mriedem: So is https://review.openstack.org/#/c/650043/ not safe to recheck? | |
| 20:06:21 | efried | we're holding RC2 for it, nah? | |
| 20:06:23 | mriedem | efried: melwitt already approved it | |
| 20:06:32 | efried | yes but it's sitting failed | |
| 20:06:32 | mriedem | but i just looked at the grenade failure, it's a known gate failure | |
| 20:06:45 | mriedem | don't need to recheck it if it's approved now | |
| 20:06:46 | efried | right, I'm asking if it's a 100% failure or if we can recheck? | |
| 20:06:49 | melwitt | it's in the zuul check queue | |
| 20:06:54 | efried | oh, okay. | |
| 20:13:30 | mriedem | for this pymysql InterfaceError, it's showing up in more than just nova, | |
| 20:13:44 | mriedem | so did something change in upper-constraints or something more global on 4/1? | |
| 20:14:56 | mriedem | https://github.com/openstack/requirements/commit/27d0b42e23f8d964464862cb5590dea870ae9fd7#diff-0bdd949ed8a7fdd4f95240bd951779c8 merged on 4/1 | |
| 20:15:48 | mriedem | i don't see anything obvious though | |
| 20:19:56 | mriedem | logan-: are you able to tell if anything changed around 4/1 wrt limestone-regionone bionic node packages? | |
| 20:20:10 | mriedem | specifically related to mysql/sqlalchemy/pymysql | |
| 20:20:17 | mriedem | although the latter 2 should be coming from pypi in our CI runs | |
| 20:20:41 | logan- | the region came back online after a p->q->r upgrade on 3/27, nothing has changed since then though. | |
| 20:21:08 | logan- | since we have been chatting, I just found a config issue that is being resolved: https://review.openstack.org/#/c/650135/ | |
| 20:22:07 | mriedem | heh what a coincidence https://review.openstack.org/#/c/641981/ | |
| 20:23:23 | logan- | hah | |
| 20:39:06 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Re-add skipsdist=True, except for lower-constraints https://review.openstack.org/650123 | |
| 20:40:29 | efried | kashyap, aspiers: FYI, I did not seed the review-day etherpad with the specs you mentioned, because it sounded like they weren't quite ready for that yet. | |
| 20:44:58 | openstackgerrit | Michael Still proposed openstack/nova master: Hacking N362: Don't abbrev/alias privsep import https://review.openstack.org/649190 | |
| 20:44:59 | openstackgerrit | Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs. https://review.openstack.org/648602 | |
| 20:44:59 | openstackgerrit | Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.path. https://review.openstack.org/648601 | |
| 20:45:00 | openstackgerrit | Michael Still proposed openstack/nova master: Add test coverage for nova.privsep.libvirt. https://review.openstack.org/648616 | |
| 20:45:00 | openstackgerrit | Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs, continued. https://review.openstack.org/648603 | |
| 20:45:01 | openstackgerrit | Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431 | |
| 20:45:01 | openstackgerrit | Michael Still proposed openstack/nova master: Add test coverage for nova.privsep.qemu. https://review.openstack.org/649191 | |
| 20:45:02 | openstackgerrit | Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435 | |
| 20:45:02 | openstackgerrit | Michael Still proposed openstack/nova master: Remove unused FP device creation and deletion methods. https://review.openstack.org/635433 | |
| 20:45:03 | openstackgerrit | Michael Still proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.openstack.org/636508 | |
| 20:45:03 | openstackgerrit | Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436 | |
| 20:45:04 | openstackgerrit | Michael Still proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.openstack.org/639281 | |
| 20:45:04 | openstackgerrit | Michael Still proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.openstack.org/639280 | |
| 20:45:05 | openstackgerrit | Michael Still proposed openstack/nova master: Move setting of device trust to privsep. https://review.openstack.org/639283 | |
| 20:45:05 | openstackgerrit | Michael Still proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.openstack.org/639282 | |
| 20:45:06 | openstackgerrit | Michael Still proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.openstack.org/639581 | |
| 20:45:06 | openstackgerrit | Michael Still proposed openstack/nova master: Move final bridge commands to privsep. https://review.openstack.org/639580 | |
| 20:45:07 | openstackgerrit | Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | |
| 20:45:46 | efried | mdbooth_: anything on this morning's weird db race topic? | |
| 20:45:53 | efried | o, it's like midnight for you. | |
| 21:09:54 | efried | http://logs.openstack.org/01/648601/5/check/openstack-tox-py36/0e12826/testr_results.html.gz | |
| 21:09:54 | efried | http://logs.openstack.org/66/649566/1/check/openstack-tox-py37/e989684/testr_results.html.gz | |
| 21:09:54 | efried | http://logs.openstack.org/53/639453/9/check/openstack-tox-py36/2b6b01f/testr_results.html.gz | |
| 21:09:54 | efried | http://logs.openstack.org/13/621313/21/check/openstack-tox-py36/ba2c848/testr_results.html.gz | |
| 21:09:54 | efried | http://logs.openstack.org/92/627892/21/check/openstack-tox-py37/66b8eab/testr_results.html.gz | |
| 21:09:54 | efried | http://logs.openstack.org/57/649757/1/check/openstack-tox-py37/a965c6b/testr_results.html.gz | |
| 21:09:54 | efried | mriedem: I'm dyin here. Can't figure out how to get logstash to return these. Here are representative examples: | |
| 21:11:20 | mriedem | think i already got it https://review.openstack.org/650136 | |
| 21:12:15 | efried | sigh | |
| 21:12:57 | efried | I was trying to include other parts of the "message". How do you do that? Whenever I use AND it comes back with nothing. | |
| 21:13:25 | efried | fixtures._fixtures.timeout.TimeoutException seems relevant. Because I would have thought that InterfaceException could be caused by any number of things. | |
| 21:13:32 | efried | InterfaceError, rather | |
| 21:15:13 | mriedem | console log entries are indexed as single lines | |