Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-04
15:53:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.openstack.org/621061
15:53:48 efried dansmith: You around to hit https://review.openstack.org/#/c/649775/ ?
15:54:08 dansmith I will, I'm just about to finish my two hours of morning calls
15:54:14 efried cool, thanks.
16:01:58 ganso alex_xu: thanks for the +W! =)
16:03:50 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/650060
16:25:23 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: prevent common kwargs from glance client failure https://review.openstack.org/650064
16:25:34 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: prevent common kwargs from glance client failure https://review.openstack.org/650065
17:12:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle Invalid exceptions as expected in attach_interface https://review.openstack.org/650077
18:32:09 openstackgerrit Adrian Chiris proposed openstack/nova master: Move get_pci_mapping_for_migration to MigrationContext https://review.openstack.org/643023
18:32:09 openstackgerrit Adrian Chiris proposed openstack/nova master: Allow driver to properly unplug VIFs on destination on confirm resize https://review.openstack.org/643024
18:33:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Do not log a warning about not using compute monitors https://review.openstack.org/650092
18:33:17 mriedem jaypipes: i know you love compute monitors ^
18:33:39 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Excluding%20nova.compute.monitors.cpu%20monitor%20virt_driver.%20Not%20in%20the%20list%20of%20enabled%20monitors%20(CONF.compute_monitors).%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
18:49:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle PortLimitExceeded in POST /servers/{server_id}/os-interface https://review.openstack.org/650097
18:52:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Handle PortLimitExceeded in POST /servers/{server_id}/os-interface https://review.openstack.org/650097
19:08:26 mriedem melwitt: have you seen this? intermittent unit test failure http://logs.openstack.org/06/641806/7/check/openstack-tox-lower-constraints/1f1fce6/job-output.txt.gz#_2019-04-03_22_36_18_975096
19:12:27 mriedem looks like this unit test starts up a conductor service using the ServiceFixture which calls self.addCleanup(self.service.kill)
19:13:56 melwitt no, haven't seen that before. I've seen the error before, I remember helping to fix a bug for that way back when
19:14:40 melwitt this one https://review.openstack.org/555093
19:15:26 melwitt and in the fail you linked, we're getting it during service.destroy() this time
19:15:55 melwitt which must mean in that flow somewhere we have a reader transaction context nested outside a writer transaction context
19:18:08 mriedem yeah not sure, also doesn't really show up much in logstash
19:18:13 mriedem so i guess i'll recheck and assume it's a fluke
19:18:27 melwitt looking at the code, so far I don't see how the nesting could be happening
19:33:55 mriedem efried_mtg: these failures http://logs.openstack.org/13/621313/21/check/openstack-tox-py36/ba2c848/testr_results.html.gz definitely started spiking after 4/1 with https://github.com/openstack/nova/commit/e5269f2fbb773953266fccbfd9df3f75e351eeb9
19:34:06 mriedem i wonder if removing kipsdist = True in tox.ini is contributing
19:34:17 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22sqlalchemy.exc.InterfaceError%3A%20(pymysql.err.InterfaceError)%20(0%2C%20'')%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
19:34:23 mriedem *skipsdist = True
19:34:58 mriedem and it's always on node provider: limestone-regionone
19:45:47 mriedem tox on the limestone node is 3.8.4 and on an ovh node that passes tox is 3.8.1
19:45:52 mriedem pymysql is the same between them
19:47:00 efried mriedem: What can I do to help?
19:47:13 mriedem hold me
19:47:36 mriedem but not in a tap out kind of way
19:47:46 mriedem i'm reporting a new bug and will push an e-r query
19:48:20 efried should I push a change adding skipdist back and see what happens?
19:49:02 mriedem idk, i wasn't involved in that lower-constraints fix so i don't know if that will screw up the lower-constraints job, but i guess you'd find out
19:49:24 efried according to the commit message, cdent thought skipdist was "otherwise superfluous"
19:49:35 efried so hopefully that means it at least won't hurt anything.
19:49:46 mriedem if lower-constraints needs usedevelop=false and skipsdist=false, it could just specify that explicitly
19:49:53 efried yuh
19:50:07 mriedem usedevelop defaults to false also, so i'm not sure why the lower-constraints job is calling it out
19:50:39 mriedem i'm currently being defeated by launchpad timeouts
19:53:21 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Re-add skipsdist=True https://review.openstack.org/650123
19:53:34 efried mriedem: ^ any way to get that to run on the offending node?
19:53:50 mriedem not that i know of
19:54:08 mriedem unless there is a way to specify that in the zuul job definition
19:54:10 mriedem fungi: ^
19:54:34 mriedem question is can we restrict a change to only run on limestone-regionone nodes
19:56:28 mriedem btw this would explain the unit test job timeouts creeping up again
19:56:45 mriedem b/c when this fails, i think there is some thread switch or something and the fixture just goes away and we timeout
19:57:15 logan- i can provide a project to run it on in limestone-regionone
19:58:20 logan- aside from setting up custom nodepool labels I don't think there is a way to set affinity in the job
19:59:34 openstackgerrit Merged openstack/nova master: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/649775
19:59:41 openstackgerrit Merged openstack/nova master: Add docs on what not to include in notifications https://review.openstack.org/650020
20:04:52 fungi mriedem: efried: logan- is correct. we'd need special node labels or nodesets
20:05:11 mriedem https://bugs.launchpad.net/nova/+bug/1823251
20:05:12 openstack Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019" [Medium,Confirmed]
20:05:28 mriedem fungi: logan-: ok i guess we'll just tinker with seeing if adding skipsdist=True back fixes the issue
20:05:31 mriedem and i'll get an e-r query up
20:06:06 efried mriedem: So is https://review.openstack.org/#/c/650043/ not safe to recheck?
20:06:21 efried we're holding RC2 for it, nah?
20:06:23 mriedem efried: melwitt already approved it
20:06:32 mriedem but i just looked at the grenade failure, it's a known gate failure
20:06:32 efried yes but it's sitting failed
20:06:45 mriedem don't need to recheck it if it's approved now
20:06:46 efried right, I'm asking if it's a 100% failure or if we can recheck?
20:06:49 melwitt it's in the zuul check queue
20:06:54 efried oh, okay.
20:13:30 mriedem for this pymysql InterfaceError, it's showing up in more than just nova,
20:13:44 mriedem so did something change in upper-constraints or something more global on 4/1?
20:14:56 mriedem https://github.com/openstack/requirements/commit/27d0b42e23f8d964464862cb5590dea870ae9fd7#diff-0bdd949ed8a7fdd4f95240bd951779c8 merged on 4/1
20:15:48 mriedem i don't see anything obvious though
20:19:56 mriedem logan-: are you able to tell if anything changed around 4/1 wrt limestone-regionone bionic node packages?
20:20:10 mriedem specifically related to mysql/sqlalchemy/pymysql
20:20:17 mriedem although the latter 2 should be coming from pypi in our CI runs
20:20:41 logan- the region came back online after a p->q->r upgrade on 3/27, nothing has changed since then though.
20:21:08 logan- since we have been chatting, I just found a config issue that is being resolved: https://review.openstack.org/#/c/650135/
20:22:07 mriedem heh what a coincidence https://review.openstack.org/#/c/641981/
20:23:23 logan- hah
20:39:06 openstackgerrit Eric Fried proposed openstack/nova master: DNM: Re-add skipsdist=True, except for lower-constraints https://review.openstack.org/650123
20:40:29 efried kashyap, aspiers: FYI, I did not seed the review-day etherpad with the specs you mentioned, because it sounded like they weren't quite ready for that yet.
20:44:58 openstackgerrit Michael Still proposed openstack/nova master: Hacking N362: Don't abbrev/alias privsep import https://review.openstack.org/649190
20:44:59 openstackgerrit Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.path. https://review.openstack.org/648601
20:44:59 openstackgerrit Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs. https://review.openstack.org/648602
20:45:00 openstackgerrit Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs, continued. https://review.openstack.org/648603
20:45:00 openstackgerrit Michael Still proposed openstack/nova master: Add test coverage for nova.privsep.libvirt. https://review.openstack.org/648616
20:45:01 openstackgerrit Michael Still proposed openstack/nova master: Add test coverage for nova.privsep.qemu. https://review.openstack.org/649191
20:45:01 openstackgerrit Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431
20:45:02 openstackgerrit Michael Still proposed openstack/nova master: Remove unused FP device creation and deletion methods. https://review.openstack.org/635433
20:45:02 openstackgerrit Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435
20:45:03 openstackgerrit Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436
20:45:03 openstackgerrit Michael Still proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.openstack.org/636508
20:45:04 openstackgerrit Michael Still proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.openstack.org/639280
20:45:04 openstackgerrit Michael Still proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.openstack.org/639281
20:45:05 openstackgerrit Michael Still proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.openstack.org/639282
20:45:05 openstackgerrit Michael Still proposed openstack/nova master: Move setting of device trust to privsep. https://review.openstack.org/639283
20:45:06 openstackgerrit Michael Still proposed openstack/nova master: Move final bridge commands to privsep. https://review.openstack.org/639580
20:45:06 openstackgerrit Michael Still proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.openstack.org/639581

Earlier   Later