| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 18:14:41 | jroll | mriedem: we had another failure for about 24-36 hours before that, so it's a bit masked, but the first instance is 2018-01-17T09:52:37.119Z in logstash | |
| 18:15:02 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Forking%20too%20fast%2C%20sleeping%5C%22%20AND%20tags%3A%5C%22screen-n-cond.txt%5C%22&from=10d | |
| 18:16:38 | mriedem | https://github.com/openstack/requirements/commit/482fca3e04b820045bb87d9b37470bc076d2216d was 1/16 | |
| 18:17:40 | jroll | right | |
| 18:18:01 | mriedem | not sure that commit should be a problem though since something has to opt into passing the new kwarg, else it defaults to the same as before | |
| 18:18:10 | mriedem | https://github.com/openstack/oslo.concurrency/compare/3.24.0...3.25.0 | |
| 18:18:33 | jroll | ah true | |
| 18:18:53 | mriedem | and i don't think we run with osprofiler enabled in this job do we? | |
| 18:18:56 | mriedem | worth checking | |
| 18:18:59 | jroll | not that I know of | |
| 18:19:10 | jroll | I don't think any ironic jobs have ever enabled that | |
| 18:19:28 | mriedem | Jan 18 04:49:32.628690 ubuntu-xenial-inap-mtl01-0001976291 nova-conductor[19600]: DEBUG oslo_service.service [None req-db5ca533-98f7-4358-8448-0201fe9a04d8 None None] profiler.enabled = False {{(pid=19600) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2883}} | |
| 18:19:33 | mriedem | yeah osprofiler isn't enabled | |
| 18:20:24 | mriedem | https://github.com/openstack/requirements/commit/34d56244a87ac2a61170ab8fa81dc86dba70fc1f was 1/17 | |
| 18:20:25 | mriedem | cffi? | |
| 18:20:53 | jroll | could be | |
| 18:21:03 | mriedem | could try reverting that back to 1.11.2 and run a depends-on with an ironic patch | |
| 18:21:11 | jroll | https://github.com/cffi/cffi/compare/1.11.2...1.11.4 | |
| 18:21:19 | jroll | nothing O_o | |
| 18:21:45 | jroll | oh, they don't tag, good | |
| 18:21:55 | mriedem | http://cffi.readthedocs.io/en/latest/whatsnew.html#v1-11-4 | |
| 18:22:26 | jroll | windows, py3, meh | |
| 18:22:44 | mriedem | that's just the stuff they call out in the release notes | |
| 18:22:50 | mriedem | but yeah | |
| 18:24:04 | mriedem | https://github.com/openstack/requirements/commit/93d488328bfc780c322338226b6c62e9141637f4 | |
| 18:24:12 | mriedem | note the "and introduces a file handel leak due to an upstream bug in pyroute2" | |
| 18:24:46 | mriedem | https://github.com/openstack/os-vif/compare/1.7.0...1.9.0 | |
| 18:24:49 | jroll | mmmm | |
| 18:26:37 | mriedem | https://github.com/openstack/os-vif/commit/570c05266fa6231a21d70f2917ac0a933ac8ce7b | |
| 18:26:46 | sean-k-mooney | i dont think os-vif is the cause as in 1.7 we did not use pyroute2 and in 1.9.0 we have disabled and use ip tools instead. | |
| 18:26:46 | jroll | doesn't look like os-vif gets upgraded, though: http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz | |
| 18:27:15 | sean-k-mooney | jroll: the only project useing os-vif are nova and kuyr-kubernetes | |
| 18:27:50 | efried | rgerganov yt? | |
| 18:27:54 | sean-k-mooney | im assumeing you dont have the later installed in this gate and you disable nova upgrade right ? | |
| 18:28:10 | mriedem | yeah os-vif is 1.7.0 in pip freeze | |
| 18:28:13 | mriedem | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/pip2-freeze.txt.gz | |
| 18:28:49 | jroll | so in this run, it started segfaulting at 05:49:48.642840 | |
| 18:28:56 | jroll | here is grenade.txt at that time-ish http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_49_47_199 | |
| 18:29:12 | jroll | gotta be one of those first few imo | |
| 18:29:35 | cdent | efried: he's a time zone before me, and usually pretty sane with regards to going home, so he's probably not around | |
| 18:30:24 | efried | cdent Swhat I figured, but it was worth a shot. Perhaps you can see it: https://review.openstack.org/#/c/533821/6/nova/scheduler/client/report.py@1399 | |
| 18:31:09 | jroll | simplejson does lots of C things: https://github.com/simplejson/simplejson/compare/v3.11.1...v3.13.2 | |
| 18:32:11 | mriedem | simplejson 3.13.2 has been in u-c for 2 months though | |
| 18:32:13 | mriedem | so it's not that | |
| 18:32:30 | mriedem | looking at things in https://github.com/openstack/requirements/commit/34d56244a87ac2a61170ab8fa81dc86dba70fc1f from 1/17 that are in that list | |
| 18:32:37 | mriedem | and which nova usess | |
| 18:33:08 | mriedem | i think that would only be babel, sphinx and cffi, and runtime code doesn't use sphinx | |
| 18:33:16 | mriedem | nor babel i don't think | |
| 18:33:20 | mriedem | so my money is on cffi | |
| 18:33:23 | jroll | right | |
| 18:33:30 | jroll | cffi isn't being upgraded at that time though | |
| 18:34:25 | cdent | efried: I think you're right | |
| 18:34:31 | efried | cdent Thanks for looking. | |
| 18:34:43 | efried | cdent I coded it all up as if I wasn't, then started writing tests... | |
| 18:34:50 | jroll | mriedem: hrm, there's also 2018-01-18 05:49:32.589 | + /opt/stack/new/grenade/projects/50_neutron/upgrade.sh:main:96 : sudo apt-get -y install python-qpid | |
| 18:35:01 | edleafe | efried: just looked, too, and I can't find any way it could be None, either | |
| 18:35:08 | efried | edleafe Thank you. | |
| 18:35:31 | sean-k-mooney | jroll: the gate runs on rabbitmq but perhaps python-qupid has a dep that upgraded something | |
| 18:35:43 | jroll | mriedem: oh, and from apt-get update about 3 minutes before crashy crashy http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_46_59_521 | |
| 18:35:48 | cdent | efried: I think one of the thing that makes me confused about the ProviderTree stuff is that it behaves as if it is a strong Type. Which may make sense in this context, but I struggle to get used to it; it changes some idioms. | |
| 18:36:05 | efried | cdent What do you mean by "strong Type"? | |
| 18:36:06 | jroll | nova definitely uses python-libvirt | |
| 18:36:20 | mriedem | qpid shouldn't be getting installed | |
| 18:36:25 | edleafe | cdent: you mean static type? | |
| 18:36:38 | sean-k-mooney | finding the lib that change however wont resolve the issue will it. this seams like a general class of problem. | |
| 18:36:58 | sean-k-mooney | python-libvirt is used by n-cpu but not the conductor | |
| 18:37:38 | jroll | there's definitely a possibility it's imported by the conductor, though | |
| 18:37:44 | jroll | in some way | |
| 18:38:05 | jroll | sean-k-mooney: yes, it's a general class of problem with a known solution: don't use system packages :P | |
| 18:38:54 | cdent | edleafe, efried: I guess static could work as well, but what I mean is that it is written as if we're in a strongly typed language and the ProviderTree itself is a (static) type, made of up of statically typed things. <- This is not a relevant though, really, I'm just trying to suss out some of the sources of my anxiety with ProviderTree so I can flush them. | |
| 18:39:04 | jroll | I gotta step away for a bit so I can eat and stuff, sorry | |
| 18:39:11 | sean-k-mooney | jroll: i dont think the conductor is aware of the hypervisor so i dont think it would ever import phyton-libvirt. unless for the livemigration events? | |
| 18:39:12 | cdent | stuff | |
| 18:39:43 | jroll | cdent: short for stuff my face :D | |
| 18:39:55 | jroll | sean-k-mooney: not on purpose, but it's a tangled web | |
| 18:40:24 | mriedem | conductor only talks to n-cpu over rpc | |
| 18:40:36 | cdent | but yes, edleafe, I'm noodling in a way that uses terms incorrectly | |
| 18:41:09 | edleafe | cdent: no worries; just wanted to be sure I understood your thoughts | |
| 18:41:12 | sean-k-mooney | mriedem: ya i think python-libvirts use shoudl be limited to just the libvirt virt driver correct | |
| 18:41:22 | mriedem | yes | |
| 18:48:07 | mriedem | jroll: for when you get back, it looks like ovs was getting updated too around the time things started crashing http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/openvswitch/ovs-vswitchd.txt.gz | |
| 18:48:14 | mriedem | 2018-01-18T05:46:42.826Z|05730|rconn|INFO|brbm<->tcp:127.0.0.1:6633: connection closed by peer | |
| 18:50:24 | mriedem | 2018-01-18T05:46:38.988Z|00007|jsonrpc|WARN|tcp:127.0.0.1:47588: receive error: Connection reset by peer | |
| 18:51:30 | mriedem | and neutron is upgraded right before ironic when things start blowing up | |
| 18:57:33 | mriedem | 2018-01-18 05:46:41.213 | + /opt/stack/new/devstack/functions-common:stop_process:1528 : sudo systemctl stop devstack@q-agt.service | |
| 19:01:59 | sean-k-mooney | mriedem: looking at the systemd service files on my local system there is no depency between q-agt or n-cpu and n-cond or the supper conductor for that matter so that service stop should not propegate to nova services | |
| 19:02:54 | sean-k-mooney | if the connect reset is in ovs-vswitchd.txt.gz it indicates teh q-agt was upgraded rather then ovs | |
| 19:03:25 | mriedem | yeah q-agt is being stopped at 5:46:42 | |
| 19:04:26 | sean-k-mooney | so its likely one (proably several) of neutron deps is common with nova | |
| 19:04:52 | sean-k-mooney | in the normal upgrade sequence nova would upgrade before neutron correct | |
| 19:05:17 | sean-k-mooney | i mean technicall i dont think it has to but i think that is correct for greade | |
| 19:05:20 | mriedem | no | |
| 19:05:24 | mriedem | neutron goes before nova | |
| 19:05:29 | sean-k-mooney | oh ok | |
| 19:06:22 | mriedem | Jan 18 05:48:49 ubuntu-xenial-inap-mtl01-0001976291 kernel: traps: nova-compute[1669] general protection ip:4a7f54 sp:7ffd13de8328 error:0 in python2.7[400000+2e9000] | |
| 19:06:35 | mriedem | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_49 | |
| 19:07:35 | mriedem | Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:07:37 | sean-k-mooney | mriedem: ya so that basically a segfault | |
| 19:07:45 | mriedem | yeah, and not helpful as to what failed | |
| 19:08:00 | mriedem | right before it: | |