| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 18:21:55 | mriedem | http://cffi.readthedocs.io/en/latest/whatsnew.html#v1-11-4 | |
| 18:22:26 | jroll | windows, py3, meh | |
| 18:22:44 | mriedem | that's just the stuff they call out in the release notes | |
| 18:22:50 | mriedem | but yeah | |
| 18:24:04 | mriedem | https://github.com/openstack/requirements/commit/93d488328bfc780c322338226b6c62e9141637f4 | |
| 18:24:12 | mriedem | note the "and introduces a file handel leak due to an upstream bug in pyroute2" | |
| 18:24:46 | mriedem | https://github.com/openstack/os-vif/compare/1.7.0...1.9.0 | |
| 18:24:49 | jroll | mmmm | |
| 18:26:37 | mriedem | https://github.com/openstack/os-vif/commit/570c05266fa6231a21d70f2917ac0a933ac8ce7b | |
| 18:26:46 | jroll | doesn't look like os-vif gets upgraded, though: http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz | |
| 18:26:46 | sean-k-mooney | i dont think os-vif is the cause as in 1.7 we did not use pyroute2 and in 1.9.0 we have disabled and use ip tools instead. | |
| 18:27:15 | sean-k-mooney | jroll: the only project useing os-vif are nova and kuyr-kubernetes | |
| 18:27:50 | efried | rgerganov yt? | |
| 18:27:54 | sean-k-mooney | im assumeing you dont have the later installed in this gate and you disable nova upgrade right ? | |
| 18:28:10 | mriedem | yeah os-vif is 1.7.0 in pip freeze | |
| 18:28:13 | mriedem | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/pip2-freeze.txt.gz | |
| 18:28:49 | jroll | so in this run, it started segfaulting at 05:49:48.642840 | |
| 18:28:56 | jroll | here is grenade.txt at that time-ish http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_49_47_199 | |
| 18:29:12 | jroll | gotta be one of those first few imo | |
| 18:29:35 | cdent | efried: he's a time zone before me, and usually pretty sane with regards to going home, so he's probably not around | |
| 18:30:24 | efried | cdent Swhat I figured, but it was worth a shot. Perhaps you can see it: https://review.openstack.org/#/c/533821/6/nova/scheduler/client/report.py@1399 | |
| 18:31:09 | jroll | simplejson does lots of C things: https://github.com/simplejson/simplejson/compare/v3.11.1...v3.13.2 | |
| 18:32:11 | mriedem | simplejson 3.13.2 has been in u-c for 2 months though | |
| 18:32:13 | mriedem | so it's not that | |
| 18:32:30 | mriedem | looking at things in https://github.com/openstack/requirements/commit/34d56244a87ac2a61170ab8fa81dc86dba70fc1f from 1/17 that are in that list | |
| 18:32:37 | mriedem | and which nova usess | |
| 18:33:08 | mriedem | i think that would only be babel, sphinx and cffi, and runtime code doesn't use sphinx | |
| 18:33:16 | mriedem | nor babel i don't think | |
| 18:33:20 | mriedem | so my money is on cffi | |
| 18:33:23 | jroll | right | |
| 18:33:30 | jroll | cffi isn't being upgraded at that time though | |
| 18:34:25 | cdent | efried: I think you're right | |
| 18:34:31 | efried | cdent Thanks for looking. | |
| 18:34:43 | efried | cdent I coded it all up as if I wasn't, then started writing tests... | |
| 18:34:50 | jroll | mriedem: hrm, there's also 2018-01-18 05:49:32.589 | + /opt/stack/new/grenade/projects/50_neutron/upgrade.sh:main:96 : sudo apt-get -y install python-qpid | |
| 18:35:01 | edleafe | efried: just looked, too, and I can't find any way it could be None, either | |
| 18:35:08 | efried | edleafe Thank you. | |
| 18:35:31 | sean-k-mooney | jroll: the gate runs on rabbitmq but perhaps python-qupid has a dep that upgraded something | |
| 18:35:43 | jroll | mriedem: oh, and from apt-get update about 3 minutes before crashy crashy http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_46_59_521 | |
| 18:35:48 | cdent | efried: I think one of the thing that makes me confused about the ProviderTree stuff is that it behaves as if it is a strong Type. Which may make sense in this context, but I struggle to get used to it; it changes some idioms. | |
| 18:36:05 | efried | cdent What do you mean by "strong Type"? | |
| 18:36:06 | jroll | nova definitely uses python-libvirt | |
| 18:36:20 | mriedem | qpid shouldn't be getting installed | |
| 18:36:25 | edleafe | cdent: you mean static type? | |
| 18:36:38 | sean-k-mooney | finding the lib that change however wont resolve the issue will it. this seams like a general class of problem. | |
| 18:36:58 | sean-k-mooney | python-libvirt is used by n-cpu but not the conductor | |
| 18:37:38 | jroll | there's definitely a possibility it's imported by the conductor, though | |
| 18:37:44 | jroll | in some way | |
| 18:38:05 | jroll | sean-k-mooney: yes, it's a general class of problem with a known solution: don't use system packages :P | |
| 18:38:54 | cdent | edleafe, efried: I guess static could work as well, but what I mean is that it is written as if we're in a strongly typed language and the ProviderTree itself is a (static) type, made of up of statically typed things. <- This is not a relevant though, really, I'm just trying to suss out some of the sources of my anxiety with ProviderTree so I can flush them. | |
| 18:39:04 | jroll | I gotta step away for a bit so I can eat and stuff, sorry | |
| 18:39:11 | sean-k-mooney | jroll: i dont think the conductor is aware of the hypervisor so i dont think it would ever import phyton-libvirt. unless for the livemigration events? | |
| 18:39:12 | cdent | stuff | |
| 18:39:43 | jroll | cdent: short for stuff my face :D | |
| 18:39:55 | jroll | sean-k-mooney: not on purpose, but it's a tangled web | |
| 18:40:24 | mriedem | conductor only talks to n-cpu over rpc | |
| 18:40:36 | cdent | but yes, edleafe, I'm noodling in a way that uses terms incorrectly | |
| 18:41:09 | edleafe | cdent: no worries; just wanted to be sure I understood your thoughts | |
| 18:41:12 | sean-k-mooney | mriedem: ya i think python-libvirts use shoudl be limited to just the libvirt virt driver correct | |
| 18:41:22 | mriedem | yes | |
| 18:48:07 | mriedem | jroll: for when you get back, it looks like ovs was getting updated too around the time things started crashing http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/openvswitch/ovs-vswitchd.txt.gz | |
| 18:48:14 | mriedem | 2018-01-18T05:46:42.826Z|05730|rconn|INFO|brbm<->tcp:127.0.0.1:6633: connection closed by peer | |
| 18:50:24 | mriedem | 2018-01-18T05:46:38.988Z|00007|jsonrpc|WARN|tcp:127.0.0.1:47588: receive error: Connection reset by peer | |
| 18:51:30 | mriedem | and neutron is upgraded right before ironic when things start blowing up | |
| 18:57:33 | mriedem | 2018-01-18 05:46:41.213 | + /opt/stack/new/devstack/functions-common:stop_process:1528 : sudo systemctl stop devstack@q-agt.service | |
| 19:01:59 | sean-k-mooney | mriedem: looking at the systemd service files on my local system there is no depency between q-agt or n-cpu and n-cond or the supper conductor for that matter so that service stop should not propegate to nova services | |
| 19:02:54 | sean-k-mooney | if the connect reset is in ovs-vswitchd.txt.gz it indicates teh q-agt was upgraded rather then ovs | |
| 19:03:25 | mriedem | yeah q-agt is being stopped at 5:46:42 | |
| 19:04:26 | sean-k-mooney | so its likely one (proably several) of neutron deps is common with nova | |
| 19:04:52 | sean-k-mooney | in the normal upgrade sequence nova would upgrade before neutron correct | |
| 19:05:17 | sean-k-mooney | i mean technicall i dont think it has to but i think that is correct for greade | |
| 19:05:20 | mriedem | no | |
| 19:05:24 | mriedem | neutron goes before nova | |
| 19:05:29 | sean-k-mooney | oh ok | |
| 19:06:22 | mriedem | Jan 18 05:48:49 ubuntu-xenial-inap-mtl01-0001976291 kernel: traps: nova-compute[1669] general protection ip:4a7f54 sp:7ffd13de8328 error:0 in python2.7[400000+2e9000] | |
| 19:06:35 | mriedem | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_49 | |
| 19:07:35 | mriedem | Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:07:37 | sean-k-mooney | mriedem: ya so that basically a segfault | |
| 19:07:45 | mriedem | yeah, and not helpful as to what failed | |
| 19:08:00 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:08:00 | mriedem | right before it: | |
| 19:08:20 | mriedem | neutron is installing stuff | |
| 19:09:36 | sean-k-mooney | well it happend 5 seconds after we start installing the swift test-requirements http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_44 | |
| 19:12:07 | sean-k-mooney | the sudo process for install the swift test requiremetns pid:15728 end at http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_54 | |
| 19:13:00 | sean-k-mooney | so the compute agent segfault while we are upgradeing the test requiremets for swift | |
| 19:14:16 | sean-k-mooney | that said nothing in https://github.com/openstack/swift/blob/master/test-requirements.txt jumps out at me | |
| 19:15:30 | mriedem | i don't know why test-requirement would be getting installed in a devstack env anyway | |
| 19:15:33 | mriedem | test-requirements should be for unit tests | |
| 19:16:00 | sean-k-mooney | mriedem: that is a dispute i have had with devstack for 2-3 releases | |
| 19:16:11 | sean-k-mooney | devstack installs all test-requiremets systemwide | |
| 19:16:55 | sdague | test-requirements also got overloaded by lots of projects as "optional requirements" and people put all their drivers in there | |
| 19:17:12 | sdague | it's never had clear definitions | |
| 19:17:30 | sdague | it was made to always install... 3 years ago? | |
| 19:17:45 | mriedem | ok so neutron reqs are installed here for queens | |
| 19:17:46 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:17:51 | sean-k-mooney | sdague: yes but if they are optional they should only be installed in the gates that need them :) | |
| 19:17:58 | mriedem | then nova-conductor segfaults Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:18:03 | mriedem | and nova-conductor is pike at this point | |
| 19:19:00 | sean-k-mooney | sdague: yep but it only bit me with the flake8 import ording plugin that was added then blacklisted to nova test-requriement that ended up brakeing our ovs-dpdk gates because ovs enabled pip but disable that check internally | |
| 19:19:45 | sean-k-mooney | mriedem: well that appears to be the same memory adress in both python2.7[400000+2e9000] | |