| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 18:27:50 | efried | rgerganov yt? | |
| 18:27:54 | sean-k-mooney | im assumeing you dont have the later installed in this gate and you disable nova upgrade right ? | |
| 18:28:10 | mriedem | yeah os-vif is 1.7.0 in pip freeze | |
| 18:28:13 | mriedem | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/pip2-freeze.txt.gz | |
| 18:28:49 | jroll | so in this run, it started segfaulting at 05:49:48.642840 | |
| 18:28:56 | jroll | here is grenade.txt at that time-ish http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_49_47_199 | |
| 18:29:12 | jroll | gotta be one of those first few imo | |
| 18:29:35 | cdent | efried: he's a time zone before me, and usually pretty sane with regards to going home, so he's probably not around | |
| 18:30:24 | efried | cdent Swhat I figured, but it was worth a shot. Perhaps you can see it: https://review.openstack.org/#/c/533821/6/nova/scheduler/client/report.py@1399 | |
| 18:31:09 | jroll | simplejson does lots of C things: https://github.com/simplejson/simplejson/compare/v3.11.1...v3.13.2 | |
| 18:32:11 | mriedem | simplejson 3.13.2 has been in u-c for 2 months though | |
| 18:32:13 | mriedem | so it's not that | |
| 18:32:30 | mriedem | looking at things in https://github.com/openstack/requirements/commit/34d56244a87ac2a61170ab8fa81dc86dba70fc1f from 1/17 that are in that list | |
| 18:32:37 | mriedem | and which nova usess | |
| 18:33:08 | mriedem | i think that would only be babel, sphinx and cffi, and runtime code doesn't use sphinx | |
| 18:33:16 | mriedem | nor babel i don't think | |
| 18:33:20 | mriedem | so my money is on cffi | |
| 18:33:23 | jroll | right | |
| 18:33:30 | jroll | cffi isn't being upgraded at that time though | |
| 18:34:25 | cdent | efried: I think you're right | |
| 18:34:31 | efried | cdent Thanks for looking. | |
| 18:34:43 | efried | cdent I coded it all up as if I wasn't, then started writing tests... | |
| 18:34:50 | jroll | mriedem: hrm, there's also 2018-01-18 05:49:32.589 | + /opt/stack/new/grenade/projects/50_neutron/upgrade.sh:main:96 : sudo apt-get -y install python-qpid | |
| 18:35:01 | edleafe | efried: just looked, too, and I can't find any way it could be None, either | |
| 18:35:08 | efried | edleafe Thank you. | |
| 18:35:31 | sean-k-mooney | jroll: the gate runs on rabbitmq but perhaps python-qupid has a dep that upgraded something | |
| 18:35:43 | jroll | mriedem: oh, and from apt-get update about 3 minutes before crashy crashy http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_46_59_521 | |
| 18:35:48 | cdent | efried: I think one of the thing that makes me confused about the ProviderTree stuff is that it behaves as if it is a strong Type. Which may make sense in this context, but I struggle to get used to it; it changes some idioms. | |
| 18:36:05 | efried | cdent What do you mean by "strong Type"? | |
| 18:36:06 | jroll | nova definitely uses python-libvirt | |
| 18:36:20 | mriedem | qpid shouldn't be getting installed | |
| 18:36:25 | edleafe | cdent: you mean static type? | |
| 18:36:38 | sean-k-mooney | finding the lib that change however wont resolve the issue will it. this seams like a general class of problem. | |
| 18:36:58 | sean-k-mooney | python-libvirt is used by n-cpu but not the conductor | |
| 18:37:38 | jroll | there's definitely a possibility it's imported by the conductor, though | |
| 18:37:44 | jroll | in some way | |
| 18:38:05 | jroll | sean-k-mooney: yes, it's a general class of problem with a known solution: don't use system packages :P | |
| 18:38:54 | cdent | edleafe, efried: I guess static could work as well, but what I mean is that it is written as if we're in a strongly typed language and the ProviderTree itself is a (static) type, made of up of statically typed things. <- This is not a relevant though, really, I'm just trying to suss out some of the sources of my anxiety with ProviderTree so I can flush them. | |
| 18:39:04 | jroll | I gotta step away for a bit so I can eat and stuff, sorry | |
| 18:39:11 | sean-k-mooney | jroll: i dont think the conductor is aware of the hypervisor so i dont think it would ever import phyton-libvirt. unless for the livemigration events? | |
| 18:39:12 | cdent | stuff | |
| 18:39:43 | jroll | cdent: short for stuff my face :D | |
| 18:39:55 | jroll | sean-k-mooney: not on purpose, but it's a tangled web | |
| 18:40:24 | mriedem | conductor only talks to n-cpu over rpc | |
| 18:40:36 | cdent | but yes, edleafe, I'm noodling in a way that uses terms incorrectly | |
| 18:41:09 | edleafe | cdent: no worries; just wanted to be sure I understood your thoughts | |
| 18:41:12 | sean-k-mooney | mriedem: ya i think python-libvirts use shoudl be limited to just the libvirt virt driver correct | |
| 18:41:22 | mriedem | yes | |
| 18:48:07 | mriedem | jroll: for when you get back, it looks like ovs was getting updated too around the time things started crashing http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/openvswitch/ovs-vswitchd.txt.gz | |
| 18:48:14 | mriedem | 2018-01-18T05:46:42.826Z|05730|rconn|INFO|brbm<->tcp:127.0.0.1:6633: connection closed by peer | |
| 18:50:24 | mriedem | 2018-01-18T05:46:38.988Z|00007|jsonrpc|WARN|tcp:127.0.0.1:47588: receive error: Connection reset by peer | |
| 18:51:30 | mriedem | and neutron is upgraded right before ironic when things start blowing up | |
| 18:57:33 | mriedem | 2018-01-18 05:46:41.213 | + /opt/stack/new/devstack/functions-common:stop_process:1528 : sudo systemctl stop devstack@q-agt.service | |
| 19:01:59 | sean-k-mooney | mriedem: looking at the systemd service files on my local system there is no depency between q-agt or n-cpu and n-cond or the supper conductor for that matter so that service stop should not propegate to nova services | |
| 19:02:54 | sean-k-mooney | if the connect reset is in ovs-vswitchd.txt.gz it indicates teh q-agt was upgraded rather then ovs | |
| 19:03:25 | mriedem | yeah q-agt is being stopped at 5:46:42 | |
| 19:04:26 | sean-k-mooney | so its likely one (proably several) of neutron deps is common with nova | |
| 19:04:52 | sean-k-mooney | in the normal upgrade sequence nova would upgrade before neutron correct | |
| 19:05:17 | sean-k-mooney | i mean technicall i dont think it has to but i think that is correct for greade | |
| 19:05:20 | mriedem | no | |
| 19:05:24 | mriedem | neutron goes before nova | |
| 19:05:29 | sean-k-mooney | oh ok | |
| 19:06:22 | mriedem | Jan 18 05:48:49 ubuntu-xenial-inap-mtl01-0001976291 kernel: traps: nova-compute[1669] general protection ip:4a7f54 sp:7ffd13de8328 error:0 in python2.7[400000+2e9000] | |
| 19:06:35 | mriedem | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_49 | |
| 19:07:35 | mriedem | Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:07:37 | sean-k-mooney | mriedem: ya so that basically a segfault | |
| 19:07:45 | mriedem | yeah, and not helpful as to what failed | |
| 19:08:00 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:08:00 | mriedem | right before it: | |
| 19:08:20 | mriedem | neutron is installing stuff | |
| 19:09:36 | sean-k-mooney | well it happend 5 seconds after we start installing the swift test-requirements http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_44 | |
| 19:12:07 | sean-k-mooney | the sudo process for install the swift test requiremetns pid:15728 end at http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_54 | |
| 19:13:00 | sean-k-mooney | so the compute agent segfault while we are upgradeing the test requiremets for swift | |
| 19:14:16 | sean-k-mooney | that said nothing in https://github.com/openstack/swift/blob/master/test-requirements.txt jumps out at me | |
| 19:15:30 | mriedem | i don't know why test-requirement would be getting installed in a devstack env anyway | |
| 19:15:33 | mriedem | test-requirements should be for unit tests | |
| 19:16:00 | sean-k-mooney | mriedem: that is a dispute i have had with devstack for 2-3 releases | |
| 19:16:11 | sean-k-mooney | devstack installs all test-requiremets systemwide | |
| 19:16:55 | sdague | test-requirements also got overloaded by lots of projects as "optional requirements" and people put all their drivers in there | |
| 19:17:12 | sdague | it's never had clear definitions | |
| 19:17:30 | sdague | it was made to always install... 3 years ago? | |
| 19:17:45 | mriedem | ok so neutron reqs are installed here for queens | |
| 19:17:46 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:17:51 | sean-k-mooney | sdague: yes but if they are optional they should only be installed in the gates that need them :) | |
| 19:17:58 | mriedem | then nova-conductor segfaults Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:18:03 | mriedem | and nova-conductor is pike at this point | |
| 19:19:00 | sean-k-mooney | sdague: yep but it only bit me with the flake8 import ording plugin that was added then blacklisted to nova test-requriement that ended up brakeing our ovs-dpdk gates because ovs enabled pip but disable that check internally | |
| 19:19:45 | sean-k-mooney | mriedem: well that appears to be the same memory adress in both python2.7[400000+2e9000] | |
| 19:22:48 | sean-k-mooney | mriedem: this is a time issue tool any of the previous pip installs could have cased it, as we cant know how long it will be betwen the module being replaced and the dangeling function pointer being deferenced | |
| 19:23:19 | sean-k-mooney | * timing issue too | |
| 19:23:29 | openstackgerrit | Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288 | |
| 19:25:47 | efried | edleafe jaypipes Just found a nice leetle bug | |
| 19:25:52 | sean-k-mooney | sdague: by the way that was not a did at you. i know why you made the change i.e the multiple code paths and the dep issue it was causeing. | |
| 19:26:25 | mriedem | hmm, so n-cpu starts failing around 5:46 and fails to connect to the ironic api | |
| 19:26:32 | efried | sdague too | |
| 19:26:33 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver https://review.openstack.org/532290 | |
| 19:26:33 | openstackgerrit | Mark Goddard proposed openstack/nova master: Send traits to ironic on server boot https://review.openstack.org/508116 | |
| 19:26:34 | efried | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L296-L302 | |
| 19:26:34 | openstackgerrit | Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288 | |
| 19:26:36 | mriedem | and i think that's around when glance is being upgraded/restarted | |