| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 18:35:43 | jroll | mriedem: oh, and from apt-get update about 3 minutes before crashy crashy http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_46_59_521 | |
| 18:35:48 | cdent | efried: I think one of the thing that makes me confused about the ProviderTree stuff is that it behaves as if it is a strong Type. Which may make sense in this context, but I struggle to get used to it; it changes some idioms. | |
| 18:36:05 | efried | cdent What do you mean by "strong Type"? | |
| 18:36:06 | jroll | nova definitely uses python-libvirt | |
| 18:36:20 | mriedem | qpid shouldn't be getting installed | |
| 18:36:25 | edleafe | cdent: you mean static type? | |
| 18:36:38 | sean-k-mooney | finding the lib that change however wont resolve the issue will it. this seams like a general class of problem. | |
| 18:36:58 | sean-k-mooney | python-libvirt is used by n-cpu but not the conductor | |
| 18:37:38 | jroll | there's definitely a possibility it's imported by the conductor, though | |
| 18:37:44 | jroll | in some way | |
| 18:38:05 | jroll | sean-k-mooney: yes, it's a general class of problem with a known solution: don't use system packages :P | |
| 18:38:54 | cdent | edleafe, efried: I guess static could work as well, but what I mean is that it is written as if we're in a strongly typed language and the ProviderTree itself is a (static) type, made of up of statically typed things. <- This is not a relevant though, really, I'm just trying to suss out some of the sources of my anxiety with ProviderTree so I can flush them. | |
| 18:39:04 | jroll | I gotta step away for a bit so I can eat and stuff, sorry | |
| 18:39:11 | sean-k-mooney | jroll: i dont think the conductor is aware of the hypervisor so i dont think it would ever import phyton-libvirt. unless for the livemigration events? | |
| 18:39:12 | cdent | stuff | |
| 18:39:43 | jroll | cdent: short for stuff my face :D | |
| 18:39:55 | jroll | sean-k-mooney: not on purpose, but it's a tangled web | |
| 18:40:24 | mriedem | conductor only talks to n-cpu over rpc | |
| 18:40:36 | cdent | but yes, edleafe, I'm noodling in a way that uses terms incorrectly | |
| 18:41:09 | edleafe | cdent: no worries; just wanted to be sure I understood your thoughts | |
| 18:41:12 | sean-k-mooney | mriedem: ya i think python-libvirts use shoudl be limited to just the libvirt virt driver correct | |
| 18:41:22 | mriedem | yes | |
| 18:48:07 | mriedem | jroll: for when you get back, it looks like ovs was getting updated too around the time things started crashing http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/openvswitch/ovs-vswitchd.txt.gz | |
| 18:48:14 | mriedem | 2018-01-18T05:46:42.826Z|05730|rconn|INFO|brbm<->tcp:127.0.0.1:6633: connection closed by peer | |
| 18:50:24 | mriedem | 2018-01-18T05:46:38.988Z|00007|jsonrpc|WARN|tcp:127.0.0.1:47588: receive error: Connection reset by peer | |
| 18:51:30 | mriedem | and neutron is upgraded right before ironic when things start blowing up | |
| 18:57:33 | mriedem | 2018-01-18 05:46:41.213 | + /opt/stack/new/devstack/functions-common:stop_process:1528 : sudo systemctl stop devstack@q-agt.service | |
| 19:01:59 | sean-k-mooney | mriedem: looking at the systemd service files on my local system there is no depency between q-agt or n-cpu and n-cond or the supper conductor for that matter so that service stop should not propegate to nova services | |
| 19:02:54 | sean-k-mooney | if the connect reset is in ovs-vswitchd.txt.gz it indicates teh q-agt was upgraded rather then ovs | |
| 19:03:25 | mriedem | yeah q-agt is being stopped at 5:46:42 | |
| 19:04:26 | sean-k-mooney | so its likely one (proably several) of neutron deps is common with nova | |
| 19:04:52 | sean-k-mooney | in the normal upgrade sequence nova would upgrade before neutron correct | |
| 19:05:17 | sean-k-mooney | i mean technicall i dont think it has to but i think that is correct for greade | |
| 19:05:20 | mriedem | no | |
| 19:05:24 | mriedem | neutron goes before nova | |
| 19:05:29 | sean-k-mooney | oh ok | |
| 19:06:22 | mriedem | Jan 18 05:48:49 ubuntu-xenial-inap-mtl01-0001976291 kernel: traps: nova-compute[1669] general protection ip:4a7f54 sp:7ffd13de8328 error:0 in python2.7[400000+2e9000] | |
| 19:06:35 | mriedem | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_49 | |
| 19:07:35 | mriedem | Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:07:37 | sean-k-mooney | mriedem: ya so that basically a segfault | |
| 19:07:45 | mriedem | yeah, and not helpful as to what failed | |
| 19:08:00 | mriedem | right before it: | |
| 19:08:00 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:08:20 | mriedem | neutron is installing stuff | |
| 19:09:36 | sean-k-mooney | well it happend 5 seconds after we start installing the swift test-requirements http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_44 | |
| 19:12:07 | sean-k-mooney | the sudo process for install the swift test requiremetns pid:15728 end at http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_54 | |
| 19:13:00 | sean-k-mooney | so the compute agent segfault while we are upgradeing the test requiremets for swift | |
| 19:14:16 | sean-k-mooney | that said nothing in https://github.com/openstack/swift/blob/master/test-requirements.txt jumps out at me | |
| 19:15:30 | mriedem | i don't know why test-requirement would be getting installed in a devstack env anyway | |
| 19:15:33 | mriedem | test-requirements should be for unit tests | |
| 19:16:00 | sean-k-mooney | mriedem: that is a dispute i have had with devstack for 2-3 releases | |
| 19:16:11 | sean-k-mooney | devstack installs all test-requiremets systemwide | |
| 19:16:55 | sdague | test-requirements also got overloaded by lots of projects as "optional requirements" and people put all their drivers in there | |
| 19:17:12 | sdague | it's never had clear definitions | |
| 19:17:30 | sdague | it was made to always install... 3 years ago? | |
| 19:17:45 | mriedem | ok so neutron reqs are installed here for queens | |
| 19:17:46 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:17:51 | sean-k-mooney | sdague: yes but if they are optional they should only be installed in the gates that need them :) | |
| 19:17:58 | mriedem | then nova-conductor segfaults Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:18:03 | mriedem | and nova-conductor is pike at this point | |
| 19:19:00 | sean-k-mooney | sdague: yep but it only bit me with the flake8 import ording plugin that was added then blacklisted to nova test-requriement that ended up brakeing our ovs-dpdk gates because ovs enabled pip but disable that check internally | |
| 19:19:45 | sean-k-mooney | mriedem: well that appears to be the same memory adress in both python2.7[400000+2e9000] | |
| 19:22:48 | sean-k-mooney | mriedem: this is a time issue tool any of the previous pip installs could have cased it, as we cant know how long it will be betwen the module being replaced and the dangeling function pointer being deferenced | |
| 19:23:19 | sean-k-mooney | * timing issue too | |
| 19:23:29 | openstackgerrit | Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288 | |
| 19:25:47 | efried | edleafe jaypipes Just found a nice leetle bug | |
| 19:25:52 | sean-k-mooney | sdague: by the way that was not a did at you. i know why you made the change i.e the multiple code paths and the dep issue it was causeing. | |
| 19:26:25 | mriedem | hmm, so n-cpu starts failing around 5:46 and fails to connect to the ironic api | |
| 19:26:32 | efried | sdague too | |
| 19:26:33 | openstackgerrit | Mark Goddard proposed openstack/nova master: Send traits to ironic on server boot https://review.openstack.org/508116 | |
| 19:26:33 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver https://review.openstack.org/532290 | |
| 19:26:34 | openstackgerrit | Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288 | |
| 19:26:34 | efried | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L296-L302 | |
| 19:26:36 | mriedem | and i think that's around when glance is being upgraded/restarted | |
| 19:26:46 | mriedem | maybe some weird uwsgi stuff? | |
| 19:27:02 | efried | edleafe jaypipes sdague The bug is that e.g. [] is a valid json payload. | |
| 19:27:52 | sdague | efried: nice | |
| 19:28:17 | jaypipes | efried: sure, I suppose. though I prefer DELETE :) | |
| 19:28:17 | sean-k-mooney | mriedem: http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/apache/error.txt.gz | |
| 19:28:33 | sean-k-mooney | keysonten is dead from 5:47 | |
| 19:29:02 | efried | jaypipes No DELETE API for aggs :( | |
| 19:29:08 | owalsh | dansmith: hey, hitting an issue with cell_v2 setup https://bugs.launchpad.net/nova/+bug/1744780 | |
| 19:29:09 | openstack | Launchpad bug 1744780 in tripleo "Nova::Cell_v2::Simple_setup/Nova_cell_v2[cell0] fails on master promote" [Critical,Triaged] | |
| 19:29:33 | jaypipes | efried: ooh, yeah... | |
| 19:29:45 | mriedem | [Thu Jan 18 05:50:49.905807 2018] [wsgi:warn] [pid 30759:tid 139966291453824] mod_wsgi: Compiled for Python/2.7.11. | |
| 19:29:47 | efried | fixing | |
| 19:29:48 | jaypipes | efried: prolly should have one, but yeah | |
| 19:29:49 | mriedem | [Thu Jan 18 05:50:49.905865 2018] [wsgi:warn] [pid 30759:tid 139966291453824] mod_wsgi: Runtime using Python/2.7.12. | |
| 19:31:07 | mriedem | mtreinish: is anything using this code yet? https://review.openstack.org/#/c/531493/ | |
| 19:31:58 | mriedem | i guess not https://review.openstack.org/#/c/531498/ | |
| 19:33:59 | melwitt | owalsh: that's a recent change that we're not allowing creation/update of a cell with a DB connection + transport URL combination that already exists | |
| 19:34:22 | owalsh | melwitt: yea, but it's not removing the target cell from the cells it checks for dups | |
| 19:34:32 | owalsh | melwitt: so it fails on itself | |
| 19:34:53 | melwitt | hm | |
| 19:37:50 | efried | jaypipes edleafe sdague https://bugs.launchpad.net/nova/+bug/1744786 FYI | |
| 19:37:51 | openstack | Launchpad bug 1744786 in OpenStack Compute (nova) "SchedulerReportClient.put with empty (not None) payload errors 415" [Undecided,New] | |
| 19:40:44 | jroll | mriedem: looks like n-cpu is because we shut down ironic-api while upgrading things (which seems weird to me, but) | |
| 19:40:55 | jroll | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_44_56_948 | |
| 19:46:03 | melwitt | owalsh: I see what you mean. so the case you're hitting is trying to update_cell with only one of DB connection or transport URL being different then? | |
| 19:46:35 | dansmith | melwitt: yeah I expect so | |