| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-22 | |||
| 19:07:35 | mriedem | Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:07:37 | sean-k-mooney | mriedem: ya so that basically a segfault | |
| 19:07:45 | mriedem | yeah, and not helpful as to what failed | |
| 19:08:00 | mriedem | right before it: | |
| 19:08:00 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:08:20 | mriedem | neutron is installing stuff | |
| 19:09:36 | sean-k-mooney | well it happend 5 seconds after we start installing the swift test-requirements http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_44 | |
| 19:12:07 | sean-k-mooney | the sudo process for install the swift test requiremetns pid:15728 end at http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_48_54 | |
| 19:13:00 | sean-k-mooney | so the compute agent segfault while we are upgradeing the test requiremets for swift | |
| 19:14:16 | sean-k-mooney | that said nothing in https://github.com/openstack/swift/blob/master/test-requirements.txt jumps out at me | |
| 19:15:30 | mriedem | i don't know why test-requirement would be getting installed in a devstack env anyway | |
| 19:15:33 | mriedem | test-requirements should be for unit tests | |
| 19:16:00 | sean-k-mooney | mriedem: that is a dispute i have had with devstack for 2-3 releases | |
| 19:16:11 | sean-k-mooney | devstack installs all test-requiremets systemwide | |
| 19:16:55 | sdague | test-requirements also got overloaded by lots of projects as "optional requirements" and people put all their drivers in there | |
| 19:17:12 | sdague | it's never had clear definitions | |
| 19:17:30 | sdague | it was made to always install... 3 years ago? | |
| 19:17:45 | mriedem | ok so neutron reqs are installed here for queens | |
| 19:17:46 | mriedem | Jan 18 05:49:36 ubuntu-xenial-inap-mtl01-0001976291 sudo[24307]: stack : TTY=unknown ; PWD=/opt/stack/old/tempest ; USER=root ; ENV=http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite ; COMMAND=/usr/local/bin/pip2.7 install -c /opt/stack/new/requirements/upper-constraints.txt -r /opt/stack/new/neutron/test-requirements.txt -e /opt/stack/new/neutron | |
| 19:17:51 | sean-k-mooney | sdague: yes but if they are optional they should only be installed in the gates that need them :) | |
| 19:17:58 | mriedem | then nova-conductor segfaults Jan 18 05:49:48 ubuntu-xenial-inap-mtl01-0001976291 kernel: nova-conductor[20023]: segfault at d0 ip 0000000000500211 sp 00007ffdae038760 error 4 in python2.7[400000+2e9000] | |
| 19:18:03 | mriedem | and nova-conductor is pike at this point | |
| 19:19:00 | sean-k-mooney | sdague: yep but it only bit me with the flake8 import ording plugin that was added then blacklisted to nova test-requriement that ended up brakeing our ovs-dpdk gates because ovs enabled pip but disable that check internally | |
| 19:19:45 | sean-k-mooney | mriedem: well that appears to be the same memory adress in both python2.7[400000+2e9000] | |
| 19:22:48 | sean-k-mooney | mriedem: this is a time issue tool any of the previous pip installs could have cased it, as we cant know how long it will be betwen the module being replaced and the dangeling function pointer being deferenced | |
| 19:23:19 | sean-k-mooney | * timing issue too | |
| 19:23:29 | openstackgerrit | Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288 | |
| 19:25:47 | efried | edleafe jaypipes Just found a nice leetle bug | |
| 19:25:52 | sean-k-mooney | sdague: by the way that was not a did at you. i know why you made the change i.e the multiple code paths and the dep issue it was causeing. | |
| 19:26:25 | mriedem | hmm, so n-cpu starts failing around 5:46 and fails to connect to the ironic api | |
| 19:26:32 | efried | sdague too | |
| 19:26:33 | openstackgerrit | Mark Goddard proposed openstack/nova master: Send traits to ironic on server boot https://review.openstack.org/508116 | |
| 19:26:33 | openstackgerrit | Mark Goddard proposed openstack/nova master: Add get_traits() method to ComputeDriver https://review.openstack.org/532290 | |
| 19:26:34 | openstackgerrit | Mark Goddard proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288 | |
| 19:26:34 | efried | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L296-L302 | |
| 19:26:36 | mriedem | and i think that's around when glance is being upgraded/restarted | |
| 19:26:46 | mriedem | maybe some weird uwsgi stuff? | |
| 19:27:02 | efried | edleafe jaypipes sdague The bug is that e.g. [] is a valid json payload. | |
| 19:27:52 | sdague | efried: nice | |
| 19:28:17 | jaypipes | efried: sure, I suppose. though I prefer DELETE :) | |
| 19:28:17 | sean-k-mooney | mriedem: http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/apache/error.txt.gz | |
| 19:28:33 | sean-k-mooney | keysonten is dead from 5:47 | |
| 19:29:02 | efried | jaypipes No DELETE API for aggs :( | |
| 19:29:08 | owalsh | dansmith: hey, hitting an issue with cell_v2 setup https://bugs.launchpad.net/nova/+bug/1744780 | |
| 19:29:09 | openstack | Launchpad bug 1744780 in tripleo "Nova::Cell_v2::Simple_setup/Nova_cell_v2[cell0] fails on master promote" [Critical,Triaged] | |
| 19:29:33 | jaypipes | efried: ooh, yeah... | |
| 19:29:45 | mriedem | [Thu Jan 18 05:50:49.905807 2018] [wsgi:warn] [pid 30759:tid 139966291453824] mod_wsgi: Compiled for Python/2.7.11. | |
| 19:29:47 | efried | fixing | |
| 19:29:48 | jaypipes | efried: prolly should have one, but yeah | |
| 19:29:49 | mriedem | [Thu Jan 18 05:50:49.905865 2018] [wsgi:warn] [pid 30759:tid 139966291453824] mod_wsgi: Runtime using Python/2.7.12. | |
| 19:31:07 | mriedem | mtreinish: is anything using this code yet? https://review.openstack.org/#/c/531493/ | |
| 19:31:58 | mriedem | i guess not https://review.openstack.org/#/c/531498/ | |
| 19:33:59 | melwitt | owalsh: that's a recent change that we're not allowing creation/update of a cell with a DB connection + transport URL combination that already exists | |
| 19:34:22 | owalsh | melwitt: yea, but it's not removing the target cell from the cells it checks for dups | |
| 19:34:32 | owalsh | melwitt: so it fails on itself | |
| 19:34:53 | melwitt | hm | |
| 19:37:50 | efried | jaypipes edleafe sdague https://bugs.launchpad.net/nova/+bug/1744786 FYI | |
| 19:37:51 | openstack | Launchpad bug 1744786 in OpenStack Compute (nova) "SchedulerReportClient.put with empty (not None) payload errors 415" [Undecided,New] | |
| 19:40:44 | jroll | mriedem: looks like n-cpu is because we shut down ironic-api while upgrading things (which seems weird to me, but) | |
| 19:40:55 | jroll | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_44_56_948 | |
| 19:46:03 | melwitt | owalsh: I see what you mean. so the case you're hitting is trying to update_cell with only one of DB connection or transport URL being different then? | |
| 19:46:35 | dansmith | melwitt: yeah I expect so | |
| 19:47:18 | melwitt | yeah, just looking at the unit tests and the case of both being different should go through. guh | |
| 19:47:54 | dansmith | melwitt: er, what I meant was I think they're likely just running update_cell all the time, expecting it to be idempotent | |
| 19:48:10 | dansmith | so it might not be different, but the pre-check finds an existing (identical) record and so it fails | |
| 19:48:17 | melwitt | oh, I see | |
| 19:48:26 | dansmith | where it should only fail if they set cell1 to the same thing as cell2 | |
| 19:49:52 | mriedem | stephenfin: jaypipes: i don't see the reason for this change at all https://review.openstack.org/#/c/534849/ | |
| 19:50:11 | mriedem | those tests all default to send 2.1 so explicitly hard-coding something isn't necessary | |
| 19:51:14 | melwitt | dansmith: do you want to cook up a patch for that or should I? | |
| 19:51:32 | dansmith | melwitt: grilling one currently | |
| 19:51:38 | melwitt | sweet | |
| 19:53:58 | jaypipes | mriedem: that was originally in one of jackie's other patches that was unrelated and I had asked her to split it out. | |
| 19:54:07 | openstackgerrit | Eric Fried proposed openstack/nova master: Report Client: PUT empty (not None) JSON data https://review.openstack.org/536545 | |
| 19:54:09 | efried | sdague jaypipes dansmith ^ | |
| 19:54:47 | efried | Would rather not put it in the dep chain of the update_provider_tree series, but it's gonna foul my test cases for update_from_provider_tree (which is how I found it) | |
| 19:54:48 | mriedem | jaypipes: ok | |
| 19:54:52 | jaypipes | mriedem: without it, patches that test the new certificates REST API functionality were failing due to the personality stuff still being around for some reason | |
| 19:55:19 | mriedem | new tests should have just overrode the personality files stuff rather than need to hack up old tests | |
| 19:56:19 | mriedem | and...the mailman is stuck outside my house | |
| 19:56:37 | openstackgerrit | Sen Yang proposed openstack/python-novaclient master: Implement hypervisor hostname exact patten match for server cold migrate https://review.openstack.org/520187 | |
| 19:56:37 | mriedem | he knows i work from home, so i guess i have to suit up now | |
| 19:59:06 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix update_cell to ignore existing identical cells https://review.openstack.org/536546 | |
| 19:59:07 | dansmith | melwitt: owalsh: ^ | |
| 20:09:36 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 20:09:43 | edleafe | mriedem: ^^ responded to your comments | |
| 20:13:49 | dansmith | melwitt: I think you meant assertFalse() when you pointed out my mind-o | |
| 20:14:46 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix update_cell to ignore existing identical cells https://review.openstack.org/536546 | |
| 20:14:56 | dansmith | melwitt: I had another bug in the test that that uncovered, but the functional code worked thankfully | |
| 20:14:57 | melwitt | dansmith: yes, heh. I was thinking 0 == no error == True for some reason | |
| 20:15:12 | dansmith | aye | |
| 20:15:45 | mtreinish | mriedem: yeah, not yet. But I need get that landed in glance soon | |
| 20:22:07 | mtreinish | mriedem: were you looking to use that option for something? | |
| 20:45:34 | owalsh | melwitt, dansmith: LTGM, thanks! | |
| 20:52:40 | ildikov | gibi: johnthetubaguy: mriedem: thank you!!! :) | |
| 20:53:11 | bauzas | mriedem: dansmith: gibi: rebased https://review.openstack.org/#/c/533642/ for cdent's comments about relnotes | |
| 21:03:37 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM to feature-classification https://review.openstack.org/533237 | |
| 21:06:16 | mriedem | mtreinish: no, trying to figure out why the ironic multinode grenade job starting having random seg faults since 1/17 | |
| 21:06:49 | pooja | Hello! Is it possible to use the Placement API in Newton with a non-cells deployment? | |
| 21:07:00 | mriedem | pooja: yes | |