| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 12:15:07 | aspiers | http://lnav.org/ | |
| 12:15:12 | kashyap | aspiers: On that other bug from Dirk -- I'll repost the patch with CentOS fix, too. And think more | |
| 12:15:29 | aspiers | kashyap: thanks. Yeah just upload a new patchset? | |
| 12:15:39 | kashyap | aspiers: Yea, brewing | |
| 12:16:04 | kashyap | aspiers: I take it that you've read my full comment on that patch; thanks | |
| 12:16:09 | kashyap | aspiers: On lnav - I noticed it, but haven't used it much | |
| 12:16:22 | aspiers | You should, it's insanely good | |
| 12:16:31 | aspiers | worth it for the histogram feature alone | |
| 12:16:34 | sean-k-mooney | yep | |
| 12:16:40 | sean-k-mooney | i found it like a year ago | |
| 12:16:43 | sean-k-mooney | its really nice | |
| 12:16:56 | aspiers | sean-k-mooney: did you know about "lnav -i extra" ? | |
| 12:17:04 | aspiers | my 3rd party repo gets included | |
| 12:17:05 | sean-k-mooney | i havent use that really but i like its ablity to filter things | |
| 12:17:24 | sean-k-mooney | no but -q should be on by default | |
| 12:17:35 | aspiers | sean-k-mooney: haha, did you see that issue? | |
| 12:17:37 | kashyap | aspiers: Installed; and began using. I heard it from sean-k-mooney, actually | |
| 12:17:54 | sean-k-mooney | is there an issue open for it | |
| 12:18:06 | kashyap | aspiers: Meanwhile, I take it that you haven't spotted anything _else_ that is spooky in https://review.opendev.org/#/c/673790/ | |
| 12:18:18 | aspiers | sean-k-mooney: https://github.com/tstack/lnav/issues/436 but it got closed because the author *still* isn't getting the point | |
| 12:18:21 | aspiers | about to comment | |
| 12:18:37 | sean-k-mooney | i have basically locked up my terminal after leaving the devstack logs open for a few days in it | |
| 12:20:07 | sean-k-mooney | anyway be right be time to grab coffee and post something by snail mail... | |
| 12:20:13 | sean-k-mooney | at least its not fax... | |
| 12:21:13 | sean-k-mooney | my insurance company will acatully accept email copies of scans provided you eventually send them the forms which is nice | |
| 12:22:20 | aspiers | Jul 29 17:06:56 devstack systemd[1]: devstack@n-sch.service: Main process exited, code=dumped, status=11/SEGV | |
| 12:22:23 | aspiers | ummm... WTF?! | |
| 12:23:03 | sean-k-mooney | that sound like the issue matt was hitting yesterday | |
| 12:23:16 | sean-k-mooney | are you using an old kernel or python 2? | |
| 12:23:21 | aspiers | python 2 | |
| 12:23:36 | aspiers | 4.12.14-95.24-default | |
| 12:23:36 | sean-k-mooney | on kernel? 4.15 ish? | |
| 12:23:40 | sean-k-mooney | ok older | |
| 12:23:59 | aspiers | is there a workaround/solution? | |
| 12:24:00 | sean-k-mooney | ya there may be an issue with one of our deps with python 2 and older kernels | |
| 12:24:26 | sean-k-mooney | we havnt track it down yet but if you can provide logs/version number it might help | |
| 12:24:53 | sean-k-mooney | you can try python3 | |
| 12:25:02 | aspiers | there's nothing useful in the log | |
| 12:25:17 | sean-k-mooney | just he seg fault | |
| 12:25:22 | aspiers | oh god | |
| 12:25:24 | aspiers | [35889.416395] cinder-volume[178403]: segfault at 7f2154837db7 ip 00007f20f299c9ca sp 00007ffea13d86d0 error 6 in libpython2.7.so.1.0[7f20f28b0000+1ac000] | |
| 12:25:26 | aspiers | not just nova | |
| 12:25:30 | sean-k-mooney | ya | |
| 12:25:42 | sean-k-mooney | are you using nested virt | |
| 12:25:48 | aspiers | no | |
| 12:26:05 | sean-k-mooney | ok so that rules out one possibelissue | |
| 12:26:48 | sean-k-mooney | matt was seeing similar isssues with python 2.7 on ubuntu 18.04 yesterday and last week | |
| 12:27:11 | aspiers | I guess I can try switching to python 3 | |
| 12:27:13 | aspiers | yikes | |
| 12:27:14 | sean-k-mooney | if leap 15 has python3 i would try that | |
| 12:27:19 | aspiers | this is not leap | |
| 12:27:24 | aspiers | sles12 sp4 | |
| 12:27:43 | aspiers | but yes leap and sles have both had python3 for years | |
| 12:28:01 | sean-k-mooney | make sure you use either 3.5 or 3.6 | |
| 12:28:20 | sean-k-mooney | we never really supported 3.4 and 3.7 may have issues | |
| 12:28:21 | aspiers | oh crap | |
| 12:28:37 | aspiers | default is 3.4.6 | |
| 12:28:42 | sean-k-mooney | 3.4 might work | |
| 12:28:47 | sean-k-mooney | we just never offially supported it | |
| 12:28:58 | sean-k-mooney | we went strait to 3.5 | |
| 12:28:58 | aspiers | ok | |
| 12:32:14 | stephenfin | gmann: Could you take a look at https://review.opendev.org/#/c/661391/ (DevStack stuff) when you've a chance? | |
| 12:34:23 | aspiers | sean-k-mooney: https://github.com/tstack/lnav/issues/436#issuecomment-516829106 | |
| 12:36:18 | gmann | stephenfin: sure. do you have nova patch to test that ? only thing i want to verify is IPv6 job i added recently on nova gate works fine. I removed the setting of novnc addresses from base devstack job and let /lib/nova handle that. | |
| 12:36:32 | gmann | stephenfin: of let me add the same job on devstack on top of that patch which will test both. | |
| 12:36:37 | gmann | s/of/or | |
| 12:46:02 | aspiers | sean-k-mooney: I have a coredump now | |
| 12:46:28 | aspiers | sean-k-mooney: also https://github.com/aspiers/lnav-formats/blob/master/oslo_log.json | |
| 12:46:41 | aspiers | although I'm not sure that was necessary, it might be in other repos | |
| 12:51:38 | stephenfin | gmann: Yeah, I think this one should do it https://review.opendev.org/#/c/667388/ | |
| 12:54:03 | gmann | stephenfin: thanks. rechecked that. | |
| 12:56:10 | sean-k-mooney | aspiers: with python 3.4? | |
| 12:56:18 | aspiers | no | |
| 12:56:29 | aspiers | http://paste.openstack.org/show/755163/ | |
| 12:57:14 | sean-k-mooney | aspiers: also did you add oslo_log support to lnav as a plugin if so i might have to try that | |
| 12:57:31 | aspiers | sean-k-mooney: that's what the link above is | |
| 12:57:46 | aspiers | sean-k-mooney: it gets installed via "lnav -i extra" | |
| 12:57:56 | aspiers | just type that | |
| 12:58:02 | sean-k-mooney | ya i clicked and got confused with your core dump comment but imm looking at teh paste now | |
| 13:03:00 | sean-k-mooney | hum well i cant really tell what cased that seg fault other thin it appers to be a result of a getattr call on a randomp python object. so ill blame eventlest by default if you can repoduce with the pytho debuger instad of gdb that might help | |
| 13:05:00 | sean-k-mooney | aspiers: can you try running the conductor with pdb | |
| 13:05:03 | aspiers | sean-k-mooney: I'm trying to install the python gdb extensions so I can get the Python stacktrace | |
| 13:05:23 | aspiers | sean-k-mooney: I guess that might work too | |
| 13:07:08 | sean-k-mooney | i think its dieing pretty early in the nova-condoctor console script that is generated via pbr/setup tools | |
| 13:07:41 | aspiers | sean-k-mooney: what's the best way to start it? | |
| 13:08:04 | aspiers | oh wow | |
| 13:08:06 | aspiers | ● devstack@n-sch.service loaded failed failed Devstack devstack@n-sch.service | |
| 13:08:08 | aspiers | ● devstack@n-super-cond.service loaded failed failed Devstack devstack@n-super-cond.service | |
| 13:08:10 | aspiers | ● devstack@s-proxy.service loaded failed failed Devstack devstack@s-proxy.service | |
| 13:08:23 | aspiers | I've had 6 crashes in the last hour | |
| 13:08:27 | sean-k-mooney | i think just "pdb /usr/bin/python /usr/bin/nova-conductor --config-file /etc/nova/nova.conf" maybe with sudo | |
| 13:11:01 | sean-k-mooney | ya one of our deps really dislikes python2.7 apparently | |
| 13:12:22 | sean-k-mooney | was this a clean sles 14 deployment | |
| 13:12:41 | sean-k-mooney | im wondering if i can quickly spin up an container or something and try and repoduce | |
| 13:17:59 | aspiers | gah | |
| 13:18:04 | aspiers | I just lost connection to the node | |
| 13:18:21 | sean-k-mooney | did you check dmesg before | |
| 13:18:33 | sean-k-mooney | there may have been other core dumps or crashes | |
| 13:19:30 | aspiers | coredumpctl shows them all | |
| 13:19:36 | aspiers | all 6 | |