| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-03-01 | |||
| 15:48:56 | dansmith | sean-k-mooney: right but it's not viewable in the browser is it? | |
| 15:49:00 | sean-k-mooney | im on a call but ill see if i can get a link | |
| 15:49:52 | gibi | ratailor_: just pull out the logic to a separate function to reduce complexity | |
| 15:50:07 | ratailor_ | gibi, ack. Thanks! | |
| 15:50:41 | ratailor_ | gibi, inner function ? | |
| 15:51:12 | bauzas | sean-k-mooney: dansmith: yeah we do have the journal https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f81/904568/14/check/tempest-integrated-compute/f81f16a/controller/logs/devstack.journal.README.txt | |
| 15:51:35 | dansmith | yeah, I know, but it's not browseable | |
| 15:51:35 | bauzas | but you need to download the file | |
| 15:51:41 | bauzas | yup | |
| 15:51:55 | dansmith | but yeah, I guess we don't need log merge if we generated the combined log from that | |
| 15:52:15 | dansmith | okay so maybe I should just script doing that from a log url and look at it not in the browser | |
| 15:52:27 | gibi | ratailor_: a private func inte view class is OK too | |
| 15:52:43 | bauzas | honestly, for debugging gate failures, I'm sometimes downloading already | |
| 15:52:47 | ratailor_ | gibi, ack. | |
| 15:53:18 | bauzas | dansmith: we already have download-logs.sh we could add some tooling in that script to do some kind of automatic merge | |
| 16:04:20 | sean-k-mooney | dansmith: ok done. so ya we have the devstack.journal.gz | |
| 16:04:31 | sean-k-mooney | which you cant browse in the web as you said | |
| 16:09:22 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | |
| 16:11:19 | bauzas | ratailor_: I need to stop now | |
| 16:11:28 | bauzas | gibi: too, I need to stop | |
| 16:11:44 | bauzas | but I can take a look on Monday morning at https://review.opendev.org/c/openstack/nova/+/904568/ | |
| 16:12:13 | ratailor_ | bauzas, isn't today last day for featurefreeze ? | |
| 16:12:31 | sean-k-mooney | it was technially yesterday | |
| 16:12:40 | bauzas | it was yesterday | |
| 16:12:57 | bauzas | but I can give you an exception due to my previous +2 | |
| 16:13:02 | bauzas | gibi found a good point | |
| 16:13:07 | ratailor_ | bauzas, ack. Thanks! | |
| 16:13:32 | ratailor_ | sean-k-mooney, gibi Thanks! | |
| 16:13:32 | bauzas | but if we can't merge it on Monday, then it will be punted to Dalmatian | |
| 16:14:06 | ratailor_ | bauzas, ack. sure. | |
| 16:14:17 | bauzas | sean-k-mooney: dansmith: about the log merging, | |
| 16:14:40 | bauzas | sean-k-mooney: dansmith: could we modify our ansible scripts to create some host logs ? | |
| 16:14:46 | bauzas | 2024-02-28 13:50:19.602722 | TASK [fetch-devstack-log-dir : Collect devstack logs] | |
| 16:15:17 | dansmith | bauzas: I'm cooking something right now, we'll see if it's helpful | |
| 16:15:51 | bauzas | dansmith: I need to go off by now but I could see it on Monday | |
| 16:16:21 | dansmith | o/ | |
| 16:18:45 | dansmith | man, we have got to get brick to stop logging tracebacks any time multipathd is missing | |
| 16:19:33 | sean-k-mooney | but how else would you know... | |
| 16:19:41 | sean-k-mooney | it is a littel excessive | |
| 16:20:12 | bauzas | dansmith: https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L9-L18 | |
| 16:20:27 | bauzas | dansmith: I think we could change this to create some host log | |
| 16:20:58 | sean-k-mooney | well we do just in binary form | |
| 16:21:09 | dansmith | bauzas: we already have them combined in the journal, and I bet duplicating them in the log dump would not be popular, so just having something I can dump a log url into in the terminal is good enough I think | |
| 16:21:09 | sean-k-mooney | https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L37-L49 | |
| 16:21:19 | dansmith | less good than in the browser, but the browser struggles anyway | |
| 16:21:29 | bauzas | sean-k-mooney: yeah but as dansmith said, you need to download the whole journal | |
| 16:21:45 | sean-k-mooney | right which woudl be the same... | |
| 16:21:46 | dansmith | https://termbin.com/mqhd | |
| 16:21:53 | bauzas | anyway, /me goes off by now | |
| 16:22:07 | dansmith | if you run that with the url copied from the zuul manifest it drops you right into journalctl with the whole thing merged | |
| 16:22:19 | dansmith | and you can use '-u devstack@n*' to get only nova stuff | |
| 16:22:19 | gibi | bauzas: ratailor_ cool, Im looking at it now | |
| 16:22:27 | sean-k-mooney | cool | |
| 16:22:33 | sean-k-mooney | i was playing iwht something like | |
| 16:22:36 | sean-k-mooney | curl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f81/904568/14/check/tempest-integrated-compute/f81f16a/controller/logs/devstack.journal.gz | zcat | lnav -q | |
| 16:22:48 | dansmith | can lnav read journal files? | |
| 16:23:15 | sean-k-mooney | kind of but it does not render them well | |
| 16:23:26 | dansmith | well, journalctl can :) | |
| 16:23:31 | sean-k-mooney | so you would proably want to pass them thorugh systemd-joural-remote | |
| 16:23:41 | dansmith | not as nice highlighting and stuff, but.. | |
| 16:24:07 | sean-k-mooney | well i normally pipe journalctl to lnav | |
| 16:24:30 | sean-k-mooney | so ya that script would be useful wehn i need ot do that | |
| 16:25:06 | dansmith | oh yeah just running that | lnav is very nice | |
| 16:25:15 | dansmith | that script I mean | |
| 16:25:20 | sean-k-mooney | ya | |
| 16:25:44 | sean-k-mooney | so we recently moved our new installer to put all the logs into the jorunal | |
| 16:25:46 | dansmith | the comment in there isn't right, let me fix it | |
| 16:26:11 | sean-k-mooney | and in those ci jobs we just are dumping /var/log/messages in to the job logs | |
| 16:26:20 | sean-k-mooney | and i have been using lnave to parst them | |
| 16:26:28 | dansmith | there: https://termbin.com/qpbv | |
| 16:26:52 | dansmith | so I can just right-click copy a url out of a zuul manifest and run that on it, optionally | lnav and it just dumps me into a merged log | |
| 16:26:54 | dansmith | that's nice | |
| 16:27:11 | gibi | bauzas: I'm +2 on the AZ series | |
| 16:29:48 | sean-k-mooney | "/usr/lib/systemd/systemd-journal-remote: No such file or directory" installing that might help | |
| 16:30:18 | dansmith | yeah | |
| 16:30:36 | dansmith | it's also in a different place on ubuntu, so I should make it use either | |
| 16:30:59 | sean-k-mooney | you could just leave it up to the path | |
| 16:31:11 | dansmith | it's not in path on either I think | |
| 16:31:27 | sean-k-mooney | oh ok odd | |
| 16:31:36 | sean-k-mooney | im installing it now but ill let you know | |
| 16:32:33 | sean-k-mooney | /usr/lib/systemd/systemd-journal-remote works on debian testing | |
| 16:32:44 | dansmith | on ubuntu it's /lib/systemd/systemd-journal-remote | |
| 16:32:58 | dansmith | oh is there a link? | |
| 16:33:37 | sean-k-mooney | rwxrwxrwx 1 root root 7 May 30 2023 /lib -> usr/lib | |
| 16:33:58 | sean-k-mooney | so ya /lib links to /usr/lib | |
| 16:33:58 | dansmith | oh heh | |
| 16:34:04 | dansmith | okay perfect then | |
| 16:34:22 | dansmith | I guess I should gist this | |
| 16:34:50 | sean-k-mooney | or we can have the ci also provide the script | |
| 16:35:07 | sean-k-mooney | i like that you dont redownload the file if its there | |
| 16:35:17 | sean-k-mooney | ../log-download.sh https://zuul.opendev.org/t/openstack/build/83d6dc665c2643f88d0025c385987c92/logs | lnav -q | |
| 16:35:21 | sean-k-mooney | that works for me nicely | |
| 16:35:54 | dansmith | https://gist.github.com/kk7ds/9a77939bd314cd4b5a35139adcdcedce | |
| 16:36:17 | sean-k-mooney | osgatelog is a better name | |
| 16:36:44 | dansmith | I also have an osfinger, that attaches to a live running job's console at the terminal | |
| 16:38:02 | sean-k-mooney | oh cool. i never got the finger gateway stuff to work localy | |
| 16:38:14 | sean-k-mooney | i didn know we had that configured for the public zuul | |
| 16:38:55 | dansmith | https://termbin.com/4mx2 | |
| 16:39:22 | dansmith | wait that's not it | |
| 16:39:25 | sean-k-mooney | oh its litrally just conenct with nc or telnet | |