| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-03-01 | |||
| 15:26:46 | gibi | ratailor_: could you drop the changes from https://review.opendev.org/c/openstack/nova/+/904568/16/nova/tests/unit/policies/test_servers.py | |
| 15:26:50 | gibi | they are not needed | |
| 15:27:00 | gibi | then I will be +2 | |
| 15:27:08 | sean-k-mooney | apprently ^ | |
| 15:27:29 | ratailor_ | gibi, sure, i will remove remaining changes there. | |
| 15:28:57 | gibi | cool | |
| 15:28:58 | opendevreview | Amit Uniyal proposed openstack/nova master: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905134 | |
| 15:35:37 | ratailor_ | gibi, how to resolve this https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d83/904568/16/check/openstack-tox-pep8/d83f39e/job-output.txt | |
| 15:35:45 | ratailor_ | nova/api/openstack/compute/views/servers.py:223:5: C901 'ViewBuilder.show' is too complex (41) | |
| 15:36:15 | ratailor_ | gibi, I think, its failing because we have too many ifs | |
| 15:39:17 | ratailor_ | we want to get req_spec during server show call when provided_az is None, but during server list, it will be passed from list_view method, so it might be some-string or '' | |
| 15:42:07 | opendevreview | Fabian Wiesel proposed openstack/nova master: Vmware: Remove uuid parameter from get_vmdk_info call https://review.opendev.org/c/openstack/nova/+/910627 | |
| 15:43:01 | dansmith | sean-k-mooney: was it you that had or was using some log aggregator thing that would collapse logs from multiple services into one? | |
| 15:46:00 | sean-k-mooney | gibi: has one tool for that bu ti also use lnav locally to do something like that | |
| 15:46:56 | dansmith | just thinking that for debugging gate fails, it might be a lot easier if we had a unified journal-style log with all the services from that host so we could open and search for a traceback, error, or instance uuid | |
| 15:47:36 | bauzas | dansmith: I generally use journalctl -f -u devstack@n-* for my controller | |
| 15:47:45 | sean-k-mooney | i think we have a full copy of the journal oputput | |
| 15:47:47 | dansmith | bauzas: right, I mean for gate fails | |
| 15:48:05 | bauzas | ah, aggregating the logfiles | |
| 15:48:56 | dansmith | sean-k-mooney: right but it's not viewable in the browser is it? | |
| 15:49:00 | sean-k-mooney | im on a call but ill see if i can get a link | |
| 15:49:52 | gibi | ratailor_: just pull out the logic to a separate function to reduce complexity | |
| 15:50:07 | ratailor_ | gibi, ack. Thanks! | |
| 15:50:41 | ratailor_ | gibi, inner function ? | |
| 15:51:12 | bauzas | sean-k-mooney: dansmith: yeah we do have the journal https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f81/904568/14/check/tempest-integrated-compute/f81f16a/controller/logs/devstack.journal.README.txt | |
| 15:51:35 | dansmith | yeah, I know, but it's not browseable | |
| 15:51:35 | bauzas | but you need to download the file | |
| 15:51:41 | bauzas | yup | |
| 15:51:55 | dansmith | but yeah, I guess we don't need log merge if we generated the combined log from that | |
| 15:52:15 | dansmith | okay so maybe I should just script doing that from a log url and look at it not in the browser | |
| 15:52:27 | gibi | ratailor_: a private func inte view class is OK too | |
| 15:52:43 | bauzas | honestly, for debugging gate failures, I'm sometimes downloading already | |
| 15:52:47 | ratailor_ | gibi, ack. | |
| 15:53:18 | bauzas | dansmith: we already have download-logs.sh we could add some tooling in that script to do some kind of automatic merge | |
| 16:04:20 | sean-k-mooney | dansmith: ok done. so ya we have the devstack.journal.gz | |
| 16:04:31 | sean-k-mooney | which you cant browse in the web as you said | |
| 16:09:22 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | |
| 16:11:19 | bauzas | ratailor_: I need to stop now | |
| 16:11:28 | bauzas | gibi: too, I need to stop | |
| 16:11:44 | bauzas | but I can take a look on Monday morning at https://review.opendev.org/c/openstack/nova/+/904568/ | |
| 16:12:13 | ratailor_ | bauzas, isn't today last day for featurefreeze ? | |
| 16:12:31 | sean-k-mooney | it was technially yesterday | |
| 16:12:40 | bauzas | it was yesterday | |
| 16:12:57 | bauzas | but I can give you an exception due to my previous +2 | |
| 16:13:02 | bauzas | gibi found a good point | |
| 16:13:07 | ratailor_ | bauzas, ack. Thanks! | |
| 16:13:32 | ratailor_ | sean-k-mooney, gibi Thanks! | |
| 16:13:32 | bauzas | but if we can't merge it on Monday, then it will be punted to Dalmatian | |
| 16:14:06 | ratailor_ | bauzas, ack. sure. | |
| 16:14:17 | bauzas | sean-k-mooney: dansmith: about the log merging, | |
| 16:14:40 | bauzas | sean-k-mooney: dansmith: could we modify our ansible scripts to create some host logs ? | |
| 16:14:46 | bauzas | 2024-02-28 13:50:19.602722 | TASK [fetch-devstack-log-dir : Collect devstack logs] | |
| 16:15:17 | dansmith | bauzas: I'm cooking something right now, we'll see if it's helpful | |
| 16:15:51 | bauzas | dansmith: I need to go off by now but I could see it on Monday | |
| 16:16:21 | dansmith | o/ | |
| 16:18:45 | dansmith | man, we have got to get brick to stop logging tracebacks any time multipathd is missing | |
| 16:19:33 | sean-k-mooney | but how else would you know... | |
| 16:19:41 | sean-k-mooney | it is a littel excessive | |
| 16:20:12 | bauzas | dansmith: https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L9-L18 | |
| 16:20:27 | bauzas | dansmith: I think we could change this to create some host log | |
| 16:20:58 | sean-k-mooney | well we do just in binary form | |
| 16:21:09 | dansmith | bauzas: we already have them combined in the journal, and I bet duplicating them in the log dump would not be popular, so just having something I can dump a log url into in the terminal is good enough I think | |
| 16:21:09 | sean-k-mooney | https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L37-L49 | |
| 16:21:19 | dansmith | less good than in the browser, but the browser struggles anyway | |
| 16:21:29 | bauzas | sean-k-mooney: yeah but as dansmith said, you need to download the whole journal | |
| 16:21:45 | sean-k-mooney | right which woudl be the same... | |
| 16:21:46 | dansmith | https://termbin.com/mqhd | |
| 16:21:53 | bauzas | anyway, /me goes off by now | |
| 16:22:07 | dansmith | if you run that with the url copied from the zuul manifest it drops you right into journalctl with the whole thing merged | |
| 16:22:19 | dansmith | and you can use '-u devstack@n*' to get only nova stuff | |
| 16:22:19 | gibi | bauzas: ratailor_ cool, Im looking at it now | |
| 16:22:27 | sean-k-mooney | cool | |
| 16:22:33 | sean-k-mooney | i was playing iwht something like | |
| 16:22:36 | sean-k-mooney | curl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f81/904568/14/check/tempest-integrated-compute/f81f16a/controller/logs/devstack.journal.gz | zcat | lnav -q | |
| 16:22:48 | dansmith | can lnav read journal files? | |
| 16:23:15 | sean-k-mooney | kind of but it does not render them well | |
| 16:23:26 | dansmith | well, journalctl can :) | |
| 16:23:31 | sean-k-mooney | so you would proably want to pass them thorugh systemd-joural-remote | |
| 16:23:41 | dansmith | not as nice highlighting and stuff, but.. | |
| 16:24:07 | sean-k-mooney | well i normally pipe journalctl to lnav | |
| 16:24:30 | sean-k-mooney | so ya that script would be useful wehn i need ot do that | |
| 16:25:06 | dansmith | oh yeah just running that | lnav is very nice | |
| 16:25:15 | dansmith | that script I mean | |
| 16:25:20 | sean-k-mooney | ya | |
| 16:25:44 | sean-k-mooney | so we recently moved our new installer to put all the logs into the jorunal | |
| 16:25:46 | dansmith | the comment in there isn't right, let me fix it | |
| 16:26:11 | sean-k-mooney | and in those ci jobs we just are dumping /var/log/messages in to the job logs | |
| 16:26:20 | sean-k-mooney | and i have been using lnave to parst them | |
| 16:26:28 | dansmith | there: https://termbin.com/qpbv | |
| 16:26:52 | dansmith | so I can just right-click copy a url out of a zuul manifest and run that on it, optionally | lnav and it just dumps me into a merged log | |
| 16:26:54 | dansmith | that's nice | |
| 16:27:11 | gibi | bauzas: I'm +2 on the AZ series | |
| 16:29:48 | sean-k-mooney | "/usr/lib/systemd/systemd-journal-remote: No such file or directory" installing that might help | |
| 16:30:18 | dansmith | yeah | |
| 16:30:36 | dansmith | it's also in a different place on ubuntu, so I should make it use either | |
| 16:30:59 | sean-k-mooney | you could just leave it up to the path | |
| 16:31:11 | dansmith | it's not in path on either I think | |
| 16:31:27 | sean-k-mooney | oh ok odd | |
| 16:31:36 | sean-k-mooney | im installing it now but ill let you know | |
| 16:32:33 | sean-k-mooney | /usr/lib/systemd/systemd-journal-remote works on debian testing | |