| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-10 | |||
| 17:13:12 | efried | Meanwhile, any idea what glorious magic makes a .txt.gz journalctl log show up in the browser with color codes translated? | |
| 17:21:33 | sdague | efried: magic yet to be written | |
| 17:22:07 | efried | sdague But it works | |
| 17:22:15 | efried | At least in my browser | |
| 17:22:19 | sdague | efried: interesting | |
| 17:22:27 | efried | Except that for powervm logs, it doesn't *quite* work. | |
| 17:22:33 | efried | sdague What, it doesn't do that for you? | |
| 17:23:13 | sdague | efried: oh, it's not color codes translated | |
| 17:23:21 | sdague | you mean the coloring of - http://logs.openstack.org/81/488381/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/2b8b331/logs/screen-c-api.txt.gz ? | |
| 17:23:40 | efried | sdague Yup. | |
| 17:23:43 | sdague | https://github.com/openstack-infra/os-loganalyze | |
| 17:24:07 | sdague | is a wsgi filter for the logs | |
| 17:24:22 | efried | ...that runs under the auspices of apached? | |
| 17:24:39 | sdague | yeh, log access is proxied through that | |
| 17:24:41 | efried | ...and would need to exist on the log server. | |
| 17:24:48 | sdague | yes | |
| 17:24:56 | sdague | it processes in real time | |
| 17:24:56 | efried | So ours must be outdated or something. Cool. esberglu ^^ ! | |
| 17:25:12 | sdague | there is a regex on the files it will attempt to do it with | |
| 17:25:17 | efried | sdague Cause for us, it shows up HTMLified, with the log filters, but no color translation. | |
| 17:25:28 | efried | http://184.172.12.213/manual/rest_logs/logs/n-cpu.txt.gz | |
| 17:25:40 | efried | (This isn't in production yet, cause we've been naming 'em .log.gz, but we're converting over) | |
| 17:25:42 | sdague | https://github.com/openstack-infra/os-loganalyze/blob/master/os_loganalyze/filter.py#L24-L31 | |
| 17:25:56 | sdague | yeh, you'd need to setup this code on apache | |
| 17:27:13 | sdague | https://github.com/openstack-infra/puppet-openstackci/blob/master/templates/logs.vhost.erb#L106 and friends to see how you'd set it up on apache | |
| 17:27:27 | sdague | it requires a few rewrite rules | |
| 17:27:49 | efried | sdague Is that not the same script that does the filtering capability? | |
| 17:30:34 | efried | sdague Okay that stack failed, on image create. | |
| 17:31:29 | efried | apache error log shows an ENOSPC in /tmp. I've got .8GB in there; is it really trying to copy the image through /tmp?? | |
| 17:40:53 | sdague | efried: it might be | |
| 17:41:03 | efried | sdague Restacking, gonna watch /tmp... | |
| 17:41:04 | sdague | mtreinish: ^^^ | |
| 17:41:14 | sdague | it might be specific on the uwsgi glance | |
| 17:41:25 | sdague | and, if so, that could be an interesting issue | |
| 17:42:30 | sdague | you all are booting a real image, not a cirros, right? | |
| 17:42:44 | sdague | I could totally imagine that issue to have slipped through because cirros | |
| 17:43:33 | cdent | excellent point, sdague, at least one interation of the uwsgiification of glance needed to do some writes to disk, but I don’t know if that version landed, and if it did where on disk | |
| 17:44:36 | efried | sdague Yeah, it's a real image - though it's pretty small (2GB) | |
| 17:45:04 | sdague | efried: yeh... but there was all manner of tricksyness to get uwsgi glance to work | |
| 17:45:23 | sdague | and I can totally imagine full write of the upload to tmp was part of that | |
| 17:45:29 | efried | How... precious. | |
| 17:49:22 | mriedem | melwitt: hmm, nova-network only starts in a cells v1 config | |
| 17:49:31 | mriedem | so yeah, i don't think we probably have to worry about that for cells v2 multi-cell | |
| 17:56:26 | efried | sdague Yup, thar she blows. During that image create command, /tmp fills quickly, then stack bails. | |
| 17:57:18 | mriedem | dude | |
| 17:57:20 | mriedem | thar she blars | |
| 17:58:01 | mriedem | bauzas: jaypipes: we want this in for pike too https://review.openstack.org/#/c/492234/ | |
| 17:58:20 | mriedem | melwitt: ^ | |
| 17:58:51 | jaypipes | mriedem: done | |
| 17:59:43 | mriedem | thanks | |
| 18:00:03 | SamYaple | any chance I will see this backport merge https://review.openstack.org/#/c/486272/ ? im not sure what novas backport policies are | |
| 18:00:05 | mriedem | melwitt: can you also help go through the set of docs changes leading up to https://review.openstack.org/#/c/491853/ ? that's the target for the docs migration for rc1 | |
| 18:01:27 | mriedem | SamYaple: maybe | |
| 18:01:38 | mriedem | there should be a push for stable branch reviews once we are clear of pike rc1 | |
| 18:03:22 | SamYaple | mriedem: ok thanks. i can't run ocata in my setup without this patch and im not a huge fan of carrying patches which im currently doing with it | |
| 18:03:38 | sdague | SamYaple: +2 on that. | |
| 18:04:15 | SamYaple | nice sdague! thanks | |
| 18:04:36 | mtreinish | sdague, cdent: we ended up doing mod_proxy with http traffic on a local port | |
| 18:04:57 | mtreinish | glance runs uwsgi in http mode and apache just forwards the traffic to the local uwsgi daemon | |
| 18:04:57 | sdague | mtreinish: yeh, file upload seems to create a full temp file though | |
| 18:05:10 | sdague | which, is actually kind of an issue | |
| 18:05:28 | cdent | yeah, that will be because of a tweak to mod proxy’s behavior | |
| 18:05:29 | sdague | from nova bugs, 120GB snapshots are a thing | |
| 18:05:51 | sdague | no one is planning N * 120GB free in /tmp | |
| 18:08:04 | cdent | I guess that means the question becomes: is anyone planning to duplicate devstack’s way of running glance? | |
| 18:08:09 | mtreinish | sdague: I'm not sure how the uwsgi change would effect things there. We ran basically the same config before just instead of uwsgi running glance it was glance standalone. The apache config was more or less the same for tls | |
| 18:08:21 | mtreinish | unless uwsgi is using /tmp as a cache | |
| 18:08:26 | cdent | it’s mod proxy | |
| 18:08:43 | sdague | cdent: that's an impact of the tls termination? | |
| 18:09:33 | efried | mtreinish Is there a uwsgi config somewhere that I can tell to use something other than /tmp ? | |
| 18:10:04 | efried | I'm trying again with TMPDIR=/var/tmp (where I have some more space) - but that'll only affect stuff running directly under the auspices of stack.sh | |
| 18:11:11 | efried | mm, unless I set it in /etc/environment. Let's see if that flies... | |
| 18:11:50 | cdent | sdague, mtreinish, efried : https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies | |
| 18:11:57 | cdent | it’s the sendcl thing | |
| 18:12:16 | cdent | that was added to the proxy config when glance switched to uwsg | |
| 18:12:45 | mtreinish | cdent: the mod_proxy config for glance should not be using sendcl | |
| 18:12:58 | cdent | did you take it back out? | |
| 18:13:08 | mtreinish | that was one of the paths I was investigating and it didn't work for various reasons (including this issue growing tmp space) | |
| 18:13:52 | melwitt | mriedem: sure thing | |
| 18:14:10 | mtreinish | cdent: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L333 is the path for glance | |
| 18:14:11 | cdent | it’s currently on line 276 of lib/apache | |
| 18:14:32 | mtreinish | cdent: right that's the path that's not glance | |
| 18:14:42 | cdent | is setenv iglobal to apache ? | |
| 18:15:31 | cdent | It looks like it may be | |
| 18:16:33 | sdague | so... if you hit uwsgi directly without the tls proxy, this wouldn't be an issue? | |
| 18:17:12 | cdent | sdague: unclear, because it may be the mod_proxy_uwsgi also attends to it? | |
| 18:18:12 | cdent | I’m not certain this is the problem, but I do think sendcl is a factor, or something like it | |
| 18:18:22 | cdent | hard to be clear what uwsgi does in the face of file uploads | |
| 18:18:32 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 18:18:43 | sdague | yeh, it's supposed to be chunk encoded upload | |
| 18:18:58 | sdague | but bouncing through proxies... things get tricky | |
| 18:19:43 | mtreinish | is there a running system with this issue? because it's easy to confirm if apache is passing chunked data to uwsgi with a tcpump on the local port | |
| 18:19:47 | cdent | if you are sending chunked but demanding content-lenght (from whatever directive) a write will be required | |
| 18:20:20 | mriedem | sdague: i've got a couple of comments in here https://review.openstack.org/#/c/491815 | |
| 18:20:21 | mtreinish | cdent: if setenv is globally scoped across different site files we probaly just need to create a virtualhost for the glance path to isolate it from the other sites | |
| 18:20:59 | mriedem | melwitt: take a look at my comments if you update | |
| 18:21:10 | mtreinish | cdent: fwiw, the glance path doesn't use mod_proxy_uwsgi | |
| 18:21:35 | cdent | I know, but it does use mod_proxy doesn’t it? At least that’s what I thought I heard you say? | |
| 18:21:43 | mtreinish | yes it does | |
| 18:21:46 | melwitt | mriedem: phew, in the nick of time | |