| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-10 | |||
| 16:59:41 | sdague | which went all 500 stack tracy | |
| 17:00:03 | sdague | yes, placement is returning 500s, however I can't find them landing in a log anywhere | |
| 17:01:08 | sdague | http://paste.openstack.org/show/618074/ horizon_access.log | |
| 17:01:38 | efried | yeah, just what I was seeing. | |
| 17:01:54 | efried | Mine were coming from the compute service trying to suss out the resource providers | |
| 17:02:03 | efried | sdague Well, if you're game to help me debug why uwsgi stack is failing, I can try restacking thusly. | |
| 17:02:11 | sdague | efried: yeh, I would do that | |
| 17:02:15 | efried | That would be a big help in general. | |
| 17:02:16 | sdague | make sure you do a ./clean.sh | |
| 17:02:21 | efried | Okay, rippinit. | |
| 17:02:35 | sdague | yeh, I don't think anything else useful can come from this install | |
| 17:03:01 | efried | I did a lot of useful stuff wrt service catalog lookups. | |
| 17:03:24 | efried | But I was breaking right at driver init to muck around in pdb | |
| 17:04:09 | efried | But now I'm working on converting the placement API over (https://review.openstack.org/#/c/492247/) so I kinda wanted to see it working :) | |
| 17:04:44 | efried | ...and you can see from ^^ that our CI isn't having any trouble with it. | |
| 17:05:51 | sdague | :) | |
| 17:05:59 | efried | (just checked the compute logs to be sure - as if we could pass without placement being happy in the first place - and there's no 500s) | |
| 17:06:46 | sdague | yeh, it's in a weird intermediate state I think, there is a reason why mod_wsgi is something i wanted to remove from the dev/test stack. It just hits a bunch of odd apachisms that don't fit well with our other assumptions | |
| 17:07:17 | efried | sdague To be clear, all I need to do to switch is remove WSGI_MODE=mod_wsgi from my local.conf? | |
| 17:07:22 | sdague | yep | |
| 17:07:57 | efried | k. Refreshing other project clones... | |
| 17:10:34 | efried | aaand stacking... | |
| 17:10:40 | melwitt | mriedem_away: I was wondering if we need to document that for multi-cell with nova-network, upcalls from compute are required for quota checks. I was thinking we might not have to because IIRC we're not supporting multi-cell + nova-network | |
| 17:11:30 | melwitt | I think single cell would be okay because there's no isolation from the API DB there | |
| 17:13:12 | efried | Meanwhile, any idea what glorious magic makes a .txt.gz journalctl log show up in the browser with color codes translated? | |
| 17:21:33 | sdague | efried: magic yet to be written | |
| 17:22:07 | efried | sdague But it works | |
| 17:22:15 | efried | At least in my browser | |
| 17:22:19 | sdague | efried: interesting | |
| 17:22:27 | efried | Except that for powervm logs, it doesn't *quite* work. | |
| 17:22:33 | efried | sdague What, it doesn't do that for you? | |
| 17:23:13 | sdague | efried: oh, it's not color codes translated | |
| 17:23:21 | sdague | you mean the coloring of - http://logs.openstack.org/81/488381/7/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/2b8b331/logs/screen-c-api.txt.gz ? | |
| 17:23:40 | efried | sdague Yup. | |
| 17:23:43 | sdague | https://github.com/openstack-infra/os-loganalyze | |
| 17:24:07 | sdague | is a wsgi filter for the logs | |
| 17:24:22 | efried | ...that runs under the auspices of apached? | |
| 17:24:39 | sdague | yeh, log access is proxied through that | |
| 17:24:41 | efried | ...and would need to exist on the log server. | |
| 17:24:48 | sdague | yes | |
| 17:24:56 | sdague | it processes in real time | |
| 17:24:56 | efried | So ours must be outdated or something. Cool. esberglu ^^ ! | |
| 17:25:12 | sdague | there is a regex on the files it will attempt to do it with | |
| 17:25:17 | efried | sdague Cause for us, it shows up HTMLified, with the log filters, but no color translation. | |
| 17:25:28 | efried | http://184.172.12.213/manual/rest_logs/logs/n-cpu.txt.gz | |
| 17:25:40 | efried | (This isn't in production yet, cause we've been naming 'em .log.gz, but we're converting over) | |
| 17:25:42 | sdague | https://github.com/openstack-infra/os-loganalyze/blob/master/os_loganalyze/filter.py#L24-L31 | |
| 17:25:56 | sdague | yeh, you'd need to setup this code on apache | |
| 17:27:13 | sdague | https://github.com/openstack-infra/puppet-openstackci/blob/master/templates/logs.vhost.erb#L106 and friends to see how you'd set it up on apache | |
| 17:27:27 | sdague | it requires a few rewrite rules | |
| 17:27:49 | efried | sdague Is that not the same script that does the filtering capability? | |
| 17:30:34 | efried | sdague Okay that stack failed, on image create. | |
| 17:31:29 | efried | apache error log shows an ENOSPC in /tmp. I've got .8GB in there; is it really trying to copy the image through /tmp?? | |
| 17:40:53 | sdague | efried: it might be | |
| 17:41:03 | efried | sdague Restacking, gonna watch /tmp... | |
| 17:41:04 | sdague | mtreinish: ^^^ | |
| 17:41:14 | sdague | it might be specific on the uwsgi glance | |
| 17:41:25 | sdague | and, if so, that could be an interesting issue | |
| 17:42:30 | sdague | you all are booting a real image, not a cirros, right? | |
| 17:42:44 | sdague | I could totally imagine that issue to have slipped through because cirros | |
| 17:43:33 | cdent | excellent point, sdague, at least one interation of the uwsgiification of glance needed to do some writes to disk, but I don’t know if that version landed, and if it did where on disk | |
| 17:44:36 | efried | sdague Yeah, it's a real image - though it's pretty small (2GB) | |
| 17:45:04 | sdague | efried: yeh... but there was all manner of tricksyness to get uwsgi glance to work | |
| 17:45:23 | sdague | and I can totally imagine full write of the upload to tmp was part of that | |
| 17:45:29 | efried | How... precious. | |
| 17:49:22 | mriedem | melwitt: hmm, nova-network only starts in a cells v1 config | |
| 17:49:31 | mriedem | so yeah, i don't think we probably have to worry about that for cells v2 multi-cell | |
| 17:56:26 | efried | sdague Yup, thar she blows. During that image create command, /tmp fills quickly, then stack bails. | |
| 17:57:18 | mriedem | dude | |
| 17:57:20 | mriedem | thar she blars | |
| 17:58:01 | mriedem | bauzas: jaypipes: we want this in for pike too https://review.openstack.org/#/c/492234/ | |
| 17:58:20 | mriedem | melwitt: ^ | |
| 17:58:51 | jaypipes | mriedem: done | |
| 17:59:43 | mriedem | thanks | |
| 18:00:03 | SamYaple | any chance I will see this backport merge https://review.openstack.org/#/c/486272/ ? im not sure what novas backport policies are | |
| 18:00:05 | mriedem | melwitt: can you also help go through the set of docs changes leading up to https://review.openstack.org/#/c/491853/ ? that's the target for the docs migration for rc1 | |
| 18:01:27 | mriedem | SamYaple: maybe | |
| 18:01:38 | mriedem | there should be a push for stable branch reviews once we are clear of pike rc1 | |
| 18:03:22 | SamYaple | mriedem: ok thanks. i can't run ocata in my setup without this patch and im not a huge fan of carrying patches which im currently doing with it | |
| 18:03:38 | sdague | SamYaple: +2 on that. | |
| 18:04:15 | SamYaple | nice sdague! thanks | |
| 18:04:36 | mtreinish | sdague, cdent: we ended up doing mod_proxy with http traffic on a local port | |
| 18:04:57 | mtreinish | glance runs uwsgi in http mode and apache just forwards the traffic to the local uwsgi daemon | |
| 18:04:57 | sdague | mtreinish: yeh, file upload seems to create a full temp file though | |
| 18:05:10 | sdague | which, is actually kind of an issue | |
| 18:05:28 | cdent | yeah, that will be because of a tweak to mod proxy’s behavior | |
| 18:05:29 | sdague | from nova bugs, 120GB snapshots are a thing | |
| 18:05:51 | sdague | no one is planning N * 120GB free in /tmp | |
| 18:08:04 | cdent | I guess that means the question becomes: is anyone planning to duplicate devstack’s way of running glance? | |
| 18:08:09 | mtreinish | sdague: I'm not sure how the uwsgi change would effect things there. We ran basically the same config before just instead of uwsgi running glance it was glance standalone. The apache config was more or less the same for tls | |
| 18:08:21 | mtreinish | unless uwsgi is using /tmp as a cache | |
| 18:08:26 | cdent | it’s mod proxy | |
| 18:08:43 | sdague | cdent: that's an impact of the tls termination? | |
| 18:09:33 | efried | mtreinish Is there a uwsgi config somewhere that I can tell to use something other than /tmp ? | |
| 18:10:04 | efried | I'm trying again with TMPDIR=/var/tmp (where I have some more space) - but that'll only affect stuff running directly under the auspices of stack.sh | |
| 18:11:11 | efried | mm, unless I set it in /etc/environment. Let's see if that flies... | |
| 18:11:50 | cdent | sdague, mtreinish, efried : https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies | |
| 18:11:57 | cdent | it’s the sendcl thing | |
| 18:12:16 | cdent | that was added to the proxy config when glance switched to uwsg | |
| 18:12:45 | mtreinish | cdent: the mod_proxy config for glance should not be using sendcl | |