| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-10 | |||
| 17:30:34 | efried | sdague Okay that stack failed, on image create. | |
| 17:31:29 | efried | apache error log shows an ENOSPC in /tmp. I've got .8GB in there; is it really trying to copy the image through /tmp?? | |
| 17:40:53 | sdague | efried: it might be | |
| 17:41:03 | efried | sdague Restacking, gonna watch /tmp... | |
| 17:41:04 | sdague | mtreinish: ^^^ | |
| 17:41:14 | sdague | it might be specific on the uwsgi glance | |
| 17:41:25 | sdague | and, if so, that could be an interesting issue | |
| 17:42:30 | sdague | you all are booting a real image, not a cirros, right? | |
| 17:42:44 | sdague | I could totally imagine that issue to have slipped through because cirros | |
| 17:43:33 | cdent | excellent point, sdague, at least one interation of the uwsgiification of glance needed to do some writes to disk, but I don’t know if that version landed, and if it did where on disk | |
| 17:44:36 | efried | sdague Yeah, it's a real image - though it's pretty small (2GB) | |
| 17:45:04 | sdague | efried: yeh... but there was all manner of tricksyness to get uwsgi glance to work | |
| 17:45:23 | sdague | and I can totally imagine full write of the upload to tmp was part of that | |
| 17:45:29 | efried | How... precious. | |
| 17:49:22 | mriedem | melwitt: hmm, nova-network only starts in a cells v1 config | |
| 17:49:31 | mriedem | so yeah, i don't think we probably have to worry about that for cells v2 multi-cell | |
| 17:56:26 | efried | sdague Yup, thar she blows. During that image create command, /tmp fills quickly, then stack bails. | |
| 17:57:18 | mriedem | dude | |
| 17:57:20 | mriedem | thar she blars | |
| 17:58:01 | mriedem | bauzas: jaypipes: we want this in for pike too https://review.openstack.org/#/c/492234/ | |
| 17:58:20 | mriedem | melwitt: ^ | |
| 17:58:51 | jaypipes | mriedem: done | |
| 17:59:43 | mriedem | thanks | |
| 18:00:03 | SamYaple | any chance I will see this backport merge https://review.openstack.org/#/c/486272/ ? im not sure what novas backport policies are | |
| 18:00:05 | mriedem | melwitt: can you also help go through the set of docs changes leading up to https://review.openstack.org/#/c/491853/ ? that's the target for the docs migration for rc1 | |
| 18:01:27 | mriedem | SamYaple: maybe | |
| 18:01:38 | mriedem | there should be a push for stable branch reviews once we are clear of pike rc1 | |
| 18:03:22 | SamYaple | mriedem: ok thanks. i can't run ocata in my setup without this patch and im not a huge fan of carrying patches which im currently doing with it | |
| 18:03:38 | sdague | SamYaple: +2 on that. | |
| 18:04:15 | SamYaple | nice sdague! thanks | |
| 18:04:36 | mtreinish | sdague, cdent: we ended up doing mod_proxy with http traffic on a local port | |
| 18:04:57 | sdague | mtreinish: yeh, file upload seems to create a full temp file though | |
| 18:04:57 | mtreinish | glance runs uwsgi in http mode and apache just forwards the traffic to the local uwsgi daemon | |
| 18:05:10 | sdague | which, is actually kind of an issue | |
| 18:05:28 | cdent | yeah, that will be because of a tweak to mod proxy’s behavior | |
| 18:05:29 | sdague | from nova bugs, 120GB snapshots are a thing | |
| 18:05:51 | sdague | no one is planning N * 120GB free in /tmp | |
| 18:08:04 | cdent | I guess that means the question becomes: is anyone planning to duplicate devstack’s way of running glance? | |
| 18:08:09 | mtreinish | sdague: I'm not sure how the uwsgi change would effect things there. We ran basically the same config before just instead of uwsgi running glance it was glance standalone. The apache config was more or less the same for tls | |
| 18:08:21 | mtreinish | unless uwsgi is using /tmp as a cache | |
| 18:08:26 | cdent | it’s mod proxy | |
| 18:08:43 | sdague | cdent: that's an impact of the tls termination? | |
| 18:09:33 | efried | mtreinish Is there a uwsgi config somewhere that I can tell to use something other than /tmp ? | |
| 18:10:04 | efried | I'm trying again with TMPDIR=/var/tmp (where I have some more space) - but that'll only affect stuff running directly under the auspices of stack.sh | |
| 18:11:11 | efried | mm, unless I set it in /etc/environment. Let's see if that flies... | |
| 18:11:50 | cdent | sdague, mtreinish, efried : https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies | |
| 18:11:57 | cdent | it’s the sendcl thing | |
| 18:12:16 | cdent | that was added to the proxy config when glance switched to uwsg | |
| 18:12:45 | mtreinish | cdent: the mod_proxy config for glance should not be using sendcl | |
| 18:12:58 | cdent | did you take it back out? | |
| 18:13:08 | mtreinish | that was one of the paths I was investigating and it didn't work for various reasons (including this issue growing tmp space) | |
| 18:13:52 | melwitt | mriedem: sure thing | |
| 18:14:10 | mtreinish | cdent: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L333 is the path for glance | |
| 18:14:11 | cdent | it’s currently on line 276 of lib/apache | |
| 18:14:32 | mtreinish | cdent: right that's the path that's not glance | |
| 18:14:42 | cdent | is setenv iglobal to apache ? | |
| 18:15:31 | cdent | It looks like it may be | |
| 18:16:33 | sdague | so... if you hit uwsgi directly without the tls proxy, this wouldn't be an issue? | |
| 18:17:12 | cdent | sdague: unclear, because it may be the mod_proxy_uwsgi also attends to it? | |
| 18:18:12 | cdent | I’m not certain this is the problem, but I do think sendcl is a factor, or something like it | |
| 18:18:22 | cdent | hard to be clear what uwsgi does in the face of file uploads | |
| 18:18:32 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 18:18:43 | sdague | yeh, it's supposed to be chunk encoded upload | |
| 18:18:58 | sdague | but bouncing through proxies... things get tricky | |
| 18:19:43 | mtreinish | is there a running system with this issue? because it's easy to confirm if apache is passing chunked data to uwsgi with a tcpump on the local port | |
| 18:19:47 | cdent | if you are sending chunked but demanding content-lenght (from whatever directive) a write will be required | |
| 18:20:20 | mriedem | sdague: i've got a couple of comments in here https://review.openstack.org/#/c/491815 | |
| 18:20:21 | mtreinish | cdent: if setenv is globally scoped across different site files we probaly just need to create a virtualhost for the glance path to isolate it from the other sites | |
| 18:20:59 | mriedem | melwitt: take a look at my comments if you update | |
| 18:21:10 | mtreinish | cdent: fwiw, the glance path doesn't use mod_proxy_uwsgi | |
| 18:21:35 | cdent | I know, but it does use mod_proxy doesn’t it? At least that’s what I thought I heard you say? | |
| 18:21:43 | mtreinish | yes it does | |
| 18:21:46 | melwitt | mriedem: phew, in the nick of time | |
| 18:22:27 | melwitt | looks like sdague is on it already | |
| 18:22:41 | sdague | mriedem: ok, so an inline change to that is going to be 4 - 6 hours to get check results back | |
| 18:22:54 | sdague | how late you staying up for the branch? | |
| 18:22:59 | sdague | vs. fix later | |
| 18:23:38 | mriedem | it's just docs, shouldn't run all jobs | |
| 18:23:44 | sdague | it doesn't | |
| 18:23:50 | sdague | the check queue is starved out | |
| 18:23:59 | melwitt | hm, yeah I didn't think about going through the check queue again. though I thought it would be fast for docs | |
| 18:24:01 | sdague | it's hours before you get a slot | |
| 18:24:20 | mriedem | are you already doing a dump of review comment follow up stuff at the end of the series? | |
| 18:24:30 | mriedem | like https://review.openstack.org/#/c/491517/ | |
| 18:24:47 | sdague | I rebased that and stuck it on the end | |
| 18:25:01 | sdague | as it was in merge conflict, and I didn't want to generate more conflicts | |
| 18:25:03 | mriedem | alright i'm fine with following up at the end of the series | |
| 18:25:10 | sdague | ok, can do | |
| 18:25:19 | mriedem | melwitt: you want the +W ? | |
| 18:25:33 | openstackgerrit | Merged openstack/nova master: Add format_dom for PCI device addresses https://review.openstack.org/491822 | |
| 18:25:55 | melwitt | mriedem: hell yes I do | |
| 18:26:00 | sdague | I also tried to set expectations here - https://review.openstack.org/#/c/492124 how we really do need to think about docs reviews a bit differently than code reviews, especially because they are merge magnets | |
| 18:27:04 | sdague | yeh, it looks like we're at 3.5 hours hold to get check nodes | |
| 18:27:23 | cdent | mtreinish: you added the sendcl, do you remember why? https://review.openstack.org/#/c/459451/ | |
| 18:27:42 | cdent | it looks like maybe you accidentally left it in | |
| 18:27:42 | melwitt | +Warthogged | |
| 18:27:54 | sdague | melwitt: thanks | |
| 18:27:54 | cdent | after choosing a different approach | |
| 18:28:16 | sdague | ok, I'll work on follow up fix patch at top of stack | |
| 18:28:53 | mtreinish | cdent: my intent there was specifically for the non-glance case. If nova or others gets a chunked request we don't want it to blow up an invalid content length error | |