| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-10 | |||
| 18:08:09 | mtreinish | sdague: I'm not sure how the uwsgi change would effect things there. We ran basically the same config before just instead of uwsgi running glance it was glance standalone. The apache config was more or less the same for tls | |
| 18:08:21 | mtreinish | unless uwsgi is using /tmp as a cache | |
| 18:08:26 | cdent | it’s mod proxy | |
| 18:08:43 | sdague | cdent: that's an impact of the tls termination? | |
| 18:09:33 | efried | mtreinish Is there a uwsgi config somewhere that I can tell to use something other than /tmp ? | |
| 18:10:04 | efried | I'm trying again with TMPDIR=/var/tmp (where I have some more space) - but that'll only affect stuff running directly under the auspices of stack.sh | |
| 18:11:11 | efried | mm, unless I set it in /etc/environment. Let's see if that flies... | |
| 18:11:50 | cdent | sdague, mtreinish, efried : https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies | |
| 18:11:57 | cdent | it’s the sendcl thing | |
| 18:12:16 | cdent | that was added to the proxy config when glance switched to uwsg | |
| 18:12:45 | mtreinish | cdent: the mod_proxy config for glance should not be using sendcl | |
| 18:12:58 | cdent | did you take it back out? | |
| 18:13:08 | mtreinish | that was one of the paths I was investigating and it didn't work for various reasons (including this issue growing tmp space) | |
| 18:13:52 | melwitt | mriedem: sure thing | |
| 18:14:10 | mtreinish | cdent: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L333 is the path for glance | |
| 18:14:11 | cdent | it’s currently on line 276 of lib/apache | |
| 18:14:32 | mtreinish | cdent: right that's the path that's not glance | |
| 18:14:42 | cdent | is setenv iglobal to apache ? | |
| 18:15:31 | cdent | It looks like it may be | |
| 18:16:33 | sdague | so... if you hit uwsgi directly without the tls proxy, this wouldn't be an issue? | |
| 18:17:12 | cdent | sdague: unclear, because it may be the mod_proxy_uwsgi also attends to it? | |
| 18:18:12 | cdent | I’m not certain this is the problem, but I do think sendcl is a factor, or something like it | |
| 18:18:22 | cdent | hard to be clear what uwsgi does in the face of file uploads | |
| 18:18:32 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 18:18:43 | sdague | yeh, it's supposed to be chunk encoded upload | |
| 18:18:58 | sdague | but bouncing through proxies... things get tricky | |
| 18:19:43 | mtreinish | is there a running system with this issue? because it's easy to confirm if apache is passing chunked data to uwsgi with a tcpump on the local port | |
| 18:19:47 | cdent | if you are sending chunked but demanding content-lenght (from whatever directive) a write will be required | |
| 18:20:20 | mriedem | sdague: i've got a couple of comments in here https://review.openstack.org/#/c/491815 | |
| 18:20:21 | mtreinish | cdent: if setenv is globally scoped across different site files we probaly just need to create a virtualhost for the glance path to isolate it from the other sites | |
| 18:20:59 | mriedem | melwitt: take a look at my comments if you update | |
| 18:21:10 | mtreinish | cdent: fwiw, the glance path doesn't use mod_proxy_uwsgi | |
| 18:21:35 | cdent | I know, but it does use mod_proxy doesn’t it? At least that’s what I thought I heard you say? | |
| 18:21:43 | mtreinish | yes it does | |
| 18:21:46 | melwitt | mriedem: phew, in the nick of time | |
| 18:22:27 | melwitt | looks like sdague is on it already | |
| 18:22:41 | sdague | mriedem: ok, so an inline change to that is going to be 4 - 6 hours to get check results back | |
| 18:22:54 | sdague | how late you staying up for the branch? | |
| 18:22:59 | sdague | vs. fix later | |
| 18:23:38 | mriedem | it's just docs, shouldn't run all jobs | |
| 18:23:44 | sdague | it doesn't | |
| 18:23:50 | sdague | the check queue is starved out | |
| 18:23:59 | melwitt | hm, yeah I didn't think about going through the check queue again. though I thought it would be fast for docs | |
| 18:24:01 | sdague | it's hours before you get a slot | |
| 18:24:20 | mriedem | are you already doing a dump of review comment follow up stuff at the end of the series? | |
| 18:24:30 | mriedem | like https://review.openstack.org/#/c/491517/ | |
| 18:24:47 | sdague | I rebased that and stuck it on the end | |
| 18:25:01 | sdague | as it was in merge conflict, and I didn't want to generate more conflicts | |
| 18:25:03 | mriedem | alright i'm fine with following up at the end of the series | |
| 18:25:10 | sdague | ok, can do | |
| 18:25:19 | mriedem | melwitt: you want the +W ? | |
| 18:25:33 | openstackgerrit | Merged openstack/nova master: Add format_dom for PCI device addresses https://review.openstack.org/491822 | |
| 18:25:55 | melwitt | mriedem: hell yes I do | |
| 18:26:00 | sdague | I also tried to set expectations here - https://review.openstack.org/#/c/492124 how we really do need to think about docs reviews a bit differently than code reviews, especially because they are merge magnets | |
| 18:27:04 | sdague | yeh, it looks like we're at 3.5 hours hold to get check nodes | |
| 18:27:23 | cdent | mtreinish: you added the sendcl, do you remember why? https://review.openstack.org/#/c/459451/ | |
| 18:27:42 | cdent | it looks like maybe you accidentally left it in | |
| 18:27:42 | melwitt | +Warthogged | |
| 18:27:54 | sdague | melwitt: thanks | |
| 18:27:54 | cdent | after choosing a different approach | |
| 18:28:16 | sdague | ok, I'll work on follow up fix patch at top of stack | |
| 18:28:53 | mtreinish | cdent: my intent there was specifically for the non-glance case. If nova or others gets a chunked request we don't want it to blow up an invalid content length error | |
| 18:29:16 | mtreinish | so I wanted sendcl for the mod_proxy_uwsgi case so it would handle those requests properly | |
| 18:29:55 | mtreinish | none of the projects using that path should be taking large chunked uploads over the api (it's only nova, cinder, and placement) | |
| 18:30:28 | mriedem | oh no | |
| 18:31:28 | mriedem | sdague: you've just started two major new ML thread wars https://review.openstack.org/#/c/491834/3/doc/source/index.rst | |
| 18:31:38 | mriedem | sdague: crisis the first: you omitted CONTAINERS! | |
| 18:31:55 | mriedem | crisis the second: OpenSack is a new competing community of cloud hacky sack based software that will kill us all! | |
| 18:32:16 | sdague | mriedem: the only in tree container support is untested libvirt lxc, right? | |
| 18:32:21 | mriedem | virtuozzo | |
| 18:32:25 | mriedem | and lxc | |
| 18:32:25 | sdague | oh, right | |
| 18:32:35 | mriedem | but yes lxc is likely a floating turd | |
| 18:32:47 | sdague | yeh, I debated it, someone else can tweak that | |
| 18:32:47 | mriedem | tried getting CI running for that several releases ago | |
| 18:32:57 | sdague | my goals, make this page navigable | |
| 18:33:02 | mriedem | "limited container support" is probably ok | |
| 18:34:29 | mriedem | heh, nova-net! https://review.openstack.org/#/c/492105/2/doc/source/figures/nova-conf-xen-flat.png,unified | |
| 18:36:55 | sdague | yeh, there is all kinds of fun that came over with that import | |
| 18:38:02 | openstackgerrit | Ilya Popov proposed openstack/nova master: Tests: Add cleanup of 'instances' directory https://review.openstack.org/491589 | |
| 18:40:23 | mriedem | yeah, i finally know about the EMC system enabler https://review.openstack.org/#/c/492105/2/doc/source/figures/emc-enabler.png,unified | |
| 18:42:35 | mtreinish | cdent: looks like you're right the file is coming from mod_proxy | |
| 18:43:08 | mtreinish | I'll write up a patch to create an isolated virtual host for the glance apache endpoint and hopefully that should isolate the sendcl param | |
| 18:43:20 | openstackgerrit | Sean Dague proposed openstack/nova master: clean ups from I6815958b2533d462a2e5d27e7be57440d9f4f40a https://review.openstack.org/492645 | |
| 18:43:33 | sdague | mriedem / melwitt ^^^ ok, go to town ripping on that | |
| 18:43:43 | sdague | I'll iterate until we get it good enough | |
| 18:45:07 | melwitt | cool, thanks | |
| 18:55:22 | sdague | melwitt: on https://review.openstack.org/#/c/491853 realize that's just the import plus reference corrections to make it build | |
| 18:56:06 | melwitt | sdague: ah, I see that now. I thought the Cells v1 vs v2 warning was new but I see now that it's not | |
| 18:56:14 | sdague | yeh | |
| 18:56:46 | sdague | let us say... the existing docs are a mixed bag. It's probably better to have them in, but I think we're going to need a doc sprint early in the cycle to clean up a bunch of this | |
| 18:57:15 | melwitt | sdague: I think the only other thing not imported is the database config section that's not mentioned as skipped in the commit message. is that one supposed to be skipped? | |
| 18:57:39 | sdague | yeh, it turned into a really goofy thing | |
| 18:57:44 | melwitt | k | |
| 18:57:49 | sdague | let me pull up a couple of links | |
| 18:58:34 | sdague | the database doc in question was this - https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/compute/database-connections.rst#L2 | |
| 18:59:04 | sdague | which besides being kind of inaccurate with the "any sqla" bit, it was also pretty vapid in what it said | |
| 18:59:37 | sdague | and was mostly a link to https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/tables/common-database.rst#L11 | |
| 18:59:49 | melwitt | cool. just making sure it wasn't missed. I'm kind of just skimming this | |
| 18:59:56 | sdague | yeh | |