| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-10 | |||
| 18:10:04 | efried | I'm trying again with TMPDIR=/var/tmp (where I have some more space) - but that'll only affect stuff running directly under the auspices of stack.sh | |
| 18:11:11 | efried | mm, unless I set it in /etc/environment. Let's see if that flies... | |
| 18:11:50 | cdent | sdague, mtreinish, efried : https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies | |
| 18:11:57 | cdent | it’s the sendcl thing | |
| 18:12:16 | cdent | that was added to the proxy config when glance switched to uwsg | |
| 18:12:45 | mtreinish | cdent: the mod_proxy config for glance should not be using sendcl | |
| 18:12:58 | cdent | did you take it back out? | |
| 18:13:08 | mtreinish | that was one of the paths I was investigating and it didn't work for various reasons (including this issue growing tmp space) | |
| 18:13:52 | melwitt | mriedem: sure thing | |
| 18:14:10 | mtreinish | cdent: https://github.com/openstack-dev/devstack/blob/master/lib/apache#L333 is the path for glance | |
| 18:14:11 | cdent | it’s currently on line 276 of lib/apache | |
| 18:14:32 | mtreinish | cdent: right that's the path that's not glance | |
| 18:14:42 | cdent | is setenv iglobal to apache ? | |
| 18:15:31 | cdent | It looks like it may be | |
| 18:16:33 | sdague | so... if you hit uwsgi directly without the tls proxy, this wouldn't be an issue? | |
| 18:17:12 | cdent | sdague: unclear, because it may be the mod_proxy_uwsgi also attends to it? | |
| 18:18:12 | cdent | I’m not certain this is the problem, but I do think sendcl is a factor, or something like it | |
| 18:18:22 | cdent | hard to be clear what uwsgi does in the face of file uploads | |
| 18:18:32 | openstackgerrit | Merged openstack/nova master: Imported Translations from Zanata https://review.openstack.org/477091 | |
| 18:18:43 | sdague | yeh, it's supposed to be chunk encoded upload | |
| 18:18:58 | sdague | but bouncing through proxies... things get tricky | |
| 18:19:43 | mtreinish | is there a running system with this issue? because it's easy to confirm if apache is passing chunked data to uwsgi with a tcpump on the local port | |
| 18:19:47 | cdent | if you are sending chunked but demanding content-lenght (from whatever directive) a write will be required | |
| 18:20:20 | mriedem | sdague: i've got a couple of comments in here https://review.openstack.org/#/c/491815 | |
| 18:20:21 | mtreinish | cdent: if setenv is globally scoped across different site files we probaly just need to create a virtualhost for the glance path to isolate it from the other sites | |
| 18:20:59 | mriedem | melwitt: take a look at my comments if you update | |
| 18:21:10 | mtreinish | cdent: fwiw, the glance path doesn't use mod_proxy_uwsgi | |
| 18:21:35 | cdent | I know, but it does use mod_proxy doesn’t it? At least that’s what I thought I heard you say? | |
| 18:21:43 | mtreinish | yes it does | |
| 18:21:46 | melwitt | mriedem: phew, in the nick of time | |
| 18:22:27 | melwitt | looks like sdague is on it already | |
| 18:22:41 | sdague | mriedem: ok, so an inline change to that is going to be 4 - 6 hours to get check results back | |
| 18:22:54 | sdague | how late you staying up for the branch? | |
| 18:22:59 | sdague | vs. fix later | |
| 18:23:38 | mriedem | it's just docs, shouldn't run all jobs | |
| 18:23:44 | sdague | it doesn't | |
| 18:23:50 | sdague | the check queue is starved out | |
| 18:23:59 | melwitt | hm, yeah I didn't think about going through the check queue again. though I thought it would be fast for docs | |
| 18:24:01 | sdague | it's hours before you get a slot | |
| 18:24:20 | mriedem | are you already doing a dump of review comment follow up stuff at the end of the series? | |
| 18:24:30 | mriedem | like https://review.openstack.org/#/c/491517/ | |
| 18:24:47 | sdague | I rebased that and stuck it on the end | |
| 18:25:01 | sdague | as it was in merge conflict, and I didn't want to generate more conflicts | |
| 18:25:03 | mriedem | alright i'm fine with following up at the end of the series | |
| 18:25:10 | sdague | ok, can do | |
| 18:25:19 | mriedem | melwitt: you want the +W ? | |
| 18:25:33 | openstackgerrit | Merged openstack/nova master: Add format_dom for PCI device addresses https://review.openstack.org/491822 | |
| 18:25:55 | melwitt | mriedem: hell yes I do | |
| 18:26:00 | sdague | I also tried to set expectations here - https://review.openstack.org/#/c/492124 how we really do need to think about docs reviews a bit differently than code reviews, especially because they are merge magnets | |
| 18:27:04 | sdague | yeh, it looks like we're at 3.5 hours hold to get check nodes | |
| 18:27:23 | cdent | mtreinish: you added the sendcl, do you remember why? https://review.openstack.org/#/c/459451/ | |
| 18:27:42 | cdent | it looks like maybe you accidentally left it in | |
| 18:27:42 | melwitt | +Warthogged | |
| 18:27:54 | sdague | melwitt: thanks | |
| 18:27:54 | cdent | after choosing a different approach | |
| 18:28:16 | sdague | ok, I'll work on follow up fix patch at top of stack | |
| 18:28:53 | mtreinish | cdent: my intent there was specifically for the non-glance case. If nova or others gets a chunked request we don't want it to blow up an invalid content length error | |
| 18:29:16 | mtreinish | so I wanted sendcl for the mod_proxy_uwsgi case so it would handle those requests properly | |
| 18:29:55 | mtreinish | none of the projects using that path should be taking large chunked uploads over the api (it's only nova, cinder, and placement) | |
| 18:30:28 | mriedem | oh no | |
| 18:31:28 | mriedem | sdague: you've just started two major new ML thread wars https://review.openstack.org/#/c/491834/3/doc/source/index.rst | |
| 18:31:38 | mriedem | sdague: crisis the first: you omitted CONTAINERS! | |
| 18:31:55 | mriedem | crisis the second: OpenSack is a new competing community of cloud hacky sack based software that will kill us all! | |
| 18:32:16 | sdague | mriedem: the only in tree container support is untested libvirt lxc, right? | |
| 18:32:21 | mriedem | virtuozzo | |
| 18:32:25 | mriedem | and lxc | |
| 18:32:25 | sdague | oh, right | |
| 18:32:35 | mriedem | but yes lxc is likely a floating turd | |
| 18:32:47 | sdague | yeh, I debated it, someone else can tweak that | |
| 18:32:47 | mriedem | tried getting CI running for that several releases ago | |
| 18:32:57 | sdague | my goals, make this page navigable | |
| 18:33:02 | mriedem | "limited container support" is probably ok | |
| 18:34:29 | mriedem | heh, nova-net! https://review.openstack.org/#/c/492105/2/doc/source/figures/nova-conf-xen-flat.png,unified | |
| 18:36:55 | sdague | yeh, there is all kinds of fun that came over with that import | |
| 18:38:02 | openstackgerrit | Ilya Popov proposed openstack/nova master: Tests: Add cleanup of 'instances' directory https://review.openstack.org/491589 | |
| 18:40:23 | mriedem | yeah, i finally know about the EMC system enabler https://review.openstack.org/#/c/492105/2/doc/source/figures/emc-enabler.png,unified | |
| 18:42:35 | mtreinish | cdent: looks like you're right the file is coming from mod_proxy | |
| 18:43:08 | mtreinish | I'll write up a patch to create an isolated virtual host for the glance apache endpoint and hopefully that should isolate the sendcl param | |
| 18:43:20 | openstackgerrit | Sean Dague proposed openstack/nova master: clean ups from I6815958b2533d462a2e5d27e7be57440d9f4f40a https://review.openstack.org/492645 | |
| 18:43:33 | sdague | mriedem / melwitt ^^^ ok, go to town ripping on that | |
| 18:43:43 | sdague | I'll iterate until we get it good enough | |
| 18:45:07 | melwitt | cool, thanks | |
| 18:55:22 | sdague | melwitt: on https://review.openstack.org/#/c/491853 realize that's just the import plus reference corrections to make it build | |
| 18:56:06 | melwitt | sdague: ah, I see that now. I thought the Cells v1 vs v2 warning was new but I see now that it's not | |
| 18:56:14 | sdague | yeh | |
| 18:56:46 | sdague | let us say... the existing docs are a mixed bag. It's probably better to have them in, but I think we're going to need a doc sprint early in the cycle to clean up a bunch of this | |
| 18:57:15 | melwitt | sdague: I think the only other thing not imported is the database config section that's not mentioned as skipped in the commit message. is that one supposed to be skipped? | |
| 18:57:39 | sdague | yeh, it turned into a really goofy thing | |
| 18:57:44 | melwitt | k | |
| 18:57:49 | sdague | let me pull up a couple of links | |
| 18:58:34 | sdague | the database doc in question was this - https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/compute/database-connections.rst#L2 | |
| 18:59:04 | sdague | which besides being kind of inaccurate with the "any sqla" bit, it was also pretty vapid in what it said | |
| 18:59:37 | sdague | and was mostly a link to https://github.com/openstack/openstack-manuals/blob/6f9fc171800e8a435011f38cd4558e900884ce86/doc/config-reference/source/tables/common-database.rst#L11 | |
| 18:59:49 | melwitt | cool. just making sure it wasn't missed. I'm kind of just skimming this | |
| 18:59:56 | sdague | yeh | |
| 19:00:05 | sdague | the common tables things was kind of a bear | |
| 19:01:22 | sdague | steven spent a day or two fixing references, and I spent another day after him doing that. There was a lot more common normalization in the openstack-manuals repo that had to be denormalized for all of this. | |
| 19:03:03 | sdague | ok, afk for a bit | |
| 19:03:12 | mtreinish | efried_afk: https://review.openstack.org/#/c/492651/ | |
| 19:03:20 | mtreinish | sdague: ^^^ that should fix the glance thing | |