From jean.lagarde at gmail.com Sun Dec 7 19:14:56 2008 From: jean.lagarde at gmail.com (Jean Lagarde) Date: Sun, 7 Dec 2008 10:14:56 -0800 Subject: [Tramline-dev] Handling timeout in squid or other cache? Message-ID: Good day, Strictly speaking this may be more of a Squid or mod_python question, but as developers/users of tramline I wonder if some of you didn't come across this and found the best solution. I am in the process of setting up tramline in a Apache --> Squid --> Zope (Plone) configuration. I am using the tramline branch from Georges Racinet (version 0.9.0) (although I've made modifications -- I'll probably send another message about that). I'm using it with the attramline product for Plone, but that should be irrelevant. When uploading files up to a certain size it worked, but I was getting HTTP 502 errors for larger files. Things work fine for files of any size when I remove Squid from the equation. Given that the tramline filter does its write of the ID to the filter at the start of the upload (in init_data), and writes nothing else until the end of the request, I suspected a timeout issue. I saw that Squid has a request_timeout config parameter that defaults to five minutes, and increasing this did fix the problem, however, I assume that increasing that timeout to a very large value may be a bad idea for other reasons, so I wonder if there is a better way to do this, and a possible fix within tramline. Moving the writing to the filter to the end of the request (in finalizeInput?) would cause the same problem I assume. I don't get the impression that there is some sort of a non-data "keep-alive" beat that can be sent through mod_python, but maybe I'm mistaken Cheers, Jean -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/tramline-dev/attachments/20081207/5d2a2631/attachment-0001.htm From jean.lagarde at gmail.com Sun Dec 7 19:35:16 2008 From: jean.lagarde at gmail.com (Jean Lagarde) Date: Sun, 7 Dec 2008 10:35:16 -0800 Subject: [Tramline-dev] pass_on(filter) VS filter.pass_on() VS filter.disable() Message-ID: Clearly some of you have dealt with and pondered this subject, so could someone recap the best understanding of these options, for Apache 2 specifically? As I wrote in a previous email, I use an Apache2 --> Squid --> Zope configuration, and we have some content/applications served directly by Apache (a PHP based forum among other things). I started up with the version of tramline from Georges Racinet (version 0.9.0), but using the pass_on function he wrote inside of tramline seemed to make Apache work a little hard for my taste, so I left all of his other changes in but used the mix of filter.pass_on() and filter.disable() found in the latest Infrae version. That worked on our development server, but when I turned this on in our development server, all POSTs to PHP stopped working (the Plone/Zope stuff worked fine). Investigating, for one I found that I had different versions of mod_python and Apache on the dev and live server (3.3.1_2 vs 3.3.1 for mod_python, and 2.2.9_5 vs 2.2.3 for apache, the most recent versions on the dev server). I realize that this is not a good situation (our webmaster is in the process of upgrading to FreeBSD 7 and has not done the live server side yet), but I noticed that tramline with the filter.disable() calls worked fine with PHP POST forms on the dev server, but not on the live one. I fixed the problem on the live server by replacing the disable() calls with pass_on() ones. So, what is your understanding of the difference between pass_on and disable? If I use versions of other components (Apache, etc.) that support it, is is preferable to use disable, and why? Thanks for any wisdom on this. If no one has a clue, then I'll ask on the mod_python list and report back. Cheers, Jean -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/tramline-dev/attachments/20081207/ae3a386a/attachment.htm From georges at racinet.fr Sun Dec 7 20:25:46 2008 From: georges at racinet.fr (Racinet Georges) Date: Sun, 7 Dec 2008 20:25:46 +0100 Subject: [Tramline-dev] pass_on(filter) VS filter.pass_on() VS filter.disable() In-Reply-To: References: Message-ID: <47FB1848-F54E-4375-B1C7-8772430844E7@racinet.fr> On Dec 7, 2008, at 7:35 PM, Jean Lagarde wrote: > Clearly some of you have dealt with and pondered this subject, so > could someone recap the best understanding of these options, for > Apache 2 specifically? > > As I wrote in a previous email, I use an Apache2 --> Squid --> Zope > configuration, and we have some content/applications served directly > by Apache (a PHP based forum among other things). I started up with > the version of tramline from Georges Racinet (version 0.9.0), but > using the pass_on function he wrote inside of tramline seemed to > make Apache work a little hard for my taste, so I left all of his > other changes in but used the mix of filter.pass_on() and > filter.disable() found in the latest Infrae version. Maybe that means I read it too fast and tried to replace all "pass_on()" with "disable()", but frankly speaking, I don't remember exactly what tests I did perform. Just got bored at some point and got back to this poor solution (told myself that the overhead would be negligible compared to the Zope stuff behind anyway). I'm of course interested in whatever would make my tramlined systems more efficient. By the way, this pass_on() python re-implementation is something I found during a web search as a workaround by someone who was trying to use tramline. I was eager to try it, and couldn't find his post again when time came to share my mods and pay proper credits. > That worked on our development server, but when I turned this on in > our development server, all POSTs to PHP stopped working (the Plone/ > Zope stuff worked fine). Investigating, for one I found that I had > different versions of mod_python and Apache on the dev and live > server (3.3.1_2 vs 3.3.1 for mod_python, and 2.2.9_5 vs 2.2.3 for > apache, the most recent versions on the dev server). I realize that > this is not a good situation (our webmaster is in the process of > upgrading to FreeBSD 7 and has not done the live server side yet), > but I noticed that tramline with the filter.disable() calls worked > fine with PHP POST forms on the dev server, but not on the live one. > I fixed the problem on the live server by replacing the disable() > calls with pass_on() ones. Ok, so the conclusion would be that PHP has nothing to do with this, but that Apache and mod_python version do, which seems to be at least logical. > So, what is your understanding of the difference between pass_on and > disable? If I use versions of other components (Apache, etc.) that > support it, is is preferable to use disable, and why? Unfortunately, the documentation I was able to find on those matters is quite sparse. I also noticed that apache 2.2.8-1ubuntu0.3 + mod_python 3.3.1-2build1 don't even need the famous apache.py patch (at least with my version). > Thanks for any wisdom on this. If no one has a clue, then I'll ask > on the mod_python list and report back. As far as I know these filter concepts are actually simple rewrapping of the internal Apache C API, so I suppose that the matter is wider than just mod_python, but the folks on that list would probably tell you more about that. Regards, GR > Cheers, > > Jean > > _______________________________________________ > Tramline-dev mailing list > Tramline-dev at codespeak.net > http://codespeak.net/mailman/listinfo/tramline-dev -- Georges Racinet Zope/CPS/Plone expertise & assistance From jean.lagarde at gmail.com Thu Dec 11 04:13:44 2008 From: jean.lagarde at gmail.com (Jean Lagarde) Date: Wed, 10 Dec 2008 19:13:44 -0800 Subject: [Tramline-dev] pass_on(filter) VS filter.pass_on() VS filter.disable() In-Reply-To: <47FB1848-F54E-4375-B1C7-8772430844E7@racinet.fr> References: <47FB1848-F54E-4375-B1C7-8772430844E7@racinet.fr> Message-ID: Thanks Georges, Here's the reply from Graham Dumpleton below, from the mod_python list. That clearly explains the difference, and yes, it would be nice to have a C version of the tramline functionality, given that each and every request passes through it. If I ever do that, I'll announce on this list, but don't hold your breath... -- Jean If you use disable(), your filter will never be called for that > request (input or output as appropriate) again. > > If you use pass_on(), then it is passing on data available to that > specific call to your filter function. > > The distinction exists because your filter function can be called > multiple times for a request. The latter is allowing you to pass on > the data for that call, but still allowing you the option to process > data in a subsequent call of the filter function. > > The disable() call is used internally to mod_python to ensure that > filter function is never called again for a specific request if the > filter function itself raises an exception of some sort. If it doesn't > do this you could end up with a loop if the error in filter function > actually generates output, for which it would then call the filter > function again, etc etc. > > FWIW, if you are concerned about performance in any way, then you > shouldn't be using mod_python to implement Apache input/output > filters. Yes you can write filters with mod_python, but it will impact > performance. For best performance you should write filters in C code > as proper Apache module. On Sun, Dec 7, 2008 at 11:25 AM, Racinet Georges wrote: > > On Dec 7, 2008, at 7:35 PM, Jean Lagarde wrote: > > > Clearly some of you have dealt with and pondered this subject, so > > could someone recap the best understanding of these options, for > > Apache 2 specifically? > > > > As I wrote in a previous email, I use an Apache2 --> Squid --> Zope > > configuration, and we have some content/applications served directly > > by Apache (a PHP based forum among other things). I started up with > > the version of tramline from Georges Racinet (version 0.9.0), but > > using the pass_on function he wrote inside of tramline seemed to > > make Apache work a little hard for my taste, so I left all of his > > other changes in but used the mix of filter.pass_on() and > > filter.disable() found in the latest Infrae version. > > Maybe that means I read it too fast and tried to replace all > "pass_on()" with "disable()", but frankly speaking, I don't remember > exactly what tests I did perform. Just got bored at some point and got > back to this poor solution (told myself that the overhead would be > negligible compared to the Zope stuff behind anyway). > I'm of course interested in whatever would make my tramlined systems > more efficient. > > By the way, this pass_on() python re-implementation is something I > found during a web search as a workaround by someone who was trying to > use tramline. I was eager to try it, and couldn't find his post again > when time came to share my mods and pay proper credits. > > > > That worked on our development server, but when I turned this on in > > our development server, all POSTs to PHP stopped working (the Plone/ > > Zope stuff worked fine). Investigating, for one I found that I had > > different versions of mod_python and Apache on the dev and live > > server (3.3.1_2 vs 3.3.1 for mod_python, and 2.2.9_5 vs 2.2.3 for > > apache, the most recent versions on the dev server). I realize that > > this is not a good situation (our webmaster is in the process of > > upgrading to FreeBSD 7 and has not done the live server side yet), > > but I noticed that tramline with the filter.disable() calls worked > > fine with PHP POST forms on the dev server, but not on the live one. > > I fixed the problem on the live server by replacing the disable() > > calls with pass_on() ones. > > Ok, so the conclusion would be that PHP has nothing to do with this, > but that Apache and mod_python version do, which seems to be at least > logical. > > > So, what is your understanding of the difference between pass_on and > > disable? If I use versions of other components (Apache, etc.) that > > support it, is is preferable to use disable, and why? > > Unfortunately, the documentation I was able to find on those matters > is quite sparse. > > I also noticed that apache 2.2.8-1ubuntu0.3 + mod_python 3.3.1-2build1 > don't even need the famous apache.py patch (at least with my version). > > > Thanks for any wisdom on this. If no one has a clue, then I'll ask > > on the mod_python list and report back. > > As far as I know these filter concepts are actually simple rewrapping > of the internal Apache C API, so I suppose > that the matter is wider than just mod_python, but the folks on that > list would probably tell you more about that. > > Regards, > > GR > > Cheers, > > > > Jean > > > > _______________________________________________ > > Tramline-dev mailing list > > Tramline-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/tramline-dev > > -- > Georges Racinet > Zope/CPS/Plone expertise & assistance > > > > > > _______________________________________________ > Tramline-dev mailing list > Tramline-dev at codespeak.net > http://codespeak.net/mailman/listinfo/tramline-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/tramline-dev/attachments/20081210/e1c9a1a2/attachment-0001.htm From georges at racinet.fr Thu Dec 11 10:02:34 2008 From: georges at racinet.fr (Racinet Georges) Date: Thu, 11 Dec 2008 10:02:34 +0100 Subject: [Tramline-dev] pass_on(filter) VS filter.pass_on() VS filter.disable() In-Reply-To: References: <47FB1848-F54E-4375-B1C7-8772430844E7@racinet.fr> Message-ID: On Dec 11, 2008, at 4:13 AM, Jean Lagarde wrote: > Thanks Georges, > > Here's the reply from Graham Dumpleton below, from the mod_python > list. That clearly explains the difference, Hi Jean, I saw the thread on mod_python, thanks for starting this. Graham's explanations are quite clear, yes. I'll do some experiments and keep track of the results in the README. > and yes, it would be nice to have a C version of the tramline > functionality, given that each and every request passes through it. > If I ever do that, I'll announce on this list, but don't hold your > breath... That'd be cool, indeed, but I'm not really sure it's worth the effort, except if you really want to learn about C filters in Apache. As for myself, I still plan to make tramline's repo hierarchical at some point, to avoid file system choking for large amounts of files (over 10^4) on ext3, but that's it. Another task still to be done would be to clean the "couple of branches around", of course. Cheers, GR > > -- Jean > > As far as I know these filter concepts are actually simple rewrapping > of the internal Apache C API, so I suppose > that the matter is wider than just mod_python, but the folks on that > list would probably tell you more about that. Seems I was wrong here, at least for disable() > Tramline-dev mailing list > Tramline-dev at codespeak.net > http://codespeak.net/mailman/listinfo/tramline-dev > > _______________________________________________ > Tramline-dev mailing list > Tramline-dev at codespeak.net > http://codespeak.net/mailman/listinfo/tramline-dev -- Georges Racinet Zope/CPS/Plone expertise & assistance