From sylvain at infrae.com Tue Aug 12 10:41:55 2008 From: sylvain at infrae.com (Sylvain Viollon) Date: Tue, 12 Aug 2008 10:41:55 +0200 Subject: [Tramline-dev] Tramline status Message-ID: <20080812104155.39c314c7@reddwarf.infrae> Hello, I work at Infrae, and I would like to have some information about Tramline, if there is someone here. First of all, I made a buildout to setup a tramline instance, with Apache, and the mod_python: https://svn.infrae.com/buildout/tramline/trunk/ This can interest some people. I use Apache 2.2, and for that I needed some change in the code, to replace all filter.pass_on() to filter.disable() in the input filter. I would like to known if there is other people which use Apache 2.2, and if they got this problem as well. In the other case, which version of Apache do you use. After, I wonder what's the purpose of the `simplefilter.py` file in the SVN. It's seems to be a test, and to be useless. In this case, I propose to delete it. At the last, I got problems with HTTP download request which ask ranges. The problem is that the file sizes seen by the client and the real server are differents, and the real server often can't handle request with ranges, since they are over-ranges. I would like to known if someone add already that problem. I think it's common if you want to host videos with Tramline. Otherwise, I found an easy-fix to disable range request on Tramline files. Best regards, Sylvain, -- Sylvain Viollon -- Infrae t +31 10 243 7051 -- http://infrae.com Hoevestraat 10 3033GC Rotterdam -- The Netherlands -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://codespeak.net/pipermail/tramline-dev/attachments/20080812/5c76b4c8/attachment.pgp From georges.racinet at viral-prod.com Tue Aug 12 11:47:42 2008 From: georges.racinet at viral-prod.com (Georges Racinet) Date: Tue, 12 Aug 2008 11:47:42 +0200 Subject: [Tramline-dev] Tramline status In-Reply-To: <20080812104155.39c314c7@reddwarf.infrae> References: <20080812104155.39c314c7@reddwarf.infrae> Message-ID: On Aug 12, 2008, at 10:41 AM, Sylvain Viollon wrote: > > Hello, > > I work at Infrae, and I would like to have some information about > Tramline, if there is someone here. Hello ! I did an integration of Tramline for my non-profit organization (http://viral-prod.com ). For that purpose, I made a few changes to Tramline, which I didn't publish yet. I tried to reach Martijn Faasen, but without success (I've heard he's doesn't work for Infrae any more). I can forward the info it to you or the list if you wish. > > > First of all, I made a buildout to setup a tramline instance, with > Apache, and the mod_python: > > https://svn.infrae.com/buildout/tramline/trunk/ starting from the latest release, 0.5.1, IIRC > > > This can interest some people. Indeed. > > > I use Apache 2.2, and for that I needed some change in the code, to > replace all filter.pass_on() to filter.disable() in the input filter. Got through that too, although I didn't realize filter.disable() would do the job. The documentation I've found on mod_python's filters is a bit sparse and seems to imply that filter.pass_on() should do the job. So I implemented (in python, that's sub-par) what it's supposed to do. > > > I would like to known if there is other people which use Apache 2.2, > and if they got this problem as well. In the other case, which version > of Apache do you use. The one from Debian Etch: 2.2.3-4+etch4 > > > After, I wonder what's the purpose of the `simplefilter.py` file in > the SVN. It's seems to be a test, and to be useless. In this case, I > propose to delete it. I'll take a look at the svn head if I get a minute. > > > At the last, I got problems with HTTP download request which ask > ranges. The problem is that the file sizes seen by the client and the > real server are differents, and the real server often can't handle > request with ranges, since they are over-ranges. Same here. Situation got worse with Firefox 3 The thing is, the range request goes through to Zope, who can't handle it properly. A quick&dirty fix is to replace it by a 200 response, according to RFC 2616 this is admissible (they write "MAY"). I started writing some code for that (using twisted libraries for header parsing) but I didn't have time to test it yet. Would be glad if we could cooperate on that point. > > > I would like to known if someone add already that problem. I think > it's common if you want to host videos with Tramline. I've heard problems with the Adobe Acrobat plugin under Windows, too. Dont' have a win machine on hand at this time. The thing is, range requests are becoming more and more popular, as it seems. > Otherwise, I > found an easy-fix to disable range request on Tramline files. Is it the same as mine? At the time I tested it, it still didn't work perfectly, but Firefox 3 was in early beta stage then. I have a question about the svn head: did the unit tests pass for you ? The ones from the version I use are broken, and that's a pain to test my 206/416 responses... Cheers, -- Georges Racinet Zope/CPS/Plone expertise & assistance From sylvain at infrae.com Tue Aug 12 12:34:43 2008 From: sylvain at infrae.com (Sylvain Viollon) Date: Tue, 12 Aug 2008 12:34:43 +0200 Subject: [Tramline-dev] Tramline status In-Reply-To: References: <20080812104155.39c314c7@reddwarf.infrae> Message-ID: <20080812123443.69d5f2d1@reddwarf.infrae> On Tue, 12 Aug 2008 11:47:42 +0200 Georges Racinet wrote: > > Hello ! > Hello, > I did an integration of Tramline for my non-profit organization > (http://viral-prod.com ). > For that purpose, I made a few changes to Tramline, which I didn't > publish yet. > I tried to reach Martijn Faasen, but without success (I've heard > he's doesn't work for Infrae any more). > I can forward the info it to you or the list if you wish. > Of course, you can forward it either to me or the list. I often see him, so I could ask his feedback. > > > > > > First of all, I made a buildout to setup a tramline instance, with > > Apache, and the mod_python: > > > > https://svn.infrae.com/buildout/tramline/trunk/ > > starting from the latest release, 0.5.1, IIRC > > > > > > This can interest some people. > > Indeed. > > > > > > > I use Apache 2.2, and for that I needed some change in the code, to > > replace all filter.pass_on() to filter.disable() in the input > > filter. > > Got through that too, although I didn't realize filter.disable() > would do the job. The documentation I've found on mod_python's > filters is a bit sparse and seems to imply that filter.pass_on() > should do the job. So I implemented (in python, that's sub-par) what > it's supposed to do. > I agree on the documentation, there is nothing. But I saw this nice method by doing a dir on my filter object, so why not to try it. I think to use disable, if it works really, that better than having a custom code which does the same. I known that the way filter are handled had change between Apache 2.0 and 2.2, so I am not surprised by that fix. > > > > > > I would like to known if there is other people which use Apache > > 2.2, and if they got this problem as well. In the other case, which > > version of Apache do you use. > > The one from Debian Etch: 2.2.3-4+etch4 So I think it's an Apache-version related issue. I will add a test in the code, if it's Apache 2.2, to use filter.disable. > > > > > At the last, I got problems with HTTP download request which ask > > ranges. The problem is that the file sizes seen by the client and > > the real server are differents, and the real server often can't > > handle request with ranges, since they are over-ranges. > > Same here. Situation got worse with Firefox 3 > The thing is, the range request goes through to Zope, who can't > handle it properly. A quick&dirty fix is to replace it by a 200 > response, according to RFC 2616 this is admissible (they write "MAY"). > Ha. I didn't think about it. I just check that for all files I send, if there is an 'Accept-Ranges' in the response, that the value is 'none', so Zope don't get at all ranges requests. That's a two-lines fix which works quite well. I used it with an old version of Zope 3, which have some problems with request with invalid ranges (I got a python exception Attribute Error). I like this fix because this don't involve Zope at all. > I started writing some code for that (using twisted libraries for > header parsing) but I didn't have time to test it yet. > Would be glad if we could cooperate on that point. > > > > > > > I would like to known if someone add already that problem. I think > > it's common if you want to host videos with Tramline. > > I've heard problems with the Adobe Acrobat plugin under Windows, > too. Dont' have a win machine on hand at this time. The thing is, > range requests are becoming more and more popular, as it seems. > That is my problem :). > > Otherwise, I > > found an easy-fix to disable range request on Tramline files. > > Is it the same as mine? At the time I tested it, it still didn't > work perfectly, but Firefox 3 was in early beta stage then. > Nope, see above. > I have a question about the svn head: did the unit tests pass for > you ? The ones from the version I use are broken, and that's a pain > to test my 206/416 responses... I didn't try yet, but I will check. Best regards, Sylvain, -- Sylvain Viollon -- Infrae t +31 10 243 7051 -- http://infrae.com Hoevestraat 10 3033GC Rotterdam -- The Netherlands -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://codespeak.net/pipermail/tramline-dev/attachments/20080812/5a2edb6e/attachment.pgp From emyr.thomas at gmail.com Wed Aug 13 18:28:04 2008 From: emyr.thomas at gmail.com (Emyr Thomas) Date: Wed, 13 Aug 2008 17:28:04 +0100 Subject: [Tramline-dev] Tramline status In-Reply-To: <20080812123443.69d5f2d1@reddwarf.infrae> References: <20080812104155.39c314c7@reddwarf.infrae> <20080812123443.69d5f2d1@reddwarf.infrae> Message-ID: <8fb874820808130928te565303ub5d25eea6515087d@mail.gmail.com> Hi Sylvain & Georges I'm Emyr Thomas, and I have taken over maintenance of tramline while Martijn is working on other projects. No development has been done on it for a while, but it's really great to hear others are actually using it too. The current (up to date) svn repo is hosted at codespeak.net/svn/rr/tramline. I'm currently looking into moving the repo to Google Code, to make it easier to give access to other developers like yourselves. I'll keep you posted via this list. Regards --Emyr On Tue, Aug 12, 2008 at 11:34 AM, Sylvain Viollon wrote: > On Tue, 12 Aug 2008 11:47:42 +0200 > Georges Racinet wrote: > > >> >> Hello ! >> > > Hello, > >> I did an integration of Tramline for my non-profit organization >> (http://viral-prod.com ). >> For that purpose, I made a few changes to Tramline, which I didn't >> publish yet. >> I tried to reach Martijn Faasen, but without success (I've heard >> he's doesn't work for Infrae any more). >> I can forward the info it to you or the list if you wish. >> > > Of course, you can forward it either to me or the list. I often see > him, so I could ask his feedback. > >> > >> > >> > First of all, I made a buildout to setup a tramline instance, with >> > Apache, and the mod_python: >> > >> > https://svn.infrae.com/buildout/tramline/trunk/ >> >> starting from the latest release, 0.5.1, IIRC >> > >> > >> > This can interest some people. >> >> Indeed. >> >> > >> > >> > I use Apache 2.2, and for that I needed some change in the code, to >> > replace all filter.pass_on() to filter.disable() in the input >> > filter. >> >> Got through that too, although I didn't realize filter.disable() >> would do the job. The documentation I've found on mod_python's >> filters is a bit sparse and seems to imply that filter.pass_on() >> should do the job. So I implemented (in python, that's sub-par) what >> it's supposed to do. >> > > I agree on the documentation, there is nothing. But I saw this nice > method by doing a dir on my filter object, so why not to try it. > > I think to use disable, if it works really, that better than having a > custom code which does the same. I known that the way filter are > handled had change between Apache 2.0 and 2.2, so I am not surprised by > that fix. > >> > >> > >> > I would like to known if there is other people which use Apache >> > 2.2, and if they got this problem as well. In the other case, which >> > version of Apache do you use. >> >> The one from Debian Etch: 2.2.3-4+etch4 > > So I think it's an Apache-version related issue. I will add a test > in the code, if it's Apache 2.2, to use filter.disable. > >> >> > >> > At the last, I got problems with HTTP download request which ask >> > ranges. The problem is that the file sizes seen by the client and >> > the real server are differents, and the real server often can't >> > handle request with ranges, since they are over-ranges. >> >> Same here. Situation got worse with Firefox 3 >> The thing is, the range request goes through to Zope, who can't >> handle it properly. A quick&dirty fix is to replace it by a 200 >> response, according to RFC 2616 this is admissible (they write "MAY"). >> > > Ha. I didn't think about it. I just check that for all files I send, > if there is an 'Accept-Ranges' in the response, that the value is > 'none', so Zope don't get at all ranges requests. That's a two-lines fix > which works quite well. > > I used it with an old version of Zope 3, which have some problems > with request with invalid ranges (I got a python exception Attribute > Error). I like this fix because this don't involve Zope at all. > >> I started writing some code for that (using twisted libraries for >> header parsing) but I didn't have time to test it yet. >> Would be glad if we could cooperate on that point. >> >> > >> > >> > I would like to known if someone add already that problem. I think >> > it's common if you want to host videos with Tramline. >> >> I've heard problems with the Adobe Acrobat plugin under Windows, >> too. Dont' have a win machine on hand at this time. The thing is, >> range requests are becoming more and more popular, as it seems. >> > > That is my problem :). > >> > Otherwise, I >> > found an easy-fix to disable range request on Tramline files. >> >> Is it the same as mine? At the time I tested it, it still didn't >> work perfectly, but Firefox 3 was in early beta stage then. >> > > Nope, see above. > >> I have a question about the svn head: did the unit tests pass for >> you ? The ones from the version I use are broken, and that's a pain >> to test my 206/416 responses... > > I didn't try yet, but I will check. > > Best regards, > > Sylvain, > > -- > Sylvain Viollon -- Infrae > t +31 10 243 7051 -- http://infrae.com > Hoevestraat 10 3033GC Rotterdam -- The Netherlands > > _______________________________________________ > Tramline-dev mailing list > Tramline-dev at codespeak.net > http://codespeak.net/mailman/listinfo/tramline-dev > > From georges.racinet at viral-prod.com Thu Aug 14 09:44:39 2008 From: georges.racinet at viral-prod.com (Georges Racinet) Date: Thu, 14 Aug 2008 09:44:39 +0200 Subject: [Tramline-dev] Tramline status In-Reply-To: <8fb874820808130928te565303ub5d25eea6515087d@mail.gmail.com> References: <20080812104155.39c314c7@reddwarf.infrae> <20080812123443.69d5f2d1@reddwarf.infrae> <8fb874820808130928te565303ub5d25eea6515087d@mail.gmail.com> Message-ID: <1A232CC0-0376-424D-8440-6C0146D28522@viral-prod.com> On Aug 13, 2008, at 6:28 PM, Emyr Thomas wrote: > Hi Sylvain & Georges Hi Emyr > > > I'm Emyr Thomas, and I have taken over maintenance of tramline while > Martijn is working on other projects. No development has been done on > it for a while, but it's really great to hear others are actually > using it too. and I was afraid the project could be dead! I should thank Sylvain for starting this thread. > The current (up to date) svn repo is hosted at > codespeak.net/svn/rr/tramline. I'm currently looking into moving the > repo to Google Code, to make it easier to give access to other > developers like yourselves. I'll keep you posted via this list. I'll post the modifications I have to propose on the list as soon as possible. Probably tomorrow or this week-end. There's nothing stellar, mind you. Cheers, -- Georges Racinet Zope/CPS/Plone expertise & assistance From georges at racinet.fr Wed Aug 20 12:02:48 2008 From: georges at racinet.fr (Racinet Georges) Date: Wed, 20 Aug 2008 12:02:48 +0200 Subject: [Tramline-dev] List of enhancement proposals Message-ID: Hi all, sorry, I've been busier than expected. Here are my proposals for Tramline enhancements. I try to keep in mind the principle that tramline is supposed to be simple and straight to the point. The idea of the pseudo file containing the tramline id is really neat: thanks to it, there's not much code to adapt for Tramlime in the application server. But that "not much" is nonzero. For instance, in my case (CPS with additional products for media files) we need to access the files for deletion and cloning, and also analysis (content extraction for indexing, ID3 tags...). I'll start with the ones I'm already using in the context of CPSTramline. I believe these enhancements together still meet the simplicity requirements of Tramline. Tell me what you think. I didn't try Railroad btw, maybe because it's supposed to be non-simple, or because I had the feeling it'd be less backend-agnostic. There's a good chance I can start porting to the svn trunk next week. By the way, I did check that the tests pass in there. DONE: - new PythonOption to set write group permission on files. Typically, an application like CPS or Plone will need to do a minimum of management, deleting, cloning, the like. I don't like running the app under the same user as Apache, and prefer resorting to groups. - new PythonOption to tramline only some of the files in a POST request. Currently this is for CPS only (hardcoded input id prefix + support for so called flexible widgets), but it could be made backend agnostic by the registration of a regexp. Among other things, this is useful if one has attached file content with some logic (ID3 extraction and pdf indexing in my case) and can't afford adapting this logic right away for all content types. Also, one might not want to clutter Tramline repository with small content like images under 100 KB (see below) IN PROGRESS: - support for Range queries and appropriate (416 and 206) responses. Things like Adobe plugin on windows, and, as it seems, Firefox 3 perform range queries, that don't behave well, because they pass through to the application server. We already discussed possible quick fixes (bypass actually) with Sylvain. PLANNED: - simple repository hierarchical structure (similar to Apache caches). ext3 does not deal happily with 30000+ files flat in a directory. Implementation would be mostly trivial, but would require migration of existing data. Implementation not breaking existing repositories would be less straightforward but nevertheless doable. Or we could have the tree structure optionall (and disabled by default). While we're at it, the id should have a fixed length (hexa dump instead of decimal would do). - Slight refactoring to single out a method to get the filesystem path from the tramline id (if possible) The backend server needs to access the files anyway (indexing, various metadata extraction, basic management). I'd prefer the backend server to import code from tramline rather than re- implementing it. Not really necessary without previous point, the path being so simple.