From p.giarrusso at gmail.com Fri Oct 1 00:11:17 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Fri, 1 Oct 2010 00:11:17 +0200 Subject: [pypy-dev] Question on the future of RPython In-Reply-To: References: <525425.61205.qm@web53702.mail.re2.yahoo.com> <827221.6704.qm@web53704.mail.re2.yahoo.com> <1285616691.5954.34.camel@localhost> <201009280157.17264.jacob@openend.se> <1285634613.5954.112.camel@localhost> <1285703348.5276.56.camel@localhost> <1285809701.5129.150.camel@localhost> Message-ID: On Thu, Sep 30, 2010 at 13:01, Armin Rigo wrote: > Hi Paolo, > On Thu, Sep 30, 2010 at 8:33 AM, Paolo Giarrusso wrote: >> My proposal, here, would be a "virtual guard", (...) > Yes, this proposal makes sense. ?It's an optimization that is > definitely done in regular JITs, and we have a "to-do" task about it > in http://codespeak.net/svn/pypy/extradoc/planning/jit.txt (where they > are called "out-of-line guards"). I see - but while I know of what for instance Java does (I had an example), here one can reuse JITted traces (up to some point) rather than just throw everything out (except maybe profiling data), add the new code path and redo all optimizations when recompiling it. I don't see how a method-at-a-time JITs could reuse more than that, and my class notes just state that code can be thrown out. My proposal is that here, also the recorded traces could be reused (if they are stored or can be recovered), exactly because one just adds a different code path and tracing JITs reason in terms of single paths. Ideally, the same trace could be recompiled (this time with a guard) the next time it is entered, without using the trace-recording interpreter, nor waiting again that it is executed N times with N > compileThreshold. Potentially, in a trace-stitching JIT like PyPy, one could even just prepend a guard to the compiled assembly/binary code (if that's position-independent, like x86-64, or can be relocated). Of course, all of this is dependent on the representation used for the "code/trace caches", because I'm not sure all needed data is kept - and this shouldn't increase too much storage requirements, especially given that invalidations might not be very frequent. So, do you have pointers in the code? Do you prefer me to ask by chat? I just checked out sources on my new dev machine (lack of disk space was one reason why I never did it). Is the History class in pypy/jit/metainterp/history.py what represents a trace? Grepping for "trace" in that folder didn't help a lot. BTW, I am not used to bother authors by asking questions when reading sources; but as far as I saw, it is considered "socially acceptable" here, isn't it? Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From arigo at tunes.org Fri Oct 1 11:31:30 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 1 Oct 2010 11:31:30 +0200 Subject: [pypy-dev] =?iso-8859-1?q?Next_sprint=3A_D=FCsseldorf=2C_end_of_O?= =?iso-8859-1?q?ctober?= Message-ID: Hi all! D?sseldorf PyPy sprint October 25th-31st 2010 ===================================================== The next PyPy sprint will be held in the Computer Science department of Heinrich-Heine Universit?t D?sseldorf from the 25th to the 31st of October 2010. This is a fully public sprint, everyone is welcome to join us. Topics and goals ---------------- Open. (to be defined more precisely) Location -------- The sprint will take place in a seminar room of the computer science department. It is in the building 25.12 of the university campus. For travel instructions see http://stups.cs.uni-duesseldorf.de/anreise/esbahn.php Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2010/people.txt From arigo at tunes.org Fri Oct 1 12:02:49 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 1 Oct 2010 12:02:49 +0200 Subject: [pypy-dev] Question on the future of RPython In-Reply-To: References: <525425.61205.qm@web53702.mail.re2.yahoo.com> <827221.6704.qm@web53704.mail.re2.yahoo.com> <1285616691.5954.34.camel@localhost> <201009280157.17264.jacob@openend.se> <1285634613.5954.112.camel@localhost> <1285703348.5276.56.camel@localhost> <1285809701.5129.150.camel@localhost> Message-ID: Hi Paolo, On Fri, Oct 1, 2010 at 12:11 AM, Paolo Giarrusso wrote: > BTW, I am not used to bother authors by asking questions when reading > sources; but as far as I saw, it is considered "socially acceptable" > here, isn't it? Yes, absolutely :-) Indeed, asking by chat (#pypy on irc.freenode.net) is the best way. Please feel free to do it. Armin From anto.cuni at gmail.com Mon Oct 4 23:56:34 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 04 Oct 2010 23:56:34 +0200 Subject: [pypy-dev] [pypy-svn] r77589 - pypy/trunk/pypy/jit/metainterp In-Reply-To: <20101004213232.2F5A1282B90@codespeak.net> References: <20101004213232.2F5A1282B90@codespeak.net> Message-ID: <4CAA4D92.9000908@gmail.com> On 04/10/10 23:32, fijal at codespeak.net wrote: > def transform(op): > from pypy.jit.metainterp.history import AbstractDescr > - # Rename CALL_PURE to CALL. > + # Rename CALL_PURE and CALL_INVARIANT to CALL. > # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. > - if op.getopnum() == rop.CALL_PURE: > + if (op.getopnum() == rop.CALL_PURE or > + op.getopnum() == rop.CALL_LOOPINVARIANT): > op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, > op.getdescr()) > elif op.getopnum() == rop.VIRTUAL_REF: Hi, I just wanted to tell that since the resoperation-refactoring, there is now a nice "copy_and_change" method to be used exactly for the case where you want to replace an operation with a "similar" one. So, the code above should be written like: op = op.copy_and_change(rop.CALL, args=op.getarglist()[1:]) ciao, Anto From fijall at gmail.com Tue Oct 5 08:42:54 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 5 Oct 2010 08:42:54 +0200 Subject: [pypy-dev] [pypy-svn] r77589 - pypy/trunk/pypy/jit/metainterp In-Reply-To: <4CAA4D92.9000908@gmail.com> References: <20101004213232.2F5A1282B90@codespeak.net> <4CAA4D92.9000908@gmail.com> Message-ID: On Mon, Oct 4, 2010 at 11:56 PM, Antonio Cuni wrote: > On 04/10/10 23:32, fijal at codespeak.net wrote: > >> ? def transform(op): >> ? ? ? from pypy.jit.metainterp.history import AbstractDescr >> - ? ?# Rename CALL_PURE to CALL. >> + ? ?# Rename CALL_PURE and CALL_INVARIANT to CALL. >> ? ? ? # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. >> - ? ?if op.getopnum() == rop.CALL_PURE: >> + ? ?if (op.getopnum() == rop.CALL_PURE or >> + ? ? ? ?op.getopnum() == rop.CALL_LOOPINVARIANT): >> ? ? ? ? ? op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? op.getdescr()) >> ? ? ? elif op.getopnum() == rop.VIRTUAL_REF: > > Hi, > I just wanted to tell that since the resoperation-refactoring, there is now a > nice "copy_and_change" method to be used exactly for the case where you want > to replace an operation with a "similar" one. > > So, the code above should be written like: > > ? ?op = op.copy_and_change(rop.CALL, args=op.getarglist()[1:]) ah, great! :) > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From andrewfr_ice at yahoo.com Tue Oct 5 15:51:05 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Tue, 5 Oct 2010 06:51:05 -0700 (PDT) Subject: [pypy-dev] PyPy JIT & C extensions, greenlet Message-ID: <192780.17766.qm@web120710.mail.ne1.yahoo.com> Hi Armin and folks: --- On Thu, 9/30/10, Armin Rigo wrote: > From: Armin Rigo > Subject: Re: [pypy-dev] PyPy JIT & C extensions, greenlet > To: "Andrew Francis" > Cc: pypy-dev at codespeak.net > Date: Thursday, September 30, 2010, 5:25 AM > Maybe I should expand on an idea posted on #pypy by > fijal.? He mentioned that he would like to try to support Stackless in > PyPy without using the stackless transform, just by using the > same low-level stack hacks that are done by greenlet.c and > optionally by Stackless Python.? This means that there would be two > different approaches we can consider to support Stackless in PyPy: That sounds like a neat idea! > ? ? tasklet-switching Python code ???tasklet-switching Python code > ? ? becomes a single loop in? ? ?? ? becomes N loops with residual calls > ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? to machine code? ? ? ? Maybe this is the core of my problems (and ability to help) - I have a superficial understanding of how Stackless works under the hood. Where do the N loops come from? I have read Christian's "Stackless Python" paper (http://www.python.org/workshops/2000-01/proceedings/papers/tismers/stackless.htm) and slowly but surely I do more stuff with the C Stackless code base. That said, I haven't found any particularly good papers on how the Python interpreter itself is architected. Maybe I should look at how the greenlet package works. Again, I would be happy to help but I am not quite sure where to start so at least I can ask intelligent questions in the pypy IRC. Cheers, Andrew From brian at slesinsky.org Thu Oct 7 05:33:26 2010 From: brian at slesinsky.org (Brian Slesinsky) Date: Wed, 6 Oct 2010 20:33:26 -0700 Subject: [pypy-dev] ssl cert expired / OS X download doesn't work In-Reply-To: References: Message-ID: Hi, I tried running pypy 1.3 on OS X but ran into a few issues: First, I got this error: $ pypy dyld: Library not loaded: /usr/lib/libssl.0.9.8.dylib ?Referenced from: /usr/local/pypy-1.3/bin/pypy ?Reason: image not found Trace/BPT trap It looks like my machine (running OS X 10.5.8) has libssl.0.9.7.dylib, but not libssl.0.9.8. (I suppose I could recompile from source, but don't feel like waiting an hour.) Also, the SSL certificate has expired for https://codespeak.net, which is why I didn't post there. Anyway, please keep up the good work. - Brian From santagada at gmail.com Thu Oct 7 06:33:19 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 7 Oct 2010 01:33:19 -0300 Subject: [pypy-dev] ssl cert expired / OS X download doesn't work In-Reply-To: References: Message-ID: On Thu, Oct 7, 2010 at 12:33 AM, Brian Slesinsky wrote: > Hi, I tried running pypy 1.3 on OS X but ran into a few issues: > > First, I got this error: > > $ pypy > dyld: Library not loaded: /usr/lib/libssl.0.9.8.dylib > ?Referenced from: /usr/local/pypy-1.3/bin/pypy > ?Reason: image not found > Trace/BPT trap > > It looks like my machine (running OS X 10.5.8) has libssl.0.9.7.dylib, > but not libssl.0.9.8. (I suppose I could recompile from source, but > don't feel like waiting an hour.) The binaries are for mac os x 10.6 intel 32 bits. Someone should look at compiling it using the SDK (so we can choose 10.5 or 10.6 compatibility). > Also, the SSL certificate has expired for https://codespeak.net, which > is why I didn't post there. Yes it expired 17 of august. -- Leonardo Santagada From holger at merlinux.eu Sun Oct 10 09:37:49 2010 From: holger at merlinux.eu (holger krekel) Date: Sun, 10 Oct 2010 09:37:49 +0200 Subject: [pypy-dev] issue tracker spam issues / switching Message-ID: <20101010073749.GB11924@trillke.net> Hi all, seems like the pypy-dev issue tracker got some spam through bot-registered users. Unless i am mistaken there is no easy way to add a captcha in addition to the e-mail confirmation registration and so i made the default role for new issue tracker users "anonymous". This means new users need to find an admin who assigns the role "User". Admins on the trackers are e.g. Armin, Maciej, Anto, Amaury, myself and probably others. Not a great solution, i know. It doesn't affect any of the existing users, though. it's probably a good occassion to push for switching to using new issue tracker software especially since there has been other complaints as well. best, holger From fijall at gmail.com Mon Oct 11 11:42:20 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 11 Oct 2010 11:42:20 +0200 Subject: [pypy-dev] issue tracker spam issues / switching In-Reply-To: <20101010073749.GB11924@trillke.net> References: <20101010073749.GB11924@trillke.net> Message-ID: On Sun, Oct 10, 2010 at 9:37 AM, holger krekel wrote: > Hi all, > > seems like the pypy-dev issue tracker got some spam through bot-registered > users. ?Unless i am mistaken there is no easy way to add a captcha in addition > to the e-mail confirmation registration and so i made the default role for new > issue tracker users "anonymous". ?This means new users need to find an admin who > assigns the role "User". ?Admins on the trackers are e.g. Armin, Maciej, Anto, > Amaury, myself and probably others. ?Not a great solution, i know. It doesn't > affect any of the existing users, though. > > it's probably a good occassion to push for switching to using new > issue tracker software especially since there has been other complaints > as well. Hey. Yes, I think so, especially if we can give handling of spam to someone else ;-) I did a bit of research at some point and personally I would like to give JIRA a try. It looks good, but it's not open source (it's free for open source users though). Cheers, fijal > > best, > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Sun Oct 17 10:29:58 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 17 Oct 2010 10:29:58 +0200 Subject: [pypy-dev] [pypy-svn] r77933 - pypy/branch/jitffi/pypy/jit/metainterp/test In-Reply-To: <20101014133223.A97D9282BEA@codespeak.net> References: <20101014133223.A97D9282BEA@codespeak.net> Message-ID: The exception still can't happen though. On Thu, Oct 14, 2010 at 3:32 PM, wrote: > Author: antocuni > Date: Thu Oct 14 15:32:22 2010 > New Revision: 77933 > > Modified: > ? pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py > Log: > update this test too > > > Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py > ============================================================================== > --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py (original) > +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py Thu Oct 14 15:32:22 2010 > @@ -77,7 +77,9 @@ > ? ? ? ? call(0, p2, ? ? ? ? ? ? descr=libffi_prepare) > ? ? ? ? call(0, p2, i0, ? ? ? ? descr=libffi_push_arg) > ? ? ? ? call(0, p2, f1, ? ? ? ? descr=libffi_push_arg) > - ? ? ? ?i3 = call(0, p2, 12345, descr=libffi_call) > + ? ? ? ?i3 = call_may_force(0, p2, 12345, descr=libffi_call) > + ? ? ? ?guard_not_forced() [] > + ? ? ? ?guard_no_exception() [] > ? ? ? ? jump(i3, f1, p2) > ? ? ? ? """ > ? ? ? ? expected = ops > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From andrewfr_ice at yahoo.com Mon Oct 25 05:12:21 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Sun, 24 Oct 2010 20:12:21 -0700 (PDT) Subject: [pypy-dev] PyPy JIT & C extensions Message-ID: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Hi Folks: > --- On Thu, 9/30/10, Armin Rigo > wrote: > > > From: Armin Rigo > > Subject: Re: [pypy-dev] PyPy JIT & C extensions, > greenlet > > To: "Andrew Francis" > > Cc: pypy-dev at codespeak.net > > Date: Thursday, September 30, 2010, 5:25 AM > > > > Maybe I should expand on an idea posted on #pypy by > > fijal. He mentioned that he would like to try to > support Stackless in PyPy without using the stackless transform, >>just by using the same low-level stack hacks that are done by greenlet.c > and optionally by Stackless Python. This means that there would be two > > different approaches we can consider to support > Stackless in PyPy: A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, install the greenlet package, run it and see what happens? After all, greenlet is a C extension module. Hopefully this would provide a base line for work to be required? I am trying this but I am having a few hiccups getting setup.py to do the right thing. Since I am a newbie, I would like to know what is wrong with this line of reasoning? Cheers, Andrew From william.leslie.ttg at gmail.com Mon Oct 25 05:36:01 2010 From: william.leslie.ttg at gmail.com (William Leslie) Date: Mon, 25 Oct 2010 14:36:01 +1100 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: <365555.71903.qm@web120710.mail.ne1.yahoo.com> References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: On 25 October 2010 14:12, Andrew Francis wrote: > A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, install the greenlet package, run it and see what happens? After all, greenlet is a C extension module. ?Hopefully this would provide a base line for work to be required? I am trying this but I am having a few hiccups getting setup.py to do the right thing. Since I am a newbie, I would like to know what is wrong with this line of reasoning? PyPy's internal representation of frames differs from that of CPython. How does greenlet determine how much of the stack to copy? -- William Leslie From arigo at tunes.org Mon Oct 25 14:08:20 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Oct 2010 14:08:20 +0200 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: <365555.71903.qm@web120710.mail.ne1.yahoo.com> References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: Hi, On Mon, Oct 25, 2010 at 5:12 AM, Andrew Francis wrote: > A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, > install the greenlet package, run it and see what happens? You will just get strange segfaults and no way to understand from outside where they come from. I can give some more details about them, but I guess it would not be too helpful. In summary, C extension modules have a chance to work with PyPy as long as they don't play strange tricks. A bient?t, Armin. From fijall at gmail.com Mon Oct 25 16:37:28 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Oct 2010 16:37:28 +0200 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: On Mon, Oct 25, 2010 at 2:08 PM, Armin Rigo wrote: > Hi, > > On Mon, Oct 25, 2010 at 5:12 AM, Andrew Francis wrote: >> A few days ago, I floated this idea in the #pypy IRC channel. Why can't we take a JITed pypy, >> install the greenlet package, run it and see what happens? > > You will just get strange segfaults and no way to understand from > outside where they come from. ?I can give some more details about > them, but I guess it would not be too helpful. ?In summary, C > extension modules have a chance to work with PyPy as long as they > don't play strange tricks. > Hey. I had a private email exchange with Andrew and I think what he wants to achieve is a working greenlet module for PyPy. Is a current C version a good start, or something else would be a good start? Cheers, fijal > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Mon Oct 25 16:50:50 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Oct 2010 16:50:50 +0200 Subject: [pypy-dev] [pypy-svn] r78267 - pypy/branch/arm-backend/pypy/jit/backend/arm/tools In-Reply-To: <20101025144559.82B2A282BE3@codespeak.net> References: <20101025144559.82B2A282BE3@codespeak.net> Message-ID: Hey. Can we make this directory 'tool' instead of 'tools' like every other tool directory? Cheers, fijal On Mon, Oct 25, 2010 at 4:45 PM, wrote: > Author: david > Date: Mon Oct 25 16:45:58 2010 > New Revision: 78267 > > Added: > ? pypy/branch/arm-backend/pypy/jit/backend/arm/tools/ > ? pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py ? (contents, props changed) > Log: > Wrapper for calling objdump on dumped memory > > Added: pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py > ============================================================================== > --- (empty file) > +++ pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py ? ? ? Mon Oct 25 16:45:58 2010 > @@ -0,0 +1,5 @@ > +#!/usr/bin/env python > +import os > +import sys > + > +os.system('objdump -D --architecture=arm --target=binary %s' % sys.argv[1]) > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From arigo at tunes.org Mon Oct 25 16:59:50 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Oct 2010 16:59:50 +0200 Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: References: <365555.71903.qm@web120710.mail.ne1.yahoo.com> Message-ID: Hi, On Mon, Oct 25, 2010 at 4:37 PM, Maciej Fijalkowski wrote: > I had a private email exchange with Andrew and I think what he wants > to achieve is a working greenlet module for PyPy. Is a current C > version a good start, or something else would be a good start? Sorry, I guess I should repeat here the argument I made on IRC. It would not work, because the GC will not find its roots. With --gcrootfinder=shadowstack, I think it will get confused by the switches -- maybe I am wrong and it works anyway; but then I know for sure that with --gcrootfinder=asmgcc it will definitely not work at all. (For reference, the JIT requires --gcrootfinder=asmgcc.) Armin From andrewfr_ice at yahoo.com Mon Oct 25 19:36:26 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Mon, 25 Oct 2010 10:36:26 -0700 (PDT) Subject: [pypy-dev] PyPy JIT & C extensions In-Reply-To: Message-ID: <733520.43651.qm@web120703.mail.ne1.yahoo.com> Hi Folks: --- On Mon, 10/25/10, Maciej Fijalkowski wrote: > From: Maciej Fijalkowski > Subject: Re: [pypy-dev] PyPy JIT & C extensions > To: "Armin Rigo" > Cc: "Andrew Francis" , pypy-dev at codespeak.net > Date: Monday, October 25, 2010, 7:37 AM > On Mon, Oct 25, 2010 at 2:08 PM, > Armin Rigo > wrote: > > Hi, > I had a private email exchange with Andrew and I think what > he wants to achieve is a working greenlet module for PyPy. Is a > current C version a good start, or something else would be a good > start? Yes this is exactly what I wish to achieve. I thought starting with the current greenlet module would be a good start. Maybe we will get lucky and not need to re-invent the wheel. I would learn the terminology (i.e., shadowstacks), see how differently the stacks are laid out, get familiar with the code base and debugging techniques. Already I see interesting questions being asked. Cheers, Andrew From tismer at stackless.com Mon Nov 1 18:02:58 2010 From: tismer at stackless.com (Christian Tismer) Date: Mon, 01 Nov 2010 18:02:58 +0100 Subject: [pypy-dev] =?iso-8859-1?q?Next_sprint=3A_D=FCsseldorf=2C_end_of_O?= =?iso-8859-1?q?ctober?= In-Reply-To: References: Message-ID: <4CCEF2C2.7040508@stackless.com> On 10/1/10 11:31 AM, Armin Rigo wrote: > Hi all! > > D?sseldorf PyPy sprint October 25th-31st 2010 > ===================================================== > ... Hi Armin, this became a trap for me. The PyPy sprints were always listed in the sprint list as well. But this time it was not, and I used the entry from 2009 without recognizing because of my reduced sight. I even had answered to that, with no reaction, obviously. That means the sprint is over, and I was about to start my travel to D?sseldorf on Friday. :-( Good that I talked to Stephan Diehl today, who made me check again. I really wanted to come and get a deeper insight into PyPy's status. Feeling pretty disappointed. ciao -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From hakan at debian.org Mon Nov 1 22:13:07 2010 From: hakan at debian.org (Hakan Ardo) Date: Mon, 1 Nov 2010 22:13:07 +0100 Subject: [pypy-dev] pypy_g_action_dispatcher Message-ID: Hi, after 1016420 iterations of the loop below, pypy_g_action_dispatcher() is called for the first time. After that it is called every 100 iterations or so. Is this intentional? import sys for i in range(10000000): if i%20 == 0: sys.stderr.write('%d\n'%i) -- H?kan Ard? From arigo at tunes.org Wed Nov 3 13:32:11 2010 From: arigo at tunes.org (Armin Rigo) Date: Wed, 3 Nov 2010 13:32:11 +0100 Subject: [pypy-dev] pypy_g_action_dispatcher In-Reply-To: References: Message-ID: Hi Hakan, On Mon, Nov 1, 2010 at 10:13 PM, Hakan Ardo wrote: > after 1016420 iterations of the loop below, pypy_g_action_dispatcher() > is called for the first time. After that it is called every 100 > iterations or so. Is this intentional? No, strange. Armin From cfbolz at gmx.de Wed Nov 3 13:38:47 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 03 Nov 2010 13:38:47 +0100 Subject: [pypy-dev] =?utf-8?q?Next_sprint=3A_D=C3=BCsseldorf=2C_end_of_Oct?= =?utf-8?q?ober?= In-Reply-To: <4CCEF2C2.7040508@stackless.com> References: <4CCEF2C2.7040508@stackless.com> Message-ID: <4CD157D7.1060701@gmx.de> On 11/01/2010 06:02 PM, Christian Tismer wrote: > On 10/1/10 11:31 AM, Armin Rigo wrote: >> Hi all! >> >> D?sseldorf PyPy sprint October 25th-31st 2010 >> ===================================================== >> > ... > > Hi Armin, > > this became a trap for me. > The PyPy sprints were always listed in the sprint list as well. > But this time it was not, and I used the entry from 2009 > without recognizing because of my reduced sight. > I even had answered to that, with no reaction, obviously. > > That means the sprint is over, and I was about to start my travel > to D?sseldorf on Friday. :-( > > Good that I talked to Stephan Diehl today, who made me check again. > > I really wanted to come and get a deeper insight into PyPy's status. > Feeling pretty disappointed. Hi Christian, If you just want to find out what is currently going on with PyPy, you could just come at any time to D?sseldorf for some days. Armin and me are here all the time, and Anto is here until end of November. Just send us a mail (off-list, I guess) to discuss timing. Cheers, Carl Friedrich From tismer at stackless.com Wed Nov 3 22:33:13 2010 From: tismer at stackless.com (Christian Tismer) Date: Wed, 03 Nov 2010 22:33:13 +0100 Subject: [pypy-dev] =?utf-8?q?Next_sprint=3A_D=C3=BCsseldorf=2C_end_of_Oct?= =?utf-8?q?ober?= In-Reply-To: <4CD157D7.1060701@gmx.de> References: <4CCEF2C2.7040508@stackless.com> <4CD157D7.1060701@gmx.de> Message-ID: <4CD1D519.6060806@stackless.com> On 11/3/10 1:38 PM, Carl Friedrich Bolz wrote: > On 11/01/2010 06:02 PM, Christian Tismer wrote: >> On 10/1/10 11:31 AM, Armin Rigo wrote: >>> Hi all! >>> >>> D?sseldorf PyPy sprint October 25th-31st 2010 >>> ===================================================== >>> >> ... >> >> Hi Armin, >> >> this became a trap for me. >> The PyPy sprints were always listed in the sprint list as well. >> But this time it was not, and I used the entry from 2009 >> without recognizing because of my reduced sight. >> I even had answered to that, with no reaction, obviously. >> >> That means the sprint is over, and I was about to start my travel >> to D?sseldorf on Friday. :-( >> >> Good that I talked to Stephan Diehl today, who made me check again. >> >> I really wanted to come and get a deeper insight into PyPy's status. >> Feeling pretty disappointed. > Hi Christian, > > If you just want to find out what is currently going on with PyPy, you > could just come at any time to D?sseldorf for some days. Armin and me > are here all the time, and Anto is here until end of November. Just send > us a mail (off-list, I guess) to discuss timing. > Oh, that's a nice offer, thanks very much. I will check my schedule and contact you in private. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From anto.cuni at gmail.com Fri Nov 5 17:30:02 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 05 Nov 2010 17:30:02 +0100 Subject: [pypy-dev] Migration to mercurial Message-ID: <4CD4310A.5030201@gmail.com> Hi all, as some of you might have noticed by watching the IRC channel, we are finally migrating to mercurial. The actual conversion of the repository will happen at some point during next week (or maybe the week after). One issue that we have to care about is how to convert the author names: in the current svn repository each author is simply indicated by its codespeak username, while with mercurial this is no longer possible as there is no longer a "centralized" server where everyone has an account. In mercurial an author is just a string, but the standard way is put both the real name and a valid email address to uniquely identify the author. This is exploited by websites like e.g. bitbucket to link the author of the commit to the bitbucket user. For doing the conversion, we will use the following usermap: http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/usermap.txt By default, we map each username to its real name, as found in the /etc/passwd file on codespeak.net. Because of the reasons I explained above, I strongly recommend everyone to put its email address there, so that it will be there in the final mercurial repository. If for any reason you don't want your real name to be in the mercurial repository, please kill the corresponding line in the file. Note that it won't be possible to change the info after the conversion has been made. ciao, Anto From holger at merlinux.eu Wed Nov 10 16:57:30 2010 From: holger at merlinux.eu (holger krekel) Date: Wed, 10 Nov 2010 16:57:30 +0100 Subject: [pypy-dev] PyPy has a non-profit foundation now Message-ID: <20101110155729.GD1009@trillke.net> Hi all, i'd like to point you to your recent joining to the Software Freedom Conservancy (SFC). We decided it's better to bite the bullet and go GPL in exchange for working with a good non-profit foundation and getting some donations. Any objections? Nah, just kidding. But what is true is that the PyPy project joined the Software Freedom Conservancy, see our posts and theirs: http://morepypy.blogspot.com/2010/11/speeding-up-pypy-by-donations.html http://sfconservancy.org/news/2010/nov/10/pypy-joins/ Many thanks to Bradley M. Kuhn from the SFC for helping to implement a smooth process so far. And now let's see if some people find some spare money so we can convene on how to spend it for PyPy improvements. Oh, and if you happen to know how to personally approach the right guys in companies to send some donations the PyPy way, no need to be shy. cheers, holger From fijall at gmail.com Wed Nov 10 21:53:38 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 10 Nov 2010 22:53:38 +0200 Subject: [pypy-dev] Memory consumption of pypy-c translate.py Message-ID: Hello. I did a bit of measurments using various techniques and pypy-c-64bit-jit and the outcome is that after a bit of annotation, where memory consumption is 1200M, we get: * 31M of mmaped code blocks for assembler * about 150M I can account for that is jit resume data * about 150M of the rest I can account for. Where is everything else? Can GC trash some memory (like a lot). Here is output of dump.py -r http://codespeak.net/~fijal/jit64.out From arigo at tunes.org Fri Nov 12 09:19:32 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 12 Nov 2010 09:19:32 +0100 Subject: [pypy-dev] Release 1.4 Message-ID: Hi all, We are preparing for the next release of PyPy. A branch was already created http://codespeak.net/svn/pypy/release/1.4.x, twice, but it may get thrown away and re-created again. I have 3 things that I would like to fix before the release: 1. Mac OS/X 64. We have there an issue with asmgcc not finding the gc roots. 2. Try to understand if there is a leak in pypy-c-jit or not. The issue is that it consumes quite a lot of memory and fijal could not, so far, really figure out where it comes from. 3. Optionally, depending on 2, implement freeing old JIT-generated assembler and supporting objects. For 3, I have in mind the following lightweight plan: put an "age" counter on each TreeLoop object, reset it to zero whenever we run the corresponding code, and create an extension of the asmgcc root finder that also resets to zero the counter of loops found in the stack. After every GC collection we increment the age counter. This should ensure that the age of any live loop never goes past 1; on the other hand, loops whose age reach some limit (like 20 or so) have not been run for a long time, and are freed. If we do it right it would free all supporting data structures, and with a __del__ somewhere we could also free the assembler itself (which may only account for 25% of ram usage). A bient?t, Armin. From phyo.arkarlwin at gmail.com Fri Nov 12 21:18:22 2010 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Sat, 13 Nov 2010 02:48:22 +0630 Subject: [pypy-dev] Release 1.4 In-Reply-To: References: Message-ID: Is python-mysql gonna work in 1.4 ? i cant wait to run my webserver via pypy-jit On Fri, Nov 12, 2010 at 2:49 PM, Armin Rigo wrote: > Hi all, > > We are preparing for the next release of PyPy. > > A branch was already created > http://codespeak.net/svn/pypy/release/1.4.x, twice, but it may get > thrown away and re-created again. I have 3 things that I would like > to fix before the release: > > 1. Mac OS/X 64. We have there an issue with asmgcc not finding the gc > roots. > > 2. Try to understand if there is a leak in pypy-c-jit or not. The > issue is that it consumes quite a lot of memory and fijal could not, > so far, really figure out where it comes from. > > 3. Optionally, depending on 2, implement freeing old JIT-generated > assembler and supporting objects. > > For 3, I have in mind the following lightweight plan: put an "age" > counter on each TreeLoop object, reset it to zero whenever we run the > corresponding code, and create an extension of the asmgcc root finder > that also resets to zero the counter of loops found in the stack. > After every GC collection we increment the age counter. This should > ensure that the age of any live loop never goes past 1; on the other > hand, loops whose age reach some limit (like 20 or so) have not been > run for a long time, and are freed. If we do it right it would free > all supporting data structures, and with a __del__ somewhere we could > also free the assembler itself (which may only account for 25% of ram > usage). > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101113/4fd68032/attachment.htm From andrewfr_ice at yahoo.com Sat Nov 13 15:32:06 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Sat, 13 Nov 2010 06:32:06 -0800 (PST) Subject: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) Message-ID: <627056.25013.qm@web120714.mail.ne1.yahoo.com> Hi: I finally got my hands on a machine big enough to compile pypy-c. I want to try pypy-c with the greenlet package. As I mentioned in previous posts, I want to see how the greenlet package blows up pypy-c and hope that it is a case of simply altering greenlets enough so it will work. I did a test run with -Ojit. It still took about 90 minutes to compile on a Mac with 4G of ram. That said, what would be the right switches to create a version can be placed in a debugger? Where does the C code go - or would this code even be useful. Any suggestions to speed up the process. Cheers, Andrew From santagada at gmail.com Sat Nov 13 15:39:11 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 13 Nov 2010 12:39:11 -0200 Subject: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) In-Reply-To: <627056.25013.qm@web120714.mail.ne1.yahoo.com> References: <627056.25013.qm@web120714.mail.ne1.yahoo.com> Message-ID: On Sat, Nov 13, 2010 at 12:32 PM, Andrew Francis wrote: > Hi: > > I finally got my hands on a machine big enough to compile pypy-c. I want to try pypy-c with the greenlet package. As I mentioned in previous posts, I want to see how the greenlet package blows up pypy-c and hope that it is a case of simply altering greenlets enough so it will work. > > I did a test run with -Ojit. It still took about 90 minutes to compile on a Mac with 4G of ram. That said, what would be the right switches to create a version can be placed in a debugger? Where does the C code go - or would this code even be useful. Any suggestions to speed up the process. One way to speed up is to use nightly builds (http://buildbot.pypy.org/nightly/trunk/) :) I still can't see how the greenlet for cpython could possibly work on pypy. If you are talking about http://codespeak.net/pypy/dist/pypy/doc/stackless.html#greenlets then I think the "only" problem is that it doesn't work with the jit right? -- Leonardo Santagada From andrewfr_ice at yahoo.com Sat Nov 13 18:40:39 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Sat, 13 Nov 2010 09:40:39 -0800 (PST) Subject: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) In-Reply-To: Message-ID: <639562.33217.qm@web120715.mail.ne1.yahoo.com> Hi Leonardo: --- On Sat, 11/13/10, Leonardo Santagada wrote: > From: Leonardo Santagada > Subject: Re: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) > To: "Andrew Francis" > Cc: pypy-dev at codespeak.net > Date: Saturday, November 13, 2010, 6:39 AM > I still can't see how the greenlet for cpython could > possibly work on pypy. If you are talking about > http://codespeak.net/pypy/dist/pypy/doc/stackless.html#greenlets > then I think the "only" problem is that it doesn't work with the > jit right? As I have stated in earlier posts, I am a newbie. However I want to help with Maciej Fijalkowski's suggestions about integrating Stackless with the JIT. I figured a way to start is to see what it would take to get greenlet style hard switching done. Since greenlet is a C extension, let's be naive, compile it and pypy-c, run it in a debugger and see exactly how it fails. Besides reading the documentation inside the greenlet.c source, I can take a similar approach with Standard C - that is run it in a debugger and see what happens. This way I get a feel for the issues and get familiar with the tools. Maybe trying to work off the greenlet source will prove to be too difficult. Right now, I have a computer that will compile pypy-c. However it takes a long time. I just want to make sure that everything is in place to carry out my plan. Also it would be nice if folks can spot problems in the approach. I know I need to learn more about stuff like shadow stacks. Also I have started to read documentation on the bytecode interpreter. Cheers, Andrew From santagada at gmail.com Sat Nov 13 19:38:30 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 13 Nov 2010 16:38:30 -0200 Subject: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) In-Reply-To: <639562.33217.qm@web120715.mail.ne1.yahoo.com> References: <639562.33217.qm@web120715.mail.ne1.yahoo.com> Message-ID: On Sat, Nov 13, 2010 at 3:40 PM, Andrew Francis wrote: > --- On Sat, 11/13/10, Leonardo Santagada wrote: >> I still can't see how the greenlet for cpython could >> possibly work on pypy. If you are talking about >> http://codespeak.net/pypy/dist/pypy/doc/stackless.html#greenlets >> then I think the "only" problem is that it doesn't work with the >> jit right? > > As I have stated in earlier posts, I am a newbie. However I want to help with Maciej Fijalkowski's suggestions about integrating Stackless with the JIT. I figured a way to start is to see what it would take to get greenlet style hard switching done. Since greenlet is a C extension, let's be naive, compile it and pypy-c, run it in a debugger and see exactly how it fails. Besides reading the documentation inside the greenlet.c source, I can take a similar approach with Standard C - that is run it in a debugger and see what happens. This way I get a feel for the issues and get familiar with the tools. > > Maybe trying to work off the greenlet source will prove to be too difficult. I will repeat what armin said, but I think you should reread what he said in the previous thread, the gist is it will probably not work and give you strange crashes and there is probably no way to fix them. I think what fijal meant is to use the ideas in greenlets.c and map them to pypy, but not directly trying to run the extension on top of pypy. Here is a list of what is crazy about the idea: - greenlets is a c module, but the stuff it does with the stack is completely outside the C API which is what pypy supports from cpython. - It will (probably) break the gc. It will not find the roots (address on the stack to memory on the heap) from functions that are taken out of the stack - The layout of the stack is probably not the same also and that will also break. > Right now, I have a computer that will compile pypy-c. However it takes a long time. I just want to make sure that everything is in place to carry out my plan. Also it would be nice if folks can spot problems in the approach. I know I need to learn more about stuff like shadow stacks. Also I have started to read documentation on the bytecode interpreter. My idea to make things faster for you: - Forget about running greenlets inside pypy-c after running and making sure that armin prediction is true (that it will just blowup), don't try to fix it. I also think that greenlets are more or less independent from the bytecode interpreter, then don't waste your time with that either. - To translate faster and using less memory turn your python to 32bit and use --no-allworkingmodules. - Look at how greenlets.c work and talk to fijal and the other on irc after/during you understand how it unwinds the stack. Write tests for it (or reuse the ones that probably exist for stackless and greenlets). - Make a solid plan on how to do it. - Have fun :) ps: I hope I was clear (and also correct) -- Leonardo Santagada From andrewfr_ice at yahoo.com Sun Nov 14 16:11:32 2010 From: andrewfr_ice at yahoo.com (Andrew Francis) Date: Sun, 14 Nov 2010 07:11:32 -0800 (PST) Subject: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) Message-ID: <64943.47473.qm@web120703.mail.ne1.yahoo.com> Hello Leonardo: --- On Sat, 11/13/10, Leonardo Santagada wrote: > From: Leonardo Santagada > Subject: Re: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) > To: "Andrew Francis" > Cc: pypy-dev at codespeak.net > Date: Saturday, November 13, 2010, 10:38 AM > On Sat, Nov 13, 2010 at 3:40 PM, > Andrew Francis > wrote: > > --- On Sat, 11/13/10, Leonardo Santagada > wrote: > I think what fijal meant is to use the ideas in greenlets.c > and map them to pypy, but not directly trying to run the extension > on top of pypy. I understood what Fijal said. I thought why not try modifying it? > Here is a list of what is crazy about the idea: > - greenlets is a c module, but the stuff it does with the > stack is completely outside the C API which is what pypy supports > from cpython. Leonardo, I don't want to sound argumentative but conceptually does this differ from what greenlets does with Standard Python? If I am not mistaken, greenlets (like Stackless Python) at some point need to do some very machine dependent low-level stuff to move stack frames to and from a heap. That is not in the C API either. > - It will (probably) break the gc. It will not find the > roots (address on the stack to memory on the heap) from functions that >are taken out of the stack - The layout of the stack is probably not the >same also and that will also break. I acknowledge all of these concerns. However what I would like to figure out is if the greenlet.c code is sufficiently modular so these parts can be isolated. I figure it would be good strategy to try to keep most of greenlet's high level Maybe folks can take a quick look at greenlet.c and off-hand state what would change. I would figure that low level routines (i.e., like g_save() and slp_restore_state(). In this fashion we have more facts to see if this approach should be nipped in the bud. > - To translate faster and using less memory turn your > python to 32bit and use --no-allworkingmodules. Great! I'll try this. > - Look at how greenlets.c work and talk I am doing this as we speak. I'll make IRC appearances once I have really has something new to add. > ps: I hope I was clear (and also correct) I thank you for your input. Right now, I need more information and to learn more. Cheers, Andrew From arigo at tunes.org Mon Nov 15 13:07:57 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 15 Nov 2010 13:07:57 +0100 Subject: [pypy-dev] Advice on Compiling Pypy-c for debugging (Stackless support) In-Reply-To: <64943.47473.qm@web120703.mail.ne1.yahoo.com> References: <64943.47473.qm@web120703.mail.ne1.yahoo.com> Message-ID: Hi Andrew, On Sun, Nov 14, 2010 at 4:11 PM, Andrew Francis wrote: >> I think what fijal meant is to use the ideas in greenlets.c >> and map them to pypy, but not directly trying to run the extension >> on top of pypy. > > I understood what Fijal said. I thought why not try modifying it? Because it will not work at all. This approach is completely doomed. And that's probably the last time you'll get an answer to the same question, so I've tried to be as clear as I possibly can :-) Armin From drsalists at gmail.com Tue Nov 16 01:25:46 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 15 Nov 2010 16:25:46 -0800 Subject: [pypy-dev] gdbm Message-ID: I've put together a gdbm wrapper module using ctypes, and tested against pypy with the included test_gdbm. Is there interest in adding it to pypy? It's at http://stromberg.dnsalias.org/svn/gdbm-ctypes/trunk/ if anyone wants to look it over. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101115/3c341da2/attachment.htm From amauryfa at gmail.com Tue Nov 16 02:22:56 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 16 Nov 2010 02:22:56 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: Message-ID: Hi, 2010/11/16 Dan Stromberg : > I've put together a gdbm wrapper module using ctypes, and tested against > pypy with the included test_gdbm.? Is there interest in adding it to pypy? > > It's at http://stromberg.dnsalias.org/svn/gdbm-ctypes/trunk/ if anyone wants > to look it over. I don't know exactly what gdbm is, but it looks very similar to dbm.py already in pypy: http://codespeak.net/svn/pypy/trunk/lib_pypy/dbm.py -- Amaury Forgeot d'Arc From drsalists at gmail.com Tue Nov 16 03:17:06 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 15 Nov 2010 18:17:06 -0800 Subject: [pypy-dev] gdbm In-Reply-To: References: Message-ID: Yes, the dbm module in pypy is basically like the bsddb module in cpython. cpython includes modules for bsddb, gdbm, and more. I tend to prefer gdbm over bsddb, because I've seen bsddb databases get corrupt too many times - EG, when a filesystem overflows. bsddb might be a little faster though; I've never compared their performance. On Mon, Nov 15, 2010 at 5:22 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2010/11/16 Dan Stromberg : > > I've put together a gdbm wrapper module using ctypes, and tested against > > pypy with the included test_gdbm. Is there interest in adding it to > pypy? > > > > It's at http://stromberg.dnsalias.org/svn/gdbm-ctypes/trunk/ if anyone > wants > > to look it over. > > I don't know exactly what gdbm is, but it looks very similar to dbm.py > already in pypy: > http://codespeak.net/svn/pypy/trunk/lib_pypy/dbm.py > > -- > Amaury Forgeot d'Arc > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101115/42491b09/attachment.htm From lac at openend.se Tue Nov 16 03:22:53 2010 From: lac at openend.se (Laura Creighton) Date: Tue, 16 Nov 2010 03:22:53 +0100 Subject: [pypy-dev] gdbm In-Reply-To: Message from "Amaury Forgeot d'Arc" of "Tue, 16 Nov 2010 02:22:56 +0100." References: Message-ID: <201011160222.oAG2Mrfv004177@theraft.openend.se> In a message of Tue, 16 Nov 2010 02:22:56 +0100, "Amaury Forgeot d'Arc" writes: >Hi, > >2010/11/16 Dan Stromberg : >> I've put together a gdbm wrapper module using ctypes, and tested agains >t >> pypy with the included test_gdbm.? Is there interest in adding it to pyp >y? >> >> It's at http://stromberg.dnsalias.org/svn/gdbm-ctypes/trunk/ if anyone >wants >> to look it over. > >I don't know exactly what gdbm is, but it looks very similar to dbm.py >already in pypy: >http://codespeak.net/svn/pypy/trunk/lib_pypy/dbm.py Both gdbm and dbm are part of the python standard library, so I'd guess we'd better have both of them. http://docs.python.org/library/gdbm.html Laura >Amaury Forgeot d'Arc From drsalists at gmail.com Tue Nov 16 04:30:11 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 15 Nov 2010 19:30:11 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <201011160222.oAG2Mrfv004177@theraft.openend.se> References: <201011160222.oAG2Mrfv004177@theraft.openend.se> Message-ID: On Mon, Nov 15, 2010 at 6:22 PM, Laura Creighton wrote: > In a message of Tue, 16 Nov 2010 02:22:56 +0100, "Amaury Forgeot d'Arc" > writes: > >Hi, > > > >2010/11/16 Dan Stromberg : > >> I've put together a gdbm wrapper module using ctypes, and tested agains > >t > >> pypy with the included test_gdbm. Is there interest in adding it to pyp > >y? > >> > >> It's at http://stromberg.dnsalias.org/svn/gdbm-ctypes/trunk/ if anyone > >wants > >> to look it over. > > > >I don't know exactly what gdbm is, but it looks very similar to dbm.py > >already in pypy: > >http://codespeak.net/svn/pypy/trunk/lib_pypy/dbm.py > > Both gdbm and dbm are part of the python standard library, so I'd guess > we'd > better have both of them. > > http://docs.python.org/library/gdbm.html > > Laura > > Sounds good to me :) BTW, it might cause confusion down the road to call something that is basically like cpython's bsddb (Berkeley DB) by the name "dbm" in pypy's library. In the cpython standard library, "dbm" is an interface to ndbm databases. These all provide the same dictionary-like interface to Python programs, but have somewhat different API's to C, and pretty different, incompatible on-disk representations. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101115/0cd72bfa/attachment.htm From anto.cuni at gmail.com Tue Nov 16 13:17:34 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 16 Nov 2010 13:17:34 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: Message-ID: <4CE2765E.2020507@gmail.com> Hi Dan, first: thanks for your help :-) On 16/11/10 03:17, Dan Stromberg wrote: > > Yes, the dbm module in pypy is basically like the bsddb module in cpython. > > cpython includes modules for bsddb, gdbm, and more. > > I tend to prefer gdbm over bsddb, because I've seen bsddb databases get > corrupt too many times - EG, when a filesystem overflows. bsddb might be a > little faster though; I've never compared their performance. So, if I understand correctly you are saying that we should rename our dbm.py to bsdb.py, and write a new dbm.py which can use either bsdb or gdbm? Sounds fine, do you feel like implementing it? :-) Moreover, I also agree with amaury that your code is very similar to the one in the current dbm.py, so we should maybe try to refactor things to share common parts between the twos. ciao, Anto From drsalists at gmail.com Tue Nov 16 17:27:03 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 16 Nov 2010 08:27:03 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <4CE2765E.2020507@gmail.com> References: <4CE2765E.2020507@gmail.com> Message-ID: On Tue, Nov 16, 2010 at 4:17 AM, Antonio Cuni wrote: > > On 16/11/10 03:17, Dan Stromberg wrote: > >> >> Yes, the dbm module in pypy is basically like the bsddb module in cpython. >> >> cpython includes modules for bsddb, gdbm, and more. >> >> I tend to prefer gdbm over bsddb, because I've seen bsddb databases get >> corrupt too many times - EG, when a filesystem overflows. bsddb might be >> a >> little faster though; I've never compared their performance. >> > > > So, if I understand correctly you are saying that we should rename our > dbm.py to bsdb.py, and write a new dbm.py which can use either bsdb or gdbm? > I think it's anydbm that can use whatever among dbm, bsddb, gdbm and dumbdbm, as it sees fit. TTBoMK, it's not until python 3.x that dbm becomes a sort of unifying module hierarchy. > Sounds fine, do you feel like implementing it? :-) > > Moreover, I also agree with amaury that your code is very similar to the > one in the current dbm.py, so we should maybe try to refactor things to > share common parts between the twos. > I'd be happy to. Would sharing based on inheritance or a more functional approach be preferred? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101116/197ef989/attachment.htm From arigo at tunes.org Tue Nov 16 17:47:35 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 16 Nov 2010 17:47:35 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> Message-ID: Hi, On Tue, Nov 16, 2010 at 5:27 PM, Dan Stromberg wrote: >> So, if I understand correctly you are saying that we should rename our >> dbm.py to bsdb.py, and write a new dbm.py which can use either bsdb or gdbm? > > I think it's anydbm that can use whatever among dbm, bsddb, gdbm and > dumbdbm, as it sees fit.? TTBoMK, it's not until python 3.x that dbm becomes > a sort of unifying module hierarchy. Yes, in Python 2.x, dbm.py is very specifically an interface to the Unix dbm library (see e.g. man dbm_open). At the level of C, the gdbm interface is some kind of extension of that. It's not related to bsddb, which has a very different interface. > Would sharing based on inheritance or a more functional approach be > preferred? I think either is fine (or even not sharing at all if it turns out to be too much of a mess in practice). A bient?t, Armin. From exarkun at twistedmatrix.com Tue Nov 16 17:38:25 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 16 Nov 2010 16:38:25 -0000 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> Message-ID: <20101116163825.2040.1287738136.divmod.xquotient.932@localhost.localdomain> On 04:27 pm, drsalists at gmail.com wrote: >On Tue, Nov 16, 2010 at 4:17 AM, Antonio Cuni >wrote: >>Sounds fine, do you feel like implementing it? :-) >> >>Moreover, I also agree with amaury that your code is very similar to >>the >>one in the current dbm.py, so we should maybe try to refactor things >>to >>share common parts between the twos. >I'd be happy to. > >Would sharing based on inheritance or a more functional approach be >preferred? No, avoid inheritance where possible. Composition is preferred. As for "functional", I don't really know what approach you're describing with that word. Jean-Paul From arigo at tunes.org Tue Nov 16 19:12:57 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 16 Nov 2010 19:12:57 +0100 Subject: [pypy-dev] Release 1.4 In-Reply-To: References: Message-ID: Hi Phyo, On Fri, Nov 12, 2010 at 9:18 PM, Phyo Arkar wrote: > Is python-mysql gonna work in 1.4 ? i cant wait to run my webserver via > pypy-jit Yes, you can use the CPython C extension module. It's a feature that already worked more or less since PyPy 1.3. See explanations in http://morepypy.blogspot.com/2010/06/pypy-13-released.html . Note that _mysql.c first needs a small patch in order to work, which is here: http://codespeak.net/svn/pypy/trunk/pypy/module/cpyext/patches/mysqldb.patch . Armin From drsalists at gmail.com Wed Nov 17 00:12:19 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 16 Nov 2010 15:12:19 -0800 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> Message-ID: On Tue, Nov 16, 2010 at 8:47 AM, Armin Rigo wrote: > Hi, > > On Tue, Nov 16, 2010 at 5:27 PM, Dan Stromberg > wrote: > >> So, if I understand correctly you are saying that we should rename our > >> dbm.py to bsdb.py, and write a new dbm.py which can use either bsdb or > gdbm? > > > > I think it's anydbm that can use whatever among dbm, bsddb, gdbm and > > dumbdbm, as it sees fit. TTBoMK, it's not until python 3.x that dbm > becomes > > a sort of unifying module hierarchy. > > Yes, in Python 2.x, dbm.py is very specifically an interface to the > Unix dbm library (see e.g. man dbm_open). At the level of C, the gdbm > interface is some kind of extension of that. It's not related to > bsddb, which has a very different interface. > Well, there's related, and then there's related. bsddb provides a bunch of operations, including a basic hash facility. The API is not intended to be the same as gdbm or ndbm (gdbm has a native interface and an ndbm compatability interface), but the concept is similar between the 3 for one part of bsddb - the part the cpython uses. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101116/92a17838/attachment-0001.htm From drsalists at gmail.com Wed Nov 17 00:24:58 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 16 Nov 2010 15:24:58 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <20101116163825.2040.1287738136.divmod.xquotient.932@localhost.localdomain> References: <4CE2765E.2020507@gmail.com> <20101116163825.2040.1287738136.divmod.xquotient.932@localhost.localdomain> Message-ID: On Tue, Nov 16, 2010 at 8:38 AM, wrote: > On 04:27 pm, drsalists at gmail.com wrote: > >On Tue, Nov 16, 2010 at 4:17 AM, Antonio Cuni > >wrote: > >>Sounds fine, do you feel like implementing it? :-) > >> > >>Moreover, I also agree with amaury that your code is very similar to > >>the > >>one in the current dbm.py, so we should maybe try to refactor things > >>to > >>share common parts between the twos. > >I'd be happy to. > > > >Would sharing based on inheritance or a more functional approach be > >preferred? > > No, avoid inheritance where possible. Composition is preferred. > Due to performance? Or "flat is better than nested" - as more of a philosophical issue? I'm not arguing for inheritance, just wanting to understand the reasoning behind it. > As for "functional", I don't really know what approach you're describing > with that word. > I mean something like what you'd do in Lisp, ML or Haskell (not that I know any of those that well). I suppose in this case it would mean functools.partial() as a decorator, or perhaps a wrapper around functools.partial() used as a decorator. Then again, maybe functools.partial() imposes its own performance penalty? That's part of why I'm wondering if avoiding inheritance is a performance-related goal. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101116/75bf0635/attachment.htm From p.giarrusso at gmail.com Wed Nov 17 03:35:22 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Wed, 17 Nov 2010 03:35:22 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> <20101116163825.2040.1287738136.divmod.xquotient.932@localhost.localdomain> Message-ID: On Wed, Nov 17, 2010 at 00:24, Dan Stromberg wrote: > > On Tue, Nov 16, 2010 at 8:38 AM, wrote: >> >> On 04:27 pm, drsalists at gmail.com wrote: >> >On Tue, Nov 16, 2010 at 4:17 AM, Antonio Cuni >> >wrote: >> >>Sounds fine, do you feel like implementing it? :-) >> >> >> >>Moreover, I also agree with amaury that your code is very similar to >> >>the >> >>one in the current dbm.py, so we should maybe try to refactor things >> >>to >> >>share common parts between the twos. >> >I'd be happy to. >> > >> >Would sharing based on inheritance or a more functional approach be >> >preferred? >> No, avoid inheritance where possible. ?Composition is preferred. > Due to performance?? Or "flat is better than nested" - as more of a > philosophical issue?? I'm not arguing for inheritance, just wanting to > understand the reasoning behind it. I answer on this because it is a general topic. Preferring composition over inheritance is a general design principle, not intended to improve performance; of course, there are cases where inheritance is good, so it needs to be further qualified; but in a very simple way, inheritance is often used where it is not appropriate. One of the reasons which come to my mind is that inheritance implies closer coupling between subclasses and superclasses. If you look for "composition vs inheritance" on Google, you will find quite a few articles. One which looks good is this one: http://www.artima.com/designtechniques/compoinh4.html (you can also browse the whole article, I link to page 4 because it's more interesting). That article also points out that you might get some performance overhead with composition instead of inheritance, which is the opposite of what you said, and that makes some sense; however, in many cases the overhead can be removed by an optimizer through inlining, and that's one reason not to worry. The other reason not to worry is that the overhead is anyway minor and thus design issues are more important - doing otherwise would be premature optimization. If you have a lot of experience with hard data which taught you that you should avoid something upfront, and that data is still valid, then you can avoid something upfront - and that's not the case here, because when you change your target implementation, you'll need to recheck about how it optimizes, and that takes long. >> As for "functional", I don't really know what approach you're describing >> with that word. > I mean something like what you'd do in Lisp, ML or Haskell (not that I know > any of those that well).? I suppose in this case it would mean > functools.partial() as a decorator, or perhaps a wrapper around > functools.partial() used as a decorator. > Then again, maybe functools.partial() imposes its own performance penalty? > That's part of why I'm wondering if avoiding inheritance is a > performance-related goal. I can't comment in detail on this, because I just had a look at the code - but performance shouldn't be the main goal. Anyway, it looks like most differences are about calling clearerr or not, and about the module name in strings; probably one could abstract that away through some other technique. Bye! -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From phyo.arkarlwin at gmail.com Wed Nov 17 06:06:17 2010 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Wed, 17 Nov 2010 11:36:17 +0630 Subject: [pypy-dev] Release 1.4 In-Reply-To: References: Message-ID: Oh thats cool ! THanks a lot i will try! On 11/17/10, Armin Rigo wrote: > Hi Phyo, > > On Fri, Nov 12, 2010 at 9:18 PM, Phyo Arkar > wrote: >> Is python-mysql gonna work in 1.4 ? i cant wait to run my webserver via >> pypy-jit > > Yes, you can use the CPython C extension module. It's a feature that > already worked more or less since PyPy 1.3. See explanations in > http://morepypy.blogspot.com/2010/06/pypy-13-released.html . Note > that _mysql.c first needs a small patch in order to work, which is > here: > http://codespeak.net/svn/pypy/trunk/pypy/module/cpyext/patches/mysqldb.patch > . > > > Armin > From fijall at gmail.com Wed Nov 17 07:32:54 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 17 Nov 2010 08:32:54 +0200 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> <20101116163825.2040.1287738136.divmod.xquotient.932@localhost.localdomain> Message-ID: On Wed, Nov 17, 2010 at 1:24 AM, Dan Stromberg wrote: > > On Tue, Nov 16, 2010 at 8:38 AM, wrote: >> >> On 04:27 pm, drsalists at gmail.com wrote: >> >On Tue, Nov 16, 2010 at 4:17 AM, Antonio Cuni >> >wrote: >> >>Sounds fine, do you feel like implementing it? :-) >> >> >> >>Moreover, I also agree with amaury that your code is very similar to >> >>the >> >>one in the current dbm.py, so we should maybe try to refactor things >> >>to >> >>share common parts between the twos. >> >I'd be happy to. >> > >> >Would sharing based on inheritance or a more functional approach be >> >preferred? >> >> No, avoid inheritance where possible. ?Composition is preferred. > > Due to performance?? Or "flat is better than nested" - as more of a > philosophical issue?? I'm not arguing for inheritance, just wanting to > understand the reasoning behind it. I think one of the points is that those are originally not related in CPython. And someone somewhere *will* use this and complain. People do crazy stuff with Python. > >> >> As for "functional", I don't really know what approach you're describing >> with that word. > > I mean something like what you'd do in Lisp, ML or Haskell (not that I know > any of those that well).? I suppose in this case it would mean > functools.partial() as a decorator, or perhaps a wrapper around > functools.partial() used as a decorator. > > Then again, maybe functools.partial() imposes its own performance penalty? > That's part of why I'm wondering if avoiding inheritance is a > performance-related goal. > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Wed Nov 17 13:53:27 2010 From: arigo at tunes.org (Armin Rigo) Date: Wed, 17 Nov 2010 13:53:27 +0100 Subject: [pypy-dev] Memory consumption of pypy-c translate.py In-Reply-To: References: Message-ID: Hi, (Answering here in addition to on IRC, for reference.) On Wed, Nov 10, 2010 at 9:53 PM, Maciej Fijalkowski wrote: > * 31M of mmaped code blocks for assembler > * about 150M I can account for that is jit resume data > * about 150M of the rest I can account for. > > Where is everything else? Can GC trash some memory (like a lot). Everything else is memory that has been freed, but not returned to the OS. (You get the same effect with CPython 2.5.) That's a bit unexpected, because we call the C-level free() on it, but it is still not returned to the OS; it seems to be caused by only a few mallocs() that occur from time to time and stay alive for long (e.g. a few 4KB pages needed by the GC for its various AddressStacks). I verified it in a custom C program that does the same calls to malloc() and free(). If at the end we force free() on the remaining few 4KB blocks, then the total memory use suddenly drops from (in this example) more than 200MB to almost 0. This seems to be the main difference between the way pypy-c and CPython 2.6 call the C-level malloc() and free(). Unsure how to fix it. (If the same program then asks for more memory again, it will reuse this memory -- the libc malloc() at least ensures this, of course.) A bient?t, Armin. From renesd at gmail.com Wed Nov 17 14:47:22 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Wed, 17 Nov 2010 13:47:22 +0000 Subject: [pypy-dev] Memory consumption of pypy-c translate.py In-Reply-To: References: Message-ID: Check out the brk syscall. malloc is allocating more memory to itself. You really need a custom malloc to change this behaviour. cheers, On Wed, Nov 17, 2010 at 12:53 PM, Armin Rigo wrote: > Hi, > > (Answering here in addition to on IRC, for reference.) > > On Wed, Nov 10, 2010 at 9:53 PM, Maciej Fijalkowski wrote: >> * 31M of mmaped code blocks for assembler >> * about 150M I can account for that is jit resume data >> * about 150M of the rest I can account for. >> >> Where is everything else? Can GC trash some memory (like a lot). > > Everything else is memory that has been freed, but not returned to the > OS. ?(You get the same effect with CPython 2.5.) That's a bit > unexpected, because we call the C-level free() on it, but it is still > not returned to the OS; it seems to be caused by only a few mallocs() > that occur from time to time and stay alive for long (e.g. a few 4KB > pages needed by the GC for its various AddressStacks). ?I verified it > in a custom C program that does the same calls to malloc() and free(). > ?If at the end we force free() on the remaining few 4KB blocks, then > the total memory use suddenly drops from (in this example) more than > 200MB to almost 0. > > This seems to be the main difference between the way pypy-c and > CPython 2.6 call the C-level malloc() and free(). ?Unsure how to fix > it. ?(If the same program then asks for more memory again, it will > reuse this memory -- the libc malloc() at least ensures this, of > course.) > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From drsalists at gmail.com Thu Nov 18 01:17:09 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 17 Nov 2010 16:17:09 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <4CE2765E.2020507@gmail.com> References: <4CE2765E.2020507@gmail.com> Message-ID: On Tue, Nov 16, 2010 at 4:17 AM, Antonio Cuni wrote: > Hi Dan, > first: thanks for your help :-) > > > On 16/11/10 03:17, Dan Stromberg wrote: > >> >> Yes, the dbm module in pypy is basically like the bsddb module in cpython. >> >> cpython includes modules for bsddb, gdbm, and more. >> >> I tend to prefer gdbm over bsddb, because I've seen bsddb databases get >> corrupt too many times - EG, when a filesystem overflows. bsddb might be >> a >> little faster though; I've never compared their performance. >> > > So, if I understand correctly you are saying that we should rename our > dbm.py to bsdb.py, and write a new dbm.py which can use either bsdb or gdbm? > Sounds fine, do you feel like implementing it? :-) > > Moreover, I also agree with amaury that your code is very similar to the > one in the current dbm.py, so we should maybe try to refactor things to > share common parts between the twos. > > ciao, > Anto > Wow, CPython's Berkeley DB interface is actually quite a bit more comprehensive and complex than I'd realized. This isn't just a matter of renaming dbm.py to bsddb.py and refactoring a bit. It's more of a time commitment to something I don't use than I'd thought. Althought pypy's current dbm.py implements something similar to cpython's Berkeley DB interface, it isn't all that similar. It uses a subset of the same on-disk representations, but the API appears to be pretty different. This is based on playing around in the unit tests and bsddb module a bit. I actually suggest: 1) svn rename'ing dbm.py into some unused directory for history's sake; it implements the ndbm _interface_ (a little oddly called "dbm" in cpython - but I believe true "dbm" is "one database per program") well, but it's not really all that similar to bsddb. 2) Adding the gdbm.py module I wrote, more or less verbatim. I got into the project of merging these two things thinking that bsddb was mostly just like the gdbm module, but bsddb is actually quite a bit more involved, and is something I've pretty much stopped using due bad experiences with bsddb database corruption from both cpython and C. Given #1 and #2 above, anydbm should continue working, due to the presence of gdbm and dumbdbm. I guess I think that if someone has a need for bsddb (and it's assorted interfaces), they probably should work on that. Sound reasonable? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101117/0122a5b7/attachment-0001.htm From anto.cuni at gmail.com Thu Nov 18 16:06:43 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 18 Nov 2010 16:06:43 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> Message-ID: <4CE54103.6060603@gmail.com> On 18/11/10 01:17, Dan Stromberg wrote: Hi Dan, [cut] > Given #1 and #2 above, anydbm should continue working, due to the presence of > gdbm and dumbdbm. > > I guess I think that if someone has a need for bsddb (and it's assorted > interfaces), they probably should work on that. > > Sound reasonable? Yes. To me it sounds reasonable. I don't know if it's correct because I never used any of those modules, but it's surely reasoable :-). I think that the best would be if you get an account on codespeak and do that in a branch: this way we can follow your commits and the if everything is fine we merge it. I personally cannot create new accounts on codespeak, so you should ask someone else for this (e.g. Armin Rigo): the easiest way is if you just come on #pypy on freenode.net and talk with us "real-time". ciao, Anto From romcheg.prihod at gmail.com Fri Nov 19 20:46:22 2010 From: romcheg.prihod at gmail.com (Roman Prykhodchenko) Date: Fri, 19 Nov 2010 21:46:22 +0200 Subject: [pypy-dev] Memory error when importing optparse module Message-ID: Hi guys, I'm trying to port my scripts to pypy but when I import the optparse module I get the error: >>>> import optparse Traceback (most recent call last): File "", line 1, in File "/opt/local/share/pypy-1.3/lib-python/2.5.2/optparse.py", line 84, in from gettext import gettext File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/gettext.py", line 49, in import locale, copy, os, re, struct, sys File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/locale.py", line 30, in from _locale import * File "/opt/local/share/pypy-1.3/pypy/lib/_locale.py", line 6, in from ctypes import (Structure, POINTER, create_string_buffer, File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/ctypes/__init__.py", line 489, in _cast = PYFUNCTYPE(py_object, c_void_p, py_object, py_object)(_cast_addr) File "/opt/local/share/pypy-1.3/pypy/lib/_ctypes/function.py", line 104, in __init__ ffiargs, ffires, self._flags_) MemoryError OS: Mac OS Snow Leopard 10.6.5 I tried to sign up your bugtracker to report a bug but the registration is broken -- I got the error message when I was trying to confirm my registration. My login is prykhodchenko --- Roman Prykhodchenko From santagada at gmail.com Sat Nov 20 05:09:00 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 20 Nov 2010 02:09:00 -0200 Subject: [pypy-dev] Needs some guidance in writing the kqueue part of select Message-ID: I'm writing the kqueue part of the select module and would like some guidance in how to fix some pieces of it. The code is here: http://paste.pocoo.org/show/293812/ and the specific problems I'm having are: 1) There is no UINTPTR_T in rffi, but INTPTR_T = SSIZE_T so can UINTPTR_T be an alias to SIZE_T? 2) How do I declare that an argument is const in rffi? But if anyone wants to take a look (or maybe more tests http://paste.pocoo.org/show/293811/) please do. I just tested kqueue, but now I need to test the kevent call, I'm planing on using sockect_pair or a simple socket from rlib (if there is no socket_pair there), -- Leonardo Santagada From fijall at gmail.com Sat Nov 20 09:09:22 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 20 Nov 2010 10:09:22 +0200 Subject: [pypy-dev] Memory error when importing optparse module In-Reply-To: References: Message-ID: Hey. Unfortunately we don't have a full-time OS X developer and it makes it harder for this platform to be fully spported. Is your python 32bit or 64bit? How did you build that executable? On Fri, Nov 19, 2010 at 9:46 PM, Roman Prykhodchenko wrote: > Hi guys, > > > I'm trying to port my scripts to pypy but when I import the optparse module I get the error: > >>>>> import optparse > Traceback (most recent call last): > ?File "", line 1, in > ?File "/opt/local/share/pypy-1.3/lib-python/2.5.2/optparse.py", line 84, in > ? from gettext import gettext > ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/gettext.py", line 49, in > ? import locale, copy, os, re, struct, sys > ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/locale.py", line 30, in > ? from _locale import * > ?File "/opt/local/share/pypy-1.3/pypy/lib/_locale.py", line 6, in > ? from ctypes import (Structure, POINTER, create_string_buffer, > ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/ctypes/__init__.py", line 489, in > ? _cast = PYFUNCTYPE(py_object, c_void_p, py_object, py_object)(_cast_addr) > ?File "/opt/local/share/pypy-1.3/pypy/lib/_ctypes/function.py", line 104, in __init__ > ? ffiargs, ffires, self._flags_) > MemoryError > > > OS: Mac OS Snow Leopard 10.6.5 > > I tried to sign up your bugtracker to report a bug but the registration is broken ?-- I got ?the error message when I was trying to confirm my registration. > My login is prykhodchenko > > > --- > Roman Prykhodchenko > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Sat Nov 20 09:10:21 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 20 Nov 2010 10:10:21 +0200 Subject: [pypy-dev] Needs some guidance in writing the kqueue part of select In-Reply-To: References: Message-ID: (part response) > > 2) How do I declare that an argument is const in rffi? > usually you don't From lac at openend.se Sat Nov 20 16:17:11 2010 From: lac at openend.se (Laura Creighton) Date: Sat, 20 Nov 2010 16:17:11 +0100 Subject: [pypy-dev] Conference in Japan Message-ID: <201011201517.oAKFHBJI022329@theraft.openend.se> ------- Forwarded Message From: To: Subject: [ANN] Call for Papers: Workshop Python for High Performance and Scientific Computing Thread-Topic: [ANN] Call for Papers: Workshop Python for High Performance and Scientific Computing Thread-Index: AcuIOaLheS34scrKT/anKkKcPOGEAA== Date: Fri, 19 Nov 2010 22:32:23 +0000 Message-ID: <64886ABF09B1924997D70320E02F7DDB03F1AE at DLREXMBX01.intra.dlr.de> Workshop Python for High Performance and Scientific Computing June 1-3, 2011, Tsukuba, Japan part of The International Conference on Computational Science (ICCS 2011) http://www.dlr.de/sc/iccs2011 =============== Call for Papers =============== Introduction - ------------ Python is an accepted high-level programming language with a growing community in academia and industry. Beside its original use as a scripting language for web applications, today Python is a general-purpose language adopted by many scientific applications like CFD, bio molecular simulation, AI, scientific visualization etc. More and more industrial domains are turning towards it as well, such as robotics, semiconductor manufacturing, automotive solutions, telecommunication, computer graphics, and games. In all fields, the use of Python for scientific, high performance parallel, and distributed computing, as well as general scripted automation is increasing. Moreover, Python is well-suited for education in scientific computing. The workshop aims at bringing together researchers and practitioners from industry and academia using Python for all aspects of high performance and scientific computing. The goal is to present Python-based scientific applications and libraries, to discuss general topics regarding the use of Python (such as language design and performance issues), and to share experience using Python in scientific computing education. Topics - ------ * Python-based scientific applications and libraries * High performance computing * Parallel Python-based programming languages * Scientific visualization * Scientific computing education * Python performance and language issues * Problem solving environments with Python * Performance analysis tools for Python application Papers/Submission - ----------------- We invite you to submit a paper of up to 10 pages formatted according to the rules of Procedia Computer Science (http://www.elsevier.com/locate/inca/719435) via the ICCS 2011 conference submission system (http://www.iccs-meeting.org/iccs2011/papers/upload.php) selecting our workshop Python for High Performance and Scientific Computing in the drop-down menu. Important Dates - --------------- Full paper submission: January 10, 2011 Notification of acceptance: February 20, 2011 Camera-ready papers: March 7, 2011 Program Committee - ----------------- * Achim Basermann, German Aerospace Center, Germany * David Beazley, Dabeaz, LLC, USA * William E. Hart, Sandia National Laboratories, USA * Konrad Hinsen, Centre de Biophysique Mol?culaire, CNRS Orl?ans, France * Andreas Kl?ckner, New York University, USA * Maurice Ling, Singapore Polytechnic, Singapore * Stuart Mitchell, The University of Auckland, New Zealand * Mike M?ller, Python Academy, Germany * Travis Oliphant, Enthought, Inc., USA * Fernando P?rez, University of California, Berkeley, USA * Massimo Di Pierro, DePaul University, USA * Marc Poinot, ONERA, France * William Scullin, Argonne National Laboratory, USA * Ga?l Varoquaux, INRIA, France Workshop Organizers - ------------------- * Chair: Andreas Schreiber, German Aerospace Center (DLR), Germany E-Mail: iccs2011 at dlr.de * Co-Chair: Guy K. Kloss, Auckland University of Technology, New Zealand - -- Andreas Schreiber Head Distributed Systems and Component Software DLR (German Aerospace Center), Simulation and Software Technology Linder Hoehe, 51147 Cologne, Germany Tel: +49 (0)2203/601-2485 http://www.dlr.de/sc Fax: +49 (0)2203/601-12485 MSN: mailto:Andreas.Schreiber at dlr.de Mobile: +49 (0)173 5231013 ICQ# 324185855 DLR on Twitter: http://twitter.com/DLR_de - -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/ ------- End of Forwarded Message From romcheg.prihod at gmail.com Sat Nov 20 20:20:32 2010 From: romcheg.prihod at gmail.com (Roman Prykhodchenko) Date: Sat, 20 Nov 2010 21:20:32 +0200 Subject: [pypy-dev] Memory error when importing optparse module In-Reply-To: References: Message-ID: I built the executable using macports (port install pypy). My python is 64 bit. I can help you to find the bug if you help me to help you. - Roman On Nov 20, 2010, at 10:09 AM, Maciej Fijalkowski wrote: > Hey. > > Unfortunately we don't have a full-time OS X developer and it makes it > harder for this platform to be fully spported. Is your python 32bit or > 64bit? How did you build that executable? > > On Fri, Nov 19, 2010 at 9:46 PM, Roman Prykhodchenko > wrote: >> Hi guys, >> >> >> I'm trying to port my scripts to pypy but when I import the optparse module I get the error: >> >>>>>> import optparse >> Traceback (most recent call last): >> File "", line 1, in >> File "/opt/local/share/pypy-1.3/lib-python/2.5.2/optparse.py", line 84, in >> from gettext import gettext >> File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/gettext.py", line 49, in >> import locale, copy, os, re, struct, sys >> File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/locale.py", line 30, in >> from _locale import * >> File "/opt/local/share/pypy-1.3/pypy/lib/_locale.py", line 6, in >> from ctypes import (Structure, POINTER, create_string_buffer, >> File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/ctypes/__init__.py", line 489, in >> _cast = PYFUNCTYPE(py_object, c_void_p, py_object, py_object)(_cast_addr) >> File "/opt/local/share/pypy-1.3/pypy/lib/_ctypes/function.py", line 104, in __init__ >> ffiargs, ffires, self._flags_) >> MemoryError >> >> >> OS: Mac OS Snow Leopard 10.6.5 >> >> I tried to sign up your bugtracker to report a bug but the registration is broken -- I got the error message when I was trying to confirm my registration. >> My login is prykhodchenko >> >> >> --- >> Roman Prykhodchenko >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> From santagada at gmail.com Sun Nov 21 00:54:32 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 20 Nov 2010 21:54:32 -0200 Subject: [pypy-dev] Needs some guidance in writing the kqueue part of select In-Reply-To: References: Message-ID: here is the latest version of the code http://paste.pocoo.org/show/294079/ Things apear to be working except that when I run the tests http://paste.pocoo.org/show/294087/ the kevent one fails. Supposedly it should give me back the event setting kevent.data to how much you can read on the file and it is returning 0 to me. Maybe I did something dumb while declaring kevent_array or in kevent_struct, but I'm kind of lost. Python 3.1 show the expected behavior (of course, using its much more high level interface). Any tips? -- Leonardo Santagada From p.giarrusso at gmail.com Sun Nov 21 10:26:04 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Sun, 21 Nov 2010 10:26:04 +0100 Subject: [pypy-dev] Needs some guidance in writing the kqueue part of select In-Reply-To: References: Message-ID: On Sun, Nov 21, 2010 at 00:54, Leonardo Santagada wrote: > here is the latest version of the code http://paste.pocoo.org/show/294079/ > > Things apear to be working except that when I run the tests > http://paste.pocoo.org/show/294087/ the kevent one fails. Supposedly > it should give me back the event setting kevent.data to how much you > can read on the file and it is returning 0 to me. > > Maybe I did something dumb while declaring kevent_array or in > kevent_struct, but I'm kind of lost. Python 3.1 show the expected > behavior (of course, using its much more high level interface). > Any tips? Have you tried using strace on the two programs, to see and compare the actual invocations? Variations of strace -e kqueue,kevent,file,network might help selecting relevant data (-e syscall1,syscall2,...,syscallN shows only those syscalls; file and network are syscall categories). If you are not on Linux but on other *BSDes (and it seems you must be, since Linux has other syscalls), you might still use strace or equivalents: http://www.cyberciti.biz/tips/ktrace-freebsd-macosx-tool-howto.html Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From fijall at gmail.com Sun Nov 21 13:20:59 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 21 Nov 2010 14:20:59 +0200 Subject: [pypy-dev] Memory error when importing optparse module In-Reply-To: References: Message-ID: I guess the first thing would be to build pypy svn yourself instead of using 1.3 and see if it helps On Sat, Nov 20, 2010 at 9:20 PM, Roman Prykhodchenko wrote: > I built the executable using macports (port install pypy). > My python is 64 bit. > > I can help you to find the bug if you help me to help you. > > > - Roman > > On Nov 20, 2010, at 10:09 AM, Maciej Fijalkowski wrote: > >> Hey. >> >> Unfortunately we don't have a full-time OS X developer and it makes it >> harder for this platform to be fully spported. Is your python 32bit or >> 64bit? How did you build that executable? >> >> On Fri, Nov 19, 2010 at 9:46 PM, Roman Prykhodchenko >> wrote: >>> Hi guys, >>> >>> >>> I'm trying to port my scripts to pypy but when I import the optparse module I get the error: >>> >>>>>>> import optparse >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ?File "/opt/local/share/pypy-1.3/lib-python/2.5.2/optparse.py", line 84, in >>> ? from gettext import gettext >>> ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/gettext.py", line 49, in >>> ? import locale, copy, os, re, struct, sys >>> ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/locale.py", line 30, in >>> ? from _locale import * >>> ?File "/opt/local/share/pypy-1.3/pypy/lib/_locale.py", line 6, in >>> ? from ctypes import (Structure, POINTER, create_string_buffer, >>> ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/ctypes/__init__.py", line 489, in >>> ? _cast = PYFUNCTYPE(py_object, c_void_p, py_object, py_object)(_cast_addr) >>> ?File "/opt/local/share/pypy-1.3/pypy/lib/_ctypes/function.py", line 104, in __init__ >>> ? ffiargs, ffires, self._flags_) >>> MemoryError >>> >>> >>> OS: Mac OS Snow Leopard 10.6.5 >>> >>> I tried to sign up your bugtracker to report a bug but the registration is broken ?-- I got ?the error message when I was trying to confirm my registration. >>> My login is prykhodchenko >>> >>> >>> --- >>> Roman Prykhodchenko >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> > > From romcheg.prihod at gmail.com Sun Nov 21 13:31:59 2010 From: romcheg.prihod at gmail.com (Roman Prykhodchenko) Date: Sun, 21 Nov 2010 14:31:59 +0200 Subject: [pypy-dev] Memory error when importing optparse module In-Reply-To: References: Message-ID: <25355127-9CE4-4DD8-B902-DCDD3B9CFB82@gmail.com> I have already built it myself. I got lots of errors and warnings during the build process and as a result I have pypy-c that doesn't work. May be that was caused by --opt=jit on my x64 platform. Will try to go with --opt=2 a bit later and report you the result. - Roman On Nov 21, 2010, at 2:20 PM, Maciej Fijalkowski wrote: > I guess the first thing would be to build pypy svn yourself instead of > using 1.3 and see if it helps > > On Sat, Nov 20, 2010 at 9:20 PM, Roman Prykhodchenko > wrote: >> I built the executable using macports (port install pypy). >> My python is 64 bit. >> >> I can help you to find the bug if you help me to help you. >> >> >> - Roman >> >> On Nov 20, 2010, at 10:09 AM, Maciej Fijalkowski wrote: >> >>> Hey. >>> >>> Unfortunately we don't have a full-time OS X developer and it makes it >>> harder for this platform to be fully spported. Is your python 32bit or >>> 64bit? How did you build that executable? >>> >>> On Fri, Nov 19, 2010 at 9:46 PM, Roman Prykhodchenko >>> wrote: >>>> Hi guys, >>>> >>>> >>>> I'm trying to port my scripts to pypy but when I import the optparse module I get the error: >>>> >>>>>>>> import optparse >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> File "/opt/local/share/pypy-1.3/lib-python/2.5.2/optparse.py", line 84, in >>>> from gettext import gettext >>>> File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/gettext.py", line 49, in >>>> import locale, copy, os, re, struct, sys >>>> File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/locale.py", line 30, in >>>> from _locale import * >>>> File "/opt/local/share/pypy-1.3/pypy/lib/_locale.py", line 6, in >>>> from ctypes import (Structure, POINTER, create_string_buffer, >>>> File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/ctypes/__init__.py", line 489, in >>>> _cast = PYFUNCTYPE(py_object, c_void_p, py_object, py_object)(_cast_addr) >>>> File "/opt/local/share/pypy-1.3/pypy/lib/_ctypes/function.py", line 104, in __init__ >>>> ffiargs, ffires, self._flags_) >>>> MemoryError >>>> >>>> >>>> OS: Mac OS Snow Leopard 10.6.5 >>>> >>>> I tried to sign up your bugtracker to report a bug but the registration is broken -- I got the error message when I was trying to confirm my registration. >>>> My login is prykhodchenko >>>> >>>> >>>> --- >>>> Roman Prykhodchenko >>>> _______________________________________________ >>>> pypy-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/pypy-dev >>>> >> >> From fijall at gmail.com Sun Nov 21 13:38:33 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 21 Nov 2010 14:38:33 +0200 Subject: [pypy-dev] Memory error when importing optparse module In-Reply-To: <25355127-9CE4-4DD8-B902-DCDD3B9CFB82@gmail.com> References: <25355127-9CE4-4DD8-B902-DCDD3B9CFB82@gmail.com> Message-ID: PyPy is known not to work on mac os x 64bit. Blame a combination of crazy licensing schemes (we can't run a buildbot in a virtual machine nor we can debug it) with awkward outdated tools available on platform. Or blame us :) Generally someone has to fix this, but as I said before we don't have an OS X developer. Precise problem is some difference in generated assembler by gcc or platform needs. On Sun, Nov 21, 2010 at 2:31 PM, Roman Prykhodchenko wrote: > I have already built it myself. I got lots of errors and warnings during the build process and as a result I have pypy-c that doesn't work. > May be that was caused by --opt=jit on my x64 platform. > Will try to go with --opt=2 a bit later and report you the result. > > > - Roman > > > > > On Nov 21, 2010, at 2:20 PM, Maciej Fijalkowski wrote: > >> I guess the first thing would be to build pypy svn yourself instead of >> using 1.3 and see if it helps >> >> On Sat, Nov 20, 2010 at 9:20 PM, Roman Prykhodchenko >> wrote: >>> I built the executable using macports (port install pypy). >>> My python is 64 bit. >>> >>> I can help you to find the bug if you help me to help you. >>> >>> >>> - Roman >>> >>> On Nov 20, 2010, at 10:09 AM, Maciej Fijalkowski wrote: >>> >>>> Hey. >>>> >>>> Unfortunately we don't have a full-time OS X developer and it makes it >>>> harder for this platform to be fully spported. Is your python 32bit or >>>> 64bit? How did you build that executable? >>>> >>>> On Fri, Nov 19, 2010 at 9:46 PM, Roman Prykhodchenko >>>> wrote: >>>>> Hi guys, >>>>> >>>>> >>>>> I'm trying to port my scripts to pypy but when I import the optparse module I get the error: >>>>> >>>>>>>>> import optparse >>>>> Traceback (most recent call last): >>>>> ?File "", line 1, in >>>>> ?File "/opt/local/share/pypy-1.3/lib-python/2.5.2/optparse.py", line 84, in >>>>> ? from gettext import gettext >>>>> ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/gettext.py", line 49, in >>>>> ? import locale, copy, os, re, struct, sys >>>>> ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/locale.py", line 30, in >>>>> ? from _locale import * >>>>> ?File "/opt/local/share/pypy-1.3/pypy/lib/_locale.py", line 6, in >>>>> ? from ctypes import (Structure, POINTER, create_string_buffer, >>>>> ?File "/opt/local/share/pypy-1.3/lib-python/modified-2.5.2/ctypes/__init__.py", line 489, in >>>>> ? _cast = PYFUNCTYPE(py_object, c_void_p, py_object, py_object)(_cast_addr) >>>>> ?File "/opt/local/share/pypy-1.3/pypy/lib/_ctypes/function.py", line 104, in __init__ >>>>> ? ffiargs, ffires, self._flags_) >>>>> MemoryError >>>>> >>>>> >>>>> OS: Mac OS Snow Leopard 10.6.5 >>>>> >>>>> I tried to sign up your bugtracker to report a bug but the registration is broken ?-- I got ?the error message when I was trying to confirm my registration. >>>>> My login is prykhodchenko >>>>> >>>>> >>>>> --- >>>>> Roman Prykhodchenko >>>>> _______________________________________________ >>>>> pypy-dev at codespeak.net >>>>> http://codespeak.net/mailman/listinfo/pypy-dev >>>>> >>> >>> > > From anto.cuni at gmail.com Tue Nov 23 08:31:36 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 23 Nov 2010 08:31:36 +0100 Subject: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config In-Reply-To: <20101123071858.17721282BD6@codespeak.net> References: <20101123071858.17721282BD6@codespeak.net> Message-ID: <4CEB6DD8.5080504@gmail.com> On 23/11/10 08:18, fijal at codespeak.net wrote: > Author: fijal > Date: Tue Nov 23 08:18:57 2010 > New Revision: 79382 > > Modified: > pypy/trunk/pypy/config/translationoption.py > Log: > Disable jit debugging by default > > > Modified: pypy/trunk/pypy/config/translationoption.py > ============================================================================== > --- pypy/trunk/pypy/config/translationoption.py (original) > +++ pypy/trunk/pypy/config/translationoption.py Tue Nov 23 08:18:57 2010 > @@ -115,7 +115,7 @@ > default="auto", cmdline="--jit-backend"), > ChoiceOption("jit_debug", "the amount of debugging dumps made by the JIT", > ["off", "profile", "steps", "detailed"], > - default="profile", # XXX for now > + default="off", > cmdline="--jit-debug"), > ChoiceOption("jit_profiler", "integrate profiler support into the JIT", > ["off", "oprofile"], Not sure it's a good idea. It's useful to see the statistics at the end of the run. I know that you can turn it on explicitly, but it's easy to forget. What about having an env variable that automatically turns jit debug on? ciao, Anto From arigo at tunes.org Tue Nov 23 17:52:18 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 23 Nov 2010 17:52:18 +0100 Subject: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config In-Reply-To: <4CEB6DD8.5080504@gmail.com> References: <20101123071858.17721282BD6@codespeak.net> <4CEB6DD8.5080504@gmail.com> Message-ID: Hi Anto, On Tue, Nov 23, 2010 at 8:31 AM, Antonio Cuni wrote: >> Disable jit debugging by default > > Not sure it's a good idea. It's useful to see the statistics at the end of the > run. ?I know that you can turn it on explicitly, but it's easy to forget. > > What about having an env variable that automatically turns jit debug on? I guess it should just be another log category, like "jit-summary". Then you can get the effect you want by setting (in the shell) PYPYLOG to "jit-summary:-". A bient?t, Armin. From fijall at gmail.com Tue Nov 23 21:31:11 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 23 Nov 2010 22:31:11 +0200 Subject: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config In-Reply-To: References: <20101123071858.17721282BD6@codespeak.net> <4CEB6DD8.5080504@gmail.com> Message-ID: On Tue, Nov 23, 2010 at 6:52 PM, Armin Rigo wrote:> Hi Anto, > > On Tue, Nov 23, 2010 at 8:31 AM, Antonio Cuni wrote: >>> Disable jit debugging by default >> >> Not sure it's a good idea. It's useful to see the statistics at the end of the >> run. ?I know that you can turn it on explicitly, but it's easy to forget. >> >> What about having an env variable that automatically turns jit debug on? > > I guess it should just be another log category, like "jit-summary". > Then you can get the effect you want by setting (in the shell) PYPYLOG > to "jit-summary:-". > This or an env variable. I find it really disruptive for everyday usage. Like py.test -k test_ displaying jit log is not exactly what I want. You can turn it on, put an alias or log it somewhere. From wlavrijsen at lbl.gov Tue Nov 23 21:48:05 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Tue, 23 Nov 2010 12:48:05 -0800 (PST) Subject: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' Message-ID: Hi, I've been banging my head on the error below. I've removed the references to lltype.direct_ptradd in my code that I'm aware of, but that didn't do the trick. Any ideas, or perhaps an implementation of bhimpl_direct_ptradd? Thanks, Wim [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "../../../translator/goal/translate.py", line 290, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/translator/driver.py", line 817, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/translator/driver.py", line 293, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/translator/driver.py", line 403, in task_pyjitpl_lltype [translation:ERROR] backend_name=self.config.translation.jit_backend, inline=True) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/jit/metainterp/warmspot.py", line 47, in apply_jit [translation:ERROR] **kwds) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/jit/metainterp/warmspot.py", line 188, in __init__ [translation:ERROR] self.metainterp_sd.finish_setup(self.codewriter, optimizer=optimizer) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/jit/metainterp/pyjitpl.py", line 1268, in finish_setup [translation:ERROR] self.blackholeinterpbuilder = BlackholeInterpBuilder(codewriter, self) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/jit/metainterp/blackhole.py", line 45, in __init__ [translation:ERROR] self.setup_insns(asm.insns) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/jit/metainterp/blackhole.py", line 67, in setup_insns [translation:ERROR] all_funcs.append(self._get_method(name, argcodes)) [translation:ERROR] File "/home/wlav/pydev/pypy-reflex/pypy/jit/metainterp/blackhole.py", line 223, in _get_method [translation:ERROR] unboundmethod = getattr(BlackholeInterpreter, 'bhimpl_' + name).im_func [translation:ERROR] AttributeError: type object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From drsalists at gmail.com Wed Nov 24 01:13:16 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 23 Nov 2010 16:13:16 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <4CE54103.6060603@gmail.com> References: <4CE2765E.2020507@gmail.com> <4CE54103.6060603@gmail.com> Message-ID: On Thu, Nov 18, 2010 at 7:06 AM, Antonio Cuni wrote: > On 18/11/10 01:17, Dan Stromberg wrote: > > Hi Dan, > > [cut] > > Given #1 and #2 above, anydbm should continue working, due to the presence >> of >> gdbm and dumbdbm. >> >> I guess I think that if someone has a need for bsddb (and it's assorted >> interfaces), they probably should work on that. >> >> Sound reasonable? >> > > Yes. To me it sounds reasonable. I don't know if it's correct because I > never used any of those modules, but it's surely reasoable :-). > > I think that the best would be if you get an account on codespeak and do > that in a branch: this way we can follow your commits and the if everything > is fine we merge it. > > I personally cannot create new accounts on codespeak, so you should ask > someone else for this (e.g. Armin Rigo): the easiest way is if you just come > on #pypy on freenode.net and talk with us "real-time". > > ciao, > Anto > I've created a branch at http://codespeak.net/svn/pypy/branch/gdbm and made the proposed changes in it. I took the liberty of creating an "attic" directory to put dbm.py in. Please let me know if there are further changes required related to this. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101123/9a986426/attachment.htm From exarkun at twistedmatrix.com Wed Nov 24 01:19:42 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 24 Nov 2010 00:19:42 -0000 Subject: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config In-Reply-To: References: <20101123071858.17721282BD6@codespeak.net> <4CEB6DD8.5080504@gmail.com> Message-ID: <20101124001942.2109.1155721242.divmod.xquotient.188@localhost.localdomain> On 23 Nov, 08:31 pm, fijall at gmail.com wrote: >On Tue, Nov 23, 2010 at 6:52 PM, Armin Rigo wrote:> >Hi Anto, >> >>On Tue, Nov 23, 2010 at 8:31 AM, Antonio Cuni >>wrote: >>>>Disable jit debugging by default >>> >>>Not sure it's a good idea. It's useful to see the statistics at the >>>end of the >>>run. ?I know that you can turn it on explicitly, but it's easy to >>>forget. >>> >>>What about having an env variable that automatically turns jit debug >>>on? >> >>I guess it should just be another log category, like "jit-summary". >>Then you can get the effect you want by setting (in the shell) PYPYLOG >>to "jit-summary:-". > >This or an env variable. I find it really disruptive for everyday >usage. Like py.test -k test_ displaying jit log is not exactly >what I want. You can turn it on, put an alias or log it somewhere. It also breaks Twisted unit tests, which spawn sys.executable as a child and then inspect output in various ways. It would be nice to be able to support the jit-summary-by-default behavior, but that probably involves adding something like a `sys.executablev` list that contains the executable and extra arguments to pass to the executable, so that the summary can *really* be turned off. Jean-Paul From cfbolz at gmx.de Wed Nov 24 09:41:12 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 24 Nov 2010 09:41:12 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> <4CE54103.6060603@gmail.com> Message-ID: <4CECCFA8.80708@gmx.de> On 11/24/2010 01:13 AM, Dan Stromberg wrote: > I've created a branch at http://codespeak.net/svn/pypy/branch/gdbm and > made the proposed changes in it. > > I took the liberty of creating an "attic" directory to put dbm.py in. Any reason not to just delete dbm.py? This attic stuff is annoying, and if necessary you can still get the file from svn history. Cheers, Carl Friedrich From cfbolz at gmx.de Wed Nov 24 10:33:48 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 24 Nov 2010 10:33:48 +0100 Subject: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' In-Reply-To: References: Message-ID: <4CECDBFC.60706@gmx.de> On 11/23/2010 09:48 PM, wlavrijsen at lbl.gov wrote: > I've been banging my head on the error below. I've removed the references > to lltype.direct_ptradd in my code that I'm aware of, but that didn't do > the trick. I told the JIT not to look into TypeConverter._get_fieldptr, see revision 79444. Afterwards it seemed to work. > Any ideas, or perhaps an implementation of bhimpl_direct_ptradd? Ideally support should be added to the JIT, yes (Armin? :-) ). But for now the above solution should work too. Cheers, Carl Friedrich From arigo at tunes.org Wed Nov 24 13:39:18 2010 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 Nov 2010 13:39:18 +0100 Subject: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config In-Reply-To: <20101124001942.2109.1155721242.divmod.xquotient.188@localhost.localdomain> References: <20101123071858.17721282BD6@codespeak.net> <4CEB6DD8.5080504@gmail.com> <20101124001942.2109.1155721242.divmod.xquotient.188@localhost.localdomain> Message-ID: Hi, On Wed, Nov 24, 2010 at 1:19 AM, wrote: >>> I guess it should just be another log category, like "jit-summary". >>> Then you can get the effect you want by setting (in the shell) PYPYLOG >>> to "jit-summary:-". >> >> This or an env variable. The PYPYLOG solution I described *is* an environment variable. What do you mean? A bient?t, Armin. From fijall at gmail.com Wed Nov 24 13:41:47 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 24 Nov 2010 14:41:47 +0200 Subject: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config In-Reply-To: References: <20101123071858.17721282BD6@codespeak.net> <4CEB6DD8.5080504@gmail.com> <20101124001942.2109.1155721242.divmod.xquotient.188@localhost.localdomain> Message-ID: On Wed, Nov 24, 2010 at 2:39 PM, Armin Rigo wrote: > Hi, > > On Wed, Nov 24, 2010 at 1:19 AM, ? wrote: >>>> I guess it should just be another log category, like "jit-summary". >>>> Then you can get the effect you want by setting (in the shell) PYPYLOG >>>> to "jit-summary:-". >>> >>> This or an env variable. > > The PYPYLOG solution I described *is* an environment variable. ?What > do you mean? > yes of course. nothing any more :) From drsalists at gmail.com Wed Nov 24 18:04:18 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 24 Nov 2010 09:04:18 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <4CECCFA8.80708@gmx.de> References: <4CE2765E.2020507@gmail.com> <4CE54103.6060603@gmail.com> <4CECCFA8.80708@gmx.de> Message-ID: One tiny reason: People tend to forget that they can pull things from SVN history, when an isolated file disappears. I'm expecting someday someone's going to want to add bsddb - when they do, dbm.py might be a decent starting point (but it likely has memory leaks that were corrected in gdbm.py). I'd be happy to delete it if that's the consensus. On Wed, Nov 24, 2010 at 12:41 AM, Carl Friedrich Bolz wrote: > On 11/24/2010 01:13 AM, Dan Stromberg wrote: > > I've created a branch at http://codespeak.net/svn/pypy/branch/gdbm and > > made the proposed changes in it. > > > > I took the liberty of creating an "attic" directory to put dbm.py in. > > Any reason not to just delete dbm.py? This attic stuff is annoying, and > if necessary you can still get the file from svn history. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101124/15aa92b3/attachment.htm From exarkun at twistedmatrix.com Wed Nov 24 19:50:15 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 24 Nov 2010 18:50:15 -0000 Subject: [pypy-dev] gdbm In-Reply-To: References: <4CE2765E.2020507@gmail.com> <4CE54103.6060603@gmail.com> <4CECCFA8.80708@gmx.de> Message-ID: <20101124185015.2109.1131897446.divmod.xquotient.218@localhost.localdomain> On 05:04 pm, drsalists at gmail.com wrote: >One tiny reason: People tend to forget that they can pull things from >SVN >history, when an isolated file disappears. I'm expecting someday >someone's >going to want to add bsddb - when they do, dbm.py might be a decent >starting >point (but it likely has memory leaks that were corrected in gdbm.py). Anyone who cannot figure out how to use svn has a negligible chance of successfully implementing the bsddb module. Jean-Paul From drsalists at gmail.com Wed Nov 24 19:56:42 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 24 Nov 2010 10:56:42 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <20101124185015.2109.1131897446.divmod.xquotient.218@localhost.localdomain> References: <4CE2765E.2020507@gmail.com> <4CE54103.6060603@gmail.com> <4CECCFA8.80708@gmx.de> <20101124185015.2109.1131897446.divmod.xquotient.218@localhost.localdomain> Message-ID: On Wed, Nov 24, 2010 at 10:50 AM, wrote: > On 05:04 pm, drsalists at gmail.com wrote: > >One tiny reason: People tend to forget that they can pull things from > >SVN > >history, when an isolated file disappears. I'm expecting someday > >someone's > >going to want to add bsddb - when they do, dbm.py might be a decent > >starting > >point (but it likely has memory leaks that were corrected in gdbm.py). > > Anyone who cannot figure out how to use svn has a negligible chance of > successfully implementing the bsddb module. > It's not really a matter of not knowing how to use SVN. Peg revisions are easy enough - even people who don't know them yet, will probably learn them easily. But if no one points out that there was once something vaguely related to what they want to do in the present,... Well, I think few would know to look for it under the name of "dbm.py", even if it occurred to them to look backward in time before getting started. That is, unless they saw this thread in the archives :) But I'm starting to get the feeling that the consensus is that it should be deleted. That's not a problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101124/05cb0a2a/attachment.htm From wlavrijsen at lbl.gov Wed Nov 24 23:41:40 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 24 Nov 2010 14:41:40 -0800 (PST) Subject: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' In-Reply-To: <4CECDBFC.60706@gmx.de> References: <4CECDBFC.60706@gmx.de> Message-ID: Hi Carl, > I told the JIT not to look into TypeConverter._get_fieldptr, see > revision 79444. Afterwards it seemed to work. yes ... I now finally have my executable again. :) Some initial numbers from bench1, which we ran during the sprint, are (and they are not as accurate as the digits seem to presume): PyROOT: 62.8 PyCintex: 84.9 pypy-c: 11.0 C++: 0.05 For reminder: PyCintex is Reflex + PyROOT; the number is the difference between an empty loop and one calling into a C++ function that basically does nothing. I'm a little surprised b/c I remember that PyROOT was only 2x slower when we measured during the sprint, but that might simply be improvement in the JIT of course, and I don't remember the C++ data point to anchor it (although I thought it was lower than a factor 200), so oh well. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From wlavrijsen at lbl.gov Thu Nov 25 01:32:20 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 24 Nov 2010 16:32:20 -0800 (PST) Subject: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' Message-ID: Hi, so I'm revising my numbers after finding out that I was using a debug version of ROOT/Reflex ... PyROOT: 48.6 PyCintex: 50.2 pypy-c: 5.5 C++: 0.05 Note that although PyROOT speeds up only marginally in opt mode, pypy-c has a huge speedup. For PyROOT, this is b/c most of the time is spent in CPython through API calls and that was already in opt mode. For pypy-c, this is telling me quite clearly that I've got to work some on capi.py. ;) Of course, the factor 6 that I saw originally (v.s. 2 during the sprint) is now a factor 10 almost ... Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From fijall at gmail.com Thu Nov 25 07:40:16 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 25 Nov 2010 08:40:16 +0200 Subject: [pypy-dev] [pypy-svn] r79480 - pypy/trunk/pypy/jit/tool In-Reply-To: <20101124163904.C308C50810@codespeak.net> References: <20101124163904.C308C50810@codespeak.net> Message-ID: Is pypy repository really a place for such files? Maybe we should keep it somewhere else? On Wed, Nov 24, 2010 at 6:39 PM, wrote: > Author: antocuni > Date: Wed Nov 24 17:39:03 2010 > New Revision: 79480 > > Added: > ? pypy/trunk/pypy/jit/tool/cpython.vmrss > Log: > add the vmrss file produced by armin's memusage.py when running ./translate.py -Ojit at rev 79456 > > > > Added: pypy/trunk/pypy/jit/tool/cpython.vmrss > ============================================================================== > --- (empty file) > +++ pypy/trunk/pypy/jit/tool/cpython.vmrss ? ? ?Wed Nov 24 17:39:03 2010 > @@ -0,0 +1,4101 @@ > +124 > +16600 > +20620 > +20640 > +22036 > +94084 > +94084 > +94108 > +94108 > +94108 > +94108 > +94108 > +94120 > +94164 > +94160 > +94160 > +94160 > +94160 > +94160 > +94216 > +110644 > +123144 > +135236 > +143680 > +148500 > +153104 > +157088 > +160640 > +164760 > +167992 > +163108 > +163232 > +163232 > +163436 > +163436 > +163436 > +163444 > +163444 > +163444 > +163448 > +163448 > +163448 > +163448 > +163448 > +163448 > +167060 > +170948 > +174432 > +177212 > +177216 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176508 > +176520 > +176520 > +176520 > +176520 > +176520 > +176520 > +176520 > +176520 > +176520 > +176540 > +176544 > +176544 > +176544 > +176544 > +176544 > +176544 > +176544 > +176544 > +176544 > +176544 > +176544 > +176544 > +179120 > +187120 > +189380 > +191052 > +192156 > +193320 > +194900 > +195860 > +198516 > +201484 > +202600 > +202600 > +202600 > +202600 > +202832 > +202832 > +202836 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +202840 > +207784 > +212136 > +216320 > +220508 > +224696 > +228876 > +245088 > +245088 > +247844 > +252032 > +256212 > +260400 > +264592 > +268776 > +272776 > +275060 > +279244 > +283428 > +287616 > +291032 > +293900 > +298080 > +302272 > +304364 > +308548 > +310644 > +312740 > +316924 > +319016 > +323208 > +325296 > +327392 > +331572 > +333668 > +335760 > +337856 > +354328 > +356424 > +358520 > +362700 > +364792 > +366892 > +368984 > +371080 > +373168 > +375260 > +377356 > +379448 > +381540 > +383636 > +383636 > +385732 > +387820 > +390032 > +391160 > +392292 > +394552 > +396816 > +397092 > +399072 > +401340 > +403600 > +405860 > +408008 > +408148 > +412640 > +414900 > +417164 > +419420 > +421680 > +423944 > +426204 > +428464 > +430724 > +432768 > +434980 > +436476 > +437932 > +440332 > +441984 > +442740 > +445152 > +447688 > +449148 > +449960 > +452436 > +454712 > +454896 > +457180 > +458888 > +459688 > +462040 > +463480 > +464408 > +466812 > +467244 > +469224 > +471096 > +471684 > +474136 > +474328 > +476508 > +478872 > +481356 > +483472 > +483780 > +486072 > +488480 > +489668 > +490888 > +493420 > +495704 > +496836 > +498116 > +500520 > +502756 > +503668 > +505400 > +507760 > +509296 > +510204 > +512764 > +514708 > +515508 > +517372 > +519764 > +520648 > +522188 > +524596 > +525524 > +527004 > +529412 > +534224 > +536632 > +538080 > +539216 > +541588 > +542560 > +543384 > +543384 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +518804 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519024 > +519028 > +519028 > +519028 > +519028 > +519028 > +519028 > +519028 > +519028 > +519028 > +519028 > +519028 > +519028 > +519032 > +519032 > +519032 > +519032 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519036 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519048 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519052 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519056 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519060 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519064 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +519068 > +520592 > +520592 > +520904 > +522740 > +522740 > +523212 > +525256 > +525256 > +525316 > +526552 > +526552 > +526560 > +528508 > +528508 > +528508 > +530040 > +530040 > +530040 > +532684 > +532684 > +532684 > +534948 > +534948 > +534948 > +538028 > +538028 > +538028 > +541404 > +541448 > +541448 > +543784 > +543784 > +543784 > +545184 > +545476 > +545768 > +545832 > +545832 > +545832 > +546016 > +546016 > +546128 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546184 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546228 > +546708 > +546708 > +546708 > +547988 > +550420 > +550420 > +550420 > +552896 > +555796 > +555796 > +555796 > +559136 > +560280 > +560280 > +560996 > +562504 > +563772 > +564672 > +564672 > +565268 > +567936 > +568884 > +569028 > +569236 > +569292 > +569292 > +570236 > +572960 > +573980 > +573980 > +574508 > +577404 > +579188 > +579188 > +579508 > +582836 > +584468 > +584468 > +585544 > +591292 > +591292 > +591292 > +595868 > +597588 > +597588 > +598404 > +602772 > +603964 > +603964 > +605488 > +609740 > +610468 > +610468 > +611884 > +616440 > +617108 > +617108 > +618156 > +622276 > +623784 > +623784 > +624128 > +624376 > +625544 > +626736 > +627112 > +627112 > +627116 > +627656 > +628836 > +628836 > +628836 > +629160 > +630180 > +630488 > +630488 > +630492 > +631288 > +632144 > +632144 > +632144 > +632172 > +632688 > +633220 > +633220 > +633220 > +633284 > +633756 > +633916 > +633916 > +633916 > +634012 > +634608 > +634608 > +634608 > +634624 > +634732 > +635144 > +635144 > +635144 > +635196 > +635680 > +635868 > +635868 > +635944 > +638440 > +639964 > +639980 > +639980 > +640056 > +641052 > +642064 > +642064 > +642064 > +642248 > +643080 > +643832 > +643832 > +643832 > +644116 > +646500 > +647424 > +648236 > +649032 > +649156 > +649156 > +649256 > +651556 > +652056 > +652504 > +652860 > +653168 > +653440 > +653876 > +654096 > +654304 > +654304 > +654304 > +654756 > +655648 > +657064 > +657064 > +657064 > +657064 > +657488 > +657756 > +657756 > +657756 > +658112 > +658736 > +658736 > +658736 > +658796 > +659304 > +659376 > +659376 > +659376 > +659848 > +661000 > +661172 > +661172 > +661236 > +662240 > +663240 > +663508 > +663508 > +663660 > +664324 > +665512 > +665764 > +665764 > +665764 > +666448 > +667692 > +668112 > +668112 > +668112 > +668624 > +669508 > +670388 > +670536 > +670536 > +670536 > +670564 > +671288 > +672268 > +672268 > +672268 > +672676 > +674504 > +675072 > +675072 > +675072 > +675156 > +675896 > +676700 > +676700 > +676700 > +676820 > +677988 > +678628 > +678628 > +678628 > +678776 > +679744 > +680400 > +680400 > +680400 > +705092 > +705156 > +705156 > +705156 > +705156 > +705156 > +705156 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705160 > +705264 > +705264 > +705264 > +705400 > +705476 > +706168 > +706292 > +706292 > +706292 > +706504 > +706568 > +706980 > +707012 > +707012 > +707012 > +707196 > +707280 > +707904 > +707904 > +707904 > +707924 > +708112 > +708176 > +708676 > +708676 > +708676 > +708696 > +708892 > +708984 > +709588 > +709588 > +709588 > +709612 > +709804 > +709848 > +710300 > +710300 > +710300 > +710676 > +710520 > +710604 > +711156 > +711156 > +711156 > +711336 > +711352 > +711576 > +712080 > +712080 > +712080 > +712300 > +712408 > +712500 > +712648 > +712648 > +712648 > +712648 > +713060 > +713300 > +713716 > +713716 > +713716 > +714072 > +714196 > +714568 > +714568 > +714568 > +714596 > +714956 > +715112 > +715808 > +715808 > +715808 > +717504 > +717628 > +717660 > +717660 > +717660 > +718620 > +719048 > +719424 > +719424 > +719424 > +719480 > +719924 > +720612 > +720612 > +720612 > +720620 > +722584 > +722848 > +722848 > +722848 > +723060 > +724108 > +724604 > +724604 > +724604 > +725108 > +726168 > +726348 > +726348 > +726348 > +727216 > +728204 > +728204 > +728204 > +728324 > +729396 > +730152 > +730152 > +730152 > +730396 > +736796 > +736800 > +736800 > +736800 > +736800 > +736800 > +736800 > +736800 > +736800 > +737136 > +738296 > +738400 > +738400 > +738400 > +739092 > +740128 > +740128 > +740128 > +740140 > +741092 > +741980 > +741980 > +741980 > +742060 > +743060 > +743796 > +743796 > +743796 > +743836 > +744440 > +745348 > +745348 > +745348 > +745400 > +746108 > +746848 > +746952 > +746952 > +746952 > +747496 > +748608 > +748744 > +749084 > +749084 > +749084 > +749172 > +750172 > +751468 > +751592 > +751592 > +751592 > +751688 > +751928 > +752152 > +752308 > +759152 > +760152 > +760152 > +760152 > +756356 > +754816 > +756356 > +756356 > +756356 > +756688 > +756688 > +756820 > +757152 > +757200 > +757400 > +757432 > +757432 > +757432 > +757632 > +757956 > +758404 > +758844 > +759480 > +760064 > +760552 > +760552 > +760552 > +760560 > +761180 > +761632 > +762288 > +762800 > +763700 > +764504 > +764716 > +764716 > +764716 > +764940 > +765388 > +765936 > +767748 > +767056 > +767300 > +767484 > +767868 > +768316 > +768316 > +768316 > +768316 > +768700 > +768828 > +768700 > +769340 > +769260 > +771008 > +771552 > +771652 > +771716 > +772580 > +772708 > +772740 > +772740 > +772740 > +772292 > +772740 > +772944 > +773188 > +773700 > +774084 > +774084 > +774404 > +774020 > +774532 > +774020 > +774596 > +774340 > +774468 > +774468 > +774468 > +774724 > +774792 > +774980 > +775368 > +775816 > +775816 > +776264 > +777480 > +778292 > +778408 > +778440 > +778440 > +778440 > +778440 > +778440 > +778440 > +778440 > +778440 > +778440 > +778440 > +778440 > +778632 > +778696 > +778952 > +779016 > +779016 > +779016 > +779016 > +780812 > +780812 > +780812 > +781068 > +781580 > +781772 > +781712 > +781868 > +782092 > +782420 > +782796 > +782796 > +782796 > +782796 > +782668 > +783128 > +783436 > +783820 > +784076 > +784332 > +784908 > +785164 > +785500 > +786188 > +786188 > +786188 > +786188 > +786188 > +786188 > +786188 > +786412 > +786896 > +787084 > +787404 > +787532 > +787724 > +787568 > +788108 > +788428 > +788748 > +788752 > +789520 > +789520 > +789520 > +788880 > +789440 > +789452 > +789516 > +790092 > +790284 > +790604 > +790860 > +791052 > +791372 > +791628 > +792012 > +792012 > +792396 > +792780 > +792780 > +792780 > +792780 > +792780 > +793228 > +793228 > +793868 > +793868 > +793996 > +793996 > +794636 > +794636 > +794636 > +795084 > +795084 > +795468 > +795532 > +795980 > +795980 > +796300 > +796364 > +796364 > +796364 > +796364 > +796748 > +797132 > +797132 > +797516 > +797644 > +797644 > +798380 > +798604 > +798860 > +798924 > +799372 > +799564 > +799756 > +799756 > +799756 > +799756 > +799816 > +800292 > +800792 > +801312 > +801816 > +802364 > +802880 > +803456 > +803660 > +803660 > +803660 > +803812 > +804388 > +804960 > +805516 > +806084 > +806668 > +807324 > +807980 > +807980 > +807980 > +807980 > +808416 > +809084 > +809784 > +810492 > +811160 > +812900 > +813476 > +813876 > +813876 > +813876 > +813876 > +814508 > +815152 > +815864 > +816556 > +817260 > +817916 > +818708 > +819272 > +819352 > +819352 > +819352 > +819352 > +819884 > +820308 > +820888 > +821012 > +821012 > +821204 > +821588 > +821588 > +821588 > +821972 > +822228 > +822164 > +822932 > +823252 > +823252 > +823252 > +823252 > +823252 > +823256 > +823612 > +823920 > +823936 > +824140 > +824168 > +824220 > +824280 > +824400 > +824664 > +825776 > +825776 > +825776 > +825776 > +832168 > +832168 > +832168 > +832168 > +832808 > +833492 > +833492 > +833492 > +833492 > +833700 > +834308 > +834428 > +834428 > +834428 > +834780 > +836216 > +836216 > +836216 > +836836 > +839308 > +839308 > +839308 > +839308 > +839416 > +840344 > +840344 > +840344 > +840344 > +841724 > +841724 > +841724 > +841724 > +843800 > +843800 > +843800 > +843800 > +845692 > +846072 > +846072 > +846072 > +846096 > +846736 > +847096 > +847156 > +847156 > +847156 > +847160 > +847180 > +847360 > +847360 > +847360 > +847360 > +847416 > +849248 > +849248 > +849248 > +849248 > +849716 > +849716 > +849716 > +849716 > +849740 > +851168 > +851168 > +851168 > +851168 > +854544 > +854544 > +854544 > +854544 > +854564 > +855332 > +855332 > +855332 > +855332 > +857092 > +857092 > +857092 > +857092 > +858000 > +859388 > +859388 > +859388 > +859388 > +861584 > +861584 > +861584 > +861584 > +861584 > +863464 > +863464 > +863464 > +863464 > +864304 > +866500 > +866500 > +866500 > +866500 > +868952 > +868952 > +868952 > +868952 > +869740 > +872108 > +872108 > +872108 > +872108 > +873208 > +875564 > +875564 > +875564 > +875564 > +878568 > +878568 > +878568 > +878568 > +878576 > +880780 > +880780 > +880780 > +880780 > +884048 > +884048 > +884048 > +884048 > +884048 > +884048 > +886516 > +886516 > +886516 > +886516 > +886516 > +886536 > +888112 > +888112 > +888112 > +888112 > +888112 > +888112 > +888656 > +888984 > +888984 > +888984 > +888984 > +888984 > +888984 > +889076 > +890288 > +890288 > +890288 > +890288 > +890288 > +890288 > +890404 > +892900 > +892900 > +892900 > +892900 > +892900 > +892900 > +892900 > +892900 > +895760 > +895760 > +895760 > +895760 > +895760 > +895760 > +895920 > +897624 > +897624 > +897624 > +897624 > +897624 > +897624 > +897624 > +897624 > +897628 > +898024 > +898024 > +898024 > +898024 > +898024 > +898060 > +900024 > +900024 > +900024 > +900024 > +900024 > +900024 > +900240 > +901436 > +901436 > +901436 > +901436 > +901436 > +901556 > +903116 > +903116 > +903116 > +903116 > +903116 > +903128 > +905084 > +905084 > +905084 > +905084 > +905084 > +905084 > +905096 > +906832 > +906832 > +906832 > +906832 > +906832 > +906832 > +908916 > +908916 > +908916 > +908916 > +908916 > +908916 > +908916 > +910720 > +910720 > +910720 > +910720 > +910720 > +910720 > +911780 > +912072 > +912072 > +912072 > +912072 > +914472 > +914472 > +914472 > +914472 > +914472 > +917120 > +917120 > +917120 > +917120 > +917120 > +919056 > +919056 > +919056 > +919056 > +919056 > +920316 > +920316 > +920316 > +920316 > +920316 > +920892 > +920892 > +920892 > +920892 > +920892 > +922996 > +922996 > +922996 > +922996 > +922996 > +925564 > +925564 > +925564 > +925564 > +925564 > +927780 > +927780 > +927780 > +927780 > +928016 > +928936 > +929048 > +930864 > +930864 > +930864 > +930864 > +932980 > +933304 > +933304 > +933304 > +933304 > +933540 > +934292 > +935452 > +935528 > +935528 > +935528 > +935528 > +935528 > +935528 > +935528 > +935528 > +935600 > +936112 > +936112 > +936208 > +936208 > +936208 > +936208 > +936308 > +936308 > +936316 > +936368 > +936368 > +936368 > +936368 > +936368 > +936368 > +936368 > +936368 > +936368 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +936376 > +937404 > +937404 > +937404 > +937404 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +939968 > +940516 > +940516 > +940516 > +940516 > +947168 > +947168 > +947168 > +947168 > +951948 > +951948 > +951948 > +951948 > +953488 > +956916 > +956916 > +956916 > +956916 > +952296 > +955376 > +953420 > +953260 > +953900 > +953516 > +953900 > +955052 > +953516 > +953516 > +954284 > +953324 > +953516 > +956208 > +956208 > +956208 > +956208 > +954668 > +948988 > +948988 > +948988 > +948988 > +948988 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +945908 > +947704 > +948068 > +948068 > +948068 > +948068 > +948068 > +948552 > +949024 > +949024 > +949024 > +949024 > +949024 > +949944 > +950492 > +950616 > +950616 > +950616 > +950616 > +951416 > +952000 > +952564 > +952564 > +952564 > +952564 > +952620 > +953296 > +953952 > +954332 > +954332 > +954332 > +954332 > +954532 > +955532 > +956216 > +956216 > +956216 > +956216 > +956216 > +956584 > +957396 > +957704 > +957704 > +957704 > +957704 > +957740 > +958620 > +959444 > +959444 > +959444 > +959444 > +959444 > +960224 > +963784 > +963868 > +963868 > +963868 > +963868 > +963872 > +964684 > +972452 > +972452 > +972452 > +972452 > +972452 > +972452 > +973220 > +974548 > +974548 > +974548 > +974548 > +974548 > +975540 > +977704 > +978792 > +978792 > +978792 > +978792 > +978792 > +981296 > +982700 > +983180 > +983180 > +983180 > +983180 > +983368 > +985060 > +987512 > +987688 > +987688 > +987688 > +987688 > +988180 > +990724 > +993084 > +993124 > +993124 > +993124 > +993124 > +993604 > +995444 > +996640 > +996640 > +996640 > +996640 > +996640 > +997892 > +999160 > +1001452 > +1001452 > +1001452 > +1001452 > +1001452 > +1002264 > +1004120 > +1004916 > +1004916 > +1004916 > +1004916 > +1004916 > +1006648 > +1010244 > +1010548 > +1010548 > +1010548 > +1010548 > +1010572 > +1011860 > +1013748 > +1017264 > +1017264 > +1017264 > +1017264 > +1017264 > +1019096 > +1021044 > +1021044 > +1021044 > +1021044 > +1021044 > +1021536 > +1023636 > +1024964 > +1024964 > +1024964 > +1024964 > +1024964 > +1025592 > +1027672 > +1029384 > +1029384 > +1029384 > +1029384 > +1029384 > +1030056 > +1032244 > +1033756 > +1033756 > +1033756 > +1033756 > +1033756 > +1034384 > +1035856 > +1036588 > +1038428 > +1038428 > +1038428 > +1038428 > +1038428 > +1039288 > +1041088 > +1042508 > +1042508 > +1042508 > +1042508 > +1042508 > +1043544 > +1045280 > +1046580 > +1046580 > +1046580 > +1046580 > +1046580 > +1047040 > +1048560 > +1049872 > +1050576 > +1050576 > +1050576 > +1050576 > +1050576 > +1052016 > +1056044 > +1057360 > +1057360 > +1057360 > +1057360 > +1057360 > +1058336 > +1059900 > +1061244 > +1061704 > +1061704 > +1061704 > +1061704 > +1061704 > +1063148 > +1063972 > +1065404 > +1067064 > +1067536 > +1067536 > +1067536 > +1067536 > +1067536 > +1069284 > +1070524 > +1072340 > +1072836 > +1072836 > +1072836 > +1072836 > +1072836 > +1073584 > +1074592 > +1076404 > +1076832 > +1076832 > +1076832 > +1076832 > +1076832 > +1078124 > +1079640 > +1080220 > +1080644 > +1080736 > +1080736 > +1080736 > +1080736 > +1080924 > +1082868 > +1083368 > +1084412 > +1084412 > +1084412 > +1084412 > +1084412 > +1085216 > +1087068 > +1087960 > +1087960 > +1087960 > +1087960 > +1087960 > +1089788 > +1093132 > +1095064 > +1095064 > +1095064 > +1095064 > +1095064 > +1095140 > +1097092 > +1098948 > +1098948 > +1098948 > +1098948 > +1098948 > +1098980 > +1100812 > +1102032 > +1102032 > +1102032 > +1102032 > +1102032 > +1105464 > +1116944 > +1119248 > +1120364 > +1120364 > +1120364 > +1120364 > +1120364 > +1121568 > +1122908 > +1123680 > +1124604 > +1125076 > +1125076 > +1125076 > +1125076 > +1125076 > +1126292 > +1128160 > +1129952 > +1129952 > +1129952 > +1129952 > +1129952 > +1130496 > +1131884 > +1133032 > +1134204 > +1135460 > +1135636 > +1135636 > +1135636 > +1135636 > +1135636 > +1136764 > +1138048 > +1139412 > +1140764 > +1141164 > +1141188 > +1142440 > +1142440 > +1142440 > +1142440 > +1142440 > +1142440 > +1143980 > +1145876 > +1146576 > +1146576 > +1146576 > +1146576 > +1146576 > +1146576 > +1147680 > +1148328 > +1148960 > +1148960 > +1148960 > +1148960 > +1148960 > +1149004 > +1150700 > +1152228 > +1153364 > +1153364 > +1153520 > +1153784 > +1154588 > +1154680 > +1154712 > +1154728 > +1154784 > +1154992 > +1155356 > +1155620 > +1155856 > +1156044 > +1156420 > +1157392 > +1158760 > +1158980 > +1158988 > +1159000 > +1162724 > +1162740 > +1162788 > +1163112 > +1163188 > +1163188 > +1163188 > +1163188 > +1163188 > +1163384 > +1165668 > +1166648 > +1166652 > +1166664 > +1166676 > +1166688 > +1166692 > +1166696 > +1166700 > +1166700 > +1172848 > +1172852 > +1174888 > +1176824 > +1176836 > +1176852 > +1176860 > +1176876 > +1176880 > +1176888 > +1176892 > +1176900 > +1176912 > +1176944 > +1177248 > +1177712 > +1178172 > +1178536 > +1178656 > +1178780 > +1178920 > +1179044 > +1179188 > +1179384 > +1180296 > +1180300 > +1180300 > +1180300 > +1180300 > +1180300 > +1180300 > +1180372 > +1180380 > +1180468 > +1180524 > +1180524 > +1180524 > +1180524 > +1180524 > +1180576 > +1180580 > +1180644 > +1180684 > +1180684 > +1180684 > +1180684 > +1180684 > +1180684 > +1180724 > +1180756 > +1180852 > +1180904 > +1180904 > +1180904 > +1180904 > +1180904 > +1180904 > +1181096 > +1181400 > +1181744 > +1181744 > +1181744 > +1181744 > +1181744 > +1181744 > +1181936 > +1181936 > +1181936 > +1181936 > +1181936 > +1181936 > +1181936 > +1181936 > +1181936 > +1181972 > +1182004 > +1182004 > +1182004 > +1182004 > +1182004 > +1182004 > +1182004 > +1182004 > +1182004 > +1182128 > +1182156 > +1182156 > +1182156 > +1182156 > +1182156 > +1182156 > +1182156 > +1182156 > +1182180 > +1182180 > +1182180 > +1182180 > +1182180 > +1182180 > +1182180 > +1182368 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1182516 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183576 > +1183596 > +1183772 > +1183772 > +1183772 > +1183772 > +1183772 > +1183772 > +1183772 > +1183772 > +1183772 > +1183776 > +1183776 > +1183776 > +1183776 > +1183776 > +1183776 > +1183776 > +1183776 > +1183776 > +1183952 > +1184000 > +1184000 > +1184000 > +1184000 > +1184000 > +1184000 > +1184000 > +1184200 > +1184832 > +1184832 > +1184832 > +1184832 > +1184832 > +1184832 > +1184928 > +1184928 > +1184940 > +1184940 > +1184940 > +1184940 > +1184940 > +1184940 > +1184940 > +1184940 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185144 > +1185196 > +1185196 > +1185196 > +1185196 > +1185196 > +1185196 > +1185196 > +1185196 > +1185196 > +1185268 > +1185268 > +1185268 > +1185268 > +1185268 > +1185268 > +1185268 > +1185268 > +1186444 > +1186776 > +1186776 > +1186776 > +1186776 > +1186776 > +1187664 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188072 > +1188168 > +1188168 > +1188168 > +1188168 > +1188168 > +1188168 > +1188168 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189240 > +1189292 > +1189292 > +1189292 > +1189292 > +1189292 > +1189292 > +1189292 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189532 > +1189704 > +1189708 > +1189708 > +1189708 > +1189708 > +1189708 > +1190160 > +1190160 > +1190160 > +1190160 > +1190160 > +1190160 > +1190160 > +1190160 > +1190160 > +1191100 > +1191100 > +1191100 > +1191100 > +1191100 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191316 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191748 > +1191772 > +1191772 > +1191772 > +1191772 > +1191772 > +1191772 > +1191772 > +1191772 > +1192964 > +1192964 > +1192964 > +1192964 > +1192964 > +1192964 > +1192964 > +1193060 > +1193060 > +1193060 > +1193060 > +1193060 > +1193060 > +1193060 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193072 > +1193076 > +1193076 > +1193076 > +1193076 > +1193076 > +1193076 > +1193124 > +1193124 > +1193360 > +1194108 > +1194108 > +1194108 > +1194108 > +1194108 > +1193380 > +1193460 > +1193460 > +1193460 > +1193460 > +1193460 > +1193460 > +1193460 > +1193460 > +1193792 > +1193792 > +1193792 > +1193792 > +1193792 > +1193792 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194000 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194048 > +1194328 > +1194328 > +1194328 > +1194328 > +1194328 > +1194328 > +1194328 > +1194328 > +1194328 > +1194328 > +1194360 > +1194360 > +1194360 > +1194360 > +1194360 > +1194360 > +1194360 > +1194508 > +1194508 > +1194508 > +1194508 > +1194508 > +1194508 > +1194512 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194668 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1194912 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196596 > +1196660 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196764 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1196948 > +1197236 > +1197236 > +1197236 > +1197236 > +1197236 > +1197236 > +1197236 > +1197236 > +1197236 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197288 > +1197376 > +1197384 > +1197596 > +1197596 > +1197596 > +1197596 > +1197596 > +1197596 > +1197596 > +1197596 > +1197588 > +1197588 > +1197588 > +1197588 > +1197588 > +1197588 > +1197588 > +1197588 > +1197564 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197460 > +1197492 > +1197508 > +1197516 > +1197516 > +1197516 > +1197516 > +1197516 > +1197516 > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From anto.cuni at gmail.com Thu Nov 25 08:45:52 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 25 Nov 2010 08:45:52 +0100 Subject: [pypy-dev] [pypy-svn] r79480 - pypy/trunk/pypy/jit/tool In-Reply-To: References: <20101124163904.C308C50810@codespeak.net> Message-ID: <4CEE1430.2@gmail.com> On 25/11/10 07:40, Maciej Fijalkowski wrote: > Is pypy repository really a place for such files? Maybe we should keep > it somewhere else? well, it's the memory usage of cpython when translating pypy. Why the pypy repo should not be the place for a pypy-related file? > On Wed, Nov 24, 2010 at 6:39 PM, wrote: >> Author: antocuni >> Date: Wed Nov 24 17:39:03 2010 >> New Revision: 79480 >> >> Added: >> pypy/trunk/pypy/jit/tool/cpython.vmrss >> Log: >> add the vmrss file produced by armin's memusage.py when running ./translate.py -Ojit at rev 79456 From anto.cuni at gmail.com Thu Nov 25 08:53:11 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 25 Nov 2010 08:53:11 +0100 Subject: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' In-Reply-To: References: Message-ID: <4CEE15E7.3030409@gmail.com> Hi Wim, On 25/11/10 01:32, wlavrijsen at lbl.gov wrote: > Hi, > > so I'm revising my numbers after finding out that I was using a debug version > of ROOT/Reflex ... > > PyROOT: 48.6 > PyCintex: 50.2 > pypy-c: 5.5 > C++: 0.05 > wow, impressive numbers :-). You said that the benchmark you used is a loop calling a c++ function that does nothing. What is the signature of that function? If you remember, in interp_cppyy there is a fast path that gives huge speedups when calling a c++ function which takes an integer and returns an integer, so depending on the signature you get very different numbers today. The good news is that nowadays the JIT has got the capability to optimise external calls in a general way: this means that as soon as we integrate it with cppyy we can have all calls as fast as the ones that go through the current fast path. ciao, Anto From fijall at gmail.com Thu Nov 25 09:24:13 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 25 Nov 2010 10:24:13 +0200 Subject: [pypy-dev] [pypy-svn] r79480 - pypy/trunk/pypy/jit/tool In-Reply-To: <4CEE1430.2@gmail.com> References: <20101124163904.C308C50810@codespeak.net> <4CEE1430.2@gmail.com> Message-ID: On Thu, Nov 25, 2010 at 9:45 AM, Antonio Cuni wrote: > On 25/11/10 07:40, Maciej Fijalkowski wrote: >> >> Is pypy repository really a place for such files? Maybe we should keep >> it somewhere else? > > well, it's the memory usage of cpython when translating pypy. Why the pypy > repo should not be the place for a pypy-related file? For the same reason why we don't put extradoc in checkout I guess. > >> On Wed, Nov 24, 2010 at 6:39 PM, ?wrote: >>> >>> Author: antocuni >>> Date: Wed Nov 24 17:39:03 2010 >>> New Revision: 79480 >>> >>> Added: >>> ? pypy/trunk/pypy/jit/tool/cpython.vmrss >>> Log: >>> add the vmrss file produced by armin's memusage.py when running >>> ./translate.py -Ojit at rev 79456 > > From arigo at tunes.org Thu Nov 25 11:18:39 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 25 Nov 2010 11:18:39 +0100 Subject: [pypy-dev] [pypy-svn] r79480 - pypy/trunk/pypy/jit/tool In-Reply-To: References: <20101124163904.C308C50810@codespeak.net> <4CEE1430.2@gmail.com> Message-ID: Hi Anto, On Thu, Nov 25, 2010 at 9:24 AM, Maciej Fijalkowski wrote: > For the same reason why we don't put extradoc in checkout I guess. I tend to agree with fijal that this file should be in extradoc/somepath... Armin From wlavrijsen at lbl.gov Thu Nov 25 17:46:48 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 25 Nov 2010 08:46:48 -0800 (PST) Subject: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' In-Reply-To: <4CEE15E7.3030409@gmail.com> References: <4CEE15E7.3030409@gmail.com> Message-ID: Hi Anto, > You said that the benchmark you used is a loop calling a c++ function that > does nothing. What is the signature of that function? > If you remember, in interp_cppyy there is a fast path that gives huge > speedups when calling a c++ function which takes an integer and returns an > integer, so depending on the signature you get very different numbers today. yes, but the fast path was disabled (commented out) when updating the branch with trunk (I needed a few casts in the JIT that were in trunk already, but not in the branch). It should thus be in addition when it's working again. (And to be sure, using a float argument makes no difference for the numbers.) Myself, I'm just working on functionality right now. A factor of almost 10 is already good enough to start working on a functional demo of real code, given that most of our analysis is I/O bound. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From holger at merlinux.eu Thu Nov 25 20:02:05 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 25 Nov 2010 20:02:05 +0100 Subject: [pypy-dev] why pypy startup is currently ~5 times slower than python? Message-ID: <20101125190205.GA1009@trillke.net> Hi, using the following script: import sys, subprocess for i in range(1000): subprocess.call([sys.executable, "--version"]) i get these results repeatedly: (0)hpk at teta:~/p/pytest$ time python2.7 r.py 2>/dev/null >/dev/null real 0m1.885s user 0m0.110s sys 0m0.480s (0)hpk at teta:~/p/pytest$ time pypy r.py 2>/dev/null >/dev/null real 0m8.763s user 0m1.730s sys 0m2.880s IOW, pypy seems to be consistently 5-6 times slower on startup. Does anybody have explanations why? When we did measurements for the Nokia Maemo device pypy was actually much faster IIRC. holger From holger at merlinux.eu Thu Nov 25 20:21:03 2010 From: holger at merlinux.eu (holger krekel) Date: Thu, 25 Nov 2010 20:21:03 +0100 Subject: [pypy-dev] faster actually (was Re: why pypy startup is currently ~5 times slower than python?) In-Reply-To: <20101125190205.GA1009@trillke.net> References: <20101125190205.GA1009@trillke.net> Message-ID: <20101125192103.GB1009@trillke.net> Well, it seems to be the version-printing. Using subprocess.call([sys.executable, "-c", "pass"]) or "print 42" or other simple commands pypy seems faster up to 2 times. Not sure care is needed for speed of --version :) cheers, holger On Thu, Nov 25, 2010 at 20:02 +0100, holger krekel wrote: > using the following script: > > import sys, subprocess > > for i in range(1000): > subprocess.call([sys.executable, "--version"]) > > i get these results repeatedly: > > (0)hpk at teta:~/p/pytest$ time python2.7 r.py 2>/dev/null >/dev/null > real 0m1.885s > user 0m0.110s > sys 0m0.480s > > (0)hpk at teta:~/p/pytest$ time pypy r.py 2>/dev/null >/dev/null > real 0m8.763s > user 0m1.730s > sys 0m2.880s > > IOW, pypy seems to be consistently 5-6 times slower on startup. > Does anybody have explanations why? When we did measurements > for the Nokia Maemo device pypy was actually much faster IIRC. > > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- From anto.cuni at gmail.com Fri Nov 26 11:48:49 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 26 Nov 2010 11:48:49 +0100 Subject: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd' In-Reply-To: References: <4CEE15E7.3030409@gmail.com> Message-ID: <4CEF9091.7010405@gmail.com> On 25/11/10 17:46, wlavrijsen at lbl.gov wrote: > yes, but the fast path was disabled (commented out) when updating the branch > with trunk (I needed a few casts in the JIT that were in trunk already, but > not in the branch). It should thus be in addition when it's working again. > (And to be sure, using a float argument makes no difference for the numbers.) > > Myself, I'm just working on functionality right now. A factor of almost 10 > is already good enough to start working on a functional demo of real code, > given that most of our analysis is I/O bound. wow, if you did not use the fast path, then the 10x factor it's even more impressive :-). I think that yesterday Carl Friedrich added support to cppyy for calling the function through the new JIT-friendly "rlib.libffi" module, so now you should get the "fast-path" performance for most of the calls. Carl, could you post the benchmarks you did please? ciao, Anto From cfbolz at gmx.de Fri Nov 26 16:31:44 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 26 Nov 2010 16:31:44 +0100 Subject: [pypy-dev] cppyy performance numbers (was: ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd') In-Reply-To: References: Message-ID: <4CEFD2E0.8050201@gmx.de> Hi Wim, hi all, On 11/25/2010 01:32 AM, wlavrijsen at lbl.gov wrote: > so I'm revising my numbers after finding out that I was using a debug > version > of ROOT/Reflex ... > > PyROOT: 48.6 > PyCintex: 50.2 > pypy-c: 5.5 > C++: 0.05 I did some benchmarks with the new fast path that I just added, it gives the following numbers: pypy-c without fast path: 5.9 (my laptop seems a bit slower than yours) pypy-c with fast path: 0.22 C++: 0.05 So Anto's and my optimization helped a lot, we got another factor of 25 faster. A bit more than a factor of 4 slower than C++, and it should work for many (non-static) method and constructor calls. > > Note that although PyROOT speeds up only marginally in opt mode, pypy-c has > a huge speedup. For PyROOT, this is b/c most of the time is spent in > CPython > through API calls and that was already in opt mode. For pypy-c, this is > telling me quite clearly that I've got to work some on capi.py. ;) > > Of course, the factor 6 that I saw originally (v.s. 2 during the sprint) is > now a factor 10 almost ... Are you sure that the units are the same? Because if yes, it is a factor of 100, not 10. Cheers, Carl Friedrich P.S.: For the people missing context, see here: http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries.html From fijall at gmail.com Fri Nov 26 19:23:45 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 26 Nov 2010 20:23:45 +0200 Subject: [pypy-dev] PyPy 1.4 released Message-ID: =============================== PyPy 1.4: Ouroboros in practice =============================== We're pleased to announce the 1.4 release of PyPy. This is a major breakthrough in our long journey, as PyPy 1.4 is the first PyPy release that can translate itself faster than CPython. Starting today, we are using PyPy more for our every-day development. So may you :) You can download it here: http://pypy.org/download.html What is PyPy ============ PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython. It's fast (`pypy 1.4 and cpython 2.6`_ comparison) Among its new features, this release includes numerous performance improvements (which made fast self-hosting possible), a 64-bit JIT backend, as well as serious stabilization. As of now, we can consider the 32-bit and 64-bit linux versions of PyPy stable enough to run `in production`_. Numerous speed achievements are described on `our blog`_. Normalized speed charts comparing `pypy 1.4 and pypy 1.3`_ as well as `pypy 1.4 and cpython 2.6`_ are available on benchmark website. For the impatient: yes, we got a lot faster! More highlights =============== * PyPy's built-in Just-in-Time compiler is fully transparent and automatically generated; it now also has very reasonable memory requirements. The total memory used by a very complex and long-running process (translating PyPy itself) is within 1.5x to at most 2x the memory needed by CPython, for a speed-up of 2x. * More compact instances. All instances are as compact as if they had ``__slots__``. This can give programs a big gain in memory. (In the example of translation above, we already have carefully placed ``__slots__``, so there is no extra win.) * `Virtualenv support`_: now PyPy is fully compatible with virtualenv_: note that to use it, you need a recent version of virtualenv (>= 1.5). * Faster (and JITted) regular expressions - huge boost in speeding up the `re` module. * Other speed improvements, like JITted calls to functions like map(). .. _virtualenv: http://pypi.python.org/pypi/virtualenv .. _`Virtualenv support`: http://morepypy.blogspot.com/2010/08/using-virtualenv-with-pypy.html .. _`in production`: http://morepypy.blogspot.com/2010/11/running-large-radio-telescope-software.html .. _`our blog`: http://morepypy.blogspot.com .. _`pypy 1.4 and pypy 1.3`: http://speed.pypy.org/comparison/?exe=1%2B41,1%2B172&ben=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=false&bas=1%2B41&chart=normal+bars .. _`pypy 1.4 and cpython 2.6`: http://speed.pypy.org/comparison/?exe=2%2B35,1%2B172&ben=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=false&bas=2%2B35&chart=normal+bars Cheers, Carl Friedrich Bolz, Antonio Cuni, Maciej Fijalkowski, Amaury Forgeot d'Arc, Armin Rigo and the PyPy team From samuele.pedroni at gmail.com Fri Nov 26 19:49:20 2010 From: samuele.pedroni at gmail.com (Samuele Pedroni) Date: Fri, 26 Nov 2010 19:49:20 +0100 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: Message-ID: On Fri, Nov 26, 2010 at 7:23 PM, Maciej Fijalkowski wrote: > =============================== > PyPy 1.4: Ouroboros in practice > =============================== > > We're pleased to announce the 1.4 release of PyPy. This is a major breakthrough > in our long journey, as PyPy 1.4 is the first PyPy release that can translate > itself faster than CPython. ?Starting today, we are using PyPy more for > our every-day development. ?So may you :) You can download it here: > > ? ?http://pypy.org/download.html > > What is PyPy > ============ > > PyPy is a very compliant Python interpreter, almost a drop-in replacement > for CPython. It's fast (`pypy 1.4 and cpython 2.6`_ comparison) > > Among its new features, this release includes numerous performance improvements > (which made fast self-hosting possible), a 64-bit JIT backend, as well > as serious stabilization. As of now, we can consider the 32-bit and 64-bit > linux versions of PyPy stable enough to run `in production`_. > > Numerous speed achievements are described on `our blog`_. Normalized speed > charts comparing `pypy 1.4 and pypy 1.3`_ as well as `pypy 1.4 and cpython 2.6`_ > are available on benchmark website. For the impatient: yes, we got a lot faster! > congratulations! From wlavrijsen at lbl.gov Fri Nov 26 20:05:07 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 26 Nov 2010 11:05:07 -0800 (PST) Subject: [pypy-dev] cppyy performance numbers (was: ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd') In-Reply-To: <4CEFD2E0.8050201@gmx.de> References: <4CEFD2E0.8050201@gmx.de> Message-ID: Hi Carl, >> PyROOT: 48.6 >> PyCintex: 50.2 >> pypy-c: 5.5 >> C++: 0.05 > > pypy-c without fast path: 5.9 (my laptop seems a bit slower than yours) > pypy-c with fast path: 0.22 > C++: 0.05 in my case, with patched reflex, using the reflex-branch of pypy with the libffi changes, I get a number of 0.41 for pypy-c. So yes, that helped a lot, even when our numbers don't seem to line up, the overall direction is definitely the way we want it. :) I'll see later when moving up to trunk makes the numbers the same (on a relative basis at least). Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From fuzzyman at gmail.com Fri Nov 26 23:15:11 2010 From: fuzzyman at gmail.com (Michael Foord) Date: Fri, 26 Nov 2010 22:15:11 +0000 Subject: [pypy-dev] pypy 1.4 and virtualenv 1.5.1 Message-ID: Hello all, Congratulations on the new release. Unfortunately it seems that virtualenv 1.5.1 and pypy 1.4 do not play well together on Mac OS X. At least the "--distribute" option doesn't work, which is needed by tox. I tried to create an issue on the PyPy issue tracker, but new registrants are unable to create issues: Here is the output on Linux: $ virtualenv -p /compile/pypy-1.4/bin/pypy pypy --distribute --no-site-packages Running virtualenv with interpreter /compile/pypy-1.4/bin/pypy New pypy-c executable in pypy/bin/pypy Also creating executable in pypy/bin/pypy-c Installing distribute....................................................................................................................................................................................done. Here is the failed output on OS X: $ virtualenv -p /compile/pypy-1.4-osx/bin/pypy pypy --distribute --no-site-packages Running virtualenv with interpreter /compile/pypy-1.4-osx/bin/pypy New pypy-c executable in pypy/bin/pypy Also creating executable in pypy/bin/pypy-c Please make sure you remove any previous custom paths from your /Volumes/Second Drive/michael/.pydistutils.cfg file. Installing distribute................................done. Complete output from command /compile/pypy/bin/pypy /compile/pypy/bin/easy_install /Library/Frameworks/Python.fra...ar.gz: Traceback (most recent call last): File "app_main.py", line 33, in run_toplevel IOError: [Errno 2] No such file or directory: '/compile/pypy/bin/easy_install' ---------------------------------------- Traceback (most recent call last): File "app_main.py", line 33, in run_toplevel File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", line 1647, in main() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", line 558, in main prompt=options.prompt) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", line 656, in create_environment install_pip(py_executable) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", line 415, in install_pip filter_stdout=_filter_setup) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", line 624, in call_subprocess % (cmd_desc, proc.returncode)) OSError: Command /compile/pypy/bin/pypy /compile/pypy/bin/easy_install /Library/Frameworks/Python.fra...ar.gz failed with error code 1 The nice thing is that for unittest2 and mock all tests pass with pypy 1.4. Not unexpected, but still nice. All the best, Michael Foord -- http://www.voidspace.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101126/d8f7588a/attachment.htm From anto.cuni at gmail.com Sat Nov 27 00:12:18 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 27 Nov 2010 00:12:18 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: <201011160222.oAG2Mrfv004177@theraft.openend.se> Message-ID: <4CF03ED2.3050801@gmail.com> On 16/11/10 04:30, Dan Stromberg wrote: > BTW, it might cause confusion down the road to call something that is > basically like cpython's bsddb (Berkeley DB) by the name "dbm" in pypy's > library. In the cpython standard library, "dbm" is an interface to ndbm > databases. These all provide the same dictionary-like interface to Python > programs, but have somewhat different API's to C, and pretty different, > incompatible on-disk representations. Hi Dan, I played a bit (veeeery quickly) with dbm on both pypy and cpython, and I'm not sure I get what you mean when you say that our dbm.py is equivalent to cpython's bsddb. E.g., I can create a db on cpython and open it from pypy, so it seems that the two modules are compatible. Moreover, I checked which libraries the links to. On CPython, it links to libdb-4.8.so: viper2 ~ $ ldd /usr/lib/python2.6/lib-dynload/dbm.so linux-gate.so.1 => (0x00884000) libdb-4.8.so => /usr/lib/libdb-4.8.so (0x00110000) libpthread.so.0 => /lib/libpthread.so.0 (0x003de000) libc.so.6 => /lib/libc.so.6 (0x003f8000) /lib/ld-linux.so.2 (0x002e0000) the pypy version first tries to open libdb.so, then libdb-4.5.so. I had to manually modify it to open version 4.8 (I agree that we should find a more general way to find it), but apart from that what I can see is that it uses the same underlying wrapper as CPython. So, to summarise: could you elaborate a bit more why we should delete dbm.py from pypy? ciao, Anto From angelflow at yahoo.com Sat Nov 27 00:03:08 2010 From: angelflow at yahoo.com (Andy) Date: Fri, 26 Nov 2010 15:03:08 -0800 (PST) Subject: [pypy-dev] PyPy-JIT memory usage compared to CPython? Message-ID: <160969.12027.qm@web111315.mail.gq1.yahoo.com> Hi, Congrats on the 1.4 release! >From the benchmark results on your site, the performance looks great. I was wondering if you have any data showing relative memory usage for the various benchmark. I'm particularly interested in the memory usage of PyPy-JIT vs. CPython when running Django. In general would PyPy-JIT increase or decrease memory usage? Thanks Andy From holger at merlinux.eu Sat Nov 27 00:19:41 2010 From: holger at merlinux.eu (holger krekel) Date: Sat, 27 Nov 2010 00:19:41 +0100 Subject: [pypy-dev] pypy 1.4 and virtualenv 1.5.1 In-Reply-To: References: Message-ID: <20101126231940.GJ1009@trillke.net> Hi Michael, On Fri, Nov 26, 2010 at 22:15 +0000, Michael Foord wrote: > Congratulations on the new release. > > Unfortunately it seems that virtualenv 1.5.1 and pypy 1.4 do not play well > together on Mac OS X. At least the "--distribute" option doesn't work, which > is needed by tox. It's not needed, with tox you can configure "distribute=False" in your test env and this would install setuptools. > I tried to create an issue on the PyPy issue tracker, but new registrants > are unable to create issues: > > Here is the output on Linux: > > $ virtualenv -p /compile/pypy-1.4/bin/pypy pypy --distribute > --no-site-packages Does that work for you without --distribute? holger > Running virtualenv with interpreter /compile/pypy-1.4/bin/pypy > New pypy-c executable in pypy/bin/pypy > Also creating executable in pypy/bin/pypy-c > Installing > distribute....................................................................................................................................................................................done. > > > Here is the failed output on OS X: > > $ virtualenv -p /compile/pypy-1.4-osx/bin/pypy pypy --distribute > --no-site-packages > Running virtualenv with interpreter /compile/pypy-1.4-osx/bin/pypy > New pypy-c executable in pypy/bin/pypy > Also creating executable in pypy/bin/pypy-c > Please make sure you remove any previous custom paths from your > /Volumes/Second Drive/michael/.pydistutils.cfg file. > Installing distribute................................done. > Complete output from command /compile/pypy/bin/pypy > /compile/pypy/bin/easy_install /Library/Frameworks/Python.fra...ar.gz: > Traceback (most recent call last): > File "app_main.py", line 33, in run_toplevel > IOError: [Errno 2] No such file or directory: '/compile/pypy/bin/easy_install' > ---------------------------------------- > Traceback (most recent call last): > File "app_main.py", line 33, in run_toplevel > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > line 1647, in > main() > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > line 558, in main > prompt=options.prompt) > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > line 656, in create_environment > install_pip(py_executable) > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > line 415, in install_pip > filter_stdout=_filter_setup) > File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > line 624, in call_subprocess > % (cmd_desc, proc.returncode)) > OSError: Command /compile/pypy/bin/pypy /compile/pypy/bin/easy_install > /Library/Frameworks/Python.fra...ar.gz failed with error code 1 > > > The nice thing is that for unittest2 and mock all tests pass with pypy 1.4. > Not unexpected, but still nice. > > All the best, > > Michael Foord > > -- > http://www.voidspace.org.uk > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- From fuzzyman at gmail.com Sat Nov 27 00:22:16 2010 From: fuzzyman at gmail.com (Michael Foord) Date: Fri, 26 Nov 2010 23:22:16 +0000 Subject: [pypy-dev] pypy 1.4 and virtualenv 1.5.1 In-Reply-To: <20101126231940.GJ1009@trillke.net> References: <20101126231940.GJ1009@trillke.net> Message-ID: On 26 November 2010 23:19, holger krekel wrote: > Hi Michael, > > On Fri, Nov 26, 2010 at 22:15 +0000, Michael Foord wrote: > > Congratulations on the new release. > > > > Unfortunately it seems that virtualenv 1.5.1 and pypy 1.4 do not play > well > > together on Mac OS X. At least the "--distribute" option doesn't work, > which > > is needed by tox. > > It's not needed, with tox you can configure "distribute=False" in your test > env > and this would install setuptools. > > > I tried to create an issue on the PyPy issue tracker, but new registrants > > are unable to create issues: > > > > Here is the output on Linux: > > > > $ virtualenv -p /compile/pypy-1.4/bin/pypy pypy --distribute > > --no-site-packages > > Does that work for you without --distribute? > Yes it does. mvt on #pypy has tracked it down to plat-mac/plist.py not being included in pypy. This error is somehow being masked when distribute is installed: 1. ~/src/distribute-0.6.14 $ mkvirtualenv --distribute --no-site-packages --python=/Users/mvantellingen/pypy/bin/pypy pypy-test2 2. Running virtualenv with interpreter /Users/mvantellingen/pypy/bin/pypy 3. New pypy-c executable in pypy-test2/bin/pypy 4. Not overwriting existing pypy-c script pypy-test2/bin/pypy-c (you must use pypy-test2/bin/pypy) 5. Installing distribute...Extracting in /var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2 6. Now working in /var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14 7. Installing Distribute 8. Traceback (most recent call last): 9. File "app_main.py", line 33, in run_toplevel 10. File "setup.py", line 37, in 11. exec(open(init_path).read(), d) 12. File "", line 8, in 13. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/setuptools/__init__.py", line 2, in 14. from setuptools.extension import Extension, Library 15. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/setuptools/extension.py", line 2, in 16. from setuptools.dist import _get_unpatched 17. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/setuptools/dist.py", line 6, in 18. from setuptools.command.install import install 19. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/setuptools/command/__init__.py", line 8, in 20. from setuptools.command import install_scripts 21. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/setuptools/command/install_scripts.py", line 3, in 22. from pkg_resources import Distribution, PathMetadata, ensure_directory 23. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/pkg_resources.py", line 685, in 24. class Environment(object): 25. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/pkg_resources.py", line 688, in Environment 26. def __init__(self, search_path=None, platform=get_supported_platform(), python=PY_MAJOR): 27. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/pkg_resources.py", line 77, in get_supported_platform 28. plat = 'macosx-%s-%s' % ('.'.join(_macosx_vers()[:2]), m.group(3)) 29. File "/private/var/folders/Dk/DkuwVxyEH4CIjPxKNOHaCE+++TI/-Tmp-/tmpjCepU2/distribute-0.6.14/pkg_resources.py", line 192, in _macosx_vers 30. import plistlib 31. ImportError: No module named plistlib 32. Something went wrong during the installation. 33. See the error message above. 34. None 35. done. 36. Complete output from command /Users/mvantellingen/virtualen.../pypy /Users/mvantellingen/virtualen...stall /Library/Python/2.6/site-packa...ar.gz: 37. Traceback (most recent call last): 38. File "app_main.py", line 33, in run_toplevel 39. IOError: [Errno 2] No such file or directory: '/Users/mvantellingen/virtualenvs/pypy-test2/bin/easy_install' 40. ---------------------------------------- 41. Traceback (most recent call last): 42. File "app_main.py", line 33, in run_toplevel 43. File "/Library/Python/2.6/site-packages/virtualenv-1.5.1-py2.6.egg/virtualenv.py", line 1647, in 44. main() 45. File "/Library/Python/2.6/site-packages/virtualenv-1.5.1-py2.6.egg/virtualenv.py", line 558, in main 46. prompt=options.prompt) 47. File "/Library/Python/2.6/site-packages/virtualenv-1.5.1-py2.6.egg/virtualenv.py", line 656, in create_environment 48. install_pip(py_executable) 49. File "/Library/Python/2.6/site-packages/virtualenv-1.5.1-py2.6.egg/virtualenv.py", line 415, in install_pip 50. filter_stdout=_filter_setup) 51. File "/Library/Python/2.6/site-packages/virtualenv-1.5.1-py2.6.egg/virtualenv.py", line 624, in call_subprocess 52. % (cmd, proc.returncode)) 53. OSError: Command ['/Users/mvantellingen/virtualenvs/pypy-test2/bin/pypy', '/Users/mvantellingen/virtualenvs/pypy-test2/bin/easy_install', '/Library/Python/2.6/site-packages/virtualenv-1.5.1-py2.6.egg/virtualenv_support/pip-0.8.1.tar.gz'] failed with error code 1 54. ~/src/distribute-0.6.14 $ All the best, Michael Foord > > holger > > > Running virtualenv with interpreter /compile/pypy-1.4/bin/pypy > > New pypy-c executable in pypy/bin/pypy > > Also creating executable in pypy/bin/pypy-c > > Installing > > > distribute....................................................................................................................................................................................done. > > > > > > Here is the failed output on OS X: > > > > $ virtualenv -p /compile/pypy-1.4-osx/bin/pypy pypy --distribute > > --no-site-packages > > Running virtualenv with interpreter /compile/pypy-1.4-osx/bin/pypy > > New pypy-c executable in pypy/bin/pypy > > Also creating executable in pypy/bin/pypy-c > > Please make sure you remove any previous custom paths from your > > /Volumes/Second Drive/michael/.pydistutils.cfg file. > > Installing distribute................................done. > > Complete output from command /compile/pypy/bin/pypy > > /compile/pypy/bin/easy_install /Library/Frameworks/Python.fra...ar.gz: > > Traceback (most recent call last): > > File "app_main.py", line 33, in run_toplevel > > IOError: [Errno 2] No such file or directory: > '/compile/pypy/bin/easy_install' > > ---------------------------------------- > > Traceback (most recent call last): > > File "app_main.py", line 33, in run_toplevel > > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > > line 1647, in > > main() > > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > > line 558, in main > > prompt=options.prompt) > > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > > line 656, in create_environment > > install_pip(py_executable) > > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > > line 415, in install_pip > > filter_stdout=_filter_setup) > > File > "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/virtualenv.py", > > line 624, in call_subprocess > > % (cmd_desc, proc.returncode)) > > OSError: Command /compile/pypy/bin/pypy /compile/pypy/bin/easy_install > > /Library/Frameworks/Python.fra...ar.gz failed with error code 1 > > > > > > The nice thing is that for unittest2 and mock all tests pass with pypy > 1.4. > > Not unexpected, but still nice. > > > > All the best, > > > > Michael Foord > > > > -- > > http://www.voidspace.org.uk > > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > -- > -- http://www.voidspace.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101126/26d12411/attachment-0001.htm From drsalists at gmail.com Sat Nov 27 01:18:29 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 26 Nov 2010 16:18:29 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <4CF03ED2.3050801@gmail.com> References: <201011160222.oAG2Mrfv004177@theraft.openend.se> <4CF03ED2.3050801@gmail.com> Message-ID: On Fri, Nov 26, 2010 at 3:12 PM, Antonio Cuni wrote: > On 16/11/10 04:30, Dan Stromberg wrote: > > BTW, it might cause confusion down the road to call something that is >> basically like cpython's bsddb (Berkeley DB) by the name "dbm" in pypy's >> library. In the cpython standard library, "dbm" is an interface to ndbm >> databases. These all provide the same dictionary-like interface to Python >> programs, but have somewhat different API's to C, and pretty different, >> incompatible on-disk representations. >> > > Hi Dan, > I played a bit (veeeery quickly) with dbm on both pypy and cpython, and I'm > not sure I get what you mean when you say that our dbm.py is equivalent to > cpython's bsddb. E.g., I can create a db on cpython and open it from pypy, > so it seems that the two modules are compatible. > > Moreover, I checked which libraries the links to. On CPython, it links to > libdb-4.8.so: > > viper2 ~ $ ldd /usr/lib/python2.6/lib-dynload/dbm.so > linux-gate.so.1 => (0x00884000) > libdb-4.8.so => /usr/lib/libdb-4.8.so (0x00110000) > libpthread.so.0 => /lib/libpthread.so.0 (0x003de000) > libc.so.6 => /lib/libc.so.6 (0x003f8000) > /lib/ld-linux.so.2 (0x002e0000) > > the pypy version first tries to open libdb.so, then libdb-4.5.so. I had to > manually modify it to open version 4.8 (I agree that we should find a more > general way to find it), but apart from that what I can see is that it uses > the same underlying wrapper as CPython. > > So, to summarise: could you elaborate a bit more why we should delete > dbm.py from pypy? > > ciao, > Anto > Looks like dbm at the API level: CPython dbm, pypy dbm Looks like dbm on disk: CPython dbm Looks like bsddb at the API level: CPython bsddb Looks like bsddb on disk: CPython bsddb, pypy dbm Don't let the common prefix fool you - libdb is Berkeley DB, while dbm is supposed to be ndbm. http://docs.python.org/library/dbm.html That is, pypy's dbm.py is perfectly self-consistent (other than a couple of likely memory leaks), but if you try to open a database from CPython using pypy's dbm module (or vice-versa), I don't believe it'll work. EG: $ /usr/local/cpython-2.7/bin/python Python 2.7 (r27:82500, Aug 2 2010, 19:15:05) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import dbm >>> d = dbm.open('d', 'n') >>> d['a'] = 'b' >>> d.close() >>> benchbox-dstromberg:/tmp/dbm-test i686-pc-linux-gnu 30890 - above cmd done 2010 Fri Nov 26 04:14 PM $ /usr/local/pypy-1.4/bin/pypy Python 2.5.2 (79529, Nov 25 2010, 20:40:03) [PyPy 1.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``casuality violations and flying'' >>>> import dbm >>>> d = dbm.open('d', 'r') Traceback (most recent call last): File "", line 1, in File "/usr/local/pypy-1.4/lib_pypy/dbm.py", line 172, in open raise error("Could not open file %s.db" % filename) error: Could not open file d.db >>>> HTH -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101126/f2cc569e/attachment.htm From phyo.arkarlwin at gmail.com Sat Nov 27 01:52:43 2010 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Sat, 27 Nov 2010 07:22:43 +0630 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: Message-ID: Grats! i am going to give web2py : http://www.web2py.com a try on pypy On 11/27/10, Samuele Pedroni wrote: > On Fri, Nov 26, 2010 at 7:23 PM, Maciej Fijalkowski > wrote: >> =============================== >> PyPy 1.4: Ouroboros in practice >> =============================== >> >> We're pleased to announce the 1.4 release of PyPy. This is a major >> breakthrough >> in our long journey, as PyPy 1.4 is the first PyPy release that can >> translate >> itself faster than CPython. Starting today, we are using PyPy more for >> our every-day development. So may you :) You can download it here: >> >> http://pypy.org/download.html >> >> What is PyPy >> ============ >> >> PyPy is a very compliant Python interpreter, almost a drop-in replacement >> for CPython. It's fast (`pypy 1.4 and cpython 2.6`_ comparison) >> >> Among its new features, this release includes numerous performance >> improvements >> (which made fast self-hosting possible), a 64-bit JIT backend, as well >> as serious stabilization. As of now, we can consider the 32-bit and 64-bit >> linux versions of PyPy stable enough to run `in production`_. >> >> Numerous speed achievements are described on `our blog`_. Normalized speed >> charts comparing `pypy 1.4 and pypy 1.3`_ as well as `pypy 1.4 and cpython >> 2.6`_ >> are available on benchmark website. For the impatient: yes, we got a lot >> faster! >> > > congratulations! > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From phyo.arkarlwin at gmail.com Sat Nov 27 02:12:18 2010 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Sat, 27 Nov 2010 07:42:18 +0630 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: Message-ID: I just tested a simple hello world page in web2py and i dont see speed improvment yet.. pages comes up in 16-18ms in normal python , and in pypy 22-25 I just run directly via command $pypy web2py.py am i doing something wrong? need some extra command line options? On 11/27/10, Phyo Arkar wrote: > Grats! > i am going to give web2py : http://www.web2py.com a try on pypy > > On 11/27/10, Samuele Pedroni wrote: >> On Fri, Nov 26, 2010 at 7:23 PM, Maciej Fijalkowski >> wrote: >>> =============================== >>> PyPy 1.4: Ouroboros in practice >>> =============================== >>> >>> We're pleased to announce the 1.4 release of PyPy. This is a major >>> breakthrough >>> in our long journey, as PyPy 1.4 is the first PyPy release that can >>> translate >>> itself faster than CPython. Starting today, we are using PyPy more for >>> our every-day development. So may you :) You can download it here: >>> >>> http://pypy.org/download.html >>> >>> What is PyPy >>> ============ >>> >>> PyPy is a very compliant Python interpreter, almost a drop-in >>> replacement >>> for CPython. It's fast (`pypy 1.4 and cpython 2.6`_ comparison) >>> >>> Among its new features, this release includes numerous performance >>> improvements >>> (which made fast self-hosting possible), a 64-bit JIT backend, as well >>> as serious stabilization. As of now, we can consider the 32-bit and >>> 64-bit >>> linux versions of PyPy stable enough to run `in production`_. >>> >>> Numerous speed achievements are described on `our blog`_. Normalized >>> speed >>> charts comparing `pypy 1.4 and pypy 1.3`_ as well as `pypy 1.4 and >>> cpython >>> 2.6`_ >>> are available on benchmark website. For the impatient: yes, we got a lot >>> faster! >>> >> >> congratulations! >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > From santagada at gmail.com Sat Nov 27 02:56:09 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Fri, 26 Nov 2010 23:56:09 -0200 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: Message-ID: On Fri, Nov 26, 2010 at 11:12 PM, Phyo Arkar wrote: > I just tested a simple hello world page in web2py and i dont see speed > improvment yet.. > > pages comes up in 16-18ms in normal python , and in pypy 22-25 > > > I just run directly via command $pypy web2py.py > > am i doing something wrong? need some extra command line options? The pypy jit is always on, what you could do is maybe warm up the page, like making 100 requests and then measuring the next 10 requests would make a more interesting benchmark. -- Leonardo Santagada From phyo.arkarlwin at gmail.com Sat Nov 27 03:09:15 2010 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Sat, 27 Nov 2010 08:39:15 +0630 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: Message-ID: libmagic python fails to work on pypy (python-magic) it uses ctypes and easy-install try to download and instaill it , but it fails. how to enable ctypes on pypy? On 11/27/10, Leonardo Santagada wrote: > On Fri, Nov 26, 2010 at 11:12 PM, Phyo Arkar > wrote: >> I just tested a simple hello world page in web2py and i dont see speed >> improvment yet.. >> >> pages comes up in 16-18ms in normal python , and in pypy 22-25 >> >> >> I just run directly via command $pypy web2py.py >> >> am i doing something wrong? need some extra command line options? > > The pypy jit is always on, what you could do is maybe warm up the > page, like making 100 requests and then measuring the next 10 requests > would make a more interesting benchmark. > > > -- > Leonardo Santagada > From drsalists at gmail.com Sat Nov 27 03:51:57 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 26 Nov 2010 18:51:57 -0800 Subject: [pypy-dev] gdbm In-Reply-To: References: <201011160222.oAG2Mrfv004177@theraft.openend.se> <4CF03ED2.3050801@gmail.com> Message-ID: On Fri, Nov 26, 2010 at 4:18 PM, Dan Stromberg wrote: > > On Fri, Nov 26, 2010 at 3:12 PM, Antonio Cuni wrote: > >> On 16/11/10 04:30, Dan Stromberg wrote: >> >> BTW, it might cause confusion down the road to call something that is >>> basically like cpython's bsddb (Berkeley DB) by the name "dbm" in pypy's >>> library. In the cpython standard library, "dbm" is an interface to ndbm >>> databases. These all provide the same dictionary-like interface to >>> Python >>> programs, but have somewhat different API's to C, and pretty different, >>> incompatible on-disk representations. >>> >> >> Hi Dan, >> I played a bit (veeeery quickly) with dbm on both pypy and cpython, and >> I'm not sure I get what you mean when you say that our dbm.py is equivalent >> to cpython's bsddb. E.g., I can create a db on cpython and open it from >> pypy, so it seems that the two modules are compatible. >> >> Moreover, I checked which libraries the links to. On CPython, it links to >> libdb-4.8.so: >> >> viper2 ~ $ ldd /usr/lib/python2.6/lib-dynload/dbm.so >> linux-gate.so.1 => (0x00884000) >> libdb-4.8.so => /usr/lib/libdb-4.8.so (0x00110000) >> libpthread.so.0 => /lib/libpthread.so.0 (0x003de000) >> libc.so.6 => /lib/libc.so.6 (0x003f8000) >> /lib/ld-linux.so.2 (0x002e0000) >> >> the pypy version first tries to open libdb.so, then libdb-4.5.so. I had >> to manually modify it to open version 4.8 (I agree that we should find a >> more general way to find it), but apart from that what I can see is that it >> uses the same underlying wrapper as CPython. >> >> So, to summarise: could you elaborate a bit more why we should delete >> dbm.py from pypy? >> >> ciao, >> Anto >> > > > Looks like dbm at the API level: CPython dbm, pypy dbm > Looks like dbm on disk: CPython dbm > Looks like bsddb at the API level: CPython bsddb > Looks like bsddb on disk: CPython bsddb, pypy dbm > > Don't let the common prefix fool you - libdb is Berkeley DB, while dbm is > supposed to be ndbm. > > http://docs.python.org/library/dbm.html > > That is, pypy's dbm.py is perfectly self-consistent (other than a couple of > likely memory leaks), but if you try to open a database from CPython using > pypy's dbm module (or vice-versa), I don't believe it'll work. EG: > > $ /usr/local/cpython-2.7/bin/python > Python 2.7 (r27:82500, Aug 2 2010, 19:15:05) > [GCC 4.4.3] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import dbm > >>> d = dbm.open('d', 'n') > >>> d['a'] = 'b' > >>> d.close() > >>> > benchbox-dstromberg:/tmp/dbm-test i686-pc-linux-gnu 30890 - above cmd done > 2010 Fri Nov 26 04:14 PM > > $ /usr/local/pypy-1.4/bin/pypy > Python 2.5.2 (79529, Nov 25 2010, 20:40:03) > [PyPy 1.4.0] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > And now for something completely different: ``casuality violations and > flying'' > >>>> import dbm > >>>> d = dbm.open('d', 'r') > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/pypy-1.4/lib_pypy/dbm.py", line 172, in open > raise error("Could not open file %s.db" % filename) > error: Could not open file d.db > >>>> > > HTH > > Interesting. My CPython 2.7 build has: $ ldd dbm.so linux-gate.so.1 => (0x009ed000) libgdbm.so.3 => /usr/lib/libgdbm.so.3 (0x00ed5000) libgdbm_compat.so.3 => /usr/lib/libgdbm_compat.so.3 (0x00269000) libpthread.so.0 => /lib/libpthread.so.0 (0x00df3000) libc.so.6 => /lib/libc.so.6 (0x00425000) /lib/ld-linux.so.2 (0x00b7b000) benchbox-dstromberg:/usr/local/cpython-2.7/lib/python2.7/lib-dynload i686-pc-linux-gnu 30430 - above cmd done 2010 Fri Nov 26 06:48 PM ...but http://docs.python.org/library/dbm.html plainly says it should be ndbm. So which is wrong? The doc, or the module that's picking gdbm or Berkeley DB, as it sees fit? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101126/a352b5be/attachment-0001.htm From drsalists at gmail.com Sat Nov 27 04:07:49 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 26 Nov 2010 19:07:49 -0800 Subject: [pypy-dev] gdbm In-Reply-To: References: <201011160222.oAG2Mrfv004177@theraft.openend.se> <4CF03ED2.3050801@gmail.com> Message-ID: On Fri, Nov 26, 2010 at 6:51 PM, Dan Stromberg wrote: > > On Fri, Nov 26, 2010 at 4:18 PM, Dan Stromberg wrote: > >> >> On Fri, Nov 26, 2010 at 3:12 PM, Antonio Cuni wrote: >> >>> On 16/11/10 04:30, Dan Stromberg wrote: >>> >>> BTW, it might cause confusion down the road to call something that is >>>> basically like cpython's bsddb (Berkeley DB) by the name "dbm" in pypy's >>>> library. In the cpython standard library, "dbm" is an interface to ndbm >>>> databases. These all provide the same dictionary-like interface to >>>> Python >>>> programs, but have somewhat different API's to C, and pretty different, >>>> incompatible on-disk representations. >>>> >>> >>> Hi Dan, >>> I played a bit (veeeery quickly) with dbm on both pypy and cpython, and >>> I'm not sure I get what you mean when you say that our dbm.py is equivalent >>> to cpython's bsddb. E.g., I can create a db on cpython and open it from >>> pypy, so it seems that the two modules are compatible. >>> >>> Moreover, I checked which libraries the links to. On CPython, it links to >>> libdb-4.8.so: >>> >>> viper2 ~ $ ldd /usr/lib/python2.6/lib-dynload/dbm.so >>> linux-gate.so.1 => (0x00884000) >>> libdb-4.8.so => /usr/lib/libdb-4.8.so (0x00110000) >>> libpthread.so.0 => /lib/libpthread.so.0 (0x003de000) >>> libc.so.6 => /lib/libc.so.6 (0x003f8000) >>> /lib/ld-linux.so.2 (0x002e0000) >>> >>> the pypy version first tries to open libdb.so, then libdb-4.5.so. I had >>> to manually modify it to open version 4.8 (I agree that we should find a >>> more general way to find it), but apart from that what I can see is that it >>> uses the same underlying wrapper as CPython. >>> >>> So, to summarise: could you elaborate a bit more why we should delete >>> dbm.py from pypy? >>> >>> ciao, >>> Anto >>> >> >> >> Looks like dbm at the API level: CPython dbm, pypy dbm >> Looks like dbm on disk: CPython dbm >> Looks like bsddb at the API level: CPython bsddb >> Looks like bsddb on disk: CPython bsddb, pypy dbm >> >> Don't let the common prefix fool you - libdb is Berkeley DB, while dbm is >> supposed to be ndbm. >> >> http://docs.python.org/library/dbm.html >> >> That is, pypy's dbm.py is perfectly self-consistent (other than a couple >> of likely memory leaks), but if you try to open a database from CPython >> using pypy's dbm module (or vice-versa), I don't believe it'll work. EG: >> >> $ /usr/local/cpython-2.7/bin/python >> Python 2.7 (r27:82500, Aug 2 2010, 19:15:05) >> [GCC 4.4.3] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import dbm >> >>> d = dbm.open('d', 'n') >> >>> d['a'] = 'b' >> >>> d.close() >> >>> >> benchbox-dstromberg:/tmp/dbm-test i686-pc-linux-gnu 30890 - above cmd done >> 2010 Fri Nov 26 04:14 PM >> >> $ /usr/local/pypy-1.4/bin/pypy >> Python 2.5.2 (79529, Nov 25 2010, 20:40:03) >> [PyPy 1.4.0] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> And now for something completely different: ``casuality violations and >> flying'' >> >>>> import dbm >> >>>> d = dbm.open('d', 'r') >> Traceback (most recent call last): >> File "", line 1, in >> File "/usr/local/pypy-1.4/lib_pypy/dbm.py", line 172, in open >> raise error("Could not open file %s.db" % filename) >> error: Could not open file d.db >> >>>> >> >> HTH >> >> Interesting. My CPython 2.7 build has: > > $ ldd dbm.so > linux-gate.so.1 => (0x009ed000) > libgdbm.so.3 => /usr/lib/libgdbm.so.3 (0x00ed5000) > libgdbm_compat.so.3 => /usr/lib/libgdbm_compat.so.3 (0x00269000) > libpthread.so.0 => /lib/libpthread.so.0 (0x00df3000) > libc.so.6 => /lib/libc.so.6 (0x00425000) > /lib/ld-linux.so.2 (0x00b7b000) > benchbox-dstromberg:/usr/local/cpython-2.7/lib/python2.7/lib-dynload > i686-pc-linux-gnu 30430 - above cmd done 2010 Fri Nov 26 06:48 PM > > ...but http://docs.python.org/library/dbm.html plainly says it should be > ndbm. > > So which is wrong? The doc, or the module that's picking gdbm or Berkeley > DB, as it sees fit? > > But then there's this at the URL above: This module can be used with the ?classic? ndbm interface, the BSD DB compatibility interface, or the GNU GDBM compatibility interface. On Unix, the *configure* script will attempt to locate the appropriate header file to simplify building this module. I suppose that means that if it can't find ndbm (which at one time was hard due to licensing, but last I heard it'd become readily available), it's free to pretend it has ndbm using something else. I'd call that puzzlingly worded - it's not the interface that's changing, but the backend implementation. But perhaps dbm.py is free to use Berkeley DB if it prefers to pretend it can never find ndbm. And perhaps I shouldn't have skimmed that page so quickly ^_^ CPython 2.7's configure script has: --with-dbmliborder=db1:db2:... order to check db backends for dbm. Valid value is a colon separated string with the backend names `ndbm', `gdbm' and `bdb'. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101126/c4a1d226/attachment.htm From ademan555 at gmail.com Sat Nov 27 05:47:21 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Fri, 26 Nov 2010 20:47:21 -0800 Subject: [pypy-dev] test_all.py -A behavior In-Reply-To: References: Message-ID: I'm having trouble running the applevel tests on a translated pypy-c (without jit). I have this problem with recent trunk and my psycopg2 compatibility branch. Translating both with: python translate.py targetpypystandalone.py --withmod-cpyext and running tests with: translator/goal/pypy-c test_all.py -A both trunk and my branch have multiple errors on module/bz2/test/test_bz2_compdecomp.py tests, one failure in module/posix/test/test_posix2.py and both hang on the 8th test on module/signal/test/test_signal.py . This wouldn't be too weird if all three of those test suites run successfully via: translator/goal/pypy-c test_all.py -A $SUITE on both my branch and trunk. I'm on Ubuntu 10.04 32 bit. I tried again with trunk just now with the same behavior. Is this a test_all.py bug? Can anyone with a pypy-c confirm this behavior? Cheers, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101126/f8c43e20/attachment.htm From anto.cuni at gmail.com Sat Nov 27 07:17:59 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 27 Nov 2010 07:17:59 +0100 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: Message-ID: <4CF0A297.9090202@gmail.com> On 27/11/10 03:09, Phyo Arkar wrote: > libmagic python fails to work on pypy (python-magic) > > it uses ctypes and easy-install try to download and instaill it , but it fails. > > how to enable ctypes on pypy? Hi Phyo, ctypes *is* enabled on pypy by default. If python-magic does not work, it can either: 1) be a pypy bug: please report it as an issue (possibly with a simple failing test and the full traceack) 2) a python-magic issue, e.g. if it plays dirty ctypes trick that cannot really be supported by pypy due to the internal differences with CPython ciao, Anto From phyo.arkarlwin at gmail.com Sun Nov 28 09:57:26 2010 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Sun, 28 Nov 2010 15:27:26 +0630 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: <4CF0A297.9090202@gmail.com> References: <4CF0A297.9090202@gmail.com> Message-ID: i got python-magic working , after i installed without easy_install (easy_install fail because it tried to install ctypes). Now what is not working is python-lxml , which is very important for my project. here are the errors: Running lxml-2.3beta1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Gg3GRA/lxml-2.3beta1/egg-dist-tmp-bwUkM2 Building lxml version 2.3.beta1. NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib src/lxml/lxml.etree.c:75: error: conflicting types for ?Py_buffer? /home/v3ss/pypy-1.4/include/object.h:19: note: previous declaration of ?Py_buffer? was here Had Anyone got lxml working in pypy successfully? On 11/27/10, Antonio Cuni wrote: > On 27/11/10 03:09, Phyo Arkar wrote: >> libmagic python fails to work on pypy (python-magic) >> >> it uses ctypes and easy-install try to download and instaill it , but it >> fails. >> >> how to enable ctypes on pypy? > > Hi Phyo, > ctypes *is* enabled on pypy by default. > > If python-magic does not work, it can either: > > 1) be a pypy bug: please report it as an issue (possibly with a simple > failing > test and the full traceack) > > 2) a python-magic issue, e.g. if it plays dirty ctypes trick that cannot > really be supported by pypy due to the internal differences with CPython > > ciao, > Anto > From fijall at gmail.com Sun Nov 28 10:48:17 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 28 Nov 2010 11:48:17 +0200 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: <4CF0A297.9090202@gmail.com> Message-ID: Hey. On Sun, Nov 28, 2010 at 10:57 AM, Phyo Arkar wrote: > i got python-magic working , after i installed without easy_install > (easy_install fail because it tried to install ctypes). great > > Now what is not working is python-lxml , which is very important for my project. lxml won't work out of the box. if you think it's important enough, you can try to port cython to generate something saner (right now what it generates won't work on cpyext). Cheers, fijal > > here are the errors: > > Running lxml-2.3beta1/setup.py -q bdist_egg --dist-dir > /tmp/easy_install-Gg3GRA/lxml-2.3beta1/egg-dist-tmp-bwUkM2 > Building lxml version 2.3.beta1. > NOTE: Trying to build without Cython, pre-generated > 'src/lxml/lxml.etree.c' needs to be available. > Using build configuration of libxslt 1.1.26 > Building against libxml2/libxslt in the following directory: /usr/lib > src/lxml/lxml.etree.c:75: error: conflicting types for ?Py_buffer? > /home/v3ss/pypy-1.4/include/object.h:19: note: previous declaration of > ?Py_buffer? was here > > > Had Anyone got lxml working in pypy successfully? > > On 11/27/10, Antonio Cuni wrote: >> On 27/11/10 03:09, Phyo Arkar wrote: >>> libmagic python fails to work on pypy (python-magic) >>> >>> it uses ctypes and easy-install try to download and instaill it , but it >>> fails. >>> >>> how to enable ctypes on pypy? >> >> Hi Phyo, >> ctypes *is* enabled on pypy by default. >> >> If python-magic does not work, it can either: >> >> 1) be a pypy bug: please report it as an issue (possibly with a simple >> failing >> test and the full traceack) >> >> 2) a python-magic issue, e.g. if it plays dirty ctypes trick that cannot >> really be supported by pypy due to the internal differences with CPython >> >> ciao, >> Anto >> > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From renesd at gmail.com Sun Nov 28 10:58:33 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Sun, 28 Nov 2010 09:58:33 +0000 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) Message-ID: Hi, what xml libraries are people using with pypy? What is working well? cu, On Sun, Nov 28, 2010 at 9:48 AM, Maciej Fijalkowski wrote: > Hey. > > On Sun, Nov 28, 2010 at 10:57 AM, Phyo Arkar > wrote: > > i got python-magic working , after i installed without easy_install > > (easy_install fail because it tried to install ctypes). > > great > > > > > Now what is not working is python-lxml , which is very important for my > project. > > lxml won't work out of the box. if you think it's important enough, > you can try to port cython to generate something saner (right now what > it generates won't work on cpyext). > > Cheers, > fijal > > > > > here are the errors: > > > > Running lxml-2.3beta1/setup.py -q bdist_egg --dist-dir > > /tmp/easy_install-Gg3GRA/lxml-2.3beta1/egg-dist-tmp-bwUkM2 > > Building lxml version 2.3.beta1. > > NOTE: Trying to build without Cython, pre-generated > > 'src/lxml/lxml.etree.c' needs to be available. > > Using build configuration of libxslt 1.1.26 > > Building against libxml2/libxslt in the following directory: /usr/lib > > src/lxml/lxml.etree.c:75: error: conflicting types for ?Py_buffer? > > /home/v3ss/pypy-1.4/include/object.h:19: note: previous declaration of > > ?Py_buffer? was here > > > > > > Had Anyone got lxml working in pypy successfully? > > > > On 11/27/10, Antonio Cuni wrote: > >> On 27/11/10 03:09, Phyo Arkar wrote: > >>> libmagic python fails to work on pypy (python-magic) > >>> > >>> it uses ctypes and easy-install try to download and instaill it , but > it > >>> fails. > >>> > >>> how to enable ctypes on pypy? > >> > >> Hi Phyo, > >> ctypes *is* enabled on pypy by default. > >> > >> If python-magic does not work, it can either: > >> > >> 1) be a pypy bug: please report it as an issue (possibly with a simple > >> failing > >> test and the full traceack) > >> > >> 2) a python-magic issue, e.g. if it plays dirty ctypes trick that cannot > >> really be supported by pypy due to the internal differences with CPython > >> > >> ciao, > >> Anto > >> > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101128/61b91832/attachment-0001.htm From fijall at gmail.com Sun Nov 28 11:09:25 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 28 Nov 2010 12:09:25 +0200 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: On Sun, Nov 28, 2010 at 11:58 AM, Ren? Dudfield wrote: > Hi, > > what xml libraries are people using with pypy?? What is working well? > > cu, PyExpat works, although it's slow (ctypes-based implementation). I know genshi has some troubles with it, someone is debugging now. Besides I don't think there are any working (unless someone wrote a pure-python one) Cheers, fijal > > > On Sun, Nov 28, 2010 at 9:48 AM, Maciej Fijalkowski > wrote: >> >> Hey. >> >> On Sun, Nov 28, 2010 at 10:57 AM, Phyo Arkar >> wrote: >> > i got python-magic working , after i installed without easy_install >> > (easy_install fail because it tried to install ctypes). >> >> great >> >> > >> > Now what is not working is python-lxml , which is very important for my >> > project. >> >> lxml won't work out of the box. if you think it's important enough, >> you can try to port cython to generate something saner (right now what >> it generates won't work on cpyext). >> >> Cheers, >> fijal >> >> > >> > here are the errors: >> > >> > Running lxml-2.3beta1/setup.py -q bdist_egg --dist-dir >> > /tmp/easy_install-Gg3GRA/lxml-2.3beta1/egg-dist-tmp-bwUkM2 >> > Building lxml version 2.3.beta1. >> > NOTE: Trying to build without Cython, pre-generated >> > 'src/lxml/lxml.etree.c' needs to be available. >> > Using build configuration of libxslt 1.1.26 >> > Building against libxml2/libxslt in the following directory: /usr/lib >> > src/lxml/lxml.etree.c:75: error: conflicting types for ?Py_buffer? >> > /home/v3ss/pypy-1.4/include/object.h:19: note: previous declaration of >> > ?Py_buffer? was here >> > >> > >> > Had Anyone got lxml working in pypy successfully? >> > >> > On 11/27/10, Antonio Cuni wrote: >> >> On 27/11/10 03:09, Phyo Arkar wrote: >> >>> libmagic python fails to work on pypy (python-magic) >> >>> >> >>> it uses ctypes and easy-install try to download and instaill it , but >> >>> it >> >>> fails. >> >>> >> >>> how to enable ctypes on pypy? >> >> >> >> Hi Phyo, >> >> ctypes *is* enabled on pypy by default. >> >> >> >> If python-magic does not work, it can either: >> >> >> >> 1) be a pypy bug: please report it as an issue (possibly with a simple >> >> failing >> >> test and the full traceack) >> >> >> >> 2) a python-magic issue, e.g. if it plays dirty ctypes trick that >> >> cannot >> >> really be supported by pypy due to the internal differences with >> >> CPython >> >> >> >> ciao, >> >> Anto >> >> >> > _______________________________________________ >> > pypy-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/pypy-dev >> > >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From amauryfa at gmail.com Sun Nov 28 11:44:40 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sun, 28 Nov 2010 11:44:40 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: Hi 2010/11/28 Maciej Fijalkowski > On Sun, Nov 28, 2010 at 11:58 AM, Ren? Dudfield wrote: > > what xml libraries are people using with pypy? What is working well? > > PyExpat works, although it's slow (ctypes-based implementation). I > know genshi has some troubles with it, someone is debugging now. > Besides I don't think there are any working (unless someone wrote a > pure-python one) > PyExpat is now a built-in module, implemented in RPython, and should have reasonable performance. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101128/5074733d/attachment.htm From anto.cuni at gmail.com Sun Nov 28 12:10:19 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 28 Nov 2010 12:10:19 +0100 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: References: <4CF0A297.9090202@gmail.com> Message-ID: <4CF2389B.90801@gmail.com> On 28/11/10 10:48, Maciej Fijalkowski wrote: >> > i got python-magic working , after i installed without easy_install >> > (easy_install fail because it tried to install ctypes). > great well, that's still a bit strange. Why does easy_install try to install ctypes, given that it's in the stdlib? From renesd at gmail.com Sun Nov 28 13:19:30 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Sun, 28 Nov 2010 12:19:30 +0000 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: <4CF2389B.90801@gmail.com> References: <4CF0A297.9090202@gmail.com> <4CF2389B.90801@gmail.com> Message-ID: Maybe it's doing a version comparison? On Sun, Nov 28, 2010 at 11:10 AM, Antonio Cuni wrote: > On 28/11/10 10:48, Maciej Fijalkowski wrote: > >> > i got python-magic working , after i installed without easy_install > >> > (easy_install fail because it tried to install ctypes). > > > great > > > well, that's still a bit strange. Why does easy_install try to install > ctypes, > given that it's in the stdlib? > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101128/a1b0f80b/attachment.htm From anto.cuni at gmail.com Mon Nov 29 10:00:09 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 29 Nov 2010 10:00:09 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: <201011160222.oAG2Mrfv004177@theraft.openend.se> <4CF03ED2.3050801@gmail.com> Message-ID: <4CF36B99.9080104@gmail.com> On 27/11/10 04:07, Dan Stromberg wrote: > This module can be used with the ?classic? ndbm interface, the BSD DB > compatibility interface, or the GNU GDBM compatibility interface. On Unix, the > *configure* script will attempt to locate the appropriate header file to > simplify building this module. > > I suppose that means that if it can't find ndbm (which at one time was hard > due to licensing, but last I heard it'd become readily available), it's free > to pretend it has ndbm using something else. > > I'd call that puzzlingly worded - it's not the interface that's changing, but > the backend implementation. But perhaps dbm.py is free to use Berkeley DB if > it prefers to pretend it can never find ndbm. And perhaps I shouldn't have > skimmed that page so quickly ^_^ > > CPython 2.7's configure script has: > > --with-dbmliborder=db1:db2:... > order to check db backends for dbm. Valid value is a > colon separated string with the backend names > `ndbm', `gdbm' and `bdb'. > > so, having a dbm.py which links to libdb is fine, and it's also what you get with cpython on ubuntu. There is still the issue of how to find the correct library name, as it seems it can vary (it was db-4.5 when the module was written, it's db-4.8 nowadays), but this is a bit orthogonal to what we are discussing here. To summarize, I think we should keep the current dbm.py to link against libdb, and integrate your gdbm.py to link against libgdbm. But before merging it to trunk, I'd like to solve the issue of code duplication between the two modules. ciao, Anto From arigo at tunes.org Mon Nov 29 11:32:59 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 29 Nov 2010 11:32:59 +0100 Subject: [pypy-dev] PyPy-JIT memory usage compared to CPython? In-Reply-To: <160969.12027.qm@web111315.mail.gq1.yahoo.com> References: <160969.12027.qm@web111315.mail.gq1.yahoo.com> Message-ID: Hi, On Sat, Nov 27, 2010 at 12:03 AM, Andy wrote: > I'm particularly interested in the memory usage of PyPy-JIT vs. CPython when running Django. In general would PyPy-JIT increase or decrease memory usage? It depends on the kind of application. PyPy with a JIT may get to 1.5x or 2x more than CPython, but it may also be smaller than CPython if your program contains mainly instances and you didn't feel like carefully putting __slots__ everywhere. A bient?t, Armin. From arigo at tunes.org Mon Nov 29 11:38:01 2010 From: arigo at tunes.org (Armin Rigo) Date: Mon, 29 Nov 2010 11:38:01 +0100 Subject: [pypy-dev] test_all.py -A behavior In-Reply-To: References: Message-ID: Hi Dan, On Sat, Nov 27, 2010 at 5:47 AM, Dan Roberts wrote: > I'm having trouble running the applevel tests on a translated pypy-c > (without jit).? I have this problem with recent trunk and my psycopg2 > compatibility branch. Translating both with: > > python translate.py targetpypystandalone.py --withmod-cpyext > > and running tests with: > > translator/goal/pypy-c test_all.py -A We run these tests nightly, and they seem to pass. You'll have to come to IRC to discuss this, or else post much more details about your problem -- e.g. does it work in a plain trunk without your compatibility branch? A bient?t, Armin. From wlavrijsen at lbl.gov Mon Nov 29 14:04:22 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Mon, 29 Nov 2010 05:04:22 -0800 (PST) Subject: [pypy-dev] double trouble with JIT Message-ID: Hi, with the Reflex branch, and the fast path enabled, I sometimes run into pbs with doubles in libffi. Sometimes the results are wrong, sometimes there's a crash, and sometimes there's: self = def _ctypes_storage_was_allocated(self): addr = ctypes.cast(self._storage, ctypes.c_void_p).value if addr in ALLOCATED: > raise Exception("internal ll2ctypes error - " "double conversion from lltype to ctypes?") E Exception: internal ll2ctypes error - double conversion from lltype to ctypes? Any ideas? Thanks, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From cfbolz at gmx.de Mon Nov 29 14:12:04 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 29 Nov 2010 14:12:04 +0100 Subject: [pypy-dev] double trouble with JIT In-Reply-To: References: Message-ID: <4CF3A6A4.7080607@gmx.de> On 11/29/2010 02:04 PM, wlavrijsen at lbl.gov wrote: > Hi, > > with the Reflex branch, and the fast path enabled, I sometimes run into > pbs with doubles in libffi. Sometimes the results are wrong, sometimes > there's a crash, and sometimes there's: > > self = > > def _ctypes_storage_was_allocated(self): > addr = ctypes.cast(self._storage, ctypes.c_void_p).value > if addr in ALLOCATED: >> raise Exception("internal ll2ctypes error - " > "double conversion from lltype to ctypes?") > E Exception: internal ll2ctypes error - double conversion from lltype to ctypes? Could you check in a test that shows the behaviour so that I can reproduce that? It is definitely possible that there is a bug in one of the layers that has nothing to do with cppyy. Cheers, Carl Friedrich From wlavrijsen at lbl.gov Mon Nov 29 14:17:12 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Mon, 29 Nov 2010 05:17:12 -0800 (PST) Subject: [pypy-dev] double trouble with JIT In-Reply-To: <4CF3A6A4.7080607@gmx.de> References: <4CF3A6A4.7080607@gmx.de> Message-ID: Hi Carl, > Could you check in a test that shows the behaviour so that I can > reproduce that? It is definitely possible that there is a bug in one of > the layers that has nothing to do with cppyy. simply run: $ cd pypy/module/cppyy/test $ pypy-c-jit ../../../test_all.py -x test_cppyy.py You may have to run it a few times to get the exception, rather than any of the other errors. Thanks, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From stefan_ml at behnel.de Mon Nov 29 14:40:49 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 29 Nov 2010 14:40:49 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: Amaury Forgeot d'Arc, 28.11.2010 11:44: > 2010/11/28 Maciej Fijalkowski > >> On Sun, Nov 28, 2010 at 11:58 AM, Ren? Dudfield wrote: >>> what xml libraries are people using with pypy? What is working well? >> >> PyExpat works, although it's slow (ctypes-based implementation). I >> know genshi has some troubles with it, someone is debugging now. >> Besides I don't think there are any working (unless someone wrote a >> pure-python one) > > PyExpat is now a built-in module, implemented in RPython, > and should have reasonable performance. Hmm, reasonable? $ ./bin/pypy -m timeit -s 'import xml.etree.ElementTree as ET' \ 'ET.parse("ot.xml")' 10 loops, best of 3: 1.27 sec per loop $ python2.7 -m timeit -s 'import xml.etree.ElementTree as ET' \ 'ET.parse("ot.xml")' 10 loops, best of 3: 486 msec per loop $ python2.7 -m timeit -s 'import xml.etree.cElementTree as ET' \ 'ET.parse("ot.xml")' 10 loops, best of 3: 33.7 msec per loop Stefan From cfbolz at gmx.de Mon Nov 29 14:50:59 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 29 Nov 2010 14:50:59 +0100 Subject: [pypy-dev] double trouble with JIT In-Reply-To: References: <4CF3A6A4.7080607@gmx.de> Message-ID: <4CF3AFC3.1040703@gmx.de> Hi Wim, On 11/29/2010 02:17 PM, wlavrijsen at lbl.gov wrote: >> Could you check in a test that shows the behaviour so that I can >> reproduce that? It is definitely possible that there is a bug in one of >> the layers that has nothing to do with cppyy. > > simply run: > > $ cd pypy/module/cppyy/test > $ pypy-c-jit ../../../test_all.py -x test_cppyy.py > > You may have to run it a few times to get the exception, rather than any of > the other errors. Given that we are not running PyPy's test suite on PyPy nightly currently, I think it would be safer to keep running the tests on top of CPython. Running the tests needs ctypes, and I don't quite trust PyPy's implementation of that yet (there are even known bugs that have not been fixed on trunk but only on a branch). You should still be able to use pypy-c-jit for translation, because for translation (and after translation) you won't need ctypes that much. Cheers, Carl Friedrich From renesd at gmail.com Mon Nov 29 14:52:07 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Mon, 29 Nov 2010 13:52:07 +0000 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: Ah, does etree work in pypy? That's just python right? On Mon, Nov 29, 2010 at 1:40 PM, Stefan Behnel wrote: > Amaury Forgeot d'Arc, 28.11.2010 11:44: > > 2010/11/28 Maciej Fijalkowski > > > >> On Sun, Nov 28, 2010 at 11:58 AM, Ren? Dudfield wrote: > >>> what xml libraries are people using with pypy? What is working well? > >> > >> PyExpat works, although it's slow (ctypes-based implementation). I > >> know genshi has some troubles with it, someone is debugging now. > >> Besides I don't think there are any working (unless someone wrote a > >> pure-python one) > > > > PyExpat is now a built-in module, implemented in RPython, > > and should have reasonable performance. > > Hmm, reasonable? > > $ ./bin/pypy -m timeit -s 'import xml.etree.ElementTree as ET' \ > 'ET.parse("ot.xml")' > 10 loops, best of 3: 1.27 sec per loop > > $ python2.7 -m timeit -s 'import xml.etree.ElementTree as ET' \ > 'ET.parse("ot.xml")' > 10 loops, best of 3: 486 msec per loop > > $ python2.7 -m timeit -s 'import xml.etree.cElementTree as ET' \ > 'ET.parse("ot.xml")' > 10 loops, best of 3: 33.7 msec per loop > > > Stefan > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101129/c44423f2/attachment.htm From wlavrijsen at lbl.gov Mon Nov 29 14:55:48 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Mon, 29 Nov 2010 05:55:48 -0800 (PST) Subject: [pypy-dev] double trouble with JIT In-Reply-To: <4CF3AFC3.1040703@gmx.de> References: <4CF3A6A4.7080607@gmx.de> <4CF3AFC3.1040703@gmx.de> Message-ID: Hi Carl, well, the real problem started when trying to use pythonify.py as that gives me a segfault in: Program received signal SIGSEGV, Segmentation fault. 0x0865b98c in pypy_g_BuiltinActivation_UwS_W_CPPOverload__run () (gdb) where #0 0x0865b98c in pypy_g_BuiltinActivation_UwS_W_CPPOverload__run () #1 0xb7933fe4 in ?? () #2 0x00000000 in ?? () I then moved to the test and the segfault there (if the exception is not seen) is the same. Hence I was hoping that the exception provided some insight into the source of the segfault. The benchmark uses .invoke(), so bypasses pythonify, but for a demo, I prefer the syntax that pythonify offers I'll do some more digging first. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From stefan_ml at behnel.de Mon Nov 29 14:58:01 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 29 Nov 2010 14:58:01 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: Ren? Dudfield, 29.11.2010 14:52: > On Mon, Nov 29, 2010 at 1:40 PM, Stefan Behnel wrote: >> Amaury Forgeot d'Arc, 28.11.2010 11:44: >>> 2010/11/28 Maciej Fijalkowski >>> >>>> On Sun, Nov 28, 2010 at 11:58 AM, Ren? Dudfield wrote: >>>>> what xml libraries are people using with pypy? What is working well? >>>> >>>> PyExpat works, although it's slow (ctypes-based implementation). I >>>> know genshi has some troubles with it, someone is debugging now. >>>> Besides I don't think there are any working (unless someone wrote a >>>> pure-python one) >>> >>> PyExpat is now a built-in module, implemented in RPython, >>> and should have reasonable performance. >> >> Hmm, reasonable? >> >> $ ./bin/pypy -m timeit -s 'import xml.etree.ElementTree as ET' \ >> 'ET.parse("ot.xml")' >> 10 loops, best of 3: 1.27 sec per loop >> >> $ python2.7 -m timeit -s 'import xml.etree.ElementTree as ET' \ >> 'ET.parse("ot.xml")' >> 10 loops, best of 3: 486 msec per loop >> >> $ python2.7 -m timeit -s 'import xml.etree.cElementTree as ET' \ >> 'ET.parse("ot.xml")' >> 10 loops, best of 3: 33.7 msec per loop > > Ah, does etree work in pypy? That's just python right? Yes, ET is plain Python. cET is not, though, as the name indicates. Stefan From alex.nanou at gmail.com Mon Nov 29 15:04:08 2010 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Mon, 29 Nov 2010 17:04:08 +0300 Subject: [pypy-dev] a possible leak in the object namespace... Message-ID: Hi All, Decided to test the 1.4 with a couple of my usecases, got quite surprising results... To keep it short, I have several usecases that involve replacing the object dictionary. Here's a simple example: ---cut--- class D(dict): def __setitem__(self, key, value): print 'updating attr "%s" to "%s"...' % (key, value) super(D, self).__setitem__(key, value) class X(object): pass x = X() x.__dict__ = D() x.a = 1 # will print a nice log string... # NOTE: this will not print anything in CPython (see P.S.) --uncut-- With the release of version 1.4, I decided to test these usecases out and benchmark them on PyPy and 15 minutes later I got results that were surprising to say the least... Expectations: 1) the normal/native namespace should have been a bit faster than the hooked object on the first run. Both cases should have leveled to about the same performance after the JIT finished it's job +/- a constant. 2) all times should have been near constant. What I got per point: 1) the object with native dict was slower by about three orders of magnitude than the object with a hooked namespace. 2) sequential write benchmark runs on the normal object did not level out, as they did with the hook, rather, they exhibited exponential times (!!) For details and actual test code see the attached file. P.S. This specific functionality is a weak point in CPython for two reasons: 1) writing to .__dict__ demands a subclass of dict (not a real problem, though over-restrictive in my view) 2) all interactions with the namespace are done directly via the Python C-API, completely neglecting the high-level dict interface. Thanks! -- Alex. -------------- next part -------------- A non-text attachment was scrubbed... Name: object_ns_hotwire.py Type: application/octet-stream Size: 2993 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101129/c2210c6b/attachment.obj From cfbolz at gmx.de Mon Nov 29 19:46:11 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 29 Nov 2010 19:46:11 +0100 Subject: [pypy-dev] a possible leak in the object namespace... In-Reply-To: References: Message-ID: <4CF3F4F3.9020204@gmx.de> Hi Alex, On 11/29/2010 03:04 PM, Alex A. Naanou wrote: > With the release of version 1.4, I decided to test these usecases out > and benchmark them on PyPy and 15 minutes later I got results that > were surprising to say the least... > > Expectations: > 1) the normal/native namespace should have been a bit faster than the > hooked object on the first run. Both cases should have leveled to > about the same performance after the JIT finished it's job +/- a > constant. > 2) all times should have been near constant. > > What I got per point: > 1) the object with native dict was slower by about three orders of > magnitude than the object with a hooked namespace. > 2) sequential write benchmark runs on the normal object did not level > out, as they did with the hook, rather, they exhibited exponential > times (!!) Don't do that then :-). > For details and actual test code see the attached file. The code you are trying is essentially this: def test(obj, count=10000): t0 = time.time() for i in xrange(count): setattr(obj, 'a' + str(i), i) t1 = time.time() # return: total, time per write return t1 - t0, (t1 - t0)/count This is not working very well with the non-overridden dict, because we don't optimize for this case at all. You are using a) lots of attributes, which we expect to be rare b) access them with setattr, which is a lot slower than a fixed attribute c) access a different attribute every loop iteration, which means the compiler has to produce one bit of code for every attribute Read this, for some hints why this is the case: http://morepypy.blogspot.com/2010/11/efficiently-implementing-python-objects.html This is in theory fixable with enough work, but I am not sure that this is a common or useful use case. If you really need to do this, just use a normal dictionary. Or show me some real-world code that does this, and might think about the case some more. Anyway, the timing behavior of the above loop is merely quadratic in the number of attributes, not exponential :-). Cheers, Carl Friedrich From alex.nanou at gmail.com Mon Nov 29 21:02:19 2010 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Mon, 29 Nov 2010 23:02:19 +0300 Subject: [pypy-dev] a possible leak in the object namespace... In-Reply-To: <4CF3F4F3.9020204@gmx.de> References: <4CF3F4F3.9020204@gmx.de> Message-ID: On Mon, Nov 29, 2010 at 21:46, Carl Friedrich Bolz wrote: > Hi Alex, > > On 11/29/2010 03:04 PM, Alex A. Naanou wrote: >> With the release of version 1.4, I decided to test these usecases out >> and benchmark them on PyPy and 15 minutes later I got results that >> were surprising to say the least... >> >> Expectations: >> 1) the normal/native namespace should have been a bit faster than the >> hooked object on the first run. Both cases should have leveled to >> about the same performance after the JIT finished it's job +/- a >> constant. >> 2) all times should have been near constant. >> >> What I got per point: >> 1) the object with native dict was slower by about three orders of >> magnitude than the object with a hooked namespace. >> 2) sequential write benchmark runs on the normal object did not level >> out, as they did with the hook, rather, they exhibited exponential >> times (!!) > > Don't do that then :-). :) > > >> For details and actual test code see the attached file. > > The code you are trying is essentially this: > > def test(obj, count=10000): > ? ? ? ?t0 = time.time() > ? ? ? ?for i in xrange(count): > ? ? ? ? ? ? ? ?setattr(obj, 'a' + str(i), i) > ? ? ? ?t1 = time.time() > ? ? ? ?# return: total, time per write > ? ? ? ?return t1 - t0, (t1 - t0)/count > > This is not working very well with the non-overridden dict, because we > don't optimize for this case at all. You are using > > ?a) lots of attributes, which we expect to be rare > ?b) access them with setattr, which is a lot slower than a fixed attribute > ?c) access a different attribute every loop iteration, which means the > compiler has to produce one bit of code for every attribute This is intentional (all three points), I did not want the jit to factor out the loop -- I wanted to time the initial attribute creation... > > Read this, for some hints why this is the case: > > http://morepypy.blogspot.com/2010/11/efficiently-implementing-python-objects.html > > This is in theory fixable with enough work, but I am not sure that this > is a common or useful use case. If you really need to do this, just use > a normal dictionary. Or show me some real-world code that does this, and > might think about the case some more. > > Anyway, the timing behavior of the above loop is merely quadratic in the > number of attributes, not exponential :-). Accepted, my mistake :) But quadratic behaviour + a three orders of magnitude increase in time it takes to create an attr is scarry... but you are right, how often does that usecase happen? :) Retested with: setattr(obj, 'a', i) The results are *allot* better and it is indeed the common case :) Thanks! > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- Alex. From alex.nanou at gmail.com Mon Nov 29 21:12:49 2010 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Mon, 29 Nov 2010 23:12:49 +0300 Subject: [pypy-dev] a possible leak in the object namespace... In-Reply-To: References: <4CF3F4F3.9020204@gmx.de> Message-ID: On Mon, Nov 29, 2010 at 23:02, Alex A. Naanou wrote: > On Mon, Nov 29, 2010 at 21:46, Carl Friedrich Bolz wrote: >> Hi Alex, >> >> On 11/29/2010 03:04 PM, Alex A. Naanou wrote: >>> With the release of version 1.4, I decided to test these usecases out >>> and benchmark them on PyPy and 15 minutes later I got results that >>> were surprising to say the least... >>> >>> Expectations: >>> 1) the normal/native namespace should have been a bit faster than the >>> hooked object on the first run. Both cases should have leveled to >>> about the same performance after the JIT finished it's job +/- a >>> constant. >>> 2) all times should have been near constant. >>> >>> What I got per point: >>> 1) the object with native dict was slower by about three orders of >>> magnitude than the object with a hooked namespace. >>> 2) sequential write benchmark runs on the normal object did not level >>> out, as they did with the hook, rather, they exhibited exponential >>> times (!!) >> >> Don't do that then :-). > > :) > >> >> >>> For details and actual test code see the attached file. >> >> The code you are trying is essentially this: >> >> def test(obj, count=10000): >> ? ? ? ?t0 = time.time() >> ? ? ? ?for i in xrange(count): >> ? ? ? ? ? ? ? ?setattr(obj, 'a' + str(i), i) >> ? ? ? ?t1 = time.time() >> ? ? ? ?# return: total, time per write >> ? ? ? ?return t1 - t0, (t1 - t0)/count >> >> This is not working very well with the non-overridden dict, because we >> don't optimize for this case at all. You are using >> >> ?a) lots of attributes, which we expect to be rare >> ?b) access them with setattr, which is a lot slower than a fixed attribute >> ?c) access a different attribute every loop iteration, which means the >> compiler has to produce one bit of code for every attribute > > This is intentional (all three points), I did not want the jit to > factor out the loop -- I wanted to time the initial attribute > creation... > > > >> >> Read this, for some hints why this is the case: >> >> http://morepypy.blogspot.com/2010/11/efficiently-implementing-python-objects.html >> >> This is in theory fixable with enough work, but I am not sure that this >> is a common or useful use case. If you really need to do this, just use >> a normal dictionary. Or show me some real-world code that does this, and >> might think about the case some more. >> >> Anyway, the timing behavior of the above loop is merely quadratic in the >> number of attributes, not exponential :-). > > Accepted, my mistake :) > > But quadratic behaviour + a three orders of magnitude increase in time > it takes to create an attr is scarry... but you are right, how often > does that usecase happen? :) > > > Retested with: > ?setattr(obj, 'a', i) > > The results are *allot* better and it is indeed the common case :) come to think of it, the results appear to be too good, it looks like the loop is optimized an only the last iteration is left in, so it's not not representative... > > > > Thanks! > >> >> Cheers, >> >> Carl Friedrich >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > > > > -- > Alex. > -- Alex. From p.giarrusso at gmail.com Mon Nov 29 21:54:24 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 29 Nov 2010 21:54:24 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: On Mon, Nov 29, 2010 at 14:40, Stefan Behnel wrote: > Amaury Forgeot d'Arc, 28.11.2010 11:44: >> 2010/11/28 Maciej Fijalkowski >> >>> On Sun, Nov 28, 2010 at 11:58 AM, Ren? Dudfield wrote: >>>> what xml libraries are people using with pypy? ?What is working well? >>> >>> PyExpat works, although it's slow (ctypes-based implementation). I >>> know genshi has some troubles with it, someone is debugging now. >>> Besides I don't think there are any working (unless someone wrote a >>> pure-python one) >> >> PyExpat is now a built-in module, implemented in RPython, >> and should have reasonable performance. > > Hmm, reasonable? > > $ ./bin/pypy -m timeit -s 'import xml.etree.ElementTree as ET' \ > ? ? ?'ET.parse("ot.xml")' > 10 loops, best of 3: 1.27 sec per loop > > $ python2.7 -m timeit -s 'import xml.etree.ElementTree as ET' \ > ? ? ?'ET.parse("ot.xml")' > 10 loops, best of 3: 486 msec per loop > > $ python2.7 -m timeit -s 'import xml.etree.cElementTree as ET' \ > ? ? ?'ET.parse("ot.xml")' > 10 loops, best of 3: 33.7 msec per loop Is any JITting expected to trigger with so few iteractions? Or does RPython saves the need for that? I tried increasing the loop count, but I couldn't, because of two different bugs somewhere (in PyPy I guess). I tried ensuring that at least 1000 iterations were displayed, but timeit doesn't work for more than 852 iterations on the attached example (found on my HD): $ pypy-trunk/pypy/translator/goal/pypy-c -m timeit -n 853 -s 'import xml.etree.ElementTree as ET' 'ET.parse("extensionNames.xml")' ImportError: No module named linecache Now, even if linecache is imported locally, linecache.py exists (located in the same path as timeit.py, i.e. lib-python/2.5.2/). Furthermore, it works fine on the Python interpreter, suggesting that the -m option might be part of the bug: import timeit a=timeit.Timer('ET.parse("extensionNames.xml")', 'import xml.etree.ElementTree as ET') a.timeit(1000) However, a bigger timing count doesn't work: >>>> a.timeit(10000) Traceback (most recent call last): File "", line 1, in File "/Users/pgiarrusso/Documents/Research/Sorgenti/PyPy/pypy-trunk/lib-python/2.5.2/timeit.py", line 161, in timeit File "", line 6, in inner File "/Users/pgiarrusso/Documents/Research/Sorgenti/PyPy/pypy-trunk/lib_pypy/xml/etree/ElementTree.py", line 862, in parse File "/Users/pgiarrusso/Documents/Research/Sorgenti/PyPy/pypy-trunk/lib_pypy/xml/etree/ElementTree.py", line 579, in parse IOError: [Errno 24] Too many open files: 'extensionNames.xml' Inspection of the pypy process confirms a leak of file handles to the XML files. Whether it is GC not being invoked, a missing destructor, or simply because the code should release file handles, I dunno. Is there a way to trigger explicit GC to workaround such issues? Warning: all this is with a 32bit PyPy-1.4 on Mac OS X. Bye -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ -------------- next part -------------- A non-text attachment was scrubbed... Name: extensionNames.xml Type: text/xml Size: 365 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101129/084bba6c/attachment.bin From amauryfa at gmail.com Mon Nov 29 21:57:44 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 29 Nov 2010 21:57:44 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: 2010/11/29 Paolo Giarrusso > Inspection of the pypy process confirms a leak of file handles to the > XML files. Whether it is GC not being invoked, a missing destructor, > or simply because the code should release file handles, I dunno. Is > there a way to trigger explicit GC to workaround such issues? > As usual: import gc gc.collect() Calling gc.collect() is indeed a good idea if the code does not explicitly close the files. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101129/7859c364/attachment.htm From piotr.skamruk at gmail.com Mon Nov 29 22:29:26 2010 From: piotr.skamruk at gmail.com (Piotr Skamruk) Date: Mon, 29 Nov 2010 22:29:26 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: simplier would be set ulimit -n to 65536 (probably in /etc/security/limits.conf) 2010/11/29 Amaury Forgeot d'Arc : > 2010/11/29 Paolo Giarrusso >> >> Inspection of the pypy process confirms a leak of file handles to the >> XML files. Whether it is GC not being invoked, a missing destructor, >> or simply because the code should release file handles, I dunno. Is >> there a way to trigger explicit GC to workaround such issues? > > As usual: > ?? ?import gc > ?? ?gc.collect() > Calling gc.collect() is indeed a good idea if the code does not explicitly > close the files. > > -- > Amaury Forgeot d'Arc > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From p.giarrusso at gmail.com Tue Nov 30 00:14:08 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Tue, 30 Nov 2010 00:14:08 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: Hi all, thanks to the tips, I verified on Mac OS X a 17% slowdown, after manually taking the best times, vs Python-2.5 (32bit). Measuring on the command line would give a 57% slowdown instead, because of lack of warmup. As a matter of fact, however, pyexpat is not involved here for PyPy, and here (v1.4) it is still implemented through ctypes (in lib_pypy/pyexpat.py), and not in RPython in pypy/rlib/. Python 2.7 may well be faster, which might explain some extra difference with Stefan's results. It looks like the two bugs should be easy to fix: - a file leak on the tested XML module, indeed - an IOException on module opening converted to "file not found" - at least in Java, file not found is a specific exception which can be distinguished from generic I/O errors. On Mon, Nov 29, 2010 at 22:29, Piotr Skamruk wrote: > simplier would be set ulimit -n to 65536 (probably in /etc/security/limits.conf) Thanks, I needed both this and the GC tips, since during a test run to run 10^4 iterations, I can't call the GC and still get meaningful results. [I'm on Mac OS X though, so ulimit -S -n 10240 is the best one can do, otherwise "Invalid argument", i.e. EINVAL, results]. Additionally, I just discovered that the ImportError on "import linecache" looks filehandle-related as well, because changing the ulimit changes the iteration count triggering the error, so it's likely an effect of the same bug. Still, the original error message should be preserved, and this should be easy to fix. In these conditions, my best results after warming up are: 0.358 ms PyPy-JIT-32bit (see below for JIT logs) 0.305 ms CPython-2.5-32bit 0.269 ms CPython-2.6-64bit 0.553 ms PyPy-64bit-noJIT, rev 79307, 21 Nov 2010 which means a 17% slowdown on comparable setups, rather than a 2x slowdown; measuring with timeit on the cmd line, instead, would give a 57% slowdown. All this is on a very small input file, the one I attached before. That's for the total of 1000 iterations, on a Core 2 Duo 2.6GHz. I don't report the average because: a) it is difficult to get something significant anyway (I don't want to code confidence intervals, and automated tools wouldn't call GC appropriately) b) I expect the deviation to be due more to unrelated load on my laptop (around 12-18% CPU) than to actual spread of the runtime. I set PYPYLOG='jit-summary:-' before the PyPy-JIT run and got this - I hope somebody can check from this whether the JIT is working successfully. [f2dd1fbaa1c2] {jit-summary Tracing: 25 0.163456 Backend: 23 0.017392 Running asm: 191214 Blackhole: 2012 TOTAL: 502.543032 ops: 68338 recorded ops: 32764 calls: 1759 guards: 18005 opt ops: 2757 opt guards: 696 forcings: 111 abort: trace too long: 2 abort: compiling: 0 abort: vable escape: 0 nvirtuals: 6693 nvholes: 1059 nvreused: 3979 Total # of loops: 18 Total # of bridges: 6 Freed # of loops: 0 Freed # of bridges: 0 [f2dd1fc141a8] jit-summary} Best regards. > 2010/11/29 Amaury Forgeot d'Arc : >> 2010/11/29 Paolo Giarrusso >>> >>> Inspection of the pypy process confirms a leak of file handles to the >>> XML files. Whether it is GC not being invoked, a missing destructor, >>> or simply because the code should release file handles, I dunno. Is >>> there a way to trigger explicit GC to workaround such issues? >> >> As usual: >> ?? ?import gc >> ?? ?gc.collect() >> Calling gc.collect() is indeed a good idea if the code does not explicitly >> close the files. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From amauryfa at gmail.com Tue Nov 30 00:45:05 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 30 Nov 2010 00:45:05 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: 2010/11/30 Paolo Giarrusso > As a matter of fact, however, pyexpat is not involved here for PyPy, > and here (v1.4) it is still implemented through ctypes (in > lib_pypy/pyexpat.py), and not in RPython in pypy/rlib/. > Did you compile pypy yourself? if the expat development files are present, the translation should build the pyexpat module: Python 2.5.2 (79656, Nov 29 2010, 21:05:28) [PyPy 1.4.0] on linux2 >>>> import pyexpat >>>> pyexpat -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101130/661c0e3e/attachment.htm From drsalists at gmail.com Tue Nov 30 02:28:20 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 29 Nov 2010 17:28:20 -0800 Subject: [pypy-dev] gdbm In-Reply-To: <4CF36B99.9080104@gmail.com> References: <201011160222.oAG2Mrfv004177@theraft.openend.se> <4CF03ED2.3050801@gmail.com> <4CF36B99.9080104@gmail.com> Message-ID: On Mon, Nov 29, 2010 at 1:00 AM, Antonio Cuni wrote: > On 27/11/10 04:07, Dan Stromberg wrote: > > > This module can be used with the ?classic? ndbm interface, the BSD DB > > compatibility interface, or the GNU GDBM compatibility interface. On > Unix, the > > *configure* script will attempt to locate the appropriate header file to > > simplify building this module. > > > > I suppose that means that if it can't find ndbm (which at one time was > hard > > due to licensing, but last I heard it'd become readily available), it's > free > > to pretend it has ndbm using something else. > > > > I'd call that puzzlingly worded - it's not the interface that's changing, > but > > the backend implementation. But perhaps dbm.py is free to use Berkeley > DB if > > it prefers to pretend it can never find ndbm. And perhaps I shouldn't > have > > skimmed that page so quickly ^_^ > > > > CPython 2.7's configure script has: > > > > --with-dbmliborder=db1:db2:... > > order to check db backends for dbm. Valid value > is a > > colon separated string with the backend names > > `ndbm', `gdbm' and `bdb'. > > > > > > so, having a dbm.py which links to libdb is fine, and it's also what you > get > with cpython on ubuntu. There is still the issue of how to find the > correct > library name, as it seems it can vary (it was db-4.5 when the module was > written, it's db-4.8 nowadays), but this is a bit orthogonal to what we are > discussing here. > > To summarize, I think we should keep the current dbm.py to link against > libdb, > and integrate your gdbm.py to link against libgdbm. But before merging it > to > trunk, I'd like to solve the issue of code duplication between the two > modules. > > ciao, > Anto > Agreed. I should have time for this sometime this week or the next. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101129/5bcc967c/attachment-0001.htm From wlavrijsen at lbl.gov Tue Nov 30 05:35:06 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Mon, 29 Nov 2010 20:35:06 -0800 (PST) Subject: [pypy-dev] double trouble with JIT In-Reply-To: <4CF3AFC3.1040703@gmx.de> References: <4CF3A6A4.7080607@gmx.de> <4CF3AFC3.1040703@gmx.de> Message-ID: Hi Carl, > Given that we are not running PyPy's test suite on PyPy nightly currently, I > think it would be safer to keep running the tests on top of CPython. of course, that begs the question how one tests pypy? I'm thinking of making the benchmarks more elaborate and use those as my post-translation tests. Btw., I solved the original segfault that I was after, but that didn't change anything for the particular error in this e-mail thread, though. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From fijall at gmail.com Tue Nov 30 08:13:41 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 30 Nov 2010 09:13:41 +0200 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: On Tue, Nov 30, 2010 at 1:45 AM, Amaury Forgeot d'Arc wrote: > 2010/11/30 Paolo Giarrusso >> >> As a matter of fact, however, pyexpat is not involved here for PyPy, >> and here (v1.4) it is still implemented through ctypes (in >> lib_pypy/pyexpat.py), and not in RPython in pypy/rlib/. It's also module/pyexpat and not rlib (rlib is for RPython libraries) > > Did you compile pypy yourself? > if the expat development files are present, the translation should build the > pyexpat module: > Python 2.5.2 (79656, Nov 29 2010, 21:05:28) > [PyPy 1.4.0] on linux2 >>>>> import pyexpat >>>>> pyexpat > > -- > Amaury Forgeot d'Arc > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Tue Nov 30 08:19:21 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 30 Nov 2010 08:19:21 +0100 Subject: [pypy-dev] gdbm In-Reply-To: References: <201011160222.oAG2Mrfv004177@theraft.openend.se> <4CF03ED2.3050801@gmail.com> <4CF36B99.9080104@gmail.com> Message-ID: <4CF4A579.3060208@gmail.com> On 30/11/10 02:28, Dan Stromberg wrote: > Agreed. > > I should have time for this sometime this week or the next. thank you! :-) From arigo at tunes.org Tue Nov 30 11:04:20 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 30 Nov 2010 11:04:20 +0100 Subject: [pypy-dev] PyPy-JIT memory usage compared to CPython? In-Reply-To: <504667.29687.qm@web111301.mail.gq1.yahoo.com> References: <504667.29687.qm@web111301.mail.gq1.yahoo.com> Message-ID: Hi, On Tue, Nov 30, 2010 at 10:58 AM, Andy wrote: > Can you explain a bit about what did you mean by "program contains mainly instances"? And what is __slots__? I meant basically: try with your particular use case, because it is hard to give one answer that fits all cases. Armin From angelflow at yahoo.com Tue Nov 30 10:58:13 2010 From: angelflow at yahoo.com (Andy) Date: Tue, 30 Nov 2010 01:58:13 -0800 (PST) Subject: [pypy-dev] PyPy-JIT memory usage compared to CPython? In-Reply-To: Message-ID: <504667.29687.qm@web111301.mail.gq1.yahoo.com> Can you explain a bit about what did you mean by "program contains mainly instances"? And what is __slots__? Thanks --- On Mon, 11/29/10, Armin Rigo wrote: > From: Armin Rigo > Subject: Re: [pypy-dev] PyPy-JIT memory usage compared to CPython? > To: "Andy" > Cc: pypy-dev at codespeak.net > Date: Monday, November 29, 2010, 5:32 AM > Hi, > > On Sat, Nov 27, 2010 at 12:03 AM, Andy > wrote: > > I'm particularly interested in the memory usage of > PyPy-JIT vs. CPython when running Django. In general would > PyPy-JIT increase or decrease memory usage? > > It depends on the kind of application.? PyPy with a > JIT may get to > 1.5x or 2x more than CPython, but it may also be smaller > than CPython > if your program contains mainly instances and you didn't > feel like > carefully putting __slots__ everywhere. > > > A bient?t, > > Armin. > From marius at pov.lt Tue Nov 30 12:42:05 2010 From: marius at pov.lt (Marius Gedminas) Date: Tue, 30 Nov 2010 13:42:05 +0200 Subject: [pypy-dev] PyPy 1.4 released In-Reply-To: <4CF2389B.90801@gmail.com> References: <4CF0A297.9090202@gmail.com> <4CF2389B.90801@gmail.com> Message-ID: <20101130114204.GA30038@fridge.pov.lt> On Sun, Nov 28, 2010 at 12:10:19PM +0100, Antonio Cuni wrote: > On 28/11/10 10:48, Maciej Fijalkowski wrote: > >> > i got python-magic working , after i installed without easy_install > >> > (easy_install fail because it tried to install ctypes). > > > great > > well, that's still a bit strange. Why does easy_install try to install ctypes, > given that it's in the stdlib? The standard library doesn't have setuptools metadata that would tell easy_install about ctypes already being available, so easy_install tries to download and install ctypes if asked to. Marius Gedminas -- I'm a shareware signature! Send $2 if you use me, $10 for a manual. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 190 bytes Desc: Digital signature Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101130/1077eaf0/attachment.pgp From cfbolz at gmx.de Tue Nov 30 18:33:45 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 30 Nov 2010 18:33:45 +0100 Subject: [pypy-dev] a possible leak in the object namespace... In-Reply-To: References: <4CF3F4F3.9020204@gmx.de> Message-ID: <4CF53579.6080504@gmx.de> Hi Alex, On 11/29/2010 09:02 PM, Alex A. Naanou wrote: > On Mon, Nov 29, 2010 at 21:46, Carl Friedrich Bolz wrote: [snip] >> a) lots of attributes, which we expect to be rare >> b) access them with setattr, which is a lot slower than a fixed attribute >> c) access a different attribute every loop iteration, which means the >> compiler has to produce one bit of code for every attribute > > This is intentional (all three points), I did not want the jit to > factor out the loop -- I wanted to time the initial attribute > creation... Yes, I fear initial attribute creation is never going to be very efficient. [snip] > > But quadratic behaviour + a three orders of magnitude increase in time > it takes to create an attr is scarry... but you are right, how often > does that usecase happen? :) I improved some things so that setattr/getattr is a bit faster. Should now only be one or two orders of magnitude :-). If you run it tomorrow with the nightly build from tonight, you should see an improvement. The quadraticness is not easily fixable without giving up all optimizations with instances of more than X (maybe 1000?) attributes. Again, I don't think this is common. And I don't want to chose an X. Cheers, Carl Friedrich From p.giarrusso at gmail.com Wed Dec 1 00:34:54 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Wed, 1 Dec 2010 00:34:54 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: On Tue, Nov 30, 2010 at 08:13, Maciej Fijalkowski wrote: > On Tue, Nov 30, 2010 at 1:45 AM, Amaury Forgeot d'Arc > wrote: >> 2010/11/30 Paolo Giarrusso >>> >>> As a matter of fact, however, pyexpat is not involved here for PyPy, >>> and here (v1.4) it is still implemented through ctypes (in >>> lib_pypy/pyexpat.py), and not in RPython in pypy/rlib/. > > It's also module/pyexpat and not rlib (rlib is for RPython libraries) > >> >> Did you compile pypy yourself? >> if the expat development files are present, the translation should build the >> pyexpat module: >> Python 2.5.2 (79656, Nov 29 2010, 21:05:28) >> [PyPy 1.4.0] on linux2 >>>>>> import pyexpat >>>>>> pyexpat >> My apologies, I self-compiled PyPy and I get the output you describe indeed. Therefore I guess that the ctypes implementation I come across in lib_pypy/pyexpat.py is probably a fallback - in case only the library, but not the headers, are present. Anyway, this does not interact with benchmarks above - Stefan, I still don't get why you complained that pyexpat is slow by showing benchmarks for another module, I guess I do not understand your email, but it asks "reasonable?" after Amaury talks about pyexpat. I'll try to benchmark it soon; a reasonable way to call pyexpat would make it simpler since I have limited time and mental energy to devote, and figuring out a non-stupid way to use it might be non-trivial without learning to use the library. Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From stefan_ml at behnel.de Wed Dec 1 08:48:41 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 01 Dec 2010 08:48:41 +0100 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: Paolo Giarrusso, 01.12.2010 00:34: > Anyway, this does not interact with benchmarks above - Stefan, I still > don't get why you complained that pyexpat is slow by showing > benchmarks for another module, I guess I do not understand your email, > but it asks "reasonable?" after Amaury talks about pyexpat. Well, in CPython, I can see little to no reason why anyone would go as low-level as pyexpat when you can use cElementTree. So IMHO the level to compare is what people would normally use rather than what people could potentially use if they were beaten hard enough. Stefan From fijall at gmail.com Wed Dec 1 09:00:12 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 1 Dec 2010 10:00:12 +0200 Subject: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released) In-Reply-To: References: Message-ID: On Wed, Dec 1, 2010 at 9:48 AM, Stefan Behnel wrote: > Paolo Giarrusso, 01.12.2010 00:34: >> Anyway, this does not interact with benchmarks above - Stefan, I still >> don't get why you complained that pyexpat is slow by showing >> benchmarks for another module, I guess I do not understand your email, >> but it asks "reasonable?" after Amaury talks about pyexpat. > > Well, in CPython, I can see little to no reason why anyone would go as > low-level as pyexpat when you can use cElementTree. So IMHO the level to > compare is what people would normally use rather than what people could > potentially use if they were beaten hard enough. > > Stefan > Hey. Sure, makes sense :-) I think one of the reasons for some slowdown is that calls from C are not jitted if they don't contain loops themselves. This doesn't explain the whole thing obviously, because there is something really wrong going on looking at numbers. From alex.nanou at gmail.com Wed Dec 1 11:23:08 2010 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Wed, 1 Dec 2010 13:23:08 +0300 Subject: [pypy-dev] a possible leak in the object namespace... In-Reply-To: <4CF53579.6080504@gmx.de> References: <4CF3F4F3.9020204@gmx.de> <4CF53579.6080504@gmx.de> Message-ID: On Tue, Nov 30, 2010 at 20:33, Carl Friedrich Bolz wrote: > Hi Alex, > > On 11/29/2010 09:02 PM, Alex A. Naanou wrote: >> >> On Mon, Nov 29, 2010 at 21:46, Carl Friedrich Bolz ?wrote: > > [snip] >>> >>> ?a) lots of attributes, which we expect to be rare >>> ?b) access them with setattr, which is a lot slower than a fixed >>> attribute >>> ?c) access a different attribute every loop iteration, which means the >>> compiler has to produce one bit of code for every attribute >> >> This is intentional (all three points), I did not want the jit to >> factor out the loop -- I wanted to time the initial attribute >> creation... > > Yes, I fear initial attribute creation is never going to be very efficient. > > [snip] >> >> But quadratic behaviour + a three orders of magnitude increase in time >> it takes to create an attr is scarry... but you are right, how often >> does that usecase happen? :) > > I improved some things so that setattr/getattr is a bit faster. Should now > only be one or two orders of magnitude :-). If you run it tomorrow with the > nightly build from tonight, you should see an improvement. That's good news! It is not uncommon, at least in my work, to write generic code that does not know of attribute names it uses, thus getattr/setattr functions are unavoidable. > The quadraticness is not easily fixable without giving up all optimizations > with instances of more than X (maybe 1000?) attributes. Again, I don't think > this is common. And I don't want to chose an X. I'd even argue that most of the cases were the number of attributes exceeds a certain "sane" N would and should be better implemented via a container... (in spite of the how I wrote the code in my first mail ;) ) I've been planning of moving most of my commercial projects to PyPy as soon as it is stable enough -- I'm tired of fighting the CPython implementation for most of the time -- we will see how it goes... too bad some of the code will be PyPy specific due to the way CPython works. Thanks! > > Cheers, > > Carl Friedrich > -- Alex. From rndblnch at gmail.com Wed Dec 1 14:18:01 2010 From: rndblnch at gmail.com (renaud blanch) Date: Wed, 1 Dec 2010 11:18:01 -0200 Subject: [pypy-dev] (no subject) Message-ID: hi, i'm trying to make some pyopengl [0] -based code [1] run on top of pypy. this is partially successful, but i need some advice to progress further. pyopengl 3.x makes use of ctypes to provide the opengl binding, and it works out of the box for simple functions (those that do not takes c-pointer to buffers of data as arguments). for the rest, the first issue is that pyopengl use two functions from the ctypes.pythonapi lib, namely PyString_AsString and PyBuffer_FromMemory. any advice on how to replace those functions to make them compatible with pypy? Mike Fletcher (pyopengl author) gave me some hints about that point: > For the first issue, those are going to require some reworking, in > essence those are "C" implemented code that happens to use Python/ctypes > as the implementation language and makes assumptions about the > data-storage for the objects (e.g. that a string is internally a > contiguous series of bytes, which is *not necessarily* true in PyPy). > We'd need to find a mechanism in PyPy that would give us that direct > memory-pointer access to be able to use it. Note: a compacting garbage > collector (or anything else that can move memory locations) will cause > problems there, so we may need to find a way to signal PyPy not to move > a given object, and to use contiguous data-arrays for their storage. thanks a lot for any advice, renaud 0. PyOpenGL 3.x / The Python OpenGL? Binding 1. opengl-programmable / a short step by step tutorial to OpenGL programmable pipeline From rndblnch at gmail.com Wed Dec 1 14:19:31 2010 From: rndblnch at gmail.com (renaud blanch) Date: Wed, 1 Dec 2010 11:19:31 -0200 Subject: [pypy-dev] pyopengl on pypy Message-ID: sorry, i forgot to put a subject. renaud From rndblnch at gmail.com Wed Dec 1 14:21:21 2010 From: rndblnch at gmail.com (renaud blanch) Date: Wed, 1 Dec 2010 11:21:21 -0200 Subject: [pypy-dev] pyopengl on pypy Message-ID: this time with a subject, the message, and my apologies for the noise, renaud On 12/1/10, renaud blanch wrote: > hi, > > i'm trying to make some pyopengl [0] -based code [1] run on top of pypy. > this is partially successful, but i need some advice to progress further. > pyopengl 3.x makes use of ctypes to provide the opengl binding, and it > works out of the box for simple functions (those that do not takes > c-pointer to buffers of data as arguments). > for the rest, the first issue is that pyopengl use two functions from > the ctypes.pythonapi lib, namely PyString_AsString and > PyBuffer_FromMemory. > any advice on how to replace those functions to make them compatible with > pypy? > > Mike Fletcher (pyopengl author) gave me some hints about that point: >> For the first issue, those are going to require some reworking, in >> essence those are "C" implemented code that happens to use Python/ctypes >> as the implementation language and makes assumptions about the >> data-storage for the objects (e.g. that a string is internally a >> contiguous series of bytes, which is *not necessarily* true in PyPy). >> We'd need to find a mechanism in PyPy that would give us that direct >> memory-pointer access to be able to use it. Note: a compacting garbage >> collector (or anything else that can move memory locations) will cause >> problems there, so we may need to find a way to signal PyPy not to move >> a given object, and to use contiguous data-arrays for their storage. > > thanks a lot for any advice, > > renaud > > 0. PyOpenGL 3.x / The Python OpenGL? Binding > > > 1. opengl-programmable / a short step by step tutorial to OpenGL > programmable pipeline > > From fijall at gmail.com Wed Dec 1 14:52:39 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 1 Dec 2010 15:52:39 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: References: Message-ID: Hey. I would suggest using array module. It allocates memory in a non-moving location and it's address can be found using buffer_info method. Cheers, fijal On Wed, Dec 1, 2010 at 3:18 PM, renaud blanch wrote: > hi, > > i'm trying to make some pyopengl [0] -based code [1] run on top of pypy. > this is partially successful, but i need some advice to progress further. > pyopengl 3.x makes use of ctypes to provide the opengl binding, and it > works out of the box for simple functions (those that do not takes > c-pointer to buffers of data as arguments). > for the rest, the first issue is that pyopengl use two functions from > the ctypes.pythonapi lib, namely PyString_AsString and > PyBuffer_FromMemory. > any advice on how to replace those functions to make them compatible with pypy? > > Mike Fletcher (pyopengl author) gave me some hints about that point: >> For the first issue, those are going to require some reworking, in >> essence those are "C" implemented code that happens to use Python/ctypes >> as the implementation language and makes assumptions about the >> data-storage for the objects (e.g. that a string is internally a >> contiguous series of bytes, which is *not necessarily* true in PyPy). >> We'd need to find a mechanism in PyPy that would give us that direct >> memory-pointer access to be able to use it. ?Note: a compacting garbage >> collector (or anything else that can move memory locations) will cause >> problems there, so we may need to find a way to signal PyPy not to move >> a given object, and to use contiguous data-arrays for their storage. > > thanks a lot for any advice, > > renaud > > 0. PyOpenGL 3.x / The Python OpenGL? Binding > > > 1. opengl-programmable / a short step by step tutorial to OpenGL > programmable pipeline > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Wed Dec 1 15:23:01 2010 From: arigo at tunes.org (Armin Rigo) Date: Wed, 1 Dec 2010 15:23:01 +0100 Subject: [pypy-dev] pyopengl on pypy In-Reply-To: References: Message-ID: Hi Renaud, On Wed, Dec 1, 2010 at 2:21 PM, renaud blanch wrote: >>> We'd need to find a mechanism in PyPy that would give us that direct >>> memory-pointer access to be able to use it. ?Note: a compacting garbage >>> collector (or anything else that can move memory locations) will cause >>> problems there, so we may need to find a way to signal PyPy not to move >>> a given object, and to use contiguous data-arrays for their storage. Indeed, this part from Mike Fletcher got the cause right, but the solution we recommend is different. It's kind of impossible in PyPy to mark objects as non-moving ("pin" them, as the garbage collection language goes). Instead you need to just use the official ctypes API, create_string_buffer(). It's equivalent, but requires one extra copy of the data. A bient?t, Armin. From amauryfa at gmail.com Wed Dec 1 15:33:47 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 1 Dec 2010 15:33:47 +0100 Subject: [pypy-dev] pyopengl on pypy In-Reply-To: References: Message-ID: Hi, 2010/12/1 renaud blanch > > i'm trying to make some pyopengl [0] -based code [1] run on top of pypy. > > this is partially successful, but i need some advice to progress further. > > pyopengl 3.x makes use of ctypes to provide the opengl binding, and it > > works out of the box for simple functions (those that do not takes > > c-pointer to buffers of data as arguments). > > for the rest, the first issue is that pyopengl use two functions from > > the ctypes.pythonapi lib, namely PyString_AsString and > > PyBuffer_FromMemory. > > any advice on how to replace those functions to make them compatible with > > pypy? > > > > Mike Fletcher (pyopengl author) gave me some hints about that point: > >> For the first issue, those are going to require some reworking, in > >> essence those are "C" implemented code that happens to use Python/ctypes > >> as the implementation language and makes assumptions about the > >> data-storage for the objects (e.g. that a string is internally a > >> contiguous series of bytes, which is *not necessarily* true in PyPy). > >> We'd need to find a mechanism in PyPy that would give us that direct > >> memory-pointer access to be able to use it. Note: a compacting garbage > >> collector (or anything else that can move memory locations) will cause > >> problems there, so we may need to find a way to signal PyPy not to move > >> a given object, and to use contiguous data-arrays for their storage. > Don't worry, with PyPy's ctypes you'll never get the address of a moving object. c_char_p makes a non-moving copy of the string. But IMO the call to pythonapi.PyString_AsString could be removed in OpenGL/arrays/strings.py: def dataPointer(value): return ctypes.cast(ctypes.c_char_p(value), ctypes.c_void_p).value I don't know about the other function, though. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101201/def6fba1/attachment.htm From rndblnch at gmail.com Wed Dec 1 18:56:54 2010 From: rndblnch at gmail.com (rndblnch) Date: Wed, 1 Dec 2010 17:56:54 +0000 (UTC) Subject: [pypy-dev] pyopengl on pypy References: Message-ID: hi, Amaury Forgeot d'Arc gmail.com> writes: > But IMO the call to?pythonapi.PyString_AsString could be removed in > OpenGL/arrays/strings.py: > > def?dataPointer(value): > ?? ?return ctypes.cast(ctypes.c_char_p(value),?ctypes.c_void_p).value great, this replacement (and also the one suggested by Armin with create_string_buffer instead of c_char_p) seams to works (i.e., runs fine on cpython). > I don't know about the other function, though. this one is only in experimental code of pyopengl, so i won't bother much for now. the next issue is now that the array data types used by pyopengl are not compatible with pypy ctypes: they are lacking (at least) _CData_input as shown in the stack trace below. any advice on that one? should i try to fix PyOpenGL ArrayDatatype, or try to patch pypy's ctypes function.py to work around that one? thanks again, renaud % pypy 01-direct.py Traceback (most recent call last): File "app_main.py", line 33, in run_toplevel File "01-direct.py", line 268, in sys.exit(main()) File "01-direct.py", line 262, in main init_texture() File "01-direct.py", line 48, in init_texture "".join(pixel(i, j, k) for i in range(width) File "/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/latebind.py", line 45, in __call__ return self._finalCall( *args, **named ) File "/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/wrapper.py", line 784, in wrapperCall result = self.wrappedOperation( *cArguments ) File "/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/platform/baseplatform.py", line 335, in __call__ return self( *args, **named ) File "/Users/Shared/src/pypy-1.4-osx/lib_pypy/_ctypes/function.py", line 166, in __call__ argtypes, argsandobjs = self._wrap_args(argtypes, args) File "/Users/Shared/src/pypy-1.4-osx/lib_pypy/_ctypes/function.py", line 281, in _wrap_args wrapped = argtype._CData_input(arg) AttributeError: type object 'ArrayDatatype' has no attribute '_CData_input' From anto.cuni at gmail.com Wed Dec 1 23:48:23 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 01 Dec 2010 23:48:23 +0100 Subject: [pypy-dev] [PyPy Status Blog] New comment on PyPy 1.4: Ouroboros in practice. In-Reply-To: <1291242102925.4c68f818-d73b-492b-a923-4edf7d1a74f7@google.com> References: <1291242102925.4c68f818-d73b-492b-a923-4edf7d1a74f7@google.com> Message-ID: <4CF6D0B7.9080900@gmail.com> On 01/12/10 23:21, Amaury Forgeot d'Arc wrote: [cut] > There is already an ongoing effort to port PyPy to Python 2.7. > > But we need some help! It's a good way to become a PyPy developer. > And no, you don't have to be a JIT expert to implement itertools.combinations > or asian codecs. Nice comment Amaury :-). What about writing a blog post explicitly asking for help? I also agree that the fast-forward branch is a good way to enter pypy, moreover we should exploit the hype we seem to have at the moment :-). ciao, Anto From sasmekoll at gmail.com Wed Nov 24 21:41:53 2010 From: sasmekoll at gmail.com (Vovan) Date: Wed, 24 Nov 2010 23:41:53 +0300 Subject: [pypy-dev] =?utf-8?b?0J/QuNC60LDQvw==?= Message-ID: <4cf7a5b5.960acc0a.33d1.2224@mx.google.com> ?????????? ???????? ???????? ???? ???????????? ?????????????????????????? ???????????? ?? ?? ??????????????, ?????????????? ???????????????? ???????????????? ?? ???????? ???????????????? ???? ???????????? ?? ?????????????????????? ?????????? ???????? ???????????? ???? ???????????? ?????? ?????????????????????????? ?? ????????????????? , ?? ???????????? ?????????????????? ??????????????, ?????????? ???????????? ?????????????? ?????????????? ?????? ?????????? ???????????????????? ??????????????????!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101124/1f05085d/attachment.htm From sasmekoll at gmail.com Wed Nov 24 21:44:49 2010 From: sasmekoll at gmail.com (Vovan) Date: Wed, 24 Nov 2010 23:44:49 +0300 Subject: [pypy-dev] =?utf-8?b?0J/QuNC60LDQvw==?= Message-ID: <4cf7a649.cc97cc0a.16f2.2037@mx.google.com> ?????????? ???????? ???????? ???? ???????????? ?????????????????????????? ???????????? ?? ?? ??????????????, ?????????????? ???????????????? ???????????????? ?? ???????? ???????????????? ???? ???????????? ?? ?????????????????????? ?????????? ???????? ???????????? ???? ???????????? ?????? ?????????????????????????? ?? ????????????????? , ?? ???????????? ?????????????????? ??????????????, ?????????? ???????????? ?????????????? ?????????????? ?????? ?????????? ???????????????????? ??????????????????!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101124/49bcdcae/attachment.htm From sasmekoll at gmail.com Wed Nov 24 21:45:46 2010 From: sasmekoll at gmail.com (Vovan) Date: Wed, 24 Nov 2010 23:45:46 +0300 Subject: [pypy-dev] =?utf-8?b?0J/QuNC60LDQvw==?= Message-ID: <4cf7a67a.4402cc0a.48f5.2225@mx.google.com> ?????????? ???????? ???????? ???? ???????????? ?????????????????????????? ???????????? ?? ?? ??????????????, ?????????????? ???????????????? ???????????????? ?? ???????? ???????????????? ???? ???????????? ?? ?????????????????????? ?????????? ???????? ???????????? ???? ???????????? ?????? ?????????????????????????? ?? ????????????????? , ?? ???????????? ?????????????????? ??????????????, ?????????? ???????????? ?????????????? ?????????????? ?????? ?????????? ???????????????????? ??????????????????!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101124/ea13295b/attachment.htm From sasmekoll at gmail.com Wed Nov 24 21:42:51 2010 From: sasmekoll at gmail.com (Vovan) Date: Wed, 24 Nov 2010 23:42:51 +0300 Subject: [pypy-dev] =?utf-8?b?0J/QuNC60LDQvw==?= Message-ID: <4cf7a5e5.948fcc0a.5118.2184@mx.google.com> ?????????? ???????? ???????? ???? ???????????? ?????????????????????????? ???????????? ?? ?? ??????????????, ?????????????? ???????????????? ???????????????? ?? ???????? ???????????????? ???? ???????????? ?? ?????????????????????? ?????????? ???????? ???????????? ???? ???????????? ?????? ?????????????????????????? ?? ????????????????? , ?? ???????????? ?????????????????? ??????????????, ?????????? ???????????? ?????????????? ?????????????? ?????? ?????????? ???????????????????? ??????????????????!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101124/b149a7c0/attachment.htm From sasmekoll at gmail.com Wed Nov 24 21:43:49 2010 From: sasmekoll at gmail.com (Vovan) Date: Wed, 24 Nov 2010 23:43:49 +0300 Subject: [pypy-dev] =?utf-8?b?0J/QuNC60LDQvw==?= Message-ID: <4cf7a616.948fcc0a.34d7.215c@mx.google.com> ?????????? ???????? ???????? ???? ???????????? ?????????????????????????? ???????????? ?? ?? ??????????????, ?????????????? ???????????????? ???????????????? ?? ???????? ???????????????? ???? ???????????? ?? ?????????????????????? ?????????? ???????? ???????????? ???? ???????????? ?????? ?????????????????????????? ?? ????????????????? , ?? ???????????? ?????????????????? ??????????????, ?????????? ???????????? ?????????????? ?????????????? ?????? ?????????? ???????????????????? ??????????????????!!! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101124/c3df3d94/attachment.htm From rndblnch at gmail.com Thu Dec 2 17:21:47 2010 From: rndblnch at gmail.com (rndblnch) Date: Thu, 2 Dec 2010 16:21:47 +0000 (UTC) Subject: [pypy-dev] pyopengl on pypy, take 2 Message-ID: hi, with your help, i have been able to patch pyopengl to make non-trivial glut/opengl interactive programs (involving shaders, vertex and pixel buffers) run on pypy. the programs: http://bitbucket.org/rndblnch/opengl-programmable/ some details about pyopengl modifications here: http://thread.gmane.org/gmane.comp.python.opengl.devel/1004/focus=1007 the only thing i add to change that is not already in the pypy svn is a minor check in _ctypes._Pointer.setcontents to make it raise a TypeError rather than an AttributeError if the value passed do not have a _ensure_objects method: class _Pointer(_CData): def setcontents(self, value): if not isinstance(value, self._type_): raise TypeError("expected %s instead of %s" % ( self._type_.__name__, type(value).__name__)) self._objects = {keepalive_key(1):value} + if not hasattr(value, "_ensure_objects"): + raise TypeError if value._ensure_objects() is not None: self._objects[keepalive_key(0)] = value._objects value = value._buffer self._buffer[0] = value a better check might be "if not isinstance(value, _CData)". i hope you will consider this minor change for inclusion into pypy's ctypes implementation. i still have other issues, but at least all my programs run and are interactive. i have not looked at the performance, but i have a question on this aspect: since most of the python code is executed as c callbacks passed to glut, should i except any gain at all? is pypy able to jit those functions wrapped by ctypes and passed down as callbacks to a c library? thanks again, renaud From sasmekoll at gmail.com Thu Nov 25 02:21:36 2010 From: sasmekoll at gmail.com (Vovan) Date: Thu, 25 Nov 2010 04:21:36 +0300 Subject: [pypy-dev] =?utf-8?b?0JbQtdC90YHQutCw0Y8g0LvQvtCz0LjQutCw?= Message-ID: <4cf7e88f.5989cc0a.4c3c.34f2@mx.google.com> ???????????? ???????????????? ????????????? ?????????? ???????????????? ???? ??????????, ?????????? ?????? ???????????????? ???????????????? ???? ???? ???????????????????? ???????????????????????? ????????????? ???? ???????????????????????? ?????????????????? ???????????????? ?????????????????? ???? ?? ???????????????? ???????????????? ?????????????????? ???????? ?????????????????????????????????? ?? ???????????????????? ???????????????? ?????? ????????????. http://samec.org.ua/?p=202 ???? ????????????? ????????????????????, ?????? ????????????????????. ?????????? ?????? ?????????????? ?????????????? ??????????????. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101125/89134871/attachment-0001.htm From renesd at gmail.com Fri Dec 3 00:28:22 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 2 Dec 2010 23:28:22 +0000 Subject: [pypy-dev] =?koi8-r?b?9sXO08vB0SDMz8fJy8E=?= In-Reply-To: <4cf7e88f.5989cc0a.4c3c.34f2@mx.google.com> References: <4cf7e88f.5989cc0a.4c3c.34f2@mx.google.com> Message-ID: >>>> a="""Want to understand women? Know the real their thoughts when they try to hide them in a completely abstract words? On the practical training of the Academy of Dating in the real world you learn to be attentive and understanding man for women. Do not believe me? Try it, you'll enjoy. I wish you a great scientific success.""" >>>> spam(a) True >>>> block(a.sender) ... 2010/11/25 Vovan > ?????? ???????? ??????? ????? ???????? ?? ?????, ????? ??? ???????? > ???????? ?? ?? ?????????? ???????????? ??????? ?? ???????????? ????????? > ???????? ????????? ?? ? ???????? ???????? ????????? ???? ????????????????? ? > ?????????? ???????? ??? ??????. > > http://samec.org.ua/?p=202 > > ?? ??????? ??????????, ??? ??????????. ????? ??? ??????? ??????? ???????. > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101202/c26d96a3/attachment.htm From sungsuha at gmail.com Fri Dec 3 11:19:53 2010 From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha) Date: Fri, 3 Dec 2010 10:19:53 +0000 (UTC) Subject: [pypy-dev] Windows source link Message-ID: http://pypy.org/download/pypy-1.4-src.zip Down for everyone or is it just me? From arigo at tunes.org Fri Dec 3 11:32:40 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 3 Dec 2010 11:32:40 +0100 Subject: [pypy-dev] Windows source link In-Reply-To: References: Message-ID: Hi Seung, On Fri, Dec 3, 2010 at 11:19 AM, Seung Soo, wrote: > http://pypy.org/download/pypy-1.4-src.zip > > Down for everyone or is it just me? Works for me. Try again? A bient?t, Armin. From fijall at gmail.com Fri Dec 3 12:02:37 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 3 Dec 2010 13:02:37 +0200 Subject: [pypy-dev] Windows source link In-Reply-To: References: Message-ID: I fixed it yesterday On Fri, Dec 3, 2010 at 12:32 PM, Armin Rigo wrote: > Hi Seung, > > On Fri, Dec 3, 2010 at 11:19 AM, Seung Soo, wrote: >> http://pypy.org/download/pypy-1.4-src.zip >> >> Down for everyone or is it just me? > > Works for me. ?Try again? > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From drsalists at gmail.com Fri Dec 3 20:04:23 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 3 Dec 2010 11:04:23 -0800 Subject: [pypy-dev] Testing if a function exists? Message-ID: How does one test if a function exists in pypy? In CPython 2.x and 3.x, it appears to be sufficient to use: 'funcname' in __bultins__ ...but that doesn't appear to work in pypy 1.4: >>>> print 'platform_version' in platform Traceback (most recent call last): File "", line 1, in TypeError: 'module' object is not iterable >>>> print 'bytes' in __builtins__ Traceback (most recent call last): File "", line 1, in TypeError: 'module' object is not iterable >>>> TIA! From alex.gaynor at gmail.com Fri Dec 3 20:07:55 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 3 Dec 2010 14:07:55 -0500 Subject: [pypy-dev] Testing if a function exists? In-Reply-To: References: Message-ID: On Fri, Dec 3, 2010 at 2:04 PM, Dan Stromberg wrote: > How does one test if a function exists in pypy? > > In CPython 2.x and 3.x, it appears to be sufficient to use: > > 'funcname' in __bultins__ > > ...but that doesn't appear to work in pypy 1.4: > > >>>> print 'platform_version' in platform > Traceback (most recent call last): > File "", line 1, in > TypeError: 'module' object is not iterable > >>>> print 'bytes' in __builtins__ > Traceback (most recent call last): > File "", line 1, in > TypeError: 'module' object is not iterable > >>>> > > TIA! > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > That doesn't work for me on CPython: alex at alex-laptop:~$ python Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> "bytes" in __builtins__ Traceback (most recent call last): File "", line 1, in TypeError: argument of type 'module' is not iterable You can use `hasattr` to check if a module has an attribute (function). Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101203/caee9603/attachment.htm From santagada at gmail.com Fri Dec 3 20:13:09 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Fri, 3 Dec 2010 17:13:09 -0200 Subject: [pypy-dev] Testing if a function exists? In-Reply-To: References: Message-ID: (I'm guessing but) In python 2.5 modules are not iterable, you can use getattr for the same effect. On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg wrote: > How does one test if a function exists in pypy? > > In CPython 2.x and 3.x, it appears to be sufficient to use: > > ? 'funcname' in __bultins__ > > ...but that doesn't appear to work in pypy 1.4: > >>>>> print 'platform_version' in platform > Traceback (most recent call last): > ?File "", line 1, in > TypeError: 'module' object is not iterable >>>>> print 'bytes' in __builtins__ > Traceback (most recent call last): > ?File "", line 1, in > TypeError: 'module' object is not iterable >>>>> -- Leonardo Santagada From fijall at gmail.com Fri Dec 3 20:37:44 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 3 Dec 2010 21:37:44 +0200 Subject: [pypy-dev] Testing if a function exists? In-Reply-To: References: Message-ID: Use: import __builtin__ and treat it as module. __builtins__ is an ugly hack that is sometimes a dict and sometimes a module and pypy has different corner cases. __builtin__ will always work On Fri, Dec 3, 2010 at 9:13 PM, Leonardo Santagada wrote: > (I'm guessing but) In python 2.5 modules are not iterable, you can use > getattr for the same effect. > > On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg wrote: >> How does one test if a function exists in pypy? >> >> In CPython 2.x and 3.x, it appears to be sufficient to use: >> >> ? 'funcname' in __bultins__ >> >> ...but that doesn't appear to work in pypy 1.4: >> >>>>>> print 'platform_version' in platform >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: 'module' object is not iterable >>>>>> print 'bytes' in __builtins__ >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: 'module' object is not iterable >>>>>> > > > > -- > Leonardo Santagada > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From jacob at openend.se Sat Dec 4 00:42:59 2010 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Sat, 4 Dec 2010 00:42:59 +0100 Subject: [pypy-dev] Idea for speed.pypy.org Message-ID: <201012040043.04547.jacob@openend.se> Something that would be nice on the speed.pypy.org site would be that each benchmark links to a page that describes the benchmark - What the benchmark is designed to measure - Why PyPy performs better/worse than CPYthon (if known) - What might be done to further improve performance (if there are ideas) - Link to the source code of the benchmark Any takers? Jacob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101204/3ba9958f/attachment.pgp From exarkun at twistedmatrix.com Sat Dec 4 04:14:55 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sat, 04 Dec 2010 03:14:55 -0000 Subject: [pypy-dev] Idea for speed.pypy.org Message-ID: <20101204031455.2109.1197734270.divmod.xquotient.1807@localhost.localdomain> On 3 Dec, 11:42 pm, jacob at openend.se wrote: >Something that would be nice on the speed.pypy.org site would be that >each >benchmark links to a page that describes the benchmark > >- What the benchmark is designed to measure >- Why PyPy performs better/worse than CPYthon (if known) >- What might be done to further improve performance (if there are >ideas) >- Link to the source code of the benchmark > >Any takers? There's a column in the database where descriptions can be added - this could either be the full description you're talking about, or a link to a page with more information. The description is rendered on hover, and below the large version of the graph (if you have a recent enough codespeed). So it may just be a matter of entering some text for each benchmark (as opposed to making changes to the codespeed software). Jean-Paul From drsalists at gmail.com Sat Dec 4 06:42:33 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 3 Dec 2010 21:42:33 -0800 Subject: [pypy-dev] Testing if a function exists? In-Reply-To: References: Message-ID: This sounds promising, though sadly, it doesn't appear to work in Python 3.x: $ /usr/local/cpython-3.1/bin/python3 -c 'import __builtin__' Traceback (most recent call last): File "", line 1, in ImportError: No module named __builtin__ benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu 22275 - above cmd done 2010 Fri Dec 03 09:40 PM $ /usr/local/cpython-3.0/bin/python3.0 -c 'import __builtin__' Traceback (most recent call last): File "", line 1, in ImportError: No module named __builtin__ benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu 22275 - above cmd done 2010 Fri Dec 03 09:40 PM $ /usr/local/cpython-2.7/bin/python -c 'import __builtin__' benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu 22275 - above cmd done 2010 Fri Dec 03 09:41 PM $ /usr/local/cpython-2.6/bin/python -c 'import __builtin__' benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu 22275 - above cmd done 2010 Fri Dec 03 09:41 PM $ /usr/local/cpython-2.5/bin/python -c 'import __builtin__' benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu 22275 - above cmd done 2010 Fri Dec 03 09:41 PM $ On Fri, Dec 3, 2010 at 11:37 AM, Maciej Fijalkowski wrote: > Use: > > import __builtin__ > > and treat it as module. __builtins__ is an ugly hack that is sometimes > a dict and sometimes a module and pypy has different corner cases. > __builtin__ will always work > > On Fri, Dec 3, 2010 at 9:13 PM, Leonardo Santagada wrote: >> (I'm guessing but) In python 2.5 modules are not iterable, you can use >> getattr for the same effect. >> >> On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg wrote: >>> How does one test if a function exists in pypy? >>> >>> In CPython 2.x and 3.x, it appears to be sufficient to use: >>> >>> ? 'funcname' in __bultins__ >>> >>> ...but that doesn't appear to work in pypy 1.4: >>> >>>>>>> print 'platform_version' in platform >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> TypeError: 'module' object is not iterable >>>>>>> print 'bytes' in __builtins__ >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> TypeError: 'module' object is not iterable >>>>>>> >> >> >> >> -- >> Leonardo Santagada >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Sat Dec 4 07:15:58 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 4 Dec 2010 08:15:58 +0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: <201012040043.04547.jacob@openend.se> References: <201012040043.04547.jacob@openend.se> Message-ID: > - Why PyPy performs better/worse than CPYthon (if known) I'm a bit worried about keeping this up to date From fijall at gmail.com Sat Dec 4 07:17:29 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 4 Dec 2010 08:17:29 +0200 Subject: [pypy-dev] Testing if a function exists? In-Reply-To: References: Message-ID: According to this: http://docs.python.org/py3k/whatsnew/3.0.html they renamed __builtin__ to builtins (with removed underscores). So probably a bunch of ifs would do On Sat, Dec 4, 2010 at 7:42 AM, Dan Stromberg wrote: > This sounds promising, though sadly, it doesn't appear to work in Python 3.x: > > $ /usr/local/cpython-3.1/bin/python3 -c 'import __builtin__' > Traceback (most recent call last): > ?File "", line 1, in > ImportError: No module named __builtin__ > benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu > 22275 - above cmd done 2010 Fri Dec 03 09:40 PM > > $ /usr/local/cpython-3.0/bin/python3.0 -c 'import __builtin__' > Traceback (most recent call last): > ?File "", line 1, in > ImportError: No module named __builtin__ > benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu > 22275 - above cmd done 2010 Fri Dec 03 09:40 PM > > $ /usr/local/cpython-2.7/bin/python -c 'import __builtin__' > benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu > 22275 - above cmd done 2010 Fri Dec 03 09:41 PM > > $ /usr/local/cpython-2.6/bin/python -c 'import __builtin__' > benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu > 22275 - above cmd done 2010 Fri Dec 03 09:41 PM > > $ /usr/local/cpython-2.5/bin/python -c 'import __builtin__' > benchbox-dstromberg:~/src/home-svn/backshift/trunk i686-pc-linux-gnu > 22275 - above cmd done 2010 Fri Dec 03 09:41 PM > > $ > > > On Fri, Dec 3, 2010 at 11:37 AM, Maciej Fijalkowski wrote: >> Use: >> >> import __builtin__ >> >> and treat it as module. __builtins__ is an ugly hack that is sometimes >> a dict and sometimes a module and pypy has different corner cases. >> __builtin__ will always work >> >> On Fri, Dec 3, 2010 at 9:13 PM, Leonardo Santagada wrote: >>> (I'm guessing but) In python 2.5 modules are not iterable, you can use >>> getattr for the same effect. >>> >>> On Fri, Dec 3, 2010 at 5:04 PM, Dan Stromberg wrote: >>>> How does one test if a function exists in pypy? >>>> >>>> In CPython 2.x and 3.x, it appears to be sufficient to use: >>>> >>>> ? 'funcname' in __bultins__ >>>> >>>> ...but that doesn't appear to work in pypy 1.4: >>>> >>>>>>>> print 'platform_version' in platform >>>> Traceback (most recent call last): >>>> ?File "", line 1, in >>>> TypeError: 'module' object is not iterable >>>>>>>> print 'bytes' in __builtins__ >>>> Traceback (most recent call last): >>>> ?File "", line 1, in >>>> TypeError: 'module' object is not iterable >>>>>>>> >>> >>> >>> >>> -- >>> Leonardo Santagada >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >> > From angelflow at yahoo.com Sat Dec 4 08:21:02 2010 From: angelflow at yahoo.com (Andy) Date: Fri, 3 Dec 2010 23:21:02 -0800 (PST) Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: <201012040043.04547.jacob@openend.se> Message-ID: <240465.94379.qm@web111314.mail.gq1.yahoo.com> The one thing that would be the most helpful is that for each test shows the memory usage comparison between PyPy-JIT & CPython. --- On Fri, 12/3/10, Jacob Hall?n wrote: > From: Jacob Hall?n > Subject: [pypy-dev] Idea for speed.pypy.org > To: pypy-dev at codespeak.net > Date: Friday, December 3, 2010, 6:42 PM > Something that would be nice on the > speed.pypy.org site would be that each > benchmark links to a page that describes the benchmark > > - What the benchmark is designed to measure > - Why PyPy performs better/worse than CPYthon (if known) > - What might be done to further improve performance (if > there are ideas) > - Link to the source code of the benchmark > > Any takers? > > Jacob > > -----Inline Attachment Follows----- > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From tobami at googlemail.com Sat Dec 4 08:57:51 2010 From: tobami at googlemail.com (Miquel Torres) Date: Sat, 4 Dec 2010 08:57:51 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: <240465.94379.qm@web111314.mail.gq1.yahoo.com> References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> Message-ID: Hi Jacob, benchmark descriptions is certainly something that could be improved upon. As exarkun said, a quick way to get *something* is to just fill out the description fields for each benchmark. I can happily do that, I just need a mail containing a list with brief descriptions for all the benchmarks ;-) I can even imaging improving it further so that there is a link to source code (your point 4) or even an info page dedicated to each benchmark. Points 2 and 3, as Maciej said, is problematic. Reasons why it performs better or worse than cpython and how it could be improved is something that may change fairly often for some benchmarks, which would be a pain to maintain. And speed.pypy.org doesn't have 3 or 4 benchmarks, it has 25 right now! @Andy: memory consumption is difficult to measure. I think someone was looking at how to best measure it? Cheers, Miquel 2010/12/4 Andy : > The one thing that would be the most helpful is that for each test shows the memory usage comparison between PyPy-JIT & CPython. > > > --- On Fri, 12/3/10, Jacob Hall?n wrote: > >> From: Jacob Hall?n >> Subject: [pypy-dev] Idea for speed.pypy.org >> To: pypy-dev at codespeak.net >> Date: Friday, December 3, 2010, 6:42 PM >> Something that would be nice on the >> speed.pypy.org site would be that each >> benchmark links to a page that describes the benchmark >> >> - What the benchmark is designed to measure >> - Why PyPy performs better/worse than CPYthon (if known) >> - What might be done to further improve performance (if >> there are ideas) >> - Link to the source code of the benchmark >> >> Any takers? >> >> Jacob >> >> -----Inline Attachment Follows----- >> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From lac at openend.se Sat Dec 4 12:05:56 2010 From: lac at openend.se (Laura Creighton) Date: Sat, 04 Dec 2010 12:05:56 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: Message from Miquel Torres of "Sat, 04 Dec 2010 08:57:51 +0100." References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> Message-ID: <201012041105.oB4B5uo1019532@theraft.openend.se> re: keeping the 'why we are slower/what we could do to fix it' info up to date -- one possibility is to make a 'why we were/what we are for release 1.4.' Then every time you make a major release, you update those fields as needed. And if major changes happen between 'what was in the last major release' vs 'what's on trunk now' you can even make a note of it -- 'fixed in rev whatever it was, when we merged in whoever it was' brank, see blog post over here'. And if you forget, well, you will catch it when the next major release comes out. Just an idea. Laura From fijall at gmail.com Sat Dec 4 13:35:18 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 4 Dec 2010 14:35:18 +0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: <201012041105.oB4B5uo1019532@theraft.openend.se> References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: On Sat, Dec 4, 2010 at 1:05 PM, Laura Creighton wrote: > re: keeping the 'why we are slower/what we could do to fix it' info up > to date -- one possibility is to make a 'why we were/what we are for > release 1.4.' ?Then every time you make a major release, you update > those fields as needed. ?And if major changes happen between 'what > was in the last major release' vs 'what's on trunk now' you can even > make a note of it -- 'fixed in rev whatever it was, when we merged in > whoever it was' brank, see blog post over here'. ?And if you forget, > well, you will catch it when the next major release comes out. > > Just an idea. > > Laura > I think the general idea "we know what's wrong" vs "we have no clue" is good. However, I would like to maybe decide what we do with tons of docs that we already have first :) From tobami at googlemail.com Wed Dec 8 19:41:51 2010 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 8 Dec 2010 19:41:51 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: Hey, I was serious when I said I would improve on benchmarks info! Anyone want to help in creating that benchmark description list? Just reply to this mail so that everyone can "review" what the benchmarks do. 2010/12/4 Maciej Fijalkowski : > On Sat, Dec 4, 2010 at 1:05 PM, Laura Creighton wrote: >> re: keeping the 'why we are slower/what we could do to fix it' info up >> to date -- one possibility is to make a 'why we were/what we are for >> release 1.4.' ?Then every time you make a major release, you update >> those fields as needed. ?And if major changes happen between 'what >> was in the last major release' vs 'what's on trunk now' you can even >> make a note of it -- 'fixed in rev whatever it was, when we merged in >> whoever it was' brank, see blog post over here'. ?And if you forget, >> well, you will catch it when the next major release comes out. >> >> Just an idea. >> >> Laura >> > > I think the general idea "we know what's wrong" vs "we have no clue" > is good. However, I would like to maybe decide what we do with tons of > docs that we already have first :) > From fijall at gmail.com Wed Dec 8 20:03:36 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 8 Dec 2010 21:03:36 +0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: On Wed, Dec 8, 2010 at 8:41 PM, Miquel Torres wrote: > Hey, I was serious when I said I would improve on benchmarks info! > > Anyone want to help in creating that benchmark description list? Just > reply to this mail so that everyone can "review" what the benchmarks > do. > Hey. I can do some stuff. I was serious about documenting why we're slow/fast on benchmarks - that maybe we should bring down our docs to a manageable number first :) Benchmarks descriptions are however unlikely to change > > 2010/12/4 Maciej Fijalkowski : >> On Sat, Dec 4, 2010 at 1:05 PM, Laura Creighton wrote: >>> re: keeping the 'why we are slower/what we could do to fix it' info up >>> to date -- one possibility is to make a 'why we were/what we are for >>> release 1.4.' ?Then every time you make a major release, you update >>> those fields as needed. ?And if major changes happen between 'what >>> was in the last major release' vs 'what's on trunk now' you can even >>> make a note of it -- 'fixed in rev whatever it was, when we merged in >>> whoever it was' brank, see blog post over here'. ?And if you forget, >>> well, you will catch it when the next major release comes out. >>> >>> Just an idea. >>> >>> Laura >>> >> >> I think the general idea "we know what's wrong" vs "we have no clue" >> is good. However, I would like to maybe decide what we do with tons of >> docs that we already have first :) >> > From santagada at gmail.com Thu Dec 9 14:14:44 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 9 Dec 2010 11:14:44 -0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: Here is a incomplete draft list: ai: chaos: create chaosgame-like fractals fannkuch: http://shootout.alioth.debian.org/u64/performance.php?test=fannkuchredux float: (this is just from looking at it) creates an array of points using circular projection and then normalizes and maximizes them. html5: meteor-contest: http://shootout.alioth.debian.org/u64/performance.php?test=meteor nbody_modified: http://shootout.alioth.debian.org/u64/performance.php?test=nbody richards: Martin Richards benchmark, implemented in many languages rietveld: A Django application benchmark [slow]spitfire[cstringio]: Spitfire is a template language, the cstringio version uses a modified engine (that uses cstringio) spambayes: Spambayes is a bayesian spam filter telco: go: A go (chess like game) computer player AI pyflate-fast: Stand-alone pure-Python DEFLATE (gzip) and bzip2 decoder/decompressor. raytrace: A raytracer renderer crypto_pyaes: A pure python implementation of AES waf: Waf is a Python-based framework for configuring, compiling and installing applications. It derives from the concepts of other build tools such as Scons, Autotools, CMake or Ant. On Wed, Dec 8, 2010 at 4:41 PM, Miquel Torres wrote: > Anyone want to help in creating that benchmark description list? Just > reply to this mail so that everyone can "review" what the benchmarks > do. -- Leonardo Santagada From jacob at openend.se Thu Dec 9 14:15:21 2010 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Thu, 9 Dec 2010 14:15:21 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> Message-ID: <201012091415.27316.jacob@openend.se> Wednesday 08 December 2010 you wrote: > On Wed, Dec 8, 2010 at 8:41 PM, Miquel Torres wrote: > > Hey, I was serious when I said I would improve on benchmarks info! > > > > Anyone want to help in creating that benchmark description list? Just > > reply to this mail so that everyone can "review" what the benchmarks > > do. > > Hey. > > I can do some stuff. I was serious about documenting why we're > slow/fast on benchmarks - that maybe we should bring down our docs to > a manageable number first :) Benchmarks descriptions are however > unlikely to change Just having the descriptions would be great. Why we are slow/fast was a wishlist item, and I can see that it is more difficult to provide this information and keep it up to date. Providing Miguel with the information can be done in the simplest way, for instance by mailing texts to this list (so that people can see which ones have been produced).. Jacob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101209/3a221a57/attachment.pgp From jacob at openend.se Thu Dec 9 15:30:35 2010 From: jacob at openend.se (Jacob =?iso-8859-15?q?Hall=E9n?=) Date: Thu, 9 Dec 2010 15:30:35 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: <201012091415.27316.jacob@openend.se> References: <201012040043.04547.jacob@openend.se> <201012091415.27316.jacob@openend.se> Message-ID: <201012091530.41121.jacob@openend.se> Extracted from what exarkun said on the IRC channel. twisted-tcp: Connects one Twised client to one Twisted server over TCP (on the loopback interface) and then writes bytes as fast as it can. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101209/2865cc1b/attachment.pgp From renesd at gmail.com Thu Dec 9 15:50:28 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 9 Dec 2010 14:50:28 +0000 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: <201012091530.41121.jacob@openend.se> References: <201012040043.04547.jacob@openend.se> <201012091415.27316.jacob@openend.se> <201012091530.41121.jacob@openend.se> Message-ID: Hello, Why is this twisted tcp benchmark slow in pypy? Does memory management play a role? Or perhaps the interface to the system calls? eg, is there no event io being used by the twisted version? I think in CPython, it would be using a c binding, and not allocating any memory as it would all be coming from memory pools. It's probably using the same pieces of memory too. cu. On Thu, Dec 9, 2010 at 2:30 PM, Jacob Hall?n wrote: > Extracted from what exarkun said on the IRC channel. > > twisted-tcp: > > Connects one Twised client to one Twisted server over TCP (on the loopback > interface) and then writes bytes as fast as it can. > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101209/4de8a1f6/attachment.htm From p.giarrusso at gmail.com Thu Dec 9 16:13:09 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Thu, 9 Dec 2010 16:13:09 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada wrote: > Here is a incomplete draft list: > [slow]spitfire[cstringio]: Spitfire is a template language, the > cstringio version uses a modified engine (that uses cstringio) > spambayes: Spambayes is a bayesian spam filter Why is [slow]spitfire slower with PyPy? Is it regex-related? I remember when, because of this, spambayes was slower (including release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4 and the head (for slowspitfire it's the opposite). For the rest, I see no significant case of slowdown of PyPy over time. http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From arigo at tunes.org Thu Dec 9 16:14:15 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 9 Dec 2010 16:14:15 +0100 Subject: [pypy-dev] Leysin Winter sprint: Jan 16-22 Message-ID: ===================================================================== PyPy Leysin Winter Sprint (16-22nd January 2011) ===================================================================== As announced on http://morepypy.blogspot.com/2010/12/leysin-winter-sprint.html, the next PyPy sprint will be in Leysin, Switzerland, for the seventh time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. ------------------------------ Goals and topics of the sprint ------------------------------ * Now that we have released 1.4, and plan to release 1.4.1 soon (possibly before the sprint), the sprint itself is going to be mainly working on fixing issues reported by various users. Of course this does not prevent people from showing up with a more precise interest in mind. If there are newcomers, we will gladly give introduction talks. * We will also work on polishing and merging the long-standing branches that are around, which could eventually lead to the next PyPy release. These branches are notably: - fast-forward (Python 2.7 support, by Benjamin, Amaury, and others) - jit-unroll-loops (improve JITting of smaller loops, by Hakan) - arm-backend (a JIT backend for ARM, by David) - jitypes2 (fast ctypes calls with the JIT, by Antonio). * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski. ----------- Exact times ----------- The work days should be 16-22 January 2011. People may arrive on the 15th already and/or leave on the 23rd. ----------------------- Location & Accomodation ----------------------- Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connexion with wireless installed. You can of course arrange your own lodging anywhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-) Please *confirm* that you are coming so that we can adjust the reservations as appropriate. The rate so far has been around 60 CHF a night all included in 2-person rooms, with breakfast. There are larger rooms too (less expensive) and maybe the possibility to get a single room if you really want to. Please register by svn: http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-winter-2011/people.txt or on the pypy-sprint mailing list if you do not yet have check-in rights: http://codespeak.net/mailman/listinfo/pypy-sprint You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around -- bring a EU-format power strip if you have one. ===================================================================== Armin Rigo From drsalists at gmail.com Thu Dec 9 17:25:40 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 9 Dec 2010 08:25:40 -0800 Subject: [pypy-dev] os.major? os.minor? Message-ID: What would be the best way to add os.major and os.minor to the os module? They appear to be C macros on some systems, so I'm guessing most FFI's aren't going to see them without a wrapper. On Linux, there's a C function that a macro is #ifdef'd to, but that seems less than amazingly portable. From arigo at tunes.org Thu Dec 9 17:56:17 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 9 Dec 2010 17:56:17 +0100 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: Hi Dan, On Thu, Dec 9, 2010 at 5:25 PM, Dan Stromberg wrote: > What would be the best way to add os.major and os.minor to the os module? It's amazing the kind of stuff there is in the 'os' module that I've never heard about before. :-) > They appear to be C macros on some systems, so I'm guessing most FFI's > aren't going to see them without a wrapper. Indeed. See for example pypy.rlib._rsocket_rffi.FD_ZERO, which is also typically a macro. There is a bit of workaround code to write, I'm afraid. This is mostly to support direct testing; if the goal is only to have it work after translation, then it does not matter that it is actually a macro. A bient?t, Armin. From drsalists at gmail.com Thu Dec 9 18:20:04 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 9 Dec 2010 09:20:04 -0800 Subject: [pypy-dev] gdbm and dbm merge - ctypes bug? Message-ID: I'm finding that my gdbm module (based on ctypes) is infrequently (but it might be consistent) giving gibberish when run on pypy 1.4, but not when run on cpython 2.6 or cpython 3.1. More specifically, sometimes the keys come up looking like undefined memory. The values associated with those keys always seem OK. I believe I am comparing apples to apples - I'm using gdbm_ctypes with the cpython's too, even though they come with a module for some doing the same thing - for the sake of comparison. Is there a known problem with pypy's ctypes? The code is at http://stromberg.dnsalias.org/cgi-bin/viewvc.cgi/gdbm-ctypes/trunk/?root=svn The tests are at http://stromberg.dnsalias.org/cgi-bin/viewvc.cgi/backshift/trunk/?root=svn , and are a bit involved right now, but I could look into finding a minimal code snippet that shows the same problem. From arigo at tunes.org Thu Dec 9 19:04:36 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 9 Dec 2010 19:04:36 +0100 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: Re-hi, I have opened an issue about the missing functions from the 'os' module: https://codespeak.net/issue/pypy-dev/issue605 I will probably get around to implement a part of them in the next few days. If you plan to do more than os.major() and os.minor(), let's coordinate there. A bient?t, Armin. From amauryfa at gmail.com Thu Dec 9 19:05:56 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 9 Dec 2010 19:05:56 +0100 Subject: [pypy-dev] gdbm and dbm merge - ctypes bug? In-Reply-To: References: Message-ID: Hi 2010/12/9 Dan Stromberg > Is there a known problem with pypy's ctypes? Yes, several of them have already been fixed in trunk, especially about memory chunks being freed too early. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101209/e0d9cc32/attachment.htm From arigo at tunes.org Thu Dec 9 19:33:35 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 9 Dec 2010 19:33:35 +0100 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: Hi, Sorry for the series of mails. I just wanted to add that after looking more at the code, it's fine if you implement major() and minor() the same way as the other posix functions (see e.g. r79941 adding os.fchdir()), calling the major() and minor() "functions" in C. If you add tests in the same places I did in r79941, then it should work even if the functions are actually macros. One place where it would not work is in translated code running on the llinterp; the other place might be the JIT trying to get the address of the "function". (If that's the case, we should fix that a bit more generally.) A bient?t, Armin. From fijall at gmail.com Thu Dec 9 21:39:52 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 9 Dec 2010 22:39:52 +0200 Subject: [pypy-dev] [pypy-svn] r79938 - in pypy/trunk/pypy: module/posix module/posix/test rpython/module translator/c/test In-Reply-To: <20101209173035.299A0282B9D@codespeak.net> References: <20101209173035.299A0282B9D@codespeak.net> Message-ID: Do we *really* need those functions available on RPython level? On Thu, Dec 9, 2010 at 7:30 PM, wrote: > Author: arigo > Date: Thu Dec ?9 18:30:33 2010 > New Revision: 79938 > > Modified: > ? pypy/trunk/pypy/module/posix/__init__.py > ? pypy/trunk/pypy/module/posix/interp_posix.py > ? pypy/trunk/pypy/module/posix/test/test_posix2.py > ? pypy/trunk/pypy/rpython/module/ll_os.py > ? pypy/trunk/pypy/translator/c/test/test_extfunc.py > Log: > Implement os.getloadavg(). ?(Phew, we really need to edit files in all > corners of the world for this...) > > > Modified: pypy/trunk/pypy/module/posix/__init__.py > ============================================================================== > --- pypy/trunk/pypy/module/posix/__init__.py ? ?(original) > +++ pypy/trunk/pypy/module/posix/__init__.py ? ?Thu Dec ?9 18:30:33 2010 > @@ -111,6 +111,8 @@ > ? ? ? ? interpleveldefs['sysconf_names'] = 'space.wrap(os.sysconf_names)' > ? ? if hasattr(os, 'ttyname'): > ? ? ? ? interpleveldefs['ttyname'] = 'interp_posix.ttyname' > + ? ?if hasattr(os, 'getloadavg'): > + ? ? ? ?interpleveldefs['getloadavg'] = 'interp_posix.getloadavg' > > ? ? for name in ['setsid', 'getuid', 'geteuid', 'getgid', 'getegid', 'setuid', > ? ? ? ? ? ? ? ? ?'seteuid', 'setgid', 'setegid', 'getpgrp', 'setpgrp', > > Modified: pypy/trunk/pypy/module/posix/interp_posix.py > ============================================================================== > --- pypy/trunk/pypy/module/posix/interp_posix.py ? ? ? ?(original) > +++ pypy/trunk/pypy/module/posix/interp_posix.py ? ? ? ?Thu Dec ?9 18:30:33 2010 > @@ -963,6 +963,17 @@ > ? ? return space.w_None > ?chown.unwrap_spec = [ObjSpace, str, "c_nonnegint", "c_nonnegint"] > > +def getloadavg(space): > + ? ?try: > + ? ? ? ?load = os.getloadavg() > + ? ?except OSError, e: > + ? ? ? ?raise OperationError(space.w_OSError, > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? space.wrap("Load averages are unobtainable")) > + ? ?return space.newtuple([space.wrap(load[0]), > + ? ? ? ? ? ? ? ? ? ? ? ? ? space.wrap(load[1]), > + ? ? ? ? ? ? ? ? ? ? ? ? ? space.wrap(load[2])]) > +getloadavg.unwrap_spec = [ObjSpace] > + > ?if _WIN: > ? ? from pypy.rlib import rwin32 > > > Modified: pypy/trunk/pypy/module/posix/test/test_posix2.py > ============================================================================== > --- pypy/trunk/pypy/module/posix/test/test_posix2.py ? ?(original) > +++ pypy/trunk/pypy/module/posix/test/test_posix2.py ? ?Thu Dec ?9 18:30:33 2010 > @@ -521,6 +521,14 @@ > ? ? ? ? ? ? ? ? assert os.WIFEXITED(status) > ? ? ? ? ? ? ? ? assert os.WEXITSTATUS(status) == exit_status > > + ? ?if hasattr(os, 'getloadavg'): > + ? ? ? ?def test_os_getloadavg(self): > + ? ? ? ? ? ?os = self.posix > + ? ? ? ? ? ?l0, l1, l2 = os.getloadavg() > + ? ? ? ? ? ?assert type(l0) is float and l0 >= 0.0 > + ? ? ? ? ? ?assert type(l1) is float and l0 >= 0.0 > + ? ? ? ? ? ?assert type(l2) is float and l0 >= 0.0 > + > ? ? if hasattr(os, 'fsync'): > ? ? ? ? def test_fsync(self): > ? ? ? ? ? ? os = self.posix > > Modified: pypy/trunk/pypy/rpython/module/ll_os.py > ============================================================================== > --- pypy/trunk/pypy/rpython/module/ll_os.py ? ? (original) > +++ pypy/trunk/pypy/rpython/module/ll_os.py ? ? Thu Dec ?9 18:30:33 2010 > @@ -730,6 +730,22 @@ > ? ? ? ? return extdef([traits.str, int, int], int, traits.ll_os_name('open'), > ? ? ? ? ? ? ? ? ? ? ? llimpl=os_open_llimpl, oofakeimpl=os_open_oofakeimpl) > > + ? ?@registering_if(os, 'getloadavg') > + ? ?def register_os_getloadavg(self): > + ? ? ? ?AP = rffi.CArrayPtr(lltype.Float) > + ? ? ? ?c_getloadavg = self.llexternal('getloadavg', [AP, rffi.INT], rffi.INT) > + > + ? ? ? ?def getloadavg_llimpl(): > + ? ? ? ? ? ?load = lltype.malloc(AP.TO, 3, flavor='raw') > + ? ? ? ? ? ?r = c_getloadavg(load, 3) > + ? ? ? ? ? ?result_tuple = load[0], load[1], load[2] > + ? ? ? ? ? ?lltype.free(load, flavor='raw') > + ? ? ? ? ? ?if r != 3: > + ? ? ? ? ? ? ? ?raise OSError > + ? ? ? ? ? ?return result_tuple > + ? ? ? ?return extdef([], (float, float, float), > + ? ? ? ? ? ? ? ? ? ? ?"ll_os.ll_getloadavg", llimpl=getloadavg_llimpl) > + > ?# ------------------------------- os.read ------------------------------- > > ? ? @registering(os.read) > > Modified: pypy/trunk/pypy/translator/c/test/test_extfunc.py > ============================================================================== > --- pypy/trunk/pypy/translator/c/test/test_extfunc.py ? (original) > +++ pypy/trunk/pypy/translator/c/test/test_extfunc.py ? Thu Dec ?9 18:30:33 2010 > @@ -755,3 +755,13 @@ > ? ? ? ? for i in range(5): > ? ? ? ? ? ? res = func(i) > ? ? ? ? ? ? assert res == os.uname()[i] > + > +if hasattr(os, 'getloadavg'): > + ? ?def test_os_getloadavg(): > + ? ? ? ?def does_stuff(): > + ? ? ? ? ? ?a, b, c = os.getloadavg() > + ? ? ? ? ? ?print a, b, c > + ? ? ? ? ? ?return a + b + c > + ? ? ? ?f = compile(does_stuff, []) > + ? ? ? ?res = f() > + ? ? ? ?assert type(res) is float and res >= 0.0 > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From benjamin at python.org Fri Dec 10 02:45:46 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 9 Dec 2010 19:45:46 -0600 Subject: [pypy-dev] [pypy-svn] r79938 - in pypy/trunk/pypy: module/posix module/posix/test rpython/module translator/c/test In-Reply-To: References: <20101209173035.299A0282B9D@codespeak.net> Message-ID: 2010/12/9 Maciej Fijalkowski : > Do we *really* need those functions available on RPython level? Even if we don't, I'd say the consistency is nice, rather than having ll implementations all over creation. -- Regards, Benjamin From arigo at tunes.org Fri Dec 10 10:27:19 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 10 Dec 2010 10:27:19 +0100 Subject: [pypy-dev] [pypy-svn] r79938 - in pypy/trunk/pypy: module/posix module/posix/test rpython/module translator/c/test In-Reply-To: References: <20101209173035.299A0282B9D@codespeak.net> Message-ID: Hi, On Fri, Dec 10, 2010 at 2:45 AM, Benjamin Peterson wrote: > 2010/12/9 Maciej Fijalkowski : >> Do we *really* need those functions available on RPython level? > > Even if we don't, I'd say the consistency is nice, rather than having > ll implementations all over creation. Agreed. Consistency is probably better than having pypy/module/posix be half written in the way it is now and half written in a lltype-specific way. A bient?t, Armin. From arigo at tunes.org Fri Dec 10 17:01:58 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 10 Dec 2010 17:01:58 +0100 Subject: [pypy-dev] Leysin Winter sprint: Jan 16-22 In-Reply-To: References: Message-ID: Hi, An additional comment (thanks Michael Foord for pointing this out to me): the Python Software Foundation (PSF) now has some money to use to help non-core-Python sprints. This means that if anyone is interested in getting some financial support for the travel to or lodging at Leysin, it might be possible from the PSF. Please write to me or to pypy-sprint at codespeak.net for more details. A bient?t, Armin. From drsalists at gmail.com Fri Dec 10 20:49:11 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 10 Dec 2010 11:49:11 -0800 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo wrote: > Hi, > > Sorry for the series of mails. ?I just wanted to add that after > looking more at the code, it's fine if you implement major() and > minor() the same way as the other posix functions (see e.g. r79941 > adding os.fchdir()), calling the major() and minor() "functions" in C. > ?If you add tests in the same places I did in r79941, then it should > work even if the functions are actually macros. ?One place where it > would not work is in translated code running on the llinterp; the > other place might be the JIT trying to get the address of the > "function". ?(If that's the case, we should fix that a bit more > generally.) > > > A bient?t, > > Armin. I attempted to add os.major and os.minor using fchdir as an example, but... I have to admit: I really don't know what to make of this error. Comments? [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 290, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/driver.py", line 812, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/driver.py", line 288, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/translator/driver.py", line 325, in task_annotate [translation:ERROR] s = annotator.build_types(self.entry_point, self.inputtypes) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py", line 103, in build_types [translation:ERROR] return self.build_graph_types(flowgraph, inputcells, complete_now=complete_now) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py", line 194, in build_graph_types [translation:ERROR] self.complete() [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py", line 250, in complete [translation:ERROR] self.processblock(graph, block) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py", line 473, in processblock [translation:ERROR] self.flowin(graph, block) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py", line 533, in flowin [translation:ERROR] self.consider_op(block.operations[i]) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py", line 716, in consider_op [translation:ERROR] argcells = [self.binding(a) for a in op.args] [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/annrpython.py", line 293, in binding [translation:ERROR] return self.bookkeeper.immutableconstant(arg) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/bookkeeper.py", line 314, in immutableconstant [translation:ERROR] return self.immutablevalue(const.value) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/bookkeeper.py", line 464, in immutablevalue [translation:ERROR] result = SomePBC([self.getdesc(x)]) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-stock-builds/pypy/annotation/bookkeeper.py", line 530, in getdesc [translation:ERROR] pyobj,)) [translation:ERROR] Exception: unexpected prebuilt constant: [translation:ERROR] Processing block: [translation:ERROR] block at 12 is a [translation:ERROR] in (pypy.module.posix.interp_posix:179)minor [translation:ERROR] containing the following operations: [translation:ERROR] v775 = simple_call((builtin_function_or_method minor), raw_device_no_0) [translation:ERROR] --end-- [translation] start debugger... From benjamin at python.org Fri Dec 10 20:50:36 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 10 Dec 2010 13:50:36 -0600 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: 2010/12/10 Dan Stromberg : > On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo wrote: >> Hi, >> >> Sorry for the series of mails. ?I just wanted to add that after >> looking more at the code, it's fine if you implement major() and >> minor() the same way as the other posix functions (see e.g. r79941 >> adding os.fchdir()), calling the major() and minor() "functions" in C. >> ?If you add tests in the same places I did in r79941, then it should >> work even if the functions are actually macros. ?One place where it >> would not work is in translated code running on the llinterp; the >> other place might be the JIT trying to get the address of the >> "function". ?(If that's the case, we should fix that a bit more >> generally.) >> >> >> A bient?t, >> >> Armin. > > I attempted to add os.major and os.minor using fchdir as an example, > but... ?I have to admit: I really don't know what to make of this > error. ?Comments? You probably failed to add an ll implementation in pypy/rpython/lltypesystem/module/ll_os.py -- Regards, Benjamin From drsalists at gmail.com Fri Dec 10 22:04:31 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 10 Dec 2010 13:04:31 -0800 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: On Fri, Dec 10, 2010 at 11:50 AM, Benjamin Peterson wrote: > 2010/12/10 Dan Stromberg : >> On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo wrote: >>> Hi, >>> >>> Sorry for the series of mails. ?I just wanted to add that after >>> looking more at the code, it's fine if you implement major() and >>> minor() the same way as the other posix functions (see e.g. r79941 >>> adding os.fchdir()), calling the major() and minor() "functions" in C. >>> ?If you add tests in the same places I did in r79941, then it should >>> work even if the functions are actually macros. ?One place where it >>> would not work is in translated code running on the llinterp; the >>> other place might be the JIT trying to get the address of the >>> "function". ?(If that's the case, we should fix that a bit more >>> generally.) >>> >>> >>> A bient?t, >>> >>> Armin. >> >> I attempted to add os.major and os.minor using fchdir as an example, >> but... ?I have to admit: I really don't know what to make of this >> error. ?Comments? > > You probably failed to add an ll implementation in > pypy/rpython/lltypesystem/module/ll_os.py Do you mean pypy/rpython/module/ll_os.py ? I have no pypy/rpython/lltypesystem/module/ll_os.py . From benjamin at python.org Sat Dec 11 02:30:25 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 10 Dec 2010 19:30:25 -0600 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: 2010/12/10 Dan Stromberg : > On Fri, Dec 10, 2010 at 11:50 AM, Benjamin Peterson wrote: >> 2010/12/10 Dan Stromberg : >>> On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo wrote: >>>> Hi, >>>> >>>> Sorry for the series of mails. ?I just wanted to add that after >>>> looking more at the code, it's fine if you implement major() and >>>> minor() the same way as the other posix functions (see e.g. r79941 >>>> adding os.fchdir()), calling the major() and minor() "functions" in C. >>>> ?If you add tests in the same places I did in r79941, then it should >>>> work even if the functions are actually macros. ?One place where it >>>> would not work is in translated code running on the llinterp; the >>>> other place might be the JIT trying to get the address of the >>>> "function". ?(If that's the case, we should fix that a bit more >>>> generally.) >>>> >>>> >>>> A bient?t, >>>> >>>> Armin. >>> >>> I attempted to add os.major and os.minor using fchdir as an example, >>> but... ?I have to admit: I really don't know what to make of this >>> error. ?Comments? >> >> You probably failed to add an ll implementation in >> pypy/rpython/lltypesystem/module/ll_os.py > > Do you mean pypy/rpython/module/ll_os.py ? ?I have no > pypy/rpython/lltypesystem/module/ll_os.py . Yes, sorry. It's easy to get those rpython/ subdirs mixed up . :) -- Regards, Benjamin From drsalists at gmail.com Sat Dec 11 05:30:32 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 10 Dec 2010 20:30:32 -0800 Subject: [pypy-dev] Trunk failing to build? Message-ID: Does today's trunk fail to build? I'm getting the following on what I believe to be an unmodified trunk: [Timer] Timings: [Timer] annotate --- 1178.9 s [Timer] rtype_lltype --- 691.9 s [Timer] pyjitpl_lltype --- 1287.3 s [Timer] backendopt_lltype --- 621.4 s [Timer] stackcheckinsertion_lltype --- 101.2 s [Timer] database_c --- 11.8 s [Timer] =========================================== [Timer] Total: --- 3892.4 s [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 290, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/driver.py", line 812, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/driver.py", line 288, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/driver.py", line 508, in task_database_c [translation:ERROR] database = cbuilder.build_database() [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/c/genc.py", line 157, in build_database [translation:ERROR] sandbox=self.config.translation.sandbox) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/translator/c/database.py", line 65, in __init__ [translation:ERROR] self.gctransformer = self.gcpolicy.transformerclass(translator) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/rpython/memory/gctransform/framework.py", line 494, in __init__ [translation:ERROR] annhelper.finish() # at this point, annotate all mix-level helpers [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/rpython/annlowlevel.py", line 240, in finish [translation:ERROR] self.finish_annotate() [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/rpython/annlowlevel.py", line 259, in finish_annotate [translation:ERROR] ann.complete_helpers(self.policy) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py", line 176, in complete_helpers [translation:ERROR] self.complete() [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py", line 250, in complete [translation:ERROR] self.processblock(graph, block) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py", line 473, in processblock [translation:ERROR] self.flowin(graph, block) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py", line 533, in flowin [translation:ERROR] self.consider_op(block.operations[i]) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py", line 735, in consider_op [translation:ERROR] raise_nicer_exception(op, str(graph)) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py", line 732, in consider_op [translation:ERROR] resultcell = consider_meth(*argcells) [translation:ERROR] File "<3972-codegen /home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/annrpython.py:770>", line 3, in consider_op_simple_call [translation:ERROR] return arg.simple_call(*args) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/unaryop.py", line 174, in simple_call [translation:ERROR] return obj.call(getbookkeeper().build_args("simple_call", args_s)) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/unaryop.py", line 677, in call [translation:ERROR] return bookkeeper.pbc_call(pbc, args) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/bookkeeper.py", line 662, in pbc_call [translation:ERROR] results.append(desc.pycall(schedule, args, s_previous_result)) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/description.py", line 275, in pycall [translation:ERROR] inputcells = self.parse_arguments(args) [translation:ERROR] File "/home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/description.py", line 255, in parse_arguments [translation:ERROR] raise TypeError, "signature mismatch: %s" % e.getmsg(self.name) [translation:ERROR] TypeError': signature mismatch: get_L2cache_linux2() takes exactly 1 argument (0 given) [translation:ERROR] .. v1047 = simple_call((function get_L2cache_linux2)) [translation:ERROR] .. '(pypy.rpython.memory.gc.env:244)estimate_best_nursery_size' [translation:ERROR] Processing block: [translation:ERROR] block at 3 is a [translation:ERROR] in (pypy.rpython.memory.gc.env:244)estimate_best_nursery_size [translation:ERROR] containing the following operations: [translation:ERROR] v1047 = simple_call((function get_L2cache_linux2)) [translation:ERROR] v1048 = simple_call((function best_nursery_size_for_L2cache), v1047) [translation:ERROR] --end-- [translation] start debugger... > /home/dstromberg/src/pypy-src/pypy-for-trunk-builds/trunk/pypy/annotation/description.py(255)parse_arguments() -> raise TypeError, "signature mismatch: %s" % e.getmsg(self.name) (Pdb+) From benjamin at python.org Sat Dec 11 05:46:54 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 10 Dec 2010 22:46:54 -0600 Subject: [pypy-dev] Trunk failing to build? In-Reply-To: References: Message-ID: 2010/12/10 Dan Stromberg : > Does today's trunk fail to build? > > I'm getting the following on what I believe to be an unmodified trunk: Yep. Hopefully fixed in r79965. -- Regards, Benjamin From drsalists at gmail.com Sat Dec 11 05:29:14 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 10 Dec 2010 20:29:14 -0800 Subject: [pypy-dev] os.major? os.minor? In-Reply-To: References: Message-ID: On Fri, Dec 10, 2010 at 5:30 PM, Benjamin Peterson wrote: > 2010/12/10 Dan Stromberg : >> On Fri, Dec 10, 2010 at 11:50 AM, Benjamin Peterson wrote: >>> 2010/12/10 Dan Stromberg : >>>> On Thu, Dec 9, 2010 at 10:33 AM, Armin Rigo wrote: >>>>> Hi, >>>>> >>>>> Sorry for the series of mails. ?I just wanted to add that after >>>>> looking more at the code, it's fine if you implement major() and >>>>> minor() the same way as the other posix functions (see e.g. r79941 >>>>> adding os.fchdir()), calling the major() and minor() "functions" in C. >>>>> ?If you add tests in the same places I did in r79941, then it should >>>>> work even if the functions are actually macros. ?One place where it >>>>> would not work is in translated code running on the llinterp; the >>>>> other place might be the JIT trying to get the address of the >>>>> "function". ?(If that's the case, we should fix that a bit more >>>>> generally.) >>>>> >>>>> >>>>> A bient?t, >>>>> >>>>> Armin. >>>> >>>> I attempted to add os.major and os.minor using fchdir as an example, >>>> but... ?I have to admit: I really don't know what to make of this >>>> error. ?Comments? >>> >>> You probably failed to add an ll implementation in >>> pypy/rpython/lltypesystem/module/ll_os.py >> >> Do you mean pypy/rpython/module/ll_os.py ? ?I have no >> pypy/rpython/lltypesystem/module/ll_os.py . > > Yes, sorry. It's easy to get those rpython/ subdirs mixed up . :) No problem. From arigo at tunes.org Sat Dec 11 12:05:13 2010 From: arigo at tunes.org (Armin Rigo) Date: Sat, 11 Dec 2010 12:05:13 +0100 Subject: [pypy-dev] Trunk failing to build? In-Reply-To: References: Message-ID: Hi, On Sat, Dec 11, 2010 at 5:46 AM, Benjamin Peterson wrote: > Yep. Hopefully fixed in r79965. Sorry, my fault. Armin From drsalists at gmail.com Sat Dec 11 17:40:42 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 11 Dec 2010 08:40:42 -0800 Subject: [pypy-dev] gdbm and dbm merge - ctypes bug? In-Reply-To: References: Message-ID: On Thu, Dec 9, 2010 at 10:05 AM, Amaury Forgeot d'Arc wrote: > Hi > 2010/12/9 Dan Stromberg >> >> Is there a known problem with pypy's ctypes? > > Yes, several of them have already been fixed in trunk, especially > about memory chunks being freed too early. > -- > Amaury Forgeot d'Arc > Using a trunk from late in the day yesterday, my gdbm_ctypes module is still giving gibberish keys sometimes, but only with pypy: find /usr/share/jpilot -print0 | (export dir=$(pwd)/save-dir-pypy; cd ../..; ./this-pypy ./backshift --save-directory "$dir" --backup) Using gdbm_ctypes /home/dstromberg/src/home-svn/backshift/trunk/treap.py:40: DeprecationWarning: setrecursionlimit deprecated sys.setrecursionlimit(min_heap_size) Creating backup id 1292083099.51-benchbox-Sat_Dec_11_07_58_19_2010-7bada40ed14b282b Inhaling filenames, readline0.__file__: /home/dstromberg/lib/readline0.py processing directory /usr/share/jpilot processing regular file /usr/share/jpilot/CalendarDB-PDat.pdb 5.88% done, 3.61 files/second, ETA Sat Dec 11 07:58:24 2010, processing regular file /usr/share/jpilot/Ma?anaDB.pdb 11.76% done, 5.55 files/second, ETA Sat Dec 11 07:58:22 2010, processing regular file /usr/share/jpilot/jpilotrc.green 17.65% done, 6.17 files/second, ETA Sat Dec 11 07:58:22 2010, processing regular file /usr/share/jpilot/jpilotrc.purple 23.53% done, 6.63 files/second, ETA Sat Dec 11 07:58:22 2010, processing regular file /usr/share/jpilot/ContactsDB-PAdd.pdb 29.41% done, 7.37 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/MemoDB.pdb 35.29% done, 8.03 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/jpilotrc.blue 41.18% done, 8.59 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/ExpenseDB.pdb 47.06% done, 8.74 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/jpilotrc.default 52.94% done, 8.86 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/AddressDB.pdb 58.82% done, 8.89 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/ToDoDB.pdb 64.71% done, 9.23 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/TasksDB-PTod.pdb 70.59% done, 9.53 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/DatebookDB.pdb 76.47% done, 9.80 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/Memo32DB.pdb 82.35% done, 9.99 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/MemosDB-PMem.pdb 88.24% done, 10.21 files/second, ETA Sat Dec 11 07:58:21 2010, processing regular file /usr/share/jpilot/jpilotrc.steel 94.12% done, 10.36 files/second, ETA Sat Dec 11 07:58:21 2010, ! ../../dumpdb $(ls -td save-dir-pypy/files/1* | head -1)/dir-usr/dir-share/dir-jpilot/files | egrep --binary-files=text -- '->' | awk ' { print $2 }' | tr -d '[A-Za-z0-9\._\-]' | sed '/^$/d' | egrep --binary-files=text . ` ? ` ??@? make[2]: *** [go] Error 1 From drsalists at gmail.com Sat Dec 11 17:41:51 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 11 Dec 2010 08:41:51 -0800 Subject: [pypy-dev] setrecursionlimit deprecation? Message-ID: BTW, what's going on with the deprecation of sys.setrecursionlimit? What should I use instead? /home/dstromberg/src/home-svn/backshift/trunk/treap.py:40: DeprecationWarning: setrecursionlimit deprecated sys.setrecursionlimit(min_heap_size) From santagada at gmail.com Sat Dec 11 18:10:08 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 11 Dec 2010 15:10:08 -0200 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: References: Message-ID: IIRC nothing, pypy knows how big the stack is (if you are using it limit stack size and not get a core dump). On Sat, Dec 11, 2010 at 2:41 PM, Dan Stromberg wrote: > BTW, what's going on with the deprecation of sys.setrecursionlimit? > What should I use instead? > > /home/dstromberg/src/home-svn/backshift/trunk/treap.py:40: > DeprecationWarning: setrecursionlimit deprecated > ?sys.setrecursionlimit(min_heap_size) > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- Leonardo Santagada From benjamin at python.org Sat Dec 11 18:10:25 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 11 Dec 2010 11:10:25 -0600 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: References: Message-ID: 2010/12/11 Dan Stromberg : > BTW, what's going on with the deprecation of sys.setrecursionlimit? > What should I use instead? You shouldn't use anything. It has no affect on pypy. FWIW, I disagree with the deprecation. I think should just be a no-op. -- Regards, Benjamin From arigo at tunes.org Sat Dec 11 18:32:26 2010 From: arigo at tunes.org (Armin Rigo) Date: Sat, 11 Dec 2010 18:32:26 +0100 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: References: Message-ID: Hi Benjamin, On Sat, Dec 11, 2010 at 6:10 PM, Benjamin Peterson wrote: > FWIW, I disagree with the deprecation. I think should just be a no-op. But then people are going to complain that their app seems to hang up on pypy, and neither they nor we have any clue what is going on --- until we figure out that they used sys.setrecursionlimit() to put a bound on recursion and catch the RuntimeError. That's at least the reason for which I suggested that calling sys.setrecursionlimit() should at least print a warning. Now I agree that maybe the message of the warning is not the clearest one. A bient?t, Armin. From anto.cuni at gmail.com Sat Dec 11 19:16:32 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 11 Dec 2010 19:16:32 +0100 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: References: Message-ID: <4D03C000.6090206@gmail.com> On 11/12/10 18:32, Armin Rigo wrote: > But then people are going to complain that their app seems to hang up > on pypy, and neither they nor we have any clue what is going on --- > until we figure out that they used sys.setrecursionlimit() to put a > bound on recursion and catch the RuntimeError. That's at least the > reason for which I suggested that calling sys.setrecursionlimit() > should at least print a warning. Now I agree that maybe the message > of the warning is not the clearest one. I was about to propose to change the message into something like "sys.setrecursionlimit is ignored by PyPy". I think this is already an improvement over the current message, but has the drawback than then people will complain that pypy might run out of stack (which is false, but not apparent by the warning message). Also, we should maybe change the class of the Warning. IIRC, DeprecationWarning is going to be ignored by default on python2.7, which means that most people won't see the warning at all once we merge fast-forward. ciao, Anto From alex.gaynor at gmail.com Sat Dec 11 19:46:35 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 11 Dec 2010 13:46:35 -0500 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: <4D03C000.6090206@gmail.com> References: <4D03C000.6090206@gmail.com> Message-ID: On Sat, Dec 11, 2010 at 1:16 PM, Antonio Cuni wrote: > On 11/12/10 18:32, Armin Rigo wrote: > > > But then people are going to complain that their app seems to hang up > > on pypy, and neither they nor we have any clue what is going on --- > > until we figure out that they used sys.setrecursionlimit() to put a > > bound on recursion and catch the RuntimeError. That's at least the > > reason for which I suggested that calling sys.setrecursionlimit() > > should at least print a warning. Now I agree that maybe the message > > of the warning is not the clearest one. > > I was about to propose to change the message into something like > "sys.setrecursionlimit is ignored by PyPy". I think this is already an > improvement over the current message, but has the drawback than then people > will complain that pypy might run out of stack (which is false, but not > apparent by the warning message). > > Also, we should maybe change the class of the Warning. IIRC, > DeprecationWarning is going to be ignored by default on python2.7, which > means > that most people won't see the warning at all once we merge fast-forward. > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Indeed, perhaps RuntimeWarning is more approrpiate? -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101211/dc31592e/attachment.htm From drsalists at gmail.com Sat Dec 11 19:59:59 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 11 Dec 2010 10:59:59 -0800 Subject: [pypy-dev] Trunk failing to build? In-Reply-To: References: Message-ID: On Sat, Dec 11, 2010 at 3:05 AM, Armin Rigo wrote: > Hi, > > On Sat, Dec 11, 2010 at 5:46 AM, Benjamin Peterson wrote: >> Yep. Hopefully fixed in r79965. > > Sorry, my fault. > > > Armin > Looks fixed. Thanks. From drsalists at gmail.com Sat Dec 11 20:17:17 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 11 Dec 2010 11:17:17 -0800 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: References: Message-ID: I originally wrote the code ( http://stromberg.dnsalias.org/~strombrg/treap/ ) for CPython, where the recursion depth was too low out of the box. It sounds like on pypy, I might actually be decreasing the recursion depth. There probably should be a "set recursion depth to at least..." function. Or I should getrecursiondepth and inspect before "increasing". On Sat, Dec 11, 2010 at 9:10 AM, Leonardo Santagada wrote: > IIRC nothing, pypy knows how big the stack is (if you are using it > limit stack size and not get a core dump). > > On Sat, Dec 11, 2010 at 2:41 PM, Dan Stromberg wrote: >> BTW, what's going on with the deprecation of sys.setrecursionlimit? >> What should I use instead? >> >> /home/dstromberg/src/home-svn/backshift/trunk/treap.py:40: >> DeprecationWarning: setrecursionlimit deprecated >> ?sys.setrecursionlimit(min_heap_size) >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > > > > -- > Leonardo Santagada > From p.giarrusso at gmail.com Sat Dec 11 21:49:29 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Sat, 11 Dec 2010 21:49:29 +0100 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: <4D03C000.6090206@gmail.com> References: <4D03C000.6090206@gmail.com> Message-ID: On Sat, Dec 11, 2010 at 19:16, Antonio Cuni wrote: > On 11/12/10 18:32, Armin Rigo wrote: > >> But then people are going to complain that their app seems to hang up >> on pypy, and neither they nor we have any clue what is going on --- >> until we figure out that they used sys.setrecursionlimit() to put a >> bound on recursion and catch the RuntimeError. ?That's at least the >> reason for which I suggested that calling sys.setrecursionlimit() >> should at least print a warning. ?Now I agree that maybe the message >> of the warning is not the clearest one. > I was about to propose to change the message into something like > "sys.setrecursionlimit is ignored by PyPy". ?I think this is already an > improvement over the current message, but has the drawback than then people > will complain that pypy might run out of stack (which is false, but not > apparent by the warning message). Do you mean that the stack size on Python is not limited (which has its problems for infinite-recursion bugs), or just that stack overflow is handled gracefully by throwing an exception? In the first case, I'd propose "stack size on PyPy cannot be limited" (and maybe additionally "sys.setrecursionlimit is ignored by PyPy"). In the second case, I'd propose "recursion depth is only limited by stack size". Maybe the call should try adjusting the stack size? However, reading os_thread.py:stack_size() [1] suggests that the stack size _is_ limited for a thread, so I guess you're in the second case. [1] codespeak.net/pypy/dist/pypy/module/thread/os_thread.py Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From benjamin at python.org Sat Dec 11 21:52:11 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 11 Dec 2010 14:52:11 -0600 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: References: <4D03C000.6090206@gmail.com> Message-ID: 2010/12/11 Paolo Giarrusso : > On Sat, Dec 11, 2010 at 19:16, Antonio Cuni wrote: >> On 11/12/10 18:32, Armin Rigo wrote: >> >>> But then people are going to complain that their app seems to hang up >>> on pypy, and neither they nor we have any clue what is going on --- >>> until we figure out that they used sys.setrecursionlimit() to put a >>> bound on recursion and catch the RuntimeError. ?That's at least the >>> reason for which I suggested that calling sys.setrecursionlimit() >>> should at least print a warning. ?Now I agree that maybe the message >>> of the warning is not the clearest one. > >> I was about to propose to change the message into something like >> "sys.setrecursionlimit is ignored by PyPy". ?I think this is already an >> improvement over the current message, but has the drawback than then people >> will complain that pypy might run out of stack (which is false, but not >> apparent by the warning message). > > Do you mean that the stack size on Python is not limited (which has > its problems for infinite-recursion bugs), or just that stack overflow > is handled gracefully by throwing an exception? The latter. -- Regards, Benjamin From arigo at tunes.org Sun Dec 12 15:14:05 2010 From: arigo at tunes.org (Armin Rigo) Date: Sun, 12 Dec 2010 15:14:05 +0100 Subject: [pypy-dev] setrecursionlimit deprecation? In-Reply-To: References: Message-ID: Hi Dan, On Sat, Dec 11, 2010 at 8:17 PM, Dan Stromberg wrote: > I originally wrote the code ( > http://stromberg.dnsalias.org/~strombrg/treap/ ) for CPython, where > the recursion depth was too low out of the box. ?It sounds like on > pypy, I might actually be decreasing the recursion depth. > > There probably should be a "set recursion depth to at least..." > function. ?Or I should getrecursiondepth and inspect before > "increasing". PyPy throws a RuntimeError anyway when the stack is getting too big, i.e. when it uses more than N kilobytes. On the other hand, setrecursiondepth() on CPython (and previously on PyPy) is based on counting the recursion depth, which is a different measure which only approximates the real stack size. We decided that it doesn't make sense to provide both on PyPy, so we killed the one that is an approximation (and makes the JIT a little bit slower). That's all. A bient?t, Armin. From tobami at googlemail.com Mon Dec 13 09:20:14 2010 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 13 Dec 2010 09:20:14 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: Thanks all for the input. I've compiled a list based on your mails, the Unladen benchmarks page (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the alioth descriptions. Here is an extract of the current speed.pypy.org admin: ai chaos Creates chaosgame-like fractals crypto_pyaes A pure python implementation of AES django Uses the Django template system to build a 150x150-cell HTML table fannkuch Indexed-access to tiny integer-sequence. The fannkuch benchmark is defined by programs in Performing Lisp Analysis of the FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig. float Creates an array of points using circular projection and then normalizes and maximizes them. Floating-point heavy. go A go (chess like game) computer player AI. html5lib Parses the HTML 5 spec using html5lib meteor-contest Searchs for solutions to shape packing puzzle. nbody_modified Double-precision N-body simulation. It models the orbits of Jovian planets, using a simple symplectic-integrator. pyflate-fast Stand-alone pure-Python DEFLATE (gzip) and bzip2 decoder/decompressor. raytrace-simple A raytracer renderer richards Medium-sized language benchmark that simulates the task dispatcher in the kernel of an operating system. rietveld A Django application benchmark. slowspitfire spambayes Runs a canned mailbox through a SpamBayes ham/spam classifier spectral-norm spitfire Uses the Spitfire template system to build a 1000x1000-cell HTML table. spitfire_cstringio Uses the Spitfire template system to build a 1000x1000-cell HTML table, using the cstringio module. telco twisted_iteration twisted_names twisted_pb twisted_tcp Connects one Twised client to one Twisted server over TCP (on the loopback interface) and then writes bytes as fast as it can. waf Python-based framework for configuring, compiling and installing applications. It derives from the concepts of other build tools such as Scons, Autotools, CMake or Ant. So the remaining descriptions are ai slowspitfire (what is the exact difference between the three spitfire benches?) spectral-norm telco twisted (most of them) Are the descriptions all right so far?. They can be made much longer if you deem it desirable. on speed.pypy.org you will currently see the descriptions in 3 places: - Changes view: A tooltip on hover over each benchmark - Timeline: a description box beneath each plot - Comparison: A tooltip over each benchmark when hovering the selection menu on the left side. Any suggestions on how to improve it further are welcome ;-) Miquel 2010/12/9 Paolo Giarrusso : > On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada wrote: >> Here is a incomplete draft list: > >> [slow]spitfire[cstringio]: Spitfire is a template language, the >> cstringio version uses a modified engine (that uses cstringio) > >> spambayes: Spambayes is a bayesian spam filter > > Why is [slow]spitfire slower with PyPy? Is it regex-related? I > remember when, because of this, spambayes was slower (including > release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4 > and the head (for slowspitfire it's the opposite). > > For the rest, I see no significant case of slowdown of PyPy over time. > http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > From tobami at googlemail.com Mon Dec 13 09:31:04 2010 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 13 Dec 2010 09:31:04 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: Oh, btw., the "normalized" stacked bars now display a warning note about its correctness, and how it must be viewed as giving results a weighting instead of them being normalized. It even includes a link to the proper paper. I hope that is enough for the strict statisticians among us ;-) See: http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars PS: there is a bug in the jqPlot plotting library when null values are present. Trying to display PyPy 1.3 results for the newer go, pyflake or raytrace will create some nasty js loops. It also has problems with autoscaling the axis sometimes. 2010/12/13 Miquel Torres : > Thanks all for the input. > I've compiled a list based on your mails, the Unladen benchmarks page > (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the > alioth descriptions. Here is an extract of the current speed.pypy.org > admin: > > ai > chaos ? Creates chaosgame-like fractals > crypto_pyaes ? ?A pure python implementation of AES > django ? ? ? ? ?Uses the Django template system to build a 150x150-cell HTML table > > fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The fannkuch > benchmark is defined by programs in Performing Lisp Analysis of the > FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig. > > float ? ? ? ? ? Creates an array of points using circular projection and then > normalizes and maximizes them. Floating-point heavy. > go ? ? ? ? ? ? ?A go (chess like game) computer player AI. > html5lib ? ? ? ?Parses the HTML 5 spec using html5lib > meteor-contest ?Searchs for solutions to shape packing puzzle. > nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models the > orbits of Jovian planets, using a simple symplectic-integrator. > pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and bzip2 > decoder/decompressor. > raytrace-simple A raytracer renderer > richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates the task > dispatcher in the kernel of an operating system. > rietveld ? ? ? ?A Django application benchmark. > slowspitfire > spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam classifier > spectral-norm > spitfire ? ? ? ?Uses the Spitfire template system to build a 1000x1000-cell HTML table. > spitfire_cstringio ? ? ?Uses the Spitfire template system to build a > 1000x1000-cell HTML table, using the cstringio module. > telco > twisted_iteration > twisted_names > twisted_pb > twisted_tcp ? ? Connects one Twised client to one Twisted server over TCP > (on the loopback interface) and then writes bytes as fast as it can. > waf ? ? Python-based framework for configuring, compiling and installing > applications. It derives from the concepts of other build tools such > as Scons, Autotools, CMake or Ant. > > > So the remaining descriptions are > ai > slowspitfire (what is the exact difference between the three spitfire benches?) > spectral-norm > telco > twisted (most of them) > > Are the descriptions all right so far?. They can be made much longer > if you deem it desirable. > > on speed.pypy.org you will currently see the descriptions in 3 places: > - Changes view: A tooltip on hover over each benchmark > - Timeline: a description box beneath each plot > - Comparison: A tooltip over each benchmark when hovering the > selection menu on the left side. > > Any suggestions on how to improve it further are welcome ;-) > > Miquel > > > 2010/12/9 Paolo Giarrusso : >> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada wrote: >>> Here is a incomplete draft list: >> >>> [slow]spitfire[cstringio]: Spitfire is a template language, the >>> cstringio version uses a modified engine (that uses cstringio) >> >>> spambayes: Spambayes is a bayesian spam filter >> >> Why is [slow]spitfire slower with PyPy? Is it regex-related? I >> remember when, because of this, spambayes was slower (including >> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4 >> and the head (for slowspitfire it's the opposite). >> >> For the rest, I see no significant case of slowdown of PyPy over time. >> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars >> -- >> Paolo Giarrusso - Ph.D. Student >> http://www.informatik.uni-marburg.de/~pgiarrusso/ >> > From cfbolz at gmx.de Mon Dec 13 10:23:07 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 13 Dec 2010 10:23:07 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: <4D05E5FB.80103@gmx.de> Hi Miquel, On 12/13/2010 09:20 AM, Miquel Torres wrote: > So the remaining descriptions are > ai > slowspitfire (what is the exact difference between the three spitfire benches?) The difference is the way the final string is built, using cstringio, or other means. I guess Maciek knows the details. > spectral-norm > telco Telco covers "the essence of a telephone company billing application" using decimal arithmetic: http://speleotrove.com/decimal/telco.html Thanks for doing this, Carl Friedrich From exarkun at twistedmatrix.com Mon Dec 13 15:51:39 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 13 Dec 2010 14:51:39 -0000 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> On 08:20 am, tobami at googlemail.com wrote: >Thanks all for the input. >I've compiled a list based on your mails, the Unladen benchmarks page >(http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >alioth descriptions. Here is an extract of the current speed.pypy.org >admin: > >[snip] >twisted_iteration Iterates a Twisted reactor as quickly as possible without doing any work. >twisted_names Runs a DNS server with Twisted Names and then issues requests to it over loopback UDP. >twisted_pb Runs a Perspective Broker server with a no-op method and invokes that method over loopback TCP with some strings, dictionaries, and tuples as arguments. From tobami at googlemail.com Mon Dec 13 16:03:24 2010 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 13 Dec 2010 16:03:24 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> Message-ID: @Carl Friedrich & exarkun: thanks, I've added those. only spectral-norm, slowspitfire and ai to go. slowspitfire is described at the Unladen page as using psyco, but it doesn't make sense in our case? 2010/12/13 : > On 08:20 am, tobami at googlemail.com wrote: >> >> Thanks all for the input. >> I've compiled a list based on your mails, the Unladen benchmarks page >> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >> alioth descriptions. Here is an extract of the current speed.pypy.org >> admin: >> >> [snip] >> twisted_iteration > > Iterates a Twisted reactor as quickly as possible without doing any work. >> >> twisted_names > > Runs a DNS server with Twisted Names and then issues requests to it over > loopback UDP. >> >> twisted_pb > > Runs a Perspective Broker server with a no-op method and invokes that method > over loopback TCP with some strings, dictionaries, and tuples as arguments. > From tobami at googlemail.com Mon Dec 13 16:08:42 2010 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 13 Dec 2010 16:08:42 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> Message-ID: sorry, I meant the opposite. To recap, according to http://code.google.com/p/unladen-swallow/wiki/Benchmarks, spitfire: psyco slowspitfire: pure python in addition we have spitfire_cstringio, which uses a c module (so it is even faster). what is vanilla spitfire in our case? 2010/12/13 Miquel Torres : > @Carl Friedrich & exarkun: thanks, I've added those. > > only spectral-norm, slowspitfire and ai to go. > > slowspitfire is described at the Unladen page as using psyco, but it > doesn't make sense in our case? > > > > 2010/12/13 ?: >> On 08:20 am, tobami at googlemail.com wrote: >>> >>> Thanks all for the input. >>> I've compiled a list based on your mails, the Unladen benchmarks page >>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >>> alioth descriptions. Here is an extract of the current speed.pypy.org >>> admin: >>> >>> [snip] >>> twisted_iteration >> >> Iterates a Twisted reactor as quickly as possible without doing any work. >>> >>> twisted_names >> >> Runs a DNS server with Twisted Names and then issues requests to it over >> loopback UDP. >>> >>> twisted_pb >> >> Runs a Perspective Broker server with a no-op method and invokes that method >> over loopback TCP with some strings, dictionaries, and tuples as arguments. >> > From fijall at gmail.com Mon Dec 13 17:43:27 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 13 Dec 2010 18:43:27 +0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> Message-ID: Hi. spitfires are confusing. slowspitfire and spitfire use ''.join(list-of-strings) where spitfire_cstringio uses cStringIO instead. spitfire and spitfire_cstringio use smaller table to render (100x100 I think) which was the default on original benchmarks slowspitfire uses 1000x1000 (which is why it used to be slower than spitfire) and was chosen by US guys to let the JIT warm up. We should remove _slow these days. On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres wrote: > sorry, I meant the opposite. To recap, according to > http://code.google.com/p/unladen-swallow/wiki/Benchmarks, > spitfire: psyco > slowspitfire: pure python > > in addition we have spitfire_cstringio, which uses a c module (so it > is even faster). > > what is vanilla spitfire in our case? > > > 2010/12/13 Miquel Torres : >> @Carl Friedrich & exarkun: thanks, I've added those. >> >> only spectral-norm, slowspitfire and ai to go. >> >> slowspitfire is described at the Unladen page as using psyco, but it >> doesn't make sense in our case? >> >> >> >> 2010/12/13 ?: >>> On 08:20 am, tobami at googlemail.com wrote: >>>> >>>> Thanks all for the input. >>>> I've compiled a list based on your mails, the Unladen benchmarks page >>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >>>> alioth descriptions. Here is an extract of the current speed.pypy.org >>>> admin: >>>> >>>> [snip] >>>> twisted_iteration >>> >>> Iterates a Twisted reactor as quickly as possible without doing any work. >>>> >>>> twisted_names >>> >>> Runs a DNS server with Twisted Names and then issues requests to it over >>> loopback UDP. >>>> >>>> twisted_pb >>> >>> Runs a Perspective Broker server with a no-op method and invokes that method >>> over loopback TCP with some strings, dictionaries, and tuples as arguments. >>> >> > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From santagada at gmail.com Mon Dec 13 19:10:45 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 13 Dec 2010 16:10:45 -0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> Message-ID: why not have only 2 versions, both with the same size table and name one spitfire_cstringio and the other spitfire_strjoin? I think it would make things clearer. On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski wrote: > Hi. > > spitfires are confusing. > > slowspitfire and spitfire use ''.join(list-of-strings) where > spitfire_cstringio uses cStringIO instead. > > spitfire and spitfire_cstringio use smaller table to render (100x100 I > think) which was the default on original benchmarks > > slowspitfire uses 1000x1000 (which is why it used to be slower than > spitfire) and was chosen by US guys to let the JIT warm up. We should > remove _slow these days. > > On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres wrote: >> sorry, I meant the opposite. To recap, according to >> http://code.google.com/p/unladen-swallow/wiki/Benchmarks, >> spitfire: psyco >> slowspitfire: pure python >> >> in addition we have spitfire_cstringio, which uses a c module (so it >> is even faster). >> >> what is vanilla spitfire in our case? >> >> >> 2010/12/13 Miquel Torres : >>> @Carl Friedrich & exarkun: thanks, I've added those. >>> >>> only spectral-norm, slowspitfire and ai to go. >>> >>> slowspitfire is described at the Unladen page as using psyco, but it >>> doesn't make sense in our case? >>> >>> >>> >>> 2010/12/13 ?: >>>> On 08:20 am, tobami at googlemail.com wrote: >>>>> >>>>> Thanks all for the input. >>>>> I've compiled a list based on your mails, the Unladen benchmarks page >>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >>>>> alioth descriptions. Here is an extract of the current speed.pypy.org >>>>> admin: >>>>> >>>>> [snip] >>>>> twisted_iteration >>>> >>>> Iterates a Twisted reactor as quickly as possible without doing any work. >>>>> >>>>> twisted_names >>>> >>>> Runs a DNS server with Twisted Names and then issues requests to it over >>>> loopback UDP. >>>>> >>>>> twisted_pb >>>> >>>> Runs a Perspective Broker server with a no-op method and invokes that method >>>> over loopback TCP with some strings, dictionaries, and tuples as arguments. >>>> >>> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- Leonardo Santagada From sasmekoll at gmail.com Mon Dec 13 03:37:19 2010 From: sasmekoll at gmail.com (Vovan) Date: Mon, 13 Dec 2010 05:37:19 +0300 Subject: [pypy-dev] =?utf-8?b?0J3QvtCy0L7QtSDRgdC+0L7QsdGJ0LXQvdC40LU=?= Message-ID: <4d06522c.4b0fcc0a.5274.1a17@mx.google.com> http://samec.org.ua/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101213/fef1f181/attachment-0001.htm From tobami at googlemail.com Mon Dec 13 21:51:59 2010 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 13 Dec 2010 21:51:59 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> Message-ID: @Maciej: it doesn't make a lot of sense. Looking at this graph: http://speed.pypy.org/comparison/?exe=2%2B35,4%2B35,1%2B172,3%2B172&ben=11,14,15&env=1&hor=false&bas=none&chart=normal+bars slowspitfire is much faster than the other two. Is that because it performs more iterations? Also, how come pypy-c-jit is faster than cpython or psyco precisely in cstringio, where performance should be dependent on cstringIO and thus be more similar across interpreters? 2010/12/13 Leonardo Santagada : > why not have only 2 versions, both with the same size table and name > one spitfire_cstringio and the other spitfire_strjoin? I think it > would make things clearer. > > > On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski wrote: >> Hi. >> >> spitfires are confusing. >> >> slowspitfire and spitfire use ''.join(list-of-strings) where >> spitfire_cstringio uses cStringIO instead. >> >> spitfire and spitfire_cstringio use smaller table to render (100x100 I >> think) which was the default on original benchmarks >> >> slowspitfire uses 1000x1000 (which is why it used to be slower than >> spitfire) and was chosen by US guys to let the JIT warm up. We should >> remove _slow these days. >> >> On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres wrote: >>> sorry, I meant the opposite. To recap, according to >>> http://code.google.com/p/unladen-swallow/wiki/Benchmarks, >>> spitfire: psyco >>> slowspitfire: pure python >>> >>> in addition we have spitfire_cstringio, which uses a c module (so it >>> is even faster). >>> >>> what is vanilla spitfire in our case? >>> >>> >>> 2010/12/13 Miquel Torres : >>>> @Carl Friedrich & exarkun: thanks, I've added those. >>>> >>>> only spectral-norm, slowspitfire and ai to go. >>>> >>>> slowspitfire is described at the Unladen page as using psyco, but it >>>> doesn't make sense in our case? >>>> >>>> >>>> >>>> 2010/12/13 ?: >>>>> On 08:20 am, tobami at googlemail.com wrote: >>>>>> >>>>>> Thanks all for the input. >>>>>> I've compiled a list based on your mails, the Unladen benchmarks page >>>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >>>>>> alioth descriptions. Here is an extract of the current speed.pypy.org >>>>>> admin: >>>>>> >>>>>> [snip] >>>>>> twisted_iteration >>>>> >>>>> Iterates a Twisted reactor as quickly as possible without doing any work. >>>>>> >>>>>> twisted_names >>>>> >>>>> Runs a DNS server with Twisted Names and then issues requests to it over >>>>> loopback UDP. >>>>>> >>>>>> twisted_pb >>>>> >>>>> Runs a Perspective Broker server with a no-op method and invokes that method >>>>> over loopback TCP with some strings, dictionaries, and tuples as arguments. >>>>> >>>> >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > > > -- > Leonardo Santagada > From anto.cuni at gmail.com Tue Dec 14 00:02:57 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 14 Dec 2010 00:02:57 +0100 Subject: [pypy-dev] Migration to mercurial Message-ID: <4D06A621.4060300@gmail.com> Hi all, finally, it's happening :-). Thanks to Ronny's work, we are going to complete the migration to mercurial very soon. The pypy buildbot is already configured to pull from the bitbucket repository: http://bitbucket.org/pypy/pypy I have tried to run a test build, and it seems to work ok. We will see tomorrow morning how the nightly runs went. If everything goes right, tomorrow we will declare the hg repo as the "official" one: the svn repo will be made read-only (as soon as armin does it :-)), the mercurial one will be synchronized to include the latest svn commits, and from that point everyone should start to commit to mercurial. If you don't have write access to the bitbucket repo, please ask me or anyone else who is an admin. A note for people working on active branches: before you can work on those, you need to perform two simple manual steps, as detailed in this howto: http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/active-branches-howto.txt ciao, Anto From benjamin at python.org Tue Dec 14 01:57:26 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 13 Dec 2010 18:57:26 -0600 Subject: [pypy-dev] Migration to mercurial In-Reply-To: <4D06A621.4060300@gmail.com> References: <4D06A621.4060300@gmail.com> Message-ID: 2010/12/13 Antonio Cuni : > Hi all, > > finally, it's happening :-). > Thanks to Ronny's work, we are going to complete the migration to mercurial > very soon. Excellent! Thanks for your work. > > The pypy buildbot is already configured to pull from the bitbucket repository: > http://bitbucket.org/pypy/pypy > > I have tried to run a test build, and it seems to work ok. We will see > tomorrow morning how the nightly runs went. > > If everything goes right, tomorrow we will declare the hg repo as the > "official" one: the svn repo will be made read-only (as soon as armin does it > :-)), the mercurial one will be synchronized to include the latest svn > commits, and from that point everyone should start to commit to mercurial. ?If > you don't have write access to the bitbucket repo, please ask me or anyone > else who is an admin. > > A note for people working on active branches: before you can work on those, > you need to perform two simple manual steps, as detailed in this howto: > http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/active-branches-howto.txt What will be happening to the other repos? ie. extradoc -- Regards, Benjamin From dmalcolm at redhat.com Tue Dec 14 01:57:04 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 13 Dec 2010 19:57:04 -0500 Subject: [pypy-dev] [PATCH] Improving readability of generated .c code Message-ID: <1292288224.24137.22.camel@radiator.bos.redhat.com> The attached patch is an attempt at extending the C code generator so that it annotates the generated .c code with file/line information for the corresponding Python sources. Method: I followed the instructions in pypy/doc/getting-started-dev.txt: cd pypy python bin/translatorshell.py >>> t = Translation(snippet.is_perfect_number) >>> t.annotate([int]) >>> f = t.compile_c() then examine the generated .c code. With this patch, the top of a generated .c function contains an inline copy of the python source, like this: bool_t pypy_g_ll_issubclass__object_vtablePtr_object_vtablePtr(struct pypy_object_vtable0 *l_subcls_0, struct pypy_object_vtable0 *l_cls_0) { bool_t l_v11; long l_v12; long l_v13; long l_v14; /* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rclass.py' * 665 : def ll_issubclass(subcls, cls): * 666 : return llop.int_between(Bool, cls.subclassrange_min, * 667 : subcls.subclassrange_min, * 668 : cls.subclassrange_max) */ goto block0; ...and for every operation that has an offset, it calculates the linenumber (using the "dis" module on the code object). For every operation that changes the line number, it emits a comment containing that Python source, so that you get code like this: block0: /* 666 : return llop.int_between(Bool, cls.subclassrange_min, */ l_v12 = RPyField(l_cls_0, ov_subclassrange_min); /* 667 : subcls.subclassrange_min, */ l_v13 = RPyField(l_subcls_0, ov_subclassrange_min); /* 668 : cls.subclassrange_max) */ l_v14 = RPyField(l_cls_0, ov_subclassrange_max); OP_INT_BETWEEN(l_v12, l_v13, l_v14, l_v11); goto block1; Hopefully this makes the generated .c much more readable. (I hope to work on the names of locals also; ideally they ought to embed the python indentifier) This is on a Fedora 13 x86_64 box, with cpython 2.6.5 Caveats: - this is my first patch to PyPy (please be gentle!): I hope I've correctly understood the various levels, but I suspect I may still be missing things at the conceptual level - haven't completed the full unit tests yet. - the patch attempts to propagate the "offset" of flowspace SpaceOperation instances in a couple of places where the offset was being discarded: - in BaseInliner.copy_operation - when generating operations in rtyper: set each of the new operations' "offset" to that of the high-level operation - I added the offset to SpaceOperation.__repr__() to help track the above down. - I also pass the reference to the func object from the flowspace FunctionGraph to the rtyper's copy (I wonder if this affect memory usage during translation?) - most functions appear correct, but there are some where I'm not at all convinced by the output (see below) - the patch is assuming that within any given generated C function, there is a single python function, and that the offsets are indexes into the bytecode for that function's code object. I suspect that this is an oversimplification, and why I'm seeing the errors that I'm seeing. - I haven't stripped some of my debugging python comments from funcgen.py in the patch (this is a work in progress). - I haven't tried doing this on a full build of the interpreter (having highly readable generated .c for this is my objective [1]). One other idea I had was to sort the blocks in the function by the bytecode offset; currently the c blocks seem to "jump around" a fair bit relative to the corresponding rpython code. Has any tuning been done on the ordering of the blocks within the generated .c code? (or is it assumed that the .c compiler will "flatten" these, for the case when a node has a single successor node?) (I wonder if similar work could be done on the JVM and CLI code generators?) As mentioned above, I think my patch is getting it wrong in a few places. Here's an example: struct pypy_rpy_string0 *pypy_g_mallocstr__Signed(long l_length_3) { ...snip... /* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rstr.py' * 35 : def mallocstr(length): * 36 : ll_assert(length >= 0, "negative string length") * 37 : r = malloc(TP, length) * 38 : if not we_are_translated() or not malloc_zero_filled: * 39 : r.hash = 0 * 40 : return r */ ...snip... block15: /* 37 : r = malloc(TP, length) */ OP_ADR_SUB(l_v202, 0, l_v234); l_v235 = (struct pypy_header0 *)l_v234; l_v236 = RPyField(l_v235, h_refcount); /* 38 : if not we_are_translated() or not malloc_zero_filled: */ OP_INT_ADD(l_v236, 1L, l_v237); RPyField(l_v235, h_refcount) = l_v237; goto block5; The C code doesn't look anything like the .py code that my patch is inserting in the above. (My current guess is that I need to be smarter about inlined operations, though that's a guess at this stage) Hope this is helpful Dave [1] http://fedoraproject.org/wiki/Features/PyPyStack is what I'm aiming at -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy-add-py-source-as-comments-to-generated-c.patch Type: text/x-patch Size: 6498 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101213/77ef01d8/attachment.bin From benjamin at python.org Tue Dec 14 02:14:04 2010 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 13 Dec 2010 19:14:04 -0600 Subject: [pypy-dev] [PATCH] Improving readability of generated .c code In-Reply-To: <1292288224.24137.22.camel@radiator.bos.redhat.com> References: <1292288224.24137.22.camel@radiator.bos.redhat.com> Message-ID: 2010/12/13 David Malcolm : > One other idea I had was to sort the blocks in the function by the > bytecode offset; currently the c blocks seem to "jump around" a fair bit > relative to the corresponding rpython code. ?Has any tuning been done on > the ordering of the blocks within the generated .c code? ?(or is it > assumed that the .c compiler will "flatten" these, for the case when a > node has a single successor node?) GCC puts everything in SSA form, so it should eliminate all pointless jumps. -- Regards, Benjamin From william.leslie.ttg at gmail.com Tue Dec 14 04:22:45 2010 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 14 Dec 2010 14:22:45 +1100 Subject: [pypy-dev] [PATCH] Improving readability of generated .c code In-Reply-To: <1292288224.24137.22.camel@radiator.bos.redhat.com> References: <1292288224.24137.22.camel@radiator.bos.redhat.com> Message-ID: On 14 December 2010 11:57, David Malcolm wrote: > One other idea I had was to sort the blocks in the function by the > bytecode offset; currently the c blocks seem to "jump around" a fair bit > relative to the corresponding rpython code. ?Has any tuning been done on > the ordering of the blocks within the generated .c code? ?(or is it > assumed that the .c compiler will "flatten" these, for the case when a > node has a single successor node?) > > (I wonder if similar work could be done on the JVM and CLI code > generators?) An unfortunate consequence of abstract interpretation is that you loose a fair bit of the structure of the code as it appears in the source. There are situations, for example, where you can't know which blocks (or which *part* of which blocks, even) are part of an inner and which are part of an outer loop. The existing code (at least when writing jitcode, but I imagine it's the same for the backend) writes the startblock, and then immediately writes its first dependency - essentially a depth-first traversal, a method as good as any in the absence of structure, but I would be interested to see what difference it makes for what cost in extra translation time. -- William Leslie From fijall at gmail.com Tue Dec 14 07:32:26 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Dec 2010 08:32:26 +0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> Message-ID: On Mon, Dec 13, 2010 at 10:51 PM, Miquel Torres wrote: > @Maciej: it doesn't make a lot of sense. Looking at this graph: > http://speed.pypy.org/comparison/?exe=2%2B35,4%2B35,1%2B172,3%2B172&ben=11,14,15&env=1&hor=false&bas=none&chart=normal+bars > > slowspitfire is much faster than the other two. Is that because it > performs more iterations? I think it's apples to oranges (they have different table sizes and different number of iterations) > > Also, how come pypy-c-jit is faster than cpython or psyco precisely in > cstringio, where performance should be dependent on cstringIO and thus > be more similar across interpreters? because having a list of small strings means you have a large (old) object referencing a lot of young objects, hence GC cost. It's not the case with cstringio where you have a single chunk of memory which does not contain GC pointers. > > > 2010/12/13 Leonardo Santagada : >> why not have only 2 versions, both with the same size table and name >> one spitfire_cstringio and the other spitfire_strjoin? I think it >> would make things clearer. >> >> >> On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski wrote: >>> Hi. >>> >>> spitfires are confusing. >>> >>> slowspitfire and spitfire use ''.join(list-of-strings) where >>> spitfire_cstringio uses cStringIO instead. >>> >>> spitfire and spitfire_cstringio use smaller table to render (100x100 I >>> think) which was the default on original benchmarks >>> >>> slowspitfire uses 1000x1000 (which is why it used to be slower than >>> spitfire) and was chosen by US guys to let the JIT warm up. We should >>> remove _slow these days. >>> >>> On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres wrote: >>>> sorry, I meant the opposite. To recap, according to >>>> http://code.google.com/p/unladen-swallow/wiki/Benchmarks, >>>> spitfire: psyco >>>> slowspitfire: pure python >>>> >>>> in addition we have spitfire_cstringio, which uses a c module (so it >>>> is even faster). >>>> >>>> what is vanilla spitfire in our case? >>>> >>>> >>>> 2010/12/13 Miquel Torres : >>>>> @Carl Friedrich & exarkun: thanks, I've added those. >>>>> >>>>> only spectral-norm, slowspitfire and ai to go. >>>>> >>>>> slowspitfire is described at the Unladen page as using psyco, but it >>>>> doesn't make sense in our case? >>>>> >>>>> >>>>> >>>>> 2010/12/13 ?: >>>>>> On 08:20 am, tobami at googlemail.com wrote: >>>>>>> >>>>>>> Thanks all for the input. >>>>>>> I've compiled a list based on your mails, the Unladen benchmarks page >>>>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >>>>>>> alioth descriptions. Here is an extract of the current speed.pypy.org >>>>>>> admin: >>>>>>> >>>>>>> [snip] >>>>>>> twisted_iteration >>>>>> >>>>>> Iterates a Twisted reactor as quickly as possible without doing any work. >>>>>>> >>>>>>> twisted_names >>>>>> >>>>>> Runs a DNS server with Twisted Names and then issues requests to it over >>>>>> loopback UDP. >>>>>>> >>>>>>> twisted_pb >>>>>> >>>>>> Runs a Perspective Broker server with a no-op method and invokes that method >>>>>> over loopback TCP with some strings, dictionaries, and tuples as arguments. >>>>>> >>>>> >>>> _______________________________________________ >>>> pypy-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/pypy-dev >>>> >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >> >> >> >> -- >> Leonardo Santagada >> > From anto.cuni at gmail.com Tue Dec 14 08:23:41 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 14 Dec 2010 08:23:41 +0100 Subject: [pypy-dev] Migration to mercurial In-Reply-To: References: <4D06A621.4060300@gmail.com> Message-ID: <4D071B7D.1030708@gmail.com> On 14/12/10 01:57, Benjamin Peterson wrote: > What will be happening to the other repos? ie. extradoc it's open to discussion, but I suppose that it's fine to migrate them as well. However, there is no hurry on this, we can do it one by one. ciao, Anto From sungsuha at gmail.com Tue Dec 14 08:24:23 2010 From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha) Date: Tue, 14 Dec 2010 07:24:23 +0000 (UTC) Subject: [pypy-dev] Migration to mercurial References: <4D06A621.4060300@gmail.com> Message-ID: Antonio Cuni gmail.com> writes: > > Hi all, > > finally, it's happening . > Thanks to Ronny's work, we are going to complete the migration to mercurial > very soon. +1 Cool. Providing binaries through "bitbucket.org/pypy/pypy/downloads" would be nice too. Are the bugtracking/code review facilities of bitbucket going to be used too? Or is the current https://codespeak.net/issue/pypy-dev/ here to stay? From fijall at gmail.com Tue Dec 14 08:29:44 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Dec 2010 09:29:44 +0200 Subject: [pypy-dev] Migration to mercurial In-Reply-To: References: <4D06A621.4060300@gmail.com> Message-ID: On Tue, Dec 14, 2010 at 9:24 AM, Seung Soo, wrote: > Antonio Cuni gmail.com> writes: > >> >> Hi all, >> >> finally, it's happening . >> Thanks to Ronny's work, we are going to complete the migration to mercurial >> very soon. > > +1 > Cool. > Providing binaries through "bitbucket.org/pypy/pypy/downloads" would be nice too. > > Are the bugtracking/code review facilities of bitbucket going to be used too? > Or is the current https://codespeak.net/issue/pypy-dev/ here to stay? > All of this will happen, it's just a one-by-one process (since they don't depend on each other). Don't worry, we'll get there :) From fijall at gmail.com Tue Dec 14 08:57:47 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Dec 2010 09:57:47 +0200 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> <20101213145139.2058.844900884.divmod.xquotient.102@localhost.localdomain> Message-ID: Hey miquel, didn't we loose colors somehow? On Tue, Dec 14, 2010 at 8:32 AM, Maciej Fijalkowski wrote: > On Mon, Dec 13, 2010 at 10:51 PM, Miquel Torres wrote: >> @Maciej: it doesn't make a lot of sense. Looking at this graph: >> http://speed.pypy.org/comparison/?exe=2%2B35,4%2B35,1%2B172,3%2B172&ben=11,14,15&env=1&hor=false&bas=none&chart=normal+bars >> >> slowspitfire is much faster than the other two. Is that because it >> performs more iterations? > > I think it's apples to oranges (they have different table sizes and > different number of iterations) > >> >> Also, how come pypy-c-jit is faster than cpython or psyco precisely in >> cstringio, where performance should be dependent on cstringIO and thus >> be more similar across interpreters? > > because having a list of small strings means you have a large (old) > object referencing a lot of young objects, hence GC cost. It's not the > case with cstringio where you have a single chunk of memory which does > not contain GC pointers. > >> >> >> 2010/12/13 Leonardo Santagada : >>> why not have only 2 versions, both with the same size table and name >>> one spitfire_cstringio and the other spitfire_strjoin? I think it >>> would make things clearer. >>> >>> >>> On Mon, Dec 13, 2010 at 2:43 PM, Maciej Fijalkowski wrote: >>>> Hi. >>>> >>>> spitfires are confusing. >>>> >>>> slowspitfire and spitfire use ''.join(list-of-strings) where >>>> spitfire_cstringio uses cStringIO instead. >>>> >>>> spitfire and spitfire_cstringio use smaller table to render (100x100 I >>>> think) which was the default on original benchmarks >>>> >>>> slowspitfire uses 1000x1000 (which is why it used to be slower than >>>> spitfire) and was chosen by US guys to let the JIT warm up. We should >>>> remove _slow these days. >>>> >>>> On Mon, Dec 13, 2010 at 5:08 PM, Miquel Torres wrote: >>>>> sorry, I meant the opposite. To recap, according to >>>>> http://code.google.com/p/unladen-swallow/wiki/Benchmarks, >>>>> spitfire: psyco >>>>> slowspitfire: pure python >>>>> >>>>> in addition we have spitfire_cstringio, which uses a c module (so it >>>>> is even faster). >>>>> >>>>> what is vanilla spitfire in our case? >>>>> >>>>> >>>>> 2010/12/13 Miquel Torres : >>>>>> @Carl Friedrich & exarkun: thanks, I've added those. >>>>>> >>>>>> only spectral-norm, slowspitfire and ai to go. >>>>>> >>>>>> slowspitfire is described at the Unladen page as using psyco, but it >>>>>> doesn't make sense in our case? >>>>>> >>>>>> >>>>>> >>>>>> 2010/12/13 ?: >>>>>>> On 08:20 am, tobami at googlemail.com wrote: >>>>>>>> >>>>>>>> Thanks all for the input. >>>>>>>> I've compiled a list based on your mails, the Unladen benchmarks page >>>>>>>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >>>>>>>> alioth descriptions. Here is an extract of the current speed.pypy.org >>>>>>>> admin: >>>>>>>> >>>>>>>> [snip] >>>>>>>> twisted_iteration >>>>>>> >>>>>>> Iterates a Twisted reactor as quickly as possible without doing any work. >>>>>>>> >>>>>>>> twisted_names >>>>>>> >>>>>>> Runs a DNS server with Twisted Names and then issues requests to it over >>>>>>> loopback UDP. >>>>>>>> >>>>>>>> twisted_pb >>>>>>> >>>>>>> Runs a Perspective Broker server with a no-op method and invokes that method >>>>>>> over loopback TCP with some strings, dictionaries, and tuples as arguments. >>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> pypy-dev at codespeak.net >>>>> http://codespeak.net/mailman/listinfo/pypy-dev >>>>> >>>> _______________________________________________ >>>> pypy-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >>> >>> >>> -- >>> Leonardo Santagada >>> >> > From kevin at movieextras.ie Tue Dec 14 11:13:46 2010 From: kevin at movieextras.ie (Kevin Gill) Date: Tue, 14 Dec 2010 10:13:46 +0000 Subject: [pypy-dev] Problem building C Extension Message-ID: <4D07435A.80608@movieextras.ie> Hi, I am new to PyPy. I built pypy using... $ python translate.py --opt=jit targetpypystandalone.py When I try to build eggs with C extensions it doesn't work. I am using pip in a virtualenv. The build uses the wrong compiler "cc" instead of "gcc" and does not use CFLAGS. For example: $ pip -v install zope.interface==3.6.1 shows this output cc -I/home/kevin/pypp/env/include -c src/zope/interface/_zope_interface_coptimizations.c -o build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o src/zope/interface/_zope_interface_coptimizations.c: In function ?verifying_changed?: src/zope/interface/_zope_interface_coptimizations.c:1349: warning: assignment makes pointer from integer without a cast cc -shared build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o -o build/lib.linux-x86_64-2.5/zope/interface/_zope_interface_coptimizations.pypy-14.so /usr/bin/ld: build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: could not read symbols: Bad value collect2: ld returned 1 exit status I did some digging through the source. I found that the compiler configuration is normally loaded via distutils/sysconfig.py, but that module was effectively disabled in pypy. I cannot override using environment variables for example because the customisation function is as follows: def customize_compiler(options): """Dummy method to let some easy_install packages that have optional C speedup components. """ pass I see from the mailing list that people are compiling C extensions. What am I doing wrong? Thanks Kevin ------------------------------------------------------------ Version of pypy that I am using. $ svn info Path: . URL: http://codespeak.net/svn/pypy/trunk Repository Root: http://codespeak.net/svn Repository UUID: fd0d7bf2-dfb6-0310-8d31-b7ecfe96aada Revision: 79796 Node Kind: directory Schedule: normal Last Changed Author: afa Last Changed Rev: 79794 Last Changed Date: 2010-12-03 21:52:49 +0000 (Fri, 03 Dec 2010) From anto.cuni at gmail.com Tue Dec 14 11:53:05 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 14 Dec 2010 11:53:05 +0100 Subject: [pypy-dev] Migration to mercurial In-Reply-To: <4D06A621.4060300@gmail.com> References: <4D06A621.4060300@gmail.com> Message-ID: <4D074C91.7020005@gmail.com> On 14/12/10 00:02, Antonio Cuni wrote: > Hi all, > > finally, it's happening :-). > Thanks to Ronny's work, we are going to complete the migration to mercurial > very soon. so, the migration is done! The svn repo is readonly, and from now the official pypy repo is this one: http://bitbucket.org/pypy/pypy There are still some rough edges to fix on buildbot, but for the rest everything seems to work fine. I would like to thank everyone who was involved in the migration, in particular Ronny who did most of the dirty work :-) ciao, Anto From fijall at gmail.com Tue Dec 14 12:49:25 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Dec 2010 13:49:25 +0200 Subject: [pypy-dev] Problem building C Extension In-Reply-To: <4D07435A.80608@movieextras.ie> References: <4D07435A.80608@movieextras.ie> Message-ID: Not to comment on issue, but it's actually pretty good that zope.interface C speedups don't work. They won't be speedups in pypy case even if they did work. On Tue, Dec 14, 2010 at 12:13 PM, Kevin Gill wrote: > Hi, > > I am new to PyPy. > > I built pypy using... > > $ python translate.py --opt=jit targetpypystandalone.py > > When I try to build eggs with C extensions it doesn't work. I am using pip in a virtualenv. The build > uses the wrong compiler "cc" instead of "gcc" and does not use CFLAGS. > > For example: > > $ pip -v install zope.interface==3.6.1 > > shows this output > > cc -I/home/kevin/pypp/env/include -c src/zope/interface/_zope_interface_coptimizations.c -o build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o > src/zope/interface/_zope_interface_coptimizations.c: In function ?verifying_changed?: > src/zope/interface/_zope_interface_coptimizations.c:1349: warning: assignment makes pointer from integer without a cast > cc -shared build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o -o build/lib.linux-x86_64-2.5/zope/interface/_zope_interface_coptimizations.pypy-14.so > /usr/bin/ld: build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC > build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: could not read symbols: Bad value > collect2: ld returned 1 exit status > > I did some digging through the source. I found that the compiler configuration is normally loaded > via distutils/sysconfig.py, but that module was effectively disabled in pypy. I cannot override > using environment variables for example because the customisation function is as follows: > > def customize_compiler(options): > ? ? """Dummy method to let some easy_install packages that have > ? ? optional C speedup components. > ? ? """ > ? ? pass > > I see from the mailing list that people are compiling C extensions. What am I doing wrong? > > Thanks > > Kevin > > ------------------------------------------------------------ > Version of pypy that I am using. > > $ svn info > Path: . > URL: http://codespeak.net/svn/pypy/trunk > Repository Root: http://codespeak.net/svn > Repository UUID: fd0d7bf2-dfb6-0310-8d31-b7ecfe96aada > Revision: 79796 > Node Kind: directory > Schedule: normal > Last Changed Author: afa > Last Changed Rev: 79794 > Last Changed Date: 2010-12-03 21:52:49 +0000 (Fri, 03 Dec 2010) > > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From cfbolz at gmx.de Tue Dec 14 15:14:13 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 14 Dec 2010 15:14:13 +0100 Subject: [pypy-dev] [PATCH] Improving readability of generated .c code In-Reply-To: <1292288224.24137.22.camel@radiator.bos.redhat.com> References: <1292288224.24137.22.camel@radiator.bos.redhat.com> Message-ID: <4D077BB5.7030102@gmx.de> Hi David, On 12/14/2010 01:57 AM, David Malcolm wrote: > The attached patch is an attempt at extending the C code generator so > that it annotates the generated .c code with file/line information for > the corresponding Python sources. > > Method: > I followed the instructions in pypy/doc/getting-started-dev.txt: > cd pypy > python bin/translatorshell.py > >>> t = Translation(snippet.is_perfect_number) > >>> t.annotate([int]) > >>> f = t.compile_c() > then examine the generated .c code. > > With this patch, the top of a generated .c function contains an inline > copy of the python source, like this: > bool_t pypy_g_ll_issubclass__object_vtablePtr_object_vtablePtr(struct pypy_object_vtable0 *l_subcls_0, struct pypy_object_vtable0 *l_cls_0) { > bool_t l_v11; long l_v12; long l_v13; long l_v14; > /* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rclass.py' > * 665 : def ll_issubclass(subcls, cls): > * 666 : return llop.int_between(Bool, cls.subclassrange_min, > * 667 : subcls.subclassrange_min, > * 668 : cls.subclassrange_max) > */ > goto block0; > > > ...and for every operation that has an offset, it calculates the > linenumber (using the "dis" module on the code object). For every > operation that changes the line number, it emits a comment containing > that Python source, so that you get code like this: > > block0: > /* 666 : return llop.int_between(Bool, cls.subclassrange_min, */ > l_v12 = RPyField(l_cls_0, ov_subclassrange_min); > /* 667 : subcls.subclassrange_min, */ > l_v13 = RPyField(l_subcls_0, ov_subclassrange_min); > /* 668 : cls.subclassrange_max) */ > l_v14 = RPyField(l_cls_0, ov_subclassrange_max); > OP_INT_BETWEEN(l_v12, l_v13, l_v14, l_v11); > goto block1; > > Hopefully this makes the generated .c much more readable. I like the general approach, though I have the vague fear that it will make the generated C code much bigger. Could you check that this is not the case? > (I hope to > work on the names of locals also; ideally they ought to embed the python > indentifier) This should already work in theory, see e.g. "l_cls_0" above, which embeds the Python variable name "cls". I guess that the information is simply lost in some of the transformation steps. If you feel like hunting down where, you can probably get better naming in some cases. > This is on a Fedora 13 x86_64 box, with cpython 2.6.5 > > Caveats: > - this is my first patch to PyPy (please be gentle!): I hope I've > correctly understood the various levels, but I suspect I may still be > missing things at the conceptual level I think you are doing great. You might have to grep around a bit for more places that create SpaceOperations, or did you look at all of them? > - haven't completed the full unit tests yet. This is the biggest problem of the patch, of course. I think that you should at least try to write tests for each of the places where you propagate the offset down one level. In addition, you probably need an integration test where you check the generated C code for the presence of the correct comments. > - the patch attempts to propagate the "offset" of flowspace > SpaceOperation instances in a couple of places where the offset was > being discarded: > - in BaseInliner.copy_operation This is wrong. If you inline one function into another, it doesn't make sense to put the offset of the inner function into the outer function. To solve this problem, you have two options: 1) when inlining, set the offset to -1, and just don't get source code then 2) make every *operation* contain a reference to the func it comes from. The memory implications of this should be checked, but are probably not too bad. > - when generating operations in rtyper: set each of the new > operations' "offset" to that of the high-level operation That makes sense. > - I added the offset to SpaceOperation.__repr__() to help track the > above down. > - I also pass the reference to the func object from the flowspace > FunctionGraph to the rtyper's copy (I wonder if this affect memory usage > during translation?) Probably not enough to care. > - most functions appear correct, but there are some where I'm not at > all convinced by the output (see below) > - the patch is assuming that within any given generated C function, > there is a single python function, and that the offsets are indexes into > the bytecode for that function's code object. I suspect that this is an > oversimplification, and why I'm seeing the errors that I'm seeing. > - I haven't stripped some of my debugging python comments from > funcgen.py in the patch (this is a work in progress). Now that we switched to mercurial, you can probably simply make a branch and work there: http://bitbucket.org/pypy/pypy/overview > - I haven't tried doing this on a full build of the interpreter > (having highly readable generated .c for this is my objective [1]). I fear it will never be "highly readable", but I think your work improves things already. > One other idea I had was to sort the blocks in the function by the > bytecode offset; currently the c blocks seem to "jump around" a fair bit > relative to the corresponding rpython code. Has any tuning been done on > the ordering of the blocks within the generated .c code? (or is it > assumed that the .c compiler will "flatten" these, for the case when a > node has a single successor node?) The C compiler should not care about the order of blocks. If it does, I am prepared to call it insane. > (I wonder if similar work could be done on the JVM and CLI code > generators?) > > As mentioned above, I think my patch is getting it wrong in a few > places. Here's an example: > struct pypy_rpy_string0 *pypy_g_mallocstr__Signed(long l_length_3) { > > ...snip... > > /* Python source '/home/david/coding/pypy-svn-anon/trunk-clean/pypy/rpython/lltypesystem/rstr.py' > * 35 : def mallocstr(length): > * 36 : ll_assert(length>= 0, "negative string length") > * 37 : r = malloc(TP, length) > * 38 : if not we_are_translated() or not malloc_zero_filled: > * 39 : r.hash = 0 > * 40 : return r > */ > > ...snip... > > block15: > /* 37 : r = malloc(TP, length) */ > OP_ADR_SUB(l_v202, 0, l_v234); > l_v235 = (struct pypy_header0 *)l_v234; > l_v236 = RPyField(l_v235, h_refcount); > /* 38 : if not we_are_translated() or not malloc_zero_filled: */ > OP_INT_ADD(l_v236, 1L, l_v237); > RPyField(l_v235, h_refcount) = l_v237; > goto block5; > > The C code doesn't look anything like the .py code that my patch is > inserting in the above. (My current guess is that I need to be smarter > about inlined operations, though that's a guess at this stage) Sounds like a correct guess, malloc is transformed into something else, and the offset of the inlined operations is basically just random, if you interpret it in the context of the outer function. Cheers, Carl Friedrich From kevin at movieextras.ie Tue Dec 14 16:33:42 2010 From: kevin at movieextras.ie (Kevin Gill) Date: Tue, 14 Dec 2010 15:33:42 +0000 Subject: [pypy-dev] Problem building C Extension Message-ID: <1292340822.28537.1410292989@webmail.messagingengine.com> On Tue, 14 Dec 2010 13:49 +0200, "Maciej Fijalkowski" wrote: > Not to comment on issue, but it's actually pretty good that > zope.interface C speedups don't work. They won't be speedups in pypy > case even if they did work. > I accept that they may not be speedups. However, I cannot be certain whether C code in an egg represents C alternative to python functionality, or whether the functionality is implemented in C. Perhaps this information is in distutils (setup.py) - I haven't noticed it. > On Tue, Dec 14, 2010 at 12:13 PM, Kevin Gill > wrote: > > Hi, > > > > I am new to PyPy. > > > > I built pypy using... > > > > $ python translate.py --opt=jit targetpypystandalone.py > > > > When I try to build eggs with C extensions it doesn't work. I am using pip in a virtualenv. The build > > uses the wrong compiler "cc" instead of "gcc" and does not use CFLAGS. > > > > For example: > > > > $ pip -v install zope.interface==3.6.1 > > > > shows this output > > > > cc -I/home/kevin/pypp/env/include -c src/zope/interface/_zope_interface_coptimizations.c -o build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o > > src/zope/interface/_zope_interface_coptimizations.c: In function ?verifying_changed?: > > src/zope/interface/_zope_interface_coptimizations.c:1349: warning: assignment makes pointer from integer without a cast > > cc -shared build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o -o build/lib.linux-x86_64-2.5/zope/interface/_zope_interface_coptimizations.pypy-14.so > > /usr/bin/ld: build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC > > build/temp.linux-x86_64-2.5/src/zope/interface/_zope_interface_coptimizations.o: could not read symbols: Bad value > > collect2: ld returned 1 exit status > > > > I did some digging through the source. I found that the compiler configuration is normally loaded > > via distutils/sysconfig.py, but that module was effectively disabled in pypy. I cannot override > > using environment variables for example because the customisation function is as follows: > > > > def customize_compiler(options): > > ? ? """Dummy method to let some easy_install packages that have > > ? ? optional C speedup components. > > ? ? """ > > ? ? pass > > > > I see from the mailing list that people are compiling C extensions. What am I doing wrong? > > > > Thanks > > > > Kevin > > > > ------------------------------------------------------------ > > Version of pypy that I am using. > > > > $ svn info > > Path: . > > URL: http://codespeak.net/svn/pypy/trunk > > Repository Root: http://codespeak.net/svn > > Repository UUID: fd0d7bf2-dfb6-0310-8d31-b7ecfe96aada > > Revision: 79796 > > Node Kind: directory > > Schedule: normal > > Last Changed Author: afa > > Last Changed Rev: 79794 > > Last Changed Date: 2010-12-03 21:52:49 +0000 (Fri, 03 Dec 2010) > > > > > > > > > > > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > From jacob at openend.se Tue Dec 14 18:04:06 2010 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 14 Dec 2010 18:04:06 +0100 Subject: [pypy-dev] Mercurial workflow Message-ID: <201012141804.06449.jacob@openend.se> Hi folks, now that the switch to Mercurial has happened, people are discovering that their Subversion workflow habits don't quite work. This is because Mercurial has a distributed philosophy, inlike svn which has the concept of the holy central server where all operations take place. In Mercurial, you do all the important operations in your local repository. If you have a central node (which is not strictly necessary), you just push ready made change sets to it. Any conflict resolving has already been done on your local machine. This means that it is very convenient to have at least 2 repositories locally (this takes hadly any extra space, since unchanged files are hard linked to be the same file in each repository). You keep one repository as a staging area to which you pull changes from the central server (or other peoples repositories) and to which you pull changes from your other local repositories. It is here you resolve any merge conflicts and it is from here you do pushes of new versions to the central pypy server on bitbucket. For every idea you have, you clone your repository and do the work for the idea in the clone. If you need to collaborate with someone else you synchronize your repo with their repo without going through the central server. Branching is a concept that is separated from the repository clones and it is just as good an idea to make branches for new features as it was in svn. When you work on an idea you should keep in mind that it is an extremely fast operation to commit your work. You can do it for a number of various reasons - whenever you have made a set of changes that you feel belong together, if your machine is flakey and you want to have a backup copy somewhere else or if you want to continue the work on a different machine. I sometimes program in bed, and it is very simple to do a pull from my workstation to my laptop. The new workflow will make the detailed changes less visible to others, since you commit work to the central server in fewer and bigger chunks. This means that there will be fewer "oops, I misspelled this" and more of a higher level view of what is being developed. I hope this helps getting your minds around how to work with Mercurial. I hope you will discover that the way it is different from svn will improve your work environment and your work habits. It has done so for me. Jacob From adimasci at gmail.com Tue Dec 14 18:42:04 2010 From: adimasci at gmail.com (Adrien Di Mascio) Date: Tue, 14 Dec 2010 18:42:04 +0100 Subject: [pypy-dev] Mercurial workflow In-Reply-To: <201012141804.06449.jacob@openend.se> References: <201012141804.06449.jacob@openend.se> Message-ID: Hi Jacob, On Tue, Dec 14, 2010 at 6:04 PM, Jacob Hall?n wrote: > now that the switch to Mercurial has happened, people are discovering that > their Subversion workflow habits don't quite work. This is because Mercurial > has a distributed philosophy, inlike svn which has the concept of the holy > central server where all operations take place. It might be relevant for you to consider the mq extension : - http://mercurial.selenic.com/wiki/MqExtension - http://mercurial.selenic.com/wiki/MqTutorial - http://mercurial.selenic.com/wiki/MqExtension I find it very convenient on a daily-basis usage. When you want to implement or fix something, you just create a new patch : $ hg qnew my_new_idea You then start to modify code and update your patch whenever you feel the current version of your code is better than the previous one : $ hq qrefresh Then, when you're happy with it, all tests pass, etc., unapply your patch, pull the new changesets from the "central" repository, and then reapply your patch : $ hg qpop # pop your patch $ hg pull central_repo_url # pull new changesets $ hg up $ hg qpush # reapply your patch $ $ hg qfinish -a # finalize and promote your patch to a changeset tracked in the repository $ hg push central_repo_url # push your changeset You can work on several patches at the same time, you can even track your patches in a mercurial repository. In my experience, it helps keeping a revision tree clean and avoid quite a few unnecessary merges. Of course there is also the "rebase" command and some other extensions I haven't tried such as pbranch but the standard "mq" extension is itself very useful. Just my two cents. Regards, Adrien. From fijall at gmail.com Tue Dec 14 20:01:39 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Dec 2010 21:01:39 +0200 Subject: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output on standard error when invoking Mercurial. In-Reply-To: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com> References: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com> Message-ID: This commit contains no tests whatsoever, it would be cooler if it did. On Tue, Dec 14, 2010 at 8:19 PM, Bitbucket wrote: > # HG changeset patch -- Bitbucket.org > # Project pypy > # URL http://bitbucket.org/pypy/pypy/overview > # User Dan Villiom Podlaski Christiansen > # Date 1292349796 -3600 > # Node ID 7b8aa74da1fb06bd5af8b37e086538e4e7d5d0a1 > # Parent ?1f81ed99bfbc17bbb3cc0920ac612218b4194f8c > Handle output on standard error when invoking Mercurial. > > Instead of just piped to the process standard error, we use a logger > to warn if 'hg' wrote anything on stderr. > > --- a/pypy/tool/version.py > +++ b/pypy/tool/version.py > @@ -13,24 +13,39 @@ def get_mercurial_info(): > ? ? hgexe = py.path.local.sysfind('hg') > > ? ? if hgexe and os.path.isdir(os.path.join(pypyroot, '.hg')): > + ? ? ? ?def maywarn(err): > + ? ? ? ? ? ?if not err: > + ? ? ? ? ? ? ? ?return > + > + ? ? ? ? ? ?from pypy.tool.ansi_print import ansi_log > + ? ? ? ? ? ?log = py.log.Producer("version") > + ? ? ? ? ? ?py.log.setconsumer("version", ansi_log) > + ? ? ? ? ? ?log.WARNING('Errors getting Mercurial information:' + err) > + > ? ? ? ? env = dict(os.environ) > ? ? ? ? # get Mercurial into scripting mode > ? ? ? ? env['HGPLAIN'] = '1' > ? ? ? ? # disable user configuration, extensions, etc. > ? ? ? ? env['HGRCPATH'] = os.devnull > > - ? ? ? ?p = Popen([str(hgexe), 'id', '-i', pypyroot], stdout=PIPE, env=env) > + ? ? ? ?p = Popen([str(hgexe), 'id', '-i', pypyroot], > + ? ? ? ? ? ? ? ? ?stdout=PIPE, stderr=PIPE, env=env) > ? ? ? ? hgid = p.stdout.read().strip() > + ? ? ? ?maywarn(p.stderr.read()) > > - ? ? ? ?p = Popen([str(hgexe), 'id', '-t', pypyroot], stdout=PIPE, env=env) > + ? ? ? ?p = Popen([str(hgexe), 'id', '-t', pypyroot], > + ? ? ? ? ? ? ? ? ?stdout=PIPE, stderr=PIPE, env=env) > ? ? ? ? hgtags = [t for t in p.stdout.read().strip().split() if t != 'tip'] > + ? ? ? ?maywarn(p.stderr.read()) > > ? ? ? ? if hgtags: > ? ? ? ? ? ? return 'PyPy', hgtags[0], hgid > ? ? ? ? else: > ? ? ? ? ? ? # use the branch instead > - ? ? ? ? ? ?p = Popen([str(hgexe), 'id', '-b', pypyroot], stdout=PIPE, env=env) > + ? ? ? ? ? ?p = Popen([str(hgexe), 'id', '-b', pypyroot], > + ? ? ? ? ? ? ? ? ? ? ?stdout=PIPE, stderr=PIPE, env=env) > ? ? ? ? ? ? hgbranch = p.stdout.read().strip() > + ? ? ? ? ? ?maywarn(p.stderr.read()) > > ? ? ? ? ? ? return 'PyPy', hgbranch, hgid > ? ? else: > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From danchr at gmail.com Tue Dec 14 20:28:34 2010 From: danchr at gmail.com (Dan Villiom Podlaski Christiansen) Date: Tue, 14 Dec 2010 20:28:34 +0100 Subject: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output on standard error when invoking Mercurial. In-Reply-To: References: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com> Message-ID: <32AFCFE5-CF32-441A-905F-0DF92BFA7964@gmail.com> On 14 Dec 2010, at 20:01, Maciej Fijalkowski wrote: > This commit contains no tests whatsoever, it would be cooler if it > did. Antonio kindly wrote a test for the functionality in an earlier changeset. Ironically, I broke it that change, but that's fixed now :) -- Dan Villiom Podlaski Christiansen danchr at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1943 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101214/bb3b33b0/attachment-0001.bin From anto.cuni at gmail.com Tue Dec 14 23:49:32 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 14 Dec 2010 23:49:32 +0100 Subject: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output on standard error when invoking Mercurial. In-Reply-To: <32AFCFE5-CF32-441A-905F-0DF92BFA7964@gmail.com> References: <20101214181946.6E7431E14AC@bitbucket02.managed.contegix.com> <32AFCFE5-CF32-441A-905F-0DF92BFA7964@gmail.com> Message-ID: <4D07F47C.3060304@gmail.com> On 14/12/10 20:28, Dan Villiom Podlaski Christiansen wrote: > On 14 Dec 2010, at 20:01, Maciej Fijalkowski wrote: > >> This commit contains no tests whatsoever, it would be cooler if it did. > > Antonio kindly wrote a test for the functionality in an earlier changeset. > Ironically, I broke it that change, but that's fixed now :) anyway, maciek on the channel mentioned a way to test it without messing with the hg repo: you can provide a custom hgexe script which returns whatever you want. Or even, restructure the code in a way that the running of the hg command is decoupled by the parsing of its results, so that you can at least test the latter. Sorry, I should have pointed this out when you asked me to review the patches, but today I was fried and had thousands of things to look at :-/ ciao, Anto From qbproger at gmail.com Tue Dec 14 23:53:41 2010 From: qbproger at gmail.com (Joe) Date: Tue, 14 Dec 2010 17:53:41 -0500 Subject: [pypy-dev] Mercurial workflow In-Reply-To: References: <201012141804.06449.jacob@openend.se> Message-ID: In reference to MqExtention, I've used Stacked Git (a git equivalent) in the past, and would generally advise against this. I like to commit after I get a piece of something working, so I know I can go back to that point if I try something and it doesn't work out. Stacked Git makes this hard. In my opinion, you're essentially by-passing a benefit of a scm tool. I wouldn't advise people to use MqExtension unless they understand that if they make a mistake they must undo it manually. Joe On Tue, Dec 14, 2010 at 12:42 PM, Adrien Di Mascio wrote: > Hi Jacob, > > On Tue, Dec 14, 2010 at 6:04 PM, Jacob Hall?n wrote: >> now that the switch to Mercurial has happened, people are discovering that >> their Subversion workflow habits don't quite work. This is because Mercurial >> has a distributed philosophy, inlike svn which has the concept of the holy >> central server where all operations take place. > > It might be relevant for you to consider the mq extension : > > - http://mercurial.selenic.com/wiki/MqExtension > - http://mercurial.selenic.com/wiki/MqTutorial > - http://mercurial.selenic.com/wiki/MqExtension > > I find it very convenient on a daily-basis usage. When you want to > implement or fix something, you just create a new patch : > > $ hg qnew my_new_idea > > You then start to modify code and update your patch whenever you feel > the current version of your code is better than the previous one : > > $ hq qrefresh > > Then, when you're happy with it, all tests pass, etc., unapply your > patch, pull the new changesets from the "central" repository, and then > reapply your patch : > > $ hg qpop ? ? ? ? ? ? ? ? ? ? ? ? ? # pop your patch > $ hg pull central_repo_url ? ? # pull new changesets > $ hg up > $ hg qpush ? ? ? ? ? ? ? ? ? ? ? ? # reapply your patch > $ > $ hg qfinish -a ? ? ? ? ? ? ? ? ? ? # finalize and promote your patch > to a changeset tracked in the repository > $ hg push central_repo_url ? # push your changeset > > You can work on several patches at the same time, you can even track > your patches in a mercurial repository. In my experience, it helps > keeping a revision tree clean and avoid quite a few unnecessary > merges. Of course there is also the "rebase" command and some other > extensions I haven't tried such as pbranch but the standard "mq" > extension is itself very useful. > > Just my two cents. > Regards, > Adrien. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From santagada at gmail.com Wed Dec 15 00:38:14 2010 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 14 Dec 2010 21:38:14 -0200 Subject: [pypy-dev] Mercurial workflow In-Reply-To: References: <201012141804.06449.jacob@openend.se> Message-ID: I would also advise against using mq for the same reason plus it is a whole set of commands to learn. What I would advise is to try to use rebase more instead of merging most of the time. Here is a link to some docs about rebase http://mercurial.selenic.com/wiki/RebaseProject On Tue, Dec 14, 2010 at 8:53 PM, Joe wrote: > In reference to MqExtention, I've used Stacked Git (a git equivalent) > in the past, and would generally advise against this. ?I like to > commit after I get a piece of something working, so I know I can go > back to that point if I try something and it doesn't work out. > Stacked Git makes this hard. ?In my opinion, you're essentially > by-passing a benefit of a scm tool. > > I wouldn't advise people to use MqExtension unless they understand > that if they make a mistake they must undo it manually. > > Joe > > On Tue, Dec 14, 2010 at 12:42 PM, Adrien Di Mascio wrote: >> Hi Jacob, >> >> On Tue, Dec 14, 2010 at 6:04 PM, Jacob Hall?n wrote: >>> now that the switch to Mercurial has happened, people are discovering that >>> their Subversion workflow habits don't quite work. This is because Mercurial >>> has a distributed philosophy, inlike svn which has the concept of the holy >>> central server where all operations take place. >> >> It might be relevant for you to consider the mq extension : >> >> - http://mercurial.selenic.com/wiki/MqExtension >> - http://mercurial.selenic.com/wiki/MqTutorial >> - http://mercurial.selenic.com/wiki/MqExtension >> >> I find it very convenient on a daily-basis usage. When you want to >> implement or fix something, you just create a new patch : >> >> $ hg qnew my_new_idea >> >> You then start to modify code and update your patch whenever you feel >> the current version of your code is better than the previous one : >> >> $ hq qrefresh >> >> Then, when you're happy with it, all tests pass, etc., unapply your >> patch, pull the new changesets from the "central" repository, and then >> reapply your patch : >> >> $ hg qpop ? ? ? ? ? ? ? ? ? ? ? ? ? # pop your patch >> $ hg pull central_repo_url ? ? # pull new changesets >> $ hg up >> $ hg qpush ? ? ? ? ? ? ? ? ? ? ? ? # reapply your patch >> $ >> $ hg qfinish -a ? ? ? ? ? ? ? ? ? ? # finalize and promote your patch >> to a changeset tracked in the repository >> $ hg push central_repo_url ? # push your changeset >> >> You can work on several patches at the same time, you can even track >> your patches in a mercurial repository. In my experience, it helps >> keeping a revision tree clean and avoid quite a few unnecessary >> merges. Of course there is also the "rebase" command and some other >> extensions I haven't tried such as pbranch but the standard "mq" >> extension is itself very useful. >> >> Just my two cents. >> Regards, >> Adrien. >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- Leonardo Santagada From p.giarrusso at gmail.com Wed Dec 15 02:09:36 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Wed, 15 Dec 2010 02:09:36 +0100 Subject: [pypy-dev] Mercurial workflow In-Reply-To: <201012141804.06449.jacob@openend.se> References: <201012141804.06449.jacob@openend.se> Message-ID: Hi, as a longer time DVCS (Git/Mercurial) user, here's some comments - just my two cents. On Tue, Dec 14, 2010 at 18:04, Jacob Hall?n wrote: > Hi folks, > In Mercurial, you do all the important operations in your local repository. If > you have a central node (which is not strictly necessary), you just push ready > made change sets to it. Any conflict resolving has already been done on your > local machine. > > This means that it is very convenient to have at least 2 repositories locally > (this takes hadly any extra space, since unchanged files are hard linked to be > the same file in each repository). If you use hg clone, only files under .hg are hard linked - and I guess they stay so only until they're touched by a commit, when the _whole_ previous content is duplicated. If you can do that also for the checked-out copy, you need to use cp -al, but it's not done by default, and you need to have advanced Unixy editors like Vim/emacs, and to set them up correctly. E.g. For Vim, you need to set backupcopy=breakhardlink (+ any other settings) or =no, otherwise any editing will affect _both_ repos (if you actually use Vim, read :help 'backupcopy' to understand your options). patch(1) is also fine, as many tools used by Linux kernel hackers. I hardly believe many other editors give you control such options. If you used a Java-based editor (say jEdit), it would even be challenging to code such a feature. > You keep one repository as a staging area to which you pull changes from the > central server (or other peoples repositories) and to which you pull changes > from your other local repositories. It is here you resolve any merge conflicts > and it is from here you do pushes of new versions to the central pypy server > on bitbucket. One could have two local heads (or something alike) and switch between them - if you need to recompile after merging and before pushing, however, things are different of course. And you might want to have a bitbucket fork for your work which you're not pushing yet, to have patches reviewed before merging. > For every idea you have, you clone your repository and do the work for the > idea in the clone. If you need to collaborate with someone else you > synchronize your repo with their repo without going through the central > server. If "your repo" means a Bitbucket fork (or any sort of publicly accessible clone, but Bitbucket has some goodies) that's fine and good. Otherwise, it might make harder for people to have a casual look at what you're working on - they need to be motivated enough to ask you to join. > When you work on an idea you should keep in mind that it is an extremely fast > operation to commit your work. You can do it for a number of various reasons - > whenever you have made a set of changes that you feel belong together, if your > machine is flakey and you want to have a backup copy somewhere else or if you > want to continue the work on a different machine. I sometimes program in bed, > and it is very simple to do a pull from my workstation to my laptop. > The new workflow will make the detailed changes less visible to others, since > you commit work to the central server in fewer and bigger chunks. This means > that there will be fewer "oops, I misspelled this" and more of a higher level > view of what is being developed. Actually, the ease of doing local commits might lead to many smaller ones. If you use hg mq and push only finished patches, or you extensively rewrite commits, then what you say is true. But there are some disadvantages because history is lost, and the use of git/hg rebase is not uncontroversial. See below for one high-profile opinion against git rebase (I don't think the git vs hg difference has any significance here): http://lwn.net/Articles/359207/ The bottom line is that there are more possible workflows, and you will maybe need to discuss which one you'll want to use, and set some policy. Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From p.giarrusso at gmail.com Wed Dec 15 02:26:09 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Wed, 15 Dec 2010 02:26:09 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: On Mon, Dec 13, 2010 at 09:31, Miquel Torres wrote: > Oh, btw., the "normalized" stacked bars now display a warning note > about its correctness, and how it must be viewed as giving results a > weighting instead of them being normalized. It even includes a link to > the proper paper. I hope that is enough for the strict statisticians > among us ;-) I see. Thanks! > See: > http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars > > PS: there is a bug in the jqPlot plotting library when null values are > present. Trying to display PyPy 1.3 results for the newer go, pyflake > or ?raytrace will create some nasty js loops. It also has problems > with autoscaling the axis sometimes. > > > > 2010/12/13 Miquel Torres : >> Thanks all for the input. >> I've compiled a list based on your mails, the Unladen benchmarks page >> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >> alioth descriptions. Here is an extract of the current speed.pypy.org >> admin: >> >> ai >> chaos ? Creates chaosgame-like fractals >> crypto_pyaes ? ?A pure python implementation of AES >> django ? ? ? ? ?Uses the Django template system to build a 150x150-cell HTML table >> >> fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The fannkuch >> benchmark is defined by programs in Performing Lisp Analysis of the >> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig. >> >> float ? ? ? ? ? Creates an array of points using circular projection and then >> normalizes and maximizes them. Floating-point heavy. >> go ? ? ? ? ? ? ?A go (chess like game) computer player AI. >> html5lib ? ? ? ?Parses the HTML 5 spec using html5lib >> meteor-contest ?Searchs for solutions to shape packing puzzle. >> nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models the >> orbits of Jovian planets, using a simple symplectic-integrator. >> pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and bzip2 >> decoder/decompressor. >> raytrace-simple A raytracer renderer >> richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates the task >> dispatcher in the kernel of an operating system. >> rietveld ? ? ? ?A Django application benchmark. >> slowspitfire >> spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam classifier >> spectral-norm >> spitfire ? ? ? ?Uses the Spitfire template system to build a 1000x1000-cell HTML table. >> spitfire_cstringio ? ? ?Uses the Spitfire template system to build a >> 1000x1000-cell HTML table, using the cstringio module. >> telco >> twisted_iteration >> twisted_names >> twisted_pb >> twisted_tcp ? ? Connects one Twised client to one Twisted server over TCP >> (on the loopback interface) and then writes bytes as fast as it can. >> waf ? ? Python-based framework for configuring, compiling and installing >> applications. It derives from the concepts of other build tools such >> as Scons, Autotools, CMake or Ant. >> >> >> So the remaining descriptions are >> ai >> slowspitfire (what is the exact difference between the three spitfire benches?) >> spectral-norm >> telco >> twisted (most of them) >> >> Are the descriptions all right so far?. They can be made much longer >> if you deem it desirable. >> >> on speed.pypy.org you will currently see the descriptions in 3 places: >> - Changes view: A tooltip on hover over each benchmark >> - Timeline: a description box beneath each plot >> - Comparison: A tooltip over each benchmark when hovering the >> selection menu on the left side. >> >> Any suggestions on how to improve it further are welcome ;-) >> >> Miquel >> >> >> 2010/12/9 Paolo Giarrusso : >>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada wrote: >>>> Here is a incomplete draft list: >>> >>>> [slow]spitfire[cstringio]: Spitfire is a template language, the >>>> cstringio version uses a modified engine (that uses cstringio) >>> >>>> spambayes: Spambayes is a bayesian spam filter >>> >>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I >>> remember when, because of this, spambayes was slower (including >>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4 >>> and the head (for slowspitfire it's the opposite). >>> >>> For the rest, I see no significant case of slowdown of PyPy over time. >>> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars >>> -- >>> Paolo Giarrusso - Ph.D. Student >>> http://www.informatik.uni-marburg.de/~pgiarrusso/ >>> >> > -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From wing1127aishi at gmail.com Wed Dec 15 18:11:28 2010 From: wing1127aishi at gmail.com (Leon Sit) Date: Wed, 15 Dec 2010 11:11:28 -0600 Subject: [pypy-dev] Interpretor for vectorized langugage Message-ID: Hi all Is pypy suitable for writing interpretor of vectorized language like Matlab, R etc which vector and matrix are first class objects? This includes matrix shape inference, and efficient linear algebra code generation. Any thoughts? Leon -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101215/2fbae5f4/attachment.htm From renesd at gmail.com Wed Dec 15 18:21:38 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Wed, 15 Dec 2010 17:21:38 +0000 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: Hi, have you seen numpy/scipy? Also see these: http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html http://ademan.wordpress.com/category/code/pypy/ cheers, On Wed, Dec 15, 2010 at 5:11 PM, Leon Sit wrote: > Hi all > > Is pypy suitable for writing interpretor of vectorized language like > Matlab, R etc which vector and matrix are first class objects? This includes > matrix shape inference, and efficient linear algebra code generation. > > Any thoughts? > > Leon > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101215/b8d6920b/attachment-0001.htm From wing1127aishi at gmail.com Wed Dec 15 18:52:55 2010 From: wing1127aishi at gmail.com (Leon Sit) Date: Wed, 15 Dec 2010 11:52:55 -0600 Subject: [pypy-dev] pypy-dev Digest, Vol 365, Issue 14 In-Reply-To: References: Message-ID: Yes I have. However I want to write an interpretor for matlab/R codebase, not using python to write numerical code. Leon Best Regards, Leon Sit Graduate Student in Computer Science at University of Chicago Message: 7 > Date: Wed, 15 Dec 2010 17:21:38 +0000 > From: Ren? Dudfield > Subject: Re: [pypy-dev] Interpretor for vectorized langugage > To: PyPy Dev > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > Hi, > > have you seen numpy/scipy? > > Also see these: > http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html > http://ademan.wordpress.com/category/code/pypy/ > > > cheers, > > > On Wed, Dec 15, 2010 at 5:11 PM, Leon Sit wrote: > > > Hi all > > > > Is pypy suitable for writing interpretor of vectorized language like > > Matlab, R etc which vector and matrix are first class objects? This > includes > > matrix shape inference, and efficient linear algebra code generation. > > > > Any thoughts? > > > > Leon > > > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://codespeak.net/pipermail/pypy-dev/attachments/20101215/b8d6920b/attachment.htm > > ------------------------------ > > _______________________________________________ > pypy-dev mailing list > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > > End of pypy-dev Digest, Vol 365, Issue 14 > ***************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101215/e56113fd/attachment.htm From tobami at googlemail.com Wed Dec 15 19:52:29 2010 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 15 Dec 2010 19:52:29 +0100 Subject: [pypy-dev] pypy-dev Digest, Vol 365, Issue 14 In-Reply-To: References: Message-ID: I think what Ren? means is that http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html is a good example of how a vectorized library can be implemented, which relates to your question of whether a vectorized language could be implemented. Regards, Miquel 2010/12/15 Leon Sit : > Yes I have. However I want to write an interpretor for matlab/R codebase, > not using python to write numerical code. > Leon > Best Regards, > > Leon Sit > Graduate Student in Computer Science at University of Chicago > > Message: 7 >> >> Date: Wed, 15 Dec 2010 17:21:38 +0000 >> From: Ren? Dudfield >> Subject: Re: [pypy-dev] Interpretor for vectorized langugage >> To: PyPy Dev >> Message-ID: >> ? ? ? ? >> Content-Type: text/plain; charset="iso-8859-1" >> >> Hi, >> >> have you seen numpy/scipy? >> >> Also see these: >> ? ?http://morepypy.blogspot.com/2009/07/pypy-numeric-experiments.html >> ? ?http://ademan.wordpress.com/category/code/pypy/ >> >> >> cheers, >> >> >> On Wed, Dec 15, 2010 at 5:11 PM, Leon Sit wrote: >> >> > Hi all >> > >> > Is pypy suitable for writing interpretor of vectorized language like >> > Matlab, R etc which vector and matrix are first class objects? This >> > includes >> > matrix shape inference, and efficient linear algebra code generation. >> > >> > Any thoughts? >> > >> > Leon >> > >> > _______________________________________________ >> > pypy-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/pypy-dev >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> http://codespeak.net/pipermail/pypy-dev/attachments/20101215/b8d6920b/attachment.htm >> >> ------------------------------ >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> >> >> End of pypy-dev Digest, Vol 365, Issue 14 >> ***************************************** > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Thu Dec 16 08:25:32 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 16 Dec 2010 08:25:32 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial Message-ID: <4D09BEEC.8030204@gmail.com> Hi Miquel, hi all, as you probably have noticed, we have recently migrated the main repo to mercurial. Now speed.pypy.org receives a revision number in the form "40046:2088ce763fc2", but of course it can no longer fetches the commit logs from the svn server. Would it be possible to fetch the commits from the bitbucket repo, please? ciao, Anto From alexandre.fayolle at logilab.fr Thu Dec 16 09:12:10 2010 From: alexandre.fayolle at logilab.fr (Alexandre Fayolle) Date: Thu, 16 Dec 2010 09:12:10 +0100 Subject: [pypy-dev] Mercurial workflow In-Reply-To: References: <201012141804.06449.jacob@openend.se> Message-ID: <201012160912.11005.alexandre.fayolle@logilab.fr> On Tuesday 14 December 2010 23:53:41 Joe wrote: > In reference to MqExtention, I've used Stacked Git (a git equivalent) > in the past, and would generally advise against this. I like to > commit after I get a piece of something working, so I know I can go > back to that point if I try something and it doesn't work out. > Stacked Git makes this hard. In my opinion, you're essentially > by-passing a benefit of a scm tool. > > I wouldn't advise people to use MqExtension unless they understand > that if they make a mistake they must undo it manually. Hi there, my 2 cents As Adrien said you can manage the patch repository itself with mercurial, so that you can quite easily undo changes that don't work for you. Also, the hg qimport command is a very useful one which enables to convert changesets to patches, which can be qfold'ed in a single larger but cleaner patch. In the end, all boils down to the style of changesets you want to see in the official repositories : lots of very small changes, with broken intermediate steps, trial and errors etc. or larger cleaner commits, which deal with a single idea and are much easier to peer review. This is not for me to decide for pypy, and both approach have benefits and drawbacks. It is however imho a worthwhile question. Kind regards, -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations D?veloppement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science From tobami at googlemail.com Thu Dec 16 09:29:10 2010 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 16 Dec 2010 09:29:10 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: <4D09BEEC.8030204@gmail.com> References: <4D09BEEC.8030204@gmail.com> Message-ID: Hi Anto, yes, that is expected, but no problem. Codespeed is designed in such a way that it can support different version control systems. It is just that there is only support for svn now ;-) So for mercurial support, I need to implement a function that takes a changeset hash, connects to the hg repo and returns a dict with 'date', 'user' and 'description' or 'summary' info. Anybody knows what library I could use for that? or do I have to parse local 'hg log' output? 2010/12/16 Antonio Cuni : > Hi Miquel, hi all, > > as you probably have noticed, we have recently migrated the main repo to > mercurial. ?Now speed.pypy.org receives a revision number in the form > "40046:2088ce763fc2", but of course it can no longer fetches the commit logs > from the svn server. > > Would it be possible to fetch the commits from the bitbucket repo, please? > > ciao, > Anto > From arigo at tunes.org Thu Dec 16 10:16:36 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 16 Dec 2010 10:16:36 +0100 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: Hi, On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield wrote: >> Is pypy suitable for writing interpretor of vectorized language like >> Matlab, R etc which vector and matrix are first class objects? This includes >> matrix shape inference, and efficient linear algebra code generation. > > have you seen numpy/scipy? The first aspect is simply if RPython would be suitable for writing an interpreter for, say, Matlab. The answer is "probably yes": PyPy would be suitable for such dynamic languages, giving you a JIT compiler for free. I don't really know how complex the core of these languages are, but I suspect not too much. Of course you are then going to hit the same problems that Ademan tries to solve for numpy/scipy, notably how to implement at least the basic linear algebra operations in such a way that the JIT can improve them. There are various goals there, e.g. to turn Python (or Matlab) code like A+B+C, adding three matrices together, into one matrix operation instead of two (as it is now: (A+B)+C). This is all a bit experimental so far. A bient?t, Armin. From anto.cuni at gmail.com Thu Dec 16 10:33:09 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 16 Dec 2010 10:33:09 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: References: <4D09BEEC.8030204@gmail.com> Message-ID: <4D09DCD5.3000602@gmail.com> Hi Miquel, On 16/12/10 09:29, Miquel Torres wrote: > Hi Anto, > > yes, that is expected, but no problem. Codespeed is designed in such a > way that it can support different version control systems. It is just > that there is only support for svn now ;-) > > So for mercurial support, I need to implement a function that takes a > changeset hash, connects to the hg repo and returns a dict with > 'date', 'user' and 'description' or 'summary' info. Anybody knows what > library I could use for that? or do I have to parse local 'hg log' > output? I think you have several options to do that. For the particular pypy case, maybe the simplest is to directly ask bitbucket for the changeset: https://bitbucket.org/pypy/pypy/changeset/c55286db0781/raw/ this has the advantage that you don't need any library but just an http request, but of course it works only for bitbucket repos. Alternatively, if you install mercurial you can then "import mercurial.commands" and use its public API from Python. Or as you said you can just execute hg log and parse the output: in this case you might be interested in the --template option, which allows you to format things exactly as you want, e.g.: $ hg log -r c55286db0781 --template '{author}\n{date|isodate}\n{desc}\n' Armin Rigo 2010-12-16 10:01 +0100 Document this publicly. see this page for more details: http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html ciao, Anto From tobami at googlemail.com Thu Dec 16 10:38:04 2010 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 16 Dec 2010 10:38:04 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: <4D09DCD5.3000602@gmail.com> References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> Message-ID: Ok, thanks Anto, I'll have a look at it this weekend! PS: If somebody has coded such a function, or feels like doing it that is also welcome ;-) 2010/12/16 Antonio Cuni : > Hi Miquel, > > On 16/12/10 09:29, Miquel Torres wrote: >> Hi Anto, >> >> yes, that is expected, but no problem. Codespeed is designed in such a >> way that it can support different version control systems. It is just >> that there is only support for svn now ;-) >> >> So for mercurial support, I need to implement a function that takes a >> changeset hash, connects to the hg repo and returns a dict with >> 'date', 'user' and 'description' or 'summary' info. Anybody knows what >> library I could use for that? or do I have to parse local 'hg log' >> output? > > I think you have several options to do that. For the particular pypy case, > maybe the simplest is to directly ask bitbucket for the changeset: > https://bitbucket.org/pypy/pypy/changeset/c55286db0781/raw/ > > this has the advantage that you don't need any library but just an http > request, but of course it works only for bitbucket repos. > > Alternatively, if you install mercurial you can then "import > mercurial.commands" and use its public API from Python. Or as you said you can > just execute hg log and parse the output: in this case you might be interested > in the --template option, which allows you to format things exactly as you > want, e.g.: > > $ hg log -r c55286db0781 --template '{author}\n{date|isodate}\n{desc}\n' > Armin Rigo > 2010-12-16 10:01 +0100 > Document this publicly. > > see this page for more details: > http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html > > ciao, > Anto > From danchr at gmail.com Thu Dec 16 11:14:14 2010 From: danchr at gmail.com (Dan Villiom Podlaski Christiansen) Date: Thu, 16 Dec 2010 11:14:14 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: <4D09DCD5.3000602@gmail.com> References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> Message-ID: <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> On 16 Dec 2010, at 10:33, Antonio Cuni wrote: > Alternatively, if you install mercurial you can then "import > mercurial.commands" and use its public API from Python. Or as you > said you can > just execute hg log and parse the output: in this case you might be > interested > in the --template option, which allows you to format things exactly > as you > want, e.g.: I'd suggest parsing the command line output of Mercurial, as it's the officially sanctioned stable API for Mercurial. The Python API, while faster, provides no stability guarantees. That being said, the following will create a repository instance and access the context for a particular changeset: >>> from mercurial import hg, ui >>> >>> repo = hg.repository(ui.ui(), 'path/to/repo') >>> ctx = repo['tip'] >>> log = ctx.description() The typical accessors for a changeset context are defined at . Please note that any use of the Python API means that the entire application is covered by the GPL. -- Dan Villiom Podlaski Christiansen danchr at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1943 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101216/1fd877d1/attachment-0001.bin From Arnd.Rechenburg at tomtom.com Thu Dec 16 12:58:31 2010 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Thu, 16 Dec 2010 12:58:31 +0100 Subject: [pypy-dev] import errors when calling pypy Message-ID: Hi, I have problems to start pypy. I got the following errors: $ pypy-1.4/bin/pypy pypy-1.4/bin/pypy: /usr/lib64/libcrypto.so.0.9.8: no version information available (required by pypy-1.4/bin/pypy) pypy-1.4/bin/pypy: /usr/lib64/libssl.so.0.9.8: no version information available (required by pypy-1.4/bin/pypy) 'import site' failed Python 2.5.2 (79529, Nov 25 2010, 23:04:06) [PyPy 1.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. ImportError: No module named atexit debug: OperationError: debug: operror-type: ImportError debug: operror-value: No module named _pypy_interact Could someone tell me how to set the environment for the libraries? I tried update PATH etc. but without success. On the system (linux SLES10, AMD 64-bit) itself python 2.5.5 is used. Any help would be appreciated. Thx in advance, Arnd Rechenburg Arnd Rechenburg * Senior Software Engineer * Tele Atlas Deutschland GmbH & Co. KG * Am Neuen Horizont 1, 31177 Harsum, Deutschland * HRA 200613 * VAT ID DE811953885 * Steuernummer 30/210/15702 * e-mail: arnd.rechenburg at teleatlas.com * direct +49 5127 40 8364 * reception +49 5127 40 80 * fax +49 5127 40 8348 * www.teleatlas.com P Please consider the environment before printing this e-mail. E-mail Confidentiality: This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s) named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. TELE ATLAS DOES NOT AMEND EXISTING AGREEMENTS OR CREATE ANY BINDING AGREEMENTS OR OBLIGATIONS VIA E-MAIL. ANY STATEMENTS OR ATTACHMENTS WITHIN THIS MESSAGE WILL NOT BE CONSIDERED BINDING. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation. Tele Atlas B.V. and its affiliates. www.teleatlas.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101216/c5923628/attachment-0001.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 9550 bytes Desc: image001.gif Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101216/c5923628/attachment-0001.gif From arigo at tunes.org Thu Dec 16 13:08:09 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 16 Dec 2010 13:08:09 +0100 Subject: [pypy-dev] import errors when calling pypy In-Reply-To: References: Message-ID: Hi Arnd, On Thu, Dec 16, 2010 at 12:58 PM, Arnd Rechenburg < Arnd.Rechenburg at tomtom.com> wrote: > I have problems to start pypy. > It is explained (or rather, at least hinted) here: http://pypy.org/download.html#installing In more words, though, the important point is that you need to *not* split up the zip or tbz2 archive that you are getting. PyPy looks for its libraries starting from the location of the "pypy" executable. You can put the uncompressed constant of the archive anywhere, but don't move "pypy" outside it. (It's fine to install a symlink to it from somewhere else.) We may "rectify" this situation when we have more of a clue about the directory layout that people would really expect, possibly including a Makefile to perform installation, for example. The current situation was deemed sufficient so far. A bient?t, Armin. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101216/f5aed236/attachment.htm From Arnd.Rechenburg at tomtom.com Thu Dec 16 13:35:39 2010 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Thu, 16 Dec 2010 13:35:39 +0100 Subject: [pypy-dev] error from function.py Message-ID: Hi, I got the following error message: File "/home/unixhome/rechenbu/output/pypy-1.4/lib-python/2.5.2/calendar.py", line 11, in import locale as _locale File "/home/unixhome/rechenbu/output/pypy-1.4/lib-python/modified-2.5.2/local e.py", line 30, in from _locale import * File "/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/_locale.py", line 8, in from ctypes_support import standard_c_lib as libc File "/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/ctypes_support.py", line 24, in standard_c_lib.__errno_location.restype = ctypes.POINTER(ctypes.c_int) File "/home/unixhome/rechenbu/output/pypy-1.4/lib-python/modified-2.5.2/ctype s/__init__.py", line 363, in __getattr__ func = self.__getitem__(name) File "/home/unixhome/rechenbu/output/pypy-1.4/lib-python/modified-2.5.2/ctype s/__init__.py", line 368, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) File "/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/_ctypes/function.py", line 112, in __init__ ptr = self._getfuncptr([], ctypes.c_int) File "/home/unixhome/rechenbu/output/pypy-1.4/lib_pypy/_ctypes/function.py", line 197, in _getfuncptr return cdll.ptr(self.name, argshapes, resshape, self._flags_) AttributeError: 'NoneType' object has no attribute 'ptr' Someone knows more about it? Regards, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101216/1679905d/attachment.htm From arigo at tunes.org Thu Dec 16 16:07:35 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 16 Dec 2010 16:07:35 +0100 Subject: [pypy-dev] error from function.py In-Reply-To: References: Message-ID: Hi, On Thu, Dec 16, 2010 at 1:35 PM, Arnd Rechenburg wrote: > ??? return cdll.ptr(self.name, argshapes, resshape, self._flags_) > > AttributeError: 'NoneType' object has no attribute 'ptr' That's a very indirect way of crashing because ctypes.util.find_library('c') returned None, i.e. it did not find your C library. I agree that this could be improved by at least reporting a clearer error. To understand the error, first note that it occurs in standard code from CPython's "ctypes.util" module. Look in lib-python/modified-2.5.2/ctypes/util.py. On Linux it seems to call "/sbin/ldconfig -p" and look in the result for "/libc.*". On my machine it would first find "/lib/libc.so.6" in this way, which is correct. In general it all looks pretty hackish, but it's not clear if any particular piece of code is to blame, given the way that the system of libraries is designed, which is not really adapted for fully dynamic lookups like needed here... A bient?t, Armin. From fijall at gmail.com Thu Dec 16 20:22:27 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 16 Dec 2010 21:22:27 +0200 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> Message-ID: On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen wrote: > On 16 Dec 2010, at 10:33, Antonio Cuni wrote: > >> Alternatively, if you install mercurial you can then "import >> mercurial.commands" and use its public API from Python. Or as you said you >> can >> just execute hg log and parse the output: in this case you might be >> interested >> in the --template option, which allows you to format things exactly as you >> want, e.g.: > > > > I'd suggest parsing the command line output of Mercurial, as it's the > officially sanctioned stable API for Mercurial. The Python API, while > faster, provides no stability guarantees. That being said, the following > will create a repository instance and access the context for a particular > changeset: > >>>> from mercurial import hg, ui >>>> >>>> repo = hg.repository(ui.ui(), 'path/to/repo') >>>> ctx = repo['tip'] >>>> log = ctx.description() > > The typical accessors for a changeset context are defined at > . > > Please note that any use of the Python API means that the entire application > is covered by the GPL. > How is it even technically possible? It does not link against mercurial and GPL specifically excludes anything about *running* your software. From fijall at gmail.com Thu Dec 16 20:20:46 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 16 Dec 2010 21:20:46 +0200 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: On Thu, Dec 16, 2010 at 11:16 AM, Armin Rigo wrote: > Hi, > > On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield wrote: >>> Is pypy suitable for writing interpretor of vectorized language like >>> Matlab, R etc which vector and matrix are first class objects? This includes >>> matrix shape inference, and efficient linear algebra code generation. >> >> have you seen numpy/scipy? > > The first aspect is simply if RPython would be suitable for writing an > interpreter for, say, Matlab. ?The answer is "probably yes": PyPy > would be suitable for such dynamic languages, giving you a JIT > compiler for free. ?I don't really know how complex the core of these > languages are, but I suspect not too much. > > Of course you are then going to hit the same problems that Ademan > tries to solve for numpy/scipy, notably how to implement at least the > basic linear algebra operations in such a way that the JIT can improve > them. ?There are various goals there, e.g. to turn Python (or Matlab) > code like A+B+C, adding three matrices together, into one matrix > operation instead of two (as it is now: (A+B)+C). ?This is all a bit > experimental so far. > > > A bient?t, > > Armin. Regarding this - I was thinking about haveing a + b - c create a bytecode that would be executed using small interpreter with a jit-merge-point and a hint "can be vectorized". From alex.gaynor at gmail.com Thu Dec 16 20:27:18 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 16 Dec 2010 14:27:18 -0500 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> Message-ID: On Thu, Dec 16, 2010 at 2:22 PM, Maciej Fijalkowski wrote: > On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen > wrote: > > On 16 Dec 2010, at 10:33, Antonio Cuni wrote: > > > >> Alternatively, if you install mercurial you can then "import > >> mercurial.commands" and use its public API from Python. Or as you said > you > >> can > >> just execute hg log and parse the output: in this case you might be > >> interested > >> in the --template option, which allows you to format things exactly as > you > >> want, e.g.: > > > > > > > > I'd suggest parsing the command line output of Mercurial, as it's the > > officially sanctioned stable API for Mercurial. The Python API, while > > faster, provides no stability guarantees. That being said, the following > > will create a repository instance and access the context for a particular > > changeset: > > > >>>> from mercurial import hg, ui > >>>> > >>>> repo = hg.repository(ui.ui(), 'path/to/repo') > >>>> ctx = repo['tip'] > >>>> log = ctx.description() > > > > The typical accessors for a changeset context are defined at > > < > http://hg.intevation.org/mercurial/file/7397a53219c9/mercurial/context.py#l97 > >. > > > > Please note that any use of the Python API means that the entire > application > > is covered by the GPL. > > > > How is it even technically possible? It does not link against > mercurial and GPL specifically excludes anything about *running* your > software. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Because the term "link" is totally meaningless in the context of Python code. Some people say this means it's impossible for Python to invoke that clause of the GPL, others say it means `import`ing something invokes it. There's no correct answer. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101216/ccb3206a/attachment-0001.htm From alex.gaynor at gmail.com Thu Dec 16 20:28:08 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 16 Dec 2010 14:28:08 -0500 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: On Thu, Dec 16, 2010 at 2:20 PM, Maciej Fijalkowski wrote: > On Thu, Dec 16, 2010 at 11:16 AM, Armin Rigo wrote: > > Hi, > > > > On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield wrote: > >>> Is pypy suitable for writing interpretor of vectorized language like > >>> Matlab, R etc which vector and matrix are first class objects? This > includes > >>> matrix shape inference, and efficient linear algebra code generation. > >> > >> have you seen numpy/scipy? > > > > The first aspect is simply if RPython would be suitable for writing an > > interpreter for, say, Matlab. The answer is "probably yes": PyPy > > would be suitable for such dynamic languages, giving you a JIT > > compiler for free. I don't really know how complex the core of these > > languages are, but I suspect not too much. > > > > Of course you are then going to hit the same problems that Ademan > > tries to solve for numpy/scipy, notably how to implement at least the > > basic linear algebra operations in such a way that the JIT can improve > > them. There are various goals there, e.g. to turn Python (or Matlab) > > code like A+B+C, adding three matrices together, into one matrix > > operation instead of two (as it is now: (A+B)+C). This is all a bit > > experimental so far. > > > > > > A bient?t, > > > > Armin. > > Regarding this - I was thinking about haveing a + b - c create a > bytecode that would be executed using small interpreter with a > jit-merge-point and a hint "can be vectorized". > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > That seems like a pretty big special case, why not work at the larger idea of cross-loop optimizations? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101216/a5a62eff/attachment.htm From danchr at gmail.com Thu Dec 16 21:05:05 2010 From: danchr at gmail.com (Dan Villiom Podlaski Christiansen) Date: Thu, 16 Dec 2010 21:05:05 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> Message-ID: <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> On 16 Dec 2010, at 20:27, Alex Gaynor wrote: > On Thu, Dec 16, 2010 at 2:22 PM, Maciej Fijalkowski > wrote: > >> On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen >> wrote: >> >>> Please note that any use of the Python API means that the entire >>> application is covered by the GPL. >> >> How is it even technically possible? It does not link against >> mercurial and GPL specifically excludes anything about *running* your >> software. > > Because the term "link" is totally meaningless in the context of > Python > code. Some people say this means it's impossible for Python to > invoke that > clause of the GPL, others say it means `import`ing something invokes > it. > There's no correct answer. Well, some people believe that any use of a GPL API constitutes a derivative work, and is thus covered by it. Mercurial even went through a licence change from GPLv2 to GPLv2+ to get rid of this ambiguity. See the links below; otherwise the convert extension (which is distributed with Mercurial) might be in violation of its own licence. Anyway, as long as your code is open, people are unlikely to care enough to do anything about it? -- Dan Villiom Podlaski Christiansen danchr at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1943 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101216/a6ccf788/attachment.bin From renesd at gmail.com Thu Dec 16 23:43:19 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 16 Dec 2010 22:43:19 +0000 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: On Thu, Dec 16, 2010 at 9:16 AM, Armin Rigo wrote: > Hi, > > On Wed, Dec 15, 2010 at 6:21 PM, Ren? Dudfield wrote: > >> Is pypy suitable for writing interpretor of vectorized language like > >> Matlab, R etc which vector and matrix are first class objects? This > includes > >> matrix shape inference, and efficient linear algebra code generation. > > > > have you seen numpy/scipy? > > The first aspect is simply if RPython would be suitable for writing an > interpreter for, say, Matlab. The answer is "probably yes": PyPy > would be suitable for such dynamic languages, giving you a JIT > compiler for free. I don't really know how complex the core of these > languages are, but I suspect not too much. > > Of course you are then going to hit the same problems that Ademan > tries to solve for numpy/scipy, notably how to implement at least the > basic linear algebra operations in such a way that the JIT can improve > them. There are various goals there, e.g. to turn Python (or Matlab) > code like A+B+C, adding three matrices together, into one matrix > operation instead of two (as it is now: (A+B)+C). This is all a bit > experimental so far. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Hi, Numexpr is really worth looking at: http://code.google.com/p/numexpr/ Especially as it combines expressions together, and does things like chunking (called tiling in the graphics world). >>> timeit a**2 + b**2 + 2*a*b 10 loops, best of 3: 35.9 ms per loop >>> ne.set_num_threads(1) # using 1 threads (on a 6-core machine) >>> timeit ne.evaluate("a**2 + b**2 + 2*a*b") 100 loops, best of 3: 9.28 ms per loop # 3.9x faster than NumPy >>> ne.set_num_threads(4) # using 4 threads (on a 6-core machine) >>> timeit ne.evaluate("a**2 + b**2 + 2*a*b") 100 loops, best of 3: 4.17 ms per loop # 8.6x faster than NumPy This approach, could be applied in the pypy jit I guess. For a vectorised xplatform run time assembler see: http://code.entropywave.com/git?p=orc.git;a=summary This allows you to write asm code in the one language, and have it compile to ARM, SSE, mmx, PPC, NEON, DSP, and even into C. Instructions that are not in one set, are emulated by multiple instructions. It's mainly used for video, sound and image processing - which are big arrays. Just some interesting vectorised thingies to look at. cya. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101216/9fc0908f/attachment.htm From p.giarrusso at gmail.com Fri Dec 17 00:12:10 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Fri, 17 Dec 2010 00:12:10 +0100 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: On Thu, Dec 16, 2010 at 10:16, Armin Rigo wrote: > Hi, >> have you seen numpy/scipy? > Of course you are then going to hit the same problems that Ademan > tries to solve for numpy/scipy, notably how to implement at least the > basic linear algebra operations in such a way that the JIT can improve > them. ?There are various goals there, e.g. to turn Python (or Matlab) > code like A+B+C, adding three matrices together, into one matrix > operation instead of two (as it is now: (A+B)+C). ?This is all a bit > experimental so far. [More about numpy/scipy]: How about the ideas from C++ expression templates? Or even better, some sort of lazy evaluation? With C++ expression templates [1], (A+B) returns something like PlusMatrix(A, B). When finally assigning to a variable, the copy constructor converts this into a real matrix; in the following line: D = (A+B) + C PlusMatrix(PlusMatrix(A, B), C) is built, and D gets the result. What they could further do, but they don't (see FAQ of uBlas [1]), is to cache the result of evaluating a template when requested; that could be cached (as in lazy evaluation - expressions like PlusMatrix are not very different from thunks, in that case) or maybe JIT compilation can recognize the constantness of this. Best regards [1] One library using them is Boost::uBlas, but the idea is much older: http://www.boost.org/doc/libs/1_45_0/libs/numeric/ublas/doc/index.htm -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From william.leslie.ttg at gmail.com Fri Dec 17 00:21:37 2010 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Fri, 17 Dec 2010 10:21:37 +1100 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: On 17 December 2010 09:43, Ren? Dudfield wrote: > [Numexpr] approach, could be applied in the pypy jit I guess. There was a lot of discussion in this direction during micronumpy experiments, but I'm not sure how well ideas were documented. A similar class of optimisation is polyhedral loop refactoring; implementing it within the framework of a tracing JIT would be new and exciting research if anyone is looking for a topic. -- William Leslie From wing1127aishi at gmail.com Fri Dec 17 02:08:00 2010 From: wing1127aishi at gmail.com (Leon Sit) Date: Thu, 16 Dec 2010 19:08:00 -0600 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: Actually.. my original idea is to parse matlab/r code and generate legal C++ code with this library http://eigen.tuxfamily.org/index.php?title=Main_Page#Documentation and use pypy to do some kind of type/shape inference on matrix object On Thu, Dec 16, 2010 at 5:21 PM, William ML Leslie < william.leslie.ttg at gmail.com> wrote: > On 17 December 2010 09:43, Ren? Dudfield wrote: > > [Numexpr] approach, could be applied in the pypy jit I guess. > > There was a lot of discussion in this direction during micronumpy > experiments, but I'm not sure how well ideas were documented. > > A similar class of optimisation is polyhedral loop refactoring; > implementing it within the framework of a tracing JIT would be new and > exciting research if anyone is looking for a topic. > > -- > William Leslie > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101216/96b428d6/attachment-0001.htm From fijall at gmail.com Fri Dec 17 07:27:31 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 17 Dec 2010 08:27:31 +0200 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 1:12 AM, Paolo Giarrusso wrote: > On Thu, Dec 16, 2010 at 10:16, Armin Rigo wrote: >> Hi, > >>> have you seen numpy/scipy? > >> Of course you are then going to hit the same problems that Ademan >> tries to solve for numpy/scipy, notably how to implement at least the >> basic linear algebra operations in such a way that the JIT can improve >> them. ?There are various goals there, e.g. to turn Python (or Matlab) >> code like A+B+C, adding three matrices together, into one matrix >> operation instead of two (as it is now: (A+B)+C). ?This is all a bit >> experimental so far. > > [More about numpy/scipy]: > How about the ideas from C++ expression templates? Or even better, > some sort of lazy evaluation? > > With C++ expression templates [1], (A+B) returns something like > PlusMatrix(A, B). When finally assigning to a variable, the copy > constructor converts this into a real matrix; in the following line: > D = (A+B) + C > PlusMatrix(PlusMatrix(A, B), C) is built, and D gets the result. > > What they could further do, but they don't (see FAQ of uBlas [1]), is > to cache the result of evaluating a template when requested; that > could be cached (as in lazy evaluation - expressions like PlusMatrix > are not very different from thunks, in that case) or maybe JIT > compilation can recognize the constantness of this. > > Best regards > > [1] One library using them is Boost::uBlas, but the idea is much older: > http://www.boost.org/doc/libs/1_45_0/libs/numeric/ublas/doc/index.htm > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev Hi. I did that (it's not published anywhere though) and it gave quite huge speedups compared to numexpr or numpy. I'll look into implementing it "properly" at some point (instead of doing that in pure python). From tobami at googlemail.com Fri Dec 17 09:27:26 2010 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 17 Dec 2010 09:27:26 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> Message-ID: Well, it wouldn't be a problem to re-license Codespeed as GPLv2+ anyway... 2010/12/16 Dan Villiom Podlaski Christiansen : > On 16 Dec 2010, at 20:27, Alex Gaynor wrote: > >> On Thu, Dec 16, 2010 at 2:22 PM, Maciej Fijalkowski >> wrote: >> >>> On Thu, Dec 16, 2010 at 12:14 PM, Dan Villiom Podlaski Christiansen >>> wrote: >>> >>>> Please note that any use of the Python API means that the entire >>>> application is covered by the GPL. >>> >>> How is it even technically possible? It does not link against >>> mercurial and GPL specifically excludes anything about *running* your >>> software. >> >> Because the term "link" is totally meaningless in the context of Python >> code. ?Some people say this means it's impossible for Python to invoke >> that >> clause of the GPL, others say it means `import`ing something invokes it. >> There's no correct answer. > > Well, some people believe that any use of a GPL API constitutes a derivative > work, and is thus covered by it. Mercurial even went through a licence > change from GPLv2 to GPLv2+ to get rid of this ambiguity. See the links > below; otherwise the convert extension (which is distributed with Mercurial) > might be in violation of its own licence. > > > > > > > > > Anyway, as long as your code is open, people are unlikely to care enough to > do anything about it? > > -- > > Dan Villiom Podlaski Christiansen > danchr at gmail.com > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Fri Dec 17 13:44:56 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 17 Dec 2010 13:44:56 +0100 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: Hi Alex, On Thu, Dec 16, 2010 at 8:28 PM, Alex Gaynor wrote: >> Regarding this - I was thinking about haveing a + b - c create a >> bytecode that would be executed using small interpreter with a >> jit-merge-point and a hint "can be vectorized". > > That seems like a pretty big special case, why not work at the larger idea > of cross-loop optimizations? We have cross-loop optimizations. Unless I'm misunderstanding you, we have already done it. That's how for example map(f, somelist) gets JITted: it is a small loop that contains a call to the "normal" loop of the interpreter for f(). This is JITted as the loop from map() containing inline code from the interpreter loop. I am a bit unsure however what is being discussed here, because fijal's comment is very terse and contains hidden issues, like when to create the bytecode (at runtime? but how can it be a green constant then? And what is the hint "can be vectorized"?). A bient?t, Armin. From exarkun at twistedmatrix.com Fri Dec 17 15:44:07 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 17 Dec 2010 14:44:07 -0000 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> Message-ID: <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain> On 08:27 am, tobami at googlemail.com wrote: >Well, it wouldn't be a problem to re-license Codespeed as GPLv2+ >anyway... >:( Jean-Paul From tobami at googlemail.com Fri Dec 17 15:48:59 2010 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 17 Dec 2010 15:48:59 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain> References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain> Message-ID: @Jean-Paul: I thought it would only be a problem if the license was GPLv3? But in any case, what is the conclusion, if any? Can the Mercurial API be used from LGPL code or not? 2010/12/17 : > On 08:27 am, tobami at googlemail.com wrote: >> >> Well, it wouldn't be a problem to re-license Codespeed as GPLv2+ anyway... > >> :( > > Jean-Paul > From exarkun at twistedmatrix.com Fri Dec 17 16:06:17 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 17 Dec 2010 15:06:17 -0000 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain> Message-ID: <20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain> On 02:48 pm, tobami at googlemail.com wrote: >@Jean-Paul: I thought it would only be a problem if the license was >GPLv3? Yes, that's right. But I got sad anyway. >But in any case, what is the conclusion, if any? Can the Mercurial API >be used from LGPL code or not? This seems unanswerable. If you believe some people, yes. If you believe other people, no. The only way to resolve the conflict is to do it, and either get sued by the copyright holder or not get sued by the copyright holder (you could argue that the outcome of the lawsuit must also be considered, but by the time you've gotten sued you probably don't care anymore). Jean-Paul From garyrob at me.com Fri Dec 17 17:08:58 2010 From: garyrob at me.com (Gary Robinson) Date: Fri, 17 Dec 2010 11:08:58 -0500 Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0 Message-ID: When I try to start PyPy, I get: bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory This is on a SuSE Linux Enterprise Server 10.3 SP2. This system does have: /lib/libbz2.so.1.0.0 As a quick experiment, I tried temporarily renaming it to /lib/libbz2.so.1.0 but that didn't change the result. I also tried installing the bzip2 package from the rpm, but it said it was already installed. Can anyone suggest a solution? -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net From arigo at tunes.org Fri Dec 17 18:16:29 2010 From: arigo at tunes.org (Armin Rigo) Date: Fri, 17 Dec 2010 18:16:29 +0100 Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0 In-Reply-To: References: Message-ID: Hi Gary, On Fri, Dec 17, 2010 at 5:08 PM, Gary Robinson wrote: > As a quick experiment, I tried temporarily renaming it to ? /lib/libbz2.so.1.0 but that didn't change the result. It's strange, what I would expect on a system is that there is a symlink from libbz2.so.1.0 to libbz2.so.1.0.0. Adding it should be enough -- did you run ldconfig after trying to rename? It is needed to refresh Linux's caches. A bient?t, Armin. From wlavrijsen at lbl.gov Fri Dec 17 18:32:11 2010 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 17 Dec 2010 09:32:11 -0800 (PST) Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0 In-Reply-To: References: Message-ID: Hi Gary, > bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open > shared object file: No such file or directory > > This is on a SuSE Linux Enterprise Server 10.3 SP2. > > This system does have: > > /lib/libbz2.so.1.0.0 > > As a quick experiment, I tried temporarily renaming it to /lib/libbz2.so.1.0 > but that didn't change the result. I also tried installing the bzip2 package > from the rpm, but it said it was already installed. > > Can anyone suggest a solution? Just guessing ... is this perhaps a 32/64b mix? I.e. are you running 64b pypy-c and is a libbz2 expected from /lib64? That's what bit me in a similar situation (albeit on a RH-clone linux install). Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From garyrob at me.com Fri Dec 17 18:36:18 2010 From: garyrob at me.com (Gary Robinson) Date: Fri, 17 Dec 2010 12:36:18 -0500 Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0 In-Reply-To: References: Message-ID: <597BA6F7-6079-4FDD-B033-26EC39F8AB3C@me.com> I did do ldconfig. I tried a symlink, copying the /lib/libbz2.so.1.0.0, and the temporary rename I mentioned. An example: dev01:~/pypy-1.4 # ln -s /lib/libbz2.so.1.0.0 /lib/libbz2.so.1.0 dev01:~/pypy-1.4 # ldconfig dev01:~/pypy-1.4 # bin/pypy bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory dev01:~/pypy-1.4 # ls -ld /lib/libbz* lrwxrwxrwx 1 root root 20 Dec 17 12:33 /lib/libbz2.so.1.0 -> /lib/libbz2.so.1.0.0 -rwxr-xr-x 1 root root 66864 Jun 16 2006 /lib/libbz2.so.1.0.0 I keep on thinking I must be doing something stupid, but it continues to elude me. (I also tried "ldconfig /lib" with no difference) -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net On Dec 17, 2010, at 12:16 PM, Armin Rigo wrote: > Hi Gary, > > On Fri, Dec 17, 2010 at 5:08 PM, Gary Robinson wrote: >> As a quick experiment, I tried temporarily renaming it to /lib/libbz2.so.1.0 but that didn't change the result. > > It's strange, what I would expect on a system is that there is a > symlink from libbz2.so.1.0 to libbz2.so.1.0.0. Adding it should be > enough -- did you run ldconfig after trying to rename? It is needed > to refresh Linux's caches. > > > A bient?t, > > Armin. From garyrob at me.com Fri Dec 17 20:28:06 2010 From: garyrob at me.com (Gary Robinson) Date: Fri, 17 Dec 2010 14:28:06 -0500 Subject: [pypy-dev] Can't run PyPy due to missing libbz2.so.1.0 In-Reply-To: References: Message-ID: > are you running 64b pypy-c > and is a libbz2 expected from /lib64 That was it! I added a symbolic link with ln -s /lib64/libbz2.so.1.0.0 /lib64/libbz2.so.1.0 and now it works. Many thanks for the help!!! -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net On Dec 17, 2010, at 12:32 PM, wlavrijsen at lbl.gov wrote: > Hi Gary, > >> bin/pypy: error while loading shared libraries: libbz2.so.1.0: cannot open >> shared object file: No such file or directory >> >> This is on a SuSE Linux Enterprise Server 10.3 SP2. >> >> This system does have: >> >> /lib/libbz2.so.1.0.0 >> >> As a quick experiment, I tried temporarily renaming it to /lib/libbz2.so.1.0 >> but that didn't change the result. I also tried installing the bzip2 package >> from the rpm, but it said it was already installed. >> >> Can anyone suggest a solution? > > Just guessing ... is this perhaps a 32/64b mix? I.e. are you running 64b pypy-c > and is a libbz2 expected from /lib64? That's what bit me in a similar situation > (albeit on a RH-clone linux install). > > Best regards, > Wim > -- > WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From p.giarrusso at gmail.com Sat Dec 18 10:11:14 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Sat, 18 Dec 2010 10:11:14 +0100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: <20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain> References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain> <20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain> Message-ID: On Fri, Dec 17, 2010 at 16:06, wrote: > On 02:48 pm, tobami at googlemail.com wrote: >>@Jean-Paul: ?I thought it would only be a problem if the license was >>GPLv3? > > Yes, that's right. ?But I got sad anyway. >>But in any case, what is the conclusion, if any? Can the Mercurial API >>be used from LGPL code or not? > > This seems unanswerable. ?If you believe some people, yes. ?If you > believe other people, no. ?The only way to resolve the conflict is to do > it, and either get sued by the copyright holder or not get sued by the > copyright holder (you could argue that the outcome of the lawsuit must > also be considered, but by the time you've gotten sued you probably > don't care anymore). Well, one point is that the matter is complex enough to make you suable. Moreover, the point that you're technically not linking seems extremely technical and specific to Python, and even to CPython - a JIT compiler links together the code, even more than it happens for C libraries. Therefore, explaining it to a judge would take an extremely expensive lawyer. However, would the Mercurial developers sue you (through some FSF-sponsorship or the like)? You can ask them if they're bothered by your license. If they're not, they might still change their mind, but at least they'll (likely) ask you to change license before suing you (so they save the money for the lawyer). Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From p.giarrusso at gmail.com Sat Dec 18 10:48:00 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Sat, 18 Dec 2010 10:48:00 +0100 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: Hi Armin, hi all, below I give my 2 cents, giving some background on advanced optimizations like cross-loop optimization and automatic vectorization, which the others are mentioning without reference. These optimizations are used for numerical code like the one discussed (and mostly specific to that) in compiled languages like C/C++/Fortran. On Fri, Dec 17, 2010 at 13:44, Armin Rigo wrote: > Hi Alex, > > On Thu, Dec 16, 2010 at 8:28 PM, Alex Gaynor wrote: >>> Regarding this - I was thinking about haveing a + b - c create a >>> bytecode that would be executed using small interpreter with a >>> jit-merge-point and a hint "can be vectorized". >> >> That seems like a pretty big special case, why not work at the larger idea >> of cross-loop optimizations? > > We have cross-loop optimizations. ?Unless I'm misunderstanding you, we > have already done it. ?That's how for example map(f, somelist) gets > JITted: it is a small loop that contains a call to the "normal" loop > of the interpreter for f(). ?This is JITted as the loop from map() > containing inline code from the interpreter loop. Cross-loop optimization are useful to optimize C/Fortran numerical code. For code like a + b - c, you want your compiler to optimize together different loops from the application (the loop implementing a+b and the one subtracting c from the result). For instance, if a, b, c are vectors, a + b - c is equivalent (after inlining) to the following code, which should be transformed into a single loop, to reduce loop overhead but more importantly to perform only one pass on d and thus improve cache locality for huge vectors, not entirely fitting in CPU caches: // a, b, c, d are vectors for i in range(1, length): d[i] = a[i] + b[i] for i in range(1, length): d[i] = d[i] - c[i] optimized result: for i in range(1, length): d[i] = a[i] + b[i] + c[i] > I am a bit unsure however what is being discussed here, because > fijal's comment is very terse and contains hidden issues, like when to > create the bytecode (at runtime? but how can it be a green constant > then? I think he wants to do something similar to regexp compilation + JITting as described on PyPy's blog. One can compile a regexp to bytecode (at runtime), but afterwards the result is constant for the regexp interpreter. In this case, I read the statement as fijal wants to compile not a regexp but the equivalent of a C++ expression template (see my other email): for a + b - c, you'd get Minus(Plus(a, b), c), and hopefully compile it directly to the efficient code above. > And what is the hint "can be vectorized"?). I can explain what "vectorized" means (it is a specific loop optimization), but I'm not sure about why you want the annotation. A reasonable possibility is to run this optimization pass only on the code with such an annotation, because most code doesn't need it. Most likely in this context, he refers to (automatic) loop vectorization, i.e., automatically converting the code to use SIMD instructions. On current processor, the following matrix code can be implemented in one SIMD sum instruction, but the compiler has to figure it out: for i in range(1, 4): c[i] = a[i] + b[i] In general, a SIMD operation operates in parallel on two short operands of vectors, rather than on two single operands. The Intel C/Fortran compilers, and more recently also GCC, can perform this rewriting without annotations from the user. See for instance: http://en.wikipedia.org/wiki/Vectorization_(parallel_computing) Introduction of: http://software.intel.com/en-us/articles/vectorization-with-the-intel-compilers-part-i/ -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From hyarion at iinet.net.au Sat Dec 18 12:10:11 2010 From: hyarion at iinet.net.au (Ben) Date: Sat, 18 Dec 2010 22:10:11 +1100 Subject: [pypy-dev] speed.pypy.org and mercurial In-Reply-To: References: <4D09BEEC.8030204@gmail.com> <4D09DCD5.3000602@gmail.com> <0B113720-AE3B-4D75-B95D-AC5A7EFE3E6D@gmail.com> <7DC65AE8-B87C-45BC-B078-754D2580020E@gmail.com> <20101217144407.2058.1688725261.divmod.xquotient.283@localhost.localdomain> <20101217150617.2058.866210621.divmod.xquotient.290@localhost.localdomain> Message-ID: <4D0C9693.2030307@iinet.net.au> On 18/12/10 8:11 PM, Paolo Giarrusso wrote: > On Fri, Dec 17, 2010 at 16:06, wrote: >> On 02:48 pm,tobami at googlemail.com wrote: >>> @Jean-Paul: I thought it would only be a problem if the license was >>> GPLv3? >> Yes, that's right. But I got sad anyway. >>> But in any case, what is the conclusion, if any? Can the Mercurial API >>> be used from LGPL code or not? >> This seems unanswerable. If you believe some people, yes. If you >> believe other people, no. The only way to resolve the conflict is to do >> it, and either get sued by the copyright holder or not get sued by the >> copyright holder (you could argue that the outcome of the lawsuit must >> also be considered, but by the time you've gotten sued you probably >> don't care anymore). > Well, one point is that the matter is complex enough to make you > suable. Moreover, the point that you're technically not linking seems > extremely technical and specific to Python, and even to CPython - a > JIT compiler links together the code, even more than it happens for C > libraries. Therefore, explaining it to a judge would take an extremely > expensive lawyer. The GPL doesn't mention linking at all. It applies to "the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language". Linking is only relevant because traditional static linking causes parts of the library to be included in compiled binaries, which automatically makes them derivative works. Downstream recipients of your code are also perfectly within their rights to combine GPL code with code released under an incompatible license; provided they never distribute such code, the GPL imposes no restrictions. So the fact that the JIT compiler links code together on the end user's system is not relevant. The GPL terms only come into play if the CodeSpeed source can be considered a "derivative work under copyright law" of Mercurial. If CodeSpeed is designed to be support several version control systems via independently installable modules, then I don't see how it could be considered a derivative work of Mercurial. The integration module might be, but to accept that the existance of such a module makes CodeSpeed a derivative work means that CodeSpeed is also a derivative work of every version control system, or at least will be as soon as a module is released for each system (which could be done by other people). I find that rather ludicrous. That said, I'm not a lawyer, and I'm applying common sense reasoning here, not knowledge of how the legal system actually treats things. -- Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101218/d5e1e1a7/attachment.htm From wing1127aishi at gmail.com Sat Dec 18 17:14:08 2010 From: wing1127aishi at gmail.com (Leon Sit) Date: Sat, 18 Dec 2010 10:14:08 -0600 Subject: [pypy-dev] Interpretor for vectorized langugage In-Reply-To: References: Message-ID: My original intent is very much describe by Paolo. I want to use the idea of template expression in c++ to apply to linear algebra code. The whole idea of template expression is that all the overloaded operation does not trigger immediate evaluation of the expression, but to return an expression instead to next operator call. After chaining the entire formula, then the loop unrolling starts to happen when caller invokes the operator[]. Sample def fun(a,b,c,d): //say they are all matrix, and the formula might not make sense but for sake of argument only return a*a+b*c*b.T/ norm(d.T*d) into low level C++ expr& fun(const expr& a, const expr& b, const expr&c, const expr& d){ return sum(prod(a*a), prod(b, prod(c, transpose(b))))/ norm(prod(transpose(d), d)); } note that the type is "expression" in both parameters and input. The reason is twofold 1) no eager evaluation on the subexpression, only evaluate the expression when the entire expression is known 2) support lazy evaluation on the expression like fun(x+y,y*y+y,x**y,y)[a,b] where only the necessary computation for the entry [a,b] are calculated. For better demo on this idea, say I have def sum3(a,b,c): return a+b+c then sum3(x+y, y^2, 2*y)[1,10] should only trigger the evaluation of return (x[1,10] + y[1,10]) + y[1,10]^2 + 2 * y[1, 10] which itself can be optimised further and nothing else should be evaluated. boost::ublas is rather a slow implementation of template expression idea and the Eigen library has way more mature and efficient implementation. Without this technique, numerical C library was never as fast as hand coded C and Fortran code because hidden temporaries were everywhere in the formula. Only after this template expressin technique, C++ matrix library are on par with fortran and C. When the expression is long enough, C++ template expression library gives much better than Lapack/atlas code because the compiler will optimise the full expression. I hand port an optimisation library from homebrew matrix library to ublas and get 20x speed up, then moving to Eigen library get another 8x speed from ublas (without using idiomatic Eigen style, only switch the type). Main reason why Eigen is so much faster is that when unrolling the loop in Eigen, SSE code are explicitly generated instead of depending compiler's auto vectorisation which is not mature and only happen to simple code, openMP are applied to matrix multiplication at the most outer loop, and data are aligned in memory for less cache misses. On Sat, Dec 18, 2010 at 3:48 AM, Paolo Giarrusso wrote: > Hi Armin, hi all, > below I give my 2 cents, giving some background on advanced > optimizations like cross-loop optimization and automatic > vectorization, which the others are mentioning without reference. > These optimizations are used for numerical code like the one discussed > (and mostly specific to that) in compiled languages like > C/C++/Fortran. > > On Fri, Dec 17, 2010 at 13:44, Armin Rigo wrote: > > Hi Alex, > > > > On Thu, Dec 16, 2010 at 8:28 PM, Alex Gaynor > wrote: > >>> Regarding this - I was thinking about haveing a + b - c create a > >>> bytecode that would be executed using small interpreter with a > >>> jit-merge-point and a hint "can be vectorized". > >> > >> That seems like a pretty big special case, why not work at the larger > idea > >> of cross-loop optimizations? > > > > We have cross-loop optimizations. Unless I'm misunderstanding you, we > > have already done it. That's how for example map(f, somelist) gets > > JITted: it is a small loop that contains a call to the "normal" loop > > of the interpreter for f(). This is JITted as the loop from map() > > containing inline code from the interpreter loop. > > Cross-loop optimization are useful to optimize C/Fortran numerical > code. For code like a + b - c, you want your compiler to optimize > together different loops from the application (the loop implementing > a+b and the one subtracting c from the result). > For instance, if a, b, c are vectors, a + b - c is equivalent (after > inlining) to the following code, which should be transformed into a > single loop, to reduce loop overhead but more importantly to perform > only one pass on d and thus improve cache locality for huge vectors, > not entirely fitting in CPU caches: > > // a, b, c, d are vectors > for i in range(1, length): > d[i] = a[i] + b[i] > for i in range(1, length): > d[i] = d[i] - c[i] > > optimized result: > > for i in range(1, length): > d[i] = a[i] + b[i] + c[i] > > > I am a bit unsure however what is being discussed here, because > > fijal's comment is very terse and contains hidden issues, like when to > > create the bytecode (at runtime? but how can it be a green constant > > then? > > I think he wants to do something similar to regexp compilation + > JITting as described on PyPy's blog. > One can compile a regexp to bytecode (at runtime), but afterwards the > result is constant for the regexp interpreter. > > In this case, I read the statement as fijal wants to compile not a > regexp but the equivalent of a C++ expression template (see my other > email): for a + b - c, you'd get Minus(Plus(a, b), c), and hopefully > compile it directly to the efficient code above. > > > And what is the hint "can be vectorized"?). > > I can explain what "vectorized" means (it is a specific loop > optimization), but I'm not sure about why you want the annotation. A > reasonable possibility is to run this optimization pass only on the > code with such an annotation, because most code doesn't need it. > > Most likely in this context, he refers to (automatic) loop > vectorization, i.e., automatically converting the code to use SIMD > instructions. On current processor, the following matrix code can be > implemented in one SIMD sum instruction, but the compiler has to > figure it out: > > for i in range(1, 4): > c[i] = a[i] + b[i] > > In general, a SIMD operation operates in parallel on two short > operands of vectors, rather than on two single operands. The Intel > C/Fortran compilers, and more recently also GCC, can perform this > rewriting without annotations from the user. > > See for instance: > http://en.wikipedia.org/wiki/Vectorization_(parallel_computing) > Introduction of: > > http://software.intel.com/en-us/articles/vectorization-with-the-intel-compilers-part-i/ > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101218/e4a4e31b/attachment-0001.htm From garyrob at me.com Sat Dec 18 23:58:15 2010 From: garyrob at me.com (Gary Robinson) Date: Sat, 18 Dec 2010 17:58:15 -0500 Subject: [pypy-dev] playing with fast-forward Message-ID: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com> I'm experimenting with the fast-forward branch. I'm actually not sure about the proper way to get it. (I have the main branch working fine.) I downloaded a nojit version from http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any jit versions listed there. I was happy to see that the method I care most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to test it with the jit. The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for the branch, but the mercurial URL shown on that page appears to be for the main project. I tried the downloads under the Source pop-up menu. For some reason, I only sporadically am able to get a complete .gz file.** But I did get one. I was able to run: python translate.py -Ojit successfully -- or at least it appeared so. But I couldn't find a bin/pypy to run!? Any suggestions how to run fast-forward with jit? ** I've used text browsers w3m and elinks on SuSE, and Safari on OS X (two different machines in different cities). Twice, the Safari download has stopped at < 500K. (It LOOKS like a successful download, it's just incomplete.) The SuSE browsers usually get 16 or 18MB and stop, and tar gets "gzip: stdin: unexpected end of file" when I try to uncompress them. -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net From alex.gaynor at gmail.com Sun Dec 19 00:00:32 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 18 Dec 2010 18:00:32 -0500 Subject: [pypy-dev] playing with fast-forward In-Reply-To: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com> References: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com> Message-ID: On Sat, Dec 18, 2010 at 5:58 PM, Gary Robinson wrote: > I'm experimenting with the fast-forward branch. I'm actually not sure about > the proper way to get it. (I have the main branch working fine.) > > I downloaded a nojit version from > http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see > any jit versions listed there. I was happy to see that the method I care > most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want > to test it with the jit. > > The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be > for the branch, but the mercurial URL shown on that page appears to be for > the main project. > > I tried the downloads under the Source pop-up menu. For some reason, I only > sporadically am able to get a complete .gz file.** But I did get one. > > I was able to run: > > python translate.py -Ojit > > successfully -- or at least it appeared so. But I couldn't find a bin/pypy > to run!? > > Any suggestions how to run fast-forward with jit? > > > ** I've used text browsers w3m and elinks on SuSE, and Safari on OS X (two > different machines in different cities). Twice, the Safari download has > stopped at < 500K. (It LOOKS like a successful download, it's just > incomplete.) The SuSE browsers usually get 16 or 18MB and stop, and tar gets > "gzip: stdin: unexpected end of file" when I try to uncompress them. > > > -- > > Gary Robinson > CTO > Emergent Discovery, LLC > personal email: garyrob at me.com > work email: grobinson at emergentdiscovery.com > Company: http://www.emergentdiscovery.com > Blog: http://www.garyrobinson.net > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > a) If you want to work with the code it's probably easier to just use a mercurial checkout, rather than downloading a tarball. b) non-release builds (aka what you built) don't generate bin/pypy, it generates pypy-c in the directory you invoked the translator from. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101218/ad302245/attachment.htm From anto.cuni at gmail.com Sun Dec 19 12:35:05 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 19 Dec 2010 12:35:05 +0100 Subject: [pypy-dev] playing with fast-forward In-Reply-To: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com> References: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com> Message-ID: <4D0DEDE9.8050300@gmail.com> On 18/12/10 23:58, Gary Robinson wrote: > I'm experimenting with the fast-forward branch. I'm actually not sure about the proper way to get it. (I have the main branch working fine.) > > I downloaded a nojit version from http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any jit versions listed there. I was happy to see that the method I care most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to test it with the jit. consider that we don't have automatic nightly builds for fast-forward, so the ones you find on that page are manually triggered, and possibly outdated. > The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for the branch, but the mercurial URL shown on that page appears to be for the main project. yes, if you clone the mercurial repo you get all the branches together. > I tried the downloads under the Source pop-up menu. For some reason, I only sporadically am able to get a complete .gz file.** But I did get one. > > I was able to run: > > python translate.py -Ojit > > successfully -- or at least it appeared so. But I couldn't find a bin/pypy to run!? as suggested by Alex, you probably have a pypy-c binary in pypy/translator/goal (I think that the translate script even says so, but I agree that it produces so much output that it might be hard to stop the message :-)) > Any suggestions how to run fast-forward with jit? I don't know how the "download source" button of bitbucket works, so I'm not even sure that you downloaded the fast-forward branch instead of the default one. The best way is to do this: $ hg clone http://bitbucket.org/pypy/pypy $ cd pypy $ hg up -r fast-forward Now you can go to pypy/translator/goal and run translate.py again. ciao, Anto From Arnd.Rechenburg at tomtom.com Mon Dec 20 19:16:26 2010 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Mon, 20 Dec 2010 19:16:26 +0100 Subject: [pypy-dev] ImportError: No module named _sqlite3 Message-ID: Hi, I tried to use sqlite3 with pypy but I got the following error: >>>> import sqlite Traceback (most recent call last): File "", line 1, in ImportError: No module named sqlite >>>> import sqlite3 Traceback (most recent call last): File "", line 1, in File "sqlite3/__init__.py", line 24, in from dbapi2 import * File "sqlite3/dbapi2.py", line 27, in from _sqlite3 import * ImportError: No module named _sqlite3 Someone knows where to put the _sqlite3.so library? Any help would be appreciated. Thx in advance, Arnd Rechenburg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101220/c144aa13/attachment.htm From garyrob at me.com Mon Dec 20 20:42:21 2010 From: garyrob at me.com (Gary Robinson) Date: Mon, 20 Dec 2010 14:42:21 -0500 Subject: [pypy-dev] playing with fast-forward In-Reply-To: <4D0DEDE9.8050300@gmail.com> References: <27BF7E8E-4504-491A-86D5-01CCF8B37260@me.com> <4D0DEDE9.8050300@gmail.com> Message-ID: <5388F994-B794-43F5-B9DE-3AE417FB2090@me.com> Many thanks for this detailed help. Since I hadn't used Mercurial before, the notes about Mercurial were particularly helpful. I did some testing on fast-forward, and it looks like even now, it basically has what I need for serious use. multiprocessing.Pool.imap_unordered() is the method I tend to use all the time for spreading a task across processors, and it works. (At least it has so far in my testing.) And of course 64-bit was implemented a couple of months ago, which I need because of the size of the data structures I often use. So this is really great news. The main thing I'm looking for now is SciPy support, but I know that's a ways off, and I can do a lot without it. Thanks again to everyone for the great work. -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net On Dec 19, 2010, at 6:35 AM, Antonio Cuni wrote: > On 18/12/10 23:58, Gary Robinson wrote: >> I'm experimenting with the fast-forward branch. I'm actually not sure about the proper way to get it. (I have the main branch working fine.) >> >> I downloaded a nojit version from http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any jit versions listed there. I was happy to see that the method I care most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to test it with the jit. > > consider that we don't have automatic nightly builds for fast-forward, so the > ones you find on that page are manually triggered, and possibly outdated. > >> The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for the branch, but the mercurial URL shown on that page appears to be for the main project. > > yes, if you clone the mercurial repo you get all the branches together. > >> I tried the downloads under the Source pop-up menu. For some reason, I only sporadically am able to get a complete .gz file.** But I did get one. >> >> I was able to run: >> >> python translate.py -Ojit >> >> successfully -- or at least it appeared so. But I couldn't find a bin/pypy to run!? > > as suggested by Alex, you probably have a pypy-c binary in > pypy/translator/goal (I think that the translate script even says so, but I > agree that it produces so much output that it might be hard to stop the > message :-)) > >> Any suggestions how to run fast-forward with jit? > > I don't know how the "download source" button of bitbucket works, so I'm not > even sure that you downloaded the fast-forward branch instead of the default one. > > The best way is to do this: > > $ hg clone http://bitbucket.org/pypy/pypy > $ cd pypy > $ hg up -r fast-forward > > Now you can go to pypy/translator/goal and run translate.py again. > > ciao, > Anto From dimaqq at gmail.com Tue Dec 21 01:59:50 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 20 Dec 2010 17:59:50 -0700 Subject: [pypy-dev] funding/popularity? Message-ID: First of all, congratulation on getting Eurostars Would be nice if companies that use Python extensively chipped in [more]. By [out of] my [@%%] calculations, each percent of perfromance improvement that pypy achieves potentially saves google many millions a year in data centre costs. What are other companies use/love python and how much? Popularity, I for one would like python to be used much more widely; it is of course a double-edges sword, on one hand more users and developers means more contributors, libraries and understanding that python is important, in terms of research, funding, etc; on the other more bad code and mess. Anyhow I see at least these problems for now: in general: 2.x/3.x dichtomy; some ext's are not ported, pypy doesn't yet run 3k (I'd be willing to work on this btw), lack of authoritative information on the subject; Since pypy is a forward-looking project, I think a push to include 3.2 (and perhaps 2.7 which would come naturally) would be very good here, at least clear statement that pypy is going there. More visibility for performance achievements would do good too. in classic apps: hard to bundle python with downloadable apps, e.g. win builds still recommend 2.5; perhaps stackless single-exe builds are the best here? A bit better on linux where dependency tracking is the responsibility of the disrto. on the web: bad support in apache, little advice for users what to use and how frameworks are structured; I reckon some pypy side projects could be great here, e.g. isolated programs within same interpreter or multiple interpreters in same process. in schools/unis: I really can't comprehend why! Anyhow I think 1.4 release is great! Though I must contend I am still to try it with most of my code, too many ext deps and very little of my python code is cpu-bound. Sidetracking... one day when pypy jit/gc/etc are all worked out, how hard would it be to use same techniques and most of backends for some unrelated language that doesn't have jit yet, e.g. php? And how hard would it be to marry two dynamic languages, so that modules from one could be used in the other? Or that modules written in rpython could be used in several langs? From william.leslie.ttg at gmail.com Tue Dec 21 03:21:49 2010 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 21 Dec 2010 13:21:49 +1100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: On 21 December 2010 11:59, Dima Tisnek wrote: > More visibility for performance achievements would do > good too. Where are pypy's performance achievements *not* visible, but should be? > Sidetracking... one day when pypy jit/gc/etc are all worked out, how > hard would it be to use same techniques and most of backends for some > unrelated language that doesn't have jit yet, e.g. php? You know that pypy already has implementations of other languages, right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't integrated with the translated pypy-c, but they show that it is not too difficult to write a runtime for any dynamic language you choose. > And how hard > would it be to marry two dynamic languages, so that modules from one > could be used in the other? Or that modules written in rpython could > be used in several langs? It's in the "interesting problems" bucket, and the effort required depends on the level of integration between languages you want. There are several projects already attempting to do decent integration between several languages, besides the approach used on the JVM, there are also GNU Guile, Racket, and Parrot, among others. It might be worth waiting to see how these different projects pan out, before spending a bunch of effort just to be an also-ran in the multi-language runtime market. However, implementing more languages in rpython has the side-effect of propagating the l * o * p problem: it introduces more and more implementations that then have to be maintained, so good cross-language integration probably belongs /outside/ pypy itself, so existing runtimes can hook into it. But it would be an interesting experiment (to marry the various interpreters pypy ships with), if you wanted to try it. My two cents. -- William Leslie From dimaqq at gmail.com Tue Dec 21 07:57:52 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 20 Dec 2010 23:57:52 -0700 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: On 20 December 2010 19:21, William ML Leslie wrote: > On 21 December 2010 11:59, Dima Tisnek wrote: >> More visibility for performance achievements would do >> good too. > > Where are pypy's performance achievements *not* visible, but should be? for example http://shootout.alioth.debian.org/ doesn't say which pypy version is used, what options, doesn't have performance figures for multithreaded/multicore also benchmarks are kinda small, most of them are not docuemented, and I couldn't find any info if the same python code was used for cpython and pypy (both shootout and speed pypy) or interpreter-specific verions were used, that is each tweaked for best performance given the known tradeoffs for each implementation.further the most standard benchmarks, pystone, etc. completely ignore the fact that real world programs are large and only a few small paths are execured often. another temp problem with speed pypy is that it's terrubly slow in ff beta. it also occasionally grinds in stable ff and opera, but I guess this can be forgiven for the sake of simplicity / developer effort. if you google for 'python performance' you don't get a single link to pypy on the first page, as a matter of fact, codespeak is poorly indexed, it took me quite some time to get some of my questions answered with a search. also if you look up 'pypy gc' you get a page on codespeak, but to interpret what the data actually means is so far beyond me. a good overview is found in the mainling list http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again slowspitfire and spambayes bits are outdated by now. the definitive good thing about pypy is that it's much easier to find out about its inner workings than that of cpython! hopefully a bit more of end-user stuff get known. let's call it pypy public outreach (feature request) > >> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >> hard would it be to use same techniques and most of backends for some >> unrelated language that doesn't have jit yet, e.g. php? > > You know that pypy already has implementations of other languages, > right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't > integrated with the translated pypy-c, but they show that it is not > too difficult to write a runtime for any dynamic language you choose. Oh I didn't know there were so many, and I mistakenly though that js was a target, not implmented langauge. In any case I read somewhere that js support was retired... > >> And how hard >> would it be to marry two dynamic languages, so that modules from one >> could be used in the other? Or that modules written in rpython could >> be used in several langs? > > It's in the "interesting problems" bucket, and the effort required > depends on the level of integration between languages you want. ?There > are several projects already attempting to do decent integration > between several languages, besides the approach used on the JVM, there > are also GNU Guile, Racket, and Parrot, among others. ?It might be > worth waiting to see how these different projects pan out, before > spending a bunch of effort just to be an also-ran in the > multi-language runtime market. > > However, implementing more languages in rpython has the side-effect of > propagating the l * o * p problem: it introduces more and more > implementations that then have to be maintained, so good > cross-language integration probably belongs /outside/ pypy itself, so > existing runtimes can hook into it. Makes perfect sense, after all any given other language hardly has the same data model as python. > > But it would be an interesting experiment (to marry the various > interpreters pypy ships with), if you wanted to try it. > > My two cents. > > -- > William Leslie > From tobami at googlemail.com Tue Dec 21 09:06:22 2010 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 21 Dec 2010 09:06:22 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Hi Dima, > another temp problem with speed pypy is that it's terrubly slow in ff > beta. it also occasionally grinds in stable ff and opera, but I guess > this can be forgiven for the sake of simplicity / developer effort. Well, speed.pypy is actually fast in all modern browsers. The problem you are referring to is probably caused by a bug in the javascript plotting library (jqPplot) that is triggered in the comparison view when there are some results with 0 values. It only appears for some plot types, but it is very annoying because it grinds the browser to a halt like you say. Is that what you meant? We are looking into it, and will fix that library if necessary. Cheers, Miquel 2010/12/21 Dima Tisnek : > On 20 December 2010 19:21, William ML Leslie > wrote: >> On 21 December 2010 11:59, Dima Tisnek wrote: >>> More visibility for performance achievements would do >>> good too. >> >> Where are pypy's performance achievements *not* visible, but should be? > > for example http://shootout.alioth.debian.org/ > doesn't say which pypy version is used, what options, doesn't have > performance figures for multithreaded/multicore > > also benchmarks are kinda small, most of them are not docuemented, and > I couldn't find any info if the same python code was used for cpython > and pypy (both shootout and speed pypy) or interpreter-specific > verions were used, that is each tweaked for best performance given the > known tradeoffs for each implementation.further the most standard > benchmarks, pystone, etc. completely ignore the fact that real world > programs are large and only a few small paths are execured often. > > another temp problem with speed pypy is that it's terrubly slow in ff > beta. it also occasionally grinds in stable ff and opera, but I guess > this can be forgiven for the sake of simplicity / developer effort. > > if you google for 'python performance' you don't get a single link to > pypy on the first page, as a matter of fact, codespeak is poorly > indexed, it took me quite some time to get some of my questions > answered with a search. also if you look up 'pypy gc' you get a page > on codespeak, but to interpret what the data actually means is so far > beyond me. > > a good overview is found in the mainling list > http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again > slowspitfire and spambayes bits are outdated by now. > > the definitive good thing about pypy is that it's much easier to find > out about its inner workings than that of cpython! > > hopefully a bit more of end-user stuff get known. > let's call it pypy public outreach (feature request) > >> >>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>> hard would it be to use same techniques and most of backends for some >>> unrelated language that doesn't have jit yet, e.g. php? >> >> You know that pypy already has implementations of other languages, >> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >> integrated with the translated pypy-c, but they show that it is not >> too difficult to write a runtime for any dynamic language you choose. > > Oh I didn't know there were so many, and I mistakenly though that js > was a target, not implmented langauge. In any case I read somewhere > that js support was retired... > >> >>> And how hard >>> would it be to marry two dynamic languages, so that modules from one >>> could be used in the other? Or that modules written in rpython could >>> be used in several langs? >> >> It's in the "interesting problems" bucket, and the effort required >> depends on the level of integration between languages you want. ?There >> are several projects already attempting to do decent integration >> between several languages, besides the approach used on the JVM, there >> are also GNU Guile, Racket, and Parrot, among others. ?It might be >> worth waiting to see how these different projects pan out, before >> spending a bunch of effort just to be an also-ran in the >> multi-language runtime market. >> >> However, implementing more languages in rpython has the side-effect of >> propagating the l * o * p problem: it introduces more and more >> implementations that then have to be maintained, so good >> cross-language integration probably belongs /outside/ pypy itself, so >> existing runtimes can hook into it. > > Makes perfect sense, after all any given other language hardly has the > same data model as python. > >> >> But it would be an interesting experiment (to marry the various >> interpreters pypy ships with), if you wanted to try it. >> >> My two cents. >> >> -- >> William Leslie >> > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From Arnd.Rechenburg at tomtom.com Tue Dec 21 09:18:47 2010 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Tue, 21 Dec 2010 09:18:47 +0100 Subject: [pypy-dev] ImportError: No module named _sqlite3 Message-ID: Hi, I saw my mistake already by myself. Regards, Arnd ________________________________ From: Arnd Rechenburg Sent: Montag, 20. Dezember 2010 19:16 To: 'pypy-dev at codespeak.net' Subject: ImportError: No module named _sqlite3 Hi, I tried to use sqlite3 with pypy but I got the following error: >>>> import sqlite Traceback (most recent call last): File "", line 1, in ImportError: No module named sqlite >>>> import sqlite3 Traceback (most recent call last): File "", line 1, in File "sqlite3/__init__.py", line 24, in from dbapi2 import * File "sqlite3/dbapi2.py", line 27, in from _sqlite3 import * ImportError: No module named _sqlite3 Someone knows where to put the _sqlite3.so library? Any help would be appreciated. Thx in advance, Arnd Rechenburg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101221/e9e062c6/attachment.htm From Arnd.Rechenburg at tomtom.com Tue Dec 21 09:58:52 2010 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Tue, 21 Dec 2010 09:58:52 +0100 Subject: [pypy-dev] long.__itemsize__ Message-ID: Hi, In my code I need something like long.__itemsize__ in Python. Any suggestions to do it with pypy? Regards, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101221/d09914ca/attachment-0001.htm From fijall at gmail.com Tue Dec 21 10:36:07 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 21 Dec 2010 11:36:07 +0200 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: References: Message-ID: On Tue, Dec 21, 2010 at 10:58 AM, Arnd Rechenburg wrote: > Hi, > > > > In my code I need something like > > long.__itemsize__ > > in Python. > > > > Any suggestions to do it with pypy? > Hey. This attribute is undocumented and untested as far as I can tell. What does it do? Cheers, fijal > > > Regards, > > Arnd > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From renesd at gmail.com Tue Dec 21 12:02:29 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 21 Dec 2010 11:02:29 +0000 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: References: Message-ID: Hi, __itemsize__ - in bytes, corresponds to item size field in the types definition structure. It's a field for types. See: http://docs.python.org/c-api/typeobj.html#tp_itemsize On Tue, Dec 21, 2010 at 9:36 AM, Maciej Fijalkowski wrote: > > On Tue, Dec 21, 2010 at 10:58 AM, Arnd Rechenburg > wrote: > > Hi, > > > > > > > > In my code I need something like > > > > long.__itemsize__ > > > > in Python. > > > > > > > > Any suggestions to do it with pypy? > > > > Hey. > > This attribute is undocumented and untested as far as I can tell. What > does it do? > > Cheers, > fijal > > > > > > > > Regards, > > > > Arnd > > > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From fijall at gmail.com Tue Dec 21 12:05:40 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 21 Dec 2010 13:05:40 +0200 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: References: Message-ID: On Tue, Dec 21, 2010 at 1:02 PM, Ren? Dudfield wrote: > Hi, > > __itemsize__ - in bytes, corresponds to item size field in the types > definition structure. > > It's a field for types. > See: > ? ?http://docs.python.org/c-api/typeobj.html#tp_itemsize > Well... Those are docs for C API. It doesn't say it's exposed at applevel nor since which version. (Also, to be precise, C API is known to be implementation specific) From anto.cuni at gmail.com Tue Dec 21 12:31:14 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 21 Dec 2010 12:31:14 +0100 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: References: Message-ID: <4D109002.2050505@gmail.com> On 21/12/10 12:05, Maciej Fijalkowski wrote: >> __itemsize__ - in bytes, corresponds to item size field in the types >> definition structure. >> >> It's a field for types. >> See: >> http://docs.python.org/c-api/typeobj.html#tp_itemsize >> > > Well... Those are docs for C API. It doesn't say it's exposed at > applevel nor since which version. (Also, to be precise, C API is known > to be implementation specific) Moreover, I don't think we could give it a sane semantics on PyPy, given that the same applevel type can be potentially implemented by many different low level structures with different sizes. ciao, Anto From fijall at gmail.com Tue Dec 21 12:31:59 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 21 Dec 2010 13:31:59 +0200 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: <4D109002.2050505@gmail.com> References: <4D109002.2050505@gmail.com> Message-ID: On Tue, Dec 21, 2010 at 1:31 PM, Antonio Cuni wrote: > On 21/12/10 12:05, Maciej Fijalkowski wrote: > >>> __itemsize__ - in bytes, corresponds to item size field in the types >>> definition structure. >>> >>> It's a field for types. >>> See: >>> ? ?http://docs.python.org/c-api/typeobj.html#tp_itemsize >>> >> >> Well... Those are docs for C API. It doesn't say it's exposed at >> applevel nor since which version. (Also, to be precise, C API is known >> to be implementation specific) > > Moreover, I don't think we could give it a sane semantics on PyPy, given that > the same applevel type can be potentially implemented by many different low > level structures with different sizes. > Not even "potentially", it actually is in some places. From arigo at tunes.org Tue Dec 21 12:37:00 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 21 Dec 2010 12:37:00 +0100 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: References: Message-ID: Hi, On Tue, Dec 21, 2010 at 9:58 AM, Arnd Rechenburg wrote: > In my code I need something like > > long.__itemsize__ > > in Python. I suppose the question is "why"? This is supposed to be the size in bytes occupied by one "element" of the type, and one element of "long" happens to be 15 bits, so that's why long.__itemsize__ is 2. It has no other meaning. If you are interested in an estimate of how many bytes some actual object takes, we have some functions in the 'gc' module, but even that is incomplete -- there are cases were it's hard to come up with a definitive answer, like objects based on "mapdicts". A bient?t, Armin. From renesd at gmail.com Tue Dec 21 13:24:37 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 21 Dec 2010 12:24:37 +0000 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: References: Message-ID: sys.getsizeof(obj) From arigo at tunes.org Tue Dec 21 13:29:28 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 21 Dec 2010 13:29:28 +0100 Subject: [pypy-dev] long.__itemsize__ In-Reply-To: References: Message-ID: Hi Ren?, On Tue, Dec 21, 2010 at 1:24 PM, Ren? Dudfield wrote: > sys.getsizeof(obj) Ah, thank you, didn't know about it. It's a 2.6 feature only though. A bient?t, Armin. From fuzzyman at gmail.com Tue Dec 21 13:59:02 2010 From: fuzzyman at gmail.com (Michael Foord) Date: Tue, 21 Dec 2010 12:59:02 +0000 Subject: [pypy-dev] Fwd: [IronPython] SciPy In-Reply-To: References: <8036074B72EB5B49B0BC19ED6F4E402047CAEB@ws-be-exchange.completegenomics.com> <6C7ABA8B4E309440B857D74348836F2E0108A168@TK5EX14MBXC135.redmond.corp.microsoft.com> Message-ID: I thought this email could be relevant for those interested in SciPy / Numpy on pypy. With enthought implementing a smaller core and using compatibility layers for alternative platforms, it would seem to be a good basis for a pypy port. ---------- Forwarded message ---------- From: Jason McCampbell Date: 20 December 2010 15:13 Subject: Re: [IronPython] SciPy To: Discussion of IronPython Hi Mark, As Dino mentioned we (Enthought) are working on refactoring Numpy into a pure "C" core with CPython and IronPython interface layers. This is largely complete and available at github (https://github.com/numpy/numpy-refactor), though the core layer is largely undocumented thus far. This is the multi-dimensional array. SciPy is in progress and we are updating it to work with the refactored numpy core and to add an IronPython interface. I assume you are looking for IronPython interfaces to SciPy as opposed to a C interface, correct? Regards, Jason On Thu, Dec 16, 2010 at 1:56 PM, Dino Viehland wrote: > Enthought has been working on getting numpy/scipy ported over to work w/ > IronPython. I believe numpy is working but I?m not sure of how far along > SciPy is. There?s a separate mailing list for this at: > > > > https://mail.enthought.com/mailman/listinfo/scipy4dotnet > > > > It?s very low traffic ? it?s usually just working through issues Enthought > has run into and either workarounds or suggested changes to IronPython. I?d > suggest sending a mail there ? they might have something you can try. > > > > > > *From:* users-bounces at lists.ironpython.com [mailto: > users-bounces at lists.ironpython.com] *On Behalf Of *Mark Senko > *Sent:* Thursday, December 16, 2010 11:49 AM > *To:* users at lists.ironpython.com > *Subject:* [IronPython] SciPy > > > > I?ve been searching for the current state of support for ?C? based > libraries, specifically SciPy (I?m just looking for a decent numerical > analysis package). The responses I?ve seen on various websites are somewhat > dated. > > What is the latest status, or is there no effort towards accommodating the > C API? Is IronClad still the best option? Any info, suggestions and warnings > would be appreciated before I start to invest a lot of time into installing > and learning these packages. > > > > *Mark Senko* > > Complete Genomics, Inc. > > 2071 Stierlin Court > > Mountain View, CA 94043 > > > > > > > > ____ > > > > The contents of this e-mail and any attachments are confidential and only for > > use by the intended recipient. Any unauthorized use, distribution or copying > > of this message is strictly prohibited. If you are not the intended recipient > > please inform the sender immediately by reply e-mail and delete this message > > from your system. Thank you for your co-operation. > > > _______________________________________________ > Users mailing list > Users at lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > _______________________________________________ Users mailing list Users at lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101221/567d69ed/attachment-0001.htm From garyrob at me.com Tue Dec 21 17:15:44 2010 From: garyrob at me.com (Gary Robinson) Date: Tue, 21 Dec 2010 11:15:44 -0500 Subject: [pypy-dev] Fwd: [IronPython] SciPy Message-ID: > I thought this email could be relevant for those interested in SciPy / Numpy > on pypy. With enthought implementing a smaller core and using compatibility > layers for alternative platforms, it would seem to be a good basis for a > pypy port. All I can do is give a BIG +1 to anything that can get numpy/scipy up more quickly. PyPy is starting to give us the speed we need for statistical/scientific work on python (the speed it still lacks compared to C or Java is made up for, for my purposes, by the ease of writing python compared to those languages). The recent 64-bit functionality lets us process a lot of data. The fast-forward branch is giving us the multiprocessing we need. (I recognize that there are other solutions, but for simple things we need to write quickly, the multiprocessing module is really sweet.) The main thing missing now is numpy/scipy. The addition of that will make PyPy a huge win in the scientific community, IMO. Anyway, I mention it in case the opinion of one person who is using Python in production for statistical processing is of interest. -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net From arigo at tunes.org Tue Dec 21 17:44:30 2010 From: arigo at tunes.org (Armin Rigo) Date: Tue, 21 Dec 2010 17:44:30 +0100 Subject: [pypy-dev] PyPy 1.4.1 Message-ID: Hi all, Here is PyPy 1.4.1 :-) Note that the Windows and the Mac OS X 64 builds are still missing; we are waiting for contributions here (amaury, mvt? :-). A bient?t, Armin. =============================== PyPy 1.4.1 =============================== We're pleased to announce the 1.4.1 release of PyPy. This release consolidates all the bug fixes that occurred since the previous release. To everyone that took the trouble to report them, we want to say thank you. http://pypy.org/download.html What is PyPy ============ PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython. Note that it still only emulates Python 2.5 by default; the ``fast-forward`` branch with Python 2.7 support is slowly getting ready but will only be integrated in the next release. In two words, the advantage of trying out PyPy instead of CPython (the default implementation of Python) is, for now, the performance. Not all programs are faster in PyPy, but we are confident that any CPU-intensive task will be much faster, at least if it runs for long enough (the JIT has a slow warm-up phase, which can take several seconds or even one minute on the largest programs). Note again that we do support compiling and using C extension modules from CPython (``pypy setup.py install``). However, this is still an alpha feature, and the most complex modules typically fail for various reasons; others work (e.g. ``PIL``) but take a serious performance hit. Also, for Mac OS X see below. Please note also that PyPy's performance was optimized almost exclusively on Linux. It seems from some reports that on Windows as well as Mac OS X (probably for different reasons) the performance might be lower. We did not investigate much so far. More highlights =============== * We migrated to Mercurial (thanks to Ronny Pfannschmidt and Antonio Cuni) for the effort) and moved to bitbucket. The new command to check out a copy of PyPy is:: hg clone http://bitbucket.org/pypy/pypy * In long-running processes, the assembler generated by old JIT-compilations is now freed. There should be no more leak, however long the process runs. * Improve a lot the performance of the ``binascii`` module, and of ``hashlib.md5`` and ``hashlib.sha``. * Made sys.setrecursionlimit() a no-op. Instead, we rely purely on the built-in stack overflow detection mechanism, which also gives you a RuntimeError -- just not at some exact recursion level. * Fix argument processing (now e.g. ``pypy -OScpass`` works like it does on CPython --- if you have a clue what it does there ``:-)`` ) * cpyext on Mac OS X: it still does not seem to work. I get systematically a segfault in dlopen(). Contributions welcome. * Fix two corner cases in the GC (one in minimark, one in asmgcc+JIT). This notably prevented "pypy translate.py -Ojit" from working on Windows, leading to crashes. * Fixed a corner case in the JIT's optimizer, leading to "Fatal RPython error: AssertionError". * Added some missing built-in functions into the 'os' module. * Fix ctypes (it was not propagating keepalive information from c_void_p). Cheers, Armin Rigo, for the rest of the team From dimaqq at gmail.com Tue Dec 21 19:07:14 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 21 Dec 2010 11:07:14 -0700 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Yes it does grind ff 4b7 to an almost halt, little cpu usage, reasonable memory size and constant disk activity for several minutes, very weird... So far, the difference between browsers is so large that I doubt it's dependent on data. Also it seems to tirgger as I remove columns progressively, thus new zero values should not appear, right? I'll invesitage some more... On 21 December 2010 01:06, Miquel Torres wrote: > Hi Dima, > >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. > > Well, speed.pypy is actually fast in all modern browsers. The problem > you are referring to is probably caused by a bug in the javascript > plotting library (jqPplot) that is triggered in the comparison view > when there are some results with 0 values. It only appears for some > plot types, but it is very annoying because it grinds the browser to a > halt like you say. Is that what you meant? > > We are looking into it, and will fix that library if necessary. > > Cheers, > Miquel > > > 2010/12/21 Dima Tisnek : >> On 20 December 2010 19:21, William ML Leslie >> wrote: >>> On 21 December 2010 11:59, Dima Tisnek wrote: >>>> More visibility for performance achievements would do >>>> good too. >>> >>> Where are pypy's performance achievements *not* visible, but should be? >> >> for example http://shootout.alioth.debian.org/ >> doesn't say which pypy version is used, what options, doesn't have >> performance figures for multithreaded/multicore >> >> also benchmarks are kinda small, most of them are not docuemented, and >> I couldn't find any info if the same python code was used for cpython >> and pypy (both shootout and speed pypy) or interpreter-specific >> verions were used, that is each tweaked for best performance given the >> known tradeoffs for each implementation.further the most standard >> benchmarks, pystone, etc. completely ignore the fact that real world >> programs are large and only a few small paths are execured often. >> >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. >> >> if you google for 'python performance' you don't get a single link to >> pypy on the first page, as a matter of fact, codespeak is poorly >> indexed, it took me quite some time to get some of my questions >> answered with a search. also if you look up 'pypy gc' you get a page >> on codespeak, but to interpret what the data actually means is so far >> beyond me. >> >> a good overview is found in the mainling list >> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >> slowspitfire and spambayes bits are outdated by now. >> >> the definitive good thing about pypy is that it's much easier to find >> out about its inner workings than that of cpython! >> >> hopefully a bit more of end-user stuff get known. >> let's call it pypy public outreach (feature request) >> >>> >>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>>> hard would it be to use same techniques and most of backends for some >>>> unrelated language that doesn't have jit yet, e.g. php? >>> >>> You know that pypy already has implementations of other languages, >>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>> integrated with the translated pypy-c, but they show that it is not >>> too difficult to write a runtime for any dynamic language you choose. >> >> Oh I didn't know there were so many, and I mistakenly though that js >> was a target, not implmented langauge. In any case I read somewhere >> that js support was retired... >> >>> >>>> And how hard >>>> would it be to marry two dynamic languages, so that modules from one >>>> could be used in the other? Or that modules written in rpython could >>>> be used in several langs? >>> >>> It's in the "interesting problems" bucket, and the effort required >>> depends on the level of integration between languages you want. ?There >>> are several projects already attempting to do decent integration >>> between several languages, besides the approach used on the JVM, there >>> are also GNU Guile, Racket, and Parrot, among others. ?It might be >>> worth waiting to see how these different projects pan out, before >>> spending a bunch of effort just to be an also-ran in the >>> multi-language runtime market. >>> >>> However, implementing more languages in rpython has the side-effect of >>> propagating the l * o * p problem: it introduces more and more >>> implementations that then have to be maintained, so good >>> cross-language integration probably belongs /outside/ pypy itself, so >>> existing runtimes can hook into it. >> >> Makes perfect sense, after all any given other language hardly has the >> same data model as python. >> >>> >>> But it would be an interesting experiment (to marry the various >>> interpreters pypy ships with), if you wanted to try it. >>> >>> My two cents. >>> >>> -- >>> William Leslie >>> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > From dmalcolm at redhat.com Tue Dec 21 19:54:36 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Tue, 21 Dec 2010 13:54:36 -0500 Subject: [pypy-dev] [PATCH] Improving readability of generated .c code In-Reply-To: <4D077BB5.7030102@gmx.de> References: <1292288224.24137.22.camel@radiator.bos.redhat.com> <4D077BB5.7030102@gmx.de> Message-ID: <1292957676.20268.110.camel@surprise> On Tue, 2010-12-14 at 15:14 +0100, Carl Friedrich Bolz wrote: > Hi David, > > On 12/14/2010 01:57 AM, David Malcolm wrote: Thanks for looking at this; sorry for the belated reply. I'm attaching the latest work-in-progress version of the patch, addressing some of the areas discussed. (snip) > > Hopefully this makes the generated .c much more readable. > > I like the general approach, though I have the vague fear that it will > make the generated C code much bigger. Could you check that this is not > the case? The .c code seems to get roughly 15% bigger (which I don't think is an issue), but the translation may be getting significantly longer (40% more time; it's a real pain to benchmark this, though). I'm also not getting as many comments as I'd like for inlining (see below), and I expect the code to grow when I fix that. Method: Control: - clean pypy hg checkout of 40149:cd083843b67a - cd pypy/translator/goal - python translate.py (i.e. without any flags) - wait for the build to complete - locate the generated code in the "testing_1" subdirectory: - invoke David A. Wheeler's "SLOCCount" tool on the "testing_1" dir (http://www.dwheeler.com/sloccount/) - run "du -h" on the "testing_1" dir Experiment: - as above, but with the attached changes to the source tree Actually, the first time I built the clean version, I was hacking on the debug tree, and lost my copy of the relevant /tmp/usession-N directory, so I set PYPY_USESSION_DIR=/tmp/clean-build and redid it. (I was also checking mail, IRC, and other stuff on the machine, but I don't think this significantly affects the timings). Size of generated source:: Output of "wc *.c":: (trimmed to final summary, giving newline, word, and byte counts) Unpatched:: 5166080 15803859 186007458 total With the patch:: 5544105 18458780 213205652 total Disk usage:: This is: du -h -c *.c omitting the individual counts Unpatched:: 178M total With the patch:: 204M total "sloccount":: Unpatched:: SLOC Directory SLOC-by-Language (Sorted) 4728922 testing_1 ansic=4728922 With the patch:: SLOC Directory SLOC-by-Language (Sorted) 4730181 testing_1 ansic=4730181 I haven't measured memory usage during conversion. The above looks fine to me (I'll happily take a 15% increase in on-disk size of files that are used at runtime only for debugging, in exchange for easier debugging). However, conversion time seems to increase roughly 40%. Unpatched: [Timer] Timings: [Timer] annotate --- 1180.5 s [Timer] rtype_lltype --- 730.7 s [Timer] backendopt_lltype --- 468.5 s [Timer] stackcheckinsertion_lltype --- 68.9 s [Timer] database_c --- 546.4 s [Timer] source_c --- 1323.5 s [Timer] compile_c --- 347.4 s [Timer] =========================================== [Timer] Total: --- 4665.9 s Patched: [Timer] Timings: [Timer] annotate --- 1428.2 s [Timer] rtype_lltype --- 913.6 s [Timer] backendopt_lltype --- 831.9 s [Timer] stackcheckinsertion_lltype --- 106.6 s [Timer] database_c --- 991.1 s [Timer] source_c --- 2385.8 s [Timer] compile_c --- 400.2 s [Timer] =========================================== [Timer] Total: --- 7057.3 s > > (I hope to > > work on the names of locals also; ideally they ought to embed the python > > indentifier) > > This should already work in theory, see e.g. "l_cls_0" above, which > embeds the Python variable name "cls". I guess that the information is > simply lost in some of the transformation steps. If you feel like > hunting down where, you can probably get better naming in some cases. I think I was wrong about this: the variables that are getting uninformative names are the temporaries; .c locals that "are" RPython locals are being named informatively. > > > This is on a Fedora 13 x86_64 box, with cpython 2.6.5 > > > > Caveats: > > - this is my first patch to PyPy (please be gentle!): I hope I've > > correctly understood the various levels, but I suspect I may still be > > missing things at the conceptual level > > I think you are doing great. You might have to grep around a bit for > more places that create SpaceOperations, or did you look at all of them? There are numerous places where ops are created, (in llops.genops iirc), and somewhere in that process, many of the resulting operations are only getting a partial "path" of CodeLoc instances (see below). I plan to look into that next. > > - haven't completed the full unit tests yet. > > This is the biggest problem of the patch, of course. I think that you > should at least try to write tests for each of the places where you > propagate the offset down one level. In addition, you probably need an > integration test where you check the generated C code for the presence > of the correct comments. The latest version of the patch adds two integration tests into pypy/translator/c/test/test_genc.py I haven't yet added unit tests for the location propagation (see below). > > - the patch attempts to propagate the "offset" of flowspace > > SpaceOperation instances in a couple of places where the offset was > > being discarded: > > - in BaseInliner.copy_operation > > This is wrong. If you inline one function into another, it doesn't make > sense to put the offset of the inner function into the outer function. > To solve this problem, you have two options: > 1) when inlining, set the offset to -1, and just don't get source code > then > 2) make every *operation* contain a reference to the func it comes > from. The memory implications of this should be checked, but are > probably not too bad. What I've done is (2): introduce a CodeLoc class representing a (code, offset) pair, and to replace the "offset" of a flowspace SpaceOperation with a new OperationLoc class, which consists of a list of 1 or more CodeLoc instances. In the simple case we have a single CodeLoc, but in the case of operations within an inlined function we have the CodeLoc of the callsite, followed by that of the operation within the inlined body. For the case of nested inlining, I'm assuming we could have an arbitrary long list of CodeLoc instances describing the OperationLoc. Having said that, within inlining most operations seem not to get part of the "path", and I'm using None for these. This is probably the area most needing further unit tests. Other changes in the patch: - I've attempted to reorder blocks within the generated C to reflect the source-level ordering of operations within the RPython. This breaks down somewhat in the presence of inlined ops with only partial paths, but it makes a best attempt. - I had a go at renaming block labels in the C, so that e.g. a return block might be "return_block3", and a raise block might be "raise_block42". - I'm now cleaning out C-style comments from the .py code when writing them into comments in the generated C code, to avoid syntax errors - two new self tests, as mentioned above. I made interactive Translation.source_c() return the .c filename to help write the tests (...snip...) > > - I haven't tried doing this on a full build of the interpreter > > (having highly readable generated .c for this is my objective [1]). > > I fear it will never be "highly readable", but I think your work > improves things already. Thanks! > > One other idea I had was to sort the blocks in the function by the > > bytecode offset; currently the c blocks seem to "jump around" a fair bit > > relative to the corresponding rpython code. Has any tuning been done on > > the ordering of the blocks within the generated .c code? (or is it > > assumed that the .c compiler will "flatten" these, for the case when a > > node has a single successor node?) > > The C compiler should not care about the order of blocks. If it does, I > am prepared to call it insane. As mentioned above, I attempted this. (..snip...) > > The C code doesn't look anything like the .py code that my patch is > > inserting in the above. (My current guess is that I need to be smarter > > about inlined operations, though that's a guess at this stage) > > Sounds like a correct guess, malloc is transformed into something else, > and the offset of the inlined operations is basically just random, if > you interpret it in the context of the outer function. I'd hoped with the OperationLoc/CodeLoc changes to be able to propagate the full hierarchy of RPython source location at an inlined callsite, but I'm not getting that yet. I plan to look at that next. However, the comments do at least now reflect true locations throughout. One other idea that occurred to me: currently in pypy/translator/c/genc.py the code spits all of the functions into a series of numbered implement.c files. I thought it might be nice to split them up somehow by namespace, so for example, all "PyUnicode_*" and "pypy_g_PyUnicode_*" functions could end up in "PyUnicode.c", "PyUnicode_1.c", etc, and implement*.c could then only contain the ones that don't match such a pattern. Does this sound reasonable, and any pointers on minimal ways of testing this? (I'm guessing the unit tests for the C database are the best guidance here). Hope this is helpful Dave -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy-add-py-source-as-comments-to-generated-c-2010-12-21-001.patch Type: text/x-patch Size: 26643 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20101221/8218cf33/attachment-0001.bin From tobami at googlemail.com Tue Dec 21 20:30:58 2010 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 21 Dec 2010 20:30:58 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: It is weird that it happens as you de-select benchmarks. Does it happen with non-beta browsers? 2010/12/21 Dima Tisnek : > Yes it does grind ff 4b7 to an almost halt, little cpu usage, > reasonable memory size and constant disk activity for several minutes, > very weird... > So far, the difference between browsers is so large that I doubt it's > dependent on data. > Also it seems to tirgger as I remove columns progressively, thus new > zero values should not appear, right? > > I'll invesitage some more... > > On 21 December 2010 01:06, Miquel Torres wrote: >> Hi Dima, >> >>> another temp problem with speed pypy is that it's terrubly slow in ff >>> beta. it also occasionally grinds in stable ff and opera, but I guess >>> this can be forgiven for the sake of simplicity / developer effort. >> >> Well, speed.pypy is actually fast in all modern browsers. The problem >> you are referring to is probably caused by a bug in the javascript >> plotting library (jqPplot) that is triggered in the comparison view >> when there are some results with 0 values. It only appears for some >> plot types, but it is very annoying because it grinds the browser to a >> halt like you say. Is that what you meant? >> >> We are looking into it, and will fix that library if necessary. >> >> Cheers, >> Miquel >> >> >> 2010/12/21 Dima Tisnek : >>> On 20 December 2010 19:21, William ML Leslie >>> wrote: >>>> On 21 December 2010 11:59, Dima Tisnek wrote: >>>>> More visibility for performance achievements would do >>>>> good too. >>>> >>>> Where are pypy's performance achievements *not* visible, but should be? >>> >>> for example http://shootout.alioth.debian.org/ >>> doesn't say which pypy version is used, what options, doesn't have >>> performance figures for multithreaded/multicore >>> >>> also benchmarks are kinda small, most of them are not docuemented, and >>> I couldn't find any info if the same python code was used for cpython >>> and pypy (both shootout and speed pypy) or interpreter-specific >>> verions were used, that is each tweaked for best performance given the >>> known tradeoffs for each implementation.further the most standard >>> benchmarks, pystone, etc. completely ignore the fact that real world >>> programs are large and only a few small paths are execured often. >>> >>> another temp problem with speed pypy is that it's terrubly slow in ff >>> beta. it also occasionally grinds in stable ff and opera, but I guess >>> this can be forgiven for the sake of simplicity / developer effort. >>> >>> if you google for 'python performance' you don't get a single link to >>> pypy on the first page, as a matter of fact, codespeak is poorly >>> indexed, it took me quite some time to get some of my questions >>> answered with a search. also if you look up 'pypy gc' you get a page >>> on codespeak, but to interpret what the data actually means is so far >>> beyond me. >>> >>> a good overview is found in the mainling list >>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >>> slowspitfire and spambayes bits are outdated by now. >>> >>> the definitive good thing about pypy is that it's much easier to find >>> out about its inner workings than that of cpython! >>> >>> hopefully a bit more of end-user stuff get known. >>> let's call it pypy public outreach (feature request) >>> >>>> >>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>>>> hard would it be to use same techniques and most of backends for some >>>>> unrelated language that doesn't have jit yet, e.g. php? >>>> >>>> You know that pypy already has implementations of other languages, >>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>>> integrated with the translated pypy-c, but they show that it is not >>>> too difficult to write a runtime for any dynamic language you choose. >>> >>> Oh I didn't know there were so many, and I mistakenly though that js >>> was a target, not implmented langauge. In any case I read somewhere >>> that js support was retired... >>> >>>> >>>>> And how hard >>>>> would it be to marry two dynamic languages, so that modules from one >>>>> could be used in the other? Or that modules written in rpython could >>>>> be used in several langs? >>>> >>>> It's in the "interesting problems" bucket, and the effort required >>>> depends on the level of integration between languages you want. ?There >>>> are several projects already attempting to do decent integration >>>> between several languages, besides the approach used on the JVM, there >>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be >>>> worth waiting to see how these different projects pan out, before >>>> spending a bunch of effort just to be an also-ran in the >>>> multi-language runtime market. >>>> >>>> However, implementing more languages in rpython has the side-effect of >>>> propagating the l * o * p problem: it introduces more and more >>>> implementations that then have to be maintained, so good >>>> cross-language integration probably belongs /outside/ pypy itself, so >>>> existing runtimes can hook into it. >>> >>> Makes perfect sense, after all any given other language hardly has the >>> same data model as python. >>> >>>> >>>> But it would be an interesting experiment (to marry the various >>>> interpreters pypy ships with), if you wanted to try it. >>>> >>>> My two cents. >>>> >>>> -- >>>> William Leslie >>>> >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >> > From dimaqq at gmail.com Tue Dec 21 22:22:15 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 21 Dec 2010 14:22:15 -0700 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: yes, same problem, but the grind is not so aweful, surviveable even, tried with opera and normal ff On 21 December 2010 12:30, Miquel Torres wrote: > It is weird that it happens as you de-select benchmarks. > > Does it happen with non-beta browsers? > > > 2010/12/21 Dima Tisnek : >> Yes it does grind ff 4b7 to an almost halt, little cpu usage, >> reasonable memory size and constant disk activity for several minutes, >> very weird... >> So far, the difference between browsers is so large that I doubt it's >> dependent on data. >> Also it seems to tirgger as I remove columns progressively, thus new >> zero values should not appear, right? >> >> I'll invesitage some more... >> >> On 21 December 2010 01:06, Miquel Torres wrote: >>> Hi Dima, >>> >>>> another temp problem with speed pypy is that it's terrubly slow in ff >>>> beta. it also occasionally grinds in stable ff and opera, but I guess >>>> this can be forgiven for the sake of simplicity / developer effort. >>> >>> Well, speed.pypy is actually fast in all modern browsers. The problem >>> you are referring to is probably caused by a bug in the javascript >>> plotting library (jqPplot) that is triggered in the comparison view >>> when there are some results with 0 values. It only appears for some >>> plot types, but it is very annoying because it grinds the browser to a >>> halt like you say. Is that what you meant? >>> >>> We are looking into it, and will fix that library if necessary. >>> >>> Cheers, >>> Miquel >>> >>> >>> 2010/12/21 Dima Tisnek : >>>> On 20 December 2010 19:21, William ML Leslie >>>> wrote: >>>>> On 21 December 2010 11:59, Dima Tisnek wrote: >>>>>> More visibility for performance achievements would do >>>>>> good too. >>>>> >>>>> Where are pypy's performance achievements *not* visible, but should be? >>>> >>>> for example http://shootout.alioth.debian.org/ >>>> doesn't say which pypy version is used, what options, doesn't have >>>> performance figures for multithreaded/multicore >>>> >>>> also benchmarks are kinda small, most of them are not docuemented, and >>>> I couldn't find any info if the same python code was used for cpython >>>> and pypy (both shootout and speed pypy) or interpreter-specific >>>> verions were used, that is each tweaked for best performance given the >>>> known tradeoffs for each implementation.further the most standard >>>> benchmarks, pystone, etc. completely ignore the fact that real world >>>> programs are large and only a few small paths are execured often. >>>> >>>> another temp problem with speed pypy is that it's terrubly slow in ff >>>> beta. it also occasionally grinds in stable ff and opera, but I guess >>>> this can be forgiven for the sake of simplicity / developer effort. >>>> >>>> if you google for 'python performance' you don't get a single link to >>>> pypy on the first page, as a matter of fact, codespeak is poorly >>>> indexed, it took me quite some time to get some of my questions >>>> answered with a search. also if you look up 'pypy gc' you get a page >>>> on codespeak, but to interpret what the data actually means is so far >>>> beyond me. >>>> >>>> a good overview is found in the mainling list >>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >>>> slowspitfire and spambayes bits are outdated by now. >>>> >>>> the definitive good thing about pypy is that it's much easier to find >>>> out about its inner workings than that of cpython! >>>> >>>> hopefully a bit more of end-user stuff get known. >>>> let's call it pypy public outreach (feature request) >>>> >>>>> >>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>>>>> hard would it be to use same techniques and most of backends for some >>>>>> unrelated language that doesn't have jit yet, e.g. php? >>>>> >>>>> You know that pypy already has implementations of other languages, >>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>>>> integrated with the translated pypy-c, but they show that it is not >>>>> too difficult to write a runtime for any dynamic language you choose. >>>> >>>> Oh I didn't know there were so many, and I mistakenly though that js >>>> was a target, not implmented langauge. In any case I read somewhere >>>> that js support was retired... >>>> >>>>> >>>>>> And how hard >>>>>> would it be to marry two dynamic languages, so that modules from one >>>>>> could be used in the other? Or that modules written in rpython could >>>>>> be used in several langs? >>>>> >>>>> It's in the "interesting problems" bucket, and the effort required >>>>> depends on the level of integration between languages you want. ?There >>>>> are several projects already attempting to do decent integration >>>>> between several languages, besides the approach used on the JVM, there >>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be >>>>> worth waiting to see how these different projects pan out, before >>>>> spending a bunch of effort just to be an also-ran in the >>>>> multi-language runtime market. >>>>> >>>>> However, implementing more languages in rpython has the side-effect of >>>>> propagating the l * o * p problem: it introduces more and more >>>>> implementations that then have to be maintained, so good >>>>> cross-language integration probably belongs /outside/ pypy itself, so >>>>> existing runtimes can hook into it. >>>> >>>> Makes perfect sense, after all any given other language hardly has the >>>> same data model as python. >>>> >>>>> >>>>> But it would be an interesting experiment (to marry the various >>>>> interpreters pypy ships with), if you wanted to try it. >>>>> >>>>> My two cents. >>>>> >>>>> -- >>>>> William Leslie >>>>> >>>> _______________________________________________ >>>> pypy-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> > From dimaqq at gmail.com Wed Dec 22 06:51:39 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 21 Dec 2010 22:51:39 -0700 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: I sort of figured it out, although I don't have a ready solution. This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8. Basically memory consumption of the plot appears to be proportional to plot area. Normal plot that you get by default at http://speed.pypy.org/comparison/ costs about 100M of browser memory consumption: opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050; opera 80M, stable ff 55M, beta ff 70M at window size 1024x600; Switching to "horizontal" produces a tall plot of same width and costs about 300~700M of browser memory: opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050; opera 350M, stable ff 370M, beta ff 370M at window size 1024x600; Suprisingly window size only matters while javascript produces the plot, and not when window is resized, even though plot it resized with the window correctly. This alone is pretty heavy, but doesn't grind the browser. What really grinds is that every time you change a tickbox on the left, a plot is redrawn and another 200M of browser memory is wasted. This is not double buffering, as next change adds yet another 200M or so and so on, it appears that either js doesn't free something, or browser caches or saves the previous page state. As memory consumption grows, at some point browser hits the wall, causes heavy swapping for some time, and I think garbage collection, because practical (but not virtual) memory usage first drops to something like 20~50M and then returns to "normal" 300M. opera ~30 seconds, stable ff about a minute, beta ff several minutes (total system mem 1G, cpu Atom @1.6GHz) Perhaps OS also plays a role in the grind, as it is clearly swapping and swaps out too much? or triggers gc too late and gc has to pull the pages back from disk to perform collection? ie8 doesn't use that much memory, as a matter of fact memory consumption starts little (40M) and changes very little (only +10M) if you go to horizonatal view; the price is very slow rendering, more than 10 seconds per column change. I'll post this on firefox bugzilla too, let's see if someone has a solution. Meanwhile perhaps pypy speed center could start with a smaller plot area (or fewer columns as that makes horizontal plot smaller) to accomodate varying hardware and system mem usage that users might have? The simplest would be a warning next to "horizontal" checkbox. On 21 December 2010 01:06, Miquel Torres wrote: > Hi Dima, > >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. > > Well, speed.pypy is actually fast in all modern browsers. The problem > you are referring to is probably caused by a bug in the javascript > plotting library (jqPplot) that is triggered in the comparison view > when there are some results with 0 values. It only appears for some > plot types, but it is very annoying because it grinds the browser to a > halt like you say. Is that what you meant? > > We are looking into it, and will fix that library if necessary. > > Cheers, > Miquel > > > 2010/12/21 Dima Tisnek : >> On 20 December 2010 19:21, William ML Leslie >> wrote: >>> On 21 December 2010 11:59, Dima Tisnek wrote: >>>> More visibility for performance achievements would do >>>> good too. >>> >>> Where are pypy's performance achievements *not* visible, but should be? >> >> for example http://shootout.alioth.debian.org/ >> doesn't say which pypy version is used, what options, doesn't have >> performance figures for multithreaded/multicore >> >> also benchmarks are kinda small, most of them are not docuemented, and >> I couldn't find any info if the same python code was used for cpython >> and pypy (both shootout and speed pypy) or interpreter-specific >> verions were used, that is each tweaked for best performance given the >> known tradeoffs for each implementation.further the most standard >> benchmarks, pystone, etc. completely ignore the fact that real world >> programs are large and only a few small paths are execured often. >> >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. >> >> if you google for 'python performance' you don't get a single link to >> pypy on the first page, as a matter of fact, codespeak is poorly >> indexed, it took me quite some time to get some of my questions >> answered with a search. also if you look up 'pypy gc' you get a page >> on codespeak, but to interpret what the data actually means is so far >> beyond me. >> >> a good overview is found in the mainling list >> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >> slowspitfire and spambayes bits are outdated by now. >> >> the definitive good thing about pypy is that it's much easier to find >> out about its inner workings than that of cpython! >> >> hopefully a bit more of end-user stuff get known. >> let's call it pypy public outreach (feature request) >> >>> >>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>>> hard would it be to use same techniques and most of backends for some >>>> unrelated language that doesn't have jit yet, e.g. php? >>> >>> You know that pypy already has implementations of other languages, >>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>> integrated with the translated pypy-c, but they show that it is not >>> too difficult to write a runtime for any dynamic language you choose. >> >> Oh I didn't know there were so many, and I mistakenly though that js >> was a target, not implmented langauge. In any case I read somewhere >> that js support was retired... >> >>> >>>> And how hard >>>> would it be to marry two dynamic languages, so that modules from one >>>> could be used in the other? Or that modules written in rpython could >>>> be used in several langs? >>> >>> It's in the "interesting problems" bucket, and the effort required >>> depends on the level of integration between languages you want. ?There >>> are several projects already attempting to do decent integration >>> between several languages, besides the approach used on the JVM, there >>> are also GNU Guile, Racket, and Parrot, among others. ?It might be >>> worth waiting to see how these different projects pan out, before >>> spending a bunch of effort just to be an also-ran in the >>> multi-language runtime market. >>> >>> However, implementing more languages in rpython has the side-effect of >>> propagating the l * o * p problem: it introduces more and more >>> implementations that then have to be maintained, so good >>> cross-language integration probably belongs /outside/ pypy itself, so >>> existing runtimes can hook into it. >> >> Makes perfect sense, after all any given other language hardly has the >> same data model as python. >> >>> >>> But it would be an interesting experiment (to marry the various >>> interpreters pypy ships with), if you wanted to try it. >>> >>> My two cents. >>> >>> -- >>> William Leslie >>> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Wed Dec 22 07:51:57 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 22 Dec 2010 08:51:57 +0200 Subject: [pypy-dev] Fwd: [IronPython] SciPy In-Reply-To: References: Message-ID: Oh wow, that's really cool. On Tue, Dec 21, 2010 at 6:15 PM, Gary Robinson wrote: >> I thought this email could be relevant for those interested in SciPy / Numpy >> on pypy. With enthought implementing a smaller core and using compatibility >> layers for alternative platforms, it would seem to be a good basis for a >> pypy port. > > All I can do is give a BIG +1 to anything that can get numpy/scipy up more quickly. > > PyPy is starting to give us the speed we need for statistical/scientific work on python (the speed it still lacks compared to C or Java is made up for, for my purposes, by the ease of writing python compared to those languages). The recent 64-bit functionality lets us process a lot of data. The fast-forward branch is giving us the multiprocessing we need. (I recognize that there are other solutions, but for simple things we need to write quickly, the multiprocessing module is really sweet.) > > The main thing missing now is numpy/scipy. The addition of that will make PyPy a huge win in the scientific community, IMO. > > Anyway, I mention it in case the opinion of one person who is using Python in production for statistical processing is of interest. > > -- > > Gary Robinson > CTO > Emergent Discovery, LLC > personal email: garyrob at me.com > work email: grobinson at emergentdiscovery.com > Company: http://www.emergentdiscovery.com > Blog: ? ?http://www.garyrobinson.net > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tobami at googlemail.com Wed Dec 22 09:24:28 2010 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 22 Dec 2010 09:24:28 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Mmm, yes, it appears that the memory is not properly freed. I can try experimenting by destroying the dom element and recreating it each time that the plot changes... Btw., instead of continuing here "polluting" the pypy-dev mailing list, we can move to http://groups.google.com/group/codespeed if needed. Thanks for reporting this. Cheers, Miquel 2010/12/22 Dima Tisnek : > I sort of figured it out, although I don't have a ready solution. > > This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8. > Basically memory consumption of the plot appears to be proportional to > plot area. > Normal plot that you get by default at > http://speed.pypy.org/comparison/ costs about 100M of browser memory > consumption: > opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050; > opera ?80M, stable ff 55M, beta ff 70M at window size 1024x600; > Switching to "horizontal" produces a tall plot of same width and costs > about 300~700M of browser memory: > opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050; > opera 350M, stable ff 370M, beta ff 370M at window size 1024x600; > > Suprisingly window size only matters while javascript produces the > plot, and not when window is resized, even though plot it resized with > the window correctly. > > This alone is pretty heavy, but doesn't grind the browser. > What really grinds is that every time you change a tickbox on the > left, a plot is redrawn and another 200M of browser memory is wasted. > This is not double buffering, as next change adds yet another 200M or > so and so on, it appears that either js doesn't free something, or > browser caches or saves the previous page state. > > As memory consumption grows, at some point browser hits the wall, > causes heavy swapping for some time, and I think garbage collection, > because practical (but not virtual) memory usage first drops to > something like 20~50M and then returns to "normal" 300M. > opera ~30 seconds, stable ff about a minute, beta ff several minutes > (total system mem 1G, cpu Atom @1.6GHz) > > Perhaps OS also plays a role in the grind, as it is clearly swapping > and swaps out too much? or triggers gc too late and gc has to pull the > pages back from disk to perform collection? > > ie8 doesn't use that much memory, as a matter of fact memory > consumption starts little (40M) and changes very little (only +10M) if > you go to horizonatal view; the price is very slow rendering, more > than 10 seconds per column change. > > I'll post this on firefox bugzilla too, let's see if someone has a solution. > > Meanwhile perhaps pypy speed center could start with a smaller plot > area (or fewer columns as that makes horizontal plot smaller) to > accomodate varying hardware and system mem usage that users might > have? > The simplest would be a warning next to "horizontal" checkbox. > > On 21 December 2010 01:06, Miquel Torres wrote: >> Hi Dima, >> >>> another temp problem with speed pypy is that it's terrubly slow in ff >>> beta. it also occasionally grinds in stable ff and opera, but I guess >>> this can be forgiven for the sake of simplicity / developer effort. >> >> Well, speed.pypy is actually fast in all modern browsers. The problem >> you are referring to is probably caused by a bug in the javascript >> plotting library (jqPplot) that is triggered in the comparison view >> when there are some results with 0 values. It only appears for some >> plot types, but it is very annoying because it grinds the browser to a >> halt like you say. Is that what you meant? >> >> We are looking into it, and will fix that library if necessary. >> >> Cheers, >> Miquel >> >> >> 2010/12/21 Dima Tisnek : >>> On 20 December 2010 19:21, William ML Leslie >>> wrote: >>>> On 21 December 2010 11:59, Dima Tisnek wrote: >>>>> More visibility for performance achievements would do >>>>> good too. >>>> >>>> Where are pypy's performance achievements *not* visible, but should be? >>> >>> for example http://shootout.alioth.debian.org/ >>> doesn't say which pypy version is used, what options, doesn't have >>> performance figures for multithreaded/multicore >>> >>> also benchmarks are kinda small, most of them are not docuemented, and >>> I couldn't find any info if the same python code was used for cpython >>> and pypy (both shootout and speed pypy) or interpreter-specific >>> verions were used, that is each tweaked for best performance given the >>> known tradeoffs for each implementation.further the most standard >>> benchmarks, pystone, etc. completely ignore the fact that real world >>> programs are large and only a few small paths are execured often. >>> >>> another temp problem with speed pypy is that it's terrubly slow in ff >>> beta. it also occasionally grinds in stable ff and opera, but I guess >>> this can be forgiven for the sake of simplicity / developer effort. >>> >>> if you google for 'python performance' you don't get a single link to >>> pypy on the first page, as a matter of fact, codespeak is poorly >>> indexed, it took me quite some time to get some of my questions >>> answered with a search. also if you look up 'pypy gc' you get a page >>> on codespeak, but to interpret what the data actually means is so far >>> beyond me. >>> >>> a good overview is found in the mainling list >>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >>> slowspitfire and spambayes bits are outdated by now. >>> >>> the definitive good thing about pypy is that it's much easier to find >>> out about its inner workings than that of cpython! >>> >>> hopefully a bit more of end-user stuff get known. >>> let's call it pypy public outreach (feature request) >>> >>>> >>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>>>> hard would it be to use same techniques and most of backends for some >>>>> unrelated language that doesn't have jit yet, e.g. php? >>>> >>>> You know that pypy already has implementations of other languages, >>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>>> integrated with the translated pypy-c, but they show that it is not >>>> too difficult to write a runtime for any dynamic language you choose. >>> >>> Oh I didn't know there were so many, and I mistakenly though that js >>> was a target, not implmented langauge. In any case I read somewhere >>> that js support was retired... >>> >>>> >>>>> And how hard >>>>> would it be to marry two dynamic languages, so that modules from one >>>>> could be used in the other? Or that modules written in rpython could >>>>> be used in several langs? >>>> >>>> It's in the "interesting problems" bucket, and the effort required >>>> depends on the level of integration between languages you want. ?There >>>> are several projects already attempting to do decent integration >>>> between several languages, besides the approach used on the JVM, there >>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be >>>> worth waiting to see how these different projects pan out, before >>>> spending a bunch of effort just to be an also-ran in the >>>> multi-language runtime market. >>>> >>>> However, implementing more languages in rpython has the side-effect of >>>> propagating the l * o * p problem: it introduces more and more >>>> implementations that then have to be maintained, so good >>>> cross-language integration probably belongs /outside/ pypy itself, so >>>> existing runtimes can hook into it. >>> >>> Makes perfect sense, after all any given other language hardly has the >>> same data model as python. >>> >>>> >>>> But it would be an interesting experiment (to marry the various >>>> interpreters pypy ships with), if you wanted to try it. >>>> >>>> My two cents. >>>> >>>> -- >>>> William Leslie >>>> >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >> > From tobami at googlemail.com Wed Dec 22 09:26:26 2010 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 22 Dec 2010 09:26:26 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: so, what about the ai and spectral-norm benchmarks. Anybody can come up with a description for them? 2010/12/15 Paolo Giarrusso : > On Mon, Dec 13, 2010 at 09:31, Miquel Torres wrote: >> Oh, btw., the "normalized" stacked bars now display a warning note >> about its correctness, and how it must be viewed as giving results a >> weighting instead of them being normalized. It even includes a link to >> the proper paper. I hope that is enough for the strict statisticians >> among us ;-) > > I see. Thanks! > >> See: >> http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars >> >> PS: there is a bug in the jqPlot plotting library when null values are >> present. Trying to display PyPy 1.3 results for the newer go, pyflake >> or ?raytrace will create some nasty js loops. It also has problems >> with autoscaling the axis sometimes. >> >> >> >> 2010/12/13 Miquel Torres : >>> Thanks all for the input. >>> I've compiled a list based on your mails, the Unladen benchmarks page >>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >>> alioth descriptions. Here is an extract of the current speed.pypy.org >>> admin: >>> >>> ai >>> chaos ? Creates chaosgame-like fractals >>> crypto_pyaes ? ?A pure python implementation of AES >>> django ? ? ? ? ?Uses the Django template system to build a 150x150-cell HTML table >>> >>> fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The fannkuch >>> benchmark is defined by programs in Performing Lisp Analysis of the >>> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig. >>> >>> float ? ? ? ? ? Creates an array of points using circular projection and then >>> normalizes and maximizes them. Floating-point heavy. >>> go ? ? ? ? ? ? ?A go (chess like game) computer player AI. >>> html5lib ? ? ? ?Parses the HTML 5 spec using html5lib >>> meteor-contest ?Searchs for solutions to shape packing puzzle. >>> nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models the >>> orbits of Jovian planets, using a simple symplectic-integrator. >>> pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and bzip2 >>> decoder/decompressor. >>> raytrace-simple A raytracer renderer >>> richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates the task >>> dispatcher in the kernel of an operating system. >>> rietveld ? ? ? ?A Django application benchmark. >>> slowspitfire >>> spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam classifier >>> spectral-norm >>> spitfire ? ? ? ?Uses the Spitfire template system to build a 1000x1000-cell HTML table. >>> spitfire_cstringio ? ? ?Uses the Spitfire template system to build a >>> 1000x1000-cell HTML table, using the cstringio module. >>> telco >>> twisted_iteration >>> twisted_names >>> twisted_pb >>> twisted_tcp ? ? Connects one Twised client to one Twisted server over TCP >>> (on the loopback interface) and then writes bytes as fast as it can. >>> waf ? ? Python-based framework for configuring, compiling and installing >>> applications. It derives from the concepts of other build tools such >>> as Scons, Autotools, CMake or Ant. >>> >>> >>> So the remaining descriptions are >>> ai >>> slowspitfire (what is the exact difference between the three spitfire benches?) >>> spectral-norm >>> telco >>> twisted (most of them) >>> >>> Are the descriptions all right so far?. They can be made much longer >>> if you deem it desirable. >>> >>> on speed.pypy.org you will currently see the descriptions in 3 places: >>> - Changes view: A tooltip on hover over each benchmark >>> - Timeline: a description box beneath each plot >>> - Comparison: A tooltip over each benchmark when hovering the >>> selection menu on the left side. >>> >>> Any suggestions on how to improve it further are welcome ;-) >>> >>> Miquel >>> >>> >>> 2010/12/9 Paolo Giarrusso : >>>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada wrote: >>>>> Here is a incomplete draft list: >>>> >>>>> [slow]spitfire[cstringio]: Spitfire is a template language, the >>>>> cstringio version uses a modified engine (that uses cstringio) >>>> >>>>> spambayes: Spambayes is a bayesian spam filter >>>> >>>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I >>>> remember when, because of this, spambayes was slower (including >>>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4 >>>> and the head (for slowspitfire it's the opposite). >>>> >>>> For the rest, I see no significant case of slowdown of PyPy over time. >>>> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars >>>> -- >>>> Paolo Giarrusso - Ph.D. Student >>>> http://www.informatik.uni-marburg.de/~pgiarrusso/ >>>> >>> >> > > > > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > From alex.gaynor at gmail.com Wed Dec 22 09:33:08 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 22 Dec 2010 02:33:08 -0600 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: ai: runs a bruce force n-queens solver Alex On Wed, Dec 22, 2010 at 2:26 AM, Miquel Torres wrote: > so, what about the ai and spectral-norm benchmarks. Anybody can come > up with a description for them? > > > 2010/12/15 Paolo Giarrusso : > > On Mon, Dec 13, 2010 at 09:31, Miquel Torres > wrote: > >> Oh, btw., the "normalized" stacked bars now display a warning note > >> about its correctness, and how it must be viewed as giving results a > >> weighting instead of them being normalized. It even includes a link to > >> the proper paper. I hope that is enough for the strict statisticians > >> among us ;-) > > > > I see. Thanks! > > > >> See: > >> > http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars > >> > >> PS: there is a bug in the jqPlot plotting library when null values are > >> present. Trying to display PyPy 1.3 results for the newer go, pyflake > >> or raytrace will create some nasty js loops. It also has problems > >> with autoscaling the axis sometimes. > >> > >> > >> > >> 2010/12/13 Miquel Torres : > >>> Thanks all for the input. > >>> I've compiled a list based on your mails, the Unladen benchmarks page > >>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the > >>> alioth descriptions. Here is an extract of the current speed.pypy.org > >>> admin: > >>> > >>> ai > >>> chaos Creates chaosgame-like fractals > >>> crypto_pyaes A pure python implementation of AES > >>> django Uses the Django template system to build a 150x150-cell > HTML table > >>> > >>> fannkuch Indexed-access to tiny integer-sequence. The > fannkuch > >>> benchmark is defined by programs in Performing Lisp Analysis of the > >>> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig. > >>> > >>> float Creates an array of points using circular projection > and then > >>> normalizes and maximizes them. Floating-point heavy. > >>> go A go (chess like game) computer player AI. > >>> html5lib Parses the HTML 5 spec using html5lib > >>> meteor-contest Searchs for solutions to shape packing puzzle. > >>> nbody_modified Double-precision N-body simulation. It models > the > >>> orbits of Jovian planets, using a simple symplectic-integrator. > >>> pyflate-fast Stand-alone pure-Python DEFLATE (gzip) and > bzip2 > >>> decoder/decompressor. > >>> raytrace-simple A raytracer renderer > >>> richards Medium-sized language benchmark that simulates > the task > >>> dispatcher in the kernel of an operating system. > >>> rietveld A Django application benchmark. > >>> slowspitfire > >>> spambayes Runs a canned mailbox through a SpamBayes ham/spam > classifier > >>> spectral-norm > >>> spitfire Uses the Spitfire template system to build a > 1000x1000-cell HTML table. > >>> spitfire_cstringio Uses the Spitfire template system to build a > >>> 1000x1000-cell HTML table, using the cstringio module. > >>> telco > >>> twisted_iteration > >>> twisted_names > >>> twisted_pb > >>> twisted_tcp Connects one Twised client to one Twisted server over > TCP > >>> (on the loopback interface) and then writes bytes as fast as it can. > >>> waf Python-based framework for configuring, compiling and > installing > >>> applications. It derives from the concepts of other build tools such > >>> as Scons, Autotools, CMake or Ant. > >>> > >>> > >>> So the remaining descriptions are > >>> ai > >>> slowspitfire (what is the exact difference between the three spitfire > benches?) > >>> spectral-norm > >>> telco > >>> twisted (most of them) > >>> > >>> Are the descriptions all right so far?. They can be made much longer > >>> if you deem it desirable. > >>> > >>> on speed.pypy.org you will currently see the descriptions in 3 places: > >>> - Changes view: A tooltip on hover over each benchmark > >>> - Timeline: a description box beneath each plot > >>> - Comparison: A tooltip over each benchmark when hovering the > >>> selection menu on the left side. > >>> > >>> Any suggestions on how to improve it further are welcome ;-) > >>> > >>> Miquel > >>> > >>> > >>> 2010/12/9 Paolo Giarrusso : > >>>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada > wrote: > >>>>> Here is a incomplete draft list: > >>>> > >>>>> [slow]spitfire[cstringio]: Spitfire is a template language, the > >>>>> cstringio version uses a modified engine (that uses cstringio) > >>>> > >>>>> spambayes: Spambayes is a bayesian spam filter > >>>> > >>>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I > >>>> remember when, because of this, spambayes was slower (including > >>>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4 > >>>> and the head (for slowspitfire it's the opposite). > >>>> > >>>> For the rest, I see no significant case of slowdown of PyPy over time. > >>>> > http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars > >>>> -- > >>>> Paolo Giarrusso - Ph.D. Student > >>>> http://www.informatik.uni-marburg.de/~pgiarrusso/ > >>>> > >>> > >> > > > > > > > > -- > > Paolo Giarrusso - Ph.D. Student > > http://www.informatik.uni-marburg.de/~pgiarrusso/ > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101222/7f377bee/attachment.htm From tobami at googlemail.com Wed Dec 22 11:19:29 2010 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 22 Dec 2010 11:19:29 +0100 Subject: [pypy-dev] Idea for speed.pypy.org In-Reply-To: References: <201012040043.04547.jacob@openend.se> <240465.94379.qm@web111314.mail.gq1.yahoo.com> <201012041105.oB4B5uo1019532@theraft.openend.se> Message-ID: @Alex: added! 2010/12/22 Alex Gaynor : > ai: runs a bruce force n-queens solver > > Alex > > On Wed, Dec 22, 2010 at 2:26 AM, Miquel Torres > wrote: >> >> so, what about the ai and spectral-norm benchmarks. Anybody can come >> up with a description for them? >> >> >> 2010/12/15 Paolo Giarrusso : >> > On Mon, Dec 13, 2010 at 09:31, Miquel Torres >> > wrote: >> >> Oh, btw., the "normalized" stacked bars now display a warning note >> >> about its correctness, and how it must be viewed as giving results a >> >> weighting instead of them being normalized. It even includes a link to >> >> the proper paper. I hope that is enough for the strict statisticians >> >> among us ;-) >> > >> > I see. Thanks! >> > >> >> See: >> >> >> >> http://speed.pypy.org/comparison/?exe=1%2B172,3%2B172,1%2BL,3%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20&env=1&hor=true&bas=2%2B35&chart=stacked+bars >> >> >> >> PS: there is a bug in the jqPlot plotting library when null values are >> >> present. Trying to display PyPy 1.3 results for the newer go, pyflake >> >> or ?raytrace will create some nasty js loops. It also has problems >> >> with autoscaling the axis sometimes. >> >> >> >> >> >> >> >> 2010/12/13 Miquel Torres : >> >>> Thanks all for the input. >> >>> I've compiled a list based on your mails, the Unladen benchmarks page >> >>> (http://code.google.com/p/unladen-swallow/wiki/Benchmarks), and the >> >>> alioth descriptions. Here is an extract of the current speed.pypy.org >> >>> admin: >> >>> >> >>> ai >> >>> chaos ? Creates chaosgame-like fractals >> >>> crypto_pyaes ? ?A pure python implementation of AES >> >>> django ? ? ? ? ?Uses the Django template system to build a >> >>> 150x150-cell HTML table >> >>> >> >>> fannkuch ? ? ? ? ? ? ? ?Indexed-access to tiny integer-sequence. The >> >>> fannkuch >> >>> benchmark is defined by programs in Performing Lisp Analysis of the >> >>> FANNKUCH Benchmark, Kenneth R. Anderson and Duane Rettig. >> >>> >> >>> float ? ? ? ? ? Creates an array of points using circular projection >> >>> and then >> >>> normalizes and maximizes them. Floating-point heavy. >> >>> go ? ? ? ? ? ? ?A go (chess like game) computer player AI. >> >>> html5lib ? ? ? ?Parses the HTML 5 spec using html5lib >> >>> meteor-contest ?Searchs for solutions to shape packing puzzle. >> >>> nbody_modified ? ? ? ? ?Double-precision N-body simulation. It models >> >>> the >> >>> orbits of Jovian planets, using a simple symplectic-integrator. >> >>> pyflate-fast ? ? ? ? ? ?Stand-alone pure-Python DEFLATE (gzip) and >> >>> bzip2 >> >>> decoder/decompressor. >> >>> raytrace-simple A raytracer renderer >> >>> richards ? ? ? ? ? ? ? ?Medium-sized language benchmark that simulates >> >>> the task >> >>> dispatcher in the kernel of an operating system. >> >>> rietveld ? ? ? ?A Django application benchmark. >> >>> slowspitfire >> >>> spambayes ? ? ? Runs a canned mailbox through a SpamBayes ham/spam >> >>> classifier >> >>> spectral-norm >> >>> spitfire ? ? ? ?Uses the Spitfire template system to build a >> >>> 1000x1000-cell HTML table. >> >>> spitfire_cstringio ? ? ?Uses the Spitfire template system to build a >> >>> 1000x1000-cell HTML table, using the cstringio module. >> >>> telco >> >>> twisted_iteration >> >>> twisted_names >> >>> twisted_pb >> >>> twisted_tcp ? ? Connects one Twised client to one Twisted server over >> >>> TCP >> >>> (on the loopback interface) and then writes bytes as fast as it can. >> >>> waf ? ? Python-based framework for configuring, compiling and >> >>> installing >> >>> applications. It derives from the concepts of other build tools such >> >>> as Scons, Autotools, CMake or Ant. >> >>> >> >>> >> >>> So the remaining descriptions are >> >>> ai >> >>> slowspitfire (what is the exact difference between the three spitfire >> >>> benches?) >> >>> spectral-norm >> >>> telco >> >>> twisted (most of them) >> >>> >> >>> Are the descriptions all right so far?. They can be made much longer >> >>> if you deem it desirable. >> >>> >> >>> on speed.pypy.org you will currently see the descriptions in 3 places: >> >>> - Changes view: A tooltip on hover over each benchmark >> >>> - Timeline: a description box beneath each plot >> >>> - Comparison: A tooltip over each benchmark when hovering the >> >>> selection menu on the left side. >> >>> >> >>> Any suggestions on how to improve it further are welcome ;-) >> >>> >> >>> Miquel >> >>> >> >>> >> >>> 2010/12/9 Paolo Giarrusso : >> >>>> On Thu, Dec 9, 2010 at 14:14, Leonardo Santagada >> >>>> wrote: >> >>>>> Here is a incomplete draft list: >> >>>> >> >>>>> [slow]spitfire[cstringio]: Spitfire is a template language, the >> >>>>> cstringio version uses a modified engine (that uses cstringio) >> >>>> >> >>>>> spambayes: Spambayes is a bayesian spam filter >> >>>> >> >>>> Why is [slow]spitfire slower with PyPy? Is it regex-related? I >> >>>> remember when, because of this, spambayes was slower (including >> >>>> release 1.3, now solved). But for spitfire, 1.3 was faster than 1.4 >> >>>> and the head (for slowspitfire it's the opposite). >> >>>> >> >>>> For the rest, I see no significant case of slowdown of PyPy over >> >>>> time. >> >>>> >> >>>> http://speed.pypy.org/comparison/?exe=2%2B35,1%2B41,1%2B172,1%2BL&ben=1,2,25,3,4,5,22,6,7,8,23,24,9,10,11,12,13,14,15,16,17,18,19,20,26&env=1&hor=true&bas=2%2B35&chart=normal+bars >> >>>> -- >> >>>> Paolo Giarrusso - Ph.D. Student >> >>>> http://www.informatik.uni-marburg.de/~pgiarrusso/ >> >>>> >> >>> >> >> >> > >> > >> > >> > -- >> > Paolo Giarrusso - Ph.D. Student >> > http://www.informatik.uni-marburg.de/~pgiarrusso/ >> > >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > "Code can always be simpler than you think, but never as simple as you want" > -- Me > From lac at openend.se Wed Dec 22 12:10:07 2010 From: lac at openend.se (Laura Creighton) Date: Wed, 22 Dec 2010 12:10:07 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: Message from Miquel Torres of "Wed, 22 Dec 2010 09:24:28 +0100." References: Message-ID: <201012221110.oBMBA7Q8021121@theraft.openend.se> In a message of Wed, 22 Dec 2010 09:24:28 +0100, Miquel Torres writes: >Btw., instead of continuing here "polluting" the pypy-dev mailing >list, we can move to http://groups.google.com/group/codespeed if >needed. I'd rather it stayed here, myself. Laura From chef at ghum.de Wed Dec 22 12:15:32 2010 From: chef at ghum.de (Massa, Harald Armin) Date: Wed, 22 Dec 2010 12:15:32 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: > > Mmm, yes, it appears that the memory is not properly freed. I can > try experimenting by destroying the dom element and recreating it each time > that the plot changes... > > Btw., instead of continuing here "polluting" the pypy-dev mailing > list, we can move to http://groups.google.com/group/codespeed if needed. > > I recommend to keep it here ... if some of the most important browser vendors have challenges with memory, everybody with memory...challenges can learn from the discussion. Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101222/f63819e6/attachment.htm From Arnd.Rechenburg at tomtom.com Wed Dec 22 18:00:10 2010 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Wed, 22 Dec 2010 18:00:10 +0100 Subject: [pypy-dev] Fatal RPython error: AssertionError Message-ID: Hi, I have the following problem: Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48) [PyPy 1.4.1] on linux2 RPython traceback: File "implement_54.c", line 21229, in send_bridge_to_backend File "implement_61.c", line 14547, in Assembler386_assemble_bridge File "implement_63.c", line 44979, in RegAlloc_prepare_bridge File "implement_64.c", line 15487, in RegAlloc__update_bindings File "implement_38.c", line 60625, in RegisterManager__check_invariants Fatal RPython error: AssertionError Any help would be appreciated. Regards, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101222/dba27e35/attachment.htm From Arnd.Rechenburg at tomtom.com Wed Dec 22 18:24:31 2010 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Wed, 22 Dec 2010 18:24:31 +0100 Subject: [pypy-dev] Fatal RPython error: AssertionError Message-ID: Hi again, It was my mistake. The problem was in 1.4.0 and is solved in 1.4.1 Regards, Arnd ________________________________ From: Arnd Rechenburg Sent: Mittwoch, 22. Dezember 2010 18:00 To: 'pypy-dev at codespeak.net' Subject: Fatal RPython error: AssertionError Hi, I have the following problem: Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48) [PyPy 1.4.1] on linux2 RPython traceback: File "implement_54.c", line 21229, in send_bridge_to_backend File "implement_61.c", line 14547, in Assembler386_assemble_bridge File "implement_63.c", line 44979, in RegAlloc_prepare_bridge File "implement_64.c", line 15487, in RegAlloc__update_bindings File "implement_38.c", line 60625, in RegisterManager__check_invariants Fatal RPython error: AssertionError Any help would be appreciated. Regards, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101222/db3949b3/attachment.htm From garyrob at me.com Wed Dec 22 22:34:12 2010 From: garyrob at me.com (Gary Robinson) Date: Wed, 22 Dec 2010 16:34:12 -0500 Subject: [pypy-dev] dlopen NotImplementedError Message-ID: <9C59B64A-15B8-4965-86B5-EC0DA57183A2@me.com> So, as of today, I'm starting to do as much of my day-to-day work in pypy as I can. The missing numpy is a problem, but one I can work around a enough amount of the time to make this effort worth my while. I don't know how, or even whether, I should report certain kinds of problems I might run into. I would appreciate some feedback on that. I want to help by giving feedback in whatever form, and to what degree, the pypy devs desire. For example, I can run some scripts in pypy 1.4 but not in pypy fast-forward because of an error involving dlopen(). Other scripts are unaffected by this problem, depending on what needs to be imported. When I try to run fast-forward in interactive mode, I get the same error, which is pasted below. Python 2.7.0 (8c2427fa1a01, Dec 19 2010, 15:58:12) [PyPy 1.4.0 with GCC 4.1.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "app_main.py", line 53, in run_toplevel File "app_main.py", line 522, in run_it File "/etc/pythonstart", line 7, in import readline File "/root/pypy/lib_pypy/readline.py", line 9, in File "/root/pypy/lib_pypy/pyrepl/readline.py", line 33, in File "/root/pypy/lib_pypy/pyrepl/unix_console.py", line 25, in File "/root/pypy/lib_pypy/pyrepl/curses.py", line 30, in File "/root/pypy/lib_pypy/_minimal_curses.py", line 12, in File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 441, in File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 353, in __init__ File "/root/pypy/lib_pypy/_ctypes/dll.py", line 7, in dlopen NotImplementedError: dlopen(None) And now for something completely different: ``- do you know berlin? - all of it? - well, just berlin'' Traceback (most recent call last): File "app_main.py", line 53, in run_toplevel File "/root/pypy/lib_pypy/_pypy_interact.py", line 27, in interactive_console File "/root/pypy/lib_pypy/pyrepl/simple_interact.py", line 27, in File "/root/pypy/lib_pypy/pyrepl/readline.py", line 33, in File "/root/pypy/lib_pypy/pyrepl/unix_console.py", line 25, in File "/root/pypy/lib_pypy/pyrepl/curses.py", line 30, in File "/root/pypy/lib_pypy/_minimal_curses.py", line 12, in File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 441, in File "/root/pypy/lib-python/modified-2.7.0/ctypes/__init__.py", line 353, in __init__ File "/root/pypy/lib_pypy/_ctypes/dll.py", line 7, in dlopen NotImplementedError: dlopen(None) -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net From garyrob at me.com Wed Dec 22 22:52:32 2010 From: garyrob at me.com (Gary Robinson) Date: Wed, 22 Dec 2010 16:52:32 -0500 Subject: [pypy-dev] dlopen NotImplementedError Message-ID: Hmm, I'm thinking the dlopen error might be my problem. I think I built python at /root/pypy, and then moved it to a more proper location. But the error message is referring to /root/pypy, so I'm guessing it has that location hardcoded because that's where I built it? I'm going to rebuild but I wanted to post this message so that you guys don't worry about it (unless rebuilding doesn't help). -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net From alex.gaynor at gmail.com Wed Dec 22 22:54:37 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 22 Dec 2010 15:54:37 -0600 Subject: [pypy-dev] dlopen NotImplementedError In-Reply-To: References: Message-ID: On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson wrote: > Hmm, I'm thinking the dlopen error might be my problem. I think I built > python at /root/pypy, and then moved it to a more proper location. But the > error message is referring to /root/pypy, so I'm guessing it has that > location hardcoded because that's where I built it? > > I'm going to rebuild but I wanted to post this message so that you guys > don't worry about it (unless rebuilding doesn't help). > > -- > > Gary Robinson > CTO > Emergent Discovery, LLC > personal email: garyrob at me.com > work email: grobinson at emergentdiscovery.com > Company: http://www.emergentdiscovery.com > Blog: http://www.garyrobinson.net > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Nope, it's a known thing, anything trying to import ctypes on fast-forward blows up ATM. If anyone knows what dlopen(None) is supposed to mean, can they let me know and I'll try to take a pass at fixing this? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101222/fa40c797/attachment.htm From dimaqq at gmail.com Wed Dec 22 23:21:52 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 22 Dec 2010 15:21:52 -0700 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Firefox people point the blame squarely at jqplot: --- Comment #4 from Boris Zbarsky (:bz) 2010-12-22 13:43:23 PST --- So what I see this page do, in horizontal mode, is create 17 canvases each of which is width="816" height="3587". That means that each of them has a backing store of 816*3587*4 = 11,707,968 bytes. So that's about 200MB of memory usage right there. I have no idea why they feel a need for 17 huge canvases, but if they want them, that's how much memory they'll take... ( https://bugzilla.mozilla.org/show_bug.cgi?id=620883 ) On 22 December 2010 01:24, Miquel Torres wrote: > Mmm, yes, it appears that the memory is not properly freed. I can try > experimenting by destroying the dom element and recreating it each > time that the plot changes... > > Btw., instead of continuing here "polluting" the pypy-dev mailing > list, we can move to http://groups.google.com/group/codespeed if > needed. > > Thanks for reporting this. > > Cheers, > Miquel > > > 2010/12/22 Dima Tisnek : >> I sort of figured it out, although I don't have a ready solution. >> >> This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8. >> Basically memory consumption of the plot appears to be proportional to >> plot area. >> Normal plot that you get by default at >> http://speed.pypy.org/comparison/ costs about 100M of browser memory >> consumption: >> opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050; >> opera ?80M, stable ff 55M, beta ff 70M at window size 1024x600; >> Switching to "horizontal" produces a tall plot of same width and costs >> about 300~700M of browser memory: >> opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050; >> opera 350M, stable ff 370M, beta ff 370M at window size 1024x600; >> >> Suprisingly window size only matters while javascript produces the >> plot, and not when window is resized, even though plot it resized with >> the window correctly. >> >> This alone is pretty heavy, but doesn't grind the browser. >> What really grinds is that every time you change a tickbox on the >> left, a plot is redrawn and another 200M of browser memory is wasted. >> This is not double buffering, as next change adds yet another 200M or >> so and so on, it appears that either js doesn't free something, or >> browser caches or saves the previous page state. >> >> As memory consumption grows, at some point browser hits the wall, >> causes heavy swapping for some time, and I think garbage collection, >> because practical (but not virtual) memory usage first drops to >> something like 20~50M and then returns to "normal" 300M. >> opera ~30 seconds, stable ff about a minute, beta ff several minutes >> (total system mem 1G, cpu Atom @1.6GHz) >> >> Perhaps OS also plays a role in the grind, as it is clearly swapping >> and swaps out too much? or triggers gc too late and gc has to pull the >> pages back from disk to perform collection? >> >> ie8 doesn't use that much memory, as a matter of fact memory >> consumption starts little (40M) and changes very little (only +10M) if >> you go to horizonatal view; the price is very slow rendering, more >> than 10 seconds per column change. >> >> I'll post this on firefox bugzilla too, let's see if someone has a solution. >> >> Meanwhile perhaps pypy speed center could start with a smaller plot >> area (or fewer columns as that makes horizontal plot smaller) to >> accomodate varying hardware and system mem usage that users might >> have? >> The simplest would be a warning next to "horizontal" checkbox. >> >> On 21 December 2010 01:06, Miquel Torres wrote: >>> Hi Dima, >>> >>>> another temp problem with speed pypy is that it's terrubly slow in ff >>>> beta. it also occasionally grinds in stable ff and opera, but I guess >>>> this can be forgiven for the sake of simplicity / developer effort. >>> >>> Well, speed.pypy is actually fast in all modern browsers. The problem >>> you are referring to is probably caused by a bug in the javascript >>> plotting library (jqPplot) that is triggered in the comparison view >>> when there are some results with 0 values. It only appears for some >>> plot types, but it is very annoying because it grinds the browser to a >>> halt like you say. Is that what you meant? >>> >>> We are looking into it, and will fix that library if necessary. >>> >>> Cheers, >>> Miquel >>> >>> >>> 2010/12/21 Dima Tisnek : >>>> On 20 December 2010 19:21, William ML Leslie >>>> wrote: >>>>> On 21 December 2010 11:59, Dima Tisnek wrote: >>>>>> More visibility for performance achievements would do >>>>>> good too. >>>>> >>>>> Where are pypy's performance achievements *not* visible, but should be? >>>> >>>> for example http://shootout.alioth.debian.org/ >>>> doesn't say which pypy version is used, what options, doesn't have >>>> performance figures for multithreaded/multicore >>>> >>>> also benchmarks are kinda small, most of them are not docuemented, and >>>> I couldn't find any info if the same python code was used for cpython >>>> and pypy (both shootout and speed pypy) or interpreter-specific >>>> verions were used, that is each tweaked for best performance given the >>>> known tradeoffs for each implementation.further the most standard >>>> benchmarks, pystone, etc. completely ignore the fact that real world >>>> programs are large and only a few small paths are execured often. >>>> >>>> another temp problem with speed pypy is that it's terrubly slow in ff >>>> beta. it also occasionally grinds in stable ff and opera, but I guess >>>> this can be forgiven for the sake of simplicity / developer effort. >>>> >>>> if you google for 'python performance' you don't get a single link to >>>> pypy on the first page, as a matter of fact, codespeak is poorly >>>> indexed, it took me quite some time to get some of my questions >>>> answered with a search. also if you look up 'pypy gc' you get a page >>>> on codespeak, but to interpret what the data actually means is so far >>>> beyond me. >>>> >>>> a good overview is found in the mainling list >>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >>>> slowspitfire and spambayes bits are outdated by now. >>>> >>>> the definitive good thing about pypy is that it's much easier to find >>>> out about its inner workings than that of cpython! >>>> >>>> hopefully a bit more of end-user stuff get known. >>>> let's call it pypy public outreach (feature request) >>>> >>>>> >>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>>>>> hard would it be to use same techniques and most of backends for some >>>>>> unrelated language that doesn't have jit yet, e.g. php? >>>>> >>>>> You know that pypy already has implementations of other languages, >>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>>>> integrated with the translated pypy-c, but they show that it is not >>>>> too difficult to write a runtime for any dynamic language you choose. >>>> >>>> Oh I didn't know there were so many, and I mistakenly though that js >>>> was a target, not implmented langauge. In any case I read somewhere >>>> that js support was retired... >>>> >>>>> >>>>>> And how hard >>>>>> would it be to marry two dynamic languages, so that modules from one >>>>>> could be used in the other? Or that modules written in rpython could >>>>>> be used in several langs? >>>>> >>>>> It's in the "interesting problems" bucket, and the effort required >>>>> depends on the level of integration between languages you want. ?There >>>>> are several projects already attempting to do decent integration >>>>> between several languages, besides the approach used on the JVM, there >>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be >>>>> worth waiting to see how these different projects pan out, before >>>>> spending a bunch of effort just to be an also-ran in the >>>>> multi-language runtime market. >>>>> >>>>> However, implementing more languages in rpython has the side-effect of >>>>> propagating the l * o * p problem: it introduces more and more >>>>> implementations that then have to be maintained, so good >>>>> cross-language integration probably belongs /outside/ pypy itself, so >>>>> existing runtimes can hook into it. >>>> >>>> Makes perfect sense, after all any given other language hardly has the >>>> same data model as python. >>>> >>>>> >>>>> But it would be an interesting experiment (to marry the various >>>>> interpreters pypy ships with), if you wanted to try it. >>>>> >>>>> My two cents. >>>>> >>>>> -- >>>>> William Leslie >>>>> >>>> _______________________________________________ >>>> pypy-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> > From exarkun at twistedmatrix.com Wed Dec 22 23:41:27 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 22 Dec 2010 22:41:27 -0000 Subject: [pypy-dev] dlopen NotImplementedError In-Reply-To: References: Message-ID: <20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain> On 09:54 pm, alex.gaynor at gmail.com wrote: >On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson wrote: >>Hmm, I'm thinking the dlopen error might be my problem. I think I >>built >>python at /root/pypy, and then moved it to a more proper location. But >>the >>error message is referring to /root/pypy, so I'm guessing it has that >>location hardcoded because that's where I built it? >> >>I'm going to rebuild but I wanted to post this message so that you >>guys >>don't worry about it (unless rebuilding doesn't help). >Nope, it's a known thing, anything trying to import ctypes on fast- >forward >blows up ATM. If anyone knows what dlopen(None) is supposed to mean, >can >they let me know and I'll try to take a pass at fixing this? From the dlopen man page: If filename is NULL, then the returned handle is for the main program. I imagine that's the case someone is trying to trigger with ctypes.LoadLibrary(None) (which is what the name `dlopen` is bound to in this context). On the other hand, maybe it's just a screw up somewhere else that causes None to show up instead of the right library name, I haven't looked at the code in question much. Jean-Paul From alex.gaynor at gmail.com Wed Dec 22 23:45:03 2010 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 22 Dec 2010 16:45:03 -0600 Subject: [pypy-dev] dlopen NotImplementedError In-Reply-To: <20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain> References: <20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain> Message-ID: On Wed, Dec 22, 2010 at 4:41 PM, wrote: > On 09:54 pm, alex.gaynor at gmail.com wrote: > >On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson wrote: > >>Hmm, I'm thinking the dlopen error might be my problem. I think I > >>built > >>python at /root/pypy, and then moved it to a more proper location. But > >>the > >>error message is referring to /root/pypy, so I'm guessing it has that > >>location hardcoded because that's where I built it? > >> > >>I'm going to rebuild but I wanted to post this message so that you > >>guys > >>don't worry about it (unless rebuilding doesn't help). > >Nope, it's a known thing, anything trying to import ctypes on fast- > >forward > >blows up ATM. If anyone knows what dlopen(None) is supposed to mean, > >can > >they let me know and I'll try to take a pass at fixing this? > > From the dlopen man page: > > If filename is NULL, then the returned handle is for the main program. > > I imagine that's the case someone is trying to trigger with > ctypes.LoadLibrary(None) (which is what the name `dlopen` is bound to in > this context). On the other hand, maybe it's just a screw up somewhere > else that causes None to show up instead of the right library name, I > haven't looked at the code in question much. > > Jean-Paul > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Ok I see the issue I think. It's coming from PyDLL(None). Not sure what the right solution is there, I know Amaury said he was working on the cpythonapi stuff. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101222/883b2ae5/attachment.htm From dimaqq at gmail.com Thu Dec 23 03:36:59 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 22 Dec 2010 19:36:59 -0700 Subject: [pypy-dev] Fwd: pypy-dev Re: funding/popularity? In-Reply-To: <765473.56989.qm@web65612.mail.ac4.yahoo.com> References: <765473.56989.qm@web65612.mail.ac4.yahoo.com> Message-ID: ---------- Forwarded message ---------- From: Isaac Gouy Dima Tisnek gmail.com> writes: > > On 20 December 2010 19:21, William ML Leslie > gmail.com> wrote: > > On 21 December 2010 11:59, Dima Tisnek gmail.com> wrote: > >> More visibility for performance achievements would do > >> good too. > > > > Where are pypy's performance achievements *not* visible, but should be? > > for example http://shootout.alioth.debian.org/ > doesn't say which pypy version is used Oh yes it does - http://shootout.alioth.debian.org/u32/compare.php?lang=pypy#about > what options, doesn't have > performance figures for multithreaded/multicore All the Python multiprocessing programs "Failed" with PyPy - http://shootout.alioth.debian.org/u32/measurements.php?lang=pypy > also benchmarks are kinda small, most of them are not docuemented, and > I couldn't find any info if the same python code was used for cpython > and pypy (both shootout and speed pypy) or interpreter-specific > verions were used, that is each tweaked for best performance given the > known tradeoffs for each implementation. The full source code is shown for all the programs - http://shootout.alioth.debian.org/u32/program.php?test=spectralnorm?=pypy&id=6 http://shootout.alioth.debian.org/u32/program.php?test=spectralnorm?=python&id=6 The GZip byte size is shown for each program, in this case 341 for both programs. Maybe the benchmarks game shows more information than you realize. From dimaqq at gmail.com Thu Dec 23 03:46:19 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 22 Dec 2010 19:46:19 -0700 Subject: [pypy-dev] 1.4.1 threading Message-ID: Hi, Somehow I'm getting exactly same performance in threaded and non-threaded code (I would expect at least some speedup), Is threading not there in this pypy release in practice or am I missing some argument to pypy? the test case is cpu-bound in python, and possibly gc-bound too From benjamin at python.org Thu Dec 23 03:49:57 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 22 Dec 2010 20:49:57 -0600 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: 2010/12/22 Dima Tisnek : > Hi, > > Somehow I'm getting exactly same performance in threaded and > non-threaded code (I would expect at least some speedup), > Is threading not there in this pypy release in practice or am I > missing some argument to pypy? PyPy has a GIL if that's what you mean. -- Regards, Benjamin From dimaqq at gmail.com Thu Dec 23 04:01:14 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 22 Dec 2010 20:01:14 -0700 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: Oh, I can't yet use alternative gc that obviates GIL then? Or am I totally confused and pypy still uses GIL for other reasons, e.g. globals dict safety? On 22 December 2010 19:49, Benjamin Peterson wrote: > 2010/12/22 Dima Tisnek : >> Hi, >> >> Somehow I'm getting exactly same performance in threaded and >> non-threaded code (I would expect at least some speedup), >> Is threading not there in this pypy release in practice or am I >> missing some argument to pypy? > > PyPy has a GIL if that's what you mean. > > > > -- > Regards, > Benjamin > From benjamin at python.org Thu Dec 23 04:05:26 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 22 Dec 2010 21:05:26 -0600 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: 2010/12/22 Dima Tisnek : > Oh, I can't yet use alternative gc that obviates GIL then? > > Or am I totally confused and pypy still uses GIL for other reasons, > e.g. globals dict safety? All of the above. -- Regards, Benjamin From sungsuha at gmail.com Thu Dec 23 06:51:26 2010 From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha) Date: Thu, 23 Dec 2010 05:51:26 +0000 (UTC) Subject: [pypy-dev] Building PyPy on Windows Message-ID: An interesting article came up on python planet: http://www.redmountainsw.com/wordpress/archives/building-pypy-on-windows Maybe with the consent of the original author and some cleaning up, we could add this to official documentation? A simple link on the getting started/download pages won't hurt windows users either. From drsalists at gmail.com Thu Dec 23 06:58:55 2010 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 22 Dec 2010 21:58:55 -0800 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: On Wed, Dec 22, 2010 at 7:05 PM, Benjamin Peterson wrote: > 2010/12/22 Dima Tisnek : >> Oh, I can't yet use alternative gc that obviates GIL then? >> >> Or am I totally confused and pypy still uses GIL for other reasons, >> e.g. globals dict safety? > > All of the above. On the topic of parallel dict safety... I've not played around with parallel dictionaries at all, but I've heard that treaps can be parallelized nicely, and they can be (and have been) set up to look much like a python dictionary. I admit, I coded one and put it at http://stromberg.dnsalias.org/~strombrg/treap/ - however, the implementation is not (yet?) suitable for parallel use. It's coded in such a way that it'll work as pure python or cython, depending on which of two m4 symbols you define. I've used the pure python version on pypy. Treaps tend to make just about everything O(log(n)), have good average performance, and are not as even performers as red-black trees (that is, they give a higher standard deviation, and a better average performance than red-black trees). And they don't play that nicely with locality of reference (caches) - things are intentionally randomized. But if you throw enough cores at them, they might still be a win compared to a big hash table with a big lock wrapped around it. From fijall at gmail.com Thu Dec 23 07:50:58 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 23 Dec 2010 08:50:58 +0200 Subject: [pypy-dev] dlopen NotImplementedError In-Reply-To: References: <20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain> Message-ID: CDLL(None) (or LoadLibrary(None)) is supposed to return to you the whole namespace (all loaded libraries). It's unsupported as far as I can tell. On Thu, Dec 23, 2010 at 12:45 AM, Alex Gaynor wrote: > > > On Wed, Dec 22, 2010 at 4:41 PM, wrote: >> >> On 09:54 pm, alex.gaynor at gmail.com wrote: >> >On Wed, Dec 22, 2010 at 3:52 PM, Gary Robinson wrote: >> >>Hmm, I'm thinking the dlopen error might be my problem. I think I >> >>built >> >>python at /root/pypy, and then moved it to a more proper location. But >> >>the >> >>error message is referring to /root/pypy, so I'm guessing it has that >> >>location hardcoded because that's where I built it? >> >> >> >>I'm going to rebuild but I wanted to post this message so that you >> >>guys >> >>don't worry about it (unless rebuilding doesn't help). >> >Nope, it's a known thing, anything trying to import ctypes on fast- >> >forward >> >blows up ATM. ?If anyone knows what dlopen(None) is supposed to mean, >> >can >> >they let me know and I'll try to take a pass at fixing this? >> >> ?From the dlopen man page: >> >> ?If filename is NULL, then the returned handle is for the main program. >> >> I imagine that's the case someone is trying to trigger with >> ctypes.LoadLibrary(None) (which is what the name `dlopen` is bound to in >> this context). ?On the other hand, maybe it's just a screw up somewhere >> else that causes None to show up instead of the right library name, I >> haven't looked at the code in question much. >> >> Jean-Paul >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > Ok I see the issue I think.? It's coming from PyDLL(None).? Not sure what > the right solution is there, I know Amaury said he was working on the > cpythonapi stuff. > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > "Code can always be simpler than you think, but never as simple as you want" > -- Me > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tobami at googlemail.com Thu Dec 23 08:29:40 2010 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 23 Dec 2010 08:29:40 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Ok, so it seems from reading comments further down that bug report that FF's garbage collection does not immediately free memory for old jqplot canvases. As I said, as soon as I find the time I will experiment with more aggressive DOM element destruction/creation and see if it improves how quickly memory is freed. I do think it will at least partly circumvent jqPlot's bug. 2010/12/22 Dima Tisnek : > Firefox people point the blame squarely at jqplot: > > --- Comment #4 from Boris Zbarsky (:bz) 2010-12-22 > 13:43:23 PST --- > So what I see this page do, in horizontal mode, is create 17 canvases each of > which is width="816" height="3587". ?That means that each of them has a backing > store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage > right there. > > I have no idea why they feel a need for 17 huge canvases, but if they want > them, that's how much memory they'll take... > > ( https://bugzilla.mozilla.org/show_bug.cgi?id=620883 ) > > On 22 December 2010 01:24, Miquel Torres wrote: >> Mmm, yes, it appears that the memory is not properly freed. I can try >> experimenting by destroying the dom element and recreating it each >> time that the plot changes... >> >> Btw., instead of continuing here "polluting" the pypy-dev mailing >> list, we can move to http://groups.google.com/group/codespeed if >> needed. >> >> Thanks for reporting this. >> >> Cheers, >> Miquel >> >> >> 2010/12/22 Dima Tisnek : >>> I sort of figured it out, although I don't have a ready solution. >>> >>> This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8. >>> Basically memory consumption of the plot appears to be proportional to >>> plot area. >>> Normal plot that you get by default at >>> http://speed.pypy.org/comparison/ costs about 100M of browser memory >>> consumption: >>> opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050; >>> opera ?80M, stable ff 55M, beta ff 70M at window size 1024x600; >>> Switching to "horizontal" produces a tall plot of same width and costs >>> about 300~700M of browser memory: >>> opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050; >>> opera 350M, stable ff 370M, beta ff 370M at window size 1024x600; >>> >>> Suprisingly window size only matters while javascript produces the >>> plot, and not when window is resized, even though plot it resized with >>> the window correctly. >>> >>> This alone is pretty heavy, but doesn't grind the browser. >>> What really grinds is that every time you change a tickbox on the >>> left, a plot is redrawn and another 200M of browser memory is wasted. >>> This is not double buffering, as next change adds yet another 200M or >>> so and so on, it appears that either js doesn't free something, or >>> browser caches or saves the previous page state. >>> >>> As memory consumption grows, at some point browser hits the wall, >>> causes heavy swapping for some time, and I think garbage collection, >>> because practical (but not virtual) memory usage first drops to >>> something like 20~50M and then returns to "normal" 300M. >>> opera ~30 seconds, stable ff about a minute, beta ff several minutes >>> (total system mem 1G, cpu Atom @1.6GHz) >>> >>> Perhaps OS also plays a role in the grind, as it is clearly swapping >>> and swaps out too much? or triggers gc too late and gc has to pull the >>> pages back from disk to perform collection? >>> >>> ie8 doesn't use that much memory, as a matter of fact memory >>> consumption starts little (40M) and changes very little (only +10M) if >>> you go to horizonatal view; the price is very slow rendering, more >>> than 10 seconds per column change. >>> >>> I'll post this on firefox bugzilla too, let's see if someone has a solution. >>> >>> Meanwhile perhaps pypy speed center could start with a smaller plot >>> area (or fewer columns as that makes horizontal plot smaller) to >>> accomodate varying hardware and system mem usage that users might >>> have? >>> The simplest would be a warning next to "horizontal" checkbox. >>> >>> On 21 December 2010 01:06, Miquel Torres wrote: >>>> Hi Dima, >>>> >>>>> another temp problem with speed pypy is that it's terrubly slow in ff >>>>> beta. it also occasionally grinds in stable ff and opera, but I guess >>>>> this can be forgiven for the sake of simplicity / developer effort. >>>> >>>> Well, speed.pypy is actually fast in all modern browsers. The problem >>>> you are referring to is probably caused by a bug in the javascript >>>> plotting library (jqPplot) that is triggered in the comparison view >>>> when there are some results with 0 values. It only appears for some >>>> plot types, but it is very annoying because it grinds the browser to a >>>> halt like you say. Is that what you meant? >>>> >>>> We are looking into it, and will fix that library if necessary. >>>> >>>> Cheers, >>>> Miquel >>>> >>>> >>>> 2010/12/21 Dima Tisnek : >>>>> On 20 December 2010 19:21, William ML Leslie >>>>> wrote: >>>>>> On 21 December 2010 11:59, Dima Tisnek wrote: >>>>>>> More visibility for performance achievements would do >>>>>>> good too. >>>>>> >>>>>> Where are pypy's performance achievements *not* visible, but should be? >>>>> >>>>> for example http://shootout.alioth.debian.org/ >>>>> doesn't say which pypy version is used, what options, doesn't have >>>>> performance figures for multithreaded/multicore >>>>> >>>>> also benchmarks are kinda small, most of them are not docuemented, and >>>>> I couldn't find any info if the same python code was used for cpython >>>>> and pypy (both shootout and speed pypy) or interpreter-specific >>>>> verions were used, that is each tweaked for best performance given the >>>>> known tradeoffs for each implementation.further the most standard >>>>> benchmarks, pystone, etc. completely ignore the fact that real world >>>>> programs are large and only a few small paths are execured often. >>>>> >>>>> another temp problem with speed pypy is that it's terrubly slow in ff >>>>> beta. it also occasionally grinds in stable ff and opera, but I guess >>>>> this can be forgiven for the sake of simplicity / developer effort. >>>>> >>>>> if you google for 'python performance' you don't get a single link to >>>>> pypy on the first page, as a matter of fact, codespeak is poorly >>>>> indexed, it took me quite some time to get some of my questions >>>>> answered with a search. also if you look up 'pypy gc' you get a page >>>>> on codespeak, but to interpret what the data actually means is so far >>>>> beyond me. >>>>> >>>>> a good overview is found in the mainling list >>>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >>>>> slowspitfire and spambayes bits are outdated by now. >>>>> >>>>> the definitive good thing about pypy is that it's much easier to find >>>>> out about its inner workings than that of cpython! >>>>> >>>>> hopefully a bit more of end-user stuff get known. >>>>> let's call it pypy public outreach (feature request) >>>>> >>>>>> >>>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>>>>>> hard would it be to use same techniques and most of backends for some >>>>>>> unrelated language that doesn't have jit yet, e.g. php? >>>>>> >>>>>> You know that pypy already has implementations of other languages, >>>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>>>>> integrated with the translated pypy-c, but they show that it is not >>>>>> too difficult to write a runtime for any dynamic language you choose. >>>>> >>>>> Oh I didn't know there were so many, and I mistakenly though that js >>>>> was a target, not implmented langauge. In any case I read somewhere >>>>> that js support was retired... >>>>> >>>>>> >>>>>>> And how hard >>>>>>> would it be to marry two dynamic languages, so that modules from one >>>>>>> could be used in the other? Or that modules written in rpython could >>>>>>> be used in several langs? >>>>>> >>>>>> It's in the "interesting problems" bucket, and the effort required >>>>>> depends on the level of integration between languages you want. ?There >>>>>> are several projects already attempting to do decent integration >>>>>> between several languages, besides the approach used on the JVM, there >>>>>> are also GNU Guile, Racket, and Parrot, among others. ?It might be >>>>>> worth waiting to see how these different projects pan out, before >>>>>> spending a bunch of effort just to be an also-ran in the >>>>>> multi-language runtime market. >>>>>> >>>>>> However, implementing more languages in rpython has the side-effect of >>>>>> propagating the l * o * p problem: it introduces more and more >>>>>> implementations that then have to be maintained, so good >>>>>> cross-language integration probably belongs /outside/ pypy itself, so >>>>>> existing runtimes can hook into it. >>>>> >>>>> Makes perfect sense, after all any given other language hardly has the >>>>> same data model as python. >>>>> >>>>>> >>>>>> But it would be an interesting experiment (to marry the various >>>>>> interpreters pypy ships with), if you wanted to try it. >>>>>> >>>>>> My two cents. >>>>>> >>>>>> -- >>>>>> William Leslie >>>>>> >>>>> _______________________________________________ >>>>> pypy-dev at codespeak.net >>>>> http://codespeak.net/mailman/listinfo/pypy-dev >>>> >>> >> > From arigo at tunes.org Thu Dec 23 12:04:32 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 23 Dec 2010 12:04:32 +0100 Subject: [pypy-dev] Building PyPy on Windows In-Reply-To: References: Message-ID: Hi Seung, On Thu, Dec 23, 2010 at 6:51 AM, Seung Soo, wrote: > An interesting article came up on python planet: > http://www.redmountainsw.com/wordpress/archives/building-pypy-on-windows I think the official documentation needs to state also how to build it with the missing modules (expat etc.). http://codespeak.net/pypy/dist/pypy/doc/windows.html has some hints, but is not complete as far as I can tell (e.g. it doesn't say how to arrange for the .h files to be found). Armin From arigo at tunes.org Thu Dec 23 12:14:00 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 23 Dec 2010 12:14:00 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Hi Dima, On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek wrote: > --- Comment #4 from Boris Zbarsky (:bz) 2010-12-22 > 13:43:23 PST --- > So what I see this page do, in horizontal mode, is create 17 canvases each of > which is width="816" height="3587". ?That means that each of them has a backing > store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage > right there. > > I have no idea why they feel a need for 17 huge canvases, but if they want > them, that's how much memory they'll take... That looks very similar to an issue with PyPy's own GC, in which ctypes.create_string_buffer() returns objects which tend to be GC'ed late. That's because the string buffer object in ctypes appears (to PyPy's GC) to be just a small object, even though it actually references a potentially large piece of raw memory. Similarly, my vague guess about the above is that the 17*11MB of memory are hold by 17 small objects which firefox's GC think don't need to be collected particularly aggressively. A bient?t, Armin. From renesd at gmail.com Thu Dec 23 14:33:50 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 23 Dec 2010 13:33:50 +0000 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? Message-ID: Hello, I think this is a case where the object returned by ctypes.create_string_buffer() could use a correct __sizeof__ method return value. If pypy supported that, then the GC's could support extensions, and 'opaque' data structures in C too a little more nicely. I think ctypes was started before __sizeof__ became available... so it seems many of it's methods are not updated yet. __sizeof__ is not mandatory, so many extensions have not been updated to support it yet. cheers, On Thu, Dec 23, 2010 at 11:14 AM, Armin Rigo wrote: > Hi Dima, > > On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek wrote: >> --- Comment #4 from Boris Zbarsky (:bz) 2010-12-22 >> 13:43:23 PST --- >> So what I see this page do, in horizontal mode, is create 17 canvases each of >> which is width="816" height="3587". ?That means that each of them has a backing >> store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage >> right there. >> >> I have no idea why they feel a need for 17 huge canvases, but if they want >> them, that's how much memory they'll take... > > That looks very similar to an issue with PyPy's own GC, in which > ctypes.create_string_buffer() returns objects which tend to be GC'ed > late. ?That's because the string buffer object in ctypes appears (to > PyPy's GC) to be just a small object, even though it actually > references a potentially large piece of raw memory. ?Similarly, my > vague guess about the above is that the 17*11MB of memory are hold by > 17 small objects which firefox's GC think don't need to be collected > particularly aggressively. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Thu Dec 23 14:38:59 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 23 Dec 2010 14:38:59 +0100 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: References: Message-ID: Hi Ren?, On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield wrote: > I think this is a case where the object returned by > ctypes.create_string_buffer() could use a correct __sizeof__ method > return value. ?If pypy supported that, then the GC's could support > extensions, and 'opaque' data structures in C too a little more > nicely. I think you are confusing levels. There is no way the GC can call some app-level Python method to get information about the objects it frees (and when would it even call it?). Remember that our GC is written at a level where it works for any interpreter for any language, not just Python. A bient?t, Armin. From p.giarrusso at gmail.com Thu Dec 23 15:09:47 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Thu, 23 Dec 2010 15:09:47 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: On Thu, Dec 23, 2010 at 12:14, Armin Rigo wrote: > Hi Dima, > > On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek wrote: >> --- Comment #4 from Boris Zbarsky (:bz) 2010-12-22 >> 13:43:23 PST --- >> So what I see this page do, in horizontal mode, is create 17 canvases each of >> which is width="816" height="3587". ?That means that each of them has a backing >> store of 816*3587*4 = 11,707,968 bytes. ?So that's about 200MB of memory usage >> right there. >> >> I have no idea why they feel a need for 17 huge canvases, but if they want >> them, that's how much memory they'll take... > > That looks very similar to an issue with PyPy's own GC, in which > ctypes.create_string_buffer() returns objects which tend to be GC'ed > late. ?That's because the string buffer object in ctypes appears (to > PyPy's GC) to be just a small object, even though it actually > references a potentially large piece of raw memory. Bigger objects are not collected sooner by typical GC algorithms* - they might just fill the heap more quickly and trigger earlier a GC invocation. >From your description, I guess that what you describe as "raw memory" is not accounted in the stats triggering GC. That would explain the behavior you describe, and suggest an easy fix. Indeed, one could wrap the raw-memory allocator to create and update such stats; then the heap-overflow check could consider them to decide whether to trigger GC. The details of the modified heap-overflow check would probably not be entirely trivial, but still doable. * "Typical" GC algorithms here includes copying, mark-sweep, and mark-compact collectors, including generational variants; even collectors having a large object space (typically reclaimed through mark-sweep) should still typically collect small objects often (by using the heuristic described above to trigger GC). Best regards >?Similarly, my > vague guess about the above is that the 17*11MB of memory are hold by > 17 small objects which firefox's GC think don't need to be collected > particularly aggressively. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From arigo at tunes.org Thu Dec 23 15:17:58 2010 From: arigo at tunes.org (Armin Rigo) Date: Thu, 23 Dec 2010 15:17:58 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Hi Paolo, On Thu, Dec 23, 2010 at 3:09 PM, Paolo Giarrusso wrote: > From your description, I guess that what you describe as "raw memory" > is not accounted in the stats triggering GC. That would explain the > behavior you describe, and suggest an easy fix. Indeed, one could wrap > the raw-memory allocator to create and update such stats; then the > heap-overflow check could consider them to decide whether to trigger > GC. The details of the modified heap-overflow check would probably not > be entirely trivial, but still doable. Absolutely. I will write it down in extradoc/planning/ for now. Armin From p.giarrusso at gmail.com Thu Dec 23 15:31:44 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Thu, 23 Dec 2010 15:31:44 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: On Thu, Dec 23, 2010 at 06:58, Dan Stromberg wrote: > On Wed, Dec 22, 2010 at 7:05 PM, Benjamin Peterson wrote: >> 2010/12/22 Dima Tisnek : >>> Oh, I can't yet use alternative gc that obviates GIL then? >>> >>> Or am I totally confused and pypy still uses GIL for other reasons, >>> e.g. globals dict safety? >> >> All of the above. > > On the topic of parallel dict safety... ?I've not played around with > parallel dictionaries at all, but I've heard that treaps can be > parallelized nicely, and they can be (and have been) set up to look > much like a python dictionary. ?I admit, I coded one and put it at > http://stromberg.dnsalias.org/~strombrg/treap/ - however, the > implementation is not (yet?) suitable for parallel use. > > It's coded in such a way that it'll work as pure python or cython, > depending on which of two m4 symbols you define. ?I've used the pure > python version on pypy. > > Treaps tend to make just about everything O(log(n)), have good average > performance, and are not as even performers as red-black trees (that > is, they give a higher standard deviation, and a better average > performance than red-black trees). ?And they don't play that nicely > with locality of reference (caches) - things are intentionally > randomized. ?But if you throw enough cores at them, they might still > be a win compared to a big hash table with a big lock wrapped around > it. That might be interesting; however, the state of the art includes many concurrent hash map alternatives which are way smarter than "a lock wrapped around". Java >=1.5 has one, by Doug Lea*. An even better (and more complex) one, by Cliff Click*, used in production on >500 cores, and presented to JavaOne, can be found by googling: nonblockinghashmap Cliff Click A C implementation can be found at: http://code.google.com/p/nbds/ Link found through this blog post of Cliff Click: http://www.azulsystems.com/blog/cliff-click/2008-11-13-some-source-forge-threads-nbhm However, I don't think you can code them in nowadays Python, because it offers neither a Java-equivalent volatile attribute nor compare-and-swap nor memory barriers, and these structures can't be implemented with plain locks. Best regards * Doug Lea wrote the memory allocator used on Linux, is behind the 1.5 java.util.concurrent package, and much more. Cliff Click is a high-performance JVM hacker. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From dimaqq at gmail.com Thu Dec 23 20:30:20 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Thu, 23 Dec 2010 12:30:20 -0700 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: References: Message-ID: Basically collecting this is hard: dict(a=range(9**9)) large list is referenced, the object that holds the only reference is small no matter how you look at it. I guess it gets harder still if there are many small live objects, as getting to this dict takes a while (easier in this simple case with generataional collector, O(n) in general case) On 23 December 2010 06:38, Armin Rigo wrote: > Hi Ren?, > > On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield wrote: >> I think this is a case where the object returned by >> ctypes.create_string_buffer() could use a correct __sizeof__ method >> return value. ?If pypy supported that, then the GC's could support >> extensions, and 'opaque' data structures in C too a little more >> nicely. > > I think you are confusing levels. ?There is no way the GC can call > some app-level Python method to get information about the objects it > frees (and when would it even call it?). ?Remember that our GC is > written at a level where it works for any interpreter for any > language, not just Python. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From p.giarrusso at gmail.com Fri Dec 24 12:39:29 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Fri, 24 Dec 2010 12:39:29 +0100 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: References: Message-ID: On Thu, Dec 23, 2010 at 20:30, Dima Tisnek wrote: > Basically collecting this is hard: > > dict(a=range(9**9)) > > large list is referenced, the object that holds the only reference is > small no matter how you look at it. First, usually (in most GC-ed languages) you can collect the list before the dict. In PyPy, if finalizers are involved (is this the case here? That'd be surprising), this is no more true. However, object size is not the point. For standard algorithms, the size of an object does not matter at all in deciding when it's collected - I already discussed this in my other email in this thread, and I noted what actually could happen in the examples described by Armin, and your examples show that it is a good property. A large object in the same heap can fill it up and trigger an earlier garbage collection. In general, if GC ran in the background (but it usually doesn't, and not in PyPy) it could make sense to free objects sooner or later, depending not on object size, but on "how much memory would be 'indirectly freed' by freeing this object". However, because of sharing, answering this question is too complex (it requires collecting data from the whole heap). Moreover, the whole thing makes no sense at all with usual, stop-the-world collectors: the app is stopped, then the whole young generation, or the whole heap, is collected, then the app is resumed. When separate heaps are involved (such as with ctypes, or with Large Object Spaces, which avoid using a copy collector for large objects), it is more complicated to ensure that the same property holds: you need to consider stats of all heaps to decide whether to trigger GC. > I guess it gets harder still if there are many small live objects, as > getting to this dict takes a while > (easier in this simple case with generataional collector, O(n) in general case) Not sure what you mean; I can make sense of it (not fully) only with an incremental collector, and they are still used seldom (especially, not in PyPy). Best regards > On 23 December 2010 06:38, Armin Rigo wrote: >> Hi Ren?, >> >> On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield wrote: >>> I think this is a case where the object returned by >>> ctypes.create_string_buffer() could use a correct __sizeof__ method >>> return value. ?If pypy supported that, then the GC's could support >>> extensions, and 'opaque' data structures in C too a little more >>> nicely. >> >> I think you are confusing levels. ?There is no way the GC can call >> some app-level Python method to get information about the objects it >> frees (and when would it even call it?). ?Remember that our GC is >> written at a level where it works for any interpreter for any >> language, not just Python. >> >> >> A bient?t, >> >> Armin. >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From tobami at googlemail.com Sat Dec 25 00:10:19 2010 From: tobami at googlemail.com (Miquel Torres) Date: Sat, 25 Dec 2010 00:10:19 +0100 Subject: [pypy-dev] Mercurial support for speed.pypy.org Message-ID: Hi all, I finally added Mercurial support to Codespeed, following Antonio's suggestion of parsing the (templated) command line output.Thanks to everyone that helped! I waited for today to give the awesome PyPy team a present. Merry Christmas! Miquel From fijall at gmail.com Sat Dec 25 10:07:20 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 25 Dec 2010 11:07:20 +0200 Subject: [pypy-dev] Mercurial support for speed.pypy.org In-Reply-To: References: Message-ID: On Sat, Dec 25, 2010 at 1:10 AM, Miquel Torres wrote: > Hi all, > > I finally added Mercurial support to Codespeed, following Antonio's > suggestion of parsing the (templated) command line output.Thanks to > everyone that helped! > > I waited for today to give the awesome PyPy team a present. > > Merry Christmas! > > Miquel That's a great present, thanks! Merry Christmas! Cheers, fijal From arigo at tunes.org Sat Dec 25 19:41:08 2010 From: arigo at tunes.org (Armin Rigo) Date: Sat, 25 Dec 2010 19:41:08 +0100 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: References: Message-ID: Hi Ren?, On Thu, Dec 23, 2010 at 8:30 PM, Dima Tisnek wrote: > Basically collecting this is hard: > > dict(a=range(9**9)) I think you missed the point of my original email. I was talking about GC-referenced objects that hold a reference to a large piece of memory allocated outside the GC. There is none here, and any GC (including PyPy's) will do a correct job in collecting this. A bient?t, Armin. From amauryfa at gmail.com Sun Dec 26 21:44:19 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sun, 26 Dec 2010 21:44:19 +0100 Subject: [pypy-dev] Fwd: Mercurial support for speed.pypy.org In-Reply-To: References: Message-ID: Hi, 2010/12/25 Miquel Torres I finally added Mercurial support to Codespeed, following Antonio's > suggestion of parsing the (templated) command line output.Thanks to > everyone that helped! > > I waited for today to give the awesome PyPy team a present. > Thanks! It seems that all changes are listed, even those who happen to be in another branch. Is it possible to filter the list and only display the "default" branch? Cheers, -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101226/24e46f50/attachment.htm From amauryfa at gmail.com Sun Dec 26 23:03:29 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sun, 26 Dec 2010 23:03:29 +0100 Subject: [pypy-dev] dlopen NotImplementedError In-Reply-To: References: <20101222224127.5919.1124137746.divmod.xquotient.242@localhost.localdomain> Message-ID: Hi, 2010/12/23 Maciej Fijalkowski > CDLL(None) (or LoadLibrary(None)) is supposed to return to you the > whole namespace (all loaded libraries). It's unsupported as far as I > can tell. It should work by now (on Unix platforms of course). -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20101226/fad9a321/attachment.htm From dimaqq at gmail.com Mon Dec 27 07:00:30 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Sun, 26 Dec 2010 23:00:30 -0700 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the constant factor for locks, etc, might make it worthwhile (paolo) non blocking hash maps: memory barriers can be quite costly too different options will have to be implemented and tested when we get there, speaking on which, is there a test dict load? does anyone have some profiling data, what portion of runtime is spent reading and writing attributes and globals anyways? while we are on the subject, is there a plan to provide different levels of sparseness for different levels of name lookup? for example globals vs builtins, first needs to be quite sparce so that builtins show through well, second hardly, because there's nowhere else to look if builtins don't have the name. then the tradeoff could be more dynamic, that is frequently accessed dicts could be e.g. more sparse and rarely accessed more compact? (obviousely more sparse is not always better, e.g. in terms of cpu cache) of course "frequently accessed" is not as easy as frequently ran code, e.g. here "class A: ...; while 1: A().func()", func lookup occurs in different objects, yet it is same logical operation. come to think of it, is there any point in polymorphic dicts, e.g. attribute access could be imeplemented as a near-perfect compact hash map if attribute names change rarely, while regular dict()s are expected to change keys often. Anyhow, back to parallel interpretation, Is there an issue or page that tracks what's needed for parallel interpretation? so far mentioned: gc, dict, c api Btw, I think that jit is more important at the moment, but time comes when jit juice has been mostly squeezed out ;-) d. On 23 December 2010 07:31, Paolo Giarrusso wrote: > On Thu, Dec 23, 2010 at 06:58, Dan Stromberg wrote: >> On Wed, Dec 22, 2010 at 7:05 PM, Benjamin Peterson wrote: >>> 2010/12/22 Dima Tisnek : >>>> Oh, I can't yet use alternative gc that obviates GIL then? >>>> >>>> Or am I totally confused and pypy still uses GIL for other reasons, >>>> e.g. globals dict safety? >>> >>> All of the above. >> >> On the topic of parallel dict safety... ?I've not played around with >> parallel dictionaries at all, but I've heard that treaps can be >> parallelized nicely, and they can be (and have been) set up to look >> much like a python dictionary. ?I admit, I coded one and put it at >> http://stromberg.dnsalias.org/~strombrg/treap/ - however, the >> implementation is not (yet?) suitable for parallel use. >> >> It's coded in such a way that it'll work as pure python or cython, >> depending on which of two m4 symbols you define. ?I've used the pure >> python version on pypy. >> >> Treaps tend to make just about everything O(log(n)), have good average >> performance, and are not as even performers as red-black trees (that >> is, they give a higher standard deviation, and a better average >> performance than red-black trees). ?And they don't play that nicely >> with locality of reference (caches) - things are intentionally >> randomized. ?But if you throw enough cores at them, they might still >> be a win compared to a big hash table with a big lock wrapped around >> it. > > That might be interesting; however, the state of the art includes many > concurrent hash map alternatives which are way smarter than "a lock > wrapped around". Java >=1.5 has one, by Doug Lea*. An even better (and > more complex) one, by Cliff Click*, used in production on >500 cores, > and presented to JavaOne, can be found by googling: > > nonblockinghashmap Cliff Click > > A C implementation can be found at: > http://code.google.com/p/nbds/ > > Link found through this blog post of Cliff Click: > http://www.azulsystems.com/blog/cliff-click/2008-11-13-some-source-forge-threads-nbhm > > However, I don't think you can code them in nowadays Python, because > it offers neither a Java-equivalent volatile attribute nor > compare-and-swap nor memory barriers, and these structures can't be > implemented with plain locks. > > Best regards > > * Doug Lea wrote the memory allocator used on Linux, is behind the 1.5 > java.util.concurrent package, and much more. Cliff Click is a > high-performance JVM hacker. > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From william.leslie.ttg at gmail.com Mon Dec 27 07:30:35 2010 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Mon, 27 Dec 2010 17:30:35 +1100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: On 27 December 2010 17:00, Dima Tisnek wrote: > while we are on the subject, is there a plan to provide different > levels of sparseness for different levels of name lookup? > for example globals vs builtins, first needs to be quite sparce so > that builtins show through well, second hardly, because there's > nowhere else to look if builtins don't have the name. > then the tradeoff could be more dynamic, that is frequently accessed > dicts could be e.g. more sparse and rarely accessed more compact? > (obviousely more sparse is not always better, e.g. in terms of cpu cache) > of course "frequently accessed" is not as easy as frequently ran code, > e.g. here "class A: ...; while 1: A().func()", func lookup occurs in > different objects, yet it is same logical operation. > > come to think of it, is there any point in polymorphic dicts, e.g. > attribute access could be imeplemented as a near-perfect compact hash > map if attribute names change rarely, while regular dict()s are > expected to change keys often. No, not really, but pypy already heavily optimises this case - see mapdicts, celldicts, and sharing dicts. Celldict is commonly used for module dictionaries, and emulates a direct pointer to the value, which can be cached along side the LOAD_GLOBAL instruction. I implemented a similar system (pre-computed array based on known module-global names) a few years back in pypy (pre-jit) and got a 6% speedup over regular dicts, but celldicts get a similar speedup and are more generally applicable. As for perfect hashing: our existing mechanism for hashing beats it, hands down. In cpython at least, I haven't checked the pypy source on this topic, the hash of a string is cached on the string object itself, which means in the case of identifiers no hash is ever computed on global lookup. The only thing that could really be faster is something like a slot on the symbol itself. Celldicts move the synchronisation point out of the hash table and into the entry for common cases, which changes the synchronisation question significantly. > Btw, I think that jit is more important at the moment, but time comes > when jit juice has been mostly squeezed out ;-) There are occasional memory-model discussions, but at the end of the day what will probably happen is the people who step up to do the work to implement it will probably also get to do most of the design work. -- William Leslie From cfbolz at gmx.de Mon Dec 27 09:31:51 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 27 Dec 2010 09:31:51 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: <4D184EF7.1040200@gmx.de> On 12/27/2010 07:00 AM, Dima Tisnek wrote: > (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the > constant factor for locks, etc, might make it worthwhile > > (paolo) non blocking hash maps: memory barriers can be quite costly too > > different options will have to be implemented and tested when we get there, > speaking on which, is there a test dict load? > does anyone have some profiling data, what portion of runtime is spent > reading and writing attributes and globals anyways? > > while we are on the subject, is there a plan to provide different > levels of sparseness for different levels of name lookup? > for example globals vs builtins, first needs to be quite sparce so > that builtins show through well, second hardly, because there's > nowhere else to look if builtins don't have the name. > then the tradeoff could be more dynamic, that is frequently accessed > dicts could be e.g. more sparse and rarely accessed more compact? > (obviousely more sparse is not always better, e.g. in terms of cpu cache) > of course "frequently accessed" is not as easy as frequently ran code, > e.g. here "class A: ...; while 1: A().func()", func lookup occurs in > different objects, yet it is same logical operation. > > come to think of it, is there any point in polymorphic dicts, e.g. > attribute access could be imeplemented as a near-perfect compact hash > map if attribute names change rarely, while regular dict()s are > expected to change keys often. All these thoughts go into the wrong direction, imo. The JIT removes nearly all dictionary accesses to global dicts, instance and class dicts. Even without the JIT, purely interpreting things, there are caches that bypass most dict lookups. The only case where improving dicts would help is for user-defined dictionaries, i.e. when you write dicts with curly braces. that is basically very hard, because the usage patterns vary widely and a lot of work would be necessary to find common uses. Cheers, Carl Friedrich From anto.cuni at gmail.com Mon Dec 27 09:35:23 2010 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 27 Dec 2010 09:35:23 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: <4D184EF7.1040200@gmx.de> References: <4D184EF7.1040200@gmx.de> Message-ID: <4D184FCB.8050404@gmail.com> On 27/12/10 09:31, Carl Friedrich Bolz wrote: > The only case where improving dicts would help is for user-defined > dictionaries, i.e. when you write dicts with curly braces. then it's easy to optimize, you just write "dict()" instead of "{}". (sorry, I could not resist :-)) From tobami at googlemail.com Mon Dec 27 11:27:03 2010 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 27 Dec 2010 11:27:03 +0100 Subject: [pypy-dev] Fwd: Mercurial support for speed.pypy.org In-Reply-To: References: Message-ID: Oh, ok. Fixed now by adding "-b default " to the hg log command. (still learning mercurial :) 2010/12/26 Amaury Forgeot d'Arc : > Hi, > > 2010/12/25 Miquel Torres >> >> I finally added Mercurial support to Codespeed, following Antonio's >> suggestion of parsing the (templated) command line output.Thanks to >> everyone that helped! >> >> I waited for today to give the awesome PyPy team a present. > > Thanks! > It seems that all changes are listed, even those who happen to be in another > branch. > Is it possible to filter the list and only display the "default" branch? > Cheers, > > -- > Amaury Forgeot d'Arc > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From p.giarrusso at gmail.com Mon Dec 27 18:25:22 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 27 Dec 2010 18:25:22 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: <4D184EF7.1040200@gmx.de> References: <4D184EF7.1040200@gmx.de> Message-ID: On Mon, Dec 27, 2010 at 09:31, Carl Friedrich Bolz wrote: > On 12/27/2010 07:00 AM, Dima Tisnek wrote: >> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the >> constant factor for locks, etc, might make it worthwhile >> >> (paolo) non blocking hash maps: memory barriers can be quite costly too >> >> different options will have to be implemented and tested when we get there, >> speaking on which, is there a test dict load? >> does anyone have some profiling data, what portion of runtime is spent >> reading and writing attributes and globals anyways? >> >> while we are on the subject, is there a plan to provide different >> levels of sparseness for different levels of name lookup? >> for example globals vs builtins, first needs to be quite sparce so >> that builtins show through well, second hardly, because there's >> nowhere else to look if builtins don't have the name. >> then the tradeoff could be more dynamic, that is frequently accessed >> dicts could be e.g. more sparse and rarely accessed more compact? >> (obviousely more sparse is not always better, e.g. in terms of cpu cache) >> of course "frequently accessed" is not as easy as frequently ran code, >> e.g. here "class A: ...; while 1: A().func()", func lookup occurs in >> different objects, yet it is same logical operation. >> >> come to think of it, is there any point in polymorphic dicts, e.g. >> attribute access could be imeplemented as a near-perfect compact hash >> map if attribute names change rarely, while regular dict()s are >> expected to change keys often. > > All these thoughts go into the wrong direction, imo. The JIT removes > nearly all dictionary accesses to global dicts, instance and class > dicts. Even without the JIT, purely interpreting things, there are > caches that bypass most dict lookups. That's very interesting. However, aren't such caches also hash maps in the end (unless you do inline caching in your interpreter, like in Ruby 1.9)? I remember reading so in PyPy's docs; moreover, that's a standard optimization for method lookup. Sharing such caches between different interpreter threads would be potentially useful, if that implied no expensive synchronization for readers - which is possible for instance on x86 (read barriers are for free), by using persistent data structures. Writes to such caches should (hopefully) be rare enough to make the extra synchronization not too expensive, however this needs benchmarking. Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From p.giarrusso at gmail.com Mon Dec 27 18:57:55 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 27 Dec 2010 18:57:55 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: On Mon, Dec 27, 2010 at 07:00, Dima Tisnek wrote: > (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the > constant factor for locks, etc, might make it worthwhile "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)? > (paolo) non blocking hash maps: memory barriers can be quite costly too Well, you just _cannot_ avoid them, whatever is your fancy data structure. You can only move most of the cost (or all of it) to write operations - which is helpful if reads are most common, a case which you discuss in your mail. == Why memory barriers are required (in more detail) == If you use persistent data structures, also called copy-on-write or purely functional (and that's the most obvious way to use a tree in a parallel setting), the whole synchronization problem is reduced to exchanging a pointer (to the root of the tree) between a writer and a reader thread. Let's consider this in the Java memory model (the state of the art). In this model the writer thread has to use an expensive StoreLoad barrier, i.e. mfence on x86 [1], costing more or less as much as a cache miss, while the reader needs just a cheap Load* barrier (for free on x86, cheap on other processor). If you don't use a volatile field, there is very little guarantee about what your program means, and it is almost always buggy; the equivalent in the new C++ memory model (which is otherwise very similar) gives completely undefined semantics (and even crashes are possible in practice, for instance if the reader thread sees a not-fully-initialized object because of the race and invokes a virtual method on it). [1] http://g.oswego.edu/dl/jmm/cookbook.html == Other stuff == > different options will have to be implemented and tested when we get there, > speaking on which, is there a test dict load? > does anyone have some profiling data, what portion of runtime is spent > reading and writing attributes and globals anyways? I'd be interested as well in such data. > Anyhow, back to parallel interpretation, Is there an issue or page > that tracks what's needed for parallel interpretation? > so far ?mentioned: gc, dict, c api > Btw, I think that jit is more important at the moment, but time comes > when jit juice has been mostly squeezed out ;-) -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From dimaqq at gmail.com Mon Dec 27 19:54:41 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 27 Dec 2010 11:54:41 -0700 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: what do you think of liburcu? lttng.org/urcu p.s. if/when I get around to profiling pypy, which is something I want to do, I'll be sure to share the results here. d. On 27 December 2010 10:57, Paolo Giarrusso wrote: > On Mon, Dec 27, 2010 at 07:00, Dima Tisnek wrote: >> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the >> constant factor for locks, etc, might make it worthwhile > > "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)? > >> (paolo) non blocking hash maps: memory barriers can be quite costly too > > Well, you just _cannot_ avoid them, whatever is your fancy data structure. > You can only move most of the cost (or all of it) to write operations > - which is helpful if reads are most common, a case which you discuss > in your mail. > > == Why memory barriers are required (in more detail) == > If you use persistent data structures, also called copy-on-write or > purely functional (and that's the most obvious way to use a tree in a > parallel setting), the whole synchronization problem is reduced to > exchanging a pointer (to the root of the tree) between a writer and a > reader thread. > Let's consider this in the Java memory model (the state of the art). > In this model the writer thread has to use an expensive StoreLoad > barrier, i.e. mfence on x86 [1], costing more or less as much as a > cache miss, while the reader needs just a cheap Load* barrier (for > free on x86, cheap on other processor). > If you don't use a volatile field, there is very little guarantee > about what your program means, and it is almost always buggy; the > equivalent in the new C++ memory model (which is otherwise very > similar) gives completely undefined semantics (and even crashes are > possible in practice, for instance if the reader thread sees a > not-fully-initialized object because of the race and invokes a virtual > method on it). > > [1] http://g.oswego.edu/dl/jmm/cookbook.html > > == Other stuff == >> different options will have to be implemented and tested when we get there, >> speaking on which, is there a test dict load? >> does anyone have some profiling data, what portion of runtime is spent >> reading and writing attributes and globals anyways? > > I'd be interested as well in such data. > >> Anyhow, back to parallel interpretation, Is there an issue or page >> that tracks what's needed for parallel interpretation? >> so far ?mentioned: gc, dict, c api > >> Btw, I think that jit is more important at the moment, but time comes >> when jit juice has been mostly squeezed out ;-) > > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > From p.giarrusso at gmail.com Mon Dec 27 20:50:08 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 27 Dec 2010 20:50:08 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: On Mon, Dec 27, 2010 at 19:54, Dima Tisnek wrote: > what do you think of liburcu? lttng.org/urcu I've studied RCU in the Linux kernel (from which URCU derives) and thought for a long time of using it for this problem. In short, once you have GC, RCU (as in the Linux kernel) becomes (almost) trivial, because RCU is almost entirely about how to delay freeing objects. The RCU-GC connection is also mentioned by Documentation/RCU/RTFP.txt in the Linux sources. So, you need to just read about persistent data structures, and remember that persistence is not the point - data structures which are just almost persistent can also be OK, if you get them right. If you replace a value in a dictionary, for instance, you don't necessarily need to keep the old version around, as long as you the value fits in a memory word so that it can be atomically replaced. http://hackerboss.com/copy-on-write-101-part-2-putting-it-to-use/ Speaking of libraries, you might want to look at this for the atomic primitives you'll need: www.hpl.hp.com/research/linux/atomic_ops/ I'm more used to the primitives of the Linux kernel, I remember I didn't fully like that library, but IIRC it was a matter of taste - the library is from Hans Boehm, which makes it trustworthy. > p.s. if/when I get around to profiling pypy, which is something I want > to do, I'll be sure to share the results here. Great, thanks! > d. > > On 27 December 2010 10:57, Paolo Giarrusso wrote: >> On Mon, Dec 27, 2010 at 07:00, Dima Tisnek wrote: >>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the >>> constant factor for locks, etc, might make it worthwhile >> >> "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)? >> >>> (paolo) non blocking hash maps: memory barriers can be quite costly too >> >> Well, you just _cannot_ avoid them, whatever is your fancy data structure. >> You can only move most of the cost (or all of it) to write operations >> - which is helpful if reads are most common, a case which you discuss >> in your mail. >> >> == Why memory barriers are required (in more detail) == >> If you use persistent data structures, also called copy-on-write or >> purely functional (and that's the most obvious way to use a tree in a >> parallel setting), the whole synchronization problem is reduced to >> exchanging a pointer (to the root of the tree) between a writer and a >> reader thread. >> Let's consider this in the Java memory model (the state of the art). >> In this model the writer thread has to use an expensive StoreLoad >> barrier, i.e. mfence on x86 [1], costing more or less as much as a >> cache miss, while the reader needs just a cheap Load* barrier (for >> free on x86, cheap on other processor). >> If you don't use a volatile field, there is very little guarantee >> about what your program means, and it is almost always buggy; the >> equivalent in the new C++ memory model (which is otherwise very >> similar) gives completely undefined semantics (and even crashes are >> possible in practice, for instance if the reader thread sees a >> not-fully-initialized object because of the race and invokes a virtual >> method on it). >> >> [1] http://g.oswego.edu/dl/jmm/cookbook.html >> >> == Other stuff == >>> different options will have to be implemented and tested when we get there, >>> speaking on which, is there a test dict load? >>> does anyone have some profiling data, what portion of runtime is spent >>> reading and writing attributes and globals anyways? >> >> I'd be interested as well in such data. >> >>> Anyhow, back to parallel interpretation, Is there an issue or page >>> that tracks what's needed for parallel interpretation? >>> so far ?mentioned: gc, dict, c api >> >>> Btw, I think that jit is more important at the moment, but time comes >>> when jit juice has been mostly squeezed out ;-) >> >> -- >> Paolo Giarrusso - Ph.D. Student >> http://www.informatik.uni-marburg.de/~pgiarrusso/ >> > -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From dimaqq at gmail.com Tue Dec 28 02:47:14 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 27 Dec 2010 18:47:14 -0700 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: Thanks for the RTFP pointer, the Triplett ref, hash tab via rt is really interesting! I think I'll have to read it again and again to get all out of it... This brings me to a new question: how much is pypy python memory model allowed to diverge from that of cpython? For example, dict lookups, etc, could be relativistic in the absense of explicit synchronization, that is, dict updates are not quaranteed to propagate [fast] until [some] therad acquires a lock [or something]. Of course the order of dict mutation has to be preserved. Would this be a worthwhile departure from GIL-based monotonic world? That is thread-safe code, that uses locks or some other mechanism [known to pypy] would see operations timely and in order, while code that doesn't synchronize its operations may see somewhat outdated shared state, as if threads executed in slightly different global order. About profiling, how do I distinguish between attribute vs globals vs builtins lookups in the profile? d. On 27 December 2010 12:50, Paolo Giarrusso wrote: > On Mon, Dec 27, 2010 at 19:54, Dima Tisnek wrote: >> what do you think of liburcu? lttng.org/urcu > > I've studied RCU in the Linux kernel (from which URCU derives) and > thought for a long time of using it for this problem. In short, once > you have GC, RCU (as in the Linux kernel) becomes (almost) trivial, > because RCU is almost entirely about how to delay freeing objects. > The RCU-GC connection is also mentioned by Documentation/RCU/RTFP.txt > in the Linux sources. > > So, you need to just read about persistent data structures, and > remember that persistence is not the point - data structures which are > just almost persistent can also be OK, if you get them right. If you > replace a value in a dictionary, for instance, you don't necessarily > need to keep the old version around, as long as you the value fits in > a memory word so that it can be atomically replaced. > http://hackerboss.com/copy-on-write-101-part-2-putting-it-to-use/ > > Speaking of libraries, you might want to look at this for the atomic > primitives you'll need: > www.hpl.hp.com/research/linux/atomic_ops/ > I'm more used to the primitives of the Linux kernel, I remember I > didn't fully like that library, but IIRC it was a matter of taste - > the library is from Hans Boehm, which makes it trustworthy. > >> p.s. if/when I get around to profiling pypy, which is something I want >> to do, I'll be sure to share the results here. > > Great, thanks! > >> d. >> >> On 27 December 2010 10:57, Paolo Giarrusso wrote: >>> On Mon, Dec 27, 2010 at 07:00, Dima Tisnek wrote: >>>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the >>>> constant factor for locks, etc, might make it worthwhile >>> >>> "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)? >>> >>>> (paolo) non blocking hash maps: memory barriers can be quite costly too >>> >>> Well, you just _cannot_ avoid them, whatever is your fancy data structure. >>> You can only move most of the cost (or all of it) to write operations >>> - which is helpful if reads are most common, a case which you discuss >>> in your mail. >>> >>> == Why memory barriers are required (in more detail) == >>> If you use persistent data structures, also called copy-on-write or >>> purely functional (and that's the most obvious way to use a tree in a >>> parallel setting), the whole synchronization problem is reduced to >>> exchanging a pointer (to the root of the tree) between a writer and a >>> reader thread. >>> Let's consider this in the Java memory model (the state of the art). >>> In this model the writer thread has to use an expensive StoreLoad >>> barrier, i.e. mfence on x86 [1], costing more or less as much as a >>> cache miss, while the reader needs just a cheap Load* barrier (for >>> free on x86, cheap on other processor). >>> If you don't use a volatile field, there is very little guarantee >>> about what your program means, and it is almost always buggy; the >>> equivalent in the new C++ memory model (which is otherwise very >>> similar) gives completely undefined semantics (and even crashes are >>> possible in practice, for instance if the reader thread sees a >>> not-fully-initialized object because of the race and invokes a virtual >>> method on it). >>> >>> [1] http://g.oswego.edu/dl/jmm/cookbook.html >>> >>> == Other stuff == >>>> different options will have to be implemented and tested when we get there, >>>> speaking on which, is there a test dict load? >>>> does anyone have some profiling data, what portion of runtime is spent >>>> reading and writing attributes and globals anyways? >>> >>> I'd be interested as well in such data. >>> >>>> Anyhow, back to parallel interpretation, Is there an issue or page >>>> that tracks what's needed for parallel interpretation? >>>> so far ?mentioned: gc, dict, c api >>> >>>> Btw, I think that jit is more important at the moment, but time comes >>>> when jit juice has been mostly squeezed out ;-) >>> >>> -- >>> Paolo Giarrusso - Ph.D. Student >>> http://www.informatik.uni-marburg.de/~pgiarrusso/ >>> >> > > > > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > From cfbolz at gmx.de Thu Dec 30 01:27:02 2010 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 30 Dec 2010 01:27:02 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: <4D184EF7.1040200@gmx.de> Message-ID: <4D1BD1D6.7010805@gmx.de> On 12/27/2010 06:25 PM, Paolo Giarrusso wrote: >> All these thoughts go into the wrong direction, imo. The JIT removes >> nearly all dictionary accesses to global dicts, instance and class >> dicts. Even without the JIT, purely interpreting things, there are >> caches that bypass most dict lookups. > > That's very interesting. However, aren't such caches also hash maps in > the end (unless you do inline caching in your interpreter, like in > Ruby 1.9)? I remember reading so in PyPy's docs; moreover, that's a > standard optimization for method lookup. Indeed, some of the caches are again hash maps, e.g. the method cache is a global hash map. However, since it is a cache of a fixed size, its implementation is a lot simpler than that of dictionaries. Some of the caches (like that of attribute access) are indeed inline caches and thus don't need hash maps at all. At some point the method cache was an inline cache as well, but that didn't seem to actually help much. In general it seems that eliminating single dict lookups is rarely worth it in PyPy. We also had an inline global dict lookup cache at some point, but it didn't give an interesting enough effect to keep it. I'm talking purely about the interpreter here, of course, the jit gets rid of all these lookups anyway. > Sharing such caches between different interpreter threads would be > potentially useful, if that implied no expensive synchronization for > readers - which is possible for instance on x86 (read barriers are for > free), by using persistent data structures. Writes to such caches > should (hopefully) be rare enough to make the extra synchronization > not too expensive, however this needs benchmarking. Again, this is all idle speculation. If PyPy wants to move towards a GIL-less implementation, a *huge* amount of problems would need to be solved first before such optimizations become important. We would need to fix the GC. We would need to think about the memory model of RPython in the presence of multi-CPU threading, and that of Python. At the moment I don't see that work happening, because nobody in the current core contributor group is qualified or interested. Yes, in a sense that's not very forward-looking, given the move to multi-core. Otoh it's not clear to me that shared-memory multithreading is really the approach that makes most sense to PyPy in its current state. Cheers, Carl Friedrich From p.giarrusso at gmail.com Fri Dec 31 17:59:05 2010 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Fri, 31 Dec 2010 17:59:05 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: On Tue, Dec 28, 2010 at 02:47, Dima Tisnek wrote: > Thanks for the RTFP pointer, the Triplett ref, hash tab via rt is > really interesting! > > I think I'll have to read it again and again to get all out of it... Ah, I remember that! Never found enough time to read up that... also I remember that I didn't like it, at the time, because it seemed to give a new name to existing design practice in the RCU community. I didn't realize that even if the article just described existing work, the generalization alone might be an important contribution. > This brings me to a new question: how much is pypy python memory model > allowed to diverge from that of cpython? In fact, Jython and IronPython already have different memory models: code.google.com/p/unladen-swallow/wiki/MemoryModel But the most important answer is that to avoid reinventing the wheel, one needs to know the state of the art, i.e. the work by Sarita Adve, Jeremy Manson, William Pugh and Hans Boehm on memory models for Java and C++. The simplest introduction I know of comes in this blog post by Jeremy Manson: http://jeremymanson.blogspot.com/2008/11/what-volatile-means-in-java.html As a matter of fact, this work is unsurprisingly often totally ignored in many discussions about Python's memory model. I'm not surprised because it's complex stuff, and understanding it doesn't matter for today's Python programming. The key concept of these models is that _properly synchronized_ programs get "sequential consistent" semantics: the programs behaves as if all memory operations were interleaved in a total order on a single coherent memory, without reorderings due to cache, compiler optimizations and so on. Sequential consistency is the key property of what you call "GIL-based monotonic world". Exactly as you suggested for dictionaries, but more in general, programs which are not properly synchronized (which exhibit so-called data races) are more-or-less left on their own. On this topic, the Java and C++ models are different, and that leaves a design space open, but one which is already somewhat smaller, and which doesn't affect the semantics of properly synchronized programs. Java tries to define the semantics of data races, C++ just gives up. * Unlike C++ and like Java, it is probably important to guarantee that programs with data races (not properly synchronized memory accesses) don't crash the interpreter or corrupt native memory. You don't want native code in the interpreter to crash or to loop if the programmer forgets synchronization - simply because that's very difficult to debug, short of running a debugger on the interpreter itself (or fixing the code without a debugger at all). It is not clear, though, if one should give specific semantics to programs with data races, other than ensure the interpreter doesn't crash: basically it looks too complex. For whoever understands the rest of the Java Memory Model, this blog post explains the semantics they use to model this. http://jeremymanson.blogspot.com/2007/08/causality-and-java-memory-model.html * Another difference between Java and C++ is about security guarantees: it is crucial in Java that untrusted code cannot read sensitive data (e.g., a password) managed by trusted code, not even through a data race. Some kinds of behavior would allow "out-of-thin-air values", which in practice might be such sensitive data, if memory is zeroed by the memory allocator but one thread still sees the old value. I believe the most up-to-date introduction, with a useful bibliography, is by S. Adve and H. Boehm [1]. Then you will probably want to read [2] and [3] about C++ and Java. Through Google Scholar, you should find you copies of all of this on CiteseerX. [1] Sarita V. Adve and Hans-J. Boehm. 2010. Memory models: a case for rethinking parallel languages and hardware. Communications of the ACM. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.158.4288 [2] Hans-J. Boehm and Sarita V. Adve. 2008. Foundations of the C++ concurrency memory model. PLDI '08 [3] Jeremy Manson, William Pugh, and Sarita V. Adve. 2005. The Java memory model. POPL '05. > For example, dict lookups, etc, could be relativistic in the absense > of explicit synchronization, that is, dict updates are not quaranteed > to propagate [fast] until [some] therad acquires a lock [or > something]. Talking about dictionaries restricts the scope unnecessarily. You want to relax these requirement for memory operations in general. > Would this be a worthwhile departure from GIL-based monotonic world? Years of research agree the answer is yes, no doubt. Otherwise several fundamental optimizations become impossible - including the usage of memory caches. To make a simple example, releasing a lock requires flushing pending writes (i.e. a cache miss, ~100 cycles nowadays). Implementing sequential consistency (what you term "monotonic world") means doing that after _each_ memory write. If you read Adve et Boehm [1], they'll elaborate more on the topic. > That is thread-safe code, that uses locks or some other mechanism > [known to pypy] would see operations timely and in order, while code > that doesn't synchronize its operations may see somewhat outdated > shared state, as if threads executed in slightly different global > order. Yes. However, the programmer can't and shouldn't synchronize, say, field lookup, so those dictionaries (and only those) need to be thread-safe - Jython uses a ConcurrentHashMap for this purpose, even too that's too thread-safe (there needs to be no thread-safety guarantee on the values of fields). > About profiling, how do I distinguish between attribute vs globals vs > builtins lookups in the profile? Others can surely answer better, since I don't know the code; I guess I'd add a "type" field to the dictionary to store such information, or maybe to the call sites, depending on what's easier to annotate; but I also guess this naive solution could require lots of work. > d. > > > On 27 December 2010 12:50, Paolo Giarrusso wrote: >> On Mon, Dec 27, 2010 at 19:54, Dima Tisnek wrote: >>> what do you think of liburcu? lttng.org/urcu >> >> I've studied RCU in the Linux kernel (from which URCU derives) and >> thought for a long time of using it for this problem. In short, once >> you have GC, RCU (as in the Linux kernel) becomes (almost) trivial, >> because RCU is almost entirely about how to delay freeing objects. >> The RCU-GC connection is also mentioned by Documentation/RCU/RTFP.txt >> in the Linux sources. >> >> So, you need to just read about persistent data structures, and >> remember that persistence is not the point - data structures which are >> just almost persistent can also be OK, if you get them right. If you >> replace a value in a dictionary, for instance, you don't necessarily >> need to keep the old version around, as long as you the value fits in >> a memory word so that it can be atomically replaced. >> http://hackerboss.com/copy-on-write-101-part-2-putting-it-to-use/ >> >> Speaking of libraries, you might want to look at this for the atomic >> primitives you'll need: >> www.hpl.hp.com/research/linux/atomic_ops/ >> I'm more used to the primitives of the Linux kernel, I remember I >> didn't fully like that library, but IIRC it was a matter of taste - >> the library is from Hans Boehm, which makes it trustworthy. >> >>> p.s. if/when I get around to profiling pypy, which is something I want >>> to do, I'll be sure to share the results here. >> >> Great, thanks! >> >>> d. >>> >>> On 27 December 2010 10:57, Paolo Giarrusso wrote: >>>> On Mon, Dec 27, 2010 at 07:00, Dima Tisnek wrote: >>>>> (dan) random binary trees: O(log2(n)) is 7.2 for builtins, though the >>>>> constant factor for locks, etc, might make it worthwhile >>>> >>>> "7.2"? What is that? Do you mean 7.2=log2(no. of builtins)? >>>> >>>>> (paolo) non blocking hash maps: memory barriers can be quite costly too >>>> >>>> Well, you just _cannot_ avoid them, whatever is your fancy data structure. >>>> You can only move most of the cost (or all of it) to write operations >>>> - which is helpful if reads are most common, a case which you discuss >>>> in your mail. >>>> >>>> == Why memory barriers are required (in more detail) == >>>> If you use persistent data structures, also called copy-on-write or >>>> purely functional (and that's the most obvious way to use a tree in a >>>> parallel setting), the whole synchronization problem is reduced to >>>> exchanging a pointer (to the root of the tree) between a writer and a >>>> reader thread. >>>> Let's consider this in the Java memory model (the state of the art). >>>> In this model the writer thread has to use an expensive StoreLoad >>>> barrier, i.e. mfence on x86 [1], costing more or less as much as a >>>> cache miss, while the reader needs just a cheap Load* barrier (for >>>> free on x86, cheap on other processor). >>>> If you don't use a volatile field, there is very little guarantee >>>> about what your program means, and it is almost always buggy; the >>>> equivalent in the new C++ memory model (which is otherwise very >>>> similar) gives completely undefined semantics (and even crashes are >>>> possible in practice, for instance if the reader thread sees a >>>> not-fully-initialized object because of the race and invokes a virtual >>>> method on it). >>>> >>>> [1] http://g.oswego.edu/dl/jmm/cookbook.html >>>> >>>> == Other stuff == >>>>> different options will have to be implemented and tested when we get there, >>>>> speaking on which, is there a test dict load? >>>>> does anyone have some profiling data, what portion of runtime is spent >>>>> reading and writing attributes and globals anyways? >>>> >>>> I'd be interested as well in such data. >>>> >>>>> Anyhow, back to parallel interpretation, Is there an issue or page >>>>> that tracks what's needed for parallel interpretation? >>>>> so far ?mentioned: gc, dict, c api >>>> >>>>> Btw, I think that jit is more important at the moment, but time comes >>>>> when jit juice has been mostly squeezed out ;-) >>>> >>>> -- >>>> Paolo Giarrusso - Ph.D. Student >>>> http://www.informatik.uni-marburg.de/~pgiarrusso/ >>>> >>> >> >> >> >> -- >> Paolo Giarrusso - Ph.D. Student >> http://www.informatik.uni-marburg.de/~pgiarrusso/ >> > -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/