From cfbolz at gmx.de Tue Jul 1 00:58:51 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 01 Jul 2008 00:58:51 +0200 Subject: [pypy-dev] channel logs Message-ID: <4869652B.5050801@gmx.de> Hi all, just wanted to note that the braintone logs seem to work at the moment, so you can read the as a stopgap measure here: http://merlinux.de/braintone/chat.freenode.net/pypy/last/50/ The format is not so nice, so I would actually really like pybot back :-). Cheers, Carl Friedrich From marius at pov.lt Tue Jul 1 11:26:01 2008 From: marius at pov.lt (Marius Gedminas) Date: Tue, 1 Jul 2008 12:26:01 +0300 Subject: [pypy-dev] channel logs In-Reply-To: <4869652B.5050801@gmx.de> References: <4869652B.5050801@gmx.de> Message-ID: <20080701092601.GA27912@fridge.pov.lt> On Tue, Jul 01, 2008 at 12:58:51AM +0200, Carl Friedrich Bolz wrote: > Hi all, > > just wanted to note that the braintone logs seem to work at the moment, > so you can read the as a stopgap measure here: > > http://merlinux.de/braintone/chat.freenode.net/pypy/last/50/ > > The format is not so nice, so I would actually really like pybot back :-). Assuming you're on FreeNode, I could easily set up IRC logs that look like this: http://mg.pov.lt/maemo-irclog/latest.log.html Marius Gedminas -- Thus spake the master programmer: "After three days without programming, life becomes meaningless." -- Geoffrey James, "The Tao of Programming" -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://codespeak.net/pipermail/pypy-dev/attachments/20080701/78248f0e/attachment.pgp From tismer at stackless.com Tue Jul 1 12:16:35 2008 From: tismer at stackless.com (Christian Tismer) Date: Tue, 01 Jul 2008 12:16:35 +0200 Subject: [pypy-dev] channel logs In-Reply-To: <20080701092601.GA27912@fridge.pov.lt> References: <4869652B.5050801@gmx.de> <20080701092601.GA27912@fridge.pov.lt> Message-ID: <486A0403.6040804@stackless.com> Marius Gedminas wrote: > On Tue, Jul 01, 2008 at 12:58:51AM +0200, Carl Friedrich Bolz wrote: >> Hi all, >> >> just wanted to note that the braintone logs seem to work at the moment, >> so you can read the as a stopgap measure here: >> >> http://merlinux.de/braintone/chat.freenode.net/pypy/last/50/ >> >> The format is not so nice, so I would actually really like pybot back :-). > > Assuming you're on FreeNode, I could easily set up IRC logs that look > like this: http://mg.pov.lt/maemo-irclog/latest.log.html My pybot is working, again. It stopped working, when I did a dist-upgrade of my debian system. It runs fine in foreground, but when I try to deamonize it, it pretends to work, but doesn't. Will try to solve that, or switch to another bot. Meanwhile pybot is running foreground in a screen session. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Tue Jul 1 18:39:35 2008 From: arigo at tunes.org (Armin Rigo) Date: Tue, 1 Jul 2008 18:39:35 +0200 Subject: [pypy-dev] Small dicts Message-ID: <20080701163935.GA7076@code0.codespeak.net> Hi all, "withsmalldicts" is not enabled by default, I suppose because we measured that it didn't help. However I've found a bug which I think means that even when the option is enabled, they won't be used in most common cases. The bug is that the option is checked in EmptyDictImplementation.setitem() but not in EmptyDictImplementation.setitem_str(): the latter always creates a non-small StrDictImplementation. Anyone feels like fixing and re-measuring this? Armin From fijall at gmail.com Fri Jul 4 01:29:26 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 4 Jul 2008 01:29:26 +0200 Subject: [pypy-dev] [pypy-svn] r56275 - pypy/extradoc/talk/ep2008 In-Reply-To: <20080703203052.A66CB169F57@codespeak.net> References: <20080703203052.A66CB169F57@codespeak.net> Message-ID: <693bc9ab0807031629sbdea4a6p8f1fd9146b210157@mail.gmail.com> On Thu, Jul 3, 2008 at 10:30 PM, wrote: > Author: hpk > Date: Thu Jul 3 22:30:52 2008 > New Revision: 56275 > > Modified: > pypy/extradoc/talk/ep2008/status.txt > Log: > review, cutting out some negative not too telling > statements and adding a bit to plans from my perspective > I don't like this changes completely. I would like to say clearly what's not in our plans. From fijall at gmail.com Fri Jul 4 06:53:29 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 4 Jul 2008 06:53:29 +0200 Subject: [pypy-dev] problems running pypy-jvm Message-ID: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> First of all, I cannot get jar to work. It's complaining about '@' in front of paths. Even if I remove it, it still cannot find Main class. When I run it by hand I get: debug: WARNING: library path not found, using compiled-in sys.path 'import site' failed Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' What's wrong? Cheers, fijal From holger at merlinux.de Fri Jul 4 08:05:59 2008 From: holger at merlinux.de (holger krekel) Date: Fri, 4 Jul 2008 08:05:59 +0200 Subject: [pypy-dev] [pypy-svn] r56275 - pypy/extradoc/talk/ep2008 In-Reply-To: <693bc9ab0807031629sbdea4a6p8f1fd9146b210157@mail.gmail.com> References: <20080703203052.A66CB169F57@codespeak.net> <693bc9ab0807031629sbdea4a6p8f1fd9146b210157@mail.gmail.com> Message-ID: <20080704060559.GL759@trillke.net> On Fri, Jul 04, 2008 at 01:29 +0200, Maciej Fijalkowski wrote: > On Thu, Jul 3, 2008 at 10:30 PM, wrote: > > Author: hpk > > Date: Thu Jul 3 22:30:52 2008 > > New Revision: 56275 > > > > Modified: > > pypy/extradoc/talk/ep2008/status.txt > > Log: > > review, cutting out some negative not too telling > > statements and adding a bit to plans from my perspective > > > > I don't like this changes completely. I would like to say clearly > what's not in our plans. maybe mention it explicitely in the "cleanup sprint" slide before? making the very last slide about what we are not going to do is not a good ending IMO. holger From anto.cuni at gmail.com Fri Jul 4 09:57:33 2008 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 04 Jul 2008 09:57:33 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> Message-ID: <486DD7ED.9090107@gmail.com> Hi Maciek, Maciej Fijalkowski wrote: > First of all, I cannot get jar to work. It's complaining about '@' in > front of paths. Even if I remove it, it still cannot find Main class. what do you exactly mean? Could you paste the error message please? > When I run it by hand I get: > > debug: WARNING: library path not found, using compiled-in sys.path > 'import site' failed > Error calling sys.excepthook: > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: cannot import name 'curdir' > > What's wrong? No clue. I've translated pypy-jvm yesterday just after the merging of the less-meta-instance branch, and it works fine: antocuni at viper goal $ ./pypy-jvm Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``nothing is true'' >>>> If I move pypy-jvm to another dir and rename my wc to prevent it to find the compiled-in sys.path, I get another error, which seems reasonable: antocuni at viper tmp $ ./pypy-jvm debug: WARNING: library path not found, using compiled-in sys.path 'import site' failed Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 Type "help", "copyright", "credits" or "license" for more information. debug: OperationError: debug: operror-type: ImportError debug: operror-value: No module named _pypy_interact Which revision is your pypy-jvm? Any clue why it doesn't find the library path? What ./pypy-jvm -c 'import sys; print sys.path' prints? ciao, Anto From victor.stinner at haypocalc.com Mon Jul 7 00:23:38 2008 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 7 Jul 2008 00:23:38 +0200 Subject: [pypy-dev] Play with fuzzing Message-ID: <200807070023.38555.victor.stinner@haypocalc.com> Hi, I wrote a fuzzing "framework" called Fusil and this week I wrote a fuzzer for Python. The idea is quite simple: for a module, - list all functions, classes and class methods - call a function with random arguments (of random types) - instanciate a class with random arguments - if the class is created correctly, call methods with random arguments Example: --------------------- 8< ----------------------------------- print "Call 39/40: linuxaudiodev.open()" try: linuxaudiodev.open( # argument 1/2 u"\u62C0\uFBD7\uB46A\u55E0\uFB7B\uD392\u7CEE", # argument 2/2 52.682, ) except Exception, err: print >>stderr, "ERROR: %s" % err --------------------- 8< ----------------------------------- I tried it on CPython 2.5 and then on CPython trunk (future 2.6). I found some bugs: http://bugs.python.org/issue3304 -> invalid call to PyMem_Free() in fileio_init() http://bugs.python.org/issue3306 -> audioop.findmax() crashs with negative length http://bugs.python.org/issue3303 -> invalid ref count on locale.strcoll() error http://bugs.python.org/issue3302 -> segfault on gettext(None) http://bugs.python.org/issue3301 -> DoS when lo is negative in bisect.insort_right() http://bugs.python.org/issue3299 -> invalid object destruction in re.finditer() Most bugs crash with a segmentation fault, abort or a denial of service. If you would like to try my fuzzer, use: (1) svn co http://fusil.hachoir.org/svn/trunk fusil (2) cd fusil (3) ./run_fusil.sh -p projects/python.py --fast --remove ALL The option --fast goes faster, --remove does remove session directory even if Python generated some files, and "ALL" test all modules. FUSIL IS NOT SAFE! So run it under a different using to avoid dangerous call to os.unlink(). Module list is hardcoded: it's the list of CPython modules written in C. More informations about Fusil: http://fusil.hachoir.org/trac -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ From brunogola at gmail.com Mon Jul 7 16:31:36 2008 From: brunogola at gmail.com (Bruno Gola) Date: Mon, 7 Jul 2008 11:31:36 -0300 Subject: [pypy-dev] Changing the default magic number Message-ID: Hi, I'm writing code to support the new 2.5 generator stuff (PEP 342) and while working on that I found a code that verifies the magic number (in pyframe.py). The default magic in PyPy is the Python2.4 one, chatting with Carl we thought it's probably a good time to change it to the 2.5 value, as we are supporting Python 2.5. What do you think? -- Bruno Fialho Marques Gola http://www.brunogola.com.br Cel: (11) 9294-5883 From arigo at tunes.org Tue Jul 8 17:28:50 2008 From: arigo at tunes.org (Armin Rigo) Date: Tue, 8 Jul 2008 17:28:50 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: References: Message-ID: <20080708152850.GA8093@code0.codespeak.net> Hi Bruno, On Mon, Jul 07, 2008 at 11:31:36AM -0300, Bruno Gola wrote: > I'm writing code to support the new 2.5 generator stuff (PEP 342) and > while working on that I found a code that verifies the magic number > (in pyframe.py). The default magic in PyPy is the Python2.4 one, > chatting with Carl we thought it's probably a good time to change it > to the 2.5 value, as we are supporting Python 2.5. The important thing here is the opcodes that changed their semantics between 2.4 and 2.5. There is the YIELD opcode which is probably the one that you're referring to; in 2.5 it pushes a result on the value stack, but in 2.4 it didn't. There is also one of the IMPORT opcodes, which pops one more argument from the stack in 2.5. So the magic version number in PyPy should be changed to 2.5's at the same time as these two opcodes change their semantics in PyPy's compiler and in pyopcode.py. A bientot, Armin From cfbolz at gmx.de Wed Jul 9 14:30:24 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 09 Jul 2008 14:30:24 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: <20080708152850.GA8093@code0.codespeak.net> References: <20080708152850.GA8093@code0.codespeak.net> Message-ID: <4874AF60.1060807@gmx.de> Hi Armin, Armin Rigo wrote: > On Mon, Jul 07, 2008 at 11:31:36AM -0300, Bruno Gola wrote: >> I'm writing code to support the new 2.5 generator stuff (PEP 342) and >> while working on that I found a code that verifies the magic number >> (in pyframe.py). The default magic in PyPy is the Python2.4 one, >> chatting with Carl we thought it's probably a good time to change it >> to the 2.5 value, as we are supporting Python 2.5. > > The important thing here is the opcodes that changed their semantics > between 2.4 and 2.5. There is the YIELD opcode which is probably the > one that you're referring to; in 2.5 it pushes a result on the value > stack, but in 2.4 it didn't. There is also one of the IMPORT opcodes, > which pops one more argument from the stack in 2.5. So the magic > version number in PyPy should be changed to 2.5's at the same time as > these two opcodes change their semantics in PyPy's compiler and in > pyopcode.py. I had the impression that we use a PyPy-specific magic number anyway, in which case it doesn't matter that we change the magic only when all the new opcode semantics are fixed. Is that wrong? Have fun at the sprint! Carl Friedrich From fijall at gmail.com Thu Jul 10 09:24:34 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 10 Jul 2008 09:24:34 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: <4874AF60.1060807@gmx.de> References: <20080708152850.GA8093@code0.codespeak.net> <4874AF60.1060807@gmx.de> Message-ID: <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> Magic in pyc files and magic as a number for pyopcode are different things IMO. On Wed, Jul 9, 2008 at 2:30 PM, Carl Friedrich Bolz wrote: > Hi Armin, > > Armin Rigo wrote: >> On Mon, Jul 07, 2008 at 11:31:36AM -0300, Bruno Gola wrote: >>> I'm writing code to support the new 2.5 generator stuff (PEP 342) and >>> while working on that I found a code that verifies the magic number >>> (in pyframe.py). The default magic in PyPy is the Python2.4 one, >>> chatting with Carl we thought it's probably a good time to change it >>> to the 2.5 value, as we are supporting Python 2.5. >> >> The important thing here is the opcodes that changed their semantics >> between 2.4 and 2.5. There is the YIELD opcode which is probably the >> one that you're referring to; in 2.5 it pushes a result on the value >> stack, but in 2.4 it didn't. There is also one of the IMPORT opcodes, >> which pops one more argument from the stack in 2.5. So the magic >> version number in PyPy should be changed to 2.5's at the same time as >> these two opcodes change their semantics in PyPy's compiler and in >> pyopcode.py. > > I had the impression that we use a PyPy-specific magic number anyway, in > which case it doesn't matter that we change the magic only when all the > new opcode semantics are fixed. Is that wrong? > > Have fun at the sprint! > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From brunogola at gmail.com Thu Jul 10 16:15:36 2008 From: brunogola at gmail.com (Bruno Gola) Date: Thu, 10 Jul 2008 11:15:36 -0300 Subject: [pypy-dev] Changing the default magic number In-Reply-To: <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> References: <20080708152850.GA8093@code0.codespeak.net> <4874AF60.1060807@gmx.de> <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> Message-ID: Hi, On Thu, Jul 10, 2008 at 4:24 AM, Maciej Fijalkowski wrote: > Magic in pyc files and magic as a number for pyopcode are different things IMO. Yes, in PyPy they are different things. The magic number in .pyc files compiled by PyPy is 1024. PyPy doesn't try to use .pyc generated by CPython and CPython will not try to use .pyc generated by PyPy (because of these magic number). But the magic I'm talking about is the default value for PyCode magic attribute. In pycode.py there is comment saying "value for Python 2.4.1" and a default value of 62061 for these attribute. These number is checked in many cases (or at least two) where, as Armin said, changes to the semantics of bytecodes have been made. It seems to me that these default value is always used, I could not find any place where a PyCode instance is created changing these value, but I may be wrong. Anyway, the semantics for YIELD_VALUE have already been changed (in 2.5-features branch) and i think it's good idea to change the other opcodes that have changed as well. The magic attribute is checked against the magic value for Python2.5 (and other) some times (probably when the opcode semantics changed and PyPy wanted to support both semantics) but as the default value is never changed (as far as I know) I think those checks are not being useful. I think we could change these value to the 2.5 one. I think it will not affect anything (only these opcodes that we are changing as well). Cheers, -- Bruno Fialho Marques Gola http://www.brunogola.com.br Cel: (11) 9294-5883 From zeroth at oddco.ca Thu Jul 10 21:57:06 2008 From: zeroth at oddco.ca (Tyler Laing) Date: Thu, 10 Jul 2008 12:57:06 -0700 Subject: [pypy-dev] Using pypy's translator? Message-ID: <3618a6e10807101257p68629f49k983a92c6643b8771@mail.gmail.com> Hello all. My name is Tyler, or Zeroth as my pseudonym. I'm currently undertaking a personal project, that seeks to explore pseudo-strict type checking for Python. More simply, it would analyze the code, and determine what methods/functions/operators each variable would need to support in the code, and then enforcing that all variables passed around follow the informal contract. It would merely enforce that the methods are present, and return the right(by right, meaning variables with the expected methods) types. It is an extension of duck typing; "If it walks like a duck, and talks like a duck, lets make sure they're expecting a duck!" I have been following the pypy project for a little while, and there are a tonne of very cool features. Of particular interest is the lexer/parser for Python. First, is it okay if I use it for my project(open source itself), and second, any advice on using it? Tutorials? Gotchas, warnings? Thanks in advance, and have lots of fun. -Tyler -- Visit my blog at http://oddco.ca/zeroth/zblog -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080710/d6c7da38/attachment.htm From holger at merlinux.de Sun Jul 13 18:00:37 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 13 Jul 2008 18:00:37 +0200 Subject: [pypy-dev] alias for --allopts? Message-ID: <20080713160037.GA759@trillke.net> Hi pypy-dev, would be good to rename the --faassen option sometime soon Martijn asked us multiple times already for it. What about "--cheetah" or "--falcon"? other suggestions? best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From cfbolz at gmx.de Sun Jul 13 18:14:04 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sun, 13 Jul 2008 18:14:04 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713160037.GA759@trillke.net> References: <20080713160037.GA759@trillke.net> Message-ID: <487A29CC.7090602@gmx.de> holger krekel wrote: > would be good to rename the --faassen option sometime soon > Martijn asked us multiple times already for it. What about > "--cheetah" or "--falcon"? other suggestions? I don't really like --falcon, falcons can only be that fast when flying downwards (crashing?). How about --sailfish (fastest fish, only very slightly slower than a Cheeta) or maybe even --blackmamba (fastest snake)? Cheers, Carl Friedrich From eric at vanrietpaap.nl Sun Jul 13 18:43:58 2008 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Sun, 13 Jul 2008 18:43:58 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <487A29CC.7090602@gmx.de> References: <20080713160037.GA759@trillke.net> <487A29CC.7090602@gmx.de> Message-ID: What about --fastest :-) cheers Eric On Sun, Jul 13, 2008 at 6:14 PM, Carl Friedrich Bolz wrote: > holger krekel wrote: > > would be good to rename the --faassen option sometime soon > > Martijn asked us multiple times already for it. What about > > "--cheetah" or "--falcon"? other suggestions? > > I don't really like --falcon, falcons can only be that fast when flying > downwards (crashing?). How about --sailfish (fastest fish, only very > slightly slower than a Cheeta) or maybe even --blackmamba (fastest snake)? > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080713/6be1fe87/attachment-0001.htm From holger at merlinux.de Sun Jul 13 18:52:47 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 13 Jul 2008 18:52:47 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <487A29CC.7090602@gmx.de> References: <20080713160037.GA759@trillke.net> <487A29CC.7090602@gmx.de> Message-ID: <20080713165247.GC759@trillke.net> On Sun, Jul 13, 2008 at 18:14 +0200, Carl Friedrich Bolz wrote: > holger krekel wrote: > > would be good to rename the --faassen option sometime soon > > Martijn asked us multiple times already for it. What about > > "--cheetah" or "--falcon"? other suggestions? > > I don't really like --falcon, falcons can only be that fast when flying > downwards (crashing?). How about --sailfish (fastest fish, only very > slightly slower than a Cheeta) or maybe even --blackmamba (fastest snake)? let's keep "blackmamba" as the release name for when we include the JIT by default :) --sailfish sounds slow to me (because "sailing" is not the fastest movement method i guess) but after reading wikipedia i kind of like it. i also like "--faastest", btw :) holger From arigo at tunes.org Sun Jul 13 23:19:01 2008 From: arigo at tunes.org (Armin Rigo) Date: Sun, 13 Jul 2008 23:19:01 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713160037.GA759@trillke.net> References: <20080713160037.GA759@trillke.net> Message-ID: <20080713211900.GA31170@code0.codespeak.net> Hi Holger, On Sun, Jul 13, 2008 at 06:00:37PM +0200, holger krekel wrote: > would be good to rename the --faassen option sometime soon > Martijn asked us multiple times already for it. What about > "--cheetah" or "--falcon"? other suggestions? "--allopts" is already an alias for it, and I changed the docs to mention it instead of --faassen. I also came up with some suggestion for the optimization options. What about introducing the --opt translation option (applying to all translations, not just targetpypystandalone) with the following possible values, which roughly correspond to the scheme used by gcc: --opt=0: all backendopt and other optimizations off; gc=boehm --opt=1: default backendopts, low inlining threadhold; gc=boehm --opt=size: same as opt=1 + remove_asserts --opt=mem: same as opt=size + gc=marknsweep (or some future low-ram gc) --opt=2: default backendopts; smallfuncset=5; gc=hybrid --opt=3: same as opt=2 + remove_asserts During a PyPy translation, we would control which pypy-specific options to use with the same scheme: a --pypyopt option (whose value is by default equal to the --opt option in order to reduce surprizes) according to a table like the following one: pypyopt option =================================== 2 3 ("objspace.opcodes.CALL_LIKELY_BUILTIN", True), 2 3 ("objspace.opcodes.CALL_METHOD", True), 3 ("translation.profopt", "..."), 2 3 ("objspace.std.withmultidict", True), 2 3 ("objspace.std.withshadowtracking", True), mem ("objspace.std.withsmallint", True), mem 2 3 ("objspace.std.withrangelist", True), 2 3 ("objspace.std.withmethodcache", True), mem 2 3 ("objspace.std.withprebuiltchar", True), 2 3 ("objspace.std.builtinshortcut", True), 2 3 ("objspace.std.optimized_list_getitem", True), 2 3 ("objspace.std.getattributeshortcut", True), mem ("objspace.std.withsharingdict", True), So this would generalize the --faassen and --allopts options. The fastest results would be obtained with --opt=3, --opt=size would try to minimize the executable size, --opt=mem would try to minimize run-time RAM usage. A bientot, Armin. From holger at merlinux.de Sun Jul 13 23:41:18 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 13 Jul 2008 23:41:18 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713211900.GA31170@code0.codespeak.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> Message-ID: <20080713214118.GE759@trillke.net> Hi Armin, On Sun, Jul 13, 2008 at 23:19 +0200, Armin Rigo wrote: > On Sun, Jul 13, 2008 at 06:00:37PM +0200, holger krekel wrote: > > would be good to rename the --faassen option sometime soon > > Martijn asked us multiple times already for it. What about > > "--cheetah" or "--falcon"? other suggestions? > > "--allopts" is already an alias for it, and I changed the docs > to mention it instead of --faassen. ah, good. i only checked that "--allopts" is already there for a long time. > I also came up with some suggestion for the optimization options. > What about introducing the --opt translation option (applying to all > translations, not just targetpypystandalone) with the following possible > values, which roughly correspond to the scheme used by gcc: > > --opt=0: all backendopt and other optimizations off; gc=boehm > --opt=1: default backendopts, low inlining threadhold; gc=boehm > --opt=size: same as opt=1 + remove_asserts > --opt=mem: same as opt=size + gc=marknsweep (or some future low-ram gc) > --opt=2: default backendopts; smallfuncset=5; gc=hybrid > --opt=3: same as opt=2 + remove_asserts i like the idea. But I find mixing in the choice of GC a bit odd. what about defaulting to gc=hybrid and getting rid of the boehm dependency? > During a PyPy translation, we would control which pypy-specific options > to use with the same scheme: a --pypyopt option (whose value is by > default equal to the --opt option in order to reduce surprizes) > according to a table like the following one: > > pypyopt option > =================================== > > 2 3 ("objspace.opcodes.CALL_LIKELY_BUILTIN", True), > 2 3 ("objspace.opcodes.CALL_METHOD", True), > 3 ("translation.profopt", "..."), > 2 3 ("objspace.std.withmultidict", True), > 2 3 ("objspace.std.withshadowtracking", True), > mem ("objspace.std.withsmallint", True), > mem 2 3 ("objspace.std.withrangelist", True), > 2 3 ("objspace.std.withmethodcache", True), > mem 2 3 ("objspace.std.withprebuiltchar", True), > 2 3 ("objspace.std.builtinshortcut", True), > 2 3 ("objspace.std.optimized_list_getitem", True), > 2 3 ("objspace.std.getattributeshortcut", True), > mem ("objspace.std.withsharingdict", True), > > So this would generalize the --faassen and --allopts options. The > fastest results would be obtained with --opt=3, --opt=size would try to > minimize the executable size, --opt=mem would try to minimize run-time > RAM usage. Looks fine to me! cheers, holger From micahel at gmail.com Mon Jul 14 00:12:09 2008 From: micahel at gmail.com (Michael Hudson) Date: Mon, 14 Jul 2008 10:12:09 +1200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713214118.GE759@trillke.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> <20080713214118.GE759@trillke.net> Message-ID: 2008/7/14 holger krekel : > Hi Armin, > > On Sun, Jul 13, 2008 at 23:19 +0200, Armin Rigo wrote: >> On Sun, Jul 13, 2008 at 06:00:37PM +0200, holger krekel wrote: >> > would be good to rename the --faassen option sometime soon >> > Martijn asked us multiple times already for it. What about >> > "--cheetah" or "--falcon"? other suggestions? >> >> "--allopts" is already an alias for it, and I changed the docs >> to mention it instead of --faassen. > > ah, good. i only checked that "--allopts" is already there > for a long time. FWIW, I think I added --allopts when I was writing up one of the EU reports, not quite being able to bring myself to write about --faassen in serious prose... Cheers, mwh From holger at merlinux.de Mon Jul 14 16:05:18 2008 From: holger at merlinux.de (holger krekel) Date: Mon, 14 Jul 2008 16:05:18 +0200 Subject: [pypy-dev] EP sprint report / blog Message-ID: <20080714140518.GE22122@trillke.net> Hi pypy-dev, not sure how many pypy-dev subscribers are also reading the Blog regularly - we have been posting there about some EuroPythion 2008 events and also the sprint report, see http://morepypy.blogspot.com/ Btw, I'd find it nice to also have a blog->pypy-dev gateway - ideally one that includes relaying e-mail / comments to and fro. I wonder which blogging software might support something like this. cheers, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From cfbolz at gmx.de Mon Jul 14 16:30:33 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 14 Jul 2008 16:30:33 +0200 Subject: [pypy-dev] EP sprint report / blog In-Reply-To: <20080714140518.GE22122@trillke.net> References: <20080714140518.GE22122@trillke.net> Message-ID: <487B6309.7010100@gmx.de> Hi Holger, holger krekel wrote: > not sure how many pypy-dev subscribers are also reading the > Blog regularly - we have been posting there about some > EuroPythion 2008 events and also the sprint report, see > http://morepypy.blogspot.com/ > > Btw, I'd find it nice to also have a blog->pypy-dev > gateway - ideally one that includes relaying > e-mail / comments to and fro. I wonder which > blogging software might support something like this. Sending e-mails to pypy-dev with the blog text is easy. I can set that up, if everybody thinks this is sensible. Also relaying the comments wouldn't be too hard. I have no clue whether somebody wrote something to push e-mail answers on the mailing list back as blog comments into the blog. Cheers, Carl Friedrich From holger at merlinux.de Mon Jul 14 16:44:16 2008 From: holger at merlinux.de (holger krekel) Date: Mon, 14 Jul 2008 16:44:16 +0200 Subject: [pypy-dev] EP sprint report / blog In-Reply-To: <487B6309.7010100@gmx.de> References: <20080714140518.GE22122@trillke.net> <487B6309.7010100@gmx.de> Message-ID: <20080714144416.GG22122@trillke.net> On Mon, Jul 14, 2008 at 16:30 +0200, Carl Friedrich Bolz wrote: > holger krekel wrote: > > not sure how many pypy-dev subscribers are also reading the > > Blog regularly - we have been posting there about some > > EuroPythion 2008 events and also the sprint report, see > > http://morepypy.blogspot.com/ > > > > Btw, I'd find it nice to also have a blog->pypy-dev > > gateway - ideally one that includes relaying > > e-mail / comments to and fro. I wonder which > > blogging software might support something like this. > > Sending e-mails to pypy-dev with the blog text is easy. I can set that > up, if everybody thinks this is sensible. Also relaying the comments > wouldn't be too hard. I have no clue whether somebody wrote something to > push e-mail answers on the mailing list back as blog comments into the blog. For relaying the comments things need to get properly threaded in mail readers and email->blog needs to work as well but quick Googling revealed nothing to me :( i find this strange with all the blogs and mailing lists out there. Without this integration, i don't know if it makes sense. but maybe someone knows some more or has other opinions. holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From arigo at tunes.org Mon Jul 14 17:27:28 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 14 Jul 2008 17:27:28 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: References: <20080708152850.GA8093@code0.codespeak.net> <4874AF60.1060807@gmx.de> <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> Message-ID: <20080714152728.GA29978@code0.codespeak.net> Hi, On Thu, Jul 10, 2008 at 11:15:36AM -0300, Bruno Gola wrote: > I think we could change these value to the 2.5 one. I think it will > not affect anything (only these opcodes that we are changing as well). I believe that we are still trying to get code objects compiled by the underlying Python to run in PyPy. This is a case where the magic number in the PyCode can be different: it's used to know if we need to emulate the 2.4 or 2.5 behavior, depending on the underlying Python's version number. It would be a bit of a mess to remove this hack, though. For example the flow object space looks at the regular Python function object's func_code, turned into PyCode, so it must be able to accept PyCode objects that come from the underlying Python's compiler... A bientot, Armin. From arigo at tunes.org Mon Jul 14 17:29:37 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 14 Jul 2008 17:29:37 +0200 Subject: [pypy-dev] Using pypy's translator? In-Reply-To: <3618a6e10807101257p68629f49k983a92c6643b8771@mail.gmail.com> References: <3618a6e10807101257p68629f49k983a92c6643b8771@mail.gmail.com> Message-ID: <20080714152937.GB29978@code0.codespeak.net> Hi, On Thu, Jul 10, 2008 at 12:57:06PM -0700, Tyler Laing wrote: > I have been following the pypy project for a little while, and there are a > tonne of very cool features. Of particular interest is the lexer/parser for > Python. First, is it okay if I use it for my project(open source itself), > and second, any advice on using it? Tutorials? Gotchas, warnings? Feel free to reuse any code. A warning is that our current lexer/parser is not too clean, though. A bientot, Armin. From arigo at tunes.org Mon Jul 14 17:32:06 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 14 Jul 2008 17:32:06 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713214118.GE759@trillke.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> <20080713214118.GE759@trillke.net> Message-ID: <20080714153206.GC29978@code0.codespeak.net> Hi Holger, On Sun, Jul 13, 2008 at 11:41:18PM +0200, holger krekel wrote: > i like the idea. But I find mixing in the choice of GC a bit odd. > what about defaulting to gc=hybrid and getting rid of > the boehm dependency? There are several reasons for wanting Boehm, but the main one is that translation is much faster and takes less memory, which is the whole point of an option like "--opt=0". The default value for --opt should probably be 2, so that by default we would get a good and Boehm-independent result. A bientot, Armin. From holger at merlinux.de Mon Jul 14 17:46:04 2008 From: holger at merlinux.de (holger krekel) Date: Mon, 14 Jul 2008 17:46:04 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080714153206.GC29978@code0.codespeak.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> <20080713214118.GE759@trillke.net> <20080714153206.GC29978@code0.codespeak.net> Message-ID: <20080714154604.GH22122@trillke.net> Hi Armin, On Mon, Jul 14, 2008 at 17:32 +0200, Armin Rigo wrote: > On Sun, Jul 13, 2008 at 11:41:18PM +0200, holger krekel wrote: > > i like the idea. But I find mixing in the choice of GC a bit odd. > > what about defaulting to gc=hybrid and getting rid of > > the boehm dependency? > > There are several reasons for wanting Boehm, but the main one is that > translation is much faster and takes less memory, which is the whole > point of an option like "--opt=0". > > The default value for --opt should probably be 2, so that by default we > would get a good and Boehm-independent result. sounds fine to me. i guess that we'd just drop "--faassen" then. holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From paskma at gmail.com Mon Jul 14 22:30:26 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Mon, 14 Jul 2008 22:30:26 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <486DD7ED.9090107@gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> Message-ID: <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> Hi pypy hackers :-), I have the same problem. pypy-jvm is unable to start in interactive mode: paskma at paskma:goal$ ./pypy-jvm 'import site' failed Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``choose your hack'' Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' Here is an investigation of sys.path, maybe it helps you: paskma at paskma:goal$ pwd /home/data/opt/svnpypy/dist/pypy/translator/goal paskma at paskma:goal$ ./pypy-c -c 'import sys; print sys.path' ['/home/data/opt/svnpypy/dist/pypy/translator/goal', '/home/data/opt/svnpypy/dist/pypy/lib', '/home/data/opt/svnpypy/dist/lib-python/modified-2.4.1', '/home/data/opt/svnpypy/dist/lib-python/2.4.1', '/usr/lib/python2.4/site-packages'] paskma at paskma:goal$ ./pypy-jvm -c 'import sys; print sys.path' 'import site' failed ['', '/home/data/opt/svnpypy/dist/pypy/lib', '/home/data/opt/svnpypy/dist/lib-python/modified-2.4.1', '/home/data/opt/svnpypy/dist/lib-python/2.4.1'] The sys.path of the pypy-jvm lacks '/home/data/opt/svnpypy/dist/pypy/translator/goal' and mainly '/usr/lib/python2.4/site-packages'. The translation of the pypy-jvm was done without any significant switches. Marek Pa?ka On Fri, Jul 4, 2008 at 09:57, Antonio Cuni wrote: > Hi Maciek, > > Maciej Fijalkowski wrote: >> First of all, I cannot get jar to work. It's complaining about '@' in >> front of paths. Even if I remove it, it still cannot find Main class. > > what do you exactly mean? Could you paste the error message please? > >> When I run it by hand I get: >> >> debug: WARNING: library path not found, using compiled-in sys.path >> 'import site' failed >> Error calling sys.excepthook: >> debug: OperationError: >> debug: operror-type: ImportError >> debug: operror-value: cannot import name 'curdir' >> >> What's wrong? > > No clue. I've translated pypy-jvm yesterday just after the merging of > the less-meta-instance branch, and it works fine: > > antocuni at viper goal $ ./pypy-jvm > Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ``nothing is true'' > >>>> > > If I move pypy-jvm to another dir and rename my wc to prevent it to find > the compiled-in sys.path, I get another error, which seems reasonable: > > antocuni at viper tmp $ ./pypy-jvm > debug: WARNING: library path not found, using compiled-in sys.path > 'import site' failed > Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: No module named _pypy_interact > > Which revision is your pypy-jvm? Any clue why it doesn't find the > library path? > What ./pypy-jvm -c 'import sys; print sys.path' prints? > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Wed Jul 16 10:25:00 2008 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 16 Jul 2008 10:25:00 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> Message-ID: <487DB05C.6040702@gmail.com> Marek Pa?ka wrote: > Hi pypy hackers :-), > > I have the same problem. pypy-jvm is unable to start in interactive mode: > > paskma at paskma:goal$ ./pypy-jvm > 'import site' failed > Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ``choose your hack'' > Error calling sys.excepthook: > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: cannot import name 'curdir' that's very weird, I've never managed to reproduce this error. Could you put pypy-jvm.jar somewhere on the web so that I can try it on my machine, please? Or you could try to set DEBUG=True in translator/goal/app_main.py, recompile and see if we get more informative messages. thank you very much! ciao, Anto From paskma at gmail.com Wed Jul 16 11:08:42 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Wed, 16 Jul 2008 11:08:42 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <487DB05C.6040702@gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> Message-ID: <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> Hi, you can obtain it from here: http://147.228.64.164/downloads/pypy-jvm.jar And I am building DEBUG=True version right now :-) Marek On Wed, Jul 16, 2008 at 10:25, Antonio Cuni wrote: > Marek Pa?ka wrote: >> >> Hi pypy hackers :-), >> >> I have the same problem. pypy-jvm is unable to start in interactive mode: >> >> paskma at paskma:goal$ ./pypy-jvm >> 'import site' failed >> Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> ``choose your hack'' >> Error calling sys.excepthook: >> debug: OperationError: >> debug: operror-type: ImportError >> debug: operror-value: cannot import name 'curdir' > > that's very weird, I've never managed to reproduce this error. > Could you put pypy-jvm.jar somewhere on the web so that I can try it on my > machine, please? > > Or you could try to set DEBUG=True in translator/goal/app_main.py, recompile > and see if we get more informative messages. > > thank you very much! > ciao, > Anto > From paskma at gmail.com Wed Jul 16 11:46:49 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Wed, 16 Jul 2008 11:46:49 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> Message-ID: <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> You can download DEBUG=True version from here: http://147.228.64.164/downloads/pypy-jvm.jar-debugOn The error message is now little more specific: paskma at paskma:goal$ ./pypy-jvm 'import site' failed Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``peephope optimizations are what a Sufficiently Smart Compiler uses'' debug: exception-type: ImportError debug: exception-value: cannot import name 'curdir' debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' Marek On Wed, Jul 16, 2008 at 11:08, Marek Pa?ka wrote: > Hi, > > you can obtain it from here: > > http://147.228.64.164/downloads/pypy-jvm.jar > > And I am building DEBUG=True version right now :-) > > Marek > > On Wed, Jul 16, 2008 at 10:25, Antonio Cuni wrote: >> Marek Pa?ka wrote: >>> >>> Hi pypy hackers :-), >>> >>> I have the same problem. pypy-jvm is unable to start in interactive mode: >>> >>> paskma at paskma:goal$ ./pypy-jvm >>> 'import site' failed >>> Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> ``choose your hack'' >>> Error calling sys.excepthook: >>> debug: OperationError: >>> debug: operror-type: ImportError >>> debug: operror-value: cannot import name 'curdir' >> >> that's very weird, I've never managed to reproduce this error. >> Could you put pypy-jvm.jar somewhere on the web so that I can try it on my >> machine, please? >> >> Or you could try to set DEBUG=True in translator/goal/app_main.py, recompile >> and see if we get more informative messages. >> >> thank you very much! >> ciao, >> Anto >> > From anto.cuni at gmail.com Wed Jul 16 12:43:03 2008 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 16 Jul 2008 12:43:03 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> Message-ID: <487DD0B7.9030708@gmail.com> Marek Pa?ka wrote: > You can download DEBUG=True version from here: > > http://147.228.64.164/downloads/pypy-jvm.jar-debugOn it works fine on my machine. > The error message is now little more specific: > > paskma at paskma:goal$ ./pypy-jvm > 'import site' failed > Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ``peephope optimizations are what a Sufficiently Smart Compiler uses'' > debug: exception-type: ImportError > debug: exception-value: cannot import name 'curdir' > debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 > Error calling sys.excepthook: > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: cannot import name 'curdir' weird. It fails when it tries to import curdir from os.path; could you try the following please? ./pypy-jvm -c 'import os.path; print os.path' ./pypy-jvm -c' import os.path; print dir(os.path)' or maybe just come on #pypy on freenode, so we can try real time debugging over irc :-). My nick is antocuni. ciao, Anto From paskma at gmail.com Wed Jul 16 13:38:38 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Wed, 16 Jul 2008 13:38:38 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <487DD0B7.9030708@gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> <487DD0B7.9030708@gmail.com> Message-ID: <4ddb710e0807160438s64a7f818g5cd1ddf9a42db966@mail.gmail.com> The trace is always the same; the os module ties to import 'curdir'. I logged to #pypy. paskma at paskma:goal$ ./pypy-jvm -c 'import os.path; print os.path' 'import site' failed debug: exception-type: ImportError debug: exception-value: cannot import name 'curdir' debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' paskma at paskma:goal$ ./pypy-jvm -c 'import os.path; print dir(os.path)' 'import site' failed debug: exception-type: ImportError debug: exception-value: cannot import name 'curdir' debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' Marek On Wed, Jul 16, 2008 at 12:43, Antonio Cuni wrote: > Marek Pa?ka wrote: >> >> You can download DEBUG=True version from here: >> >> http://147.228.64.164/downloads/pypy-jvm.jar-debugOn > > it works fine on my machine. > >> The error message is now little more specific: >> >> paskma at paskma:goal$ ./pypy-jvm >> 'import site' failed >> Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> ``peephope optimizations are what a Sufficiently Smart Compiler uses'' >> debug: exception-type: ImportError >> debug: exception-value: cannot import name 'curdir' >> debug: exception-tb: >> /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 >> Error calling sys.excepthook: >> debug: OperationError: >> debug: operror-type: ImportError >> debug: operror-value: cannot import name 'curdir' > > weird. It fails when it tries to import curdir from os.path; could you try > the following please? > > ./pypy-jvm -c 'import os.path; print os.path' > ./pypy-jvm -c' import os.path; print dir(os.path)' > > > or maybe just come on #pypy on freenode, so we can try real time debugging > over irc :-). My nick is antocuni. > > ciao, > Anto > From ranjithkannikara at gmail.com Thu Jul 17 09:08:31 2008 From: ranjithkannikara at gmail.com (ranjith kannikara) Date: Thu, 17 Jul 2008 12:38:31 +0530 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. Message-ID: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> I have taken the gsoc 08 project of porting zope2 to python2.5. Through my way to the successful completetion of the project I have to implement Restricted python in Zope2. Chris Withers, consultant of simplistix have volunteered to give guidance and help in the RPython implementation I was also suggested by him to engage with the PyPy guys on their mailing list. and work with their guidance to re-implement RestrictedPython in a way that doesn't use AST hacks. Being a student I am not much familiar with the RPython yet need some help to get the RPython implementaion started.. How can I get started with the RPython implementation.. From holger at merlinux.eu Thu Jul 17 09:25:08 2008 From: holger at merlinux.eu (holger krekel) Date: Thu, 17 Jul 2008 09:25:08 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> Message-ID: <20080717072508.GF16827@trillke.net> Hello Ranjith, names are confusing here. PyPy's "RPython" is something else than Zope2's Restricted PYthon. I guess Chris Withers was refering to the two lightning talks that he and then Armin/Maciej gave at EuroPython 2008 both carrying "Restricted Python" in their title. For the future, I suggest to avoid the "R" term and rather talk about _sandboxing_ Python code which is what both talks/concepts were about. Indeed, PyPy provides a robust way of sandboxing its Python Interpreter, see http://codespeak.net/pypy/dist/pypy/doc/sandbox.html I am not sure how well this fits in your "porting zope2 to python2.5" project - but to me it looks like a good idea to go for using pypy-sandbox in the future instead of trying to get Zope2's "RestrictedPython" secure and ported to 2.5. cheers, holger On Thu, Jul 17, 2008 at 12:38 +0530, ranjith kannikara wrote: > I have taken the gsoc 08 project of porting zope2 to python2.5. > Through my way to the successful completetion of the project I have to > implement Restricted python in Zope2. Chris Withers, consultant of > simplistix have volunteered to give guidance and help in the RPython > implementation I was also suggested by him to engage with the PyPy > guys on their mailing list. and work with their guidance to > re-implement RestrictedPython in a way that doesn't use AST hacks. > Being a student I am not much familiar with the RPython yet need some > help to get the RPython implementaion started.. > > How can I get started with the RPython implementation.. -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From holger at merlinux.eu Thu Jul 17 11:40:09 2008 From: holger at merlinux.eu (holger krekel) Date: Thu, 17 Jul 2008 11:40:09 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <487EF825.2020704@simplistix.co.uk> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> Message-ID: <20080717094009.GA22440@trillke.net> Hey Chris, all, On Thu, Jul 17, 2008 at 08:43 +0100, Chris Withers wrote: > holger krekel wrote: >> names are confusing here. > > Indeed, I think Ranjith can be forgiven for getting confused given the > proliferation of similar names in this area ;-) > > For clarity, shall we agree to refer to things as: > > RestrictedPython - the current Zope 2 implementation found at > http://pypi.python.org/pypi/RestrictedPython > > Sandboxing - the technique of getting a secure and restricted python > environment running. > > RPython - some special in the pypy world ;-) "some special", aha :) i just added a FAQ entry: http://codespeak.net/pypy/dist/pypy/doc/faq.html#does-rpython-have-anything-to-do-with-zope-s-restricted-python >> Indeed, PyPy provides a robust way of sandboxing its Python >> Interpreter, see >> >> http://codespeak.net/pypy/dist/pypy/doc/sandbox.html > > Indeed, although as it is, I'm not sure how good a fit it is. > >> I am not sure how well this fits in your "porting zope2 to >> python2.5" project - but to me it looks like a good idea to go for >> using pypy-sandbox in the future instead of trying to get Zope2's >> "RestrictedPython" secure and ported to 2.5. > > Well, the rough feature request list is: > > - something that behaves like a python function > - can be pickled this is basically about storing user-provided scripts in the zodb, right? > - prevents people stripping security proxies from objects that are > passed in to the function hum, pypy-sandbox currently virtualizes at a low level (access to the Operating Systems files, io, etc.) and provides a *very* trustable way of running untrusted code. For providing higher and python level virtualization more work is needed so it might indeed not directly fit some use cases that RestrictedPython tries to satisfy. I am interested in this as are others, i think. Could you or someone else point to some concrete usages and use cases of the current RestrictedPython? > - prevents people importing things they shouldn't > - prevents people from getting hold of un-security-proxies-objects > - executes as fast as possible > - each function plays in its own sandbox and can be called many many > times, often simultaneously in different threads > - limits the amount of memory and processor they can hog (nice to have) the rest, i think, is quite doable, particularly limiting RAM/CPU resources. > I'm sure others will pitch in if I've forgotten anything... best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From chris at simplistix.co.uk Fri Jul 18 17:28:11 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 18 Jul 2008 16:28:11 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20080717094009.GA22440@trillke.net> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> Message-ID: <4880B68B.3080803@simplistix.co.uk> holger krekel wrote: >> - something that behaves like a python function >> - can be pickled > > this is basically about storing user-provided scripts > in the zodb, right? ...or relational database, or, etc ;-) >> - prevents people stripping security proxies from objects that are >> passed in to the function > > hum, pypy-sandbox currently virtualizes at a low level > (access to the Operating Systems files, io, etc.) and > provides a *very* trustable way of running untrusted code. Right, which is exactly what we want ;-) The security proxying is handled by a different method entirely. (The proxy objects are c extensions and prevent accessing attributes based on a pluggable set of rules) However, with access to unrestricted imports, all you'd need to do is import the unwrap method from the proxy wrapping module and you'd be fre of the jail ;-) > For providing higher and python level virtualization more work > is needed so it might indeed not directly fit some use cases > that RestrictedPython tries to satisfy. I am interested in > this as are others, i think. Could you or someone else point to > some concrete usages and use cases of the current RestrictedPython? - Download and install Zope 2.11.1. - Add a "Script (Python)" using the Zope management interface. - Write some python code or look at the default python code that starts off in the python script. The use case is basically threefold: - general distrust of code authored through a web browser (what if someone managed to break in somehow and insert source code for a PythonSript (the things Zope calls "Script (Python)" in its management interface), etc) - provide an environment where objects that shouldn't be accessible by a particular user are filtered by the environment and where objects can also prevent their attributes from being written to or read. (the user here doesn't write code, they are running other code... as a programmer, rather than having to write security checks all the time yourself, the environment does it for you) - allow customers to write scripts where they do so in an environment where they can't access things that they shouldn't be able to, be it data, or functions like os.system, etc. There's also a "nice to have" rather than a requirement of providing an environment where code can be safely run without it accidentally consuming all memory or processing resources on a machine. >> - prevents people importing things they shouldn't >> - prevents people from getting hold of un-security-proxies-objects >> - executes as fast as possible >> - each function plays in its own sandbox and can be called many many >> times, often simultaneously in different threads >> - limits the amount of memory and processor they can hog (nice to have) > > the rest, i think, is quite doable, particularly > limiting RAM/CPU resources. cool :-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From jacobidiego at gmail.com Sat Jul 19 05:00:06 2008 From: jacobidiego at gmail.com (Diego Jacobi) Date: Sat, 19 Jul 2008 00:00:06 -0300 Subject: [pypy-dev] I have a problem interpreting pypy functionality. Message-ID: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> Hi. I am trying to decifer what exactly is the result that i got from using pypy. As little as i understand, it can translate code from python to C. I have followed the "Getting started" guide, but it makes it too long to see the pypy features that i am confused. First at all, i have had some issues from trying to compile pypy into pypy-c, it claims for some error with cc, but cc is on my system. And i am using debian lenny, which is still under development. Im not sure if that can cause the problem. So, i couldnt see the code generated or the optimiced program running. I have found pypy, by looking for a translator from python to C. But what i was looking for is a translator which gives me a code without a python dependency. Maybe using stdlibs, or glib, or something, and libffi, so the python wrappers are converted too. In some of the tests in the guide i see that after compiling with t.compile_c() i can directly use the function in python. So, it is a python-module compiler? Also the guide doesnt mention where is saved the resultant module. I just want to translate a simple algoritm which i am using to test the differences of speed between multiple languages: http://code.google.com/p/languagebenchmarks/ By now i only have the implementation with only Python (without external optimizations) and standard C, showing that python is only between 20 and 40 times slower than C and a promedium of 35, which is not much. I want to add the code generated by pypy to compare how good is optimiced in relation to C and Python, and in the future to other languages. The algorithm is very linear and doesnt make use of threads or anything too much complicated. It does extensive use of lists and strings. But when running the translator with the main.py file i have an error in an initial line: #python pypy-dist/pypy/translator/goal/translate.py main.py File "main.py", line 6, in N_entradas = int(sys.argv[1]) ValueError: invalid literal for int() with base 10: 'main.py' The program is called with 2 integers determinning the number of bits of input and the number of bits of output in an electronic true table which the algoritm should optimice with the QuineMcClusky algorithm. The true table is passed through stdin. By eample: ./main.py 3 2 00010 00101 01011 0111- 100-- 101-1 11000 11100 That is a random table of example. An electronic should understand that, but the fact is that the algoritm have to convert that in a set of lists of string and work with them iterativelly, until it gets the best possible optimization. What i understand of the error line, is that it cant convert 'main.py' to integer. So seems like argv is like ['python pypy-dist/pypy/.....','main.py'] Does the translator haves to run the code to translate it? The 'main.py' should be the first argument received by the program not the second one. How can i pass any argument in the right order? Thanks in advance. Diego. From nshepperd at gmail.com Mon Jul 21 08:41:47 2008 From: nshepperd at gmail.com (Neil Shepperd) Date: Mon, 21 Jul 2008 16:41:47 +1000 Subject: [pypy-dev] I have a problem interpreting pypy functionality. In-Reply-To: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> References: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> Message-ID: <1216622507.6622.17.camel@neil> Hi, On Sat, 2008-07-19 at 00:00 -0300, Diego Jacobi wrote: > What i understand of the error line, is that it cant convert 'main.py' > to integer. > So seems like argv is like ['python pypy-dist/pypy/.....','main.py'] > > Does the translator haves to run the code to translate it? > The 'main.py' should be the first argument received by the program not > the second one. > > How can i pass any argument in the right order? > > Thanks in advance. > Diego. The pypy translator is not a *module* translator. It takes an entry-point function (equivalent to main() in C) and converts that to C. Take a look at pypy/translator/goal/target-nopstandalone.py to see how it is defined. The important thing is that any code you want to translate into C must be placed in a function (and obviously used by the entry-point function). Code in the module body itself will be executed immediately by the translator (under Python interpreter). > ?In some of the tests in the guide i see that after compiling with > t.compile_c() i can directly use the function in python. The function is converted to C and compiled to an extension module (DLL). This module is saved in a temporary directory somewhere (/tmp). It's probably not a good idea to try to open that file itself, better to just use the (compiled) function returned by compile_c(). Hope I make sense. Neil. From jacobidiego at gmail.com Wed Jul 23 17:59:33 2008 From: jacobidiego at gmail.com (Diego Jacobi) Date: Wed, 23 Jul 2008 12:59:33 -0300 Subject: [pypy-dev] I have a problem interpreting pypy functionality. In-Reply-To: <20080723154921.4ef56425.simon@arrowtheory.com> References: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> <20080721232918.d1aea981.simon@arrowtheory.com> <5de034af0807212150k6b8e584ei8cdc220814ddff84@mail.gmail.com> <20080723001312.e40d994b.simon@arrowtheory.com> <5de034af0807220903g35268176h8c38fdc7917a7ede@mail.gmail.com> <20080723135210.f4105b9a.simon@arrowtheory.com> <5de034af0807222237m52100bbapb098e7954b1167c3@mail.gmail.com> <20080723154921.4ef56425.simon@arrowtheory.com> Message-ID: <5de034af0807230859y5a5ac975o2e95a3ea3dc934ec@mail.gmail.com> 2008/7/23 Simon Burton : > On Wed, 23 Jul 2008 02:37:20 -0300 > "Diego Jacobi" wrote: > >> >> Is something about a negative slice, i replaced all my negative slices >> but the error still saying the same. > > try adding assert(index>=0) statements. > >> >> Is it possible an error of pypy? or it is sure that it is my code. Why >> doesnt point me to the line being parsed? > > It is a "global" error. Very common when compiling rpython. > > Simon. > Thanks that solved the problem, i will have to write lots of asserts. I think that this tells me that i can not continue with this: File "/home/data/diego/Desktop/Comparativa/PyPy/pypy-dist/pypy/annotation/annrpython.py", line 317, in ondegenerated raise AnnotatorError(msgstr) pypy.tool.error.AnnotatorError: annotation of v0 degenerated to SomeObject() v0 = getattr(terms_0, ('sort')) Happened at file /home/data/diego/Desktop/Comparativa/PyPy/QuineMcClusky.py line 51 ==> terms.sort( __sort_condition ) marks = [ False for x in range(len(ones)) ] .. SomeObject() origin: block at 240 op=0 Previous annotation: (none) There was previusly a lambda function on sort, i changed that for a normal function but the error is the same. If i am not wrong that means that sorting is not supported. Cheers. PD: i recently noted that i am sending thoose mails directly to you, sorry. From chris at simplistix.co.uk Fri Jul 25 18:15:41 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 25 Jul 2008 17:15:41 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> Message-ID: <4889FC2D.3000609@simplistix.co.uk> Hi All, Martijn Faassen wrote: > I think it's unlikely path to try to make pypy-sandbox, a special > version of PyPy, work in a Zope 2 context right now. I'm inclined to agree, but I'd avoid writing it off until the PyPy guys have agreed. I would be surprised if they can pull something out of their box of tricks here, we may just need to find the right way to phrase the problem ;-) At the very least, we should be trying to find the "right" way for realms beyond Zope 2, which it sounds like both myself and Stefan are encountering... > I think there are two possible paths: > > * try to patch up RestrictedPython by closing any new Python 2.5 > related holes. This would require figuring out what these are. This has been "the plan", it has, unfortunately, met prettymuch universal derision from the PyPy guys and silence from Jim :-( > * reimplement RestrictedPython using zope 3 security proxies This is only half the problem, though... > Zope 3 security proxies right now do the following: > > 1) control any attribute access/setting by asking a security policy > whether it's allowed > > 2) if you call a method on a proxied object, the result will be proxied as well > > 3) if you pass something into a method on a proxied object, the thing > passed as an argument will be proxied as well. > > I think to implement something like Restricted Python we would need 1 > (with a Zope 2 specific custom security policy) and 2, but not 3. 3 > causes proxies to start spreading through Zope 2 code Why would they cause havoc? They'd only result from stuff in a python script calling back into Zope, surely? It's a shame we can't form a better "blood/brain barrier" that proxies on the way in and de-proxies on the way out... > The hope of this strategy is that it's easier to maintain in the face > of new Python versions than the current RestrictedPython approach. Right, but there's still the language problems, of which importing is the most serious, but I'm sure there are more lurking there... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From holger at merlinux.eu Fri Jul 25 20:46:09 2008 From: holger at merlinux.eu (holger krekel) Date: Fri, 25 Jul 2008 20:46:09 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <4889FC2D.3000609@simplistix.co.uk> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> Message-ID: <20080725184608.GK22324@trillke.net> Hi Chris, Martijn, all, On Fri, Jul 25, 2008 at 17:15 +0100, Chris Withers wrote: > Hi All, > > Martijn Faassen wrote: > > I think it's unlikely path to try to make pypy-sandbox, a special > > version of PyPy, work in a Zope 2 context right now. > > I'm inclined to agree, but I'd avoid writing it off until the PyPy guys > have agreed. I would be surprised if they can pull something out of > their box of tricks here, we may just need to find the right way to > phrase the problem ;-) > > At the very least, we should be trying to find the "right" way for > realms beyond Zope 2, which it sounds like both myself and Stefan are > encountering... Here are some of my thoughts on the topic - Armin, who did most of the current sandbox impl is on vacation like are others btw. There is not too much of explicit "pypy group-think" on the topic yet :) Basically I see packaging and conceptual concerns. For the first, i think we can assume that we can have a easy_install pypy-python-sandbox which would install a nice sandboxed interpreter executable that you could drive from an application via some nice API. You could control what the sandbox sees as files, and its cpu and ram usage. This basically is doable today. What the sandboxed code currently could't easily do is interact with app objects from the driving process. However, I think we could provide means to represent app objects within the sandboxed program and let the parent/app process see and decide about operation on them. (see e.g. transparent proxies here http://codespeak.net/pypy/dist/pypy/doc/objspace-proxies.html#transparent-proxies for some existing tech). This might resemble what Martijn proposed by reimplementing restricted python on top of security proxies. However, you'd probably need to care a lot less regarding controling code paths that lead to IO like crafting a __builtins__ that doesn't have open etc. You could also mindlessly allow the sandboxed code to import python modules, including ones that your app provides for manipulations because you can be sure that no IO is going to escape. And for this you wouldn't need to mess with __import__ but could allow file access to .py files from the standard lib directory plus ones from app directories. I think the remaining crucial point is how to conveniently define and implement a python object protocol for the proxy interaction layer. This is a two-sided question: what is convenient for the driving app and what is safely doable for our generated pypy-sandbox interpreter. best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From whycode at gmail.com Mon Jul 28 02:11:15 2008 From: whycode at gmail.com (rahul garg) Date: Sun, 27 Jul 2008 18:11:15 -0600 Subject: [pypy-dev] numpy and pypy : info required Message-ID: Hi. I am writing a compiler called unPython (http://code.google.com/p/unpython) a very young and immature compiler for compiling Python to C. I am presenting the compiler at SciPy conference and need to write a paper about it. Under related work, I want to describe PyPy and also the status of NumPy support in PyPy. Can someone comment on whats the status of NumPy support in PyPy? Btw, unPython is a compiler I am writing for my Masters thesis. It compiles Python to C code with particular focus on NumPy. In fact currently, not much of Python is supported. One particular focus of unPython is parallel computing .. in particular OpenMP like parallel-for loops. I will release a new (much better) version over the next 1-2 weeks and will let people here know as well if its not considered spam :) So if you want to try out the compiler, I would request you to wait 1-2 weeks. thanks, rahul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080727/d104c218/attachment.htm From chris at simplistix.co.uk Tue Jul 29 00:22:11 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 28 Jul 2008 23:22:11 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> Message-ID: <488E4693.3060403@simplistix.co.uk> Martijn Faassen wrote: >>> * try to patch up RestrictedPython by closing any new Python 2.5 >>> related holes. This would require figuring out what these are. >> This has been "the plan", it has, unfortunately, met prettymuch universal >> derision from the PyPy guys and silence from Jim :-( > > Well, the PyPy people aren't familiar with this code base. It's been > proven enough technology for the last 8 years or so, so I suspect this > plan is at least feasible, if hardly perfect. Well, I dunno. Part of me is trying to get to the bottom of why the PyPy guys hate RestrictedPython so much. It seems a robust way of providing a secured (as opposed to sandboxed) pthon environment and while it requires work when the AST changes, I've yet to see someone suggest a suitable alternative, as this discussion has kindof showed ;-) >> It's a shame we can't form a better "blood/brain barrier" that proxies on >> the way in and de-proxies on the way out... > > Uh, Chris, that was exactly what I was talking about. You could do so, > but you'd need to alter the way proxies work. I'm not sure where you could generically plug in the un-proxying code... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Jul 29 00:24:12 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 28 Jul 2008 23:24:12 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <20080725184608.GK22324@trillke.net> <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> Message-ID: <488E470C.7000903@simplistix.co.uk> Martijn Faassen wrote: > Just for context, the Python 2.5 porting project is happening this > summer as part of the google summer of code. We're talking about a > timeline of weeks here. We're also talking about an important feature > used by *many* for the last 7 or 8 years or so in a stable release of > Zope 2. There's lots of code floating around that uses this feature. > The new code would need to run on the platforms that Zope 2 runs on. Agreed. > perspective. But from the current Zope 2 perspective, this looks like > a rather long-term development path with a lot of bugs to work out. Agreed. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From holger at merlinux.eu Tue Jul 29 10:29:11 2008 From: holger at merlinux.eu (holger krekel) Date: Tue, 29 Jul 2008 10:29:11 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <20080725184608.GK22324@trillke.net> <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> Message-ID: <20080729082911.GB9619@trillke.net> Hi Martijn, Chris, On Mon, Jul 28, 2008 at 11:09 +0200, Martijn Faassen wrote: > On Fri, Jul 25, 2008 at 8:46 PM, holger krekel wrote: > [description of how this might work] > > Cool, thanks for the description. you are welcome! > Just for context, the Python 2.5 porting project is happening this > summer as part of the google summer of code. We're talking about a > timeline of weeks here. We're also talking about an important feature > used by *many* for the last 7 or 8 years or so in a stable release of > Zope 2. There's lots of code floating around that uses this feature. > The new code would need to run on the platforms that Zope 2 runs on. > > If I were to write a new version of sandboxed Python in a new > framework, this sounds potentially useful. I know Chris is interested > in such technology aside from the interest from the Zope 2 > perspective. But from the current Zope 2 perspective, this looks like > a rather long-term development path with a lot of bugs to work out. Indeed, what i described is not likely to come to happen in the next weeks. > The pay-off would be a more secure version of Python Script in Zope 2 > that is hopefully more easy to port to new Python versions. That would > be nice, but the risks involved in this project are also rather huge, > and unfortunately I don't think the pay-off is huge enough to warrant > a lot of investment in time by (rare) Zope 2 hackers. I'd be happy if > one of them proved me wrong, but it doesn't seem to be a good idea to > let the new Python 2.5 compatible version of Zope 2 be dependent on > unfinished PyPy technology.... Makes sense to me as it stands. I am sure that when the time comes we'll find good use uses for a robust virtualized embeddable Python Interpreter. Having people or parties already asking for it (or better yet, investing into it :-) would certainly accelerate it. cheers, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From paskma at gmail.com Tue Jul 29 12:55:33 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Tue, 29 Jul 2008 12:55:33 +0200 Subject: [pypy-dev] Unable to translate locks Message-ID: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> Hi, PyPy hackers, I am playing with Python translation to C, especially with threads. I am using start_new_thread function provided by pypy.module.thread.ll_thread. My example (see mytasks.py in attachment) works fine. But when I tried to use locks (provided by the same module), I got an exception during the translation (see mytasks_err.py for the source code and translation_trace.txt). End of the trace: [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/c/database.py", line 156, in getcontainernode [translation:ERROR] node = nodefactory(self, T, container, **buildkwds) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/c/node.py", line 860, in opaquenode_factory [translation:ERROR] raise Exception("don't know about %r" % (T,)) [translation:ERROR] Exception: don't know about I have no idea whats the problem, translation of pypy interpreter with thread module enabled works perfectly. Maybe it is caused by the fact that pypy interpreter uses some special annotation policy, is it possible? Please, I would be thankful for any suggestions/comments what to do to make the locks work. Marek Pa?ka -------------- next part -------------- A non-text attachment was scrubbed... Name: mytasks.py Type: text/x-python Size: 338 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20080729/36582f73/attachment.py -------------- next part -------------- A non-text attachment was scrubbed... Name: mytasks_err.py Type: text/x-python Size: 403 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20080729/36582f73/attachment-0001.py -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: translation_trace.txt Url: http://codespeak.net/pipermail/pypy-dev/attachments/20080729/36582f73/attachment.txt From d.mcneil at qmul.ac.uk Wed Jul 30 00:51:59 2008 From: d.mcneil at qmul.ac.uk (Douglas McNeil) Date: Tue, 29 Jul 2008 23:51:59 +0100 (BST) Subject: [pypy-dev] math calls and errno In-Reply-To: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> References: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> Message-ID: A while back I noticed that some math-related rpython-produced C was much slower than it should have been. After I figured out what was going on, I set it aside, but I see someone mentioned doing some numpy stuff on IRC today so I dug up the tests. Note that the following slowdown only applies to translated code, so ctypes interfaces to numpy itself are immune, and the slowdown doesn't have much effect on pypy-c, so the audience for this is probably limited. Summary: in typical cases the way pypy treats errno causes math calls in rpython to take over a third longer than they should. This can be repaired by (carefully) inlining the errno functions. Details follow. -- A simple rpython target that does nothing but loop and add the results of sin() is much slower than the same code after being translated by shedskin, which gave the same time as the equivalent cython code, which gave the same time as the equivalent handwritten C. def f(): z = 0.0 for i in xrange(100000): for j in xrange(500): z += sin(float(i+j)) return z gcc 4.2.2 (-O3 -fomit-frame-pointer): rpython: 4.160 s shedskin 0.0.28: 3.043 s cython: 3.041 s C: 3.034 s The rpython/C gap persisted with the Intel compiler: icc 10.1 (") rpython: 3.296 s C 2.118 s so clearly the rpython code was doing something that the others weren't which the compilers particularly disliked and the obvious suspect was the error handling. But that should be on the order of a few percent, not an extra third or half. Turns out the errno calls aren't being inlined -- which makes sense, they're external to the implement_*.c files -- and if you force it they're often optimized away. It's quite fragile, thanks to volatility. But replacing the start of pypy_g_ll_math_ll_math_sin with something like volatile int *errno_loc = &errno; block0: l_v591 = (long)(0L); *(errno_loc) = l_v591; l_v590 = sin(l_x_1); l_v589 = *(errno_loc); OP_INT_IS_TRUE(l_v589, l_v593); if (l_v593) { goto block2; } l_v597 = l_v590; goto block1; you get pypy_g_ll_math_ll_math_sin: .L370: pushl %ebx # subl $8, %esp #, call __errno_location # fldl 16(%esp) # l_x_1 movl %eax, %ebx #, D.11955 movl $0, (%eax) #,* D.11955 fstpl (%esp) # call sin # movl (%ebx), %eax #* D.11955, l_v589 testl %eax, %eax # l_v589 je .L372 #, which is at least much improved over the original, and unlike my first few attempts I think this one correctly survives being optimized under both gcc and icc. (Of course what's actually executed in the tests is the version of this which gets inlined into entry_point, but ll_math_sin gets inlined into entry_point in all versions, so that's not causing the difference.) And we're still calling __errno_location like we should. Anyway, now we have: gcc 4.2.2 std rpython 4.160 s gcc 4.2.2 pure C 3.034 s gcc 4.2.2 errno-inlined rpython 3.068 s icc 10.1 std rpython 3.296 s icc 10.1 pure C 2.118 s icc 10.1 errno-inlined rpython 2.244 s and that's much better, especially with gcc. Standard disclaimers apply: this worked on my linux x86 system, with the above compiler versions, under an almost new moon.. and gcc is known for dramatic variations between versions. If it works for anyone else I'd be pleasantly surprised. That said, icc agrees. Unfortunately, as you might expect, this has very modest effects on pypy-c (i.e. barely detectable over the noise, due to the overhead), so I don't know if there will be interest in modifying the C backend to change it. It does improve things considerably for rpythonic math module writers, though. FWIW. Doug, pypy-math-sig member-in-waiting -- Queen Mary College, University of London "Still creating worlds.. Mathematical Sciences, Astronomy Unit .. but now with an accent!" From arigo at tunes.org Wed Jul 30 09:32:52 2008 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 Jul 2008 09:32:52 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <488E4693.3060403@simplistix.co.uk> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> <488E4693.3060403@simplistix.co.uk> Message-ID: <20080730073252.GA662@code0.codespeak.net> Hi Chris, On Mon, Jul 28, 2008 at 11:22:11PM +0100, Chris Withers wrote: > Well, I dunno. Part of me is trying to get to the bottom of why the PyPy > guys hate RestrictedPython so much. Sorry to have given this impression so much. I can only speak for myself: "hating" is a bit too strong; genereally speaking I have little interest for the "syntactic" side of programming language or language feature implementations (like AST restrictions or manipulations), and I am more interested in solutions that work at the "semantic" level, i.e. at the level of the behavior of objects (like Zope's security proxies, PyPy's various object spaces, and (at a lower level) PyPy's sandboxing). If I have real critics about RestrictedPython it's that it's not Python at all - it's a seriously limited sublanguage. Now I'm sure there are use cases for such a thing, and unlike Python's dead rexec, I can imagine that it can be made safe with enough care. Personally I have not much interested in that (but nothing against people that do). Sorry for the over-reaction. A bientot, Armin. From arigo at tunes.org Wed Jul 30 09:34:54 2008 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 Jul 2008 09:34:54 +0200 Subject: [pypy-dev] Unable to translate locks In-Reply-To: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> References: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> Message-ID: <20080730073454.GB662@code0.codespeak.net> Hi, On Tue, Jul 29, 2008 at 12:55:33PM +0200, Marek Pa?ka wrote: > paskma at paskma:goal$ ./translate.py mytasks_err.py I think you need to specify the --thread option: ./translate.py --thread mytasks_err.py A bientot, Armin From chris at simplistix.co.uk Thu Jul 31 17:06:16 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 31 Jul 2008 16:06:16 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20080729082911.GB9619@trillke.net> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <20080725184608.GK22324@trillke.net> <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> <20080729082911.GB9619@trillke.net> Message-ID: <4891D4E8.9090802@simplistix.co.uk> holger krekel wrote: > I am sure that when the time comes we'll find good use uses > for a robust virtualized embeddable Python Interpreter. > Having people or parties already asking for it (or better yet, > investing into it :-) would certainly accelerate it. What sort of investment would be needed for something that worked like RestrictedPython but was implemented in a way that you guys would be happy to support and also had controllable memory and processor usage and that behaved like a normal python library from the end user's point of view? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Thu Jul 31 17:15:02 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 31 Jul 2008 16:15:02 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20080730073252.GA662@code0.codespeak.net> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> <488E4693.3060403@simplistix.co.uk> <20080730073252.GA662@code0.codespeak.net> Message-ID: <4891D6F6.5010607@simplistix.co.uk> Armin Rigo wrote: > feature implementations (like AST restrictions or manipulations), and I > am more interested in solutions that work at the "semantic" level, i.e. > at the level of the behavior of objects (like Zope's security proxies, > PyPy's various object spaces, and (at a lower level) PyPy's sandboxing). Well, this is good to hear :-) > If I have real critics about RestrictedPython it's that it's not Python > at all - it's a seriously limited sublanguage. I wonder how much of this is based on the problem I set in Vilnius? Because of the rough edges of RestrictedPython and the limited resources I had to get the challenge ready, I think the actual problem I set was a lot harder than I meant. Maciej was showing me that in the environment I specified, you couldn't even interate over a tuple of integers :-/ > Now I'm sure there are > use cases for such a thing, In it's normal environment, the use case for Restricted Python is huge: - you don't need to create classes as you get all the objects you could want to manipulate from the result of the Zope environment - it's *really nice* having an environment where you can write python but where all the security checks are done for you without having to do explicit checks: >>> x.a Unauthorized: You are not allowed to access 'a'. - it would be *really really nice* if such an environment prevented you chewing through excess memory and processing power, which RestrictedPython certainly doesn't offer... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From simon at arrowtheory.com Fri Aug 1 04:59:29 2008 From: simon at arrowtheory.com (Simon Burton) Date: Fri, 1 Aug 2008 12:59:29 +1000 Subject: [pypy-dev] numpy and pypy : info required In-Reply-To: References: Message-ID: <20080801125929.8937138a.simon@arrowtheory.com> On Sun, 27 Jul 2008 18:11:15 -0600 "rahul garg" wrote: > Hi. > > I am writing a compiler called unPython (http://code.google.com/p/unpython) > a very young and immature compiler for compiling Python to C. > I am presenting the compiler at SciPy conference and need to write a paper > about it. Under related work, I want to describe PyPy and also the status of > NumPy support in PyPy. > Can someone comment on whats the status of NumPy support in PyPy? > ... Do you still want this information ? I am largely responsible for the numpy inspired code in pypy. So far it has turned out to be more of an experiment to get to know the pypy codebase, than actual "numpy support". Simon. From arigo at tunes.org Sun Aug 3 15:13:49 2008 From: arigo at tunes.org (Armin Rigo) Date: Sun, 3 Aug 2008 15:13:49 +0200 Subject: [pypy-dev] new -O option Message-ID: <20080803131349.GA3155@code0.codespeak.net> Hi all, I'm going to merge the "opt-option" branch; I'll write a blog post entry when it's done. It removes the "--faassen" and "--allopts" options to translate.py and adds instead the "--opt=#" option, which can also be spelled "-O#", where "#" is one of 0, 1, size, mem, 2 or 3. Levels from 0 to 3 are trade-offs between the time+memory it takes to translate, and the efficiency of the produced executable. Levels "size" and "mem" try (or should try in the future) to minimize the executable size and the runtime memory usage, respectively. The level is a global option of translate.py but it also influences PyPy-specific options in targetpypystandalone.py; for example, a bit arbitrarily, geninterp is disabled with -O0. The specific config options enabled or disabled by each level can depend on e.g. the selected backend; see the two functions set_opt_level() and set_pypy_opt_level(). We have a number of open branches; does it make sense, in an effort to reduce confusion, to apply this change to some of them too? A bientot, Armin From lawfulfalafel at gmail.com Mon Aug 4 00:11:08 2008 From: lawfulfalafel at gmail.com (lawful falafel) Date: Sun, 3 Aug 2008 18:11:08 -0400 Subject: [pypy-dev] problem with svn-checkout pypy Message-ID: I checked out a copy of pypy today and upon trying to run "python pypy-dist/pypy/bin/py.py" I get this error: k at k-desktop:~$ python pypy-dist/pypy/bin/py.py [cbuild:execute] cc -O3 -fomit-frame-pointer -pthread -c gcctest.c -o gcctest.o Traceback (most recent call last): File "pypy-dist/pypy/bin/py.py", line 147, in sys.exit(main_(sys.argv)) File "pypy-dist/pypy/bin/py.py", line 64, in main_ space = option.make_objspace(config) File "/home/k/pypy-dist/pypy/tool/option.py", line 48, in make_objspace space = Space(config) File "/home/k/pypy-dist/pypy/interpreter/baseobjspace.py", line 237, in __init__ self.initialize() File "/home/k/pypy-dist/pypy/objspace/std/objspace.py", line 71, in initialize self.model = StdTypeModel(self.config) File "/home/k/pypy-dist/pypy/objspace/std/model.py", line 88, in __init__ import pypy.objspace.std.marshal_impl # install marshal multimethods File "/home/k/pypy-dist/pypy/objspace/std/marshal_impl.py", line 21, in from pypy.rlib.rstruct import ieee File "/home/k/pypy-dist/pypy/rlib/rstruct/__init__.py", line 8, in from pypy.rlib.rstruct.formatiterator import FormatIterator File "/home/k/pypy-dist/pypy/rlib/rstruct/formatiterator.py", line 2, in from pypy.rlib.rstruct.nativefmttable import native_is_bigendian File "/home/k/pypy-dist/pypy/rlib/rstruct/nativefmttable.py", line 8, in from pypy.rpython.tool import rffi_platform File "/home/k/pypy-dist/pypy/rpython/tool/rffi_platform.py", line 5, in from pypy.rpython.lltypesystem import rffi File "/home/k/pypy-dist/pypy/rpython/lltypesystem/rffi.py", line 338, in NUMBER_TYPES = setup() File "/home/k/pypy-dist/pypy/rpython/lltypesystem/rffi.py", line 330, in setup tp = platform.inttype(name.upper(), c_name, signed) File "/home/k/pypy-dist/pypy/rpython/tool/rfficache.py", line 49, in inttype bits = sizeof_c_type(c_name, **kwds) * 8 File "/home/k/pypy-dist/pypy/rpython/tool/rfficache.py", line 38, in sizeof_c_type return int(ask_gcc(question, **kwds)) File "/home/k/pypy-dist/pypy/rpython/tool/rfficache.py", line 34, in ask_gcc return build_executable_cache([c_file], eci) File "/home/k/pypy-dist/pypy/tool/gcc_cache.py", line 23, in build_executable_cache result = py.process.cmdexec(build_executable(c_files, eci)) File "/home/k/pypy-dist/pypy/translator/tool/cbuild.py", line 621, in build_executable compiler.build(noerr=noerr) File "/home/k/pypy-dist/pypy/translator/tool/cbuild.py", line 574, in build raise CompilationError(data) pypy.translator.tool.cbuild.CompilationError: gcctest.c:3:20: error: stdlib.h: No such file or directory gcctest.c:4:23: error: sys/types.h: No such file or directory gcctest.c: In function 'main': gcctest.c:11: warning: incompatible implicit declaration of built-in function 'printf' gcctest.c:13:2: warning: no newline at end of file command 'cc' failed with exit status 1 My gcc -v is this: gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7) And my python version is this: Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Can anyone suggest how I might fix this? Thanks! p.s. I know this is a pretty noob question, and I am not sure if a development mailing list is the right place to post it, but right now it seems like the only potential place I can find. Am I wrong in this assumption, and is there another list dedicated to people trying to get into pypy? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080803/274357a1/attachment.htm From exarkun at divmod.com Mon Aug 4 02:21:43 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Sun, 3 Aug 2008 20:21:43 -0400 Subject: [pypy-dev] problem with svn-checkout pypy In-Reply-To: Message-ID: <20080804002143.29191.961003596.divmod.quotient.10275@ohm> On Sun, 3 Aug 2008 18:11:08 -0400, lawful falafel wrote: >I checked out a copy of pypy today and upon trying to run "python >pypy-dist/pypy/bin/py.py" I get this error: > >k at k-desktop:~$ python pypy-dist/pypy/bin/py.py >[cbuild:execute] cc -O3 -fomit-frame-pointer -pthread -c gcctest.c -o >gcctest.o >Traceback (most recent call last): > [snip] > raise CompilationError(data) >pypy.translator.tool.cbuild.CompilationError: gcctest.c:3:20: error: >stdlib.h: No such file or directory >gcctest.c:4:23: error: sys/types.h: No such file or directory >gcctest.c: In function 'main': >gcctest.c:11: warning: incompatible implicit declaration of built-in >function 'printf' >gcctest.c:13:2: warning: no newline at end of file >command 'cc' failed with exit status 1 Notice that the reason the cc command failed is that "sys/types.h" could not be found. This probably means you're on a system that lacks a full "development environment". As it seems you're on Ubuntu, a convenient way to get a lot of the requirements for PyPy is probably this command, run as root: apt-get build-dep python2.5 This is because a lot of the build dependencies of PyPy and CPython overlap. This won't necessarily give you everything you need to do everything with PyPy (for example, it won't install libgc or libgc-dev, which is needed in order for a boehm-based transation to work) but it's probably a good start. > [snip] > >p.s. I know this is a pretty noob question, and I am not sure if a >development mailing list is the right place to post it, but right now it >seems like the only potential place I can find. Am I wrong in this >assumption, and is there another list dedicated to people trying to get into >pypy? > I think this is a perfectly fine question for this list. :) Jean-Paul From holger at merlinux.eu Mon Aug 4 09:03:01 2008 From: holger at merlinux.eu (holger krekel) Date: Mon, 4 Aug 2008 09:03:01 +0200 Subject: [pypy-dev] new -O option In-Reply-To: <20080803131349.GA3155@code0.codespeak.net> References: <20080803131349.GA3155@code0.codespeak.net> Message-ID: <20080804070301.GK9619@trillke.net> Hi Armin, On Sun, Aug 03, 2008 at 15:13 +0200, Armin Rigo wrote: > Hi all, > > I'm going to merge the "opt-option" branch; I'll write a blog post entry > when it's done. It removes the "--faassen" and "--allopts" options to > translate.py and adds instead the "--opt=#" option, which can also be > spelled "-O#", where "#" is one of 0, 1, size, mem, 2 or 3. Levels from > 0 to 3 are trade-offs between the time+memory it takes to translate, and > the efficiency of the produced executable. Levels "size" and "mem" try > (or should try in the future) to minimize the executable size and the > runtime memory usage, respectively. The level is a global option of > translate.py but it also influences PyPy-specific options in > targetpypystandalone.py; for example, a bit arbitrarily, geninterp is > disabled with -O0. The specific config options enabled or disabled by > each level can depend on e.g. the selected backend; see the two > functions set_opt_level() and set_pypy_opt_level(). looks good. i particularly like that a single "-O#" sets things nicely both for translation and the target. > We have a number of open branches; does it make sense, in an effort to > reduce confusion, to apply this change to some of them too? Maybe, yes. Reminds me: I'd really like to see better tool support for working with branches. eazysvn from Marius Gedminas might help a bit (haven't tried) but i am rather thinking of something that helps me efficiently managing e.g. my "codespeak.net/svn/pypy" checkout, including quick overviews of active branches, easy merging (without knowing numbers) etc. Some might think "dvcs!" here but it's a while until we go there, i think, and i even consider somewhat orthogonal to having support for my particular work flow. cheers, holger From exarkun at divmod.com Mon Aug 4 13:29:33 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Mon, 4 Aug 2008 07:29:33 -0400 Subject: [pypy-dev] new -O option In-Reply-To: <20080804070301.GK9619@trillke.net> Message-ID: <20080804112933.29191.2084489418.divmod.quotient.10437@ohm> On Mon, 4 Aug 2008 09:03:01 +0200, holger krekel wrote: >Hi Armin, > >On Sun, Aug 03, 2008 at 15:13 +0200, Armin Rigo wrote: >> Hi all, >> >> I'm going to merge the "opt-option" branch; I'll write a blog post entry >> when it's done. It removes the "--faassen" and "--allopts" options to >> translate.py and adds instead the "--opt=#" option, which can also be >> spelled "-O#", where "#" is one of 0, 1, size, mem, 2 or 3. Levels from >> 0 to 3 are trade-offs between the time+memory it takes to translate, and >> the efficiency of the produced executable. Levels "size" and "mem" try >> (or should try in the future) to minimize the executable size and the >> runtime memory usage, respectively. The level is a global option of >> translate.py but it also influences PyPy-specific options in >> targetpypystandalone.py; for example, a bit arbitrarily, geninterp is >> disabled with -O0. The specific config options enabled or disabled by >> each level can depend on e.g. the selected backend; see the two >> functions set_opt_level() and set_pypy_opt_level(). > >looks good. i particularly like that a single "-O#" sets >things nicely both for translation and the target. > >> We have a number of open branches; does it make sense, in an effort to >> reduce confusion, to apply this change to some of them too? > >Maybe, yes. Reminds me: I'd really like to see better tool >support for working with branches. eazysvn from Marius Gedminas >might help a bit (haven't tried) but i am rather thinking of >something that helps me efficiently managing >e.g. my "codespeak.net/svn/pypy" checkout, including quick >overviews of active branches, easy merging (without knowing >numbers) etc. Some might think "dvcs!" here but it's a while >until we go there, i think, and i even consider somewhat orthogonal >to having support for my particular work flow. > Combinator does some of these things. http://divmod.org/trac/wiki/DivmodCombinator Jean-Paul From arigo at tunes.org Mon Aug 4 16:17:36 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 Aug 2008 16:17:36 +0200 Subject: [pypy-dev] new -O option In-Reply-To: <20080803131349.GA3155@code0.codespeak.net> References: <20080803131349.GA3155@code0.codespeak.net> Message-ID: <20080804141736.GA7151@code0.codespeak.net> reHi, On Sun, Aug 03, 2008 at 03:13:49PM +0200, Armin Rigo wrote: > I'm going to merge the "opt-option" branch; I'll write a blog post entry > when it's done. While I'm at it, should I remove --allworkingmodules and make it the default? Should this also include the thread module? If additionally the default value for --opt would be 3, then just typing ./translate.py would produce a pypy-c executable of the "most recommended and most useful" kind. A bientot, Armin. From anto.cuni at gmail.com Mon Aug 4 16:23:29 2008 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 04 Aug 2008 16:23:29 +0200 Subject: [pypy-dev] new -O option In-Reply-To: <20080804141736.GA7151@code0.codespeak.net> References: <20080803131349.GA3155@code0.codespeak.net> <20080804141736.GA7151@code0.codespeak.net> Message-ID: <489710E1.7020802@gmail.com> Armin Rigo wrote: > While I'm at it, should I remove --allworkingmodules and make it the > default? Should this also include the thread module? If additionally > the default value for --opt would be 3, then just typing ./translate.py > would produce a pypy-c executable of the "most recommended and most > useful" kind. +1 The only drawback I see is that with --allworkingmodules the translations takes longer to complete, but I agree that the casual user probably cares more about functionality than translation speed. Does --allworkingmodules also increase the ram usage? ciao, anto From santagada at gmail.com Mon Aug 4 16:49:59 2008 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 4 Aug 2008 11:49:59 -0300 Subject: [pypy-dev] new -O option In-Reply-To: <489710E1.7020802@gmail.com> References: <20080803131349.GA3155@code0.codespeak.net> <20080804141736.GA7151@code0.codespeak.net> <489710E1.7020802@gmail.com> Message-ID: <0E2AF1A2-934D-423E-BBEE-3294647CC100@gmail.com> On 04/08/2008, at 11:23, Antonio Cuni wrote: > Armin Rigo wrote: > > >> While I'm at it, should I remove --allworkingmodules and make it the >> default? Should this also include the thread module? If >> additionally >> the default value for --opt would be 3, then just typing ./ >> translate.py >> would produce a pypy-c executable of the "most recommended and most >> useful" kind. > > +1 > > The only drawback I see is that with --allworkingmodules the > translations takes longer to complete, but I agree that the casual > user > probably cares more about functionality than translation speed. > > Does --allworkingmodules also increase the ram usage? just do a reverse switch, --withoutextramodules or something. -- Leonardo Santagada From cfbolz at gmx.de Wed Aug 6 20:57:36 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 06 Aug 2008 20:57:36 +0200 Subject: [pypy-dev] Stdlib porting In-Reply-To: References: Message-ID: <4899F420.90205@gmx.de> Hi Bruno (ccing pypy-dev)! Bruno Gola wrote: > After having some problems I finished PEP 328 and I'm starting to work > on stdlib now. > > However, there is one test that's still failing and I can't fix it :(. > It's in interpreter/test/test_nestedscope.py (lambda_in_genexpr), any > hints? No clue yet, will take a look. > About the stdlib, do you have any suggestion about how/where to begin? > > I'm looking at http://docs.python.org/whatsnew/modules.html but it > seems that most of the changes will not affect my work (as most of the > modules are written in pure python). > > My plans initially are: > - copy the stdlib from CPython 2.5.2 > - apply the changes from modified-2.4.1 Yes, exactly. You should add a new directory lib-python/2.5.2 and import CPython's stdlib there. Then you need to carefully look at the changes in modified-2.4.1 and check how they need to be applied to the relevant 2.5.2 module. > - rewrite any new module written in C* This one has a much lower priority than getting the pure-Python modules to a sensible state. > * well, taking a better look, it seems that the new modules are > wsgiref, xml.etree package, hashlib (all in python), ctypes and > sqlite3 (in C, but it seems too that there are some work already done > on those two modules, am I right?). sqlite3 and ctypes should be done already. Some work on hashlib was started, afair. It is in the lib dir, I think? > Of course tests are in my plan as well. They'd better be :-). I guess part of the porting work should really be to check that the stdlib tests keep passing, which is quite important because they also check the other 2.5 features that you implemented. > Sorry for not being online on IRC today while working, I'm behind a > proxy and it isn't a friendly proxy :-) You know that you can use other ports to connect to freenode? I think even 8000 works, which most procies let through. Cheers, Carl Friedrich From santagada at gmail.com Wed Aug 6 22:43:54 2008 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 6 Aug 2008 17:43:54 -0300 Subject: [pypy-dev] Stdlib porting In-Reply-To: <4899F420.90205@gmx.de> References: <4899F420.90205@gmx.de> Message-ID: <7BB313A8-F49F-42C3-BFB6-36BCA69F5F44@gmail.com> On 06/08/2008, at 15:57, Carl Friedrich Bolz wrote: >> >> Sorry for not being online on IRC today while working, I'm behind a >> proxy and it isn't a friendly proxy :-) > > You know that you can use other ports to connect to freenode? I think > even 8000 works, which most procies let through. There are many ports, but to really get trough stupid proxies I've been using mibbit[1][2] a web client for irc. [1] http://www.mibbit.com/ [2] https://www.mibbit.com/ -- Leonardo Santagada From rajb at rice.edu Thu Aug 14 20:36:23 2008 From: rajb at rice.edu (Raj Bandyopadhyay) Date: Thu, 14 Aug 2008 13:36:23 -0500 Subject: [pypy-dev] Building pypy-c on mac os x Message-ID: <48A47B27.8040202@rice.edu> Hi all I was wondering if pypy-c can be built on Mac OS X, and if there are any special options to use? I tried building it on a Linux/x86 machine and it worked fine (took a while to build), but on a mac, it seems to bring the machine down completely (the machine becomes non-responsive during the process). This is a mac running Leopard with Intel core-duo processors. Is this a known issue? Thanks Raj From fijall at gmail.com Thu Aug 14 21:14:29 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 14 Aug 2008 21:14:29 +0200 Subject: [pypy-dev] Building pypy-c on mac os x In-Reply-To: <48A47B27.8040202@rice.edu> References: <48A47B27.8040202@rice.edu> Message-ID: <693bc9ab0808141214n28c140a8r483d432947ff5817@mail.gmail.com> You need about 1.1-1.2G of RAM (double for 64 bit machine) and it should work fine on Mac OS X. (This is more or less the same for linux). Cheers, fijal On Thu, Aug 14, 2008 at 8:36 PM, Raj Bandyopadhyay wrote: > Hi all > > I was wondering if pypy-c can be built on Mac OS X, and if there are any > special options to use? I tried building it on a Linux/x86 machine and > it worked fine (took a while to build), but on a mac, it seems to bring > the machine down completely (the machine becomes non-responsive during > the process). > > This is a mac running Leopard with Intel core-duo processors. > > Is this a known issue? > > Thanks > Raj > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From paskma at gmail.com Fri Aug 15 00:35:45 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Fri, 15 Aug 2008 00:35:45 +0200 Subject: [pypy-dev] Unable to translate locks In-Reply-To: <4ddb710e0807300910t454c1b97keee3096b72d56832@mail.gmail.com> References: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> <20080730073454.GB662@code0.codespeak.net> <4ddb710e0807300910t454c1b97keee3096b72d56832@mail.gmail.com> Message-ID: <4ddb710e0808141535l2cdf143ar5b14fc0d68c5f9fa@mail.gmail.com> Hi, well, I've solved this problem. The point is that the lock in my example is created at import time, before translation. Works fine if the initialization is moved into a function that can be translated. Marek Pa?ka On Wed, Jul 30, 2008 at 18:10, Marek Pa?ka wrote: > Hi! > > I tried this option and it does not change anything. In fact, this > option is not even needed for pypy interpreter translation, with > --withmod-thread the pypy interpreter is compiled successfully and the > thread module is usable regardless of the --thread option. As I > understand, the --thread option only affects runtime behavior, e.g., > ensures that GC is aware of multiple threads or so. Or am I wrong? > > It seems, there is some magic used when the thread module is > translated. Something that affects annotation... I don't know. I spent > several days debugging it but PyPy is rather complex. > > Cheers, > > Marek Pa?ka > > On Wed, Jul 30, 2008 at 09:34, Armin Rigo wrote: >> Hi, >> >> On Tue, Jul 29, 2008 at 12:55:33PM +0200, Marek Pa?ka wrote: >>> paskma at paskma:goal$ ./translate.py mytasks_err.py >> >> I think you need to specify the --thread option: >> >> ./translate.py --thread mytasks_err.py >> >> >> A bientot, >> >> Armin >> > From me at andy.durdin.net Sat Aug 16 02:44:22 2008 From: me at andy.durdin.net (Andrew Durdin) Date: Fri, 15 Aug 2008 17:44:22 -0700 Subject: [pypy-dev] Building pypy-c on mac os x In-Reply-To: <693bc9ab0808141214n28c140a8r483d432947ff5817@mail.gmail.com> References: <48A47B27.8040202@rice.edu> <693bc9ab0808141214n28c140a8r483d432947ff5817@mail.gmail.com> Message-ID: On 14 Aug 2008, at 12:14, "Maciej Fijalkowski" wrote: > You need about 1.1-1.2G of RAM (double for 64 bit machine) and it > should work fine on Mac OS X >> >> >> >> >> On my iBook with 1.25G total RAM, it ran out of free RAM and started swapping heavily, so I killed it. On my iMac (core duo, Leopard, 2G RAM) it built successfully in a reasonable time; however I was not trying to do anything else at the time. It used about 1.1G of RAM and about the same of virtual memory IIRC. Andrew. From brunogola at gmail.com Tue Aug 19 05:49:23 2008 From: brunogola at gmail.com (Bruno Gola) Date: Tue, 19 Aug 2008 00:49:23 -0300 Subject: [pypy-dev] Translate pypy-c Message-ID: Hi! I'm working on branch 2.5-features and trying to translate pypy-c. Probably some of the changes I made are not RPython and the translator crashes. Some of theses changes I've already found and fixed, but now I can't find out how to fix this error (traceback http://paste.pocoo.org/show/82666/). The traceback points to checkFlag() in pyassem.py, but it's identical to trunk. I've written some code that uses this method (visitImport() and visitFrom() in pycodegen.py). I don't know much about the translator. Does anyone have any idea about what's wrong? Thanks, -- Bruno Fialho Marques Gola http://www.brunogola.com.br Cel: (11) 9294-5883 From brunogola at gmail.com Tue Aug 19 06:17:52 2008 From: brunogola at gmail.com (Bruno Gola) Date: Tue, 19 Aug 2008 01:17:52 -0300 Subject: [pypy-dev] Translate pypy-c In-Reply-To: References: Message-ID: Sorry, the right traceback: http://paste.pocoo.org/show/82667/ Anyway, mwhudson pointed that it's maybe a bug in checkFlag() (that wasn't called until now). It returns 1 or None. Trying to translate again. -- Bruno Fialho Marques Gola http://www.brunogola.com.br Cel: (11) 9294-5883 On Tue, Aug 19, 2008 at 12:49 AM, Bruno Gola wrote: > Hi! > > I'm working on branch 2.5-features and trying to translate pypy-c. > Probably some of the changes I made are not RPython and the translator > crashes. > > Some of theses changes I've already found and fixed, but now I can't > find out how to fix this error (traceback > http://paste.pocoo.org/show/82666/). > > The traceback points to checkFlag() in pyassem.py, but it's identical > to trunk. I've written some code that uses this method (visitImport() > and visitFrom() in pycodegen.py). > > I don't know much about the translator. Does anyone have any idea > about what's wrong? > > Thanks, > -- > Bruno Fialho Marques Gola > http://www.brunogola.com.br > Cel: (11) 9294-5883 > From cfbolz at gmx.de Tue Aug 19 11:40:25 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 19 Aug 2008 11:40:25 +0200 Subject: [pypy-dev] Translate pypy-c In-Reply-To: References: Message-ID: <48AA9509.90705@gmx.de> Bruno Gola wrote: > Sorry, the right traceback: http://paste.pocoo.org/show/82667/ > > Anyway, mwhudson pointed that it's maybe a bug in checkFlag() (that > wasn't called until now). It returns 1 or None. Trying to translate > again. checkFlag was rather silly, it should just return a boolean, but it returned 1 or None, which isn't possible in RPython. I changed it. There are more problems along the lines of the first problem you had, maybe I'll fix those if I have time. Cheers, Carl Friedrich From holger at merlinux.eu Fri Aug 22 13:08:14 2008 From: holger at merlinux.eu (holger krekel) Date: Fri, 22 Aug 2008 13:08:14 +0200 Subject: [pypy-dev] ANN: pylib/pytest 0.9.2 released Message-ID: <20080822110814.GQ9619@trillke.net> Welcome to the 0.9.2 py lib and py.test release - mainly fixing Windows issues, providing better packaging and integration with setuptools. Summry of main features: * py.test: cross-project testing tool with many advanced features * py.execnet: ad-hoc code distribution to SSH, Socket and local sub processes * py.magic.greenlet: micro-threads on standard CPython ("stackless-light") * py.path: path abstractions over local and subversion files * rich documentation of py's exported API * tested against Linux, Win32, OSX, works on python 2.3-2.6 good entry points: Pypi pages: http://pypi.python.org/pypi/py/ Download/Install: http://codespeak.net/py/0.9.2/download.html Documentation/API: http://codespeak.net/py/0.9.2/index.html the CHANGELOG excerpt for the 0.9.1 -> 0.9.2 transition: * refined installation and metadata, created new setup.py, now based on setuptools/ez_setup (thanks to Ralf Schmitt for his support). * improved the way of making py.* scripts available in windows environments, they are now added to the Scripts directory as ".cmd" files. * py.path.svnwc.status() now is more complete and uses xml output from the 'svn' command if available (Guido Wesdorp) * fix for py.path.svn* to work with svn 1.5 (Chris Lamb) * fix path.relto(otherpath) method on windows to use normcase for checking if a path is relative. * py.test's traceback is better parseable from editors (follows the filenames:LINENO: MSG convention) (thanks to Osmo Salomaa) * fix to javascript-generation, "py.test --runbrowser" should work more reliably now * removed previously accidentally added py.test.broken and py.test.notimplemented helpers. * there now is a py.__version__ attribute best and have fun, holger krekel -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From ondrej at certik.cz Sun Aug 24 16:08:29 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Sun, 24 Aug 2008 16:08:29 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy Message-ID: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> Hi, I am using pypy in Debian: ondra at fuji:~/sympy$ python2.4 Python 2.4.5 (#2, Jul 20 2008, 20:55:34) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sympy >>> ondra at fuji:~/sympy$ pypy Python 2.4.1 (pypy 1.0.0 build 55235) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``the 'super' keyword is not that huggable'' >>>> import sympy Traceback (most recent call last): File "", line 1, in File "/home/ondra/sympy/sympy/__init__.py", line 16, in from sympy.core import * File "/home/ondra/sympy/sympy/core/__init__.py", line 13, in from function import Lambda, WildFunction, Derivative, diff, FunctionClass, \ File "/home/ondra/sympy/sympy/core/function.py", line 361, in class WildFunction(Function, Atom): File "/home/ondra/sympy/sympy/core/function.py", line 68, in __new__ return type.__new__(cls, name, bases, attrdict) TypeError: instance layout conflicts in multiple inheritance >>>> Do you know what this error means and how should we fix sympy so that it works on top of pypy? Ondrej From fijall at gmail.com Mon Aug 25 10:23:55 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Aug 2008 10:23:55 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> Message-ID: <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> Works for me: fijal at zygfryd:~$ pypy-c Python 2.4.1 (pypy 1.0.0 build 57237) on linux2 Type "help", "copyright", "credits" or "license" for more information. Welcome to rlcompleter2 0.96 for nice experiences hit multiple times And now for something completely different: ``"peephope" optimizations are what an optimistic Compiler uses'' >>>> import sympy >>>> If you have old pypy build try pypy-c --oldstyle, otherwise you can retranslate pypy. On Sun, Aug 24, 2008 at 4:08 PM, Ondrej Certik wrote: > Hi, > > I am using pypy in Debian: > > ondra at fuji:~/sympy$ python2.4 > Python 2.4.5 (#2, Jul 20 2008, 20:55:34) > [GCC 4.3.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import sympy >>>> > ondra at fuji:~/sympy$ pypy > Python 2.4.1 (pypy 1.0.0 build 55235) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ``the 'super' keyword is not that huggable'' >>>>> import sympy > Traceback (most recent call last): > File "", line 1, in > File "/home/ondra/sympy/sympy/__init__.py", line 16, in > from sympy.core import * > File "/home/ondra/sympy/sympy/core/__init__.py", line 13, in > from function import Lambda, WildFunction, Derivative, diff, > FunctionClass, \ > File "/home/ondra/sympy/sympy/core/function.py", line 361, in > class WildFunction(Function, Atom): > File "/home/ondra/sympy/sympy/core/function.py", line 68, in __new__ > return type.__new__(cls, name, bases, attrdict) > TypeError: instance layout conflicts in multiple inheritance >>>>> > > > Do you know what this error means and how should we fix sympy so that > it works on top of pypy? > > Ondrej > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Mon Aug 25 10:24:30 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Aug 2008 10:24:30 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> Message-ID: <693bc9ab0808250124uc4d3765w767c962f65e8e79d@mail.gmail.com> Ah, sorry misread. Your pypy (from debian) seems to be a bit old. Please use svn version. On Sun, Aug 24, 2008 at 4:08 PM, Ondrej Certik wrote: > Hi, > > I am using pypy in Debian: > From ondrej at certik.cz Mon Aug 25 10:44:47 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 25 Aug 2008 10:44:47 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> Message-ID: <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> On Mon, Aug 25, 2008 at 10:23 AM, Maciej Fijalkowski wrote: > Works for me: > > fijal at zygfryd:~$ pypy-c > Python 2.4.1 (pypy 1.0.0 build 57237) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > Welcome to rlcompleter2 0.96 > for nice experiences hit multiple times > And now for something completely different: ``"peephope" optimizations are what > an optimistic Compiler uses'' >>>>> import sympy >>>>> Hm, doesn't seem to work for me: $ svn co http://codespeak.net/svn/pypy/dist pypy-dist $ cd sympy/ $ python ~/repos/pypy-dist/pypy/bin/py.py faking PyPy 1.0.0 in StdObjSpace on top of Python 2.5.2 (startuptime: 10.83 secs) >>>> import sympy Traceback (application-level): File "", line 1 in import sympy File "/home/ondra/sympy/sympy/__init__.py", line 16 in from sympy.core import * File "/home/ondra/sympy/sympy/core/__init__.py", line 13 in from function import Lambda, WildFunction, Derivative, diff, FunctionClass, \ File "/home/ondra/sympy/sympy/core/function.py", line 361 in class WildFunction(Function, Atom): File "/home/ondra/sympy/sympy/core/function.py", line 68 in __new__ return type.__new__(cls, name, bases, attrdict) TypeError: instance layout conflicts in multiple inheritance >>>> > > If you have old pypy build try pypy-c --oldstyle, otherwise you can > retranslate pypy. Old pypy + oldstyle doesn't seem to help: $ pypy --oldstyle Python 2.4.1 (pypy 1.0.0 build 55235) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``I'm sorry, could you please not agree with the cat as well?'' >>>> import sympy Traceback (most recent call last): File "", line 1, in File "/home/ondra/sympy/sympy/__init__.py", line 16, in from sympy.core import * File "/home/ondra/sympy/sympy/core/__init__.py", line 13, in from function import Lambda, WildFunction, Derivative, diff, FunctionClass, \ File "/home/ondra/sympy/sympy/core/function.py", line 361, in class WildFunction(Function, Atom): File "/home/ondra/sympy/sympy/core/function.py", line 68, in __new__ return type.__new__(cls, name, bases, attrdict) TypeError: instance layout conflicts in multiple inheritance >>>> So now I am trying to compile pypy-svn, but I doubt it will work if it doesn't work on top of python. Which sympy version are you using? Ondrej From fijall at gmail.com Mon Aug 25 10:51:18 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Aug 2008 10:51:18 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> Message-ID: <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> > > > So now I am trying to compile pypy-svn, but I doubt it will work if it > doesn't work on top of python. Which sympy version are you using? > > Ondrej > 0.4.2-1, I'll try newer one. From fijall at gmail.com Mon Aug 25 11:19:20 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Aug 2008 11:19:20 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> Message-ID: <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> Ok, I can reproduce it with newer version of sympy. Not sure what's wrong yet. On Mon, Aug 25, 2008 at 10:51 AM, Maciej Fijalkowski wrote: >> >> >> So now I am trying to compile pypy-svn, but I doubt it will work if it >> doesn't work on top of python. Which sympy version are you using? >> >> Ondrej >> > > 0.4.2-1, I'll try newer one. > From ondrej at certik.cz Mon Aug 25 11:45:26 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 25 Aug 2008 11:45:26 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> Message-ID: <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> On Mon, Aug 25, 2008 at 11:19 AM, Maciej Fijalkowski wrote: > Ok, I can reproduce it with newer version of sympy. Not sure what's wrong yet. Exactly. That's what I wanted to ask you what's wrong. :) I think we are using some stupid hacks, but if you could enlighten me how to fix sympy so that it works on pypy, it'd be awesome. Ondrej From fijall at gmail.com Mon Aug 25 12:08:19 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Aug 2008 12:08:19 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> Message-ID: <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> I'm not sure whether it's sympy's fault or pypy's fault. It might be that pypy is too strict about that. Arguments look like this: (Pdb++) bases[0].__mro__ (Function, , , ) (Pdb++) bases[1].__mro__ (, , , ) (Pdb++) cls (Pdb++) cls.__mro__ (, , , , ) (Pdb++) And it seems that bases[0] and bases[1] are incompatible (why?) On Mon, Aug 25, 2008 at 11:45 AM, Ondrej Certik wrote: > On Mon, Aug 25, 2008 at 11:19 AM, Maciej Fijalkowski wrote: >> Ok, I can reproduce it with newer version of sympy. Not sure what's wrong yet. > > Exactly. That's what I wanted to ask you what's wrong. :) > > I think we are using some stupid hacks, but if you could enlighten me > how to fix sympy so that it works on pypy, it'd be awesome. > > Ondrej > From ondrej at certik.cz Mon Aug 25 12:50:35 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 25 Aug 2008 12:50:35 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> Message-ID: <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> On Mon, Aug 25, 2008 at 12:08 PM, Maciej Fijalkowski wrote: > I'm not sure whether it's sympy's fault or pypy's fault. It might be > that pypy is too strict about that. > > Arguments look like this: > > (Pdb++) bases[0].__mro__ > (Function, , 'sympy.core.assumptions.AssumeMeths'>, ) > (Pdb++) bases[1].__mro__ > (, , > , ) > (Pdb++) cls > > (Pdb++) cls.__mro__ > (, 'sympy.core.basic.BasicMeta'>, , > , ) > (Pdb++) > > And it seems that bases[0] and bases[1] are incompatible (why?) Don't know. This patch fixes it: diff --git a/sympy/core/function.py b/sympy/core/function.py index 76a0f5b..a99c21d 100644 --- a/sympy/core/function.py +++ b/sympy/core/function.py @@ -362,7 +362,7 @@ class Function(Basic): x = sympify(x) return cls(x).diff(x, n).subs(x, 0) * x**n / C.Factorial(n) -class WildFunction(Function, Atom): +class WildFunction(Function): """ WildFunction() matches any expression but another WildFunction() XXX is this as intended, does it work ? But now there is this problem: >>>> import sympy Traceback (most recent call last): File "", line 1, in File "/home/ondra/repos/sympy/sympy/__init__.py", line 22, in from concrete import * File "/home/ondra/repos/sympy/sympy/concrete/__init__.py", line 2, in from products import product, Product File "/home/ondra/repos/sympy/sympy/concrete/products.py", line 5, in from sympy.simplify import powsimp File "/home/ondra/repos/sympy/sympy/simplify/__init__.py", line 10, in from rewrite import cancel, trim, apart File "/home/ondra/repos/sympy/sympy/simplify/rewrite.py", line 22, in def cancel(f, *symbols): TypeError: threaded() takes no non-keyword argument (1 given) Any idea what's wrong here? Here is the code: @threaded() def cancel(f, *symbols): """Cancel common factors in a given rational function. and: def threaded(**flags): Seems a-ok to me. Is pypy using some strict things? I don't know, but this patch fixes it: diff --git a/sympy/utilities/decorator.py b/sympy/utilities/decorator.py index 0043de1..f812b51 100644 --- a/sympy/utilities/decorator.py +++ b/sympy/utilities/decorator.py @@ -3,7 +3,7 @@ from sympy.core.add import Add from sympy.core.sympify import sympify from sympy.core.relational import Relational -def threaded(**flags): +def threaded(*args, **flags): Now it imports. Wow! >>>> import sympy >>>> from sympy import var >>>> var("x y z") (x, y, z) >>>> x**2 x**2 >>>> from sympy import sin >>>> e = sin(x) >>>> e.series(x, 0, 5) x - 1/6*x**3 + O(x**5) Guys, this is awesome! I did some timings: $ ~/repos/pypy-dist/pypy/translator/goal/pypy-c Python 2.4.1 (pypy 1.0.0 build 57617) on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``Python 300'' >>>> from sympy import * >>>> var("x") x >>>> from timeit import default_timer as clock >>>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 3.31447815895 Compare to python: $ python Python 2.5.2 (r252:60911, Aug 8 2008, 09:22:44) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from sympy import * >>> var("x") x >>> from timeit import default_timer as clock >>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 0.704444885254 So pypy is roughly 4.7x slower on this particular thing. Do you have any plans to release pypy? I think it's getting very useful. Ondrej From fijall at gmail.com Mon Aug 25 13:04:34 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Aug 2008 13:04:34 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> Message-ID: <693bc9ab0808250404n2019ecc7k7e2231cea3af6cad@mail.gmail.com> > > So pypy is roughly 4.7x slower on this particular thing. > > Do you have any plans to release pypy? I think it's getting very useful. > > Ondrej > Yes, we have some plans to have a release at some point soon. There are no details though. Note that if you do this: from sympy import * var('x') from timeit import default_timer as clock t = clock(); a= sin(x).series(x, 0, 100); t = clock()-t; print t var('y') t = clock(); a= sin(y).series(y, 0, 100); t = clock()-t; print gap between pypy and cpython is getting smaller (some startup time is particularly large). I can do a bit profiling if you like. Regarding threaded - I think you should not assume that your argument is passed as keyword arg when using decorator I think. The other one looks like bug in pypy, but I'm not 100% sure. Cheers, fijal From ondrej at certik.cz Mon Aug 25 13:28:49 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 25 Aug 2008 13:28:49 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <693bc9ab0808250404n2019ecc7k7e2231cea3af6cad@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> <693bc9ab0808250404n2019ecc7k7e2231cea3af6cad@mail.gmail.com> Message-ID: <85b5c3130808250428i78d05763wc9a2bda072e98a70@mail.gmail.com> On Mon, Aug 25, 2008 at 1:04 PM, Maciej Fijalkowski wrote: >> >> So pypy is roughly 4.7x slower on this particular thing. >> >> Do you have any plans to release pypy? I think it's getting very useful. >> >> Ondrej >> > > Yes, we have some plans to have a release at some point soon. There > are no details though. Note that if you do this: > > from sympy import * > var('x') > from timeit import default_timer as clock > t = clock(); a= sin(x).series(x, 0, 100); t = clock()-t; print t > var('y') > t = clock(); a= sin(y).series(y, 0, 100); t = clock()-t; print > > gap between pypy and cpython is getting smaller (some startup time is > particularly large). I can do a bit profiling if you like. Use SYMPY_USE_CACHE=no when doing repeated profiling, otherwise you are just measuring how fast our cache is. ondra at pc232:~/repos/sympy$ SYMPY_USE_CACHE=no ~/repos/pypy-dist/pypy/translator/goal/pypy-c Python 2.4.1 (pypy 1.0.0 build 57617) on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``- PyPy status blog: http://morepypy.blogspot.com/'' >>>> from sympy import * >>>> var('x') x >>>> from timeit import default_timer as clock >>>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 3.56681609154 >>>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 3.2662088871 >>>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 3.22522711754 >>>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 3.24221301079 >>>> ondra at pc232:~/repos/sympy$ SYMPY_USE_CACHE=no python Python 2.5.2 (r252:60911, Aug 8 2008, 09:22:44) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from sympy import * >>> var('x') x >>> from timeit import default_timer as clock >>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 0.83996295929 >>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 0.739042043686 >>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 0.73944401741 >>> t = clock(); a= sin(x).series(x, 0, 1000); t = clock()-t; print t 0.739789009094 We are in the process of speeding up sympy, you can also play with our experimental core that doesn't use caching: $ git clone git://github.com/certik/sympyx.git $ cd sympyx $ ~/repos/pypy-dist/pypy/translator/goal/pypy-c Python 2.4.1 (pypy 1.0.0 build 57617) on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``"peephope" optimizations are what an optimistic Compiler uses'' >>>> from sympy import var I: import sympy_pyx ... fail (No module named sympy_pyx) W: can't import sympy_pyx -- will be pure python >>>> var("x y z") (x, y, z) >>>> e = (x+y+z+1)**40 >>>> from timeit import default_timer as clock >>>> t = clock(); a= e.expand(); t = clock()-t; print t 2.6539349556 >>>> t = clock(); a= e.expand(); t = clock()-t; print t 2.73331212997 >>>> t = clock(); a= e.expand(); t = clock()-t; print t 2.76934981346 >>>> $ python Python 2.5.2 (r252:60911, Aug 8 2008, 09:22:44) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from sympy import var I: import sympy_pyx ... fail (No module named sympy_pyx) W: can't import sympy_pyx -- will be pure python >>> var("x y z") (x, y, z) >>> e = (x+y+z+1)**40 >>> from timeit import default_timer as clock >>> t = clock(); a= e.expand(); t = clock()-t; print t 0.867521047592 >>> t = clock(); a= e.expand(); t = clock()-t; print t 1.13780593872 >>> t = clock(); a= e.expand(); t = clock()-t; print t 1.13062214851 >>> t = clock(); a= e.expand(); t = clock()-t; print t 1.14683890343 >>> t = clock(); a= e.expand(); t = clock()-t; print t 1.17529797554 So for that pypy is only 2x to 3x slower. BTW, when you compile our core using Cython, it get's way faster: $ make cython --convert-range sympy_pyx.pyx gcc -I/usr/include/python2.5 -I/usr/include/python2.5 -g -O0 -fPIC -c -o sympy_pyx.o sympy_pyx.c gcc -shared sympy_pyx.o -o sympy_pyx.so $ python Python 2.5.2 (r252:60911, Aug 8 2008, 09:22:44) [GCC 4.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from sympy import var I: import sympy_pyx ... ok >>> var("x y z") (Symbol(x), Symbol(y), Symbol(z)) >>> e = (x+y+z+1)**40 >>> from timeit import default_timer as clock >>> t = clock(); a= e.expand(); t = clock()-t; print t 0.214432001114 >>> t = clock(); a= e.expand(); t = clock()-t; print t 0.271343946457 >>> t = clock(); a= e.expand(); t = clock()-t; print t 0.239308834076 Do you have any plans for supporting writing C extensions to pypy? Because as you can see above, the speed is imho only possible when writing it in C. Well, if RPython could produce as optimized code as Cython, then it could be an option. Any ideas on that? Ondrej From arigo at tunes.org Mon Aug 25 16:04:57 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Aug 2008 16:04:57 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> Message-ID: <20080825140456.GA17156@code0.codespeak.net> Hi Ondrej, On Sun, Aug 24, 2008 at 04:08:29PM +0200, Ondrej Certik wrote: > TypeError: instance layout conflicts in multiple inheritance Not completely surprizingly, our multiple inheritance "instance layout conflict" condition is not 100% the same as CPython's (as the latter is really obscure and based on conditions like the size of C-level structures). Still, I think that this case should work, so it looks like a bug in PyPy. Filed issue #390: https://codespeak.net/issue/pypy-dev/issue390 Armin From ondrej at certik.cz Mon Aug 25 16:20:40 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 25 Aug 2008 16:20:40 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <20080825140456.GA17156@code0.codespeak.net> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <20080825140456.GA17156@code0.codespeak.net> Message-ID: <85b5c3130808250720l303ef822s20f0707e791558ba@mail.gmail.com> On Mon, Aug 25, 2008 at 4:04 PM, Armin Rigo wrote: > Hi Ondrej, > > On Sun, Aug 24, 2008 at 04:08:29PM +0200, Ondrej Certik wrote: >> TypeError: instance layout conflicts in multiple inheritance > > Not completely surprizingly, our multiple inheritance "instance layout > conflict" condition is not 100% the same as CPython's (as the latter is > really obscure and based on conditions like the size of C-level > structures). Still, I think that this case should work, so it looks > like a bug in PyPy. > > Filed issue #390: https://codespeak.net/issue/pypy-dev/issue390 Thanks Armin for looking into this. Btw, if it helps, ironpython also fails on conflict1.py from #390: $ ipy conflict1.py Traceback (most recent call last): File conflict1, line unknown, in Initialize File mscorlib, line unknown, in get_Item KeyError: The given key was not present in the dictionary. Also I don't like mutliple inheritance, so we should just fix sympy and be done with it. Unfortunately IronPython fails later too and Jython as well: $ /home/ondra/ext/jython/jython Jython 2.5a1+ (asm:4943:4945, Jul 15 2008, 15:30:04) [Java HotSpot(TM) Server VM (Sun Microsystems Inc.)] on java1.5.0_15 Type "help", "copyright", "credits" or "license" for more information. >>> import sympy Traceback (most recent call last): File "", line 1, in File "sympy/__init__.py", line 16, in from sympy.core import * File "sympy/core/__init__.py", line 4, in from basic import Basic, S, C, sympify File "sympy/core/basic.py", line 236, in class Basic(AssumeMeths): TypeError: Error when calling the metaclass bases mro() returned base with unsuitable layout ('Basic') So this suggests that sympy code is not standard. Ondrej From fijall at gmail.com Mon Aug 25 22:26:00 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 25 Aug 2008 22:26:00 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808250428i78d05763wc9a2bda072e98a70@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250123n4bca4adfr69020b01d331d96a@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> <693bc9ab0808250404n2019ecc7k7e2231cea3af6cad@mail.gmail.com> <85b5c3130808250428i78d05763wc9a2bda072e98a70@mail.gmail.com> Message-ID: <693bc9ab0808251326n6f2acf35la4b4c73b3194db94@mail.gmail.com> > > > > Do you have any plans for supporting writing C extensions to pypy? > Because as you can see above, the speed is imho only possible when > writing it in C. Well, if RPython could produce as optimized code as > Cython, then it could be an option. Any ideas on that? > > > Ondrej > There are no immediate plans for supporting C extensions to pypy besides via ctypes or rpython. RPython is fairly fast these days, try it out if you like (big fat warning, rpython is an obscure language). Also usecases like sympy one should see relatively good speedups via JIT. Did you try using psyco? Cheers, fijal From ondrej at certik.cz Mon Aug 25 23:29:03 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 25 Aug 2008 23:29:03 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <693bc9ab0808251326n6f2acf35la4b4c73b3194db94@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> <693bc9ab0808250404n2019ecc7k7e2231cea3af6cad@mail.gmail.com> <85b5c3130808250428i78d05763wc9a2bda072e98a70@mail.gmail.com> <693bc9ab0808251326n6f2acf35la4b4c73b3194db94@mail.gmail.com> Message-ID: <85b5c3130808251429r6c16dc6an355c4ed872bb974@mail.gmail.com> On Mon, Aug 25, 2008 at 10:26 PM, Maciej Fijalkowski wrote: >> >> >> >> Do you have any plans for supporting writing C extensions to pypy? >> Because as you can see above, the speed is imho only possible when >> writing it in C. Well, if RPython could produce as optimized code as >> Cython, then it could be an option. Any ideas on that? >> >> >> Ondrej >> > > There are no immediate plans for supporting C extensions to pypy > besides via ctypes or rpython. RPython is fairly fast these days, try > it out if you like (big fat warning, rpython is an obscure language). > Also usecases like sympy one should see relatively good speedups via > JIT. Did you try using psyco? Psyco gives pretty good results (but not as good as Cython), but as I understood, it is not developed anymore as I thought pypy is the way to go. Yes, Cython is also a new language basically, but it's very easy to learn and easy to debug, easy to compile and one is able to blend C and Python very easily and naturally, so it's my choice. But on the other hand, I really think that one should just write what he wants in Python, maybe give it couple hints and pypy (or anything else) should be clever enough to optimize it. Cython is not perfect, but one can get C level speed now. They plan to allow a pure python syntax, so that the same code actually also runs in Python. But anyway, pypy has my thumbs up. 3x slower than CPython is not *that* bad, so imho if you could release, it'd be awesome. Ondrej From seob at gmx.ch Tue Aug 26 17:35:06 2008 From: seob at gmx.ch (Severin) Date: Tue, 26 Aug 2008 17:35:06 +0200 Subject: [pypy-dev] master thesis with pypy Message-ID: Hello everybody, I am Severin - a student at ETH in Zurich. Currently i'm enrolled at the Masters program in computer science. The next 6 month, starting from the 15th of September I will do my thesis. This more or less means working on a project. Since me and my mentor (Nicholas Matsakis) both share interest in python we came up with the idea to do some work with the pypy project. We started to talk about what we could do and what would be interesting for us. In some way we wanted to '*improve multiprocessor capabilities*' in pypy. We had some ideas so far, but we are not yet fixed on something. I was wondering if somebody of you has something that is of particular interest to the pypy community. Some ideas that might be interesting to look at for me. Any kind of input or idea is welcome... Best regards, Severin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080826/c67cc772/attachment.htm From sanxiyn at gmail.com Tue Aug 26 18:21:50 2008 From: sanxiyn at gmail.com (Seo Sanghyeon) Date: Wed, 27 Aug 2008 01:21:50 +0900 Subject: [pypy-dev] master thesis with pypy In-Reply-To: References: Message-ID: <5b0248170808260921j7c2e4bb4hb97980d7e10d8818@mail.gmail.com> 2008/8/27 Severin : > We started to talk about what we could do and what would be interesting for > us. In some way we wanted to 'improve multiprocessor capabilities' in pypy. > We had some ideas so far, but we are not yet fixed on something. > > I was wondering if somebody of you has something that is of particular > interest to the pypy community. Some ideas that might be interesting to look > at for me. Any kind of input or idea is welcome... A random idea. Implementing python-safethread monitors for PyPy? http://code.google.com/p/python-safethread/wiki/Monitors -- Seo Sanghyeon From fijall at gmail.com Tue Aug 26 18:30:56 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 Aug 2008 18:30:56 +0200 Subject: [pypy-dev] master thesis with pypy In-Reply-To: References: Message-ID: <693bc9ab0808260930u74ea5ed8k8fbed6869d0d7d32@mail.gmail.com> Hello. Can you explain a bit in which direction would you like to go? Or is the topic vague for a reason to experiment a bit? Would you rather want say, create some GILless locking mechanism in pypy or rather implement some cool multi-process framework based on what is done already? Cheers, fijal On Tue, Aug 26, 2008 at 5:35 PM, Severin wrote: > Hello everybody, > > I am Severin - a student at ETH in Zurich. Currently i'm enrolled at the > Masters program in computer science. The next 6 month, starting from the > 15th of September I will do my thesis. This more or less means working on a > project. Since me and my mentor (Nicholas Matsakis) both share interest in > python we came up with the idea to do some work with the pypy project. > > We started to talk about what we could do and what would be interesting for > us. In some way we wanted to 'improve multiprocessor capabilities' in pypy. > We had some ideas so far, but we are not yet fixed on something. > > I was wondering if somebody of you has something that is of particular > interest to the pypy community. Some ideas that might be interesting to look > at for me. Any kind of input or idea is welcome... > > Best regards, > Severin > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From simon at arrowtheory.com Tue Aug 26 22:27:15 2008 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 26 Aug 2008 16:27:15 -0400 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808251429r6c16dc6an355c4ed872bb974@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <85b5c3130808250144h7491af7ct44466b50eb5e6afa@mail.gmail.com> <693bc9ab0808250151h70833903ub9a4d28476ebc90d@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> <693bc9ab0808250404n2019ecc7k7e2231cea3af6cad@mail.gmail.com> <85b5c3130808250428i78d05763wc9a2bda072e98a70@mail.gmail.com> <693bc9ab0808251326n6f2acf35la4b4c73b3194db94@mail.gmail.com> <85b5c3130808251429r6c16dc6an355c4ed872bb974@mail.gmail.com> Message-ID: <20080826162715.e67b96ed.simon@arrowtheory.com> On Mon, 25 Aug 2008 23:29:03 +0200 "Ondrej Certik" wrote: > Cython is not perfect, but one can get C level speed now. They plan to > allow a pure python syntax, so that the same code actually also runs > in Python. What do you mean here ? Cython already allows pure python syntax. Simon. From ondrej at certik.cz Wed Aug 27 00:19:06 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 27 Aug 2008 00:19:06 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <20080826162715.e67b96ed.simon@arrowtheory.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <693bc9ab0808250219o5e409d6am126cca7e62c126aa@mail.gmail.com> <85b5c3130808250245mf171848q6dc4d336d3fc01@mail.gmail.com> <693bc9ab0808250308u57e9efb2obf67781ddd7ebcf3@mail.gmail.com> <85b5c3130808250350k48495900xdafc1db237c50f4b@mail.gmail.com> <693bc9ab0808250404n2019ecc7k7e2231cea3af6cad@mail.gmail.com> <85b5c3130808250428i78d05763wc9a2bda072e98a70@mail.gmail.com> <693bc9ab0808251326n6f2acf35la4b4c73b3194db94@mail.gmail.com> <85b5c3130808251429r6c16dc6an355c4ed872bb974@mail.gmail.com> <20080826162715.e67b96ed.simon@arrowtheory.com> Message-ID: <85b5c3130808261519s12942b34na75f98fe0660a45c@mail.gmail.com> On Tue, Aug 26, 2008 at 10:27 PM, Simon Burton wrote: > On Mon, 25 Aug 2008 23:29:03 +0200 > "Ondrej Certik" wrote: > >> Cython is not perfect, but one can get C level speed now. They plan to >> allow a pure python syntax, so that the same code actually also runs >> in Python. > > What do you mean here ? Cython already allows pure python syntax. Does it? Wow, didn't know about it. How do you write this in pure python syntax? def f(int i, int j): return i*j ? What I mean is to use some decorator, or some other (python) syntax to represent the "int" in there. Ondrej From pedronis at openend.se Wed Aug 27 11:23:35 2008 From: pedronis at openend.se (Samuele Pedroni) Date: Wed, 27 Aug 2008 11:23:35 +0200 Subject: [pypy-dev] Pypy IRC meeting fri 17:00 UTC on #pypy (freenode.net) Message-ID: <48B51D17.9020201@openend.se> Hi all, since EuroPython there's been some thinking and planning about doing a new release of PyPy 1.1 before the end of the year with a focus related to the recent work on compatibility with applications and CPython and stability. We think that an important step in the process is to improve and consolidate our testing infrastructure with the goal of making it easy to spot regressions. After some people availability checking on #pypy we agreed on scheduling a meeting: this Friday 29 Aug at 17:00 UTC (19:00 CEST) on #pypy IRC channel (freenode.net) The topics to discuss are: - which machines do we have available or we need (we would like at least to cover Linux x86-32 and windows 32bits) - requirements for our testing infrastructure - status of buildbot related work, what would it take to use buildbot for all out testing needs? - things to do first This is an open meeting, anybody interested in topics is invited to participate. It should not last more than 1 hour. We are also planning on having a sprint in October around the testing/release topics and others, we may also try at this meeting to finalize location and dates for that. regards. From arigo at tunes.org Thu Aug 28 13:44:42 2008 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Aug 2008 13:44:42 +0200 Subject: [pypy-dev] Pypy IRC meeting fri 17:00 UTC on #pypy (freenode.net) In-Reply-To: <48B51D17.9020201@openend.se> References: <48B51D17.9020201@openend.se> Message-ID: <20080828114442.GA20581@code0.codespeak.net> Hi all, On Wed, Aug 27, 2008 at 11:23:35AM +0200, Samuele Pedroni wrote: > this Friday 29 Aug at 17:00 UTC (19:00 CEST) Let me point out that time again, as it's not at 17:00 CEST as it usually was: it is at 19:00 CEST. Armin From paskma at gmail.com Thu Aug 28 16:05:37 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Thu, 28 Aug 2008 16:05:37 +0200 Subject: [pypy-dev] pypy-jvm translation, patch Message-ID: <4ddb710e0808280705r768d68e3h3316a11382e9db30@mail.gmail.com> Hi, today, I tried to translate pypy-jvm, but the translation failed with the trace below. The following patch makes the translation to be completed: Index: pypy/translator/jvm/metavm.py =================================================================== --- pypy/translator/jvm/metavm.py (revision 57667) +++ pypy/translator/jvm/metavm.py (working copy) @@ -93,6 +93,7 @@ (ootype.Signed, ootype.UnsignedLongLong): jvm.I2L, (ootype.SignedLongLong, ootype.Signed): jvm.L2I, (ootype.UnsignedLongLong, ootype.SignedLongLong): None, + (ootype.UnsignedLongLong, ootype.Unsigned): jvm.L2I, } class _CastPrimitive(MicroInstruction): However, I am not really sure whether the patch is really correct :-) Pypy-jvm does not work anyway, it is the problem I reported several weeks ago: paskma at paskma:goal$ ./pypy-jvm 'import site' failed Python 2.4.1 (pypy 1.0.0 build 100000) on linux2 Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``nothing is true'' Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' (Don't be confused by the build number, I am using git-svn.) Marek Pa?ka The traceback: [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 278, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/driver.py", line 797, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/tool/taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/driver.py", line 268, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/driver.py", line 709, in task_source_jvm [translation:ERROR] self.jvmsource = self.gen.generate_source() [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/genjvm.py", line 266, in generate_source [translation:ERROR] GenOO.generate_source(self) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/genoo.py", line 68, in generate_source [translation:ERROR] self.gen_pendings() [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/genoo.py", line 83, in gen_pendings [translation:ERROR] node.render(self.ilasm) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/node.py", line 899, in render [translation:ERROR] method.render(gen) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/function.py", line 129, in render [translation:ERROR] self.render_normal_block(block) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/node.py", line 473, in render_normal_block [translation:ERROR] return OOFunction.render_normal_block(self, block) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/function.py", line 218, in render_normal_block [translation:ERROR] self._render_op(op) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/node.py", line 574, in _render_op [translation:ERROR] OOFunction._render_op(self, op) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/function.py", line 351, in _render_op [translation:ERROR] instr_list.render(self.generator, op) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/metavm.py", line 247, in render [translation:ERROR] instr.render(generator, op) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/metavm.py", line 62, in render [translation:ERROR] jmethod.return_type, op.result) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/metavm.py", line 21, in _invoke_method [translation:ERROR] gen.load(arg) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/generator.py", line 495, in load [translation:ERROR] return render_sub_op(value, self.db, self) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/function.py", line 24, in render_sub_op [translation:ERROR] instr_list.render(generator, op) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/oosupport/metavm.py", line 247, in render [translation:ERROR] instr.render(generator, op) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/jvm/metavm.py", line 102, in render [translation:ERROR] opcode = CASTS[(FROM, TO)] [translation:ERROR] KeyError: (, ) From nvetoshkin at naumen.ru Thu Aug 28 17:13:58 2008 From: nvetoshkin at naumen.ru (=?UTF-8?B?0JLQtdGC0L7RiNC60LjQvSDQndC40LrQuNGC0LA=?=) Date: Thu, 28 Aug 2008 21:13:58 +0600 Subject: [pypy-dev] translate.py failes Message-ID: <48B6C0B6.6060802@naumen.ru> Hi, everybody. Tried to play with pypy, but translating to c failes because of lack of memory. Used command: python translate.py. OS: gentoo linux, amd64 RAM: 4Gb Traceback attached to the letter. nekto0n -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: report Url: http://codespeak.net/pipermail/pypy-dev/attachments/20080828/74ef7df2/attachment.diff From arigo at tunes.org Thu Aug 28 17:31:10 2008 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Aug 2008 17:31:10 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <48B6C0B6.6060802@naumen.ru> References: <48B6C0B6.6060802@naumen.ru> Message-ID: <20080828153110.GA8912@code0.codespeak.net> Hi, On Thu, Aug 28, 2008 at 09:13:58PM +0600, ???????????????? ???????????? wrote: > Tried to play with pypy, but translating to c failes because of lack of > memory. This is a known issue. We will try to implement a workaround. For now you need to have more swap available. Note that the swap will not actually be used if you have >= 2GB of RAM; it is really a limitation of some Linux configurations in which a process that consumes about half of the total amount of memory+swap cannot spawn new processes because os.fork() fails. A bientot, Armin. From nvetoshkin at naumen.ru Thu Aug 28 18:34:51 2008 From: nvetoshkin at naumen.ru (Vetoshkin Nikita) Date: Thu, 28 Aug 2008 22:34:51 +0600 Subject: [pypy-dev] translate.py failes In-Reply-To: <20080828153110.GA8912@code0.codespeak.net> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> Message-ID: <48B6D3AB.4050801@naumen.ru> Is there a direction, where I could digg to help? Something to begin with. I don't have enough experience but I'd like to take part (even very small) in this project. Armin Rigo wrote: > Hi, > > On Thu, Aug 28, 2008 at 09:13:58PM +0600, ???????????????? ???????????? wrote: >> Tried to play with pypy, but translating to c failes because of lack of >> memory. > > This is a known issue. We will try to implement a workaround. For now > you need to have more swap available. Note that the swap will not > actually be used if you have>= 2GB of RAM; it is really a limitation of > some Linux configurations in which a process that consumes about half of > the total amount of memory+swap cannot spawn new processes because > os.fork() fails. > > > A bientot, > > Armin. -- --- ? ?????????, ???????? ?????? ??????? ????????? ?????? IP-????????? ????????? ????? ???????? Naumen From fijall at gmail.com Fri Aug 29 12:09:20 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 29 Aug 2008 12:09:20 +0200 Subject: [pypy-dev] commit bot Message-ID: <693bc9ab0808290309o78597f4fi4c4fe961ceef5828@mail.gmail.com> what about moving from CIA to keenan (the bot that is sitting on #pypy-commits and did not have a single downtime since last few months) for commit messages on #pypy? CIA was down again yesterday. cheers, fijal From tismer at stackless.com Fri Aug 29 17:02:49 2008 From: tismer at stackless.com (Christian Tismer) Date: Fri, 29 Aug 2008 17:02:49 +0200 Subject: [pypy-dev] Pypy IRC meeting fri 17:00 UTC on #pypy (freenode.net) In-Reply-To: <20080828114442.GA20581@code0.codespeak.net> References: <48B51D17.9020201@openend.se> <20080828114442.GA20581@code0.codespeak.net> Message-ID: <48B80F99.6050300@stackless.com> Armin Rigo wrote: > Hi all, > > On Wed, Aug 27, 2008 at 11:23:35AM +0200, Samuele Pedroni wrote: >> this Friday 29 Aug at 17:00 UTC (19:00 CEST) > > Let me point out that time again, as it's not at 17:00 CEST as it > usually was: it is at 19:00 CEST. I have to say that it would be extremely helpful NOT to repeat the wrong time in the unchanged subject line. Recognized that too late and made appointments. Really wanted to attend, but I'm reading IRC rarely in the hospital. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Fri Aug 29 17:47:08 2008 From: tismer at stackless.com (Christian Tismer) Date: Fri, 29 Aug 2008 17:47:08 +0200 Subject: [pypy-dev] Pypy IRC meeting fri 17:00 UTC on #pypy (freenode.net) In-Reply-To: <48B80F99.6050300@stackless.com> References: <48B51D17.9020201@openend.se> <20080828114442.GA20581@code0.codespeak.net> <48B80F99.6050300@stackless.com> Message-ID: <48B819FC.60209@stackless.com> Christian Tismer wrote: > Armin Rigo wrote: >> Hi all, >> >> On Wed, Aug 27, 2008 at 11:23:35AM +0200, Samuele Pedroni wrote: >>> this Friday 29 Aug at 17:00 UTC (19:00 CEST) >> Let me point out that time again, as it's not at 17:00 CEST as it >> usually was: it is at 19:00 CEST. > > I have to say that it would be extremely helpful NOT to repeat > the wrong time in the unchanged subject line. Recognized that too > late and made appointments. > Really wanted to attend, but I'm reading IRC rarely in the hospital. Argh. Argh Argh, of course. Nothing was wrong with the title of this posting, but still it was extremely misleading for me. Will be more careful next time, but this one went wrong, sorry. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Fri Aug 29 17:55:11 2008 From: arigo at tunes.org (Armin Rigo) Date: Fri, 29 Aug 2008 17:55:11 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <48B6D3AB.4050801@naumen.ru> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> Message-ID: <20080829155511.GA9583@code0.codespeak.net> Hi, On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote: > Is there a direction, where I could digg to help? Something to begin with. > I don't have enough experience but I'd like to take part (even very > small) in this project. Sure. Do you understand the issue? It's related to the way a Unix process starts a new subprocess: os.fork() followed by os.execv() in the child process. The issue is that os.fork() fails if the parent process' total memory is more than half of your RAM+swap, because it tries to create a copy of the process, and the copy would also need more than half of your RAM+swap. (In reality, os.fork() doesn't copy the memory at all, but creates "shared pages" of memory so that memory pages are duplicated only when one of the two processes really modifies it; but still os.fork() has to mark all the memory as *potentially* used, and raises MemoryError if there isn't enough.) A solution might be to split translate.py in two processes: translate.py would start a subprocess when it starts, and then do all the work (consuming a lot of RAM); but when it needs to start new processes, e.g. call gcc, it would instead ask the subprocess to start the new processes. More precisely, the code in pypy.translator.tool.cbuild would stop using distutils directly, and instead ask the subprocess to use distutils. It looks even easy to do with the help of a py.execnet.PopenGateway() created when translate.py starts (see the py lib documentation at http://codespeak.net/py/). A bientot, Armin From arigo at tunes.org Fri Aug 29 18:00:55 2008 From: arigo at tunes.org (Armin Rigo) Date: Fri, 29 Aug 2008 18:00:55 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <85b5c3130808250720l303ef822s20f0707e791558ba@mail.gmail.com> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <20080825140456.GA17156@code0.codespeak.net> <85b5c3130808250720l303ef822s20f0707e791558ba@mail.gmail.com> Message-ID: <20080829160055.GB9583@code0.codespeak.net> Hi Ondrej, On Mon, Aug 25, 2008 at 04:20:40PM +0200, Ondrej Certik wrote: > class Basic(AssumeMeths): > TypeError: Error when calling the metaclass bases > mro() returned base with unsuitable layout ('Basic') > > So this suggests that sympy code is not standard. That's obscure. It probably points to a bug in Jython and IronPython, or just some feature that is really obscure but that PyPy implements like CPython by chance. I fixed the multiple inheritance bug as well as the bug shown by the @threaded() decorator. Since rev 57656, pypy-c can import sympy. (I didn't try to do anything with it, though.) A bientot, Armin. From holger at merlinux.eu Fri Aug 29 20:38:01 2008 From: holger at merlinux.eu (holger krekel) Date: Fri, 29 Aug 2008 20:38:01 +0200 Subject: [pypy-dev] commit bot In-Reply-To: <693bc9ab0808290309o78597f4fi4c4fe961ceef5828@mail.gmail.com> References: <693bc9ab0808290309o78597f4fi4c4fe961ceef5828@mail.gmail.com> Message-ID: <20080829183801.GD9619@trillke.net> Hi Maciej, On Fri, Aug 29, 2008 at 12:09 +0200, Maciej Fijalkowski wrote: > what about moving from CIA to keenan (the bot that is sitting on > #pypy-commits and did not have a single downtime since last few > months) for commit messages on #pypy? CIA was down again yesterday. I think we should give CIA some more chances to recover from the resource problem they have. it's a bit of effort to move to yet another solution (roundup detectors, changing/testing codespeak config etc.). but good to know that on #pypy-commits one can get reliable dedicated pypy commits. holger From ondrej at certik.cz Fri Aug 29 21:11:31 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 29 Aug 2008 21:11:31 +0200 Subject: [pypy-dev] works in python2.4, fails in pypy In-Reply-To: <20080829160055.GB9583@code0.codespeak.net> References: <85b5c3130808240708h20845b41y9b4db4627f5964dc@mail.gmail.com> <20080825140456.GA17156@code0.codespeak.net> <85b5c3130808250720l303ef822s20f0707e791558ba@mail.gmail.com> <20080829160055.GB9583@code0.codespeak.net> Message-ID: <85b5c3130808291211q217785d9s4539f1bc1eb94ecd@mail.gmail.com> On Fri, Aug 29, 2008 at 6:00 PM, Armin Rigo wrote: > Hi Ondrej, > > On Mon, Aug 25, 2008 at 04:20:40PM +0200, Ondrej Certik wrote: >> class Basic(AssumeMeths): >> TypeError: Error when calling the metaclass bases >> mro() returned base with unsuitable layout ('Basic') >> >> So this suggests that sympy code is not standard. > > That's obscure. It probably points to a bug in Jython and IronPython, > or just some feature that is really obscure but that PyPy implements > like CPython by chance. Yeah. We should try to fix sympy, so that we don't use obscure features. > > I fixed the multiple inheritance bug as well as the bug shown by the > @threaded() decorator. Since rev 57656, pypy-c can import sympy. (I > didn't try to do anything with it, though.) Thanks for fixing it. Ondrej From fijall at gmail.com Sat Aug 30 10:29:35 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 30 Aug 2008 10:29:35 +0200 Subject: [pypy-dev] commit bot In-Reply-To: <20080829183801.GD9619@trillke.net> References: <693bc9ab0808290309o78597f4fi4c4fe961ceef5828@mail.gmail.com> <20080829183801.GD9619@trillke.net> Message-ID: <693bc9ab0808300129r110f070ax4306204d346e7dec@mail.gmail.com> On Fri, Aug 29, 2008 at 8:38 PM, holger krekel wrote: > Hi Maciej, > > On Fri, Aug 29, 2008 at 12:09 +0200, Maciej Fijalkowski wrote: >> what about moving from CIA to keenan (the bot that is sitting on >> #pypy-commits and did not have a single downtime since last few >> months) for commit messages on #pypy? CIA was down again yesterday. > > I think we should give CIA some more chances > to recover from the resource problem they have. > it's a bit of effort to move to yet another solution (roundup > detectors, changing/testing codespeak config etc.). Personally I think we gave it too much chances. Also I would like to rely on roundup detectors etc via CIA and only do: 1. disable commit messages from CIA 2. move keenan from #pypy-commits to #pypy that's super easy and simplifies stuff already. > > but good to know that on #pypy-commits > one can get reliable dedicated pypy commits. > > holger > From nvetoshkin at naumen.ru Tue Sep 2 20:40:58 2008 From: nvetoshkin at naumen.ru (Vetoshkin Nikita) Date: Wed, 03 Sep 2008 00:40:58 +0600 Subject: [pypy-dev] translate.py failes In-Reply-To: <20080829155511.GA9583@code0.codespeak.net> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> Message-ID: <48BD88BA.3010900@naumen.ru> Thanks a lot, Armin. I know how os.fork() works and about copy-on-write too, so as soon as I got time, I'll try to fix it. My interest in translating and playing then with pypy was exactly py.execnet =) Armin Rigo wrote: > Hi, > > On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote: >> Is there a direction, where I could digg to help? Something to begin with. >> I don't have enough experience but I'd like to take part (even very >> small) in this project. > > Sure. Do you understand the issue? It's related to the way a Unix > process starts a new subprocess: os.fork() followed by os.execv() in the > child process. The issue is that os.fork() fails if the parent process' > total memory is more than half of your RAM+swap, because it tries to > create a copy of the process, and the copy would also need more than > half of your RAM+swap. (In reality, os.fork() doesn't copy the memory > at all, but creates "shared pages" of memory so that memory pages are > duplicated only when one of the two processes really modifies it; but > still os.fork() has to mark all the memory as *potentially* used, and > raises MemoryError if there isn't enough.) > > A solution might be to split translate.py in two processes: translate.py > would start a subprocess when it starts, and then do all the work > (consuming a lot of RAM); but when it needs to start new processes, e.g. > call gcc, it would instead ask the subprocess to start the new > processes. More precisely, the code in pypy.translator.tool.cbuild > would stop using distutils directly, and instead ask the subprocess to > use distutils. It looks even easy to do with the help of a > py.execnet.PopenGateway() created when translate.py starts (see the py > lib documentation at http://codespeak.net/py/). > > > A bientot, > > Armin -- --- ? ?????????, ???????? ?????? ??????? ????????? ?????? IP-????????? ????????? ????? ???????? Naumen From seob at gmx.ch Wed Sep 3 11:31:01 2008 From: seob at gmx.ch (Severin) Date: Wed, 3 Sep 2008 11:31:01 +0200 Subject: [pypy-dev] master thesis with pypy In-Reply-To: <5b0248170808260921j7c2e4bb4hb97980d7e10d8818@mail.gmail.com> References: <5b0248170808260921j7c2e4bb4hb97980d7e10d8818@mail.gmail.com> Message-ID: safethread sound like something interesting. are there already any safthread ideas in pypy? In general, working on GILless ideas sounds more interesting than doing something on multiprocess framworks. It's a master thesis, that means there is room for experimental stuff, but in the end half a year is quite short, so the room is limited again :) was there any attempt to remove the GIL in pypy so far? -- Severin On Tue, Aug 26, 2008 at 6:21 PM, Seo Sanghyeon wrote: > 2008/8/27 Severin : > > We started to talk about what we could do and what would be interesting > for > > us. In some way we wanted to 'improve multiprocessor capabilities' in > pypy. > > We had some ideas so far, but we are not yet fixed on something. > > > > I was wondering if somebody of you has something that is of particular > > interest to the pypy community. Some ideas that might be interesting to > look > > at for me. Any kind of input or idea is welcome... > > A random idea. Implementing python-safethread monitors for PyPy? > http://code.google.com/p/python-safethread/wiki/Monitors > > -- > Seo Sanghyeon > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080903/4d7ab7a7/attachment.htm From fijall at gmail.com Wed Sep 3 11:44:29 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 3 Sep 2008 11:44:29 +0200 Subject: [pypy-dev] master thesis with pypy In-Reply-To: References: <5b0248170808260921j7c2e4bb4hb97980d7e10d8818@mail.gmail.com> Message-ID: <693bc9ab0809030244i10c50ae0t8bd1dc18da4a90d1@mail.gmail.com> On Wed, Sep 3, 2008 at 11:31 AM, Severin wrote: > safethread sound like something interesting. are there already any safthread > ideas in pypy? > In general, working on GILless ideas sounds more interesting than doing > something on multiprocess framworks. > It's a master thesis, that means there is room for experimental stuff, but > in the end half a year is quite short, so the room is limited again :) > was there any attempt to remove the GIL in pypy so far? > -- > Severin > None that I know of at least. I think any serious GIL-removal proposal requires some deeper thinking about what exactly are the expected semantics. Cheers, fijal From rhamph at gmail.com Wed Sep 3 20:15:37 2008 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 3 Sep 2008 12:15:37 -0600 Subject: [pypy-dev] master thesis with pypy In-Reply-To: <693bc9ab0809030244i10c50ae0t8bd1dc18da4a90d1@mail.gmail.com> References: <5b0248170808260921j7c2e4bb4hb97980d7e10d8818@mail.gmail.com> <693bc9ab0809030244i10c50ae0t8bd1dc18da4a90d1@mail.gmail.com> Message-ID: On Wed, Sep 3, 2008 at 3:44 AM, Maciej Fijalkowski wrote: > On Wed, Sep 3, 2008 at 11:31 AM, Severin wrote: >> safethread sound like something interesting. are there already any safthread >> ideas in pypy? >> In general, working on GILless ideas sounds more interesting than doing >> something on multiprocess framworks. >> It's a master thesis, that means there is room for experimental stuff, but >> in the end half a year is quite short, so the room is limited again :) >> was there any attempt to remove the GIL in pypy so far? >> -- >> Severin >> > > None that I know of at least. I think any serious GIL-removal proposal > requires some deeper thinking about what exactly are the expected > semantics. Oi, how did I miss this thread before... Python-safethread's design of course includes most of this deep thinking. -- Adam Olsen, aka Rhamphoryncus From nvetoshkin at naumen.ru Wed Sep 3 21:43:31 2008 From: nvetoshkin at naumen.ru (Vetoshkin Nikita) Date: Thu, 04 Sep 2008 01:43:31 +0600 Subject: [pypy-dev] translate.py failes In-Reply-To: <20080829155511.GA9583@code0.codespeak.net> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> Message-ID: <48BEE8E3.6010503@naumen.ru> Playing with py.execnet failes too. Full traceback attached. P.S. Seens like a hard start with my trials in PyPy =) Armin Rigo wrote: > Hi, > > On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote: >> Is there a direction, where I could digg to help? Something to begin with. >> I don't have enough experience but I'd like to take part (even very >> small) in this project. > > Sure. Do you understand the issue? It's related to the way a Unix > process starts a new subprocess: os.fork() followed by os.execv() in the > child process. The issue is that os.fork() fails if the parent process' > total memory is more than half of your RAM+swap, because it tries to > create a copy of the process, and the copy would also need more than > half of your RAM+swap. (In reality, os.fork() doesn't copy the memory > at all, but creates "shared pages" of memory so that memory pages are > duplicated only when one of the two processes really modifies it; but > still os.fork() has to mark all the memory as *potentially* used, and > raises MemoryError if there isn't enough.) > > A solution might be to split translate.py in two processes: translate.py > would start a subprocess when it starts, and then do all the work > (consuming a lot of RAM); but when it needs to start new processes, e.g. > call gcc, it would instead ask the subprocess to start the new > processes. More precisely, the code in pypy.translator.tool.cbuild > would stop using distutils directly, and instead ask the subprocess to > use distutils. It looks even easy to do with the help of a > py.execnet.PopenGateway() created when translate.py starts (see the py > lib documentation at http://codespeak.net/py/). > > > A bientot, > > Armin -- Nikita -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: execnet_fail.txt Url: http://codespeak.net/pipermail/pypy-dev/attachments/20080904/54587572/attachment.txt From pedronis at openend.se Wed Sep 3 21:52:05 2008 From: pedronis at openend.se (Samuele Pedroni) Date: Wed, 03 Sep 2008 21:52:05 +0200 Subject: [pypy-dev] meeting summary Re: Pypy IRC meeting fri 17:00 UTC on #pypy (freenode.net) In-Reply-To: <48B51D17.9020201@openend.se> References: <48B51D17.9020201@openend.se> Message-ID: <48BEEAE5.1050606@openend.se> Samuele Pedroni wrote: > Hi all, > > since EuroPython there's been some thinking and planning about doing a > new release of PyPy 1.1 before the end of the year with a focus related > to the recent work on compatibility with applications and CPython and > stability. > > We think that an important step in the process is to improve and > consolidate our testing infrastructure with the goal of making it easy > to spot regressions. > > After some people availability checking on #pypy we agreed on scheduling > a meeting: > > this Friday 29 Aug at 17:00 UTC (19:00 CEST) > on #pypy IRC channel (freenode.net) > I have checked in a brief summary of the meeting as: http://codespeak.net/svn/pypy/extradoc/planning/1.1/testing_infrastructure.txt regards. From fijall at gmail.com Wed Sep 3 21:52:37 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 3 Sep 2008 21:52:37 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <48BEE8E3.6010503@naumen.ru> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> <48BEE8E3.6010503@naumen.ru> Message-ID: <693bc9ab0809031252j57c4610bh77e37c5db1143359@mail.gmail.com> I'm not completele sure what you're trying to do. Can you explain in a bit more detail? Also feel free to drop by on IRC for a live discussion. cheers, fijal On Wed, Sep 3, 2008 at 9:43 PM, Vetoshkin Nikita wrote: > Playing with py.execnet failes too. Full traceback attached. > > P.S. Seens like a hard start with my trials in PyPy =) > > Armin Rigo wrote: >> >> Hi, >> >> On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote: >>> >>> Is there a direction, where I could digg to help? Something to begin >>> with. >>> I don't have enough experience but I'd like to take part (even very >>> small) in this project. >> >> Sure. Do you understand the issue? It's related to the way a Unix >> process starts a new subprocess: os.fork() followed by os.execv() in the >> child process. The issue is that os.fork() fails if the parent process' >> total memory is more than half of your RAM+swap, because it tries to >> create a copy of the process, and the copy would also need more than >> half of your RAM+swap. (In reality, os.fork() doesn't copy the memory >> at all, but creates "shared pages" of memory so that memory pages are >> duplicated only when one of the two processes really modifies it; but >> still os.fork() has to mark all the memory as *potentially* used, and >> raises MemoryError if there isn't enough.) >> >> A solution might be to split translate.py in two processes: translate.py >> would start a subprocess when it starts, and then do all the work >> (consuming a lot of RAM); but when it needs to start new processes, e.g. >> call gcc, it would instead ask the subprocess to start the new >> processes. More precisely, the code in pypy.translator.tool.cbuild >> would stop using distutils directly, and instead ask the subprocess to >> use distutils. It looks even easy to do with the help of a >> py.execnet.PopenGateway() created when translate.py starts (see the py >> lib documentation at http://codespeak.net/py/). >> >> >> A bientot, >> >> Armin > > -- > Nikita > >>>>> gw = py.execnet.PopenGateway() > faking > faking > Traceback (most recent call last): > File "./pypy/bin/py.py", line 152, in > sys.exit(main_(sys.argv)) > File "./pypy/bin/py.py", line 138, in main_ > con.interact(banner) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line > 133, in interact > code.InteractiveConsole.interact(self, banner) > File "/usr/lib64/python2.5/code.py", line 239, in interact > more = self.push(line) > File "/usr/lib64/python2.5/code.py", line 261, in push > more = self.runsource(source, self.filename) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line > 192, in runsource > main.run_toplevel(self.space, doit, verbose=self.verbose) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/main.py", line 103, in > run_toplevel > f() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line > 184, in doit > code.exec_code(self.space, self.w_globals, self.w_globals) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in > exec_code > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 695, > in LOAD_ATTR > w_value = f.space.getattr(w_obj, w_attributename) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669, > in getattr > return DescrOperation.getattr(self, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 134, in getattr > return space._handle_getattribute(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 147, in _handle_getattribute > return space.get_and_call_function(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 89, in get_and_call_function > return descr.funccall(w_obj, *args_w) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 56, > in funccall > return code.fastcall_2(self.space, self, args_w[0], args_w[1]) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in > fastcall_2 > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, > in CALL_FUNCTION > w_result = f.space.call_valuestack(w_function, nargs, f) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 736, in call_valuestack > return w_func.funccall_valuestack(nargs, frame) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81, > in funccall_valuestack > frame.peekvalue(0)) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in > fastcall_2 > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, > in CALL_FUNCTION > w_result = f.space.call_valuestack(w_function, nargs, f) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 736, in call_valuestack > return w_func.funccall_valuestack(nargs, frame) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81, > in funccall_valuestack > frame.peekvalue(0)) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in > fastcall_2 > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, > in CALL_FUNCTION > w_result = f.space.call_valuestack(w_function, nargs, f) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 736, in call_valuestack > return w_func.funccall_valuestack(nargs, frame) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 88, > in funccall_valuestack > frame.peekvalue(0)) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663, > in fastcall_4 > w_result = self.fastfunc_4(space, w1, w2, w3, w4) > File "", line 3, in > fastfunc_importhook_4 > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 180, in importhook > w_mod = absolute_import(space, modulename, 0, w_fromlist, > tentative=0) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 192, in absolute_import > w_fromlist, tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 211, in _absolute_import > tentative=tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 272, in load_part > w(partname)) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 90, in try_import_mod > magic, timestamp, stream.readall()) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 499, in load_compiled_module > code_w.exec_code(space, w_dic, w_dic) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in > exec_code > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 760, > in IMPORT_NAME > w_locals, w_fromlist) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 712, in call_function > return w_func.funccall(*args_w) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 62, > in funccall > args_w[1], args_w[2], args_w[3]) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663, > in fastcall_4 > w_result = self.fastfunc_4(space, w1, w2, w3, w4) > File "", line 3, in > fastfunc_importhook_4 > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 180, in importhook > w_mod = absolute_import(space, modulename, 0, w_fromlist, > tentative=0) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 192, in absolute_import > w_fromlist, tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 211, in _absolute_import > tentative=tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 272, in load_part > w(partname)) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 90, in try_import_mod > magic, timestamp, stream.readall()) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 499, in load_compiled_module > code_w.exec_code(space, w_dic, w_dic) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in > exec_code > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 769, > in IMPORT_STAR > import_all_from(f.space, w_module, w_locals) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 839, > in appcaller > return space.call_args(w_func, args) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 98, in call_args > return w_obj.call_args(args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37, > in call_args > return self.code.funcrun(self, args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 483, > in funcrun > return BuiltinCode.funcrun_obj(self, func, None, args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 491, > in funcrun_obj > w_result = activation._run(space, scope_w) > File "", line 3, in > _run_UWS_ObjSpace_W_Root_W_Root > File > "/var/tmp/pypy-svn/dist/pypy/_cache/pyopcode_4922ecf55c7db5d81d0c4a63f92bdd44.py", > line 59, in import_all_from > w_2 = space.getattr(w_module, gs___dict__) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669, > in getattr > return DescrOperation.getattr(self, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 134, in getattr > return space._handle_getattribute(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 140, in _handle_getattribute > return space.get_and_call_function(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 89, in get_and_call_function > return descr.funccall(w_obj, *args_w) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 69, > in funccall > return self.call_args(Arguments(self.space, list(args_w))) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37, > in call_args > return self.code.funcrun(self, args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 61, in > funcrun > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 171, in > run > return self.space.wrap(result) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 475, > in wrap > items_w = [(self.wrap(k), self.wrap(v)) for (k, v) in x.iteritems()] > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 522, > in wrap > return fake_object(self, x) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 20, in > fake_object > ft = fake_type(x) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 58, in > fake_type > faked_type = really_build_fake_type(cpy_type) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 108, in > really_build_fake_type > assert cpy_type.__base__ is basestring > AssertionError > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From nvetoshkin at naumen.ru Wed Sep 3 21:59:39 2008 From: nvetoshkin at naumen.ru (Vetoshkin Nikita) Date: Thu, 04 Sep 2008 01:59:39 +0600 Subject: [pypy-dev] translate.py failes In-Reply-To: <693bc9ab0809031252j57c4610bh77e37c5db1143359@mail.gmail.com> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> <48BEE8E3.6010503@naumen.ru> <693bc9ab0809031252j57c4610bh77e37c5db1143359@mail.gmail.com> Message-ID: <48BEECAB.5020004@naumen.ru> Oh, sorry. Here are the details: launching pypy: ./pypy/bin/py.py then doing this: import py gw = py.execnet.PopenGateway() and getting that long long traceback. Machine: Gentoo Linux x64 Python 2.5.2 Would be glad to provide any useful information. I'm not very familiar with IRC, what is the server and channel to communicate on pypy related issues? Maciej Fijalkowski wrote: > I'm not completele sure what you're trying to do. Can you explain in a > bit more detail? Also feel free to drop by on IRC for a live > discussion. > > cheers, > fijal > > On Wed, Sep 3, 2008 at 9:43 PM, Vetoshkin Nikita wrote: >> Playing with py.execnet failes too. Full traceback attached. >> >> P.S. Seens like a hard start with my trials in PyPy =) >> >> Armin Rigo wrote: >>> Hi, >>> >>> On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote: >>>> Is there a direction, where I could digg to help? Something to begin >>>> with. >>>> I don't have enough experience but I'd like to take part (even very >>>> small) in this project. >>> Sure. Do you understand the issue? It's related to the way a Unix >>> process starts a new subprocess: os.fork() followed by os.execv() in the >>> child process. The issue is that os.fork() fails if the parent process' >>> total memory is more than half of your RAM+swap, because it tries to >>> create a copy of the process, and the copy would also need more than >>> half of your RAM+swap. (In reality, os.fork() doesn't copy the memory >>> at all, but creates "shared pages" of memory so that memory pages are >>> duplicated only when one of the two processes really modifies it; but >>> still os.fork() has to mark all the memory as *potentially* used, and >>> raises MemoryError if there isn't enough.) >>> >>> A solution might be to split translate.py in two processes: translate.py >>> would start a subprocess when it starts, and then do all the work >>> (consuming a lot of RAM); but when it needs to start new processes, e.g. >>> call gcc, it would instead ask the subprocess to start the new >>> processes. More precisely, the code in pypy.translator.tool.cbuild >>> would stop using distutils directly, and instead ask the subprocess to >>> use distutils. It looks even easy to do with the help of a >>> py.execnet.PopenGateway() created when translate.py starts (see the py >>> lib documentation at http://codespeak.net/py/). >>> >>> >>> A bientot, >>> >>> Armin >> -- >> Nikita >> >>>>>> gw = py.execnet.PopenGateway() >> faking >> faking >> Traceback (most recent call last): >> File "./pypy/bin/py.py", line 152, in >> sys.exit(main_(sys.argv)) >> File "./pypy/bin/py.py", line 138, in main_ >> con.interact(banner) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line >> 133, in interact >> code.InteractiveConsole.interact(self, banner) >> File "/usr/lib64/python2.5/code.py", line 239, in interact >> more = self.push(line) >> File "/usr/lib64/python2.5/code.py", line 261, in push >> more = self.runsource(source, self.filename) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line >> 192, in runsource >> main.run_toplevel(self.space, doit, verbose=self.verbose) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/main.py", line 103, in >> run_toplevel >> f() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line >> 184, in doit >> code.exec_code(self.space, self.w_globals, self.w_globals) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in >> exec_code >> return frame.run() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in >> run >> return self.execute_frame() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, >> in execute_frame >> executioncontext) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, >> in dispatch >> next_instr = self.handle_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, >> in handle_bytecode >> next_instr = self.dispatch_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, >> in dispatch_bytecode >> res = getattr(self, methodname)(oparg, next_instr) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 695, >> in LOAD_ATTR >> w_value = f.space.getattr(w_obj, w_attributename) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669, >> in getattr >> return DescrOperation.getattr(self, w_obj, w_name) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line >> 134, in getattr >> return space._handle_getattribute(w_descr, w_obj, w_name) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line >> 147, in _handle_getattribute >> return space.get_and_call_function(w_descr, w_obj, w_name) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line >> 89, in get_and_call_function >> return descr.funccall(w_obj, *args_w) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 56, >> in funccall >> return code.fastcall_2(self.space, self, args_w[0], args_w[1]) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in >> fastcall_2 >> return frame.run() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in >> run >> return self.execute_frame() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, >> in execute_frame >> executioncontext) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, >> in dispatch >> next_instr = self.handle_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, >> in handle_bytecode >> next_instr = self.dispatch_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, >> in dispatch_bytecode >> res = getattr(self, methodname)(oparg, next_instr) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, >> in CALL_FUNCTION >> w_result = f.space.call_valuestack(w_function, nargs, f) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line >> 736, in call_valuestack >> return w_func.funccall_valuestack(nargs, frame) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81, >> in funccall_valuestack >> frame.peekvalue(0)) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in >> fastcall_2 >> return frame.run() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in >> run >> return self.execute_frame() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, >> in execute_frame >> executioncontext) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, >> in dispatch >> next_instr = self.handle_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, >> in handle_bytecode >> next_instr = self.dispatch_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, >> in dispatch_bytecode >> res = getattr(self, methodname)(oparg, next_instr) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, >> in CALL_FUNCTION >> w_result = f.space.call_valuestack(w_function, nargs, f) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line >> 736, in call_valuestack >> return w_func.funccall_valuestack(nargs, frame) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81, >> in funccall_valuestack >> frame.peekvalue(0)) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in >> fastcall_2 >> return frame.run() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in >> run >> return self.execute_frame() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, >> in execute_frame >> executioncontext) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, >> in dispatch >> next_instr = self.handle_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, >> in handle_bytecode >> next_instr = self.dispatch_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, >> in dispatch_bytecode >> res = getattr(self, methodname)(oparg, next_instr) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, >> in CALL_FUNCTION >> w_result = f.space.call_valuestack(w_function, nargs, f) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line >> 736, in call_valuestack >> return w_func.funccall_valuestack(nargs, frame) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 88, >> in funccall_valuestack >> frame.peekvalue(0)) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663, >> in fastcall_4 >> w_result = self.fastfunc_4(space, w1, w2, w3, w4) >> File "", line 3, in >> fastfunc_importhook_4 >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 180, in importhook >> w_mod = absolute_import(space, modulename, 0, w_fromlist, >> tentative=0) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 192, in absolute_import >> w_fromlist, tentative) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 211, in _absolute_import >> tentative=tentative) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 272, in load_part >> w(partname)) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 90, in try_import_mod >> magic, timestamp, stream.readall()) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 499, in load_compiled_module >> code_w.exec_code(space, w_dic, w_dic) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in >> exec_code >> return frame.run() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in >> run >> return self.execute_frame() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, >> in execute_frame >> executioncontext) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, >> in dispatch >> next_instr = self.handle_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, >> in handle_bytecode >> next_instr = self.dispatch_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, >> in dispatch_bytecode >> res = getattr(self, methodname)(oparg, next_instr) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 760, >> in IMPORT_NAME >> w_locals, w_fromlist) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line >> 712, in call_function >> return w_func.funccall(*args_w) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 62, >> in funccall >> args_w[1], args_w[2], args_w[3]) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663, >> in fastcall_4 >> w_result = self.fastfunc_4(space, w1, w2, w3, w4) >> File "", line 3, in >> fastfunc_importhook_4 >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 180, in importhook >> w_mod = absolute_import(space, modulename, 0, w_fromlist, >> tentative=0) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 192, in absolute_import >> w_fromlist, tentative) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 211, in _absolute_import >> tentative=tentative) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 272, in load_part >> w(partname)) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 90, in try_import_mod >> magic, timestamp, stream.readall()) >> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", >> line 499, in load_compiled_module >> code_w.exec_code(space, w_dic, w_dic) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in >> exec_code >> return frame.run() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in >> run >> return self.execute_frame() >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, >> in execute_frame >> executioncontext) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, >> in dispatch >> next_instr = self.handle_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, >> in handle_bytecode >> next_instr = self.dispatch_bytecode(co_code, next_instr, ec) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, >> in dispatch_bytecode >> res = getattr(self, methodname)(oparg, next_instr) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 769, >> in IMPORT_STAR >> import_all_from(f.space, w_module, w_locals) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 839, >> in appcaller >> return space.call_args(w_func, args) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line >> 98, in call_args >> return w_obj.call_args(args) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37, >> in call_args >> return self.code.funcrun(self, args) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 483, >> in funcrun >> return BuiltinCode.funcrun_obj(self, func, None, args) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 491, >> in funcrun_obj >> w_result = activation._run(space, scope_w) >> File "", line 3, in >> _run_UWS_ObjSpace_W_Root_W_Root >> File >> "/var/tmp/pypy-svn/dist/pypy/_cache/pyopcode_4922ecf55c7db5d81d0c4a63f92bdd44.py", >> line 59, in import_all_from >> w_2 = space.getattr(w_module, gs___dict__) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669, >> in getattr >> return DescrOperation.getattr(self, w_obj, w_name) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line >> 134, in getattr >> return space._handle_getattribute(w_descr, w_obj, w_name) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line >> 140, in _handle_getattribute >> return space.get_and_call_function(w_descr, w_obj, w_name) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line >> 89, in get_and_call_function >> return descr.funccall(w_obj, *args_w) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 69, >> in funccall >> return self.call_args(Arguments(self.space, list(args_w))) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37, >> in call_args >> return self.code.funcrun(self, args) >> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 61, in >> funcrun >> return frame.run() >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 171, in >> run >> return self.space.wrap(result) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 475, >> in wrap >> items_w = [(self.wrap(k), self.wrap(v)) for (k, v) in x.iteritems()] >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 522, >> in wrap >> return fake_object(self, x) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 20, in >> fake_object >> ft = fake_type(x) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 58, in >> fake_type >> faked_type = really_build_fake_type(cpy_type) >> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 108, in >> really_build_fake_type >> assert cpy_type.__base__ is basestring >> AssertionError >> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> -- Nikita From cfbolz at gmx.de Wed Sep 3 22:01:24 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 03 Sep 2008 22:01:24 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <48BEE8E3.6010503@naumen.ru> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> <48BEE8E3.6010503@naumen.ru> Message-ID: <48BEED14.6030507@gmx.de> Vetoshkin Nikita wrote: > Playing with py.execnet failes too. Full traceback attached. It seems you are running py.execnet on py.py. Why? py.execnet is rather independent on PyPy and works fine on CPython. Cheers, Carl Friedrich From cfbolz at gmx.de Wed Sep 3 22:29:14 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 03 Sep 2008 22:29:14 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <48BEEE73.7010809@naumen.ru> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> <48BEE8E3.6010503@naumen.ru> <48BEED14.6030507@gmx.de> <48BEEE73.7010809@naumen.ru> Message-ID: <48BEF39A.9030406@gmx.de> Vetoshkin Nikita wrote: > Thanks! > I thought it should work on top of PyPy doesn't it? It probably should, but on py.py it is bound to be slow and I am not sure it works that on the remote site a py.py is started correctly. Anyway, if you work on py.execnet for helping the translation process then you don't need to run it on py.py anyway, but on CPython. Cheers, Carl Friedrich From calen.pennington at gmail.com Thu Sep 4 04:21:53 2008 From: calen.pennington at gmail.com (Calen Pennington) Date: Wed, 3 Sep 2008 22:21:53 -0400 Subject: [pypy-dev] Possible starting points Message-ID: <2d702fee0809031921i66eb06d0t962a7fc2fcca5ded@mail.gmail.com> Hi, everybody, I've been intrigued by pypy for a while =, but haven't really been able to find a good place to start contributing on my own, so I was wondering if anyone had suggestions. As far as what I'm interested in, I'd really like to see pypy be more accessible to mainstream python users, and to be closer to an easy drop in replacement for CPython (although it seems unlikely that it would become the standard python distribution any time soon). I know c extensions are one blocker to that, as well as feature completeness (although i don't really know the details of where pypy stands with regards to 2.5 (or 2.6/3.0, for that matter)). That said, I'm also interested in language design/implementation in general, so I wouldn't be averse to working on other parts of the interpreter/translator chain either. So, any thoughts as to what reasonable starting points would be? Thanks -Cale From holger at merlinux.eu Thu Sep 4 09:34:08 2008 From: holger at merlinux.eu (holger krekel) Date: Thu, 4 Sep 2008 09:34:08 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <48BEE8E3.6010503@naumen.ru> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> <48BEE8E3.6010503@naumen.ru> Message-ID: <20080904073408.GZ9619@trillke.net> Hi Vetoshkin, i think for translate.py you rather need to run py.execnet over CPython or through a translated pypy-c - both of which should work fine. I think you ran it through py.py which means starting a PyPy interpreter on top of CPython. Then running py.execnet.PopenGateway() would need to create another process that also runs through py.py - would be very slow - not sure it's worth trying to make it work. So i suggest you continue your experiments with plain CPython or the translated pypy-c. best & thanks, holger On Thu, Sep 04, 2008 at 01:43 +0600, Vetoshkin Nikita wrote: > Playing with py.execnet failes too. Full traceback attached. > > P.S. Seens like a hard start with my trials in PyPy =) > > Armin Rigo wrote: >> Hi, >> >> On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote: >>> Is there a direction, where I could digg to help? Something to begin with. >>> I don't have enough experience but I'd like to take part (even very >>> small) in this project. >> >> Sure. Do you understand the issue? It's related to the way a Unix >> process starts a new subprocess: os.fork() followed by os.execv() in the >> child process. The issue is that os.fork() fails if the parent process' >> total memory is more than half of your RAM+swap, because it tries to >> create a copy of the process, and the copy would also need more than >> half of your RAM+swap. (In reality, os.fork() doesn't copy the memory >> at all, but creates "shared pages" of memory so that memory pages are >> duplicated only when one of the two processes really modifies it; but >> still os.fork() has to mark all the memory as *potentially* used, and >> raises MemoryError if there isn't enough.) >> >> A solution might be to split translate.py in two processes: translate.py >> would start a subprocess when it starts, and then do all the work >> (consuming a lot of RAM); but when it needs to start new processes, e.g. >> call gcc, it would instead ask the subprocess to start the new >> processes. More precisely, the code in pypy.translator.tool.cbuild >> would stop using distutils directly, and instead ask the subprocess to >> use distutils. It looks even easy to do with the help of a >> py.execnet.PopenGateway() created when translate.py starts (see the py >> lib documentation at http://codespeak.net/py/). >> >> >> A bientot, >> >> Armin > > -- > Nikita > >>>> gw = py.execnet.PopenGateway() > faking > faking > Traceback (most recent call last): > File "./pypy/bin/py.py", line 152, in > sys.exit(main_(sys.argv)) > File "./pypy/bin/py.py", line 138, in main_ > con.interact(banner) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line > 133, in interact > code.InteractiveConsole.interact(self, banner) > File "/usr/lib64/python2.5/code.py", line 239, in interact > more = self.push(line) > File "/usr/lib64/python2.5/code.py", line 261, in push > more = self.runsource(source, self.filename) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line > 192, in runsource > main.run_toplevel(self.space, doit, verbose=self.verbose) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/main.py", line 103, in > run_toplevel > f() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line > 184, in doit > code.exec_code(self.space, self.w_globals, self.w_globals) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in > exec_code > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 695, > in LOAD_ATTR > w_value = f.space.getattr(w_obj, w_attributename) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669, > in getattr > return DescrOperation.getattr(self, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 134, in getattr > return space._handle_getattribute(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 147, in _handle_getattribute > return space.get_and_call_function(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 89, in get_and_call_function > return descr.funccall(w_obj, *args_w) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 56, > in funccall > return code.fastcall_2(self.space, self, args_w[0], args_w[1]) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in > fastcall_2 > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, > in CALL_FUNCTION > w_result = f.space.call_valuestack(w_function, nargs, f) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 736, in call_valuestack > return w_func.funccall_valuestack(nargs, frame) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81, > in funccall_valuestack > frame.peekvalue(0)) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in > fastcall_2 > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, > in CALL_FUNCTION > w_result = f.space.call_valuestack(w_function, nargs, f) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 736, in call_valuestack > return w_func.funccall_valuestack(nargs, frame) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81, > in funccall_valuestack > frame.peekvalue(0)) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in > fastcall_2 > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881, > in CALL_FUNCTION > w_result = f.space.call_valuestack(w_function, nargs, f) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 736, in call_valuestack > return w_func.funccall_valuestack(nargs, frame) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 88, > in funccall_valuestack > frame.peekvalue(0)) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663, > in fastcall_4 > w_result = self.fastfunc_4(space, w1, w2, w3, w4) > File "", line 3, in > fastfunc_importhook_4 > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 180, in importhook > w_mod = absolute_import(space, modulename, 0, w_fromlist, > tentative=0) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 192, in absolute_import > w_fromlist, tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 211, in _absolute_import > tentative=tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 272, in load_part > w(partname)) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 90, in try_import_mod > magic, timestamp, stream.readall()) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 499, in load_compiled_module > code_w.exec_code(space, w_dic, w_dic) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in > exec_code > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 760, > in IMPORT_NAME > w_locals, w_fromlist) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line > 712, in call_function > return w_func.funccall(*args_w) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 62, > in funccall > args_w[1], args_w[2], args_w[3]) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663, > in fastcall_4 > w_result = self.fastfunc_4(space, w1, w2, w3, w4) > File "", line 3, in > fastfunc_importhook_4 > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 180, in importhook > w_mod = absolute_import(space, modulename, 0, w_fromlist, > tentative=0) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 192, in absolute_import > w_fromlist, tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 211, in _absolute_import > tentative=tentative) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 272, in load_part > w(partname)) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 90, in try_import_mod > magic, timestamp, stream.readall()) > File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py", > line 499, in load_compiled_module > code_w.exec_code(space, w_dic, w_dic) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in > exec_code > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in > run > return self.execute_frame() > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117, > in execute_frame > executioncontext) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79, > in dispatch > next_instr = self.handle_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89, > in handle_bytecode > next_instr = self.dispatch_bytecode(co_code, next_instr, ec) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235, > in dispatch_bytecode > res = getattr(self, methodname)(oparg, next_instr) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 769, > in IMPORT_STAR > import_all_from(f.space, w_module, w_locals) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 839, > in appcaller > return space.call_args(w_func, args) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 98, in call_args > return w_obj.call_args(args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37, > in call_args > return self.code.funcrun(self, args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 483, > in funcrun > return BuiltinCode.funcrun_obj(self, func, None, args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 491, > in funcrun_obj > w_result = activation._run(space, scope_w) > File "", line 3, in > _run_UWS_ObjSpace_W_Root_W_Root > File > "/var/tmp/pypy-svn/dist/pypy/_cache/pyopcode_4922ecf55c7db5d81d0c4a63f92bdd44.py", > line 59, in import_all_from > w_2 = space.getattr(w_module, gs___dict__) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669, > in getattr > return DescrOperation.getattr(self, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 134, in getattr > return space._handle_getattribute(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 140, in _handle_getattribute > return space.get_and_call_function(w_descr, w_obj, w_name) > File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line > 89, in get_and_call_function > return descr.funccall(w_obj, *args_w) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 69, > in funccall > return self.call_args(Arguments(self.space, list(args_w))) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37, > in call_args > return self.code.funcrun(self, args) > File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 61, in > funcrun > return frame.run() > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 171, in > run > return self.space.wrap(result) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 475, > in wrap > items_w = [(self.wrap(k), self.wrap(v)) for (k, v) in x.iteritems()] > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 522, > in wrap > return fake_object(self, x) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 20, in > fake_object > ft = fake_type(x) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 58, in > fake_type > faked_type = really_build_fake_type(cpy_type) > File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 108, in > really_build_fake_type > assert cpy_type.__base__ is basestring > AssertionError > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From cfbolz at gmx.de Thu Sep 4 09:43:08 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 4 Sep 2008 09:43:08 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <20080904073408.GZ9619@trillke.net> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> <48BEE8E3.6010503@naumen.ru> <20080904073408.GZ9619@trillke.net> Message-ID: <348899050809040043l2cb67084i3dd5960a1e1d8c78@mail.gmail.com> Hi Holger, 2008/9/4 holger krekel : > i think for translate.py you rather need to run py.execnet over > CPython or through a translated pypy-c - both of which > should work fine. > > I think you ran it through py.py which means starting > a PyPy interpreter on top of CPython. Then running > py.execnet.PopenGateway() would need to create another > process that also runs through py.py - would be very > slow - not sure it's worth trying to make it work. > > So i suggest you continue your experiments with > plain CPython or the translated pypy-c. Well, suggesting to use pypy-c isn't really helpful, since his original problem was that PyPy doesn't translate :-). Cheers, Carl Friedrich From holger at merlinux.eu Thu Sep 4 09:54:47 2008 From: holger at merlinux.eu (holger krekel) Date: Thu, 4 Sep 2008 09:54:47 +0200 Subject: [pypy-dev] translate.py failes In-Reply-To: <348899050809040043l2cb67084i3dd5960a1e1d8c78@mail.gmail.com> References: <48B6C0B6.6060802@naumen.ru> <20080828153110.GA8912@code0.codespeak.net> <48B6D3AB.4050801@naumen.ru> <20080829155511.GA9583@code0.codespeak.net> <48BEE8E3.6010503@naumen.ru> <20080904073408.GZ9619@trillke.net> <348899050809040043l2cb67084i3dd5960a1e1d8c78@mail.gmail.com> Message-ID: <20080904075447.GA9619@trillke.net> On Thu, Sep 04, 2008 at 09:43 +0200, Carl Friedrich Bolz wrote: > Hi Holger, > > 2008/9/4 holger krekel : > > i think for translate.py you rather need to run py.execnet over > > CPython or through a translated pypy-c - both of which > > should work fine. > > > > I think you ran it through py.py which means starting > > a PyPy interpreter on top of CPython. Then running > > py.execnet.PopenGateway() would need to create another > > process that also runs through py.py - would be very > > slow - not sure it's worth trying to make it work. > > > > So i suggest you continue your experiments with > > plain CPython or the translated pypy-c. > > Well, suggesting to use pypy-c isn't really helpful, since his > original problem was that PyPy doesn't translate :-). good point - actually i just wanted to make the point that the translation tool chain (and py.execnet) should work fine on top of pypy-c, even if py.py fails. FWIW, I didn't see the other answers before i posted - using mutt over a high latency line i no fun, btw. holger From arigo at tunes.org Thu Sep 4 14:23:43 2008 From: arigo at tunes.org (Armin Rigo) Date: Thu, 4 Sep 2008 14:23:43 +0200 Subject: [pypy-dev] master thesis with pypy In-Reply-To: References: <5b0248170808260921j7c2e4bb4hb97980d7e10d8818@mail.gmail.com> <693bc9ab0809030244i10c50ae0t8bd1dc18da4a90d1@mail.gmail.com> Message-ID: <20080904122343.GA26189@code0.codespeak.net> Hi, On Wed, Sep 03, 2008 at 12:15:37PM -0600, Adam Olsen wrote: > Python-safethread's design of course includes most of this deep thinking. Jython includes another possible outcome of deep thinking about this. In general there are various possible approaches. A bientot, Armin. From cfbolz at gmx.de Thu Sep 4 14:45:44 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 04 Sep 2008 14:45:44 +0200 Subject: [pypy-dev] Possible starting points In-Reply-To: <2d702fee0809031921i66eb06d0t962a7fc2fcca5ded@mail.gmail.com> References: <2d702fee0809031921i66eb06d0t962a7fc2fcca5ded@mail.gmail.com> Message-ID: <48BFD878.6040509@gmx.de> Hi and welcome! Calen Pennington wrote: > I've been intrigued by pypy for a while =, but haven't really been > able to find a good place to start contributing on my own, so I was > wondering if anyone had suggestions. As far as what I'm interested in, > I'd really like to see pypy be more accessible to mainstream python > users, and to be closer to an easy drop in replacement for CPython > (although it seems unlikely that it would become the standard python > distribution any time soon). I know c extensions are one blocker to > that, as well as feature completeness (although i don't really know > the details of where pypy stands with regards to 2.5 (or 2.6/3.0, for > that matter)). 2.5 should be mostly implemented, there are still a couple of bugs left and the branch is still not merged, but we should get there soon. Something that would be quite useful is to port some of the remaining extension modules of CPython to be either pure Python or RPython (where Python is usually preferred). Some of the easy candidates are csv, maybe curses (using ctypes). [snip] Cheers, Carl Friedrich From holger at merlinux.eu Fri Sep 5 14:31:06 2008 From: holger at merlinux.eu (holger krekel) Date: Fri, 5 Sep 2008 14:31:06 +0200 Subject: [pypy-dev] next pypy sprint 5-13th october Message-ID: <20080905123105.GD9619@trillke.net> ======================================================== D??sseldorf PyPy sprint 5-13th October, 2008 ======================================================== The PyPy team is heading for a new sprint meetup, this time aiming at a 1.1 release and to work on integrating PyPy better with small devices. We are also open to other sprint topics! The sprint is going to take place at the Computer Science department of Heinrich-Heine Universit??t D??sseldorf (Germany). Infos in a nutshell: arrival day: Sunday, 5th October departure day: Monday, 13th October sprint days: 6-12th october with breaks Location: Seminar room of the computer science department, situated in the building 25.12 of the university campus, second floor. For travel instructions see http://stups.cs.uni-duesseldorf.de/anreise/esbahn.php ------------------------------ Accomodation ------------------------------ We may be able to fit some people into private flats and otherwise there are several hotels, one recommended one is the http://www.hotel-hillesheim.de/ where we are investigating regarding a cheaper group rate. Please register ASAP if you want to benefit. There are also some hotels that are used by visitors to the university and are quite close to the uni. None of us have been at them, so we have no experience of how good they are. 1. Hotel Bl??ttler http://www.hotel-blaettler.com/ 2. Hotel Mooren http://www.hotel-haus-mooren.de/ 3. Hotel Europa can't seem to find a website 4. Schloss Mickeln This one is the guest house of the university in a very beautiful castle. Supposed to be extremely nice, but also expensive. http://www.uni-duesseldorf.de/home/gaeste/schlossmickeln ----------------------- Funding your travel ----------------------- We have a limited budget for funding travels. If you want to participate but need travel funding please register and send a note to the pypy-sprint mailing list. We'll see what we can do to help. ------------ Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be sent to that list. Please register by adding yourself on the following list (via svn): http://codespeak.net/svn/pypy/extradoc/sprintinfo/october-2008/people.txt or announceyourself on the pypy-sprint mailing list if you do not yet have check-in rights: http://codespeak.net/mailman/listinfo/pypy-sprint --------------------------------------- Preparation (if you feel it is needed): --------------------------------------- * read the `getting-started`_ pages on http://codespeak.net/pypy * for inspiration, overview and technical status you are welcome to read `the technical reports available and other relevant documentation`_ * please direct any technical and/or development oriented questions to pypy-dev at codespeak.net and any sprint organizing/logistical questions to pypy-sprint at codespeak.net We are looking forward to meet you at the D??sseldorf PyPy sprint! The PyPy team .. See also .. .. _getting-started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`the technical reports available and other relevant documentation`: http://codespeak.net/pypy/dist/pypy/doc/index.html .. _`EuroPython schedule`: http://europython.org/timetable -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From fijall at gmail.com Fri Sep 5 15:25:19 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 5 Sep 2008 15:25:19 +0200 Subject: [pypy-dev] next pypy sprint 5-13th october In-Reply-To: <20080905123105.GD9619@trillke.net> References: <20080905123105.GD9619@trillke.net> Message-ID: <693bc9ab0809050625m2008a9e0i51ae4b57102f8b90@mail.gmail.com> > .. See also .. > .. _`EuroPython schedule`: http://europython.org/timetable ^^^^^^^^^^^^^^^ This may be less necessary than one might expect :) From holger at merlinux.eu Wed Sep 10 13:48:59 2008 From: holger at merlinux.eu (holger krekel) Date: Wed, 10 Sep 2008 13:48:59 +0200 Subject: [pypy-dev] build machines/doc directory Message-ID: <20080910114859.GF9619@trillke.net> Hi all, i've created a new doc directory to start documenting build machines, test, benchmark environments. svnlocation is https://codespeak.net/svn/pypy/build/doc I added an initial 'ssh_config' file which contains the hosts available to pypy developers. Please help complete. I also wrote a small script in svn/py/trunk/contrib/sysinfo.py to quickly collect remote system information. Combine the two and run "sysinfo.py -f ssh_config" and you'll get an overview of what we have (you can also use "sysinfo.py HOST1 HOST2 ..." explicitely) BIGDOG2 fqdn: bigdog2 BIGDOG2 sys.platform: linux2 BIGDOG2 sys.version_info: (2, 5, 2, 'final', 0) BIGDOG2 Memory: 8191804 Swap: 5116660 BIGDOG2 number of cpus: 4 BIGDOG2 cpu model AMD Phenom(tm) 9950 Quad-Core Processor WYVERN fqdn: wyvern.cs.uni-duesseldorf.de WYVERN sys.platform: linux2 WYVERN sys.version_info: (2, 4, 4, 'final', 0) WYVERN Memory: 5187968 Swap: 8032492 WYVERN number of cpus: 8 WYVERN cpu model Intel(R) Xeon(TM) CPU 3.20GHz COBRA fqdn: cobra.cs.uni-duesseldorf.de COBRA sys.platform: linux2 COBRA sys.version_info: (2, 4, 4, 'final', 0) COBRA Memory: 2059980 Swap: 5148136 COBRA number of cpus: 8 COBRA cpu model Intel(R) Xeon(TM) CPU 3.00GHz TUATARA fqdn: tuatara.cs.uni-duesseldorf.de TUATARA sys.platform: darwin TUATARA sys.version_info: (2, 3, 5, 'final', 0) T20 fqdn: jython01. T20 sys.platform: sunos5 T20 sys.version_info: (2, 4, 4, 'final', 0) Feel free to extend the sysinfo script - i guess having more information about windows/OSX/Solaris platforms and about available python versions / executables would be nice. Usually one can obtain this info through files or command line options so it shouldn't be hard to get this information automatically and add it to the sysinfo script. best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From jacob at openend.se Fri Sep 12 00:13:04 2008 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Fri, 12 Sep 2008 00:13:04 +0200 Subject: [pypy-dev] Switching to a distributed version control system Message-ID: <200809120013.04799.jacob@openend.se> I think that it would be a suitable point in time to switch to a new version control system right after the 1.1 release. The first question to ask is of course why we should switch at all. While the distributed version control systems allow a workflow where people maintan their own repositories and there is a designated role of integrator, I don't think we need such a workflow at this point in time. It may very well be the model to use in the future, when we have a production usable system, but right now this feature has no direct appeal. The compelling reason to switch is in my opinion the superior support for branches that the DVCS's provide. Creating a branch is a very cheap operation and merging it to the trunk or whatever branch is far superior to what SVN provides. I think this feature would change the way we are working and improve our productivity by a significant factor. There are a few other arguments in favour of a switch. People working through GPRS and off-line would have an easier time handling branches and updates. It would be possible to do sprints without a working internet access. There are, in my opinion, 3 viable choices of DVCS for PyPy: - git - hg (mercurial) - bzr I think they would all be an improvement over SVN and they all have their strengths and weaknesses. In favour of bzr and hg is the fact that they are written in Python, with core parts in C. Git is all C. Git currently requires a cygwin environment to run on Windows, hg and bzr appear to have native windows versions. Git is the fastest of the lot with hg in second place. Bzr is still a fair bit slower, though this is being worked on. Hg is really good at keeping the repositories small, with git in second place. Speaking for bzr is the fact that we have Michael Hudson in the PyPy community, and he seems to be a guru on bzr by now. Hg seems to be a little more tedious in its command set than the other two. Git used to be rather obscure, but is these days very straight forward to use. Git and bzr have very good visualization tools for showing the splitting and merging of branches. Git seems to be best at showing exactly what changed between 2 versions of the code (even 2 versions that are not on the same subtree). The strongest argument in favour of git seems to me to be the rebase feature, which allows one to make a branch for a new feature, work on the branch and then update the base of the branch to branch off at a later point in time. I haven't identified this feature in hg and bzr, but then I haven't read all the documentation in detail. The one feature of svn that we would miss is the inclusion of foreign version controlled trees, like we do with the pylib tree. We would have to do this in a different way than before, since none of the systems have this feature. I'm not sure it makes sense to have the close svn coupling between the projects any more, in any case. The effort of learning any of the systems seems to be quite insignificant. getting up to the level of svn is a matter of 15 minutes and learning the whole range of commands in a tool is not a big effort. There is of course the hooks that send mail and blurb in IRC, but all 3 systems seem to have at least as powerful hooks as svn. Jacob From micahel at gmail.com Fri Sep 12 00:55:36 2008 From: micahel at gmail.com (Michael Hudson) Date: Fri, 12 Sep 2008 10:55:36 +1200 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: <200809120013.04799.jacob@openend.se> References: <200809120013.04799.jacob@openend.se> Message-ID: 2008/9/12 Jacob Hall?n : > I think that it would be a suitable point in time to switch to a new version > control system right after the 1.1 release. > > The first question to ask is of course why we should switch at all. > > While the distributed version control systems allow a workflow where people > maintan their own repositories and there is a designated role of integrator, > I don't think we need such a workflow at this point in time. It may very well > be the model to use in the future, when we have a production usable system, > but right now this feature has no direct appeal. I would imagine there would still be a 'trunk' branch somewhere that would be where releases get made from and so on. > The compelling reason to switch is in my opinion the superior support for > branches that the DVCS's provide. Creating a branch is a very cheap operation > and merging it to the trunk or whatever branch is far superior to what SVN > provides. I think this feature would change the way we are working and > improve our productivity by a significant factor. > > There are a few other arguments in favour of a switch. People working through > GPRS and off-line would have an easier time handling branches and updates. It > would be possible to do sprints without a working internet access. > > There are, in my opinion, 3 viable choices of DVCS for PyPy: > > - git > > - hg (mercurial) > > - bzr I'd agree with this list, afaik, there's nothing else out there to take seriously at present. Darcs with it's cherry picking and lack of enforcing a linear history would be interesting, but I'd be a bit wary of using it seriously, tbh. > I think they would all be an improvement over SVN and they all have their > strengths and weaknesses. In favour of bzr and hg is the fact that they are > written in Python, with core parts in C. Git is all C. Git currently requires > a cygwin environment to run on Windows, hg and bzr appear to have native > windows versions. Git is the fastest of the lot with hg in second place. Bzr > is still a fair bit slower, though this is being worked on. I can say here that I work all day every day with bzr on a codebase (Launchpad) that's actually a pretty similar size to pypy -- 5000 files in 300 directories, a few tens of thousands of revisions -- and performance is mostly fine. > Hg is really good > at keeping the repositories small, with git in second place. It's worth mentioning how very bad svn is here: svn keeps a separate copy of every file in the working tree! I think that a copy of the tree containing the entire history with any of bzr, hg or git will only be a little larger than a svn checkout. > Speaking for bzr > is the fact that we have Michael Hudson in the PyPy community, and he seems > to be a guru on bzr by now. Right, I'd be more than happy to "consult" on a bzr migration. Another thing for bzr is that people can put branches up on Launchpad, even if the main branches lived on codespeak and required ssh access to commit too. I don't want to pimp bzr too hard, as I actually have very little experience with the other systems. > Hg seems to be a little more tedious in its > command set than the other two. Git used to be rather obscure, but is these > days very straight forward to use. Git and bzr have very good visualization > tools for showing the splitting and merging of branches. Git seems to be best > at showing exactly what changed between 2 versions of the code (even 2 > versions that are not on the same subtree). I'm not sure what you mean by this last point? > The strongest argument in favour of git seems to me to be the rebase feature, > which allows one to make a branch for a new feature, work on the branch and > then update the base of the branch to branch off at a later point in time. I > haven't identified this feature in hg and bzr, but then I haven't read all > the documentation in detail. This point is somewhat controversial in DVCS circles, and is viewed as a non-feature in the Bazaar camp at least. The issue is that if you make a branch, then publish it, then rebase it onto a later version of trunk, anyone who has based work on the earlier work is left high and dry. > The one feature of svn that we would miss is the inclusion of foreign version > controlled trees, like we do with the pylib tree. We would have to do this in > a different way than before, since none of the systems have this feature. I'm > not sure it makes sense to have the close svn coupling between the projects > any more, in any case. Right. bzr has some experimental support for 'nested trees', which would replace this functionality, but realistically it's not going to be ready for prime time any time soon. > The effort of learning any of the systems seems to be quite insignificant. > getting up to the level of svn is a matter of 15 minutes and learning the > whole range of commands in a tool is not a big effort. And I would say that bzr (and I suspect this applies to the others) is actually a lot nicer to use than svn. It seems to have a much more 'humane' interface, compared to svn's, which always makes me think of barbed wire and pointy metal things for some reason. > There is of course the hooks that send mail and blurb in IRC, but all 3 > systems seem to have at least as powerful hooks as svn. Right. There is also the issue of how to do a conversion. codespeak's svn repository is large and old and kind of scary in some quarters, and the variety of ways that merges have been done in pypy's past means that an import that preserves the merge information seems pretty unlikely for any tool. I'll try to get a bzr-svn import of pypy going again, though. Cheers, mwh From santagada at gmail.com Fri Sep 12 01:23:10 2008 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 11 Sep 2008 20:23:10 -0300 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: References: <200809120013.04799.jacob@openend.se> Message-ID: <818F2865-B24E-4ECB-947E-D1BDB37B8949@gmail.com> On Sep 11, 2008, at 7:55 PM, Michael Hudson wrote: > 2008/9/12 Jacob Hall?n : >> There are, in my opinion, 3 viable choices of DVCS for PyPy: >> >> - git >> >> - hg (mercurial) >> >> - bzr > > I'd agree with this list, afaik, there's nothing else out there to > take seriously at present. Darcs with it's cherry picking and lack of > enforcing a linear history would be interesting, but I'd be a bit wary > of using it seriously, tbh. I think we could remove git for the discussion, as its lack of windows support (and no mac packaging besides ports) makes it a very high barrier of entry for now and future windows developers. Isn't cfbolz also somewhat involved in bzr in the past? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080911/8c65a885/attachment.htm From sanxiyn at gmail.com Fri Sep 12 01:56:05 2008 From: sanxiyn at gmail.com (Seo Sanghyeon) Date: Fri, 12 Sep 2008 08:56:05 +0900 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: <200809120013.04799.jacob@openend.se> References: <200809120013.04799.jacob@openend.se> Message-ID: <5b0248170809111656r43b24122t83d8ecaf4f1655d4@mail.gmail.com> 2008/9/12 Jacob Hall?n : > The one feature of svn that we would miss is the inclusion of foreign version > controlled trees, like we do with the pylib tree. We would have to do this in > a different way than before, since none of the systems have this feature. I'm > not sure it makes sense to have the close svn coupling between the projects > any more, in any case. Mercurial has "forest" extension which allows nested Mercurial repositories. http://www.selenic.com/mercurial/wiki/index.cgi/ForestExtension It is an external extension, but it is used by high profile projects like Sun's OpenJDK. -- Seo Sanghyeon From ndbecker2 at gmail.com Fri Sep 12 17:14:57 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 12 Sep 2008 11:14:57 -0400 Subject: [pypy-dev] Switching to a distributed version control system References: <200809120013.04799.jacob@openend.se> <818F2865-B24E-4ECB-947E-D1BDB37B8949@gmail.com> Message-ID: hg is getting rebase, it was implemented under SOC, IIRC. From dirkjan at ochtman.nl Sat Sep 13 14:53:28 2008 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 13 Sep 2008 14:53:28 +0200 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: <200809120013.04799.jacob@openend.se> References: <200809120013.04799.jacob@openend.se> Message-ID: <48CBB7C8.1000601@ochtman.nl> Hi there, I'm a Mercurial committer who also listens in on #pypy some time. I remember we had a discussion about this subject with Armin and Michael a few weeks ago, where it was stated that the conversion process would likely be scary and that SVN was working alright for this project... Jacob Hall?n wrote: > to be a guru on bzr by now. Hg seems to be a little more tedious in its command set than the other two. Git used to be rather obscure, but is these I don't know why hg seems tedious. hg's commands have been designed to be rather a lot like SVN where this is possible. > days very straight forward to use. Git and bzr have very good visualization tools for showing the splitting and merging of branches. Git seems to be best at showing exactly what changed between 2 versions of the code (even 2 versions that are not on the same subtree). hg has also has fairly good visualization tools. We have a command-line graph in the form of the glog extension, and in the upcoming release we will feature a canvas-based graph in the included web interface: http://hg.xavamedia.nl/mercurial/crew/graph/ In addition, there are a TortoiseHG tool for Windows which also provides some integration/GUI on Linux (PyGTK-based). > The strongest argument in favour of git seems to me to be the rebase feature, which allows one to make a branch for a new feature, work on the branch and then update the base of the branch to branch off at a later point in time. I haven't identified this feature in hg and bzr, but then I haven't read all the documentation in detail. In the next release, hg will come with a rebase extension. hg also comes with the very powerful mq extension, which allows developers to keep versioned patch queues around, which can be detached from the repo and distributed separately (and which allows for easily updating, splitting and merging of in-development patches/changesets). > The one feature of svn that we would miss is the inclusion of foreign version controlled trees, like we do with the pylib tree. We would have to do this in a different way than before, since none of the systems have this feature. I'm not sure it makes sense to have the close svn coupling between the projects any more, in any case. Mercurial has the forest extension, which does this for OpenJDK, and NetBeans, I think. In fact, I think Mercurial is of the DVCS's the one with the most large projects: OpenJDK, NetBeans and Mozilla all use hg. > There is of course the hooks that send mail and blurb in IRC, but all 3 systems seem to have at least as powerful hooks as svn. In addition, hg has a very easy-to-use extension model, allowing Python developers to easily extend the functionality available in the tool. In fairness, Mercurial's support for branches is a little different than most people are used to; either people can use separate clones for branches (which use hardlinks, so they aren't too expensive), or changesets get committed to a branch name, but since history in hg is immutable (I think this is largely true for any DVCS, but hg can be a little more strict about it), this means branches cannot currently be deleted. Branches that have been merged back into another branch can be kept out of command output, though. Anyway, I'd like to help out with the conversion and infrastructure, should PyPy chose hg. Cheers, Dirkjan From arigo at tunes.org Sat Sep 13 16:29:36 2008 From: arigo at tunes.org (Armin Rigo) Date: Sat, 13 Sep 2008 16:29:36 +0200 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: <200809120013.04799.jacob@openend.se> References: <200809120013.04799.jacob@openend.se> Message-ID: <20080913142936.GA29969@code0.codespeak.net> Hi, On Fri, Sep 12, 2008 at 12:13:04AM +0200, Jacob Hall?n wrote: > I think that it would be a suitable point in time to switch to a new version > control system right after the 1.1 release. I haven't read this thread in detail, but I should quickly mention that subversion works quite well enough from my point of view. An obstacle to switching is the amount of integration efforts that was done, mainly on codespeak, between the repository and various other facilities. It is ok to discuss alternatives, but we need to keep in mind that someone will have to stand up and redo all that work on codespeak. A bientot, Armin. From exarkun at divmod.com Sat Sep 13 17:03:57 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Sat, 13 Sep 2008 11:03:57 -0400 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: <20080913142936.GA29969@code0.codespeak.net> Message-ID: <20080913150357.29191.554380514.divmod.quotient.25524@ohm> On Sat, 13 Sep 2008 16:29:36 +0200, Armin Rigo wrote: >Hi, > >On Fri, Sep 12, 2008 at 12:13:04AM +0200, Jacob Hall?n wrote: >> I think that it would be a suitable point in time to switch to a new version >> control system right after the 1.1 release. > >I haven't read this thread in detail, but I should quickly mention that >subversion works quite well enough from my point of view. > >An obstacle to switching is the amount of integration efforts that was >done, mainly on codespeak, between the repository and various other >facilities. It is ok to discuss alternatives, but we need to keep in >mind that someone will have to stand up and redo all that work on >codespeak. > In order to try to simplify discussions about changing tools, Twisted has tried to start building up a centralized list of the important features we use (still something of a work in progress): http://twistedmatrix.com/trac/wiki/WorkflowRequirements If someone writes up something similar for PyPy then it might be easier to talk about what tools can be replaced, what the consequences are, etc. Jean-Paul From sdouche at gmail.com Sat Sep 13 21:32:06 2008 From: sdouche at gmail.com (Sebastien Douche) Date: Sat, 13 Sep 2008 21:32:06 +0200 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: <200809120013.04799.jacob@openend.se> References: <200809120013.04799.jacob@openend.se> Message-ID: <5e1183fa0809131232h7ce094ebrbfa64d82e24c3dc4@mail.gmail.com> On Fri, Sep 12, 2008 at 00:13, Jacob Hall?n wrote: > There are, in my opinion, 3 viable choices of DVCS for PyPy: > > - git > > - hg (mercurial) > > - bzr Hi there :) > I think they would all be an improvement over SVN and they all have their > strengths and weaknesses. In favour of bzr and hg is the fact that they are > written in Python, with core parts in C. Git is all C. Git currently requires > a cygwin environment to run on Windows, hg and bzr appear to have native > windows versions. Git is the fastest of the lot with hg in second place. Bzr > is still a fair bit slower, though this is being worked on. Hg is really good > at keeping the repositories small, with git in second place. Excellent quick guide : http://www.infoq.com/articles/dvcs-guide > The strongest argument in favour of git seems to me to be the rebase feature, > which allows one to make a branch for a new feature, work on the branch and > then update the base of the branch to branch off at a later point in time. I > haven't identified this feature in hg and bzr, but then I haven't read all > the documentation in detail. Don't miss "extensions" (but these are officials, include in core): - Bisec http://www.selenic.com/mercurial/wiki/index.cgi/BisectExtension - Graph log (show grah of the tree, "ascii art") http://www.selenic.com/mercurial/wiki/index.cgi/GraphlogExtension - Hgk (same above, but with tcl/tk, copy of git's one) http://www.selenic.com/mercurial/wiki/index.cgi/HgkExtension - Mq (wonderfull Hg version of "quilt', must have!) http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension - Rebase (warning: available only to the next stable version) http://www.selenic.com/mercurial/wiki/index.cgi/RebaseExtension - Transplant ("transplant" patches from another branch or repositor) http://www.selenic.com/mercurial/wiki/index.cgi/TransplantExtension > The one feature of svn that we would miss is the inclusion of foreign version > controlled trees, like we do with the pylib tree. Do you means externals? The features provided by externals in Subversion are not currently available in Mercurial, NestedRepositories is a proposal to integrate this extension into Mercurial: http://www.selenic.com/mercurial/wiki/index.cgi/NestedRepositories -- Seb From micahel at gmail.com Sun Sep 14 13:07:16 2008 From: micahel at gmail.com (Michael Hudson) Date: Sun, 14 Sep 2008 23:07:16 +1200 Subject: [pypy-dev] please read my OSDC paper! Message-ID: Hi, As some of you know, I'm speaking about PyPy and OSDC in Sydney in December. As the deadline for submitting the associated paper is tomorrow, it seems like time to get some feedback :) You can see the paper (in reST format) here: http://codespeak.net/pypy/extradoc/talk/osdc2008/paper.txt If anyone has any comments, I'd be grateful for feedback. I'm going to be asleep for most of the time between now on the deadline, so please either commit comments into the file or reply to this mail and I'll take them into account before submitting tomorrow. Cheers, mwh From pedronis at openend.se Thu Sep 18 14:23:57 2008 From: pedronis at openend.se (Samuele Pedroni) Date: Thu, 18 Sep 2008 14:23:57 +0200 Subject: [pypy-dev] work on testing infrastructure using py.lib trunk and buildbot In-Reply-To: <48BEEAE5.1050606@openend.se> References: <48B51D17.9020201@openend.se> <48BEEAE5.1050606@openend.se> Message-ID: <48D2485D.2090607@openend.se> Hi all, Samuele Pedroni wrote: > I have checked in a brief summary of the meeting as: > > http://codespeak.net/svn/pypy/extradoc/planning/1.1/testing_infrastructure.txt > > some work has been started to implement the strategy described in the summary to improve our testing infrastructure and consolidate it using buildbot for all our testing needs. The experimental buildbot master setup we are playing with lives in: http://codespeak.net/svn/pypy/build/bot2/ this setup is going to be used for a buildmaster running on wyvern. As described in the summary the main idea is to use the new event model on py.lib trunk to produce structured logfiles that can be used from the buildbot master to produce consolidated test result pages. Code doing that lives at: - http://codespeak.net/svn/py/trunk/contrib/filelog/ The integration between pypy and py.lib trunk py.test has been worked on in: - http://codespeak.net/svn/pypy/branch/pypy-pytrunk/ (this points through an external to the filelog code for it to be readily available) so far only the major issues have been tackled and we are still using the compatibility layer present in the py.test, medium term we want to switch away from using the deprecated methods. On the buildbot side: - there's the start of a status page that looks like the current one we use (the htmlconftest based one) but can merge results coming from multiple builders - and definitions of simple builds to run some of pypy own tests I have checked in a TODO file with what is likely needed for people interested in helping or following what is going on: http://codespeak.net/svn/pypy/build/bot2/TODO Samuele. From ondrej at certik.cz Fri Sep 19 16:32:38 2008 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 19 Sep 2008 16:32:38 +0200 Subject: [pypy-dev] Switching to a distributed version control system In-Reply-To: <5e1183fa0809131232h7ce094ebrbfa64d82e24c3dc4@mail.gmail.com> References: <200809120013.04799.jacob@openend.se> <5e1183fa0809131232h7ce094ebrbfa64d82e24c3dc4@mail.gmail.com> Message-ID: <85b5c3130809190732y1d80a11s3ab1d893650a704e@mail.gmail.com> On Sat, Sep 13, 2008 at 9:32 PM, Sebastien Douche wrote: > On Fri, Sep 12, 2008 at 00:13, Jacob Hall?n wrote: >> There are, in my opinion, 3 viable choices of DVCS for PyPy: >> >> - git >> >> - hg (mercurial) >> >> - bzr > > Hi there :) > >> I think they would all be an improvement over SVN and they all have their >> strengths and weaknesses. In favour of bzr and hg is the fact that they are >> written in Python, with core parts in C. Git is all C. Git currently requires >> a cygwin environment to run on Windows, hg and bzr appear to have native >> windows versions. Git is the fastest of the lot with hg in second place. Bzr >> is still a fair bit slower, though this is being worked on. Hg is really good >> at keeping the repositories small, with git in second place. > > Excellent quick guide : http://www.infoq.com/articles/dvcs-guide > >> The strongest argument in favour of git seems to me to be the rebase feature, >> which allows one to make a branch for a new feature, work on the branch and >> then update the base of the branch to branch off at a later point in time. I >> haven't identified this feature in hg and bzr, but then I haven't read all >> the documentation in detail. > > Don't miss "extensions" (but these are officials, include in core): > > - Bisec > http://www.selenic.com/mercurial/wiki/index.cgi/BisectExtension > > - Graph log (show grah of the tree, "ascii art") > http://www.selenic.com/mercurial/wiki/index.cgi/GraphlogExtension > > - Hgk (same above, but with tcl/tk, copy of git's one) > http://www.selenic.com/mercurial/wiki/index.cgi/HgkExtension > > - Mq (wonderfull Hg version of "quilt', must have!) > http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension > > - Rebase (warning: available only to the next stable version) > http://www.selenic.com/mercurial/wiki/index.cgi/RebaseExtension > > - Transplant ("transplant" patches from another branch or repositor) > http://www.selenic.com/mercurial/wiki/index.cgi/TransplantExtension > >> The one feature of svn that we would miss is the inclusion of foreign version >> controlled trees, like we do with the pylib tree. > > Do you means externals? The features provided by externals in > Subversion are not currently available in Mercurial, > NestedRepositories is a proposal to integrate this extension into > Mercurial: > http://www.selenic.com/mercurial/wiki/index.cgi/NestedRepositories And here is a translation table to git if you later decide (like us:) that git is better for you: http://wiki.sympy.org/wiki/Git_hg_rosetta_stone Also git has the excellent "git svn", so you can use git with pypy right now and commit using svn (git svn dcommit). Ondrej From fijall at gmail.com Mon Sep 22 17:48:46 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 22 Sep 2008 17:48:46 +0200 Subject: [pypy-dev] 2.5 and 2.4 compatibility Message-ID: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com> I'm working on 2.5 compatibility branch right now. Due to incompatible changes, I suggest that we say we don't provide --pyversion option any more and simply compile 2.5 compatible interpreter. I don't really see benefits of providing 2.4 right now (after a brief discussion with Armin). If anyone objects, please do that now. Cheers, fijal