From arigo at tunes.org  Wed Jan  2 12:55:11 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 2 Jan 2008 12:55:11 +0100
Subject: [pypy-dev] llvm.gcroot
Message-ID: <20080102115511.GA24609@code0.codespeak.net>

Hi Eric, hi all,

A field report about the 'llvmgcroot' branch...

I've been playing with ways to remove the "shadow stack" of GC roots.
More precisely, with our own framework GCs, the problem is that in C
there is no portable way to walk the current stack and locate the local
variables containing pointers to GC-managed objects (let alone modify
these pointers safely, in case of a moving GC).  So what we do at the
moment is to push/pop all such pointers in a custom shadow stack around
each call, everywhere.

In LLVM there is a very recently implemented primitive called
llvm.gcroot() which allows us to declare local variables which contain
pointers to GC-managed objects.  Then with the help of a few pages of
custom C++ code to dynamically link with LLVM's native code backend
'llc', we can put extra data into the assembler that describes where
these local variables are and how big each function's stack frame is.
This is not platform-specific so far, in the sense that the same C++
code works on any platform.  There is still a bit of platform-specific
code, which is how exactly to interpret this data to walk the stack from
our GC's collect() methods.

For more info:
  C++ plugin for llc (see README.txt):
    http://codespeak.net/svn/user/arigo/hack/pypy-hack/stackrootwalker
  PyPy branch:
    http://codespeak.net/svn/pypy/branch/llvmgcroot
  with the platform-specific stack walking code:
    http://codespeak.net/svn/pypy/branch/llvmgcroot/pypy/rpython/memory/gctransform/llvmgcroot.py

This gives some modest speed-up over a plain llvm translation (which is
slower than a C translation, which is itself slower than a --llvm-via-c,
but the latter doesn't support llvm.gcroot(), obviously, as it goes
through C as well).  To activate this in the branch, use translate.py
--llvmgcroot.

Then I started playing with something similar for genc by parsing the .s
files produced by gcc.  Completely platform-specific, obviously.  To
active this in the branch, use translate.py --llvmgcroot --backend=c
(sorry if it's a bit confusing).  This uses the special 'asm' keyword of
gcc to generate markers that are found by
http://codespeak.net/svn/pypy/branch/llvmgcroot/pypy/translator/c/trackgcroot.py
and we get data added to the .s file in the same format as the custom
C++ GC plugin for LLVM.

Definitely an obscure hack, though it's kind of safe in the sense that
if gcc does too advanced optimizations it will not give obscure
segfaults, but clean crashes in trackgcroot.py itself.  It gives the
same modest speed-ups for pypy-c.

In both the C and the LLVM case, the presence of the gcroot markers
currently prevent some low-level optimizations by, respecitvely, llc and
gcc.  The reason is that the local variables that are marked as GC roots
cannot be simply "fast" variables that the compiler is free to put in
any register and move around as needed.  I'm not sure how to fix this
for llc, but for gcc I have the project to try to extend trackgcroot.py.

Finally, if we're talking about maximum performance, it might be
possible (given enough time and obscure hacking) to use trackgcroot.py
on the assembler files produced by gcc during a --llvm-via-c
translation...  by convincing LLVM to put in the C code things that look
like calls, but that are really macros that expand to the asm()
markers...


A bientot,

Armin.

From santagada at gmail.com  Thu Jan  3 22:34:42 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 3 Jan 2008 19:34:42 -0200
Subject: [pypy-dev] Fwd: [pypy-sprint] AVM2 / Tamarin backend at the sprint
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
Message-ID: <2DB03D38-8277-488D-9F37-CD5C150C63DF@gmail.com>

The right place to discuss this is pypy-dev

Begin forwarded message:

> From: Toby Watson <toby at thetobe.com>
> Date: 3 de janeiro de 2008 19h18min11s GMT-02:00
> To: pypy-sprint at codespeak.net
> Subject: [pypy-sprint] AVM2 / Tamarin backend at the sprint
>
> Hi Everyone,
>
> I have a PyPy related idea - one I'd like to explore at the upcoming
> sprint in Leysin.
>
> Briefly the idea is to target Adobe's AVM2/AVMplus, the virtual
> machine shipped in Flash and donated as Tamarin (to be the Mozilla/
> Firefox JS engine). This in turn would open a number of possibilities,
> not least would be seeing a >>>> prompt sitting in a Flash app on a
> web page.
>
> As I understand it from some of the PyPy CLI/JVM docs I have been
> reading it is possible certainly to generate an interpreter, but
> possibly also a compiler for any particular platform, in this case
> using the oobackend.
>
> Just quickly, because I know many people do not know about (or
> care! ;-) what is happening in Flash, the recent version 9 release
> introduced:
>
> * A new virtual machine (AVM2)
> * With JITter compilation
> * Support for prototype and class-based ECMAscript styles (should be
> suitable as Python target)
> * Dynamic bytecode generation _and loading_ is possible (should be
> handy)
> * ActionScript 3/ECMAScript has strong typing and other more mature
> language features
>
> FYI:
> * Bytecode is called Actionscript Byte Code (ABC)
> * Another language (HaXe) has already targeted this VM successfully.
>
> With such a 'Flython' interpreter I would hope in time to be able to
> bind to Adobe's native and library classes and hence by able to :
>
> * Compile swf files from Python (for Flash) source
> * Build Flex apps (Adobe's GUI framework)
> * Target AIR (Flash desktop apps)
>
> About myself, my interests:
>
> 30-something developer of many, many interactive systems using a lot
> of different technology from web Python/PHP, through ActionScript to
> OpenGL/C++, C#, etc. I have written many small parts of compilers and
> language transformers, but never a whole compiler. I have experience
> with Agile development environments and Test-driven development.
>
> I've started digging around using Adobe's flex compiler, the open-
> source Tamarin engine, plus my own small swf decompiler and lots of
> public information to understand how the AVM2 VM works and Adobe's
> compiler infrastructure - separate compilation, linking, the bytecode
> etc.
>
> I am actually interested in developing new language/programming styles
> atop PyPy/Flython (if it did exist) and I like the Flash VM because it
> is web-borne and so widely distributed - essentially people could try
> out experiments in their browser without having to download/build
> anything.
>
> I don't know how much is possible in a week and of course much depends
> on me, but my goal would be to get a skeleton for the interpreter/
> translator and to sound out what all the issues are in targetting the
> AVM, binding to native classes and so on.
>
> I was considering writing another compiler specifically for this task,
> but after looking at Jython, IronPython, Boo and PyPy it became
> apparent that PyPy is quite mature now, may save a lot of time, and
> ultimately produce a more interesting result.
>
> I would hope that by adding another backend I could contribute to the
> fleshing-out of PyPy, help participate in the general maturing of the
> project and hopefully give the group a cool demo too.
>
> So...I hope to see some of you at the sprint and look forward to some
> interesting discussions!
>
> Kind regards,
> Toby


--
Leonardo Santagada




From cfbolz at gmx.de  Thu Jan  3 22:40:35 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 03 Jan 2008 22:40:35 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
Message-ID: <477D5653.1020007@gmx.de>

Hi Toby,

(ccing pypy-dev)

Toby Watson wrote:
> I have a PyPy related idea - one I'd like to explore at the upcoming  
> sprint in Leysin.
> 
> Briefly the idea is to target Adobe's AVM2/AVMplus, the virtual  
> machine shipped in Flash and donated as Tamarin (to be the Mozilla/ 
> Firefox JS engine). This in turn would open a number of possibilities,  
> not least would be seeing a >>>> prompt sitting in a Flash app on a  
> web page.
> 
> As I understand it from some of the PyPy CLI/JVM docs I have been  
> reading it is possible certainly to generate an interpreter, but  
> possibly also a compiler for any particular platform, in this case  
> using the oobackend.
> 
> Just quickly, because I know many people do not know about (or  
> care! ;-) what is happening in Flash, the recent version 9 release  
> introduced:
> 
> * A new virtual machine (AVM2)
> * With JITter compilation
> * Support for prototype and class-based ECMAscript styles (should be  
> suitable as Python target)
> * Dynamic bytecode generation _and loading_ is possible (should be  
> handy)
> * ActionScript 3/ECMAScript has strong typing and other more mature  
> language features
> 
> FYI:
> * Bytecode is called Actionscript Byte Code (ABC)
> * Another language (HaXe) has already targeted this VM successfully.
> 
> With such a 'Flython' interpreter I would hope in time to be able to  
> bind to Adobe's native and library classes and hence by able to :
> 
> * Compile swf files from Python (for Flash) source
> * Build Flex apps (Adobe's GUI framework)
> * Target AIR (Flash desktop apps)
> 
> About myself, my interests:
> 
> 30-something developer of many, many interactive systems using a lot  
> of different technology from web Python/PHP, through ActionScript to  
> OpenGL/C++, C#, etc. I have written many small parts of compilers and  
> language transformers, but never a whole compiler. I have experience  
> with Agile development environments and Test-driven development.
> 
> I've started digging around using Adobe's flex compiler, the open- 
> source Tamarin engine, plus my own small swf decompiler and lots of  
> public information to understand how the AVM2 VM works and Adobe's  
> compiler infrastructure - separate compilation, linking, the bytecode  
> etc.
> 
> I am actually interested in developing new language/programming styles  
> atop PyPy/Flython (if it did exist) and I like the Flash VM because it  
> is web-borne and so widely distributed - essentially people could try  
> out experiments in their browser without having to download/build  
> anything.
> 
> I don't know how much is possible in a week and of course much depends  
> on me, but my goal would be to get a skeleton for the interpreter/ 
> translator and to sound out what all the issues are in targetting the  
> AVM, binding to native classes and so on.
> 
> I was considering writing another compiler specifically for this task,  
> but after looking at Jython, IronPython, Boo and PyPy it became  
> apparent that PyPy is quite mature now, may save a lot of time, and  
> ultimately produce a more interesting result.
> 
> I would hope that by adding another backend I could contribute to the  
> fleshing-out of PyPy, help participate in the general maturing of the  
> project and hopefully give the group a cool demo too.
> 
> So...I hope to see some of you at the sprint and look forward to some  
> interesting discussions!

I think this task is probably sprint-sized and relevant to PyPy. The 
question is whether some of the core-devs would like to work on it, so 
you can pair with that person.

Then there are technical questions too, for example whether the tamarin 
VM is good enough to actually run all the full PyPy Python interpreter 
on top of it.

Cheers,

Carl Friedrich

From santagada at gmail.com  Thu Jan  3 22:54:18 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Thu, 3 Jan 2008 19:54:18 -0200
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D5653.1020007@gmx.de>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
Message-ID: <F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>


>> Flash, the recent version 9 release
>> introduced:
>>
>> * A new virtual machine (AVM2)
>> * With JITter compilation
>> * Support for prototype and class-based ECMAscript styles (should be
>> suitable as Python target)
>> * Dynamic bytecode generation _and loading_ is possible (should be
>> handy)
>> * ActionScript 3/ECMAScript has strong typing and other more mature
>> language features

Also the latest version does a form of cache of libs, so in the future  
you go to a site with a pypy interpreter and then go to another one  
you download pypy only once if the library was corretly built.

>> FYI:
>> * Bytecode is called Actionscript Byte Code (ABC)
>> * Another language (HaXe) has already targeted this VM successfully.

Is there any assembler for ABC? maybe haxe has one (do they target  
avm2? I always thought it was only the old flash vm).

>> With such a 'Flython' interpreter I would hope in time to be able to
>> bind to Adobe's native and library classes and hence by able to :
>>
>> * Compile swf files from Python (for Flash) source
>> * Build Flex apps (Adobe's GUI framework)
>> * Target AIR (Flash desktop apps)
>>


I think this is a good idea, probably avm2 can run at least RPython  
and maybe this could be the idea, just like the javascript backend.  
This could be quite easier than supporting full python and pypy  
interpreter.

(I don't know if you have read any docs on the pypy site, but RPyton  
is the restricted python in wich the pypy interpreter is made on,  
suporting only it on avm2 would make programing for flex fun).

[]'s
--
Leonardo Santagada



From toby at thetobe.com  Thu Jan  3 23:07:29 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 22:07:29 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D5653.1020007@gmx.de>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
Message-ID: <312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>

>> [much snippage of intro to Tamarin/AVM2]

> I think this task is probably sprint-sized and relevant to PyPy. The  
> question is whether some of the core-devs would like to work on it,  
> so you can pair with that person.
>
> Then there are technical questions too, for example whether the  
> tamarin VM is good enough to actually run all the full PyPy Python  
> interpreter on top of it.
>
> Cheers,
>
> Carl Friedrich

Hi,

What do you think may be the tripping points?

- some datatype support problems? surely this is mostly a case of  
emulating the correct types modulo gross problems with the 'native'  
AVM types.

- speed. The new VM is a lot faster than the old one and has a JITer.  
I know that this has a go at all non-constructor methods as classes  
are loaded and can be very fast. AS3 is now running things like the  
Papervision3D library which is pretty CPU intensive, though I admit  
making particular type of requirements on the VM.

- some piece of machinery like the bytecode verifier getting in the  
way? AFAIK this just does relatively simple overflow checks and the  
like.

- I've also seen small schemes/LISPs running on the AVM so there's  
some confidence there.
Infact I've generated my own images using a basic stack machine  
interpreter (a bit like the PyPy JIT test toy). Additionally I've  
generated code at runtime, which is then dynamically loaded, bound to  
and executed to make the same images.

With regard to support I was mainly not expecting any unless this  
effort started to show some progress on my part and interest from  
others. However I expected to benefit from any intro talks (I think  
some were muted on 13/14?) and talking to other developers who have  
written backends. Not to mention general environment of  
industriousness ;-) Of course I'd be delighted if anyone else did  
become interested in it!

cheers,
Toby

From cfbolz at gmx.de  Thu Jan  3 23:11:42 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 03 Jan 2008 23:11:42 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
	<312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>
Message-ID: <477D5D9E.9060505@gmx.de>

Toby Watson wrote:
>>> [much snippage of intro to Tamarin/AVM2]
> 
>> I think this task is probably sprint-sized and relevant to PyPy. The 
>> question is whether some of the core-devs would like to work on it, so 
>> you can pair with that person.
>>
>> Then there are technical questions too, for example whether the 
>> tamarin VM is good enough to actually run all the full PyPy Python 
>> interpreter on top of it.
>>
>> Cheers,
>>
>> Carl Friedrich
> 
> Hi,
> 
> What do you think may be the tripping points?
> 
> - some datatype support problems? surely this is mostly a case of 
> emulating the correct types modulo gross problems with the 'native' AVM 
> types.
> 
> - speed. The new VM is a lot faster than the old one and has a JITer. I 
> know that this has a go at all non-constructor methods as classes are 
> loaded and can be very fast. AS3 is now running things like the 
> Papervision3D library which is pretty CPU intensive, though I admit 
> making particular type of requirements on the VM.

My gut feeling is the pure size of our Python interpreter, but maybe I 
am pessimistic. I think in terms of features we should be fine.

> - some piece of machinery like the bytecode verifier getting in the way? 
> AFAIK this just does relatively simple overflow checks and the like.
> 
> - I've also seen small schemes/LISPs running on the AVM so there's some 
> confidence there.
> Infact I've generated my own images using a basic stack machine 
> interpreter (a bit like the PyPy JIT test toy). Additionally I've 
> generated code at runtime, which is then dynamically loaded, bound to 
> and executed to make the same images.
> 
> With regard to support I was mainly not expecting any unless this effort 
> started to show some progress on my part and interest from others. 
> However I expected to benefit from any intro talks (I think some were 
> muted on 13/14?) and talking to other developers who have written 
> backends. Not to mention general environment of industriousness ;-) Of 
> course I'd be delighted if anyone else did become interested in it!

Well, the usual mode of programming at a sprint (and one of the main 
points of doing one) is to work on a problem in pairs. That's why it 
would be nice if we found somebody else who is interested in this. Of 
course the general environment of industriousness and being able to ask 
questions all the time already helps :-).

Cheers,

Carl Friedrich

From toby at thetobe.com  Thu Jan  3 23:42:26 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 22:42:26 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
Message-ID: <A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>

> Also the latest version does a form of cache of libs, so in the  
> future you go to a site with a pypy interpreter and then go to  
> another one you download pypy only once if the library was corretly  
> built.

Yup! Could be cool. Although my software often changes too much for  
caching to help ;-)

>>> FYI:
>>> * Bytecode is called Actionscript Byte Code (ABC)
>>> * Another language (HaXe) has already targeted this VM successfully.
>
> Is there any assembler for ABC? maybe haxe has one (do they target  
> avm2? I always thought it was only the old flash vm).

Certainly haXe has a generator library which I've used before, I've  
rewritten that lib in AS3 before so a Python port is not out of the  
question.

haXe does target AVM2: http://tinyurl.com/33t2ox . Certainly there's  
plenty of reference to it in the compiler source (O'Caml) that I've  
been looking at today.

Also the Adobe Flex 3 (will be OSS at some point) compiler can be  
coerced to emit a kind of Intermediate Representation, though I don't  
think it will assemble from it. Looks like this:

// ++StartProgram
// ++StartMethod $init
AllocTemp 1
LoadThis [1]
PushScope [0]
FindProperty print 3 { (package-internal) public test.as$0(private) }  
is_strict=true is_qualified=false is_attr=false [1]
PushString hello world [2]
CallProperty print 3 { (package-internal) public test.as$0(private) }  
1 is_qualified=false is_super=false is_attr=false [1]
CheckType * [1]
StoreRegister 1, * [0]
LoadRegister 1, * [1]
Return [0]
FreeTemp 1
// --FinishMethod $init
// --FinishProgram

> I think this is a good idea, probably avm2 can run at least RPython  
> and maybe this could be the idea, just like the javascript backend.  
> This could be quite easier than supporting full python and pypy  
> interpreter.
>
> (I don't know if you have read any docs on the pypy site, but RPyton  
> is the restricted python in wich the pypy interpreter is made on,  
> suporting only it on avm2 would make programing for flex fun).

I'm busy reading them now. I started with the recent some of the  
Google presentation and am now looking at http://codespeak.net/pypy/dist/pypy/doc/cli-backend.html 
  and Antonio Cuni's thesis on the CLI backend. I'll happily look at  
any other recommendations.

I know of the idea of RPython and am familar with similar in SmallTalk  
from their squeak translator. That said I haven't yet stumbled across  
any document saying what the Restrictions are...

oh here, "2.1   RPython Definition, not"

> suporting only it [RPython] on avm2 would make programing for flex  
> fun).


I think you've hit on a key point. This aspect for me would be about  
producing something that is fun...if it turns into something  
productive and worthy later on then so be it. I'm also optimistic that  
the VM could - in time - support some decent optimizations.

But baby steps first...

Toby




From toby at thetobe.com  Thu Jan  3 23:42:52 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 22:42:52 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D5D9E.9060505@gmx.de>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>
	<477D5653.1020007@gmx.de>
	<312F3771-1248-4B2C-9787-DD364A14A153@thetobe.com>
	<477D5D9E.9060505@gmx.de>
Message-ID: <88DAF02E-E783-4E40-A3B2-CF700B0BB83D@thetobe.com>

>> - speed. The new VM is a lot faster than the old one and has a  
>> JITer. I know that this has a go at all non-constructor methods as  
>> classes are loaded and can be very fast. AS3 is now running things  
>> like the Papervision3D library which is pretty CPU intensive,  
>> though I admit making particular type of requirements on the VM.
>
> My gut feeling is the pure size of our Python interpreter, but maybe  
> I am pessimistic. I think in terms of features we should be fine.

I did once have to change a few of the SWF defaults like  
max_stack_depth and is_this_script_dead_timeout? for a client project  
involving 100's of classes. But that was with the old VM ;-)

> Well, the usual mode of programming at a sprint (and one of the main  
> points of doing one) is to work on a problem in pairs. That's why it  
> would be nice if we found somebody else who is interested in this.  
> Of course the general environment of industriousness and being able  
> to ask questions all the time already helps :-).

Agreed that pairing is often super-productive.

Toby

From toby at thetobe.com  Fri Jan  4 00:12:01 2008
From: toby at thetobe.com (Toby Watson)
Date: Thu, 3 Jan 2008 23:12:01 +0000
Subject: [pypy-dev] AVM2 / Tamarin backend at the sprint
Message-ID: <21849469-E463-4860-9462-394787EDC786@thetobe.com>

Since there seems to be a little discussion here, may I ask:

What would be a useful code / piece of understanding to bring along?

Things I've been thinking about:

* A Python lib for generating ABC byte code, ABC byte code files (to  
run on Tamarin) and SWFs (for Flash).

* An understanding of how class linking / loading environment works in  
Tamarin / AVM2.

* ??? What else to figure out on the target side ???

I'm thinking that I want to generate ABC and produce ABC files. They  
are executable by the avmplus test runtime which is part of Tamarin.  
There is also an Actionscript 3 unit test framework, so some  
arrangement of these could be used to run a test suite.

That said I'd like to go further to wrappering the ABC bytecode in a  
SWF because I think it will have a lot more impact / is cuter at this  
time to be able to see something in a browser.

For anyone who's curious:

AVM2 overview:
www.adobe.com/devnet/actionscript/ articles/avm2overview.pdf

hxasm - assembler library (with example)
http://haxe.org/hxasm

ABC bytecodes here: (referenced from above)
http://www.anotherbigidea.com/javaswf/avm2/AVM2Instructions.html

cheers,
Toby

From anto.cuni at gmail.com  Fri Jan  4 00:20:04 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 04 Jan 2008 00:20:04 +0100
Subject: [pypy-dev] JVM integration
In-Reply-To: <9c0bb8a00712310914k17265838w26666b09b29fce0d@mail.gmail.com>
References: <9c0bb8a00712310914k17265838w26666b09b29fce0d@mail.gmail.com>
Message-ID: <477D6DA4.6040308@gmail.com>

Paul deGrandis wrote:
> Anto, Niko, and All,

HI Paul,

> I have begun working on PyPy again and am starting on jvm integration.
>  After talking briefly with Carl on IRC, I looked at the code in CLI.
> 
> I have a few questions:
> It appears as though the general approach is to identify entry points
> into CLI code (via dlls), use reflection, and make calls.  This is
> also later used to make (r)python modules into dlls themselves
> (allowing for bidirectional integration).  Is this correct?  

I'm not sure to understand what you mean, so I briefly explain how 
things work for CLI:

   1. (interp-level) you can use .NET libraries from RPython; this means 
that in theory you can do in RPython everything you do in C# (modulo the 
fact that the support is not complete right now, but it could be in the 
future)

   2. (app-level) you can use .NET libraries from pypy-cli; to do this, 
you go through the clr module: part of such a module is written in 
RPython and contains calls to .NET reflection API, which are used to 
dynamically access .NET libraries from python

Obviously, point 2 depends on point 1 to work.

In the source code, point 1 is implemented by translator/cli/dotnet.py, 
while point 2 is implemented by module/clr/*.py.

In both cases, probably a lot of code can be shared between the CLI and 
JVM versions; so, the first thing to do would be to split dotnet.py into 
a backend-independent part (probably to be put in translator/oosupport/) 
and the CLI specific part, then write the JVM specific code. The same 
for the code in module/clr/.

 > Also, how
 > does the notion of PythonNet play into this (ie: where do you get the
 > CLR module from)?

Look again at the point (1) above; ok, we can use .NET libraries in 
RPython; consider this RPython snippet:

def main():
     System.Console.WriteLine("Hello World")

when we translate it trough gencli, we get a .NET exe whose bytecode 
contains a call to System.Console::WriteLine, and everything is fine.

On the other hand, one of the greatest features of RPython is that we 
can test RPython programs directly in CPython, without the need to 
translate them; as you probably already know, if you type e.g. py.py you 
run the PyPy interpreter on top of CPython, which is extremely good for 
testing.

Now, suppose you want to run the snippet above on top of CPython, i.e. 
*without* translating it; how can it work? CPython can't call 
System.Console.WriteLine, can it?

This is where PythonNet come to the action: it allows CPython to access 
.NET libraries; so, even if we can't run that snippet directly on 
CPython, we can run it on PythonNet (which is nothing else than a thin 
wrapper of CPython).

To be clearer, the only role that PythonNet plays is to allow testing 
without the need of a translation. If you look at 
translator/cli/test/test_dotnet.py, you can see the TestPythonnet class: 
this class runs all the tests in TestDotnetRtyping to ensure that every 
snippet behave the same when translated and when interpreted.

There are equivalent of PythonNet for JVM; I think the most promising is 
JPype, which Carl Friedrich tried some time ago.

> If this is not the current approach, is there a document that someone
> can point me to that talks about this in further detail?
> Lastly, as work for integration on the JVM begins, a larger goal is
> the generalize the approach.  Has any work been done in the CLI/CLR to
> generalize the approach so far?

I wrote that code with JVM in mind, so I think it wouldn't be too hard 
to split the CLR specific parts from the general purpose parts.


ciao Anto

From anto.cuni at gmail.com  Fri Jan  4 00:29:18 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 04 Jan 2008 00:29:18 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>	<477D5653.1020007@gmx.de>	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
	<A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
Message-ID: <477D6FCE.9060700@gmail.com>

Hi Toby,

Toby Watson wrote:

> I'm busy reading them now. I started with the recent some of the  
> Google presentation and am now looking at http://codespeak.net/pypy/dist/pypy/doc/cli-backend.html 
>   and Antonio Cuni's thesis on the CLI backend. I'll happily look at  
> any other recommendations.

I was just writing you a mail pointing to that documents, but I see you 
already found them :-).

Be warn that my thesis is a bit outdated, because a lot of things have 
been improved since then. In particular, we added a new backend for the 
JVM which shares most of the code with the CLI backend; the good news is 
that almost certainly such code can be reused also for a new tamarin 
backend, meaning that writing it would be much easier than what was 
writing gencli and genjvm.

If you want to look at the source, the backend independent code is 
located in translator/oosupport/; each of the files there contains 
classes that are subclassed by the code in translator/{cli,jvm}.

Feel free to ask questions either here or on #pypy, I'll be glad to answer.

ciao Anto

From toby at thetobe.com  Fri Jan  4 01:06:18 2008
From: toby at thetobe.com (Toby Watson)
Date: Fri, 4 Jan 2008 00:06:18 +0000
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <477D6FCE.9060700@gmail.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>	<477D5653.1020007@gmx.de>	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
	<A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
	<477D6FCE.9060700@gmail.com>
Message-ID: <CE83EE4C-0AE3-4E13-872C-CF6C37AE1F1A@thetobe.com>

Hi Antonio,

Thanks for the advice and pointers into the code.

Would you say this is still a fair assessment of the tasks that have  
to be done to target a new backend? : (pulled from PyPy[cli-backend])

	? map ootypesystem's types to CLI Common Type System's types;

	? map ootypesystem's low level operation to CLI instructions;

	? map Python exceptions to CLI exceptions;

	? write a code generator that translates a flow graph into a list of  
CLI instructions;

	? write a class generator that translates ootypesystem classes into  
CLI classes.

Was this pre or post the refactoring you describe?

cheers,
Toby


On 3 Jan 2008, at 23:29, Antonio Cuni wrote:

> Hi Toby,
>
> Toby Watson wrote:
>
>> I'm busy reading them now. I started with the recent some of the   
>> Google presentation and am now looking at http://codespeak.net/pypy/dist/pypy/doc/cli-backend.html 
>>    and Antonio Cuni's thesis on the CLI backend. I'll happily look  
>> at  any other recommendations.
>
> I was just writing you a mail pointing to that documents, but I see  
> you already found them :-).
>
> Be warn that my thesis is a bit outdated, because a lot of things  
> have been improved since then. In particular, we added a new backend  
> for the JVM which shares most of the code with the CLI backend; the  
> good news is that almost certainly such code can be reused also for  
> a new tamarin backend, meaning that writing it would be much easier  
> than what was writing gencli and genjvm.
>
> If you want to look at the source, the backend independent code is  
> located in translator/oosupport/; each of the files there contains  
> classes that are subclassed by the code in translator/{cli,jvm}.
>
> Feel free to ask questions either here or on #pypy, I'll be glad to  
> answer.
>
> ciao Anto


From drfarina at gmail.com  Fri Jan  4 10:46:40 2008
From: drfarina at gmail.com (Daniel Farina)
Date: Fri, 04 Jan 2008 01:46:40 -0800
Subject: [pypy-dev] interrupts, preemption, threads
Message-ID: <1199440000.8858.9.camel@tenacity>

Hello list,

I wasn't able to locate any of the keywords ['interrupt',
'preemption', 'thread'] in the web documentation, so I come to
the list to ask what is roughly the status of these things in the
generated PyPy virtual machines on whatever applicable platforms.

What I am not asking about is PyPy's Python implemention of
threads, but rather support for those writing their own
interpreters.

Example scenarios:

- timeouts on arbitrary code fragments (conflated by the fact
that non-terminating interrupt-disabling code will allow code to
ignore the timeout, but I digress)

- writing a preemptive scheduler of closures

- making use of two processors 

- triggering garbage collections

fdr



From arigo at tunes.org  Fri Jan  4 12:35:50 2008
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 4 Jan 2008 12:35:50 +0100
Subject: [pypy-dev] interrupts, preemption, threads
In-Reply-To: <1199440000.8858.9.camel@tenacity>
References: <1199440000.8858.9.camel@tenacity>
Message-ID: <20080104113550.GA26548@code0.codespeak.net>

Hi Daniel,

On Fri, Jan 04, 2008 at 01:46:40AM -0800, Daniel Farina wrote:
> I wasn't able to locate any of the keywords ['interrupt',
> 'preemption', 'thread'] in the web documentation, so I come to
> the list to ask what is roughly the status of these things in the
> generated PyPy virtual machines on whatever applicable platforms.

We only have limited support for threads at the moment.  I think we have
no specific support for high-level backends (JVM, CLI).  On the
low-level backends (C, LLVM), you can use thread primitives to start
threads and use locks as you would do in C.  In addition there is
support in the translator for automatically inserting custom code before
and after every external function call.  We use this in PyPy's Python
interpreter: it uses a global interpreter lock (GIL), which is released
and reacquired around each external function call.  (This is similar to
CPython but we don't have to put the release/reacquire code everywhere
by hand.)

> - timeouts on arbitrary code fragments
> - writing a preemptive scheduler of closures
> - making use of two processors 
> - triggering garbage collections

We have no specific support for all this.  Multiple threads will run on
multiple processors just like in C.  Timeouts can be done in a
platform-specific way with signals.  For the rest, our custom garbage
collectors are not thread-safe, at least yet; only the Boehm collector
is, but its performance is not very good in multithreaded programs.


A bientot,

Armin.

From drfarina at gmail.com  Fri Jan  4 13:03:55 2008
From: drfarina at gmail.com (Daniel Farina)
Date: Fri, 04 Jan 2008 04:03:55 -0800
Subject: [pypy-dev] interrupts, preemption, threads
In-Reply-To: <20080104113550.GA26548@code0.codespeak.net>
References: <1199440000.8858.9.camel@tenacity>
	<20080104113550.GA26548@code0.codespeak.net>
Message-ID: <1199448235.8858.16.camel@tenacity>


On Fri, 2008-01-04 at 12:35 +0100, Armin Rigo wrote:
> Hi Daniel,
> 
> On Fri, Jan 04, 2008 at 01:46:40AM -0800, Daniel Farina wrote:
> > I wasn't able to locate any of the keywords ['interrupt',
> > 'preemption', 'thread'] in the web documentation, so I come to
> > the list to ask what is roughly the status of these things in the
> > generated PyPy virtual machines on whatever applicable platforms.
> 
> We only have limited support for threads at the moment.  I think we have
> no specific support for high-level backends (JVM, CLI).  On the
> low-level backends (C, LLVM), you can use thread primitives to start
> threads and use locks as you would do in C.

That seems like a pretty useful subset already...

>   In addition there is
> support in the translator for automatically inserting custom code before
> and after every external function call.  We use this in PyPy's Python
> interpreter: it uses a global interpreter lock (GIL), which is released
> and reacquired around each external function call.  (This is similar to
> CPython but we don't have to put the release/reacquire code everywhere
> by hand.)
> 

Nifty. Noted.

> > - timeouts on arbitrary code fragments
> > - writing a preemptive scheduler of closures
> > - making use of two processors 
> > - triggering garbage collections
> 
> We have no specific support for all this.  Multiple threads will run on
> multiple processors just like in C.  Timeouts can be done in a
> platform-specific way with signals.  For the rest, our custom garbage
> collectors are not thread-safe, at least yet; only the Boehm collector
> is, but its performance is not very good in multithreaded programs.
> 

Is there going to be some cross-platform support for this? 

As for examples, I imagine pypy/module/signal[/test] is the place to go,
correct?
 
Thanks,
fdr



From arigo at tunes.org  Fri Jan  4 15:24:34 2008
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 4 Jan 2008 15:24:34 +0100
Subject: [pypy-dev] interrupts, preemption, threads
In-Reply-To: <1199448235.8858.16.camel@tenacity>
References: <1199440000.8858.9.camel@tenacity>
	<20080104113550.GA26548@code0.codespeak.net>
	<1199448235.8858.16.camel@tenacity>
Message-ID: <20080104142434.GA7682@code0.codespeak.net>

Hi Daniel,

On Fri, Jan 04, 2008 at 04:03:55AM -0800, Daniel Farina wrote:
> > > - timeouts on arbitrary code fragments
> > > - writing a preemptive scheduler of closures
> > > - making use of two processors 
> > > - triggering garbage collections
> > 
> > We have no specific support for all this.  Multiple threads will run on
> > multiple processors just like in C.  Timeouts can be done in a
> > platform-specific way with signals.  For the rest, our custom garbage
> > collectors are not thread-safe, at least yet; only the Boehm collector
> > is, but its performance is not very good in multithreaded programs.
> > 
> 
> Is there going to be some cross-platform support for this? 

Thread-safe GCs are probably the biggest blocker.  To some extend we
will also need something for PyPy, although at first we can probably
just rely on the GIL to avoid concurrent GC calls.  So all the more
advanced stuff - including thread-safe or even truly concurrent GC - is
not really likely to be done any time soon by any of us, I imagine.  On
the other hand we're always open to contributions :-)

> As for examples, I imagine pypy/module/signal[/test] is the place to go,
> correct?

Yes, interp_signal.py contains all the necessary declaration, although
they are mixed up with gluing code specific to PyPy's Python
interpreter.  The first step would be to split this file and move the
general RPython-only code to pypy/rlib/rsignal.py, and keep only the
gluing in interp_signal.py; this is what we have already done with other
modules like _socket.


A bientot,

Armin.

From anto.cuni at gmail.com  Fri Jan  4 16:04:44 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 04 Jan 2008 16:04:44 +0100
Subject: [pypy-dev] [pypy-sprint] AVM2 / Tamarin backend at the sprint
In-Reply-To: <CE83EE4C-0AE3-4E13-872C-CF6C37AE1F1A@thetobe.com>
References: <1E748735-C3DC-4E9E-8604-4F097DBE489C@thetobe.com>	<477D5653.1020007@gmx.de>	<F4AB649C-4606-4B6A-A857-92BBADC67192@gmail.com>
	<A99EC969-4E59-46DF-B0D0-63D1D5D50C81@thetobe.com>
	<477D6FCE.9060700@gmail.com>
	<CE83EE4C-0AE3-4E13-872C-CF6C37AE1F1A@thetobe.com>
Message-ID: <477E4B0C.2020008@gmail.com>

Toby Watson wrote:
> Hi Antonio,

Hi Toby,

> Thanks for the advice and pointers into the code.
> 
> Would you say this is still a fair assessment of the tasks that have to 
> be done to target a new backend? : (pulled from PyPy[cli-backend])
> 
>     ? map ootypesystem's types to CLI Common Type System's types;
> 
>     ? map ootypesystem's low level operation to CLI instructions;
> 
>     ? map Python exceptions to CLI exceptions;
> 
>     ? write a code generator that translates a flow graph into a list of 
> CLI instructions;
> 
>     ? write a class generator that translates ootypesystem classes into 
> CLI classes.
> 
> Was this pre or post the refactoring you describe?

this list was written before the refactoring, but I think it's more or 
less still valid.

First of all, you need a strong understanding of both ootypesystem and 
the type system of the VM you want to targets: then you can think how to 
map the types; for CLI and JVM it was mostly straightforward, but maybe 
it wouldn't be so for AVM2, e.g. if it doesn't support classes in the 
same way ootypesystem expects.

After you have mapped types, mapping instructions would be just a matter 
of coding but it shouldn't be too hard.

The paragraph about exceptions is mostly historical; at least at the 
beginning, you shouldn't need to do anything special about exceptions as 
long as you keep the hierarchy of RPython exceptions separated from the 
hierarchy of the VM exceptions.

The cool thing is that the hardest point (code generator) have been 
already implemented in oosupport, and it's very easy to subclass it for 
targeting yet another VM.

If you want to have a closer look to how each point is implemented, here 
are some pointers in the code:

   * mapping types
       - for cli, it's cli/cts.py; the main entry-point is the function 
lltype_to_cts

       - for jvm, it's jvm/typesystem.py; however, the main entry-point 
is the function lltype_to_cts in jvm/database.py


   * mapping operations: see cli/opcodes.py and jvm/opcodes.py

   * code generator: the base class is in oosupport/function.py, 
subclassed in cli/function.py and jvm/nodes.py (class GraphFunction); 
the Function class knows how to deal with graphs, but it delegates the 
actual code generation to something specific to each backend; for cli, 
it's in cli/ilgenerator.py, for jvm it's the GraphFunction class itself.

   * class generator: cli/class_.py and the Class class in jvm/node.py

ciao Anto

From exarkun at divmod.com  Thu Jan 10 15:15:13 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 10 Jan 2008 09:15:13 -0500
Subject: [pypy-dev] CLI on Windows
In-Reply-To: 0
Message-ID: <20080110141513.31425.1333039041.divmod.quotient.3157@ohm>

Hello all,

I've played around a little bit with PyPy and Mono on Linux.  Now I'm
wondering what the requirements are to use/test the CLI features of PyPy
on Windows.

Can anyone give a list of the dependencies PyPy/CLI has on Windows?

Jean-Paul

From anto.cuni at gmail.com  Thu Jan 10 18:13:48 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 10 Jan 2008 18:13:48 +0100
Subject: [pypy-dev] CLI on Windows
In-Reply-To: <20080110141513.31425.1333039041.divmod.quotient.3157@ohm>
References: <20080110141513.31425.1333039041.divmod.quotient.3157@ohm>
Message-ID: <4786524C.90707@gmail.com>

Jean-Paul Calderone wrote:
> Hello all,
> 
> I've played around a little bit with PyPy and Mono on Linux.  Now I'm
> wondering what the requirements are to use/test the CLI features of PyPy
> on Windows.
> 
> Can anyone give a list of the dependencies PyPy/CLI has on Windows?

Hi Jean-Paul,

the only requirement is that .NET 2.0 is installed and that csc.exe and 
ilasm.exe are on the path. I've never tried with newer versions of .NET, 
it might work or not.

Moreover, some time ago I encountered one problem with microsoft ilasm 
which prevented pypy-cli to be compiled; I don't know if the problem 
it's still there. The workaround is to use mono's ilasm to assemble the 
bytecode, then run the resulting exe as usual; if pypy finds that mono 
is installed, it will automatically use mono's ilasm instead of 
microsoft's one.

ciao Anto

From toby at thetobe.com  Fri Jan 11 12:19:08 2008
From: toby at thetobe.com (Toby Watson)
Date: Fri, 11 Jan 2008 11:19:08 +0000
Subject: [pypy-dev] Blog / Parser
Message-ID: <310691A2-30B6-4E29-BDC3-238D7F8719F6@thetobe.com>

Hi,

I've just read the blog post, "Visualizing a Python tokenizer" and it  
reminded me of this:

"OMeta: an object oriented language for pattern matching"
http://www.cs.ucla.edu/~awarth/papers/dls07.pdf

OMeta is an extension and generalisation of the idea of PEGs*. It  
provides a nice way to describe a language both at the character level  
(tokens), the grammar itself and productions into the AST. Finally the  
grammars are extensible (possibly from within the language itself).

The implementation is discussed in "Packrat Parsers Can Support Left  
Recursion" and there is some discussion of the performance there. http://www.vpri.org/pdf/packrat_TR-2007-002.pdf

I wonder whether the same idea behind PyPy can be applied to the  
grammar. Write a program in some language (a python version of OMeta  
for instance) which is then transformed by the translator, compiler,  
or JIT into something that runs fast.

What could be nice about this is bringing the tokenising and parsing  
closer in spirit to the heart of PyPy, writing 'nicer' code, and  
providing a (I think tantalising) way to try new syntax going forward.

And there are things to play with on this page:
http://www.cs.ucla.edu/~awarth/ometa/ometa-js/

* Parsing Expression Grammar


With regard to railroad diagrams (I think that's what they're called):

There used to be a script that generated them - it's mentioned at the  
top of the python grammar file, and here http://www.python.org/search/hypermail/python-1994q3/0294.html

But I've seen discussion elsewhere that it has been lost :(

How about this? http://www.informatik.uni-freiburg.de/~thiemann/haskell/ebnf2ps/README


cheers,
Toby

From cfbolz at gmx.de  Fri Jan 11 22:22:13 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 11 Jan 2008 22:22:13 +0100
Subject: [pypy-dev] Blog / Parser
In-Reply-To: <310691A2-30B6-4E29-BDC3-238D7F8719F6@thetobe.com>
References: <310691A2-30B6-4E29-BDC3-238D7F8719F6@thetobe.com>
Message-ID: <4787DE05.30601@gmx.de>

Hi Toby,

Toby Watson wrote:
 > I've just read the blog post, "Visualizing a Python tokenizer" and it
 > reminded me of this:
 >
 > "OMeta: an object oriented language for pattern matching"
 > http://www.cs.ucla.edu/~awarth/papers/dls07.pdf
 >
 > OMeta is an extension and generalisation of the idea of PEGs*. It
 > provides a nice way to describe a language both at the character level
 > (tokens), the grammar itself and productions into the AST. Finally the
 > grammars are extensible (possibly from within the language itself).

Yes, I know of OMeta and this paper. It contains some nice things, but
in my opinion there is quite some hype around it too (and the paper is
at least missing a good "Prior work" section):

  - semantic-wise (apart from supporting left-recursion) their parsers
    are not more powerful than simple recursive-descent parsers with
    backtracking

  - it's not clear that the memoizing that packrat parsers do is a good
    idea for practical parsers in general. See this paper:
http://www.cs.mu.oz.au/research/mercury/information/papers.html#packrat-padl08

  - Prolog supported matching on non-stream data structures long ago
    _and_ is more powerful. It is also as efficient as packrat parsers if
    you use tabling (which is not a standard Prolog feature admittedly).

  - while I have not actually tried OMeta I somehow fear that debugging
    these programs if they behave unexpectedly is extremely annoying.
    Backtracking with first-match semantics can lead to unpleasant
    surprises and it's not always easy to see what is going on.

So what is left is some nice syntax, the inheritance support (which you
can rather easily get with a manually written recursive descent parser
too) and left recursion support.

 > The implementation is discussed in "Packrat Parsers Can Support Left
 > Recursion" and there is some discussion of the performance there. 
http://www.vpri.org/pdf/packrat_TR-2007-002.pdf

I have only glanced at it, I guess I should read it in more detail.
FWIW, the left-recursion support is basically exactly what a tabling
Prolog does, just less general.

 > I wonder whether the same idea behind PyPy can be applied to the
 > grammar. Write a program in some language (a python version of OMeta
 > for instance) which is then transformed by the translator, compiler,
 > or JIT into something that runs fast.

Maybe, yes. FWIW, we support simple packrat parsers, see here:

http://codespeak.net/pypy/dist/pypy/doc/rlib.html#parsing

It's far from OMeta, just a fairly boring parser generator. It contains
good support for regular expressions, though, which is where the diagram
from the blog post comes from.

I also experimented a bit with something OMeta-like in PyPy, you can
look at the tests here:

http://codespeak.net/svn/pypy/dist/pypy/rlib/parsing/test/test_pypackrat.py

It supports the left recursion and the composability. I am not sure it
is a good idea, though. The resulting code is RPython and can be
translated to mostly efficient C.

 > What could be nice about this is bringing the tokenising and parsing
 > closer in spirit to the heart of PyPy, writing 'nicer' code, and
 > providing a (I think tantalising) way to try new syntax going forward.

We don't really use any advanced stuff for our Python parser because we
want to be able to use the CPython's grammar files, which contain LL(1)
grammars anyway, so why bother with anything advanced? If someone wants
to support user-modifiable grammars or something like that he can
probably do so easily with PyPy. The PyPy core team is usually keeping
out of such "language-designy" topics (although I guess I would be
interested at least on the implementation-side of things if somebody
stepped forward to go in such a direction).

 >
 > And there are things to play with on this page:
 > http://www.cs.ucla.edu/~awarth/ometa/ometa-js/
 >
 > * Parsing Expression Grammar
 >
 >
 > With regard to railroad diagrams (I think that's what they're called):
 >
 > There used to be a script that generated them - it's mentioned at the
 > top of the python grammar file, and here 
http://www.python.org/search/hypermail/python-1994q3/0294.html
 >
 > But I've seen discussion elsewhere that it has been lost :(
 >
 > How about this? 
http://www.informatik.uni-freiburg.de/~thiemann/haskell/ebnf2ps/README

ebnf2ps seems to be using haskell that is no longer liked by modern 
haskell compilers. I don't know any haskell but whacking at the source a 
bit produced some sort of executable, haven't tried to use it yet.

Cheers,

Carl Friedrich

From michaelschneider at current.net  Sat Jan 12 21:12:17 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Sat, 12 Jan 2008 15:12:17 -0500
Subject: [pypy-dev] Performance Regression with SVN version 50525
Message-ID: <47891F21.7080100@current.net>

Hello All,


The pypy c translation time doubled for me with SVN revision 50525



(2:13:48 PM) *bigdog:* 50524:
(2:13:48 PM) *bigdog:* Timer] Timings:
(2:13:48 PM) *bigdog:* Timer] annotate - 318.9 s
(2:13:48 PM) *bigdog:* Timer] rtype_lltype - 174.2 s
(2:13:48 PM) *bigdog:* Timer] backendopt_lltype - 183.3 s
(2:13:48 PM) *bigdog:* Timer] stackcheckinsertion_lltype - 62.6 s
(2:13:48 PM) *bigdog:* Timer] database_c - 128.2 s
(2:13:48 PM) *bigdog:* Timer] source_c - 69.3 s
(2:13:48 PM) *bigdog:* Timer] compile_c - 0.1 s
(2:13:48 PM) *bigdog:* Timer] ========================================
(2:13:48 PM) *bigdog:* Timer] Total: - 936.6 s


(2:55:29 PM) *bigdog:* 50525 timings:
(2:55:32 PM) *bigdog:* [Timer] Timings:
(2:55:32 PM) *bigdog:* [Timer] annotate - 335.2 s
(2:55:32 PM) *bigdog:* [Timer] rtype_lltype - 174.3 s
(2:55:32 PM) *bigdog:* [Timer] backendopt_lltype - 183.5 s
(2:55:32 PM) *bigdog:* [Timer] stackcheckinsertion_lltype - 1066.0 s
(2:55:32 PM) *bigdog:* [Timer] database_c - 126.3 s
(2:55:32 PM) *bigdog:* [Timer] source_c - 70.4 s
(2:55:32 PM) *bigdog:* [Timer] compile_c - 0.1 s
(2:55:32 PM) *bigdog:* [Timer] =========================================
(2:55:32 PM) *bigdog:* [Timer] Total: - 1955.8 s



The translation spent inordinate amount of time in:
(2:37:52 PM) *bigdog:* 
..+*%+................................*%%..................................... 

(2:37:52 PM) *bigdog:* [backendopt:malloc] removed 3922 simple mallocs 
in total
(2:37:52 PM) *bigdog:* [backendopt:mergeifblocks] starting to merge if 
blocks
(2:37:52 PM) *bigdog:* 
..+*%+................................*%%..............
(2:37:52 PM) *bigdog:* [translation:info] inserting stack checks...
(2:37:52 PM) *bigdog:* ..
(2:37:52 PM) *bigdog:* [rtyper] -=- specialized 6 more blocks -=-



Translation memory use went from ~350 meg to ~1.3 gb  (memory use are 
not exact, I just looked at the perfomrnace monitor when runnning)

Hope this is helpful,
Mike

From michaelschneider at current.net  Sun Jan 13 16:26:12 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Sun, 13 Jan 2008 10:26:12 -0500
Subject: [pypy-dev] Performance Regression with SVN version 50525
In-Reply-To: <47891F21.7080100@current.net>
References: <47891F21.7080100@current.net>
Message-ID: <478A2D94.5070207@current.net>

Looks like Carls' fix took care of the issue:

(10:23:58 AM) *bigdog:* cfbolts: Translation timings for 50557
(10:24:01 AM) *bigdog:* [Timer] Timings:
(10:24:01 AM) *bigdog:* [Timer] annotate - 329.5 s
(10:24:01 AM) *bigdog:* [Timer] rtype_lltype - 172.7 s
(10:24:01 AM) *bigdog:* [Timer] backendopt_lltype - 182.5 s
(10:24:01 AM) *bigdog:* [Timer] stackcheckinsertion_lltype - 12.3 s
(10:24:01 AM) *bigdog:* [Timer] database_c - 124.8 s
(10:24:01 AM) *bigdog:* [Timer] source_c - 67.7 s
(10:24:01 AM) *bigdog:* [Timer] compile_c - 0.1 s
(10:24:01 AM) *bigdog:* [Timer] ========================================
(10:24:01 AM) *bigdog:* [Timer] Total: - 889.5 s


Very good work, amazing turnaround time.

Thanks,
Mike

Michael Schneider wrote:
> Hello All,
>
>
> The pypy c translation time doubled for me with SVN revision 50525
>
>
>
> (2:13:48 PM) *bigdog:* 50524:
> (2:13:48 PM) *bigdog:* Timer] Timings:
> (2:13:48 PM) *bigdog:* Timer] annotate - 318.9 s
> (2:13:48 PM) *bigdog:* Timer] rtype_lltype - 174.2 s
> (2:13:48 PM) *bigdog:* Timer] backendopt_lltype - 183.3 s
> (2:13:48 PM) *bigdog:* Timer] stackcheckinsertion_lltype - 62.6 s
> (2:13:48 PM) *bigdog:* Timer] database_c - 128.2 s
> (2:13:48 PM) *bigdog:* Timer] source_c - 69.3 s
> (2:13:48 PM) *bigdog:* Timer] compile_c - 0.1 s
> (2:13:48 PM) *bigdog:* Timer] ========================================
> (2:13:48 PM) *bigdog:* Timer] Total: - 936.6 s
>
>
> (2:55:29 PM) *bigdog:* 50525 timings:
> (2:55:32 PM) *bigdog:* [Timer] Timings:
> (2:55:32 PM) *bigdog:* [Timer] annotate - 335.2 s
> (2:55:32 PM) *bigdog:* [Timer] rtype_lltype - 174.3 s
> (2:55:32 PM) *bigdog:* [Timer] backendopt_lltype - 183.5 s
> (2:55:32 PM) *bigdog:* [Timer] stackcheckinsertion_lltype - 1066.0 s
> (2:55:32 PM) *bigdog:* [Timer] database_c - 126.3 s
> (2:55:32 PM) *bigdog:* [Timer] source_c - 70.4 s
> (2:55:32 PM) *bigdog:* [Timer] compile_c - 0.1 s
> (2:55:32 PM) *bigdog:* [Timer] =========================================
> (2:55:32 PM) *bigdog:* [Timer] Total: - 1955.8 s
>
>
>
> The translation spent inordinate amount of time in:
> (2:37:52 PM) *bigdog:* 
> ..+*%+................................*%%..................................... 
>
> (2:37:52 PM) *bigdog:* [backendopt:malloc] removed 3922 simple mallocs 
> in total
> (2:37:52 PM) *bigdog:* [backendopt:mergeifblocks] starting to merge if 
> blocks
> (2:37:52 PM) *bigdog:* 
> ..+*%+................................*%%..............
> (2:37:52 PM) *bigdog:* [translation:info] inserting stack checks...
> (2:37:52 PM) *bigdog:* ..
> (2:37:52 PM) *bigdog:* [rtyper] -=- specialized 6 more blocks -=-
>
>
>
> Translation memory use went from ~350 meg to ~1.3 gb  (memory use are 
> not exact, I just looked at the perfomrnace monitor when runnning)
>
> Hope this is helpful,
> Mike
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>   


From michaelschneider at current.net  Mon Jan 14 05:34:20 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Sun, 13 Jan 2008 23:34:20 -0500
Subject: [pypy-dev] windows C translation progress - complete with patches
 for 2/3 of the problems
In-Reply-To: <47891F21.7080100@current.net>
References: <47891F21.7080100@current.net>
Message-ID: <478AE64C.1080706@current.net>

Hello All,

Hope you are having fun at the sprint.

I have been working on getting the windows translation going again.

There are 3 issues, and I have fixes for 2 of them

1) translator/c/src/stack.h  includes a unix only include file without 
guards,  this is an easy fix.  This include is not used, simply delete 
this #include

#include <unistd.h>

2) translator/c/src/ll_strtod.h   apply this patch to add 
#include<winsock2.h> during a windows compile.  The patch at then end of 
this email will fix
the winsoc2 related compile errors


3) The generated code module_0.c  #includes  <sys/wait.h>   rumor has a 
fix is close for this, so I did not work this.

                                module_0.c
                                module_0.c(859) : fatal error C1083: 
Cannot open include file: 'sys/wait.h': No
                                such file or directory



Can some kind soul please review and commit these changes?


Thanks, and happy skiing/snowboarding/lounging in a nice place,
Mike

----------------------------------------------------------------------------------------------------------

m
#------------------------------------------ start 
patch-------------------------------------------
Index: ll_strtod.h

===================================================================

--- ll_strtod.h    (revision 50566)

+++ ll_strtod.h    (working copy)

@@ -1,6 +1,10 @@

 #ifndef LL_STRTOD_H
 #define LL_STRTOD_H
 
+#ifdef MS_WINDOWS
+#include <winsock2.h>
+#endif
+
 #include <locale.h>
 #include <ctype.h>
 #include <string.h>










From alexandre.fayolle at logilab.fr  Mon Jan 14 09:52:37 2008
From: alexandre.fayolle at logilab.fr (Alexandre Fayolle)
Date: Mon, 14 Jan 2008 09:52:37 +0100
Subject: [pypy-dev] pylib patch to enable mips and mipsel greenlets
Message-ID: <20080114085237.GD11580@logilab.fr>

Hi,

I've received a patch by Thiemo Seufer <ths at networkno.de> to port pylib
greenlets to mips and mipsel linux. 

It is available for examination at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=459520

-- 
Alexandre Fayolle                              LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Informatique scientifique:               http://www.logilab.fr/science
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20080114/deeba165/attachment.pgp 

From cfbolz at gmx.de  Mon Jan 14 13:33:11 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 14 Jan 2008 13:33:11 +0100
Subject: [pypy-dev] windows C translation progress - complete with
 patches for 2/3 of the problems
In-Reply-To: <478AE64C.1080706@current.net>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
Message-ID: <478B5687.9010705@gmx.de>

Hi Mike,

Michael Schneider wrote:
> Hope you are having fun at the sprint.

yes, apart from the usual head-wall interactions :-).

> I have been working on getting the windows translation going again.
> 
> There are 3 issues, and I have fixes for 2 of them
> 
> 1) translator/c/src/stack.h  includes a unix only include file without 
> guards,  this is an easy fix.  This include is not used, simply delete 
> this #include
> 
> #include <unistd.h>
> 
> 2) translator/c/src/ll_strtod.h   apply this patch to add 
> #include<winsock2.h> during a windows compile.  The patch at then end of 
> this email will fix
> the winsoc2 related compile errors

I committed both of these, thanks a lot!


> 3) The generated code module_0.c  #includes  <sys/wait.h>   rumor has a 
> fix is close for this, so I did not work this.
> 
>                                 module_0.c
>                                 module_0.c(859) : fatal error C1083: 
> Cannot open include file: 'sys/wait.h': No
>                                 such file or directory

According to Maciek this should be fixed as of revision 50588

Thanks again,

Carl Friedrich

From michaelschneider at current.net  Tue Jan 15 05:29:53 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Mon, 14 Jan 2008 23:29:53 -0500
Subject: [pypy-dev] windows pypy C translation status - 3 compile classes
 fixe - simple link issue remaining
In-Reply-To: <478B5687.9010705@gmx.de>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
	<478B5687.9010705@gmx.de>
Message-ID: <478C36C1.5040400@current.net>

Good News, things are progressing nicely.

1) the 3 classes are compile errors have been address, and the source 
files now compile (note: header cleanup may still occur)

2) The link is very close, the emmited link command includes a m.lib 
command which is an error on visual studio links..


I am sorry, I don't understand how the link line is emmited.  Can 
someone familiar with this please remove the m.lib from the generated 
link line for visual studio 2003?

Thanks
Mike


PS.  I will be happy to test the fix



---------------------------------------- compile and link line from 
simple Visual Studio Project 2003, that includes a sin(90.0) ------------

 
Command Lines
 
 
Creating temporary file "c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000005.rsp" with contents
[
/Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /D "_MBCS" /Gm /EHsc /RTC1 /MLd /Yu"stdafx.h" /Fp"Debug/simplemath.pch" /Fo"Debug/" /Fd"Debug/vc70.pdb" /W3 /c /Wp64 /ZI /TP
".\simplemath.cpp"
]
Creating command line "cl.exe @"c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000005.rsp" /nologo"
Creating temporary file "c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000006.rsp" with contents
[
/Od /D "WIN32" /D "_DEBUG" /D "_CONSOLE" /D "_MBCS" /Gm /EHsc /RTC1 /MLd /Yc"stdafx.h" /Fp"Debug/simplemath.pch" /Fo"Debug/" /Fd"Debug/vc70.pdb" /W3 /c /Wp64 /ZI /TP
".\stdafx.cpp"
]
Creating command line "cl.exe @"c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000006.rsp" /nologo"
Creating temporary file "c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000007.rsp" with contents
[
/OUT:"Debug/simplemath.exe" /INCREMENTAL /NOLOGO /DEBUG /PDB:"Debug/simplemath.pdb" /SUBSYSTEM:CONSOLE /MACHINE:X86  kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib
".\debug\stdafx.obj"
".\debug\simplemath.obj"
]
Creating command line "link.exe @"c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\RSP000007.rsp""
 
Output Window
 
 
Compiling...
stdafx.cpp
Compiling...
simplemath.cpp
Linking...
 
Results
 
 
Build log was saved at "file://c:\Documents and Settings\scmikes\My Documents\Visual Studio Projects\simplemath\simplemath\Debug\BuildLog.htm"
simplemath - 0 error(s), 0 warning(s)



-------------------------------------end simple visual studio project 
-----------------------------------------------------------------------------------------------



------------------------------------- current pypy compile and link line 
---------------------------------------------------------------------

.... compile other .obj's cut   \pypy-dist\pypy\translator\c 
/Tcmodule_2.c /Fomodule_2.obj [cbuild:execute] C:\Program 
Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox 
/MD /W3 /GX /DNDEBUG -Ic:\python25\include 
-IH:\pypy\pypy-dist\pypy\translator\c /Tcmodule_3.c /Fomodule_3.obj 
[cbuild:execute] C:\Program Files\Microsoft Visual Studio .NET 
2003\Vc7\bin\link.exe /DEBUG /nologo /INCREMENTAL:NO gc_pypy.lib m.lib 
h:\temp\usession-27\testing_1\testing_1.obj 
h:\temp\usession-27\testing_1\structimpl.obj 
h:\temp\usession-27\testing_1\nonfuncnodes.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_1.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_2.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_3.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_4.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_5.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_6.obj 
h:\temp\usession-27\testing_1\nonfuncnodes_7.obj 
h:\temp\usession-27\testing_1\implement.obj 
h:\temp\usession-27\testing_1\implement_1.obj 
h:\temp\usession-27\testing_1\implement_2.obj 
h:\temp\usession-27\testing_1\implement_3.obj 
h:\temp\usession-27\testing_1\implement_4.obj 
h:\temp\usession-27\testing_1\implement_5.obj 
h:\temp\usession-27\testing_1\implement_6.obj 
h:\temp\usession-27\testing_1\implement_7.obj 
h:\temp\usession-27\testing_1\implement_8.obj 
h:\temp\usession-27\testing_1\implement_9.obj 
h:\temp\usession-27\testing_1\implement_10.obj 
h:\temp\usession-27\testing_1\implement_11.obj 
h:\temp\usession-27\testing_1\implement_12.obj 
h:\temp\usession-27\testing_1\implement_13.obj 
h:\temp\usession-27\testing_1\implement_14.obj 
h:\temp\usession-27\module_cache\module_0.obj 
h:\temp\usession-27\module_cache\module_1.obj 
h:\temp\usession-27\module_cache\module_2.obj 
h:\temp\usession-27\module_cache\module_3.obj 
/OUT:h:\temp\usession-27\testing_1\testing_1.exe -=- specialized 0 more 
blocks -=- [backendopt:inlining] phase with threshold factor: 32.

--------------------------------- end pypy link line 
-----------------------------------------------------------------

From michaelschneider at current.net  Tue Jan 15 21:29:00 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Tue, 15 Jan 2008 15:29:00 -0500
Subject: [pypy-dev] windows C translation update
In-Reply-To: <478B5687.9010705@gmx.de>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
	<478B5687.9010705@gmx.de>
Message-ID: <478D178C.9020202@current.net>

Wow,

Thanks to everyone for their efforts.   The compile errors have been 
addressed, and the first couple of layers of the link-issue onion
have been peeled.

We are down to this link issue on windows

Mike

Note: this translation does not include the allmodules flag,  we did the 
first trial balloon of that today, and there are some issues in the mmap 
area. 
Those issues can until we finish this first level of success.
----------------------------------------------------------------------------------------------

module_0.c
module_0.c(885) : warning C4013: 'WCOREDUMP' undefined; assuming extern 
returnin
g int
module_0.c(890) : warning C4013: 'WIFCONTINUED' undefined; assuming 
extern retur
ning int
module_0.c(895) : warning C4013: 'WIFSTOPPED' undefined; assuming extern 
returni
ng int
module_0.c(900) : warning C4013: 'WIFSIGNALED' undefined; assuming 
extern return
ing int
module_0.c(905) : warning C4013: 'WIFEXITED' undefined; assuming extern 
returnin
g int
module_0.c(910) : warning C4013: 'WEXITSTATUS' undefined; assuming 
extern return
ing int
module_0.c(915) : warning C4013: 'WSTOPSIG' undefined; assuming extern 
returning
 int
module_0.c(920) : warning C4013: 'WTERMSIG' undefined; assuming extern 
returning
 int
module_1.c
module_1.c(860) : warning C4273: '_errno' : inconsistent dll linkage
module_2.c
module_3.c
module_3.c(859) : warning C4273: '__p__environ' : inconsistent dll linkage
   Creating library h:\temp\usession-36\testing_1\testing_1.lib and 
object h:\te
mp\usession-36\testing_1\testing_1.exp
module_0.obj : error LNK2019: unresolved external symbol _WCOREDUMP 
referenced i
n function _pypy_macro_wrapper_WCOREDUMP
module_0.obj : error LNK2019: unresolved external symbol _WIFCONTINUED 
reference
d in function _pypy_macro_wrapper_WIFCONTINUED
module_0.obj : error LNK2019: unresolved external symbol _WIFSTOPPED 
referenced
in function _pypy_macro_wrapper_WIFSTOPPED
module_0.obj : error LNK2019: unresolved external symbol _WIFSIGNALED 
referenced
 in function _pypy_macro_wrapper_WIFSIGNALED
module_0.obj : error LNK2019: unresolved external symbol _WIFEXITED 
referenced i
n function _pypy_macro_wrapper_WIFEXITED
module_0.obj : error LNK2019: unresolved external symbol _WEXITSTATUS 
referenced
 in function _pypy_macro_wrapper_WEXITSTATUS
module_0.obj : error LNK2019: unresolved external symbol _WSTOPSIG 
referenced in
 function _pypy_macro_wrapper_WSTOPSIG
module_0.obj : error LNK2019: unresolved external symbol _WTERMSIG 
referenced in
 function _pypy_macro_wrapper_WTERMSIG
h:\temp\usession-36\testing_1\testing_1.exe : fatal error LNK1120: 8 
unresolved
externals


From fijall at gmail.com  Tue Jan 15 22:35:36 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 15 Jan 2008 22:35:36 +0100
Subject: [pypy-dev] windows C translation update
In-Reply-To: <478D178C.9020202@current.net>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>
	<478B5687.9010705@gmx.de> <478D178C.9020202@current.net>
Message-ID: <693bc9ab0801151335y1bf21410y8d1d38bcf60c0dae@mail.gmail.com>

On Jan 15, 2008 9:29 PM, Michael Schneider <michaelschneider at current.net>
wrote:

> Wow,
>
> Thanks to everyone for their efforts.   The compile errors have been
> addressed, and the first couple of layers of the link-issue onion
> have been peeled.
>
> We are down to this link issue on windows
>
> This one should be fixed by 50653.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080115/627ac684/attachment.htm 

From michaelschneider at current.net  Wed Jan 16 04:56:01 2008
From: michaelschneider at current.net (Michael Schneider)
Date: Tue, 15 Jan 2008 22:56:01 -0500
Subject: [pypy-dev] windows C translation update
In-Reply-To: <693bc9ab0801151335y1bf21410y8d1d38bcf60c0dae@mail.gmail.com>
References: <47891F21.7080100@current.net> <478AE64C.1080706@current.net>	
	<478B5687.9010705@gmx.de> <478D178C.9020202@current.net>
	<693bc9ab0801151335y1bf21410y8d1d38bcf60c0dae@mail.gmail.com>
Message-ID: <478D8051.4020506@current.net>

Maciej ,

Good work,  I updated to trunk, and the translation completes 
successfully :-)

Thanks
Mike

Maciej Fijalkowski wrote:
>
>
> On Jan 15, 2008 9:29 PM, Michael Schneider 
> <michaelschneider at current.net <mailto:michaelschneider at current.net>> 
> wrote:
>
>     Wow,
>
>     Thanks to everyone for their efforts.   The compile errors have been
>     addressed, and the first couple of layers of the link-issue onion
>     have been peeled.
>
>     We are down to this link issue on windows
>
> This one should be fixed by 50653.


From Martin.vonLoewis at hpi.uni-potsdam.de  Tue Jan 22 13:26:32 2008
From: Martin.vonLoewis at hpi.uni-potsdam.de (=?iso-8859-1?Q?Martin_von_L=F6wis?=)
Date: Tue, 22 Jan 2008 13:26:32 +0100
Subject: [pypy-dev] Workshop on Self-sustaining Systems (S3)
Message-ID: <5AC0F51A926CD442BC08D55DA7DAB1D40868182F@3mxma1r.hpi.uni-potsdam.de>

*** Workshop on Self-sustaining Systems (S3) 2008 ***

May 15-16, 2008
Potsdam, Germany
http://www.swa.hpi.uni-potsdam.de/s3/

-- Call for papers:

The Workshop on Self-sustaining Systems (S3) is a forum for discussion
of topics relating to computer systems and languages that are able to
bootstrap, implement, modify, and maintain themselves. One property of
these systems is that their implementation is based on small but
powerful abstractions; examples include (amongst others)
Squeak/Smalltalk, COLA, Klein/Self, PyPy/Python, Rubinius/Ruby, and
Lisp. Such systems are the engines of their own replacement, giving
researchers and developers great power to experiment with, and explore
future directions from within, their own small language kernels.

S3 will be take place May 15-16, 2008 at the Hasso-Plattner-Institute in
Potsdam, Germany. It is an exciting opportunity for researchers and
practitioners interested in self-sustaining systems to meet and share
their knowledge, experience, and ideas for future research and development.

-- Invited talk:

Ian Piumarta: Late-bound Object Lambda Architectures (Viewpoints
Research Institute, USA)

-- Submissions and proceedings:

S3 invites submissions of high-quality papers reporting original
research, or describing innovative contributions to, or experience with,
self-sustaining systems, their implementation, and their application.
Papers that depart significantly from established ideas and practices
are particularly welcome.

Submissions must not have been published previously and must not be
under review for any another refereed event or publication. The program
committee will evaluate each contributed paper based on its relevance,
significance, clarity, and originality. Revised papers will be published
as post-proceedings in the Springer LNCS series.

Papers should be submitted electronically via EasyChair at
http://www.easychair.org/conferences/?conf=s3 in PDF format. Submissions
must be written in English (the official language of the workshop) and
must not exceed 20 pages. They should use the LNCS format, templates for
which are available at http://www.springer.de/comp/lncs/authors.html.

-- Venue:

Hasso-Plattner-Institut (Potsdam, Germany)

-- Important dates:

Submission of papers: February 15, 2008
Author notification: April 11, 2008
Revised papers due: April 25, 2008

S3 workshop: May 15-16, 2008

Final papers for LNCS post-proceedings due: June 6, 2008

-- Chairs:

* Robert Hirschfeld (Hasso-Plattner-Institut Potsdam, Germany)
* Kim Rose (Viewpoints Research Institute, USA)

-- Program committee:

* Johan Brichau, Universite Catholique de Louvain, Belgium
* Pascal Costanza, Vrije Universiteit Brussel, Belgium
* Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium
* Stephane Ducasse, INRIA Lille, France
* Michael Haupt, Hasso-Plattner-Institut, Germany
* Robert Hirschfeld, Hasso-Plattner-Institut, Germany
* Dan Ingalls, Sun Microsystems Laboratories, USA
* Martin von L?wis, Hasso-Plattner-Institut, Germany
* Hidehiko Masuhara, University of Tokyo, Japan
* Ian Piumarta, Viewpoints Research Institute, USA
* David Ungar, IBM, USA

-- Registration fees:

Early (until April 18, 2008)
* Regular participants: EUR 160
* Students: EUR 80

Late (after April 18, 2008)
* Regular participants: EUR 170
* Students: EUR 90 

From arigo at tunes.org  Wed Jan 23 18:54:10 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 23 Jan 2008 18:54:10 +0100
Subject: [pypy-dev] Loops for gcc
Message-ID: <20080123175409.GA29424@code0.codespeak.net>

Hi all,

Some results seem to show that GCC is more happy optimizing "for" and
"while" loops than the equivalent loops written with a bunch of goto's
like we produce in our C backend.  Interestingly, it seems that what
makes GCC unhappy is not the bunch of goto's, but only the absence of
the explicit C-level loop.  Example:

For this piece of C code as someone would write by hand:

        while (n > 0) {
            ...body...
        }

we generate the following instead, which defeats GCC's loop-specific
optimizations:

      block1:
        cond = n > 0;
        if (cond) goto block2;
        goto block3;

      block2:
        ...body...
        goto block1;

The smallest change I could find that make GCC produce good code again
is to modify block1 and block2 like this:

      block1:
        cond = n > 0;
        while (cond) {      /* trick */
            goto block2;
          block1_back:
            cond = n > 0;   /* the operations in block1 are repeated */
        }
        goto block3;

      block2:
        ...body...
        goto block1_back;   /* changed */

GCC doesn't seem to mind that the real body of the loop is not in the
loop at all but somewhere else; I guess it just follows the chain of
goto's and treat the result as equivalent to having the source code
directly within the loop's { }.

So I wrote a quick hack that detects (for now only the innermost) loops
in graphs.  More specifically it finds "loop head" blocks like block1,
and "loop-closing links" like the link from block2 to block1.  It then
produces code like above.  It's in the loops-for-gcc branch.  It's quite
obscure but seems to give a consistent 5% speed-up on pypy-c running
richards...


A bientot,

Armin.

From martin at martincmartin.com  Fri Jan 25 16:30:17 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 10:30:17 -0500
Subject: [pypy-dev] Two problems
Message-ID: <479A0089.3000400@martincmartin.com>

Hi,

I'm really impressed with the C code generated for some functions I'd 
like to use in my project.  You guys rock!

I've run into two problems this morning.  In pypy-1.0.0, this:

import re
import os, sys
from pypy.translator.interactive import Translation
import py

def mytest():
    x = re.search('bar', 'foobar')
    return x.start()

t = Translation(mytest)
t.annotate()

throws an exception:

   File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line 
567, in simple_call
     return bltn.analyser(bltn.s_self, *args)
   File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line 
19, in __call__
     return self.im_func(firstarg, *args, **kwds)
TypeError': method_get() takes exactly 3 arguments (2 given)
         .. v1 = simple_call((builtin_function_or_method get), v0)
         .. '(sre:213)_compile_star2'


This is an InstanceMethod where im_self is a SomeDict and args is a 
tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, 
'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). 
It seems to be calling a method_get() with two args, SomeDict and 
SomeTuple, whereas it wants 3.  Any idea how to fix this?

Thinking it might be fixed in svn, I got the latest code using:

svn co http://codespeak.net/svn/pypy/dist pypy-dist

Then tried to start pypy using:

python pypy-dist/pypy/bin/py.py

But got:

   File "/ita/downloaded/pypy-dist/pypy/module/posix/__init__.py", line 
3, in ?
     from pypy.rpython.module.ll_os import RegisterOs
   File "/ita/downloaded/pypy-dist/pypy/rpython/module/ll_os.py", line 127
     @registering_if(os, 'execv')
     ^
SyntaxError: invalid syntax

This is all on 64 bit Linux, python 2.4.1.

Any idea how I can get my simple function to compile?

Best,
Martin


From fijal at genesilico.pl  Fri Jan 25 16:40:34 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 16:40:34 +0100
Subject: [pypy-dev] Two problems
In-Reply-To: <479A0089.3000400@martincmartin.com>
References: <479A0089.3000400@martincmartin.com>
Message-ID: <479A02F2.8030109@genesilico.pl>

Martin C. Martin wrote:
> Hi,
>
> I'm really impressed with the C code generated for some functions I'd 
> like to use in my project.  You guys rock!
>
> I've run into two problems this morning.  In pypy-1.0.0, this:
>
> import re
> import os, sys
> from pypy.translator.interactive import Translation
> import py
>
> def mytest():
>     x = re.search('bar', 'foobar')
>     return x.start()
>
> t = Translation(mytest)
> t.annotate()
>
> throws an exception:
>
>    File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line 
> 567, in simple_call
>      return bltn.analyser(bltn.s_self, *args)
>    File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line 
> 19, in __call__
>      return self.im_func(firstarg, *args, **kwds)
> TypeError': method_get() takes exactly 3 arguments (2 given)
>          .. v1 = simple_call((builtin_function_or_method get), v0)
>          .. '(sre:213)_compile_star2'
>
>   
re is not RPythonic, you cannot use it.
> This is an InstanceMethod where im_self is a SomeDict and args is a 
> tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, 
> 'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). 
> It seems to be calling a method_get() with two args, SomeDict and 
> SomeTuple, whereas it wants 3.  Any idea how to fix this?
>
> Thinking it might be fixed in svn, I got the latest code using:
>
> svn co http://codespeak.net/svn/pypy/dist pypy-dist
>
> Then tried to start pypy using:
>
> python pypy-dist/pypy/bin/py.py
>
> But got:
>
>    File "/ita/downloaded/pypy-dist/pypy/module/posix/__init__.py", line 
> 3, in ?
>      from pypy.rpython.module.ll_os import RegisterOs
>    File "/ita/downloaded/pypy-dist/pypy/rpython/module/ll_os.py", line 127
>      @registering_if(os, 'execv')
>      ^
> SyntaxError: invalid syntax
>
> This is all on 64 bit Linux, python 2.4.1.
>
> Any idea how I can get my simple function to compile?
>
> Best,
> Martin
>   
this is valid 2.4 syntax (it does not work on 2.3 though), are you sure 
you're using 2.4?

Also, 64 bit might only work on svn (and only *might*)

Cheers,
fijal


:.


From martin at martincmartin.com  Fri Jan 25 16:40:44 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 10:40:44 -0500
Subject: [pypy-dev] Two problems
In-Reply-To: <20080125073621.d6b88167.sburton@ewtllc.com>
References: <479A0089.3000400@martincmartin.com>
	<20080125073621.d6b88167.sburton@ewtllc.com>
Message-ID: <479A02FC.7010403@martincmartin.com>

So it's not possible for RPython to call regular, non-RPython?  Is there 
some wrapper I could write to let RPython call regular python code?

Best,
Martin

Simon Burton wrote:
> On Fri, 25 Jan 2008 10:30:17 -0500
> "Martin C. Martin" <martin at martincmartin.com> wrote:
> 
>> Hi,
>>
>> I'm really impressed with the C code generated for some functions I'd 
>> like to use in my project.  You guys rock!
>>
>> I've run into two problems this morning.  In pypy-1.0.0, this:
>>
>> import re
>> import os, sys
>> from pypy.translator.interactive import Translation
>> import py
>>
>> def mytest():
>>     x = re.search('bar', 'foobar')
>>     return x.start()
>>
>> t = Translation(mytest)
>> t.annotate()
>>
>> throws an exception:
>>
>>    File "/ita/downloaded/pypy-1.0.0/pypy/annotation/unaryop.py", line 
>> 567, in simple_call
>>      return bltn.analyser(bltn.s_self, *args)
>>    File "/ita/downloaded/pypy-1.0.0/pypy/tool/instancemethod.py", line 
>> 19, in __call__
>>      return self.im_func(firstarg, *args, **kwds)
>> TypeError': method_get() takes exactly 3 arguments (2 given)
>>          .. v1 = simple_call((builtin_function_or_method get), v0)
>>          .. '(sre:213)_compile_star2'
>>
>>
>> This is an InstanceMethod where im_self is a SomeDict and args is a 
>> tuple with a single element, a SomeTuple whose "const" is (<type 'str'>, 
>> 'bar', 0) and whose "items" is (SomeObject, SomeString, SomeInteger). 
>> It seems to be calling a method_get() with two args, SomeDict and 
>> SomeTuple, whereas it wants 3.  Any idea how to fix this?
> 
> I'm not sure you can use module "re" in rpython code.
> 
> The error is because somewhere there is a .get(XX) method call on a dict.
> In rpython the default arg for dict.get must be specified.
> 
> Simon.
> 
> 

From fijal at genesilico.pl  Fri Jan 25 16:46:27 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 16:46:27 +0100
Subject: [pypy-dev] Two problems
In-Reply-To: <479A02FC.7010403@martincmartin.com>
References: <479A0089.3000400@martincmartin.com>	<20080125073621.d6b88167.sburton@ewtllc.com>
	<479A02FC.7010403@martincmartin.com>
Message-ID: <479A0453.2060400@genesilico.pl>

Martin C. Martin wrote:
> So it's not possible for RPython to call regular, non-RPython?  Is there 
> some wrapper I could write to let RPython call regular python code?
>   
Not easily. You can call normal python code, provided that you run pypy 
interpreter, not cpython. This means that stuff like re are working, but 
stuff like pygtk or any 3rd party modules doesn't.

Also calling is very different. You can take a look at module/_demo if 
you like, but there are obstacles, like you cannot compile only your 
module, you need each time to compile whole interpreter (which takes 
~30minutes on decent machine). It's very unlikely you want to use it 
that way :)


:.


From martin at martincmartin.com  Fri Jan 25 16:48:42 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 10:48:42 -0500
Subject: [pypy-dev] Two problems
In-Reply-To: <479A0453.2060400@genesilico.pl>
References: <479A0089.3000400@martincmartin.com>	<20080125073621.d6b88167.sburton@ewtllc.com>
	<479A02FC.7010403@martincmartin.com>
	<479A0453.2060400@genesilico.pl>
Message-ID: <479A04DA.8070401@martincmartin.com>



Maciek Fijalkowski wrote:
> Martin C. Martin wrote:
>> So it's not possible for RPython to call regular, non-RPython?  Is 
>> there some wrapper I could write to let RPython call regular python code?
>>   
> Not easily. You can call normal python code, provided that you run pypy 
> interpreter, not cpython. This means that stuff like re are working, but 
> stuff like pygtk or any 3rd party modules doesn't.
> 
> Also calling is very different. You can take a look at module/_demo if 
> you like, but there are obstacles, like you cannot compile only your 
> module, you need each time to compile whole interpreter (which takes 
> ~30minutes on decent machine). It's very unlikely you want to use it 
> that way :)

:)  Ok, thanks.

Best,
Martin

From martin at martincmartin.com  Fri Jan 25 21:32:38 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 15:32:38 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
Message-ID: <479A4766.1060006@martincmartin.com>

Hi,

There seems to be a lot of overhead when passing a large string (23 Meg) 
to C compiled RPython code.  For example, this code:

def small(text):
    return 3

t = Translation(small)
t.annotate()
t.rtype()
f3 = t.compile_c()

st = time.time()
z = f3(xml)
print time.time() - st

Whereas parsing the 16,000 XML elements using a regular expression only 
took 22 msec.  Even reading the text file inside the compiled RPython is 
faster than passing it.

Here's the code from rstr.py that (seems to be) doing the conversion. 
Any idea how I'd put timing code in there to see what's taking all the 
time?  Any idea how to speed it up?

class __extend__(pairtype(PyObjRepr, AbstractStringRepr)):
     def convert_from_to((r_from, r_to), v, llops):
         v_len = llops.gencapicall('PyString_Size', [v], resulttype=Signed)
         cstr = inputconst(Void, STR)
         v_result = llops.genop('malloc_varsize', [cstr, v_len],
                                resulttype=Ptr(STR))
         cchars = inputconst(Void, "chars")
         v_chars = llops.genop('getsubstruct', [v_result, cchars],
                               resulttype=Ptr(STR.chars))
         llops.gencapicall('PyString_ToLLCharArray', [v, v_chars])
         string_repr = llops.rtyper.type_system.rstr.string_repr
         v_result = llops.convertvar(v_result, string_repr, r_to)
         return v_result

Best,
Martin



From fijal at genesilico.pl  Fri Jan 25 22:04:38 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 22:04:38 +0100
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A4766.1060006@martincmartin.com>
References: <479A4766.1060006@martincmartin.com>
Message-ID: <479A4EE6.3030609@genesilico.pl>

Martin C. Martin wrote:
> Hi,
>
> There seems to be a lot of overhead when passing a large string (23 Meg) 
> to C compiled RPython code.  For example, this code:
>
> def small(text):
>     return 3
>
> t = Translation(small)
> t.annotate()
> t.rtype()
> f3 = t.compile_c()
>
> st = time.time()
> z = f3(xml)
> print time.time() - st
>
>   
This is wrong. You should even get a warning, the proper command is 
t.annotate([str]).

Besides, this is not the official way of writing rpython standalone 
programs. The official way is to go to pypy/translator/goal and for 
example modify targetnopstandalone for a standalone target (the entry 
function is entry_point).

You should translate this by ./translate.py targetnopstandalone.py (or 
whatever target you choose). You can even use some fancy options (like 
different gcs). In your example the xml was converted to python object, 
which will never happen the official way.

Cheers,
fijal


:.


From martin at martincmartin.com  Fri Jan 25 22:40:49 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 16:40:49 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A4EE6.3030609@genesilico.pl>
References: <479A4766.1060006@martincmartin.com>
	<479A4EE6.3030609@genesilico.pl>
Message-ID: <479A5761.2030903@martincmartin.com>



Maciek Fijalkowski wrote:
> Martin C. Martin wrote:
>> Hi,
>>
>> There seems to be a lot of overhead when passing a large string (23 
>> Meg) to C compiled RPython code.  For example, this code:
>>
>> def small(text):
>>     return 3
>>
>> t = Translation(small)
>> t.annotate()
>> t.rtype()
>> f3 = t.compile_c()
>>
>> st = time.time()
>> z = f3(xml)
>> print time.time() - st
>>
>>   
> This is wrong. You should even get a warning, the proper command is 
> t.annotate([str]).

Oops, yes, I've been working with variations of this all day, and I 
hadn't actually compiled & run the example in the email, although I'd 
done something equivalent.

> Besides, this is not the official way of writing rpython standalone 
> programs.

Thanks, but I'm not trying to write a standalone program, I need to call 
some 3rd party libraries.  For example, the string comes from one of a 
couple dozen of socket connections, managed by Twisted.  So I just want 
my inner loop in RPython.  The inner loop turns XML into a MySQL 
statement, which the main python program can then send to a database.

So I need to get a big string into RPython, and a smaller (but still 
pretty big) string out of it.

I see some other targets in there for shared libraries.  The docs 
mention that translate.py takes a --gc=generation argument, but when I 
try that I get:

$ python ./translator/goal/translate.py --gc=generation fun3.py
Usage
=====
   translate [options] [target] [target-specific-options]

translate: error: invalid value generation for option gc

Am I specifying it wrong?

Thanks,
Martin

> The official way is to go to pypy/translator/goal and for 
> example modify targetnopstandalone for a standalone target (the entry 
> function is entry_point).
> 
> You should translate this by ./translate.py targetnopstandalone.py (or 
> whatever target you choose). You can even use some fancy options (like 
> different gcs). In your example the xml was converted to python object, 
> which will never happen the official way.
> 
> Cheers,
> fijal
> 
> 
> :.
> 

From fijal at genesilico.pl  Fri Jan 25 22:45:35 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 22:45:35 +0100
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A5761.2030903@martincmartin.com>
References: <479A4766.1060006@martincmartin.com>
	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com>
Message-ID: <479A587F.7040807@genesilico.pl>


> Thanks, but I'm not trying to write a standalone program, I need to 
> call some 3rd party libraries.  For example, the string comes from one 
> of a couple dozen of socket connections, managed by Twisted.  So I 
> just want my inner loop in RPython.  The inner loop turns XML into a 
> MySQL statement, which the main python program can then send to a 
> database.
You can do it that way, true. But it's pretty unofficial and 
unsupported. Also you can only pass very simple types (ie list or dict 
will not work).
>
> Am I specifying it wrong?
>
You're using too old version (you should just check it out from svn)

:.


From martin at martincmartin.com  Fri Jan 25 22:53:49 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 16:53:49 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A587F.7040807@genesilico.pl>
References: <479A4766.1060006@martincmartin.com>
	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com>
	<479A587F.7040807@genesilico.pl>
Message-ID: <479A5A6D.7030906@martincmartin.com>



Maciek Fijalkowski wrote:
> 
>> Thanks, but I'm not trying to write a standalone program, I need to 
>> call some 3rd party libraries.  For example, the string comes from one 
>> of a couple dozen of socket connections, managed by Twisted.  So I 
>> just want my inner loop in RPython.  The inner loop turns XML into a 
>> MySQL statement, which the main python program can then send to a 
>> database.
> You can do it that way, true. But it's pretty unofficial and 
> unsupported. Also you can only pass very simple types (ie list or dict 
> will not work).

Ah, thanks for letting me know.  And thanks for all your help today!

Best,
Martin


From jgilbert.python at gmail.com  Fri Jan 25 23:06:03 2008
From: jgilbert.python at gmail.com (Josh Gilbert)
Date: Fri, 25 Jan 2008 17:06:03 -0500
Subject: [pypy-dev] How to learn RPython as a general purpose language
Message-ID: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>

I tried downloading the archives of this list and grepping, but
couldn't find an answer to this question. If it's been discussed to
death and I missed it I apologize.

I realize that it is not recommended to use RPython as a general
purpose language, a front end to C if you will. My question is how one
might learn how to do so, regardless. I'm also interested in reasons
why it shouldn't be done (preferably ones that can be resolved).

## References
I've found a comment on LtU suggesting that it is extremely hard to
learn, yet a valuable skill if you can cut it:
http://lambda-the-ultimate.org/node/1260#comment-13972

I've also found a hello world example from 2006 (humorously labeled part 1):
http://radix.twistedmatrix.com/2006/12/how-to-do-something-with-rpython-part-1.html


Josh Gilbert.

From martin at martincmartin.com  Fri Jan 25 23:19:15 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Fri, 25 Jan 2008 17:19:15 -0500
Subject: [pypy-dev] How to learn RPython as a general purpose language
In-Reply-To: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
Message-ID: <479A6063.6020001@martincmartin.com>

In RPython, you can't call into other libraries, be they C or Python. 
Also, calling RPython from Python is "pretty unofficial and 
unsupported."  Given that much of modern programming centers around 
connecting various libraries together, this makes RPython a poor choice 
for most practical work.

At the moment, the most straight forward way to learn RPython is to 
learn Python, then learn the restrictions of RPython, since RPython is a 
subset of Python.  The easiest way to do this is, basically, keep 
writing Python and seeing what the translator will and won't accept.

Please correct me if I'm wrong; I've only spent a few days looking at 
RPython.

Best,
Martin

Josh Gilbert wrote:
> I tried downloading the archives of this list and grepping, but
> couldn't find an answer to this question. If it's been discussed to
> death and I missed it I apologize.
> 
> I realize that it is not recommended to use RPython as a general
> purpose language, a front end to C if you will. My question is how one
> might learn how to do so, regardless. I'm also interested in reasons
> why it shouldn't be done (preferably ones that can be resolved).
> 
> ## References
> I've found a comment on LtU suggesting that it is extremely hard to
> learn, yet a valuable skill if you can cut it:
> http://lambda-the-ultimate.org/node/1260#comment-13972
> 
> I've also found a hello world example from 2006 (humorously labeled part 1):
> http://radix.twistedmatrix.com/2006/12/how-to-do-something-with-rpython-part-1.html
> 
> 
> Josh Gilbert.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev

From fijal at genesilico.pl  Fri Jan 25 23:29:15 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 25 Jan 2008 23:29:15 +0100
Subject: [pypy-dev] How to learn RPython as a general purpose language
In-Reply-To: <479A6063.6020001@martincmartin.com>
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
	<479A6063.6020001@martincmartin.com>
Message-ID: <479A62BB.3090108@genesilico.pl>

Martin C. Martin wrote:
> In RPython, you can't call into other libraries, be they C or Python. 
> Also, calling RPython from Python is "pretty unofficial and 
> unsupported."  Given that much of modern programming centers around 
> connecting various libraries together, this makes RPython a poor choice 
> for most practical work.
>   
You can call C libraries from rpython. There is even a doc here 
http://codespeak.net/pypy/dist/pypy/doc/rffi.html. Calling from Python 
to RPython is indeed pretty troublesome.
> At the moment, the most straight forward way to learn RPython is to 
> learn Python, then learn the restrictions of RPython, since RPython is a 
> subset of Python.  The easiest way to do this is, basically, keep 
> writing Python and seeing what the translator will and won't accept.
>
> Please correct me if I'm wrong; I've only spent a few days looking at 
> RPython.
>
> Best,
> Martin
>   
I think this is bad decision. Especially that translator might accept 
something and start behaving differently. Some docs are here 
http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#rpython, but 
it's indeed hard to deal with. RPython is similiar to C++ in a sense 
that one can write books about tricks. The main strangeness is that 
RPython is constructed out of live objects, which means that during 
initialization all python dynamism is allowed, but not after. This is 
sometimes hard to follow and also it's evolving a bit in time. I would 
suggest looking at various smaller language implementations (lang/ 
subdirectory in svn) and come to #pypy on freenode and ask. But it would 
be better if you have good reason why exactly you want to use it.

Cheers,
fijal

:.


From 2007b at usenet.alexanderweb.de  Fri Jan 25 23:52:05 2008
From: 2007b at usenet.alexanderweb.de (Alexander Schremmer)
Date: Fri, 25 Jan 2008 23:52:05 +0100
Subject: [pypy-dev] How to learn RPython as a general purpose language
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
	<479A6063.6020001@martincmartin.com>
Message-ID: <mp5q65-nb8.ln1@tuxpaddy.alexanderweb.homeip.net>

Martin C. Martin wrote:

> At the moment, the most straight forward way to learn RPython is to
> learn Python, then learn the restrictions of RPython, since RPython is a
> subset of Python.  The easiest way to do this is, basically, keep
> writing Python and seeing what the translator will and won't accept.
> 
> Please correct me if I'm wrong; I've only spent a few days looking at
> RPython.

No, that's pretty much correct. Another way to describe RPython is this
analogy:

Syntax and semantics of Python, speed of C, restrictions of Java and
compiler error messages as penetrable as MUMPS (*)

Kind regards,
Alexander


(*) http://en.wikipedia.org/wiki/MUMPS#Sample_Programs


From richardemslie at gmail.com  Sat Jan 26 00:10:32 2008
From: richardemslie at gmail.com (Richard Emslie)
Date: Fri, 25 Jan 2008 23:10:32 +0000
Subject: [pypy-dev] How to learn RPython as a general purpose language
In-Reply-To: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
Message-ID: <479A6C68.70300@gmail.com>

Hi Josh,

Josh Gilbert wrote:
> 
> ## References
> I've found a comment on LtU suggesting that it is extremely hard to
> learn, yet a valuable skill if you can cut it:
> http://lambda-the-ultimate.org/node/1260#comment-13972
> 

I'm not sure I said it is 'extremely hard' or it is a 'valuable skill' - 
but i definitely said:

"""
Python is a subset of Python. The rules as to what the subset is vague 
indeed, but in short writing code without using dynamic language 
features post import time (ie the static-ness is based on code objects 
not python code) should be enough. These rules are fairly well 
documented and unlikely to change significantly in the future. The 
problem (IMHO) of promoting it is a language is that the tool chain is 
extremely non-user friendly and it is necessary to have a good 
understanding of how the flow graphs are created and how the several 
annotation passes work. All you will get is a stack trace in some 
obscure area of the annotation code for a compile error. :-) If one 
takes the time to learn how the above works then you have pretty useful 
language at your disposal.
"""

Well I wrote that many moons ago.  And is more or less still true - 
except I understood pypy's tool chain better than I do now.  After 
spending about 4-5 months rewriting a quite large application from 
python to rpython - I can say this about having a 'pretty useful 
language at your disposal':

(a) the translated rpython code was faster than python

(b) some parts were pretty much a copy and paste of the original python 
code and just a few tweaks to make it compile

(c) some parts were a complete rewrite

(d) the time taken for (b) and (c) was no less than it would of taken to 
write the code in say java or c++.

(e) the end result is still some nice python code that was still 
readable and easy to modify.

(f) a lot of niceties of using python go out the window such as no long 
waits for compiles, nice stack traces, run time introspection, no 
segmentation faults, etc etc

(a), (b) and (e) are advantages (c), (d) and (f) are disadvantages.  Of 
course if your application is interpretative and can make use of say the 
jit (and similar arguments for other features like the stackless 
transformation), then (d) would be a massive (colossal) effort that you 
get for free via pypy.  (f) does also have the advantage than rpython is 
python and during development one can run with some interpreter.

YMMV

Cheers - Richard




From cfbolz at gmx.de  Sat Jan 26 11:27:27 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sat, 26 Jan 2008 11:27:27 +0100
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479A5761.2030903@martincmartin.com>
References: <479A4766.1060006@martincmartin.com>	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com>
Message-ID: <479B0B0F.608@gmx.de>

Martin C. Martin wrote:
> 
> Maciek Fijalkowski wrote:
>> Martin C. Martin wrote:
>>> Hi,
>>>
>>> There seems to be a lot of overhead when passing a large string (23 
>>> Meg) to C compiled RPython code.  For example, this code:
>>>
>>> def small(text):
>>>     return 3
>>>
>>> t = Translation(small)
>>> t.annotate()
>>> t.rtype()
>>> f3 = t.compile_c()
>>>
>>> st = time.time()
>>> z = f3(xml)
>>> print time.time() - st
>>>
>>>   
>> This is wrong. You should even get a warning, the proper command is 
>> t.annotate([str]).
> 
> Oops, yes, I've been working with variations of this all day, and I 
> hadn't actually compiled & run the example in the email, although I'd 
> done something equivalent.
> 
>> Besides, this is not the official way of writing rpython standalone 
>> programs.
> 
> Thanks, but I'm not trying to write a standalone program, I need to call 
> some 3rd party libraries.  For example, the string comes from one of a 
> couple dozen of socket connections, managed by Twisted.  So I just want 
> my inner loop in RPython.  The inner loop turns XML into a MySQL 
> statement, which the main python program can then send to a database.
> 
> So I need to get a big string into RPython, and a smaller (but still 
> pretty big) string out of it.

Couldn't you just use a subprocess, read the string from stdin and write 
the result to stdout? It's quite likely that this is not slower than the 
way strings are passed in and out now and has many advantages. You would 
need to use os.read and os.write, since sys.stdin/stdout is not 
supported in RPython, but apart from that it should work fine.

One of them is that if you use the Translation class, your RPython 
program will use reference counting, which is our slowest GC. If you use 
a subprocess you get the benefits of our much better generational GC.

Cheers,

Carl Friedrich


From martin at martincmartin.com  Sun Jan 27 09:51:46 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Sun, 27 Jan 2008 03:51:46 -0500
Subject: [pypy-dev] Converting from Python strings to C strings slow?
In-Reply-To: <479B0B0F.608@gmx.de>
References: <479A4766.1060006@martincmartin.com>	<479A4EE6.3030609@genesilico.pl>
	<479A5761.2030903@martincmartin.com> <479B0B0F.608@gmx.de>
Message-ID: <479C4622.1090409@martincmartin.com>



Carl Friedrich Bolz wrote:
> Martin C. Martin wrote:
>>
>> Thanks, but I'm not trying to write a standalone program, I need to 
>> call some 3rd party libraries.  For example, the string comes from one 
>> of a couple dozen of socket connections, managed by Twisted.  So I 
>> just want my inner loop in RPython.  The inner loop turns XML into a 
>> MySQL statement, which the main python program can then send to a 
>> database.
>>
>> So I need to get a big string into RPython, and a smaller (but still 
>> pretty big) string out of it.
> 
> Couldn't you just use a subprocess, read the string from stdin and write 
> the result to stdout? It's quite likely that this is not slower than the 
> way strings are passed in and out now and has many advantages. You would 
> need to use os.read and os.write, since sys.stdin/stdout is not 
> supported in RPython, but apart from that it should work fine.
> 
> One of them is that if you use the Translation class, your RPython 
> program will use reference counting, which is our slowest GC. If you use 
> a subprocess you get the benefits of our much better generational GC.

What I'm really looking for is a way to write most of my applications in 
a dynamic language (because its more productive to write & maintain), 
then if and when performance is a problem, have some way to speed it up. 
  PyPy promises to do this even before performance is a problem, which 
will be great!

Until that comes, I was hoping for a language where I could give some 
hints to the compiler or runtime to speed it up.  Things like "although 
this binding could change each time through the loop, it doesn't 
actually change, so there's no need to do a hash lookup for every 
access."  Or "this variable is always an int."

The only language I know of that can do that is Lisp, which is a strong 
possibility.  But Lisp's syntax is more verbose and low level than 
modern dynamic languages, it doesn't have as many libraries, it doesn't 
have an IDE with auto completion, or a good source level debugger.  I 
had hoped Groovy would be like that, with its optional typing and Java 
inspired syntax and semantics, but sadly, the developers valued 
dynamism, however rarely used, over performance.

So the next best thing is to rewrite the performance critical parts in 
some other language.  I had hoped RPython would be that language for 
Python, but it turns out not to be.  I could rewrite in C++, but the 
semantics of C++ are very different than Python, so interfacing the two 
becomes verbose and awkward.  The ctypes module looks good for calling C 
libraries that weren't originally designed to work with Python.  But it 
doesn't have a good way (or any way?) to manipulate Python objects from 
C.  Even Java's JNI makes for a lot of boilerplate code to translate 
back and forth.

So it looks like my best bet may be Groovy, which interacts with Java 
seamlessly.  A year ago, when I last checked, the IDEs weren't up to the 
job, but may that's changed.

And once PyPy is done, that may be an even better solution.

Best,
Martin

From simon at arrowtheory.com  Tue Jan 29 18:06:22 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Tue, 29 Jan 2008 09:06:22 -0800
Subject: [pypy-dev] real-time gc ?
Message-ID: <20080129090622.351039ab.simon@arrowtheory.com>


I've taken some statistics on how the boehm gc performs with our current application,
it has terrible worst case performance, often stopping for hundreds of mS at a time.

I tried calling GC_enable_incremental() but that seems to cause the code to explode.

I don't have any idea how to obtain similar statistics for the refcounting gc, which
is also non-realtime. Perhaps it is in fact worse than boehm in this regard.

Over-all CPU usage is significantly less with boehm, but a lot can happen in 300mS of
a garbage collection.

I don't think it's feasable compiling the pypy interpreter with refcounting, it takes
a truly vast amount of time (some hours...)

It would be good using the pypy interpreter to script our rpython codebase, but this
gc issue seems to be a blocker.

Simon.

From atveit at gmail.com  Tue Jan 29 19:31:16 2008
From: atveit at gmail.com (Amund Tveit)
Date: Tue, 29 Jan 2008 19:31:16 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
Message-ID: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>

Hi, I wrote a blog entry yesterday about using rpython for the great
computer language shootout recursive program, and got a RunTime error
when I input on the commandline 11 (1 to 10 goes fine). Just did an
svn update of pypy to version 51115 and still get the error, here is
the program:

blog entry:
http://amundblog.blogspot.com/2008/01/rpython-gclb-benchmark-recursive.html

source file:
recursive.py
# The Computer Language Shootout
# http://shootout.alioth.debian.org/
# based on bearophile's psyco program
# slightly modified by Isaac Gouy

def Ack(x, y):
    if x == 0: return y+1
    if y == 0: return Ack(x-1, 1)
    return Ack(x-1, Ack(x, y-1))

def Fib(n):
    if n < 2: return 1
    return Fib(n-2) + Fib(n-1)

def FibFP(n):
    if n < 2.0: return 1.0
    return FibFP(n-2.0) + FibFP(n-1.0)

def Tak(x, y, z):
    if y < x: return Tak( Tak(x-1,y,z), Tak(y-1,z,x), Tak(z-1,x,y) )
    return z

def TakFP(x, y, z):
    if y < x: return TakFP( TakFP(x-1.0,y,z), TakFP(y-1.0,z,x),
TakFP(z-1.0,x,y) )
    return z

from sys import argv, setrecursionlimit
setrecursionlimit(20000)

def main(argv):
    n = int(argv[1]) - 1
    print "Ack(3,%d):" % (n+1), Ack(3, n+1)
    print "Fib(" + str(28.0+n) + "," + str(FibFP(28.0+n))
    print "Tak(%d,%d,%d): %d" % (3*n, 2*n, n, Tak(3*n, 2*n, n))
    print "Fib(3):", Fib(3)
    print "Tak(3.0,2.0,1.0):", TakFP(3.0, 2.0, 1.0)
    return 0

from pypy.translator.interactive import Translation
t = Translation(main, standalone=True, gc='ref')
t.source(backend='c')
path = t.compile()
print path

Best regards,
Amund

From amauryfa at gmail.com  Tue Jan 29 19:57:43 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 29 Jan 2008 19:57:43 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
In-Reply-To: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
References: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
Message-ID: <e27efe130801291057j7c831a8fwcd4d990203e74c35@mail.gmail.com>

Hello,

Amund Tveit wrote:
> Hi, I wrote a blog entry yesterday about using rpython for the great
> computer language shootout recursive program, and got a RunTime error
> when I input on the commandline 11 (1 to 10 goes fine). Just did an
> svn update of pypy to version 51115 and still get the error, here is
> the program:
>
> blog entry:
> http://amundblog.blogspot.com/2008/01/rpython-gclb-benchmark-recursive.html
>
> source file:
> recursive.py
> # The Computer Language Shootout
> # http://shootout.alioth.debian.org/
> # based on bearophile's psyco program
> # slightly modified by Isaac Gouy
>
> def Ack(x, y):
>     if x == 0: return y+1
>     if y == 0: return Ack(x-1, 1)
>     return Ack(x-1, Ack(x, y-1))
...
>
> from sys import argv, setrecursionlimit
> setrecursionlimit(20000)
>
> def main(argv):
>     n = int(argv[1]) - 1
>     print "Ack(3,%d):" % (n+1), Ack(3, n+1)
>     print "Fib(" + str(28.0+n) + "," + str(FibFP(28.0+n))
>     print "Tak(%d,%d,%d): %d" % (3*n, 2*n, n, Tak(3*n, 2*n, n))
>     print "Fib(3):", Fib(3)
>     print "Tak(3.0,2.0,1.0):", TakFP(3.0, 2.0, 1.0)
>     return 0
>
> from pypy.translator.interactive import Translation
> t = Translation(main, standalone=True, gc='ref')
> t.source(backend='c')
> path = t.compile()
> print path

This has nothing to do with pypy, and I get the same error with a
normal python interpreter: the Ackermann function is gigantically
recursive, and 20000 is probably not enough for Ack(3, 11).

-- 
Amaury Forgeot d'Arc

From cfbolz at gmx.de  Tue Jan 29 20:37:02 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 29 Jan 2008 20:37:02 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
In-Reply-To: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
References: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
Message-ID: <479F805E.20901@gmx.de>

Hi Amund,

Amund Tveit wrote:
> Hi, I wrote a blog entry yesterday about using rpython for the great
> computer language shootout recursive program, and got a RunTime error
> when I input on the commandline 11 (1 to 10 goes fine). Just did an
> svn update of pypy to version 51115 and still get the error, here is
> the program:

The runtime error is a stack overflow. RPython is checking for stack 
overflows when a recursive call is occuring (which is probably also the 
reason why the RPython version is slower). It is a bit conservative in 
doing that, which is why C still works for 11, but not the RPython version.

To simulate an infinite stack you could additionally pass in the option 
stackless=True and gc="generation" instead of ="ref" to the Translation 
class. This will make the program slightly slower, but should allow you 
very large arguments (argument sizes that probably make the normal C 
version segfault).

Cheers,

Carl Friedirch

From ae at amundtveit.info  Tue Jan 29 20:44:36 2008
From: ae at amundtveit.info (Amund Tveit)
Date: Tue, 29 Jan 2008 20:44:36 +0100
Subject: [pypy-dev] reg. GCLS recursive test.
In-Reply-To: <479F805E.20901@gmx.de>
References: <be52387e0801291031h6ea4eafcv4ff93be62b8d0502@mail.gmail.com>
	<479F805E.20901@gmx.de>
Message-ID: <be52387e0801291144t1b11479ex8a0a63293a05b42b@mail.gmail.com>

thanks, will try that.

(used stackless python a while back for simulation -
http://amundtveit.info/publications/2003/zereal.pdf )

Amund


2008/1/29, Carl Friedrich Bolz <cfbolz at gmx.de>:
>
> Hi Amund,
>
> Amund Tveit wrote:
> > Hi, I wrote a blog entry yesterday about using rpython for the great
> > computer language shootout recursive program, and got a RunTime error
> > when I input on the commandline 11 (1 to 10 goes fine). Just did an
> > svn update of pypy to version 51115 and still get the error, here is
> > the program:
>
> The runtime error is a stack overflow. RPython is checking for stack
> overflows when a recursive call is occuring (which is probably also the
> reason why the RPython version is slower). It is a bit conservative in
> doing that, which is why C still works for 11, but not the RPython
> version.
>
> To simulate an infinite stack you could additionally pass in the option
> stackless=True and gc="generation" instead of ="ref" to the Translation
> class. This will make the program slightly slower, but should allow you
> very large arguments (argument sizes that probably make the normal C
> version segfault).
>
> Cheers,
>
> Carl Friedirch
>



-- 
Amund Tveit
http://amundtveit.info/ - +47 416 26 572
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080129/7898a851/attachment-0001.htm 

From arigo at tunes.org  Mon Feb  4 11:26:47 2008
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 4 Feb 2008 11:26:47 +0100
Subject: [pypy-dev] How to learn RPython as a general purpose language
In-Reply-To: <479A6C68.70300@gmail.com>
References: <1b0ce1b60801251406i10d0ef49kc033325f428a667c@mail.gmail.com>
	<479A6C68.70300@gmail.com>
Message-ID: <20080204102646.GA30303@code0.codespeak.net>

Hi Richard,

On Fri, Jan 25, 2008 at 11:10:32PM +0000, Richard Emslie wrote:
> (a) the translated rpython code was faster than python
> 
> (b) some parts were pretty much a copy and paste of the original python 
> code and just a few tweaks to make it compile
> 
> (c) some parts were a complete rewrite
> 
> (d) the time taken for (b) and (c) was no less than it would of taken to 
> write the code in say java or c++.
> 
> (e) the end result is still some nice python code that was still 
> readable and easy to modify.
> 
> (f) a lot of niceties of using python go out the window such as no long 
> waits for compiles, nice stack traces, run time introspection, no 
> segmentation faults, etc etc

Thanks for this great summary of RPython :-)

In particular, it's all too easy to overlook (c) at a first glance.  Let
me paraphrase this again: <strong> it's very likely that some parts of a
typical Python programs cannot be simply tweaked until they are RPython!
</strong>

RPython is indeed designed to be some kind of new language in which to
write algorithmic'ish code from scratch.  It was not designed as a way
to incrementally speed up parts of an existing Python program.  The
Python <-> RPython interfaces are there mostly for historical reasons
and being deprecated because they get in the way of other features that
we need more right now.  These interfaces could be cleanly re-developed
if there are people interested in pushing RPython there, but given the
priorities and interests of the small group of people currently working
on it, it doesn't look like it is going to happen soon.


A bientot,

Armin

From arigo at tunes.org  Mon Feb  4 11:58:12 2008
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 4 Feb 2008 11:58:12 +0100
Subject: [pypy-dev] real-time gc ?
In-Reply-To: <20080129090622.351039ab.simon@arrowtheory.com>
References: <20080129090622.351039ab.simon@arrowtheory.com>
Message-ID: <20080204105812.GB30303@code0.codespeak.net>

Hi Simon,

On Tue, Jan 29, 2008 at 09:06:22AM -0800, Simon Burton wrote:
> I've taken some statistics on how the boehm gc performs with our current application,
> it has terrible worst case performance, often stopping for hundreds of mS at a time.

What is the context?  It's unclear to me if you're talking about PyPy
the Python interpreter, another stand-alone RPython program, or a
CPython extension module translated from RPython.

In all cases but the latter you should definitely try our own GCs
instead of Boehm.  The generational GC in particular, while not
real-time in any sense of the word, is probably the one with the least
amount of long pauses - most collections are very quick.


A bientot,

Armin

From Ben.Young at sungard.com  Wed Feb  6 13:25:21 2008
From: Ben.Young at sungard.com (Ben.Young at sungard.com)
Date: Wed, 6 Feb 2008 12:25:21 -0000
Subject: [pypy-dev] FW: PyPy JIT and other languages
Message-ID: <A08C3DE85B750543B814BD2E7C33AEA75C3989@EMEA-EXCHANGE01.internal.sungard.corp>

Hi Guys,

 

I'm playing around with a bit of language design, and was wondering if
the JIT is able to support languages that run by traversing an AST,
rather than by interpreting bytecode? The reason is, my toy language is
able to mutate itself at runtime, and so the AST that's running is also
exposed at applevel.

 

Also, can any of you give a 2 sentence description of what a rainbow
interpreter is? It looks like a very low level bytecode interpreter, but
I'm sure it's more than that.

 

Still following the project after all these years!

 

Cheers,

Ben

 

Ben Young - Senior Software Engineer
SunGard - Enterprise House, Vision Park, Histon, Cambridge, CB24 9ZR
Tel +44 1223 266042 - Main +44 1223 266100 - http://www.sungard.com/

CONFIDENTIALITY:  This email (including any attachments) may contain
confidential, proprietary and privileged information, and unauthorized
disclosure or use is prohibited.  If you received this email in error,
please notify the sender and delete this email from your system.  Thank
you. 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080206/acdac9be/attachment.htm 

From cfbolz at gmx.de  Wed Feb  6 15:58:29 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 06 Feb 2008 15:58:29 +0100
Subject: [pypy-dev] FW: PyPy JIT and other languages
In-Reply-To: <A08C3DE85B750543B814BD2E7C33AEA75C3989@EMEA-EXCHANGE01.internal.sungard.corp>
References: <A08C3DE85B750543B814BD2E7C33AEA75C3989@EMEA-EXCHANGE01.internal.sungard.corp>
Message-ID: <47A9CB15.9060908@gmx.de>

Hi Ben,

Ben.Young at sungard.com wrote:
> I?m playing around with a bit of language design, and was wondering if 
> the JIT is able to support languages that run by traversing an AST, 
> rather than by interpreting bytecode?

That is possible in principle, yes.

> The reason is, my toy language is 
> able to mutate itself at runtime, and so the AST that?s running is also 
> exposed at applevel.

So the AST can be changed by the program too? Or only be inspected? If 
the AST is changeable it would be harder to support. The JIT could do 
something like generating code for every version of the AST it sees or 
so. However, currently there is no mechanism for throwing away existing 
machine code, so if the code is changed and run very often, you get 
memory leaks.

You could do other stuff like generating machine code only as long as 
the AST is not changed and fall back to interpreting once it is. In the 
future we might add ways to remove generated code later (by giving the 
interpreter a way to trigger that or by removing code that is rarely 
executed).


> Also, can any of you give a 2 sentence description of what a rainbow 
> interpreter is? It looks like a very low level bytecode interpreter, but 
> I?m sure it?s more than that.

The rainbow interpreter is the component that will replace the 
timeshifter when it is done. The timeshifter transforms an interpreter 
into a compiler and is a rather complex component. The rainbow 
interpreter takes a different approach where the interpreter flow graphs 
are interpreted to generate source code. The hope is that this 
interpreter is simpler than the transformation to allow experimentation 
with different approaches to figure out which things work.

Cheers,

Carl Friedrich

From Ben.Young at sungard.com  Wed Feb  6 16:05:14 2008
From: Ben.Young at sungard.com (Ben.Young at sungard.com)
Date: Wed, 6 Feb 2008 15:05:14 -0000
Subject: [pypy-dev] FW: PyPy JIT and other languages
In-Reply-To: <47A9CB15.9060908@gmx.de>
References: <A08C3DE85B750543B814BD2E7C33AEA75C3989@EMEA-EXCHANGE01.internal.sungard.corp>
	<47A9CB15.9060908@gmx.de>
Message-ID: <A08C3DE85B750543B814BD2E7C33AEA75C3A96@EMEA-EXCHANGE01.internal.sungard.corp>

Hi Carl Friedrich,

Thanks for the reply. The idea is that in most cases the AST will settle
down at some point (or could even be eagerly transformed), but there is
no guarantee that user nodes will do this if they don't want to. I think
a JIT will work 90% of the time. I considered rewriting it in c#  and
having nodes compile themselves to IL if they can prove no children can
do any manipulation, but I'd rather play with PyPy if possible!

I'm look to use the CLI backend anyway, as you get a nice standard
library built in anyway.

Thanks for the details on the rainbow interpreter. I'm looking forward
to the JIT being pushed forward again!

Cheers,
Ben



Ben Young - Senior Software Engineer
SunGard - Enterprise House, Vision Park, Histon, Cambridge, CB24 9ZR
Tel +44 1223 266042 - Main +44 1223 266100 - http://www.sungard.com/

CONFIDENTIALITY:  This email (including any attachments) may contain
confidential, proprietary and privileged information, and unauthorized
disclosure or use is prohibited.  If you received this email in error,
please notify the sender and delete this email from your system.  Thank
you. 

-----Original Message-----
From: Carl Friedrich Bolz [mailto:cfbolz at gmx.de] 
Sent: 06 February 2008 14:58
To: python at theyoungfamily.co.uk
Cc: pypy-dev at codespeak.net
Subject: Re: [pypy-dev] FW: PyPy JIT and other languages

Hi Ben,

Ben.Young at sungard.com wrote:
> I'm playing around with a bit of language design, and was wondering if

> the JIT is able to support languages that run by traversing an AST, 
> rather than by interpreting bytecode?

That is possible in principle, yes.

> The reason is, my toy language is 
> able to mutate itself at runtime, and so the AST that's running is
also 
> exposed at applevel.

So the AST can be changed by the program too? Or only be inspected? If 
the AST is changeable it would be harder to support. The JIT could do 
something like generating code for every version of the AST it sees or 
so. However, currently there is no mechanism for throwing away existing 
machine code, so if the code is changed and run very often, you get 
memory leaks.

You could do other stuff like generating machine code only as long as 
the AST is not changed and fall back to interpreting once it is. In the 
future we might add ways to remove generated code later (by giving the 
interpreter a way to trigger that or by removing code that is rarely 
executed).


> Also, can any of you give a 2 sentence description of what a rainbow 
> interpreter is? It looks like a very low level bytecode interpreter,
but 
> I'm sure it's more than that.

The rainbow interpreter is the component that will replace the 
timeshifter when it is done. The timeshifter transforms an interpreter 
into a compiler and is a rather complex component. The rainbow 
interpreter takes a different approach where the interpreter flow graphs

are interpreted to generate source code. The hope is that this 
interpreter is simpler than the transformation to allow experimentation 
with different approaches to figure out which things work.

Cheers,

Carl Friedrich




From stargaming at gmail.com  Sat Feb  9 21:04:03 2008
From: stargaming at gmail.com (Stargaming)
Date: Sat, 9 Feb 2008 21:04:03 +0100
Subject: [pypy-dev] Python on the Lego NXT
Message-ID: <20080209210403.6fa11a2d@gmail.com>

Hi PyPyers,

Lego has been developing robotics for quite a while now. Their latest
robot is the "Lego Mindstorms NXT." Unfortunately, its programming
capabilities are quite restricted_ -- with the standard firmware_ (open
source, BTW), state of the art programming is done in a C-like
language, NXC_, translated to an assembly-esque language, NBC_.

Now I want to use Python rather than NXC, but it currently only works
via a socket_ (which requires USB or Bluetooth) and live bytecode
instructions. I figured this could be possible with PyPy.

Previous advice I got from #pypy pointed out that this is indeed an
application where PyPy could be used but the Python interpreter *might*
be too heavy for this little machine. I have asked_ an unofficial Lego
Robotics mailing list as well (no reply yet).

There are several layers one could translate Python code (or rather
flow graphs?) to: Either NXC, a language pretty close to C. Chances are
high that translation to this layer might be trivial (tokenize Python
and generate NXC instructions), no PyPy involved at all. The next layer
is NBC, assembly code for the standard compiler. This can be compiled
to NXT bytecode_, which is rather close to real PC opcodes (not Python
bytecode, unfortunately). See my posting to the Lego Robotics mailing
list for detailed descriptions of these proposals.

As already stated in the Lego ML, translating the whole Python
interpreter might be too heavy for the NXT_ brick's storage (see below,
one quarter MB). Translating every program itself (and thus, writing
them in RPython), might be too expensive (considering translation
time). Writing a firmware just seems crazy.

My question: Do you think this would be a Good Thing to do with PyPy?
While I wrote these two postings, the idea of just tokenizing Python
and generating NXC began looking better with every second. Is there any
obvious solution I'm missing?

Sorry-for-the-slight-OT-ly yours,
Stargaming


Hardware_ details: Processors

  * Atmel 32-bit ARM processor
    * 256 KB Flash
    * 64 KB RAM
    * 48 MHz
    * (handles USB/Bluetooth, display/sound)
  * Atmel 8-bit AVR co-processor
    * 4 KB Flash
    * 512 Byte RAM
    * 8 MHz
    * (handles raw I/O)

Quick glossary:

    NXC
      Not eXactly C, C-like programming language for the NXT
    NBC
      Next Byte Code, Assembler-like language for the NXT
    NXT
      The robot itself :-)
  

.. _restricted: http://www.teamhassenplug.org/NXT/NXTSoftware.html
.. _firmware: http://mindstorms.lego.com/Overview/OpenSource.aspx
.. _NXC: http://bricxcc.sourceforge.net/nbc/nxcdoc/index.html
.. _NBC: http://bricxcc.sourceforge.net/nbc/
.. _socket: http://home.comcast.net/~dplau/nxt_python/index.html
.. _asked:
http://permalink.gmane.org/gmane.comp.hardware.lego.robotics/7193 
.. _bytecode:
http://cache.lego.com/upload/contentTemplating/MindstormsOverview/otherfiles/2057/LEGO%20MINDSTORMS%20NXT%20Executable%20File%20Specification(1)_9143FED4-3FF8-40B1-A06F-78B530347A59.zip 
.. _NXT: http://mindstorms.lego.com/Overview/NXTreme.aspx 
.. _Hardware:
http://cache.lego.com/upload/contentTemplatingMindstormsOverview/otherfiles/2057/LEGO%20MINDSTORMS%20NXT%20Hardware%20Developer%20Kit(3)_7A0CF630-CCE5-4AAF-91FA-D1E7C911817C.zip

.. All mentioned brands belong to their respective owners. ;)

From fijal at genesilico.pl  Mon Feb 11 17:00:29 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Mon, 11 Feb 2008 17:00:29 +0100
Subject: [pypy-dev] buffer protocol.
Message-ID: <47B0711D.1030402@genesilico.pl>

I feel a bit clueless about buffer protocol. Current implemntation in 
pypy is purely app-level, which means it's slow, but more importantly, 
one cannot get raw, C-level address of it. I've tried approaching the 
problem, but it seems that there are several serious issues.

* what's the future of buffer protocol?
* do people use that? ctypes tests are relying on it. anybody knows 
ctypes module implementation that relies on it?
* testing suite sucks. we've got two tests, cpython is not any better in 
this regard.

where to proceed?

:.


From zaz at alum.com  Fri Feb 15 07:59:32 2008
From: zaz at alum.com (daol)
Date: Fri, 15 Feb 2008 00:59:32 -0600
Subject: [pypy-dev] localized help
Message-ID: <ee18778d0802142259h61b2dde1r9f5faeaefc517e2c@mail.gmail.com>

Hi there, I'm after selection of a terminal project for school, but the fact
now is that I'm not versed in pypy or even language/compilers
design&implementation, still is an area that attract me and I would like to
do "something" useful/interesting (who not?) in pypy (for the moment the
more viable project is implement a language on top of pypy, still I don't
know if it will turn into other approach).

Thus the option that I have is to find people near where I live that I can
talk to and perhaps meet not only online, my hope is that some of you know
some people in M?xico.


Have a nice day or night.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080215/0616643b/attachment.htm 

From Ben.Young at sungard.com  Fri Feb 15 11:57:18 2008
From: Ben.Young at sungard.com (Ben.Young at sungard.com)
Date: Fri, 15 Feb 2008 10:57:18 -0000
Subject: [pypy-dev] (no subject)
Message-ID: <A08C3DE85B750543B814BD2E7C33AEA76832F6@EMEA-EXCHANGE01.internal.sungard.corp>

Hi,

 

I was trying to access the web based irc client at
http://starship.python.net/crew/mwh/irc/irc.cgi, that Michael very
kindly set up for me, but it doesn't appear to work anymore. Does anyone
know a way of accessing #pypy through the web?

 

The question I was going to ask is "are you working with Thomas Heller
on the ctypes port?" It seems to me he would be able to sort out a lot
of you lifetime / conversion questions very quickly? Have you tried to
contact him before?

 

Cheers,

Ben

 

Ben Young - Senior Software Engineer
SunGard - Enterprise House, Vision Park, Histon, Cambridge, CB24 9ZR
Tel +44 1223 266042 - Main +44 1223 266100 - http://www.sungard.com/

CONFIDENTIALITY:  This email (including any attachments) may contain
confidential, proprietary and privileged information, and unauthorized
disclosure or use is prohibited.  If you received this email in error,
please notify the sender and delete this email from your system.  Thank
you. 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080215/9b4dfbdf/attachment.htm 

From fijal at genesilico.pl  Fri Feb 15 20:14:20 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 15 Feb 2008 20:14:20 +0100
Subject: [pypy-dev] (no subject)
In-Reply-To: <A08C3DE85B750543B814BD2E7C33AEA76832F6@EMEA-EXCHANGE01.internal.sungard.corp>
References: <A08C3DE85B750543B814BD2E7C33AEA76832F6@EMEA-EXCHANGE01.internal.sungard.corp>
Message-ID: <47B5E48C.10809@genesilico.pl>

Ben.Young at sungard.com wrote:
>
> Hi,
>
>  
>
> The question I was going to ask is ?are you working with Thomas Heller 
> on the ctypes port?? It seems to me he would be able to sort out a lot 
> of you lifetime / conversion questions very quickly? Have you tried to 
> contact him before?
>
>  
>
> Cheers,
>
> Ben
>
Hey. We did not talk with Thomas Heller at all. No idea why. I suppose 
our problems are slightly different, ie. we're implementing everything 
on app-level based on a very raw libffi interface. We also have 
different problems (like moving gc) etc. Not sure if he would be interested.

Cheers,
fijal



:.


From cfbolz at gmx.de  Fri Feb 15 22:11:54 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 15 Feb 2008 22:11:54 +0100
Subject: [pypy-dev] (no subject)
In-Reply-To: <47B5E48C.10809@genesilico.pl>
References: <A08C3DE85B750543B814BD2E7C33AEA76832F6@EMEA-EXCHANGE01.internal.sungard.corp>
	<47B5E48C.10809@genesilico.pl>
Message-ID: <47B6001A.1050909@gmx.de>

Maciek Fijalkowski wrote:
> Ben.Young at sungard.com wrote:
>> The question I was going to ask is ?are you working with Thomas Heller 
>> on the ctypes port?? It seems to me he would be able to sort out a lot 
>> of you lifetime / conversion questions very quickly? Have you tried to 
>> contact him before?
>>
>>
> Hey. We did not talk with Thomas Heller at all. No idea why. I suppose 
> our problems are slightly different, ie. we're implementing everything 
> on app-level based on a very raw libffi interface. We also have 
> different problems (like moving gc) etc. Not sure if he would be interested.

While that is true, we should also mention that we are reporting back
bugs we found in the original ctypes implementation and plan to
contribute the additional tests we are writing for our ctypes
implementation.

Cheers,

Carl Friedrich

From lac at openend.se  Sat Feb 16 11:34:09 2008
From: lac at openend.se (Laura Creighton)
Date: Sat, 16 Feb 2008 11:34:09 +0100
Subject: [pypy-dev] Showed up in the mail
Message-ID: <200802161034.m1GAY9N3031463@theraft.openend.se>


Somebody's making a book about how technology gets done.
If we want to write up Sprint Driven Development and try to get it
out to the world, this could be a place to do it.

Laura

------- Forwarded Message

Return-Path: imeti2008 at mail.infocybereng.org
Delivery-Date: Sat Feb 16 02:40:38 2008
Return-Path: <imeti2008 at mail.infocybereng.org>
Subject: Inviting you to share your opinion and to participate in a project on Engineering and Meta-engineering
Date: Fri, 15 Feb 2008 19:31:35 -0400
MIME-Version: 1.0


Dear Laura Creighton,

We would like to inform you about the "Project on an Integral and
Integrative Perspective on Engineering and Meta-Engineering" which
will mainly contain several publications, empirical research, and
conferences organization. We appreciate if you can share your opinion
with us with regards to this issue.  You can do so by filling the
inquiry form at:
http://www.iiis.org/inquiry/b.asp?t=u11&e=LAC at OPENEND.SE It will
require no more than one minute of your valuable time.

We would also like to inform you that, in the context of this project,
we have already decided to 1) publish a book with the tentative title
of "Reflections on Engineering and Meta-Engineering for the XXI
Century", and 2) organize The First International Multi-Conference on
Engineering and Technological Innovation (IMETI 2008;
http://www.infocybereng.org/imeti2008). Consequently, any information
regarding your potential participation in any of these two activities
will be highly appreciated.

The rational supporting and motivating this project can be found in
the article "The Essence of Engineering and Meta-Engineering: A Work
in Progress" available at
http://www.iiis.org/Nagib-Callaos/Engineering-and-Meta-Engineering

Best regards,

Prof. Nagib C.Callaos
IMETI 2008 General Chair
www.sciiis.org/Nagib-Callaos

If you wish to be removed from this mailing list, please send an email
to imeti.remove at mail.infocybereng.org with REMOVE IMETI in the subject
line Address: Torre Profesional La California, Av. Francisco de
Miranda, Caracas, Venezuela.

------- End of Forwarded Message


From Martin.vonLoewis at hpi.uni-potsdam.de  Sun Feb 17 19:06:34 2008
From: Martin.vonLoewis at hpi.uni-potsdam.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 17 Feb 2008 19:06:34 +0100
Subject: [pypy-dev] Deadline extension: Workshop on Self-sustaining Systems
	(S3) 2008
Message-ID: <47B877AA.9010502@hpi.uni-potsdam.de>

Please note that the deadline for paper submissions to the Workshop on 
Self-sustaining Systems (S3) 2008 has been extended. The new submission 
deadline in now

     *** February 24, 2008 ***

For more information on S3, please visit 
http://www.swa.hpi.uni-potsdam.de/s3/

Regards,
Kim Rose and Robert Hirschfeld


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=


[Call for Papers]


*** Workshop on Self-sustaining Systems (S3) 2008 ***

May 15-16, 2008
Potsdam, Germany
http://www.swa.hpi.uni-potsdam.de/s3/

-- Call for papers:

The Workshop on Self-sustaining Systems (S3) is a forum for discussion 
of topics relating to computer systems and languages that are able to 
bootstrap, implement, modify, and maintain themselves. One property of 
these systems is that their implementation is based on small but 
powerful abstractions; examples include (amongst others) 
Squeak/Smalltalk, COLA, Klein/Self, PyPy/Python, Rubinius/Ruby, and 
Lisp. Such systems are the engines of their own replacement, giving 
researchers and developers great power to experiment with, and explore 
future directions from within, their own small language kernels.

S3 will be take place May 15-16, 2008 at the Hasso-Plattner-Institute in 
Potsdam, Germany. It is an exciting opportunity for researchers and 
practitioners interested in self-sustaining systems to meet and share 
their knowledge, experience, and ideas for future research and development.

-- Invited talk:

Ian Piumarta: Late-bound Object Lambda Architectures (Viewpoints 
Research Institute, USA)

-- Submissions and proceedings:

S3 invites submissions of high-quality papers reporting original 
research, or describing innovative contributions to, or experience with, 
self-sustaining systems, their implementation, and their application. 
Papers that depart significantly from established ideas and practices 
are particularly welcome.

Submissions must not have been published previously and must not be 
under review for any another refereed event or publication. The program 
committee will evaluate each contributed paper based on its relevance, 
significance, clarity, and originality. Revised papers will be published 
as post-proceedings in the Springer LNCS series.

Papers should be submitted electronically via EasyChair at 
http://www.easychair.org/conferences/?conf=s3 in PDF format. Submissions 
must be written in English (the official language of the workshop) and 
must not exceed 20 pages. They should use the LNCS format, templates for 
which are available at http://www.springer.de/comp/lncs/authors.html.

-- Venue:

Hasso-Plattner-Institut (Potsdam, Germany)

-- Important dates:

Submission of papers: February 24, 2008
Author notification: April 11, 2008
Revised papers due: April 25, 2008

S3 workshop: May 15-16, 2008

Final papers for LNCS post-proceedings due: June 6, 2008

-- Chairs:

* Robert Hirschfeld (Hasso-Plattner-Institut Potsdam, Germany)
* Kim Rose (Viewpoints Research Institute, USA)

-- Program committee:

* Johan Brichau, Universite Catholique de Louvain, Belgium
* Pascal Costanza, Vrije Universiteit Brussel, Belgium
* Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium
* Stephane Ducasse, INRIA Lille, France
* Michael Haupt, Hasso-Plattner-Institut, Germany
* Robert Hirschfeld, Hasso-Plattner-Institut, Germany
* Dan Ingalls, Sun Microsystems Laboratories, USA
* Martin von L?wis, Hasso-Plattner-Institut, Germany
* Hidehiko Masuhara, University of Tokyo, Japan
* Ian Piumarta, Viewpoints Research Institute, USA
* David Ungar, IBM, USA

-- Registration fees:

Early (until April 18, 2008)
* Regular participants: EUR 160
* Students: EUR 80

Late (after April 18, 2008)
* Regular participants: EUR 170
* Students: EUR 90



From lac at openend.se  Mon Feb 18 19:07:56 2008
From: lac at openend.se (Laura Creighton)
Date: Mon, 18 Feb 2008 19:07:56 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
Message-ID: <200802181807.m1II7uVd021024@theraft.openend.se>

When Sun announced this, I sent mail to John Rose at Sun, to see if
work on the Da Vinci Machine http://openjdk.java.net/projects/mlvm/
for Python and PyPy support would be acceptable.

The answer is favourable.

I wonder, who would be interested in making a team to work on the
challenge?  And what would you think would make a good proposal?

Laura

From holger at merlinux.de  Mon Feb 18 21:13:28 2008
From: holger at merlinux.de (holger krekel)
Date: Mon, 18 Feb 2008 21:13:28 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: <200802181807.m1II7uVd021024@theraft.openend.se>
References: <200802181807.m1II7uVd021024@theraft.openend.se>
Message-ID: <20080218201328.GJ17687@solar.trillke>

Hi Laura, hi everyone interested in PyPy/JVM and oo backends, 

as it happens, Carl Friedrich, Maciej and me discussed
about it this morning :)  We know that Anto is interested 
in the Da-Vinci JVM challenge.  So I sent a note to Anto and 
Niko asking about their availability - they are in vacation
right now.  

A possible idea to go on might be this: 

a) interested people get together to work out technical PyPy/Da-Vinci 
   related promises given a project span of 6 months (18th March 
   till 4th August 2008).  

   Deadline for proposals is 2nd March.  We can send 
   drafts earlier to Sun's mailing list.  See 
   already existing proposals here: 

   http://mail.openjdk.java.net/pipermail/challenge-discuss/2008-February/thread.html

   Note: Sun grants money at the end of the project (~August 2008), 
   depending on project outcome and ranking/comparison with other
   projects. 

b) we could plan for 2-3 PyPy sprints with a major subtopic of implementing 
   the proposal's promises.  let's aim at travel+subsistence 
   reimbursment for proposal active people. 

c) developers working on proposal topics could note down their 
   hours and we use this a indication for money distribution in 
   case of success.  (Sun ranks the implemented projects at the 
   end of the project and money is distributed according to it) 

I can offer to help with writing a proposal and also with communication 
and organisation of administrative and legal issues, probably
through "merlinux", a company driven by long time 
PyPy and py.test people, see http://merlinux.de . 

However, IMO the main current issue is to come up with possible 
goals and technical promises.  Any comments and ideas about
possible PyPy/JVM Da-Vinci technical goals?  

Otherwise i guess we may need to wait for Anto and 
Niko getting back. 

best & cheers, 

holger


On Mon, Feb 18, 2008 at 19:07 +0100, Laura Creighton wrote:
> When Sun announced this, I sent mail to John Rose at Sun, to see if
> work on the Da Vinci Machine http://openjdk.java.net/projects/mlvm/
> for Python and PyPy support would be acceptable.
> 
> The answer is favourable.
> 
> I wonder, who would be interested in making a team to work on the
> challenge?  And what would you think would make a good proposal?
> 
> Laura
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 

-- 
Holger Krekel - freelance manager and programmer 
merlinux GmbH: http://merlinux.de 
py.test: http://pytest.org 
py lib: http://pylib.org 
PyPy: http://codespeak.net/pypy 

From lac at openend.se  Tue Feb 19 00:37:43 2008
From: lac at openend.se (Laura Creighton)
Date: Tue, 19 Feb 2008 00:37:43 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: Message from holger krekel <holger@merlinux.de> of "Mon,
	18 Feb 2008 21:13:28 +0100." <20080218201328.GJ17687@solar.trillke> 
References: <200802181807.m1II7uVd021024@theraft.openend.se>
	<20080218201328.GJ17687@solar.trillke> 
Message-ID: <200802182337.m1INbhHW032479@theraft.openend.se>

In a message of Mon, 18 Feb 2008 21:13:28 +0100, holger krekel writes:

<lots of good stuff>

<what looks to me like a completely workable plan>

>However, IMO the main current issue is to come up with possible 
>goals and technical promises.  Any comments and ideas about
>possible PyPy/JVM Da-Vinci technical goals?  

yes, we need to hear from Samuele who is on John Rose's lead team for
making adaptation to the JVM as suits dynamic languages.  How this
and the Da Vinci Machine hang together is not clear to me --
at least the last time I thought I understood it, Samuele made it
clear to me that my model of what was going on was a grossly
oversimplified model of what really was.

>Otherwise i guess we may need to wait for Anto and 
>Niko getting back. 

Maybe, but maybe we ought to get started ahead of time.  I happen to
know Samuele's workload for 2 weeks from now -- we have an in
house design sprint.  If we could get some things settled ahead of
time, this week, then this might be easier for him.

Also, one reason I posted this here was that I was interested in hearing
from people who are on this list, but so far not large contributors
to the PyPy code base.   Interested Lurkers, as it were.  Are
there any of you out there?  Does this sound like something any of you
are interested in?

Laura

>
>best & cheers, 
>
>holger

From chris at chris-lamb.co.uk  Wed Feb 20 18:39:01 2008
From: chris at chris-lamb.co.uk (Chris Lamb)
Date: Wed, 20 Feb 2008 17:39:01 +0000
Subject: [pypy-dev] Sun's Open JDK Challenge
 http://openjdk.java.net/challenge/
In-Reply-To: <200802182337.m1INbhHW032479@theraft.openend.se>
References: <200802181807.m1II7uVd021024@theraft.openend.se>
	<20080218201328.GJ17687@solar.trillke>
	<200802182337.m1INbhHW032479@theraft.openend.se>
Message-ID: <20080220173901.07bcd88b@sakaki.chris-lamb.co.uk>

Laura Creighton wrote:

> Interested Lurkers, as it were.  Are there any of you out there?  Does
> this sound like something any of you are interested in?

It sounds very interesting, but it seems that projects must be contributed
under Sun's "Contributor Agreement", which doesn't sound particularly
awesome IMHO.


Regards,

-- 
Chris Lamb, UK                                       chris at chris-lamb.co.uk
                                                            GPG: 0x634F9A20
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20080220/402226cc/attachment.pgp 

From jacob at openend.se  Wed Feb 20 18:47:09 2008
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Wed, 20 Feb 2008 18:47:09 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: <20080220173901.07bcd88b@sakaki.chris-lamb.co.uk>
References: <200802181807.m1II7uVd021024@theraft.openend.se>
	<200802182337.m1INbhHW032479@theraft.openend.se>
	<20080220173901.07bcd88b@sakaki.chris-lamb.co.uk>
Message-ID: <200802201847.10043.jacob@openend.se>

onsdagen den 20 februari 2008 skrev Chris Lamb:
> Laura Creighton wrote:
> > Interested Lurkers, as it were.  Are there any of you out there?  Does
> > this sound like something any of you are interested in?
>
> It sounds very interesting, but it seems that projects must be contributed
> under Sun's "Contributor Agreement", which doesn't sound particularly
> awesome IMHO.
>
>
> Regards,

I can't really see what is wrong with it. They pay you to produce code. They 
can do anything they like with it, including selling it or giving it away for 
free. You can do anything you like with it, including selling it, or giving 
it away for free.

Sounds reasonable to me.

Jacob Hall?n

From cfbolz at gmx.de  Wed Feb 20 19:34:04 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 20 Feb 2008 19:34:04 +0100
Subject: [pypy-dev] Sun's Open JDK
	Challenge	http://openjdk.java.net/challenge/
In-Reply-To: <200802201847.10043.jacob@openend.se>
References: <200802181807.m1II7uVd021024@theraft.openend.se>	<200802182337.m1INbhHW032479@theraft.openend.se>	<20080220173901.07bcd88b@sakaki.chris-lamb.co.uk>
	<200802201847.10043.jacob@openend.se>
Message-ID: <47BC729C.8070407@gmx.de>

Jacob Hall?n wrote:
> onsdagen den 20 februari 2008 skrev Chris Lamb:
>> Laura Creighton wrote:
>>> Interested Lurkers, as it were.  Are there any of you out there?  Does
>>> this sound like something any of you are interested in?
>> It sounds very interesting, but it seems that projects must be contributed
>> under Sun's "Contributor Agreement", which doesn't sound particularly
>> awesome IMHO.
>>
>>
>> Regards,
> 
> I can't really see what is wrong with it. They pay you to produce code. They 
> can do anything they like with it, including selling it or giving it away for 
> free. You can do anything you like with it, including selling it, or giving 
> it away for free.
> 
> Sounds reasonable to me.

I am not sure it is practical to put parts of PyPy under this agreement.
Which parts would that be? All of it (which would be hard, since we need
to find all contributors and have them sign it)? Only the new bits which
are rather useless by themselves?

Cheers,

Carl Friedrich

From pedronis at openend.se  Wed Feb 20 19:45:34 2008
From: pedronis at openend.se (Samuele Pedroni)
Date: Wed, 20 Feb 2008 19:45:34 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: <47BC729C.8070407@gmx.de>
References: <200802181807.m1II7uVd021024@theraft.openend.se>	<200802182337.m1INbhHW032479@theraft.openend.se>	<20080220173901.07bcd88b@sakaki.chris-lamb.co.uk>	<200802201847.10043.jacob@openend.se>
	<47BC729C.8070407@gmx.de>
Message-ID: <47BC754E.4020000@openend.se>

Carl Friedrich Bolz wrote:
> Jacob Hall?n wrote:
>   
>> onsdagen den 20 februari 2008 skrev Chris Lamb:
>>     
>>> Laura Creighton wrote:
>>>       
>>>> Interested Lurkers, as it were.  Are there any of you out there?  Does
>>>> this sound like something any of you are interested in?
>>>>         
>>> It sounds very interesting, but it seems that projects must be contributed
>>> under Sun's "Contributor Agreement", which doesn't sound particularly
>>> awesome IMHO.
>>>
>>>
>>> Regards,
>>>       
>> I can't really see what is wrong with it. They pay you to produce code. They 
>> can do anything they like with it, including selling it or giving it away for 
>> free. You can do anything you like with it, including selling it, or giving 
>> it away for free.
>>
>> Sounds reasonable to me.
>>     
>
> I am not sure it is practical to put parts of PyPy under this agreement.
> Which parts would that be? All of it (which would be hard, since we need
> to find all contributors and have them sign it)? Only the new bits which
> are rather useless by themselves?
>   
the rule as such makes most sense if the code is going into the Open JDK 
itself,
I'm not sure how it would apply to stuff in another codebase that uses 
the Open JDK.





From cfbolz at gmx.de  Wed Feb 20 19:50:08 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 20 Feb 2008 19:50:08 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: <47BC754E.4020000@openend.se>
References: <200802181807.m1II7uVd021024@theraft.openend.se>	<200802182337.m1INbhHW032479@theraft.openend.se>	<20080220173901.07bcd88b@sakaki.chris-lamb.co.uk>	<200802201847.10043.jacob@openend.se>
	<47BC729C.8070407@gmx.de> <47BC754E.4020000@openend.se>
Message-ID: <47BC7660.9040905@gmx.de>

Samuele Pedroni wrote:
> Carl Friedrich Bolz wrote:
>> I am not sure it is practical to put parts of PyPy under this agreement.
>> Which parts would that be? All of it (which would be hard, since we need
>> to find all contributors and have them sign it)? Only the new bits which
>> are rather useless by themselves?
>>   
> the rule as such makes most sense if the code is going into the Open JDK
> itself,
> I'm not sure how it would apply to stuff in another codebase that uses
> the Open JDK.

The problem is that all entries into the open JDK challenge have to be
put under the contributor agreement. I agree that it might not make
sense for PyPy since it is not going into the JDK. But that is what the
rules are. Maybe we should send a mail asking whether other models might
be possible.

Cheers,

Carl Friedrich

From jacob at openend.se  Wed Feb 20 20:32:12 2008
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Wed, 20 Feb 2008 20:32:12 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: <47BC7660.9040905@gmx.de>
References: <200802181807.m1II7uVd021024@theraft.openend.se>
	<47BC754E.4020000@openend.se> <47BC7660.9040905@gmx.de>
Message-ID: <200802202032.12553.jacob@openend.se>

onsdagen den 20 februari 2008 skrev Carl Friedrich Bolz:
> Samuele Pedroni wrote:
> > Carl Friedrich Bolz wrote:
> >> I am not sure it is practical to put parts of PyPy under this agreement.
> >> Which parts would that be? All of it (which would be hard, since we need
> >> to find all contributors and have them sign it)? Only the new bits which
> >> are rather useless by themselves?
> >
> > the rule as such makes most sense if the code is going into the Open JDK
> > itself,
> > I'm not sure how it would apply to stuff in another codebase that uses
> > the Open JDK.
>
> The problem is that all entries into the open JDK challenge have to be
> put under the contributor agreement. I agree that it might not make
> sense for PyPy since it is not going into the JDK. But that is what the
> rules are. Maybe we should send a mail asking whether other models might
> be possible.
>
> Cheers,
>
> Carl Friedrich

No, this is not quite true. You need to sign the contributor agreement, but 
that agreement only covers what goes into the Sun codebase. Wether they want 
to support something that goes into another codebase is an open question.

Jacob


From lac at openend.se  Wed Feb 20 20:57:28 2008
From: lac at openend.se (Laura Creighton)
Date: Wed, 20 Feb 2008 20:57:28 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: Message from Jacob =?iso-8859-1?q?Hall=E9n?= <jacob@openend.se> 
	of "Wed, 20 Feb 2008 20:32:12 +0100."
	<200802202032.12553.jacob@openend.se> 
References: <200802181807.m1II7uVd021024@theraft.openend.se>
	<47BC754E.4020000@openend.se> <47BC7660.9040905@gmx.de>
	<200802202032.12553.jacob@openend.se> 
Message-ID: <200802201957.m1KJvSZp010320@theraft.openend.se>

I can ask John Rose about it again.  Suggestions on how exactly to
word what we want to do?  What I think we want to do is to develop
under the MIT license and then, should there be parts of it that
just drop into the JDK, immediately relicense those bits under
the Sun Contributors license.

Is this correct?

Laura


From cfbolz at gmx.de  Wed Feb 20 22:19:33 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 20 Feb 2008 22:19:33 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: <200802202032.12553.jacob@openend.se>
References: <200802181807.m1II7uVd021024@theraft.openend.se>
	<47BC754E.4020000@openend.se> <47BC7660.9040905@gmx.de>
	<200802202032.12553.jacob@openend.se>
Message-ID: <47BC9965.3020002@gmx.de>

Jacob Hall?n wrote:
> onsdagen den 20 februari 2008 skrev Carl Friedrich Bolz:
>> Samuele Pedroni wrote:
>>> Carl Friedrich Bolz wrote:
>>>> I am not sure it is practical to put parts of PyPy under this agreement.
>>>> Which parts would that be? All of it (which would be hard, since we need
>>>> to find all contributors and have them sign it)? Only the new bits which
>>>> are rather useless by themselves?
>>> the rule as such makes most sense if the code is going into the Open JDK
>>> itself,
>>> I'm not sure how it would apply to stuff in another codebase that uses
>>> the Open JDK.
>> The problem is that all entries into the open JDK challenge have to be
>> put under the contributor agreement. I agree that it might not make
>> sense for PyPy since it is not going into the JDK. But that is what the
>> rules are. Maybe we should send a mail asking whether other models might
>> be possible.
>>
>> Cheers,
>>
>> Carl Friedrich
> 
> No, this is not quite true. You need to sign the contributor agreement, but 
> that agreement only covers what goes into the Sun codebase. Wether they want 
> to support something that goes into another codebase is an open question.

Please read the challenge rules at:
http://openjdk.java.net/challenge/rules/

Section 3 states:

B. The Entry, in its entirety, must be contributed under the terms of
the SCA.

Cheers,

Carl Friedrich

From jacob at openend.se  Wed Feb 20 23:04:41 2008
From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Wed, 20 Feb 2008 23:04:41 +0100
Subject: [pypy-dev] Sun's Open JDK Challenge
	http://openjdk.java.net/challenge/
In-Reply-To: <47BC9965.3020002@gmx.de>
References: <200802181807.m1II7uVd021024@theraft.openend.se>
	<200802202032.12553.jacob@openend.se> <47BC9965.3020002@gmx.de>
Message-ID: <200802202304.41107.jacob@openend.se>

onsdagen den 20 februari 2008 skrev Carl Friedrich Bolz:
> Jacob Hall?n wrote:
> > onsdagen den 20 februari 2008 skrev Carl Friedrich Bolz:
> >> Samuele Pedroni wrote:
> >>> Carl Friedrich Bolz wrote:
> >>>> I am not sure it is practical to put parts of PyPy under this
> >>>> agreement. Which parts would that be? All of it (which would be hard,
> >>>> since we need to find all contributors and have them sign it)? Only
> >>>> the new bits which are rather useless by themselves?
> >>>
> >>> the rule as such makes most sense if the code is going into the Open
> >>> JDK itself,
> >>> I'm not sure how it would apply to stuff in another codebase that uses
> >>> the Open JDK.
> >>
> >> The problem is that all entries into the open JDK challenge have to be
> >> put under the contributor agreement. I agree that it might not make
> >> sense for PyPy since it is not going into the JDK. But that is what the
> >> rules are. Maybe we should send a mail asking whether other models might
> >> be possible.
> >>
> >> Cheers,
> >>
> >> Carl Friedrich
> >
> > No, this is not quite true. You need to sign the contributor agreement,
> > but that agreement only covers what goes into the Sun codebase. Wether
> > they want to support something that goes into another codebase is an open
> > question.
>
> Please read the challenge rules at:
> http://openjdk.java.net/challenge/rules/
>
> Section 3 states:
>
> B. The Entry, in its entirety, must be contributed under the terms of
> the SCA.
>
> Cheers,
>
> Carl Friedrich


Ok, I stand corrected. I just read what was said in conjunction with the SCA.

Then it seems to make no sense to submit.

Jacob

From simon at arrowtheory.com  Thu Feb 21 16:23:43 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Thu, 21 Feb 2008 10:23:43 -0500
Subject: [pypy-dev] real-time gc ?
In-Reply-To: <20080204105812.GB30303@code0.codespeak.net>
References: <20080129090622.351039ab.simon@arrowtheory.com>
	<20080204105812.GB30303@code0.codespeak.net>
Message-ID: <20080221102343.3ab6f0ec.simon@arrowtheory.com>

On Mon, 4 Feb 2008 11:58:12 +0100
Armin Rigo <arigo at tunes.org> wrote:

> Hi Simon,
> 
> On Tue, Jan 29, 2008 at 09:06:22AM -0800, Simon Burton wrote:
> > I've taken some statistics on how the boehm gc performs with our current application,
> > it has terrible worst case performance, often stopping for hundreds of mS at a time.
> 
> What is the context?  It's unclear to me if you're talking about PyPy
> the Python interpreter, another stand-alone RPython program, or a
> CPython extension module translated from RPython.

Either of the first two.

> 
> In all cases but the latter you should definitely try our own GCs
> instead of Boehm.  The generational GC in particular, while not
> real-time in any sense of the word, is probably the one with the least
> amount of long pauses - most collections are very quick.

I am hoping to be working with top of tree again soon and will let you know what i find.

thankyou,

Simon.

> 
> 
> A bientot,
> 
> Armin

From niko at alum.mit.edu  Fri Feb 22 19:54:52 2008
From: niko at alum.mit.edu (Niko Matsakis)
Date: Fri, 22 Feb 2008 19:54:52 +0100
Subject: [pypy-dev] Catching MemoryError in RPython
Message-ID: <30CB7D83-7F05-4014-9CF3-E79624525997@alum.mit.edu>

I cannot seem to generate code that catches OOM errors in RPython.  I  
wrote a test like this:

     def test_memoryerror_due_to_oom(self):
         def fn():
             try:
                 lst = []
                 for i in range(sys.maxint): lst.append(i)
             except Exception: #MemoryError:
                 return "OK"
         assert self.interpret(fn, []) == "OK"

but the try:/except: seems to have disappeared by the time the JVM  
backend is invoked (it's not in the flow graphs anywhere).  I assume  
that it is getting stripped away by a deadcode pass at some point,  
though I'm not sure where that might be.  I haven't managed to concoct  
code that can fool it into keeping the exception block yet, but I'm  
pretty sure that if I only could, then the JVM backend would handle it  
correctly... :)

If you comment out the test that causes this out of memory condition,  
then test_builtin.py *almost* passes in pypy-jvm.  There are still two  
errors, though.  :)


Niko


From arigo at tunes.org  Sat Feb 23 13:44:03 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 23 Feb 2008 13:44:03 +0100
Subject: [pypy-dev] Catching MemoryError in RPython
In-Reply-To: <30CB7D83-7F05-4014-9CF3-E79624525997@alum.mit.edu>
References: <30CB7D83-7F05-4014-9CF3-E79624525997@alum.mit.edu>
Message-ID: <20080223124403.GA24894@code0.codespeak.net>

Hi Niko,

On Fri, Feb 22, 2008 at 07:54:52PM +0100, Niko Matsakis wrote:
> I cannot seem to generate code that catches OOM errors in RPython.

A known issue: the flow space doesn't consider that almost any operation
could raise MemoryError, and thus the except clause is never seen by it.
The same issue applies to try:finally:.

The workaround is to use a function call:

    def fn():
        try:
            do_stuff()
        except MemoryError:
            return True
        else:
            return False

    def do_stuff():
        ...


A bientot,

Armin.

From holger at merlinux.de  Sat Feb 23 14:33:23 2008
From: holger at merlinux.de (holger krekel)
Date: Sat, 23 Feb 2008 14:33:23 +0100
Subject: [pypy-dev] Catching MemoryError in RPython
In-Reply-To: <20080223124403.GA24894@code0.codespeak.net>
References: <30CB7D83-7F05-4014-9CF3-E79624525997@alum.mit.edu>
	<20080223124403.GA24894@code0.codespeak.net>
Message-ID: <20080223133323.GB17687@solar.trillke>

Hi Armin, 

On Sat, Feb 23, 2008 at 13:44 +0100, Armin Rigo wrote:
> On Fri, Feb 22, 2008 at 07:54:52PM +0100, Niko Matsakis wrote:
> > I cannot seem to generate code that catches OOM errors in RPython.
> 
> A known issue: the flow space doesn't consider that almost any operation
> could raise MemoryError, and thus the except clause is never seen by it.
> The same issue applies to try:finally:.
> 
> The workaround is to use a function call:
> 
>     def fn():
>         try:
>             do_stuff()
>         except MemoryError:
>             return True
>         else:
>             return False
> 
>     def do_stuff():
>         ...

the "range" and "lst.append" in Niko's example are not 
considered to be function calls?  Is it (also) the explicit 
catching of MemoryError here that is important? 

holger

From holger at merlinux.de  Sun Feb 24 14:48:17 2008
From: holger at merlinux.de (holger krekel)
Date: Sun, 24 Feb 2008 14:48:17 +0100
Subject: [pypy-dev] pypy/jvm Da-Vinci proposal
Message-ID: <20080224134817.GH17687@solar.trillke>

Hi all, 

it seems that Anto and Niko are back - so together with CF 
we now all plan to be around IRC at  

    #pypy-sync monday 25th Feb afternoon (GMT).  

Topic is to discuss and maybe work on a PyPy proposal for 
Sun's Da-Vinci challenge.  

To cut the copyright/legal discussion short for now: let me try 
to draft a mail to Sun after or during the discussion tomorrow. 

Let's for now rather worry about the technical side: which PyPy/JVM/Da-Vinci 
related promises would we like to make for for the next six months?  

Note that Anto has created some interesting PyPy/JVM related
information to be found e.g. here: 

    http://codespeak.net/pypy/extradoc/planning/roadmap/goal_jython_ironpython_replacement.html

I suggest to use these as starting points and topics for more thinking 
and discussion - feel free to commit there, post here or otherwise 
discuss outside of regulated meetings, please. 

For those interested to see existing proposal drafts, 
here is one from today that deals with a "meta compiler for 
non-python languages": 

    http://mail.openjdk.java.net/pipermail/challenge-discuss/2008-February/000024.html

best & see you all, 

holger

-- 
Holger Krekel - freelance manager and programmer 
merlinux GmbH: http://merlinux.de 
py.test: http://pytest.org 
PyPy: http://codespeak.net/pypy 

From jacob at openend.se  Sun Feb 24 21:51:01 2008
From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=)
Date: Sun, 24 Feb 2008 21:51:01 +0100
Subject: [pypy-dev] Micro benchmarks on i386 and ppc
Message-ID: <200802242151.01261.jacob@openend.se>

Using the outputs of Leonardo's test run on i386 and a run of the same 
revision on Tuatara (ppc), I have calculated the speed ratios between the two 
architectures. For each ratio, there are 4 numbers that can vary:
- The speed of CPython for the test on ppc
- The speed of CPython for the test on i386
- The speed of PyPy for the test on ppc
- The speed of PyPy for the test on i386

In the list below, the first number is the ratio, with smaller numbers meaning 
that PyPy is doing better on i386 and larger meaning that PyPy is relatively 
better on ppc. The first number in parenthesis after the name of the test is 
relative performance to CPython on i386, the second number in parenthesis is 
relative performance on ppc.

I'm a bit surprised that some tests have a huge difference in relative 
performance on the two platforms. Maybe something worth looking into.

0.41 test_unicode.test_bad_case_python2_5() (0.11), (0.27)
0.47 test_unicode.test_find_worstcase() (0.58), (1.24)
0.47 test_unicode.test_find_pattern1() (0.37), (0.79)
0.53 test_unicode.test_count_worstcase() (0.48), (0.90)
0.70 test_unicode.test_find_pattern8() (0.80), (1.14)
0.71 test_unicode.test_find_pattern2() (0.80), (1.13)
0.71 test_unicode.test_find_pattern4() (0.80), (1.13)
0.71 test_unicode.test_find_pattern16() (0.81), (1.14)
0.76 test_dict.test_dict_instance_setattr_instance_dict() (1.96), (2.59)
0.76 test_exception.test_reraise() (1.24), (1.63)
0.79 test_list.test_list_setitem() (2.06), (2.61)
0.79 test_dict.test_dict_setitem1() (3.43), (4.32)
0.83 test_exception.test_raise_builtin_exception() (1.91), (2.31)
0.84 test_count1.test_call_nested_function() (1.80), (2.15)
0.84 test_exception.test_raise_user_exception() (1.91), (2.27)
0.84 test_bltn.test_call_sin() (1.24), (1.47)
0.86 test_count1.test_count_in_attr() (1.70), (1.97)
0.87 test_count1.test_call_nested_function_other_count() (1.86), (2.15)
0.87 test_formatting.test_dict_formatting() (2.12), (2.45)
0.87 test_formatting.test_simple_formatting() (0.60), (0.69)
0.87 test_count1.test_count_in_slot() (2.37), (2.72)
0.87 test_create1.test_simple_loop_with_old_style_class_creation() (2.72), 
(3.12)
0.88 test_count1.test_call_nested_function_many_args() (1.77), (2.02)
0.88 test_count1.test_call_function() (1.88), (2.13)
0.88 test_dict.test_dict_setitem2() (2.34), (2.65)
0.88 test_count1.test_call_function_with_arguments_in_cellvars() (1.29), 
(1.46)
0.89 test_count1.test_call_function_without_arguments_in_cellvars() (1.47), 
(1.66)
0.90 test_formatting.test_format_long() (0.43), (0.48)
0.90 test_create1.test_simple_loop_with_new_style_class_creation() (3.54), 
(3.93)
0.90 test_exception.test_except_specific_builtin_exception() (1.79), (1.98)
0.90 test_exception.test_except_specific_user_exception() (1.89), (2.09)
0.90 test_dict.test_dict_instance_setnewattr_instance_dict() (2.00), (2.21)
0.92 test_bltn.test_call_fabs() (1.33), (1.44)
0.93 test_create1.test_simple_loop_with_new_style_class_new() (1.89), (2.03)
0.93 test_dict.test_dict_creation_mode1() (1.93), (2.07)
0.94 test_dict.test_dict_instance_getattr_instance_dict() (2.18), (2.33)
0.95 test_exception.test_except_multiple_user_exception() (3.01), (3.17)
0.95 test_bltn.test_isinstance3() (1.60), (1.68)
0.95 test_count1.test_call_method_of_new_style_class() (1.62), (1.70)
0.95 test_count1.test_loop_unrolled() (1.25), (1.31)
0.97 test_exception.test_except_multiple_builtin_exception() (2.67), (2.76)
0.98 test_count1.test_loop() (1.93), (1.96)
0.99 test_exception.test_try_except_else() (2.24), (2.27)
0.99 test_create1.test_simple_loop() (2.60), (2.63)
1.00 test_count1.test_loop_other_count() (2.16), (2.17)
1.00 test_count1.test_count_in_dict() (2.06), (2.06)
1.00 test_exception.test_try_except_finally() (2.14), (2.13)
1.01 test_exception.test_try_except() (2.27), (2.25)
1.01 test_bltn.test_isinstance1() (1.36), (1.34)
1.02 test_bltn.test_isinstance2() (1.34), (1.32)
1.02 test_dict.test_dict_class_dict_getmethod() (1.58), (1.55)
1.03 test_dict.test_dict_creation_mode2() (2.02), (1.96)
1.05 test_count1.test_call_method_of_old_style_class() (1.59), (1.51)
1.06 test_exception.test_instantiate_builtin_exception() (1.11), (1.05)
1.06 test_list.test_list_getitem() (3.12), (2.95)
1.06 test_list.test_list_append() (2.39), (2.25)
1.07 test_dict.test_dict_getitem() (2.45), (2.29)
1.10 test_exception.test_instantiate_user_exception() (1.34), (1.22)
1.12 test_formatting.test_format_unicode() (0.45), (0.40)
1.15 test_list.test_list_slice() (2.80), (2.44)
1.16 test_dict.test_dict_raw_range() (1.68), (1.45)
1.17 test_count1.test_count_in_global2() (2.14), (1.83)
1.17 test_count1.test_count_in_global() (2.13), (1.82)
1.21 test_formatting.test_number_formatting() (1.62), (1.34)
1.23 test_dict.test_dict_creation_mode3() (1.03), (0.84)
1.31 test_formatting.test_repr_formatting() (4.61), (3.53)
1.51 test_list.test_list_extend() (1.40), (0.93)
2.03 test_dispatch.test_dispatch_nop() (1.36), (0.67)


Jacob

From tjreedy at udel.edu  Tue Feb 26 05:12:41 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Feb 2008 23:12:41 -0500
Subject: [pypy-dev] Sun's Open JDK
	Challengehttp://openjdk.java.net/challenge/
References: <200802181807.m1II7uVd021024@theraft.openend.se><47BC754E.4020000@openend.se>
	<47BC7660.9040905@gmx.de><200802202032.12553.jacob@openend.se>
	<47BC9965.3020002@gmx.de>
Message-ID: <fq03jq$vdu$1@ger.gmane.org>


"Carl Friedrich Bolz" <cfbolz at gmx.de> wrote in message 
news:47BC9965.3020002 at gmx.de...
|B. The Entry, in its entirety, must be contributed under the terms of the 
SCA.

As I read the rules, the proposal is intended to be a new (original) idea, 
to be implemented by writing *new* code.  The new code constitutes the 
entry.  The prize consitutes payment for the new code.

It appears to me that the entry may depend on open source code that is not 
part of the entry itself.  "Each Project must use only open source and free 
software tools and libraries with the exception of any non-free, encumbered 
binaries that are a part of the OpenJDK code base."  They clearly cannot 
expect entrants to submit already licensed code to their SCA.

I would simply be clear in the proposal what new work would constitute the 
'entry' and what old work not a part of the 'entry' it might draw upon.

>From the previous characterization of the SCA given in this thread, I would 
expect the the entry author(s) could also contribute it to the PyPy project 
if appropriate (ie, dual licence it).

tjr






From arigo at tunes.org  Tue Feb 26 15:54:23 2008
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 26 Feb 2008 15:54:23 +0100
Subject: [pypy-dev] Catching MemoryError in RPython
In-Reply-To: <20080223133323.GB17687@solar.trillke>
References: <30CB7D83-7F05-4014-9CF3-E79624525997@alum.mit.edu>
	<20080223124403.GA24894@code0.codespeak.net>
	<20080223133323.GB17687@solar.trillke>
Message-ID: <20080226145423.GA18888@code0.codespeak.net>

Hi Holger,

On Sat, Feb 23, 2008 at 02:33:23PM +0100, holger krekel wrote:
> the "range" and "lst.append" in Niko's example are not 
> considered to be function calls?

Good point.  The range() and lst.append() are function calls, so the
flow space keeps the MemoryError catching path.  (That's not the case
for the creation of the empty list done in the loop.)  However, what
occurs is that the rtyper kills these paths because rlist.py says that
append() cannot raise.  This could be changed in the rtyper to
systematically keep the MemoryError path (not just for append() but
generally).  However I think that going for half-fixes like this one is
not a good idea.  Either we do a 100% solution that would include the
flow space issue, or we keep the special rule about how we are supposed
to catch MemoryError in RPython.


A bientot,

Armin.

From arigo at tunes.org  Tue Feb 26 15:59:52 2008
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 26 Feb 2008 15:59:52 +0100
Subject: [pypy-dev] pypy/jvm Da-Vinci proposal
In-Reply-To: <20080224134817.GH17687@solar.trillke>
References: <20080224134817.GH17687@solar.trillke>
Message-ID: <20080226145952.GA20066@code0.codespeak.net>

Hi Holger,

On Sun, Feb 24, 2008 at 02:48:17PM +0100, holger krekel wrote:
>     #pypy-sync monday 25th Feb afternoon (GMT).  

Oups, too late :-)  I'd have been mostly lurking, though.

> For those interested to see existing proposal drafts, 
> here is one from today that deals with a "meta compiler for 
> non-python languages": 
> 
>     http://mail.openjdk.java.net/pipermail/challenge-discuss/2008-February/000024.html

It's for non-*Java* languages :-)


Armin

From anto.cuni at gmail.com  Tue Feb 26 20:33:29 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 26 Feb 2008 20:33:29 +0100
Subject: [pypy-dev] First draft of the openjdk proposal
Message-ID: <58e316a40802261133s163b7af0j87a13fba0728b46c@mail.gmail.com>

Hi all,

I just finished to write the first draft of the openjdk proposal; you
can find it under extradoc/proposal/openjdk-challenge.txt.

It would be nice to hear comments, remarks and suggetions from whoever
is interested as soon as possible, since the deadline is close (2nd of
march).

ciao,
Anto

From paul.degrandis at gmail.com  Wed Feb 27 15:15:22 2008
From: paul.degrandis at gmail.com (Paul deGrandis)
Date: Wed, 27 Feb 2008 09:15:22 -0500
Subject: [pypy-dev] First draft of the openjdk proposal
In-Reply-To: <58e316a40802261133s163b7af0j87a13fba0728b46c@mail.gmail.com>
References: <58e316a40802261133s163b7af0j87a13fba0728b46c@mail.gmail.com>
Message-ID: <9c0bb8a00802270615j19d02ee6we727f42eefb0312b@mail.gmail.com>

Thanks for doing this Anto!
I'll take a look at it later today or tomorrow

I just wanted to say, I totally plan on working the openjdk/DaVinci
challenge.  I've just been busy with the term papers and projects (the
term ends in two weeks), but soon I'll be ready to tear it up.

Paul

On 2/26/08, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Hi all,
>
>  I just finished to write the first draft of the openjdk proposal; you
>  can find it under extradoc/proposal/openjdk-challenge.txt.
>
>  It would be nice to hear comments, remarks and suggetions from whoever
>  is interested as soon as possible, since the deadline is close (2nd of
>  march).
>
>  ciao,
>  Anto
>  _______________________________________________
>  pypy-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/pypy-dev
>

From charles.nutter at sun.com  Wed Feb 27 18:42:48 2008
From: charles.nutter at sun.com (Charles Oliver Nutter)
Date: Wed, 27 Feb 2008 11:42:48 -0600
Subject: [pypy-dev] First draft of the openjdk proposal
In-Reply-To: <58e316a40802261133s163b7af0j87a13fba0728b46c@mail.gmail.com>
References: <58e316a40802261133s163b7af0j87a13fba0728b46c@mail.gmail.com>
Message-ID: <47C5A118.3080602@sun.com>

Antonio Cuni wrote:
> Hi all,
> 
> I just finished to write the first draft of the openjdk proposal; you
> can find it under extradoc/proposal/openjdk-challenge.txt.
> 
> It would be nice to hear comments, remarks and suggetions from whoever
> is interested as soon as possible, since the deadline is close (2nd of
> march).

Sorry I didn't get back to you here, but I just replied on the challenge 
list.

- Charlie

From regmi.amit at gmail.com  Thu Feb 28 18:42:26 2008
From: regmi.amit at gmail.com (amit)
Date: Thu, 28 Feb 2008 12:42:26 -0500
Subject: [pypy-dev] Seeking ideas for SoC 2008 participation with PyPy
Message-ID: <47C6F282.3040800@gmail.com>

Hi,

I am looking for some project Idea, that I could take up for SoC 2008.
I have a little PyPy experience, that involved working on the 
"clr-module-improvements" branch and was mentored by Antonio Cuni and Carl, 
which involved
    -   Enable importing of .NET libraries in PyPy.
    -   Enable Iterations on .NET objects
    -   Add Support for Generic Classes.

I would be glad to know if there could be some unexplored PyPy corner which 
would be easier for me to take up based on the work I did.
Waiting in anticipation.

Best Regards,
Amit

From regmi.amit at gmail.com  Fri Feb 29 14:51:31 2008
From: regmi.amit at gmail.com (amit regmi)
Date: Fri, 29 Feb 2008 08:51:31 -0500
Subject: [pypy-dev] Seeking ideas for SoC 2008 participation with PyPy
Message-ID: <df184c060802290551o7b334ccn30510591a15034c8@mail.gmail.com>

Hi,

I am looking for project Ideas, that I could take up for SoC 2008.
I have a little PyPy experience, working on the
"clr-module-improvements" branch and was mentored by Antonio Cuni and Carl,
and involved
   -   Enable importing of .NET libraries in PyPy.
   -   Enable Iterations on .NET objects
   -   Add Support for Generic Classes.

I would be glad to know if there could be some unexplored PyPy corner which
would be easier for me to take up based on the work I did.
Waiting in anticipation.

Best Regards,
Amit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20080229/cf4ab752/attachment.htm 

From stephan.diehl at gmx.net  Mon Mar  3 16:48:16 2008
From: stephan.diehl at gmx.net (Stephan Diehl)
Date: Mon, 3 Mar 2008 16:48:16 +0100
Subject: [pypy-dev] Tim Bray Blog
Message-ID: <200803031648.16761.stephan.diehl@gmx.net>

Hi all,

just found this (at reddit, what a time waster :-) ):
http://www.tbray.org/ongoing/When/200x/2008/03/03/Python-at-Sun
At the end, he's saying:
"Quick Python trivia question: Near as I can tell, Guido works half-time on 
Python over at Google. Is there anyone in the world, aside from Frank and 
Ted, getting paid to work full-time on Python?"
Within pypy, there are several people to work full time on python, aren't 
there?

Stephan

From cfbolz at gmx.de  Mon Mar  3 17:04:38 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 03 Mar 2008 17:04:38 +0100
Subject: [pypy-dev] Tim Bray Blog
In-Reply-To: <200803031648.16761.stephan.diehl@gmx.net>
References: <200803031648.16761.stephan.diehl@gmx.net>
Message-ID: <47CC2196.3020100@gmx.de>

Stephan Diehl wrote:
> Hi all,
> 
> just found this (at reddit, what a time waster :-) ):
> http://www.tbray.org/ongoing/When/200x/2008/03/03/Python-at-Sun
> At the end, he's saying:
> "Quick Python trivia question: Near as I can tell, Guido works half-time on 
> Python over at Google. Is there anyone in the world, aside from Frank and 
> Ted, getting paid to work full-time on Python?"
> Within pypy, there are several people to work full time on python, aren't 
> there?

Not since the EU project ended, no.

Cheers,

Carl Friedrich

From lac at openend.se  Mon Mar  3 19:05:27 2008
From: lac at openend.se (Laura Creighton)
Date: Mon, 03 Mar 2008 19:05:27 +0100
Subject: [pypy-dev] Tim Bray Blog
In-Reply-To: Message from Stephan Diehl <stephan.diehl@gmx.net> of "Mon,
	03 Mar 2008 16:48:16 +0100." <200803031648.16761.stephan.diehl@gmx.net>
References: <200803031648.16761.stephan.diehl@gmx.net> 
Message-ID: <200803031805.m23I5RZj012230@theraft.openend.se>

In a message of Mon, 03 Mar 2008 16:48:16 +0100, Stephan Diehl writes:
>Hi all,
>
>just found this (at reddit, what a time waster :-) ):
>http://www.tbray.org/ongoing/When/200x/2008/03/03/Python-at-Sun
>At the end, he's saying:
>"Quick Python trivia question: Near as I can tell, Guido works half-time 
>on 
>Python over at Google. Is there anyone in the world, aside from Frank and
> 
>Ted, getting paid to work full-time on Python?"
>Within pypy, there are several people to work full time on python, aren't
> 
>there?
>
>Stephan

There was as one time, and we hope there will be again.  But our EU
funding ran out last year.

Laura


From arigo at tunes.org  Wed Mar  5 19:13:44 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 5 Mar 2008 19:13:44 +0100
Subject: [pypy-dev] buffer protocol.
In-Reply-To: <47B0711D.1030402@genesilico.pl>
References: <47B0711D.1030402@genesilico.pl>
Message-ID: <20080305181344.GA13311@code0.codespeak.net>

Hi Maciek,

On Mon, Feb 11, 2008 at 05:00:29PM +0100, Maciek Fijalkowski wrote:
> I feel a bit clueless about buffer protocol. Current implemntation in 
> pypy is purely app-level, which means it's slow, but more importantly, 
> one cannot get raw, C-level address of it. I've tried approaching the 
> problem, but it seems that there are several serious issues.
> 
> * what's the future of buffer protocol?
> * do people use that? ctypes tests are relying on it. anybody knows 
> ctypes module implementation that relies on it?
> * testing suite sucks. we've got two tests, cpython is not any better in 
> this regard.
> 
> where to proceed?

The future of the buffer protocol, in Python 3000, is to be a bit
cleaned up and then made considerably more complicated and powerful
(being in this respect similar to most changes in Python 3000).

The PEP is here:

    http://svn.python.org/projects/peps/trunk/pep-3118.txt

I think we could still use this as a basis for thought.  An interesting
thing is that for any object supporting the buffer protocol, a Python
object of the new type "memoryview" can be built.  This is similar to
the existing "buffer" Python type but cleaner in the sense that it is
equivalent to the internal data structures -- in fact I'm not sure why
Python 3000 needs both an internal data structure and the memoryview
object type.

What we could do is thus add an interp-level class MemoryView which is a
subclass of Wrappable with an API to allow interp-level accesses.  To
get such a MemoryView from any Python object we'd use a space method,
space.memoryview_w().  A trivial extension to Python would be to have a
special method name associated to this, like '__memoryview__()',
allowing any Python type (built-in or user-defined) to return a
MemoryView for itself.

The API of MemoryView could be loosely based on (but much much simpler
than) the one in the PEP.


A bientot,

Armin.

From jgustak at gmail.com  Fri Mar  7 14:46:13 2008
From: jgustak at gmail.com (Jakub Gustak)
Date: Fri, 7 Mar 2008 14:46:13 +0100
Subject: [pypy-dev] summer of code 2008
Message-ID: <a26b53540803070546u71a97d5dq8ab22f669fb81fb0@mail.gmail.com>

Hello.

I am wondering if pypy is going to take part in Google Summer of Code this year.

I have an idea, which I would like to hack on (and already talked a
little about it with Armin).
But I sill am not sure if it makes much sense.

I will paste what I send to Armin some time ago:

msg << EOM
I have this weird idea of creating (yet another) scheme dialect on top of pypy
scheme interpreter.

Main features of this dialect would be:
 * strict (in definition) distinction between pure functions, and functions
  with side-effect.
 * parallelism support:
 * e.g. (parallel (f1 ...) (f2 ...)) would compute functions f1 and f2 on
       different processors (if available) and return list of results.
of course,
       f1 and f2 have to be pure functions in this case.
 * interpreter would start as many processes as processors available and
       parallel should schedule functions computation on this "nodes".
 * communication between nodes should be as much transparent as possible,
       so we can create nodes on different machines.
 * there is also another interesting pattern: map-reduce, or scatter-gather,
       which would be nice to have, if parallel back-end would be
capable of this.

We could go further, and even avoid using (parallel) construction, and
parallelize every pure function. Of course that will make sense, if computation
cost is greater than cost of spawning.

I am interested, in this, because lot of people are talking, that functional
programming is <the way> to control parallelism/concurrency, but i haven't seen
any functional language where programming "parallel applications" is like
eating cream and butter.

Similar idea is implemented in haskell, but haskell is lazy, and it works
somehow different:
http://www.haskell.org/ghc/docs/latest/html/users_guide/lang-parallel.html
http://www.haskell.org/ghc/docs/latest/html/libraries/parallel/Control-Parallel.html
EOM

Cheers,
Jakub Gustak

From niko at alum.mit.edu  Sat Mar  8 15:07:02 2008
From: niko at alum.mit.edu (Niko Matsakis)
Date: Sat, 8 Mar 2008 15:07:02 +0100
Subject: [pypy-dev] SPy (almost) works in the JVM
Message-ID: <CC6CDE4E-9D8F-48DE-AEBD-3FAF8796BD44@alum.mit.edu>

Just wanted to mention that I fixed several bugs holding SPy  
[Smalltalk in PyPy] back from working in the JVM.  The "mini.image"  
code now runs, but I'm not sure it does precisely the right thing, as  
the number of hash marks ("#") that are output differs in the JVM and  
CLI.  The biggest error was the fact that we were converting byte  
arrays into Java Strings using the default encoding, which I think is  
UTF-8 (or something).  The end result was that we did not read binary  
files correctly (hence the assertion errors in image decoding).

Here is the output of the JVM and the CLI on my machine:

;./targetimageloadingsmalltalk-jvm ../../lang/smalltalk/mini.image
################<pypy.lang.smalltalk.model.W_CompiledMethod object>
Going to execute 78 toplevel bytecodes
#####################################################################################################################################994035 
  bytecodes/sec; 70776 sends/sec

;./targetimageloadingsmalltalk-cli ../../lang/smalltalk/mini.image
################<pypy.lang.smalltalk.model.W_CompiledMethod object>
Going to execute 78 toplevel bytecodes
#################################################################################################################################################################################################2418379 
  bytecodes/sec; 106874 sends/sec

Anyway, getting closer...


Niko

From lac at openend.se  Sun Mar  9 10:24:21 2008
From: lac at openend.se (Laura Creighton)
Date: Sun, 09 Mar 2008 10:24:21 +0100
Subject: [pypy-dev] Possibly of use in the quest to parse pypy better
Message-ID: <200803090924.m299OL8x004075@theraft.openend.se>


------- Forwarded Message

From: Andrew Dalke <dalke at dalkescientific.com>
Date: Sun, 9 Mar 2008 03:29:29 +0100
To: testing-in-python at lists.idyll.org
X-Mailer: Apple Mail (2.753)
Subject: Re: [TIP] branch coverage

In late January I mentioned I was working on a parser for Python  
using PLY as the parser engine.  I wanted to experiment with branch  
coverage in Python by reading the source .py file, building a parse  
tree, adding instrumentation to the AST, and saving .pyc files with  
the instrumentation.  Then run the regression tests and see which  
branches weren't covered.

I'm not there yet.

What I have done is write a Python parser using PLY.  It's called  
"python4ply" and is available at:
   http://dalkescientific.com/Python/python4ply.html

What took even longer than writing the code was writing the  
tutorial.  It shows examples of how to change the lexer, parser, AST,  
and bytecode generation.  I worked through two examples  where I  
instrumented the code to improve testing.

The first of these assumes there are problems like this

     assert 0 not in results, "problem in: %r" % data

where the "data" variable should be "results".  Assert statements are  
rarely fully tested and so prone to failure if they ever occur.  My  
example transforms the code to something like

_$assert = Y
assert X, _$assert

where '$' is something that normal Python code can't generate but AST  
manipulation code can.

Details at: http://dalkescientific.com/Python/python4ply- 
tutorial.html#assert


The second shows a way to add statement coverage without using  
settrace.  Details at:

     http://dalkescientific.com/Python/python4ply-tutorial.html#coverage


I didn't get to the point where I could do branch coverage.  I didn't  
have a good enough use cases so wasn't sure what I should implement.   
Perhaps you've got an idea of how to do it, or something else  
involving parsing Python code?  (If so, take a look at the _ast  
module from the standard library.  It's probably a better solution.)   
If so, I hope this grammar and code helps.


Download:  http://dalkescientific.com/Python/python4ply.html
Tutorial:  http://dalkescientific.com/Python/python4ply-tutorial.html

				Andrew
				dalke at dalkescientific.com

------- End of Forwarded Message


From verwaest at iam.unibe.ch  Mon Mar 10 00:53:26 2008
From: verwaest at iam.unibe.ch (Toon Verwaest)
Date: Mon, 10 Mar 2008 00:53:26 +0100
Subject: [pypy-dev] module SomeObject setattr.
Message-ID: <20080310005326.teyya1pbwkc4g8g4@www.iam.unibe.ch>

Hi,

I'm not yet fully comfortable with the PyPy code so I just lay it out here:

I have written the following code in the Squeak VM (should change in 
the future but its just to show the "problem"):

<file objtable.py:>
objects = []

<file shadow.py:>
objtable.objects += [objofspecifictype]

This seems not to be translatable since this is translated to:
vX = setattr(objtable-module, 'objects', vY).

generally setattr are done on SomeInstance which nicely returns None -> 
SomeImpossibleValue -> the annotator is happy.
However, modules appear to be instances of SomeObject on which setattr 
returns a SomeObject -> type of vX gets broken.

(and the bug received is not very obvious since it says that the 
original type was (none)...)

cheers,
Toon

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From micahel at gmail.com  Mon Mar 10 01:02:44 2008
From: micahel at gmail.com (Michael Hudson)
Date: Mon, 10 Mar 2008 13:02:44 +1300
Subject: [pypy-dev] Fwd:  module SomeObject setattr.
In-Reply-To: <aa0dfd860803091702g1fc7435v17e639266643a0ea@mail.gmail.com>
References: <20080310005326.teyya1pbwkc4g8g4@www.iam.unibe.ch>
	<aa0dfd860803091702g1fc7435v17e639266643a0ea@mail.gmail.com>
Message-ID: <aa0dfd860803091702i6052f35ah4f1a8d0c6f8e5400@mail.gmail.com>

This was supposed to go to the list too...

---------- Forwarded message ----------
From: Michael Hudson <micahel at gmail.com>
Date: 10 Mar 2008 13:02
Subject: Re: [pypy-dev] module SomeObject setattr.
To: Toon Verwaest <verwaest at iam.unibe.ch>


On 10/03/2008, Toon Verwaest <verwaest at iam.unibe.ch> wrote:
 > Hi,
 >
 >  I'm not yet fully comfortable with the PyPy code so I just lay it out here:
 >
 >  I have written the following code in the Squeak VM (should change in
 >  the future but its just to show the "problem"):
 >
 >  <file objtable.py:>
 >  objects = []
 >
 >  <file shadow.py:>
 >  objtable.objects += [objofspecifictype]
 >
 >  This seems not to be translatable since this is translated to:
 >  vX = setattr(objtable-module, 'objects', vY).
 >
 >  generally setattr are done on SomeInstance which nicely returns None ->
 >  SomeImpossibleValue -> the annotator is happy.
 >  However, modules appear to be instances of SomeObject


This is, already, your problem: modules are not supported by RPython
 (anything that ends up as SomeObject() means translation is not going
 to work).

 It's a bit peculiar I guess that the module gets examined here, though
 I can see why it happens.  I think if you write
 objtable.objects.extend(...) you'll be happier.


 > on which setattr
 >  returns a SomeObject -> type of vX gets broken.


If anything, this should break even more noisily.


 >  (and the bug received is not very obvious since it says that the
 >  original type was (none)...)


I can believe this part :/

 Cheers,
 mwh

From arigo at tunes.org  Mon Mar 10 22:24:33 2008
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 10 Mar 2008 22:24:33 +0100
Subject: [pypy-dev] SPy (almost) works in the JVM
In-Reply-To: <CC6CDE4E-9D8F-48DE-AEBD-3FAF8796BD44@alum.mit.edu>
References: <CC6CDE4E-9D8F-48DE-AEBD-3FAF8796BD44@alum.mit.edu>
Message-ID: <20080310212433.GA26040@code0.codespeak.net>

Hi Niko,

On Sat, Mar 08, 2008 at 03:07:02PM +0100, Niko Matsakis wrote:
> the number of hash marks ("#") that are output differs in the JVM and  
> CLI.

That's fine: the Smalltalk code in mini.image is a benchmark that runs
until some total amount of time is elapsed, so the number of iterations
(and #'s) varies according to the performance.


A bientot,

Armin

From arigo at tunes.org  Mon Mar 10 22:34:39 2008
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 10 Mar 2008 22:34:39 +0100
Subject: [pypy-dev] module SomeObject setattr.
In-Reply-To: <20080310005326.teyya1pbwkc4g8g4@www.iam.unibe.ch>
References: <20080310005326.teyya1pbwkc4g8g4@www.iam.unibe.ch>
Message-ID: <20080310213439.GB26040@code0.codespeak.net>

Hi Toon,

On Mon, Mar 10, 2008 at 12:53:26AM +0100, Toon Verwaest wrote:
> <file objtable.py:>
> objects = []
> 
> <file shadow.py:>
> objtable.objects += [objofspecifictype]

You cannot mutate global lists in RPython code.  (It's fine to do it
outside the RPython-compiled function, of course.)  If you need global
state it should be put on a prebuilt global instance of a class that you
defined.

Note btw that the Smalltalk interpreter in PyPy should probably be
modified to have a 'space' object passed around; the 'space' would play
precisely the role of the global state.  That's far cleaner than what we
came up with during the Bern sprint, which is to patch module-global
objects from various places.  For example, all the globals in
objspace.py should go on such a 'space', and the 'space' would be
populated either by the image loader or manually for smaller tests.


A bientot,

Armin.

From arigo at tunes.org  Thu Mar 13 09:39:32 2008
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 13 Mar 2008 09:39:32 +0100
Subject: [pypy-dev] Next sprints
Message-ID: <20080313083931.GA8571@code0.codespeak.net>

Hi all,

This e-mail is about starting the discussion for ideas and plans about
the next sprints.

What we have planned at the moment:

* The next PyPy sprint is going to be a small one at PyCon
  (http://us.pycon.org/2008/sprints/projects/#pypy).

* In Leysin, beginning of April, will probably be held a "Paper writing"
  sprint.  Confirmation and more details should come soon (from Michael
  Leuschel and Carl Friedrich).

Another idea I now about is a follow-up Smalltalk-in-RPython (SPy)
sprint (nothing concrete though).

It seems that we could starting planning for a regular PyPy development
sprint soon.  I'm never tired of inviting people to Leysin but I guess
we could pick another location now (additionally, there are many places
where Spring is a nice season, but Leysin is not one of them).


A bientot,

Armin

From fijal at genesilico.pl  Fri Mar 14 01:39:07 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 14 Mar 2008 01:39:07 +0100
Subject: [pypy-dev] Next sprints
In-Reply-To: <20080313083931.GA8571@code0.codespeak.net>
References: <20080313083931.GA8571@code0.codespeak.net>
Message-ID: <47D9C92B.5090407@genesilico.pl>

Armin Rigo wrote:
> Hi all,
>
> This e-mail is about starting the discussion for ideas and plans about
> the next sprints.
>
> What we have planned at the moment:
>
> * The next PyPy sprint is going to be a small one at PyCon
>   (http://us.pycon.org/2008/sprints/projects/#pypy).
>
> * In Leysin, beginning of April, will probably be held a "Paper writing"
>   sprint.  Confirmation and more details should come soon (from Michael
>   Leuschel and Carl Friedrich).
>
> Another idea I now about is a follow-up Smalltalk-in-RPython (SPy)
> sprint (nothing concrete though).
>
> It seems that we could starting planning for a regular PyPy development
> sprint soon.  I'm never tired of inviting people to Leysin but I guess
> we could pick another location now (additionally, there are many places
> where Spring is a nice season, but Leysin is not one of them).
>   
I've got an idea of organizing sprint in Poland. Ideal would be summer 
holiday (easier to organize a room on any uni). I can always organize 
something in Warsaw, but I think I'll look for different places (next to 
my climbing place sounds like a perfect hit).

Cheers,
fijal


:.


From niko at alum.mit.edu  Fri Mar 14 18:41:55 2008
From: niko at alum.mit.edu (Niko Matsakis)
Date: Fri, 14 Mar 2008 18:41:55 +0100
Subject: [pypy-dev] summer of code 2008
In-Reply-To: <a26b53540803070546u71a97d5dq8ab22f669fb81fb0@mail.gmail.com>
References: <a26b53540803070546u71a97d5dq8ab22f669fb81fb0@mail.gmail.com>
Message-ID: <9AD5C127-5A97-4D9D-9012-FFABFDA97B70@alum.mit.edu>

> I have an idea, which I would like to hack on (and already talked a
> little about it with Armin).
> But I sill am not sure if it makes much sense.

I'm not sure about the technical details, but it sounds like a great  
idea.  I think PyPy could benefit from more projects that explore  
parallelism.


Niko

From cfbolz at gmx.de  Mon Mar 17 17:23:55 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 17 Mar 2008 17:23:55 +0100
Subject: [pypy-dev] Summer of Code
Message-ID: <47DE9B1B.3070104@gmx.de>

Hi all,

there was some discussion about possible Summer of Code topics on the
#pypy channel today. We had several project ideas:

 - work on the JIT assembler backends. There are several possible
   projects here, like writing a much simpler i386 backend and adding
   float support to that; or writing a x64 backend; or writing a new
   i386 backend that uses the more modern bits of the processor for
   arithmetic (like SSE and friends).

 - support ctypes on more backends. right now ctypes is supported only
   when compiling PyPy to C. A nice project would be to support it when
   compiling to .NET. That's not too hard, the only thing needed is to
   port a small module that does the actual invocation of external
   libraries (a related project would be to port this module to Jython
   or IronPython to get support for ctypes there).

 - support some of the more recent features of CPython. A nice project
   would be to support all of the remaining CPython 2.5 features and
   port over the changes to the standard library (which includes
   re-writing some of the C-coded modules in RPython or pure Python).
   Another possibility would be to even start looking into the 2.6
   features.

 - Anto proposed to make the javascript backendopt more useful by
   turning it into something more frameworkish. I am sceptical of the
   viability of the task, it would need a very clever student and
   probably a lot of work too.

What do you all think of these ideas? Are there other ideas around? I
know that Jakub posted something a while ago (haven't looked in detail
yet, though). In my opinion we should try to avoid "yet another
incomplete interpreter implentation" for this year.

The timeplan is to collect some more ideas, then write a better
description of them and make a blog post about it (say around Thursday).
Sounds like a plan?

Cheers,

Carl Friedrich

From cfbolz at gmx.de  Mon Mar 17 17:35:56 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 17 Mar 2008 17:35:56 +0100
Subject: [pypy-dev] Summer of Code
In-Reply-To: <47DE9B1B.3070104@gmx.de>
References: <47DE9B1B.3070104@gmx.de>
Message-ID: <47DE9DEC.1000007@gmx.de>

Carl Friedrich Bolz wrote:
>  - Anto proposed to make the javascript backendopt more useful by
                                          ^ should be "backend",


From anto.cuni at gmail.com  Mon Mar 17 21:31:01 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 17 Mar 2008 21:31:01 +0100
Subject: [pypy-dev] Summer of Code
In-Reply-To: <47DE9B1B.3070104@gmx.de>
References: <47DE9B1B.3070104@gmx.de>
Message-ID: <47DED505.6030303@gmail.com>

Carl Friedrich Bolz wrote:

> What do you all think of these ideas? Are there other ideas around?

maybe another possible task could be to port an existing external 
library to pypy, either by writing a mixed module or by using ctypes; 
possible choices are pygtk, postgresql/mysql DB API, pygame, etc.

I don't know whether they are small enough to fit into SoC, though.

ciao Anto

From tjreedy at udel.edu  Mon Mar 17 23:15:55 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 17 Mar 2008 18:15:55 -0400
Subject: [pypy-dev] Summer of Code
References: <47DE9B1B.3070104@gmx.de>
Message-ID: <frmqiq$4vp$1@ger.gmane.org>


"Carl Friedrich Bolz" <cfbolz at gmx.de> wrote in message 
news:47DE9B1B.3070104 at gmx.de...
||   Another possibility would be to even start looking into the 2.6
|   features.

As near as I can tell, 2.6 is pretty much the union of 2.5 and 3.0.
Most of the new features that do not change old code have been backported. 
Many (most?) of the changes that do will be available as future imports.
So a feature-complete 3.0 might be easier than a feature-complete 2.6.

tjr




From santagada at gmail.com  Mon Mar 17 23:30:13 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Mon, 17 Mar 2008 19:30:13 -0300
Subject: [pypy-dev] Summer of Code
In-Reply-To: <47DE9B1B.3070104@gmx.de>
References: <47DE9B1B.3070104@gmx.de>
Message-ID: <0833D64F-B872-43EA-8610-518D9C70E708@gmail.com>


On 17/03/2008, at 13:23, Carl Friedrich Bolz wrote:
> What do you all think of these ideas? Are there other ideas around? I
> know that Jakub posted something a while ago (haven't looked in detail
> yet, though). In my opinion we should try to avoid "yet another
> incomplete interpreter implentation" for this year.


- Threads using the new gc seems pretty important
- Numeric on pypy - the scientific comunity could use pypy because  
they don't need many extensions, mostly scipy wich numeric is the center
- Porting ZODB to pypy - this might show a lot of bugs of pypy as it  
is a really complex extension.

[]'s
--
Leonardo Santagada





From stephan.diehl at gmx.net  Tue Mar 18 08:23:39 2008
From: stephan.diehl at gmx.net (Stephan Diehl)
Date: Tue, 18 Mar 2008 08:23:39 +0100
Subject: [pypy-dev] Summer of Code
In-Reply-To: <47DE9B1B.3070104@gmx.de>
References: <47DE9B1B.3070104@gmx.de>
Message-ID: <200803180823.39315.stephan.diehl@gmx.net>

Am Montag, 17. M?rz 2008 17:23:55 schrieb Carl Friedrich Bolz:
[...]
>
> What do you all think of these ideas? Are there other ideas around? I
> know that Jakub posted something a while ago (haven't looked in detail
> yet, though). In my opinion we should try to avoid "yet another
> incomplete interpreter implentation" for this year.
>
Maybe Jakub could be persuaded to implement his ideas as a python extension.

Just my 2c

Stephan 


From cfbolz at gmx.de  Tue Mar 18 10:53:46 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 18 Mar 2008 10:53:46 +0100
Subject: [pypy-dev] Summer of Code
In-Reply-To: <0833D64F-B872-43EA-8610-518D9C70E708@gmail.com>
References: <47DE9B1B.3070104@gmx.de>
	<0833D64F-B872-43EA-8610-518D9C70E708@gmail.com>
Message-ID: <47DF912A.7070501@gmx.de>

Leonardo Santagada wrote:
> On 17/03/2008, at 13:23, Carl Friedrich Bolz wrote:
>> What do you all think of these ideas? Are there other ideas around? I
>> know that Jakub posted something a while ago (haven't looked in detail
>> yet, though). In my opinion we should try to avoid "yet another
>> incomplete interpreter implentation" for this year.
>
>
> - Threads using the new gc seems pretty important

That's true, but I fear that this is a very hard (probably too hard)
task.

> - Numeric on pypy - the scientific comunity could use pypy because
> they don't need many extensions, mostly scipy wich numeric is the center
> - Porting ZODB to pypy - this might show a lot of bugs of pypy as it
> is a really complex extension.

Both of these are potentially very annoying. Either you write a
PyPy-specific extension module (using rffi), then you have tons of fun
because we don't have separate compilation yet. Or you do this with
ctypes, which probably has way too much overhead for computation-bound
applications, especially with our slow implementation.

I guess rewriting big projects in ctypes should only be done for
projects where it seems likely that the original authors of the project
will later care for the ctypes rewrite. Otherwise you get the effect
that now happens for pygame: the ctypes rewrite of it is more or less
abandoned.

Cheers,

Carl Friedrich

From jgustak at gmail.com  Tue Mar 18 15:34:05 2008
From: jgustak at gmail.com (Jakub Gustak)
Date: Tue, 18 Mar 2008 15:34:05 +0100
Subject: [pypy-dev] Summer of Code
In-Reply-To: <47DE9B1B.3070104@gmx.de>
References: <47DE9B1B.3070104@gmx.de>
Message-ID: <a26b53540803180734g779d1f58n9db4e08773a743dc@mail.gmail.com>

>  What do you all think of these ideas? Are there other ideas around? I
>  know that Jakub posted something a while ago (haven't looked in detail
>  yet, though). In my opinion we should try to avoid "yet another
>  incomplete interpreter implentation" for this year.

I fully agree.

Still, what I proposed was not attempt to create another one, but keep
existing one alive.
It is bothering me, that scheme interpreter is incomplete, but the
problem is, i could not
find a use case for it (except of Thousand Parsecs internal scripting
language - if none
other - faster - scheme implementation available).

So if PyPy is such a nice platform to experiment, let's do it, and
play with different
approaches to implement parallel paradigms, or whatsoever. There is no
need for yet another
scheme interpreter (even complete) if it doesn't have any interesting features.
That's my 3 cents for now.

Cheers,
Jakub Gustak

From paul.degrandis at gmail.com  Tue Mar 18 15:57:00 2008
From: paul.degrandis at gmail.com (Paul deGrandis)
Date: Tue, 18 Mar 2008 10:57:00 -0400
Subject: [pypy-dev]  Summer of Code
In-Reply-To: <9c0bb8a00803180756i5988842fyc3d5e18d806bdba3@mail.gmail.com>
References: <47DE9B1B.3070104@gmx.de>
	<a26b53540803180734g779d1f58n9db4e08773a743dc@mail.gmail.com>
	<9c0bb8a00803180756i5988842fyc3d5e18d806bdba3@mail.gmail.com>
Message-ID: <9c0bb8a00803180757u2e35d356t76131f0e7f8a7c8f@mail.gmail.com>

I agree that new approaches to concurrency would be cool.  Possible
 SoCers might start by looking at PARELY (an actor model impl for
 python that runs on PyPy as is).
 http://osl.cs.uiuc.edu/parley/

 Regards,

Paul



 On 3/18/08, Jakub Gustak <jgustak at gmail.com> wrote:
 > >  What do you all think of these ideas? Are there other ideas around? I
 >  >  know that Jakub posted something a while ago (haven't looked in detail
 >  >  yet, though). In my opinion we should try to avoid "yet another
 >  >  incomplete interpreter implentation" for this year.
 >
 >
 > I fully agree.
 >
 >  Still, what I proposed was not attempt to create another one, but keep
 >  existing one alive.
 >  It is bothering me, that scheme interpreter is incomplete, but the
 >  problem is, i could not
 >  find a use case for it (except of Thousand Parsecs internal scripting
 >  language - if none
 >  other - faster - scheme implementation available).
 >
 >  So if PyPy is such a nice platform to experiment, let's do it, and
 >  play with different
 >  approaches to implement parallel paradigms, or whatsoever. There is no
 >  need for yet another
 >  scheme interpreter (even complete) if it doesn't have any
interesting features.
 >  That's my 3 cents for now.
 >
 >  Cheers,
 >
 > Jakub Gustak
 >
 > _______________________________________________
 >  pypy-dev at codespeak.net
 >  http://codespeak.net/mailman/listinfo/pypy-dev
 >

From arigo at tunes.org  Tue Mar 18 19:45:06 2008
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 18 Mar 2008 19:45:06 +0100
Subject: [pypy-dev] Summer of Code
In-Reply-To: <47DF912A.7070501@gmx.de>
References: <47DE9B1B.3070104@gmx.de>
	<0833D64F-B872-43EA-8610-518D9C70E708@gmail.com>
	<47DF912A.7070501@gmx.de>
Message-ID: <20080318184506.GA29606@code0.codespeak.net>

Hi Carl,

On Tue, Mar 18, 2008 at 10:53:46AM +0100, Carl Friedrich Bolz wrote:
> > - Threads using the new gc seems pretty important
> 
> That's true, but I fear that this is a very hard (probably too hard)
> task.

No, the basic task is only to ensure that threads and our new GCs can be
used in the same pypy-c.  This doesn't mean that the GCs should be
multithreaded or even thread-safe, if we take care of only allocating
objects when the GIL is held (Global Interpreter Lock).

A fully multithreaded GC is step two.  It would be good for RPython
programs that are more freely multithreaded, like Alexander's attempt to
just remove the GIL from pypy-c and not care about crashes (using
explicit locks at application-level to prevent them).  Getting there is
more work indeed.  Any student with JikesRVM MMTk experience around? :-)


A bientot,

Armin.

From anto.cuni at gmail.com  Sun Mar 23 11:01:54 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 23 Mar 2008 11:01:54 +0100
Subject: [pypy-dev] Problems with modified socket.py
In-Reply-To: <6815022a0803221535k2b186650l5c9a77f07d6f9770@mail.gmail.com>
References: <6815022a0803221535k2b186650l5c9a77f07d6f9770@mail.gmail.com>
Message-ID: <47E62A92.3020606@gmail.com>

Leandro Lameiro wrote:
> Hi Antonio.
> 
> My name is Leandro Lameiro and I've been a long time observer of PyPy. 
> This weekend I tried to see where I could contribute to PyPy, so I 
> decided to try and run some apps and see what is missing.

Hi Leandro.

Thank you for your interest in PyPy; every contribution is welcome, 
either patches or bug report!

I'm CCing pypy-dev, so that other developers can read it as well.

> The first thing I tried to run was ez_setup.py ( 
> http://peak.telecommunity.com/dist/ez_setup.py ), a simple script that 
> downloads and installs setuptools. It is a really short script but it 
> uses various modules such as md5, socket, urllib, os etc.
> 
> After translating PyPy (with allworkingmodules and socket switches) I 
> found that ez_setup.py will raise and exception in PyPy but not in 
> CPython (tried with 2.4 and 2.5).
> 
> Here is the traceback:
> 
> (virtualenv)lameiro at liria:~/projetos/ez_setup$ python ez_setup.py
> Downloading 
> http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c8-py2.4.egg
> Traceback (most recent call last):
>  File "?", line 34, in run_toplevel
>  File "ez_setup.py", line 267, in <module>
>    main(sys.argv[1:])
>  File "ez_setup.py", line 204, in main
>    os.unlink(egg)
>  File "ez_setup.py", line 198, in main
>    egg = download_setuptools(version, delay=0)
>  File "ez_setup.py", line 152, in download_setuptools
>    if src: src.close()
>  File "/home/lameiro/projetos/pypy-dist/lib-python/2.4.1/urllib.py", 
> line 804, in close
>    if self.fp: self.fp.close()
>  File 
> "/home/lameiro/projetos/pypy-dist/lib-python/modified-2.4.1/socket.py", 
> line 242, in close
>    s._drop()
> AttributeError: 'HTTPResponse' object has no attribute '_drop'
> (virtualenv)lameiro at liria:~/projetos/ez_setup$
> 
> After investigating a little, I found that you modified socket.py and 
> introduced this _drop call and also defined it in socket._closedsocket 
> class (an empty method), but that doesn't seem to be enough since other 
> types are going there (in this case HTTPResponse, but maybe others too).

indeed, you are right; the relevant _drop method is defined in 
pypy-dist/pypy/module/_socket/interp_socket.py (it's called _drop_w), 
but I guess we should take care of cases like this in which the object 
doesn't provide a _drop method.

> The revision in which this modification was introduced is 
> https://codespeak.net/viewvc/pypy/dist/lib-python/modified-2.4.1/socket.py?r1=16842&r2=34429 
> <https://codespeak.net/viewvc/pypy/dist/lib-python/modified-2.4.1/socket.py?r1=16842&r2=34429> 
> , and I suppose it was you because the commiter login is "ac". :)

sorry, your guess is wrong :-).
My login name is "antocuni", "ac" stands for Anders Chrigstr?m.
That's also why I don't know much about this and I'm CCing pypy-dev :-)

> Anyway, thanks for the great work and I hope the next time I can 
> contribute a patch instead of a bug report.

thank to you again for the help!
Feel free to come to #pypy on freenode if you want to talk with us by IRC.

ciao,
Anto

From jgustak at gmail.com  Wed Mar 26 15:04:27 2008
From: jgustak at gmail.com (Jakub Gustak)
Date: Wed, 26 Mar 2008 15:04:27 +0100
Subject: [pypy-dev]  summer of code 2008
In-Reply-To: <a26b53540803260703y7eec7193o7de647b6e8d65d8@mail.gmail.com>
References: <a26b53540803070546u71a97d5dq8ab22f669fb81fb0@mail.gmail.com>
	<9AD5C127-5A97-4D9D-9012-FFABFDA97B70@alum.mit.edu>
	<a26b53540803260703y7eec7193o7de647b6e8d65d8@mail.gmail.com>
Message-ID: <a26b53540803260704i5b732dd1v33fd5146244d289@mail.gmail.com>

>  I'm not sure about the technical details, but it sounds like a grea
>  idea.  I think PyPy could benefit from more projects that explore
>  parallelism.

Still. No sign of PyPy at the PSF ideas page.
Is there any chance, PyPy will participate?
__
Jakub Gustak

From fijal at genesilico.pl  Fri Mar 28 06:01:50 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Fri, 28 Mar 2008 06:01:50 +0100
Subject: [pypy-dev] enchancing flow objspace
Message-ID: <47EC7BBE.30400@genesilico.pl>

what about making flow objspace understand xxx and XXX as global 
variables that always raise XxxError() instead of exploding? would 
simplify translation greatly.

:.


From cfbolz at gmx.de  Fri Mar 28 10:32:55 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 28 Mar 2008 10:32:55 +0100
Subject: [pypy-dev] summer of code 2008
In-Reply-To: <a26b53540803260704i5b732dd1v33fd5146244d289@mail.gmail.com>
References: <a26b53540803070546u71a97d5dq8ab22f669fb81fb0@mail.gmail.com>	<9AD5C127-5A97-4D9D-9012-FFABFDA97B70@alum.mit.edu>	<a26b53540803260703y7eec7193o7de647b6e8d65d8@mail.gmail.com>
	<a26b53540803260704i5b732dd1v33fd5146244d289@mail.gmail.com>
Message-ID: <47ECBB47.3040307@gmx.de>

Hi Jakub,

Jakub Gustak wrote:
>>  I'm not sure about the technical details, but it sounds like a grea
>>  idea.  I think PyPy could benefit from more projects that explore
>>  parallelism.
> 
> Still. No sign of PyPy at the PSF ideas page.
> Is there any chance, PyPy will participate?

Sorry, due to some extreme disorganization we haven't done much for SoC
this year. However, if you have a good proposal it seems likely that one
of us will mentor it, if you register. We also updated the project ideas
page a bit:

http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html

We should probably put a link to that to the PSF wiki page.

Cheers,

Carl Friedrich

From cfbolz at gmx.de  Fri Mar 28 10:42:01 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 28 Mar 2008 10:42:01 +0100
Subject: [pypy-dev] enchancing flow objspace
In-Reply-To: <47EC7BBE.30400@genesilico.pl>
References: <47EC7BBE.30400@genesilico.pl>
Message-ID: <47ECBD69.90304@gmx.de>

Maciek Fijalkowski wrote:
> what about making flow objspace understand xxx and XXX as global 
> variables that always raise XxxError() instead of exploding? would 
> simplify translation greatly.

weeeeeell. That sounds like an extreme hack to me. Maybe we should do
something else, like 'assert 0, "XXX"' instead of just putting XXXs
everywhere.

Cheers,

Carl Friedrich


From anto.cuni at gmail.com  Fri Mar 28 10:53:58 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 28 Mar 2008 10:53:58 +0100
Subject: [pypy-dev] enchancing flow objspace
In-Reply-To: <47ECBD69.90304@gmx.de>
References: <47EC7BBE.30400@genesilico.pl> <47ECBD69.90304@gmx.de>
Message-ID: <47ECC036.8000207@gmail.com>

Carl Friedrich Bolz wrote:

> weeeeeell. That sounds like an extreme hack to me. Maybe we should do
> something else, like 'assert 0, "XXX"' instead of just putting XXXs
> everywhere.

+1


From grobinson at goombah.com  Fri Mar 28 15:33:49 2008
From: grobinson at goombah.com (Gary Robinson)
Date: Fri, 28 Mar 2008 10:33:49 -0400
Subject: [pypy-dev] JIT SoC
Message-ID: <20080328103349439579.a77430c3@goombah.com>

I saw the suggestion "A good project would be to give the 386 backend a good refactoring, simplifying it, then add support for missing features like floating-point arithmetic." on the list of PyPy independent project suggestions.

I'm wondering two things:

1) Is it realistically possible that after the SoC, if that work was taken on by someone, PyPy would be usable in an environment where high-speed floating-point calcs were needed? (The environment I have in mind wouldn't be putting much other stress on python -- it doesn't need a GUI, etc.) It sounds to me like it's far too big a project, but I don't know the existing code at all so I don't want to assume -- it would be great if it could be done.

2) I understand Google is suppling $5000 per project ($4500 for the student). Is there any reason an outside company can't throw a little extra money into the pot for a particular task to sweeten the motivation for that task to be done in the SoC?

-- 

Gary Robinson
CTO
Emergent Music, LLC
grobinson at goombah.com
207-942-3463
Company: http://www.goombah.com
Blog:    http://www.garyrobinson.net

From cfbolz at gmx.de  Fri Mar 28 15:42:53 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 28 Mar 2008 15:42:53 +0100
Subject: [pypy-dev] JIT SoC
In-Reply-To: <20080328103349439579.a77430c3@goombah.com>
References: <20080328103349439579.a77430c3@goombah.com>
Message-ID: <47ED03ED.7060408@gmx.de>

Gary Robinson wrote:
> I saw the suggestion "A good project would be to give the 386 backend
> a good refactoring, simplifying it, then add support for missing
> features like floating-point arithmetic." on the list of PyPy
> independent project suggestions.
> 
> I'm wondering two things:
> 
> 1) Is it realistically possible that after the SoC, if that work was
> taken on by someone, PyPy would be usable in an environment where
> high-speed floating-point calcs were needed? (The environment I have
> in mind wouldn't be putting much other stress on python -- it doesn't
> need a GUI, etc.) It sounds to me like it's far too big a project,
> but I don't know the existing code at all so I don't want to assume
> -- it would be great if it could be done.

That really depends. In the mid-term this should be possible, and one of
the things missing is indeed a better assembler backend (since right now
we have no float support at all). Note however that PyPy doesn't support
Numeric Python, so depending what you want to do this might be a problem.

> 2) I understand Google is suppling $5000 per project ($4500 for the
> student). Is there any reason an outside company can't throw a little
> extra money into the pot for a particular task to sweeten the
> motivation for that task to be done in the SoC?

I guess there is no reason, apart from the fact that it sounds a bit
unfair to other students participating in SoC. What I could imagine is
having a sprint during the SoC-period and asking that company to pay for
the travel of the student or something similar. If you are concretely
interested, you might want to take it off-list and discuss with the
caretaker group: pypy-ct at codespeak.net

Cheers,

Carl Friedrich

From lac at openend.se  Fri Mar 28 17:09:14 2008
From: lac at openend.se (Laura Creighton)
Date: Fri, 28 Mar 2008 17:09:14 +0100
Subject: [pypy-dev] JIT SoC
In-Reply-To: Message from Carl Friedrich Bolz <cfbolz@gmx.de> 
	of "Fri, 28 Mar 2008 15:42:53 +0100." <47ED03ED.7060408@gmx.de> 
References: <20080328103349439579.a77430c3@goombah.com>
	<47ED03ED.7060408@gmx.de> 
Message-ID: <200803281609.m2SG9Er7009372@theraft.openend.se>

In a message of Fri, 28 Mar 2008 15:42:53 +0100, Carl Friedrich Bolz writes:
>Gary Robinson wrote:
>> 2) I understand Google is suppling $5000 per project ($4500 for the
>> student). Is there any reason an outside company can't throw a little
>> extra money into the pot for a particular task to sweeten the
>> motivation for that task to be done in the SoC?

Another thing to note -- it is possible to fund the PyPy project 
directly.  If the idea of paying the people who are already working on the
project -- but aren't students -- to do things you would like done
has appeal, we can probably  work something out.

Laura Creighton


From brunogola at gmail.com  Sat Mar 29 04:30:54 2008
From: brunogola at gmail.com (Bruno Gola)
Date: Sat, 29 Mar 2008 00:30:54 -0300
Subject: [pypy-dev] PyPy GSoC Application: Supporting Python 2.5 features in
	PyPy
Message-ID: <b27f1a340803282030o28c4a5e2yfbc801df234afe5@mail.gmail.com>

Hi,

I'm a under-grad student (3th year) from "Centro Universit?rio SENAC"
from S?o Palo, Brazil. I'm very interested in the PyPy project and
since last year I have been planning to get involved. Well, this plans
has transformed into my graduation project that I'll start working
next semester. My graduation project idea is to study two different
implementations (CPython (C/Static/lower-level) and Pypy
(Python/Dynamic/higher-level)) of a programming language (Python) and
compare the difficult to hack both. I intend to study both and suggest
optimizations to the implementations and show that it is easier to
modify the higher-level implementation.

Well, I told you this because my GSoC application is all about the
Pypy interpreter (or the Pypy implementation of Python ) ;).

I wrote some ideas to the project, so I ask if anyone could review it.


PROPOSAL:

Support new features of Python 2.5 in PyPy
==========================================

* Motivation and goal

	Pypy's interpreter currently supports some of the "new" Python
features introduced in Python 2.5. Also, the Python 2.5 changes to
standard library have not been ported yet.
	
	This proposal goal is to bring 2.5 features and changes to Pypy interpreter.

* Features

	Some features are already supported (for example __index__). I'm
proposing to make Pypy support all the remaining features from 2.5.
	
	There are changes made to the python standard library[1] that haven't
been ported yet because of the lack of 2.5 features, so I'm proposing
to port this changes (some major modules added are sqlite3, ctypes and
ElementTree) too.
	
	So, what I'm proposing:
		* Support all Python 2.5 features
		* "Sync" PyPy's standard library with CPython

[1] http://docs.python.org/whatsnew/modules.html	

* Coding	

	I started looking into the list of what is new in Python 2.5 to see
if any new feature would need a special attention. My plans are to
start coding the changes to the language (not to the stdlib) because I
think it doesn't make much sense to implement the stdlib without full
language support and it seems that porting the stdlib will not be much
trouble (not as much as supporting some features :-)).
	
	I intend to start coding some simple feature (like PEP-342 that seems
easy to support) so I can study the PyPy interpreter better. Then I'll
code the other features (PEPs 314[is this really needed in Pypy?]],
328, 342, 343[or what is missing] and 352). [am I missing any other
feature?].
	
	With all features supported (and tested :-)) I'll code the stdlib
changes. At this point I think I'll be familiar with the interpreter
code so I pretend to port first the sqlite3 module.
	
	I like follow TDD (Test-driven development) so tests will not be a
problem (I mean, it will be part of the final code, of course :)).

PROPOSAL END.

Thanks,
-- 
Bruno Fialho Marques Gola <brunogola at gmail.com>
bgola` @ irc.freenode.org
http://www.brunogola.com.br
Cel: (11) 9294-5883

From brunogola at gmail.com  Sat Mar 29 04:33:09 2008
From: brunogola at gmail.com (Bruno Gola)
Date: Sat, 29 Mar 2008 00:33:09 -0300
Subject: [pypy-dev] PyPy GSoC Application: Supporting Python 2.5
	features in PyPy
In-Reply-To: <b27f1a340803282030o28c4a5e2yfbc801df234afe5@mail.gmail.com>
References: <b27f1a340803282030o28c4a5e2yfbc801df234afe5@mail.gmail.com>
Message-ID: <b27f1a340803282033v419855e3m53dda87ed476d9a1@mail.gmail.com>

ta ai

On Sat, Mar 29, 2008 at 12:30 AM, Bruno Gola <brunogola at gmail.com> wrote:
> Hi,
>
>  I'm a under-grad student (3th year) from "Centro Universit?rio SENAC"
>  from S?o Palo, Brazil. I'm very interested in the PyPy project and
>  since last year I have been planning to get involved. Well, this plans
>  has transformed into my graduation project that I'll start working
>  next semester. My graduation project idea is to study two different
>  implementations (CPython (C/Static/lower-level) and Pypy
>  (Python/Dynamic/higher-level)) of a programming language (Python) and
>  compare the difficult to hack both. I intend to study both and suggest
>  optimizations to the implementations and show that it is easier to
>  modify the higher-level implementation.
>
>  Well, I told you this because my GSoC application is all about the
>  Pypy interpreter (or the Pypy implementation of Python ) ;).
>
>  I wrote some ideas to the project, so I ask if anyone could review it.
>
>
>  PROPOSAL:
>
>  Support new features of Python 2.5 in PyPy
>  ==========================================
>
>  * Motivation and goal
>
>         Pypy's interpreter currently supports some of the "new" Python
>  features introduced in Python 2.5. Also, the Python 2.5 changes to
>  standard library have not been ported yet.
>
>         This proposal goal is to bring 2.5 features and changes to Pypy interpreter.
>
>  * Features
>
>         Some features are already supported (for example __index__). I'm
>  proposing to make Pypy support all the remaining features from 2.5.
>
>         There are changes made to the python standard library[1] that haven't
>  been ported yet because of the lack of 2.5 features, so I'm proposing
>  to port this changes (some major modules added are sqlite3, ctypes and
>  ElementTree) too.
>
>         So, what I'm proposing:
>                 * Support all Python 2.5 features
>                 * "Sync" PyPy's standard library with CPython
>
>  [1] http://docs.python.org/whatsnew/modules.html
>
>  * Coding
>
>         I started looking into the list of what is new in Python 2.5 to see
>  if any new feature would need a special attention. My plans are to
>  start coding the changes to the language (not to the stdlib) because I
>  think it doesn't make much sense to implement the stdlib without full
>  language support and it seems that porting the stdlib will not be much
>  trouble (not as much as supporting some features :-)).
>
>         I intend to start coding some simple feature (like PEP-342 that seems
>  easy to support) so I can study the PyPy interpreter better. Then I'll
>  code the other features (PEPs 314[is this really needed in Pypy?]],
>  328, 342, 343[or what is missing] and 352). [am I missing any other
>  feature?].
>
>         With all features supported (and tested :-)) I'll code the stdlib
>  changes. At this point I think I'll be familiar with the interpreter
>  code so I pretend to port first the sqlite3 module.
>
>         I like follow TDD (Test-driven development) so tests will not be a
>  problem (I mean, it will be part of the final code, of course :)).
>
>  PROPOSAL END.
>
>  Thanks,
>  --
>  Bruno Fialho Marques Gola <brunogola at gmail.com>
>  bgola` @ irc.freenode.org
>  http://www.brunogola.com.br
>  Cel: (11) 9294-5883
>



-- 
Bruno Fialho Marques Gola <brunogola at gmail.com>
http://www.brunogola.com.br
Cel: (11) 9294-5883

From lac at openend.se  Sat Mar 29 07:48:18 2008
From: lac at openend.se (Laura Creighton)
Date: Sat, 29 Mar 2008 07:48:18 +0100
Subject: [pypy-dev] Looks like Macij's talk was a hit with the attendees
Message-ID: <200803290648.m2T6mI2H005271@theraft.openend.se>

Alas, I still cannot get them to remove my name from the list of
presenters.

Forward from Python-organisers

From: Brian Ray <bray at sent.com>
To: pycon-organizers Pycon-Organizers <Pycon-organizers at python.org>

Since we are nearing the 50% participation mark on survey collection,  
I think it is fair to start analyzing the results. Once the paper  
forms are hand entered, I am happy to say, we will pass 50% user  
participation in the survey. This alone, speaks volumes.

Here is a link, if you prefer raw data view:
	
	<http://tinyurl.com/2eoab2>

<snip>

OVERVIEW
========

On the highest level, here are some comparisons between the last year  
data (found here <http://wiki.python.org/moin/PyCon2007/Feedback>):

  - 500+ Surveys were filled out this time, versus 150, last time
  - There were slightly more first timers this year 60.6% than last  
year 56%
  - Overall satisfaction dropped slightly from 3.62/4 to 3.28/4
  - Satisfaction with keynotes dropped from 3.28/4 to 2.78/4
  - Satisfaction with talks dropped from 3.12/4 to 2.81/4
  - Satisfaction with the network dropped sharply from 3.01/4 to 2.19/4
  - Satisfaction with the food dropped from  3.0/4 to 2.61/4
  - Likelihood of attending next year dropped from 3.4/4 to 3.26/4
  - There was an increase of attendance of tutorials from 31.1% to 44.4%

The results on preference of Only weekdays, including one weekend  
days, and including two weekend days, remained about the same.

The top 5 talks (including keynotes) were:

  - Keynote: Guido van Rossum 	41.4%	173
  - Core Python Containers -- Under the Hood      Mr. Raymond D  
Hettinger	20.1%	84
  - Keynote: Brian 'Fitz' Fitzpatrick	18.9%	79
  - The State of Django      Adrian Holovaty	17.7%	74
  - Keynote: Van Lindberg 	16.3%	68

The top 10 talks not including keynotes:

  - Core Python Containers -- Under the Hood      Mr. Raymond D  
Hettinger	20.1%	84
  - The State of Django      Adrian Holovaty	17.7%	74
  - Sights and sounds with pyglet  Mr. Alex Holkner	14.8%	62
  - The State of PyPy    Maciej Fijalkowski (merlinux GmbH) ; Ms.  
Laura A Creighton (Open End AB)	13.6%	57
  - SQLAlchemy 0.4 and Beyond    Mike Bayer	13.2%	55
  - Using PyGame and PySight to Create an Interactive Halloween  
Activity    Mr. John Harrison (Insight Industries)	12.9%	54
  - Consuming HTML     Ian Bicking (The Open Planning Project)	11.2%	47
  - nose: testing for the lazy coder    Jason Pellerin (Leapfrog  
Online)	10.5%	44
  - IronPython: The Road Ahead     Mr. Jim Hugunin (Microsoft)	10.5%	44
  - Tahoe: A Robust Distributed Secure Filesystem      Brian Warner	 
10.3%	43
------- End of Forwarded Message


From lac at openend.se  Sat Mar 29 07:49:59 2008
From: lac at openend.se (Laura Creighton)
Date: Sat, 29 Mar 2008 07:49:59 +0100
Subject: [pypy-dev] Looks like Macij's talk was a hit with the attendees
In-Reply-To: Message from Laura Creighton <lac@openend.se> of "Sat,
	29 Mar 2008 07:48:18 +0100."
	<200803290648.m2T6mI2H005271@theraft.openend.se> 
References: <200803290648.m2T6mI2H005271@theraft.openend.se> 
Message-ID: <200803290649.m2T6nx8N005375@theraft.openend.se>


And looks like I cannot spell this morning, sorry about that Maciej

Laura


From 2007b at usenet.alexanderweb.de  Sat Mar 29 14:41:09 2008
From: 2007b at usenet.alexanderweb.de (Alexander Schremmer)
Date: Sat, 29 Mar 2008 14:41:09 +0100
Subject: [pypy-dev] PyPy GSoC Application: Supporting Python 2.5
	features in PyPy
References: <b27f1a340803282030o28c4a5e2yfbc801df234afe5@mail.gmail.com>
Message-ID: <mgt1c5-mgv.ln1@tuxpaddy.alexanderweb.homeip.net>

Bruno Gola wrote:


> Pypy's interpreter currently supports some of the "new" Python
> features introduced in Python 2.5. Also, the Python 2.5 changes to
> standard library have not been ported yet.

There is a complete list of missing things here (see Catch up with CPython
2.5/2.6"):
http://codespeak.net/pypy/extradoc/planning/roadmap/goal_cpython_replacement.html

I guess it would be wise to refer to the tasks in your application.

Kind regards,
Alexander


From fijal at genesilico.pl  Mon Mar 31 17:23:52 2008
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Mon, 31 Mar 2008 17:23:52 +0200
Subject: [pypy-dev] [pypy-svn] r53194 - pypy/branch/jit-hotpath/pypy/tool
In-Reply-To: <20080331085144.9C0EA1684DA@codespeak.net>
References: <20080331085144.9C0EA1684DA@codespeak.net>
Message-ID: <47F10208.2030500@genesilico.pl>

arigo at codespeak.net wrote:
> Author: arigo
> Date: Mon Mar 31 10:51:44 2008
> New Revision: 53194
>
> Modified:
>    pypy/branch/jit-hotpath/pypy/tool/udir.py
> Log:
> This logic seems slightly broken - I end up with tons of different
> usession names in my /tmp, not just branch names.  Anyway I prefer
> the old behavior by far, so with this checkin I can set
> PYPY_USESSION_BASENAME to a simple '-' and get it back.
>
>
> Modified: pypy/branch/jit-hotpath/pypy/tool/udir.py
> ==============================================================================
> --- pypy/branch/jit-hotpath/pypy/tool/udir.py	(original)
> +++ pypy/branch/jit-hotpath/pypy/tool/udir.py	Mon Mar 31 10:51:44 2008
> @@ -17,9 +17,11 @@
>      else:
>          return basename.split('/')[-2]
>  
> -try:
> -    basename = '-' + svn_info(py.path.svnwc(py.magic.autopath().dirpath()).info().url) + '-'
> -except:
> -    basename = '-'
> +basename = os.environ.get('PYPY_USESSION_BASENAME')
> +if not basename:
> +    try:
> +        basename = '-' + svn_info(py.path.svnwc(py.magic.autopath().dirpath()).info().url) + '-'
> +    except:
> +        basename = '-'
>  
>  udir = local.make_numbered_dir(prefix='usession' + basename, keep=3)
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>
> :.
>
>   

That's because it contained tons of bugs and over checkins you got 
different dirs :-( I hope it's fixed by now

:.