From fijall at gmail.com  Sun Oct  5 16:22:43 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 5 Oct 2008 07:22:43 -0700
Subject: [pypy-dev] sprint
Message-ID: <693bc9ab0810050722n62818c9eh3fe32edf6b7727c7@mail.gmail.com>

Hello from the third world.

I failed to embark the train today. It seems that these days you need
to buy ticket at least 3 days in advance in order to get to berlin. I
managed though to get some place on ukrainian train, which goes 3h
longer, but at least have some places. I hope to be in ddorf tomorrow
around lunch.

Cheers,
fijal

From ondrej at certik.cz  Thu Oct  9 02:19:35 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 9 Oct 2008 02:19:35 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
Message-ID: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>

Hi,

if you want to get many bugreports, here's how. :)

$ git clone git://git.sympy.org/sympy.git
$ cd sympy
$ python
>>> import sympy
>>> sympy.test("sympy/core")
[...]
============ tests finished: 292 passed, 9 xfailed in 1.40 seconds =============
True
>>>


This works in python2.4, python2.5 (there are some failures in
python2.6) and it *doesn't* require py.test, just pure python.

Now try pypy:

$ ../pypy-dist/pypy-c    # compile pypy to C and then to executable
Python 2.4.1 (pypy 1.0.0 build 58853) on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``don't post links to patents
here''
>>>> import sympy
>>>> sympy.test("sympy/core")
[ dozens pages of errors ]


To ease debugging, use bin/test:

$ ../pypy-dist/pypy-c bin/test sympy/core/

and use it like py.test.

Basically, most of them reduce to this problem:

$ python
Python 2.4.5 (#2, Sep 29 2008, 15:31:42)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sympy import Function
>>> f = Function("f")
>>>

$ ../pypy-dist/pypy-c
Python 2.4.1 (pypy 1.0.0 build 58853) on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``things will be nice and stuff''
>>>> from sympy import Function
>>>> f = Function("f")
[boom ...]
    l.append(iter_copy(i))
  File "/home/ondra/repos/sympy/sympy/core/multidimensional.py", line
43, in iter_copy
    l.append(iter_copy(i))
  File "/home/ondra/repos/sympy/sympy/core/multidimensional.py", line
43, in iter_copy
    l.append(iter_copy(i))
  File "/home/ondra/repos/sympy/sympy/core/multidimensional.py", line
42, in iter_copy
    if hasattr(i, "__iter__"):
RuntimeError: internal error: <RuntimeError object at 0xb03b60>
>>>>




Ondrej

From ondrej at certik.cz  Fri Oct 10 13:00:12 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 13:00:12 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
Message-ID: <85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>

On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I'm working on 2.5 compatibility branch right now. Due to incompatible
> changes, I suggest that we say we don't provide --pyversion option any
> more and simply compile 2.5 compatible interpreter. I don't really see
> benefits of providing 2.4 right now (after a brief discussion with
> Armin).
>
> If anyone objects, please do that now.

Just a curious question --- what kind of features are python2.4 only
that doesn't work in python2.5? I though that except hashing,
python2.5 is backwards compatible.

Ondrej

From ondrej at certik.cz  Fri Oct 10 13:12:19 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 13:12:19 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
Message-ID: <85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>

On Thu, Oct 9, 2008 at 2:19 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi,
>
> if you want to get many bugreports, here's how. :)
>
> $ git clone git://git.sympy.org/sympy.git
> $ cd sympy
> $ python
>>>> import sympy
>>>> sympy.test("sympy/core")
> [...]
> ============ tests finished: 292 passed, 9 xfailed in 1.40 seconds =============
> True
>>>>
>
>
> This works in python2.4, python2.5 (there are some failures in
> python2.6) and it *doesn't* require py.test, just pure python.

Holger asked me why we are not using py.test anymore and any
particular feedback. I am not subscribed to py-dev, so I just post it
here.

We still use py.test. But I also want to have something simple
(current implementation has 335 lines) that we can include with sympy,
because py.test is not so easy to install --- you need to install
setuptools, which in particular require python to be compiled with
zlib. So while py.test is fine let's say in Debian or Ubuntu, it was
quite a pain for me to install it 6x on our buildbots:

http://buildbot.sympy.org/waterfall

(E.g. I compiled python2.4, 2.5, 2.6 from python.org, then I realized
that setuptools cannot be installed, because python didn't compile
with the zlib module, because I didn't have the zlib-dev Debian
package installed, so I fixed that, recompiled python, installed
setuptools 6x and then installed py.test 6x).

And we are also porting sympy to jython and it's just easier to just
install sympy and do "sympy.test()" to test that all is ok.

We also thought we could just include py.test in sympy, but py.test
depends on the py lib and it is big. So I just rewrote the necessary
thing from scratch and it can do what I want.

Ondrej

From cfbolz at gmx.de  Fri Oct 10 13:23:21 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 10 Oct 2008 13:23:21 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
Message-ID: <48EF3B29.8040301@gmx.de>

Ondrej Certik wrote:
> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>> changes, I suggest that we say we don't provide --pyversion option any
>> more and simply compile 2.5 compatible interpreter. I don't really see
>> benefits of providing 2.4 right now (after a brief discussion with
>> Armin).
>>
>> If anyone objects, please do that now.
> 
> Just a curious question --- what kind of features are python2.4 only
> that doesn't work in python2.5? I though that except hashing,
> python2.5 is backwards compatible.

I don't think there are any (big ones, at least). But that's just a
further argument for removing the --pyversion option, isn't it?

Cheers,

Carl Friedrich

From fijall at gmail.com  Fri Oct 10 14:05:20 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 10 Oct 2008 14:05:20 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <48EF3B29.8040301@gmx.de>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
	<48EF3B29.8040301@gmx.de>
Message-ID: <693bc9ab0810100505l653f2858taa1ba15dc6b70374@mail.gmail.com>

On Fri, Oct 10, 2008 at 1:23 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Ondrej Certik wrote:
>> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>>> changes, I suggest that we say we don't provide --pyversion option any
>>> more and simply compile 2.5 compatible interpreter. I don't really see
>>> benefits of providing 2.4 right now (after a brief discussion with
>>> Armin).
>>>
>>> If anyone objects, please do that now.
>>
>> Just a curious question --- what kind of features are python2.4 only
>> that doesn't work in python2.5? I though that except hashing,
>> python2.5 is backwards compatible.
>
> I don't think there are any (big ones, at least). But that's just a
> further argument for removing the --pyversion option, isn't it?
>
> Cheers,
>
> Carl Friedrich

For example incompatible module APIs. They're kind of the same from
user level, but the way they interact inside (for example sre and
_sre) has changed.

Cheers,
fijal

From ondrej at certik.cz  Fri Oct 10 14:10:54 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 14:10:54 +0200
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <693bc9ab0810100505l653f2858taa1ba15dc6b70374@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
	<48EF3B29.8040301@gmx.de>
	<693bc9ab0810100505l653f2858taa1ba15dc6b70374@mail.gmail.com>
Message-ID: <85b5c3130810100510l737996cax3689d78f42eb4b1@mail.gmail.com>

On Fri, Oct 10, 2008 at 2:05 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Oct 10, 2008 at 1:23 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>> Ondrej Certik wrote:
>>> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>>>> changes, I suggest that we say we don't provide --pyversion option any
>>>> more and simply compile 2.5 compatible interpreter. I don't really see
>>>> benefits of providing 2.4 right now (after a brief discussion with
>>>> Armin).
>>>>
>>>> If anyone objects, please do that now.
>>>
>>> Just a curious question --- what kind of features are python2.4 only
>>> that doesn't work in python2.5? I though that except hashing,
>>> python2.5 is backwards compatible.
>>
>> I don't think there are any (big ones, at least). But that's just a
>> further argument for removing the --pyversion option, isn't it?
>>
>> Cheers,
>>
>> Carl Friedrich
>
> For example incompatible module APIs. They're kind of the same from
> user level, but the way they interact inside (for example sre and
> _sre) has changed.

I see. Thanks!

Ondrej

From holger at merlinux.eu  Fri Oct 10 15:04:47 2008
From: holger at merlinux.eu (holger krekel)
Date: Fri, 10 Oct 2008 15:04:47 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>
Message-ID: <20081010130447.GN6829@trillke.net>

Hi Ondrej, 

On Fri, Oct 10, 2008 at 13:12 +0200, Ondrej Certik wrote:
> On Thu, Oct 9, 2008 at 2:19 AM, Ondrej Certik <ondrej at certik.cz> wrote:
> > Hi,
> >
> > if you want to get many bugreports, here's how. :)
> >
> > $ git clone git://git.sympy.org/sympy.git
> > $ cd sympy
> > $ python
> >>>> import sympy
> >>>> sympy.test("sympy/core")
> > [...]
> > ============ tests finished: 292 passed, 9 xfailed in 1.40 seconds =============
> > True
> >>>>
> >
> >
> > This works in python2.4, python2.5 (there are some failures in
> > python2.6) and it *doesn't* require py.test, just pure python.
> 
> Holger asked me why we are not using py.test anymore and any
> particular feedback. I am not subscribed to py-dev, so I just post it
> here.

thanks - i am cross-posting this to py-dev. 
 
> We still use py.test. But I also want to have something simple
> (current implementation has 335 lines) that we can include with sympy,
> because py.test is not so easy to install --- you need to install
> setuptools, which in particular require python to be compiled with
> zlib. So while py.test is fine let's say in Debian or Ubuntu, it was
> quite a pain for me to install it 6x on our buildbots:
> 
> http://buildbot.sympy.org/waterfall

> (E.g. I compiled python2.4, 2.5, 2.6 from python.org, then I realized
> that setuptools cannot be installed, because python didn't compile
> with the zlib module, because I didn't have the zlib-dev Debian
> package installed, so I fixed that, recompiled python, installed
> setuptools 6x and then installed py.test 6x).

Hum, that's kind of bothersome, i agree.  I wonder if going
away from setuptools makes sense, shouldn't be hard. 

> And we are also porting sympy to jython and it's just easier to just
> install sympy and do "sympy.test()" to test that all is ok.

right, makes sense. 
 
> We also thought we could just include py.test in sympy, but py.test
> depends on the py lib and it is big. So I just rewrote the necessary
> thing from scratch and it can do what I want.

Raymond Hettinger at some point also wrote a pytest one-file
script with a subset of features.  Probably providing 
something like that officially makes sense.   Also 
i'd like to work a bit on minimizing py lib (i plan
to factor some parts out into plugins soon) - most of it
is actually used from py.test. 

best,

holger

-- 
collaborative expert contracting: http://merlinux.eu 
PyPy  Python/Compiler tool chain: http://codespeak.net/pypy 
pylib py.test/greenlets/svn APIs: http://pylib.org 

From ondrej at certik.cz  Fri Oct 10 15:24:34 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 10 Oct 2008 15:24:34 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <20081010130447.GN6829@trillke.net>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<85b5c3130810100412o2dec14d7rcd353f268c10d4f8@mail.gmail.com>
	<20081010130447.GN6829@trillke.net>
Message-ID: <85b5c3130810100624v59f9385eg282ea2864d624618@mail.gmail.com>

>> We also thought we could just include py.test in sympy, but py.test
>> depends on the py lib and it is big. So I just rewrote the necessary
>> thing from scratch and it can do what I want.
>
> Raymond Hettinger at some point also wrote a pytest one-file
> script with a subset of features.  Probably providing
> something like that officially makes sense.   Also
> i'd like to work a bit on minimizing py lib (i plan
> to factor some parts out into plugins soon) - most of it
> is actually used from py.test.


Thanks for the work on py.test. The user interface is still I think
the best from all the testing frameworks around. I think it's cool to
have something small & simple that projects can just copy verbatim,
and then full featured py.test, that one can use when it is installed.
In fact, that's how I use it.

Ondrej

From arigo at tunes.org  Sat Oct 11 17:14:57 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 11 Oct 2008 17:14:57 +0200
Subject: [pypy-dev] switched dist -> trunk
Message-ID: <20081011151456.GA19291@code0.codespeak.net>

Hi all,

We finally created the path http://codespeak.net/svn/pypy/trunk, which
contains the in-development 2.5-compatible version of PyPy.  This closes
our many-year-old issue 30 :-)

The http://codespeak.net/svn/pypy/dist path is still available, and it's
fine to continue checking in there for some more days, but please switch
to using 'trunk' for development when convenient.

The plan is to merge the changes from 'dist' into 'trunk' in a few days,
after which we'll use 'dist' only as a kind of mirror of 'trunk'
containing read-only snapshots of stable revisions.


A bientot,

Armin

From arigo at tunes.org  Sat Oct 11 19:34:26 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 11 Oct 2008 19:34:26 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
Message-ID: <20081011173426.GA1699@code0.codespeak.net>

Hi,

On Thu, Oct 09, 2008 at 02:19:35AM +0200, Ondrej Certik wrote:
>     if hasattr(i, "__iter__"):
> RuntimeError: internal error: <RuntimeError object at 0xb03b60>

This occurs because our 'str' type has an '__iter__' special method.  It
turns out that CPython's does not.  I suppose we could remove our
__iter__, but I would rather suggest that sympy's code is fragile there:
as soon as someone realizes that in CPython it is possible to speed up
iteration over strings by creating a stringiterator type, then CPython
will grow a 'str.__iter__' as well, and the same infinite recursion will
occur in sympy...

A cleaner way to say "is x iterable?" would be to try to call iter(x)
and see if it raises TypeError or not.


A bientot,

Armin

From ondrej at certik.cz  Sat Oct 11 19:50:25 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 11 Oct 2008 19:50:25 +0200
Subject: [pypy-dev] sympy tests: pypy vs cpython incompatibilities
In-Reply-To: <20081011173426.GA1699@code0.codespeak.net>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<20081011173426.GA1699@code0.codespeak.net>
Message-ID: <85b5c3130810111050q3a929eavb771235a41638579@mail.gmail.com>

On Sat, Oct 11, 2008 at 7:34 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On Thu, Oct 09, 2008 at 02:19:35AM +0200, Ondrej Certik wrote:
>>     if hasattr(i, "__iter__"):
>> RuntimeError: internal error: <RuntimeError object at 0xb03b60>
>
> This occurs because our 'str' type has an '__iter__' special method.  It
> turns out that CPython's does not.  I suppose we could remove our
> __iter__, but I would rather suggest that sympy's code is fragile there:
> as soon as someone realizes that in CPython it is possible to speed up
> iteration over strings by creating a stringiterator type, then CPython
> will grow a 'str.__iter__' as well, and the same infinite recursion will
> occur in sympy...
>
> A cleaner way to say "is x iterable?" would be to try to call iter(x)
> and see if it raises TypeError or not.

Indeed, thanks very much for the tip. We'll fix that, that's
definitely something that should be fixed in sympy, I created a new
issue for that:

http://code.google.com/p/sympy/issues/detail?id=1153

Ondrej

From ondrej at certik.cz  Sat Oct 11 23:42:47 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 11 Oct 2008 23:42:47 +0200
Subject: [pypy-dev] [sympy] Re: sympy tests: pypy vs cpython
	incompatibilities
In-Reply-To: <4db014670810111113v73392f26k9781616993dd7d84@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<20081011173426.GA1699@code0.codespeak.net>
	<85b5c3130810111050q3a929eavb771235a41638579@mail.gmail.com>
	<4db014670810111113v73392f26k9781616993dd7d84@mail.gmail.com>
Message-ID: <85b5c3130810111442v11fdaaa9na3cac6eed8b20f54@mail.gmail.com>

On Sat, Oct 11, 2008 at 8:13 PM, didier deshommes <dfdeshom at gmail.com> wrote:
>
> On Sat, Oct 11, 2008 at 1:50 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>>> as soon as someone realizes that in CPython it is possible to speed up
>>> iteration over strings by creating a stringiterator type, then CPython
>>> will grow a 'str.__iter__' as well, and the same infinite recursion will
>>> occur in sympy...
>>>
>>> A cleaner way to say "is x iterable?" would be to try to call iter(x)
>>> and see if it raises TypeError or not.
>>
>> Indeed, thanks very much for the tip. We'll fix that, that's
>> definitely something that should be fixed in sympy, I created a new
>> issue for that:
>
> Hi,
> I've faced the same problem and checking if x has the __getitem__
> attribute has worked for me. The rationale being that any iterable has
> a way to access its individual items.

Thanks Didier for the tip. Both iter("p") and "p".__getitem__ behaves
the same in both python and pypy. But "p".__iter__ behaves differently
and our code assumed that "p" is not iterable, so I just implemented a
function:

http://code.google.com/p/sympy/issues/detail?id=1153#c1

+def is_iterable(x):
+    try:
+        iter(x)
+        if isinstance(x, str) and len(x) == 1:
+            return False
+        return True
+    except TypeError:
+        return False


Which fixes the problem above. Unfortunately, then there are other
unrelated problems.

Armin, is there some way to tell pypy-c to stop on ctrl-C? I am
getting hangups and the only way out is to kill the pypy-c program.
It'd be much more convenient if I could just do ctrl-C and it would
print the stacktrace.

I'll be continuing porting sympy to pypy some other time, so anyone
feel free to take over it. You just need to apply the patch in the
issue 1153 and then try to make all tests to pass.

Thanks,
Ondrej

From micahel at gmail.com  Sun Oct 12 12:13:53 2008
From: micahel at gmail.com (Michael Hudson)
Date: Sun, 12 Oct 2008 23:13:53 +1300
Subject: [pypy-dev] 2.5 and 2.4 compatibility
In-Reply-To: <85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
References: <693bc9ab0809220848s1131bd40i14b36e1016d255f8@mail.gmail.com>
	<85b5c3130810100400w75078901qc7e4b7975ec52a94@mail.gmail.com>
Message-ID: <aa0dfd860810120313u63119a34g7a4b50bdb7e0418f@mail.gmail.com>

2008/10/11 Ondrej Certik <ondrej at certik.cz>:
> On Mon, Sep 22, 2008 at 5:48 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> I'm working on 2.5 compatibility branch right now. Due to incompatible
>> changes, I suggest that we say we don't provide --pyversion option any
>> more and simply compile 2.5 compatible interpreter. I don't really see
>> benefits of providing 2.4 right now (after a brief discussion with
>> Armin).
>>
>> If anyone objects, please do that now.
>
> Just a curious question --- what kind of features are python2.4 only
> that doesn't work in python2.5? I though that except hashing,
> python2.5 is backwards compatible.

New style exceptions break a few things.  A few deprecated things were
removed -- OverflowWarning, for example.

The idea that Python 3 was the first release of Python that isn't
backwards compatible is slightly misleading :)

Cheers,
mwh

From dw at botanicus.net  Tue Oct 14 18:39:56 2008
From: dw at botanicus.net (David Wilson)
Date: Tue, 14 Oct 2008 17:39:56 +0100
Subject: [pypy-dev] Sprint Discussions: C++ Library Bindings
Message-ID: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>

Hi there,

I just read about this on the PyPy status blog and find myself
wondering about the strategies you considered before settling on the
Reflex approach.

I've had an extension to ctypes in mind for quite some time that
involves simply leaving the debugging segments present in a shared
library in order to be able to introspect them and provide type-safe
access to the code within.

The idea is simple: implement a DWARF/stabs/etc. interpreter that can
construct Python types and function proxies to represent what is
present in the shared library. Initially I wanted this for C, so that
a Python programmer needn't manually declare structs using quirky
ctypes class definitions, but I can't see why a (rather more complex)
implementation couldn't reconstruct C++ classes with their associated
class/member functions.

The obvious benefit is that a C/C++ library need not be modified at
all in order to benefit, other than not being stripped of debugging
information during its build. Linux distributions are slowly gaining
infrastructure that makes this almost zero cost (e.g. the Debian
python-dbg package).

I've had the DWARF specification sitting on my desktop for about 2
months now, waiting for me to validate my idea.

I'm wondering if you've considered this approach, and if so, why you
discarded it.

Thanks,


David.

From fijall at gmail.com  Wed Oct 15 23:53:53 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 15 Oct 2008 23:53:53 +0200
Subject: [pypy-dev] buildbot problem/misunderstanding.
Message-ID: <693bc9ab0810151453m38a255beqf3ba8977e45e2c55@mail.gmail.com>

I've been starring at report (here:
http://codespeak.net:8099/builders/own-linux-x86-32/builds/16/steps/shell_2/logs/stdio)
and I don't understand why there is only one test in translator/c. Am
I missing something? (search for translator/c/test/test_support to see
what I mean). I'm confused.

Cheers,
fijal

From fijall at gmail.com  Wed Oct 15 23:54:48 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 15 Oct 2008 23:54:48 +0200
Subject: [pypy-dev] buildbot problem/misunderstanding.
In-Reply-To: <693bc9ab0810151453m38a255beqf3ba8977e45e2c55@mail.gmail.com>
References: <693bc9ab0810151453m38a255beqf3ba8977e45e2c55@mail.gmail.com>
Message-ID: <693bc9ab0810151454r17564304t756a52688854064e@mail.gmail.com>

Ah, sorry for spam, got it.

On Wed, Oct 15, 2008 at 11:53 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I've been starring at report (here:
> http://codespeak.net:8099/builders/own-linux-x86-32/builds/16/steps/shell_2/logs/stdio)
> and I don't understand why there is only one test in translator/c. Am
> I missing something? (search for translator/c/test/test_support to see
> what I mean). I'm confused.
>
> Cheers,
> fijal
>

From arigo at tunes.org  Thu Oct 16 12:53:37 2008
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 16 Oct 2008 12:53:37 +0200
Subject: [pypy-dev] [sympy] Re: sympy tests: pypy vs cpython
	incompatibilities
In-Reply-To: <85b5c3130810111442v11fdaaa9na3cac6eed8b20f54@mail.gmail.com>
References: <85b5c3130810081719i4c337f9at9117a79419e62dfb@mail.gmail.com>
	<20081011173426.GA1699@code0.codespeak.net>
	<85b5c3130810111050q3a929eavb771235a41638579@mail.gmail.com>
	<4db014670810111113v73392f26k9781616993dd7d84@mail.gmail.com>
	<85b5c3130810111442v11fdaaa9na3cac6eed8b20f54@mail.gmail.com>
Message-ID: <20081016105337.GA9516@code0.codespeak.net>

Hi Ondrej,

On Sat, Oct 11, 2008 at 11:42:47PM +0200, Ondrej Certik wrote:
> +def is_iterable(x):
> +    try:
> +        iter(x)
> +        if isinstance(x, str) and len(x) == 1:
> +            return False
> +        return True
> +    except TypeError:
> +        return False

I don't see why there is a check for len(x); the same difference exists
for strings of any length, not just characters.

> Armin, is there some way to tell pypy-c to stop on ctrl-C? I am
> getting hangups and the only way out is to kill the pypy-c program.
> It'd be much more convenient if I could just do ctrl-C and it would
> print the stacktrace.

That's what it should do.  Do you have a way to reproduce the problem?
Just running the sympy tests with the above patch?


A bientot,

Armin

From irving at naml.us  Fri Oct 17 05:50:36 2008
From: irving at naml.us (Geoffrey Irving)
Date: Thu, 16 Oct 2008 20:50:36 -0700
Subject: [pypy-dev] C++ library bindings
Message-ID: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>

Hello,

I posted a response to your blog post on C++ library bindings, and
wanted to continue the discussion further via email if anyone's
interested.  I just signed up for the mailing list, so apologies if I
missed a lot of previous discussion.  I'll say up front that it's
unlikely that I'll be able to devote any actual coding effort to this,
so feel free to tell me to get lost if you have plenty of ideas and
not enough manpower. :)

I started out writing C++ bindings using Boost.Python, and was very
happy with it for a long time.  It's strongest point is the ability to
wrap libraries that were never designed with python in mind,
specifically code with poor and inflexible ownership semantics.
Internally, this means that C++ objects are exposed indirectly through
a holder object containing either an inline copy of the C++ object or
any type of pointer holding the object.  Every access to the object
has to go through runtime dispatch in order to work with any possible
holder type.  The holder also contains the logic for ownership and
finalization.  For example, Boost.Python can return a reference to a
field inside another object, in which case the holder will keep a
reference to the parent object to keep it alive as long as the field
reference lives.

The problem with this generality is that it produces a huge amount of
object code (wrapping a single function in Boost.Python can add 10k to
the object file), and adds a lot of runtime indirection.

Assuming that one is writing C++ bindings because of speed issues,
it'd be nice if this extra layer of memory indirection and runtime
dispatch was exposed to the (eventual) JIT.  In order to do that, pypy
would have to be capable of handling pointers to raw memory containing
non-python objects (is already true due to ctypes stuff?), with
separate information about type and ownership.  For example, if you
have bindings for a C++ vector class and a C++ array containing the
vectors, a "reference" to an individual vector in the array is really
three different pieces:

1. The actual pointer to the vector.
2. A type structure containing functions to be called with the pointer
(1) as an argument.
3. A list of references to other objects that need to stay alive while
this reference lives.

If pypy and the JIT ends up able to treat these pieces separately,
it'd be a significant performance win over libraries wrapped with
CPython.

The other main source of slowness and complexity in Boost.Python is
overloading support, but I think that part is fairly straightforward
to handle in the python level.  All Boost.Python does internally is
loop over the set of functions registered for a given name, and for
each one loop over the arguments calling into its converter registry
to see if the python object can be converted to the C++ type.

As I mentioned in the blog comment, a lot of these issues come up in
contexts outside C++, like numpy.  Internally numpy represents
operations like addition as a big list of optimized routines to call
depending on the stored data type.  Functions in these tables are
called on raw pointers to memory, which is fundamental since numpy
arrays can refer to memory inside objects from C++, Fortran, mmap,
etc.  It'd be really awesome if the type dispatch step could be
written in python but still call into optimized C code for the final
arithmetic.

The other major issue is safety: if a lot of overloading and dispatch
code is going to be written in python, it'd be nice to shield that
code from segfaults.  I think you can get a long way there just by
having a consistent scheme for boxing the three components above
(pointer, type, and reference info), a way to label C function
pointers with type information, a small RPython layer that did simple
type-checked calls (with no support for overloading or type
conversion).  I just wrote a C++ analogue to this last part as a
minimal replacement for Boost.Python, so I could try to formulate what
I mean in pseudocode if there's interest.  There'd be some amount of
duplicate type checking if higher level layers such as overload
resolution were written in application level python, but that
duplication should be amenable to elimination by the JIT.

That's enough for now.  I'll look forward to the discussion.  Most of
my uses of python revolve heavily around C++ bindings, so it's
exciting to see that you're starting to think about it even if it's a
long way off.

Geoffrey

From fijall at gmail.com  Sat Oct 18 18:07:34 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 18 Oct 2008 18:07:34 +0200
Subject: [pypy-dev] [pypy-svn] r59181 -
	pypy/branch/cbuild-refactor/pypy/translator/platform
In-Reply-To: <20081017131928.2B0D316A151@codespeak.net>
References: <20081017131928.2B0D316A151@codespeak.net>
Message-ID: <693bc9ab0810180907w1b3945f5t4ad03522fb388f0d@mail.gmail.com>

On Fri, Oct 17, 2008 at 3:19 PM,  <afa at codespeak.net> wrote:
> Author: afa
> Date: Fri Oct 17 15:19:27 2008
> New Revision: 59181
>
> Modified:
>   pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
> Log:
> Log message about updated environment
>
>
> Modified: pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
> ==============================================================================
> --- pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     (original)
> +++ pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     Fri Oct 17 15:19:27 2008
> @@ -38,6 +38,7 @@
>         key, value = line.split('=', 1)
>         if key.upper() in ['PATH', 'INCLUDE', 'LIB']:
>             os.environ[key] = value
> +    log.msg("Updated environment with %s" % (vcvars,))
>
>  try:
>     _install_msvc_env()
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>

Can you please make sure that os.environ is cleaned afterwards? thanks

From fijall at gmail.com  Sat Oct 18 18:42:02 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sat, 18 Oct 2008 18:42:02 +0200
Subject: [pypy-dev] [pypy-svn] r59181 -
	pypy/branch/cbuild-refactor/pypy/translator/platform
In-Reply-To: <693bc9ab0810180907w1b3945f5t4ad03522fb388f0d@mail.gmail.com>
References: <20081017131928.2B0D316A151@codespeak.net>
	<693bc9ab0810180907w1b3945f5t4ad03522fb388f0d@mail.gmail.com>
Message-ID: <693bc9ab0810180942m34e3a6d9qfc3c557713ec9764@mail.gmail.com>

On Sat, Oct 18, 2008 at 6:07 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Oct 17, 2008 at 3:19 PM,  <afa at codespeak.net> wrote:
>> Author: afa
>> Date: Fri Oct 17 15:19:27 2008
>> New Revision: 59181
>>
>> Modified:
>>   pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
>> Log:
>> Log message about updated environment
>>
>>
>> Modified: pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py
>> ==============================================================================
>> --- pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     (original)
>> +++ pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py     Fri Oct 17 15:19:27 2008
>> @@ -38,6 +38,7 @@
>>         key, value = line.split('=', 1)
>>         if key.upper() in ['PATH', 'INCLUDE', 'LIB']:
>>             os.environ[key] = value
>> +    log.msg("Updated environment with %s" % (vcvars,))
>>
>>  try:
>>     _install_msvc_env()
>> _______________________________________________
>> pypy-svn mailing list
>> pypy-svn at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-svn
>>
>
> Can you please make sure that os.environ is cleaned afterwards? thanks
>

Ah, xoraxax just pointed out that this is once-per-import, so it's
completely fine.

From jurgis.pralgauskis at gmail.com  Mon Oct 20 13:32:23 2008
From: jurgis.pralgauskis at gmail.com (Jurgis Pralgauskis)
Date: Mon, 20 Oct 2008 14:32:23 +0300
Subject: [pypy-dev] py2php
Message-ID: <34f4097d0810200432y51321e86t4588958e1393f45e@mail.gmail.com>

Hello,

I'd like to make life easier for writing simple php scripts
(as I am teaching python and web technologies, but my students (and I)
complain, that python web hosting is still not widelly affordable).

I imagine, that php is mostly similar to javascript for what's in pypy now
first I need very basic things: variables, loops, some string_methods,
echo and read/write to file.

I looked around https://codespeak.net/viewvc/pypy/dist/pypy/translator/js/

as I see, things would be easy to change as in  jts.py  and sbuiltin.py,
but I dont' see where loop structure representation is generated...

maybe smb could construct a bare php translator structure,
which I could start filling up and experimenting
if this smb is busy right now, I could wait :)

Thanks in advance

ps.: another idea:
can pypy quite easily help to visualize the list (or other structure)
manipulation for teaching purposes
it would be kind of watchvariables functionality
but more the way of https://launchpad.net/algorithms-animator

-- 
Jurgis Pralgauskis
tel: 8-616 77613;
jabber: jurgis at akl.lt; skype: dz0rdzas;
Don't worry, be happy and make things better ;)

From arigo at tunes.org  Wed Oct 22 18:51:41 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 22 Oct 2008 18:51:41 +0200
Subject: [pypy-dev] switched dist -> trunk
In-Reply-To: <20081011151456.GA19291@code0.codespeak.net>
References: <20081011151456.GA19291@code0.codespeak.net>
Message-ID: <20081022165141.GA14520@code0.codespeak.net>

Hi all,

On Sat, Oct 11, 2008 at 05:14:57PM +0200, Armin Rigo wrote:
> The plan is to merge the changes from 'dist' into 'trunk' in a few days,
> after which we'll use 'dist' only as a kind of mirror of 'trunk'
> containing read-only snapshots of stable revisions.

Done.  There is now a pre-commit hook that rejects commits inside
pypy/dist.  Please continue development on pypy/trunk.

SVN reminder on how to switch your existing working copy:

    $ svn info
    URL: http://codespeak.net/svn/pypy/dist
    # ^^^ check this URL! if it's not exactly .../pypy/dist, your
    #     current working directory is wrong!

    $ svn switch http://codespeak.net/svn/pypy/trunk


A bientot,

Armin.

From fijall at gmail.com  Thu Oct 23 14:19:06 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 23 Oct 2008 14:19:06 +0200
Subject: [pypy-dev] Sprint Discussions: C++ Library Bindings
In-Reply-To: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
References: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
Message-ID: <693bc9ab0810230519o1f6880c0qcb893ae5d8437f06@mail.gmail.com>

On Tue, Oct 14, 2008 at 6:39 PM, David Wilson <dw at botanicus.net> wrote:
> Hi there,
>
> I just read about this on the PyPy status blog and find myself
> wondering about the strategies you considered before settling on the
> Reflex approach.

We're not settling on it. It's rather "first to try".

>
> I've had an extension to ctypes in mind for quite some time that
> involves simply leaving the debugging segments present in a shared
> library in order to be able to introspect them and provide type-safe
> access to the code within.
>
> The idea is simple: implement a DWARF/stabs/etc. interpreter that can
> construct Python types and function proxies to represent what is
> present in the shared library. Initially I wanted this for C, so that
> a Python programmer needn't manually declare structs using quirky
> ctypes class definitions, but I can't see why a (rather more complex)
> implementation couldn't reconstruct C++ classes with their associated
> class/member functions.

The problem with such approach is usually that this is too low level.
Gaining back lost abstraction level is usually hard and sometimes
impossible. I cannot say exactly that this approach is not worth
following - I'll just try to use reflex first as it seems people are
using it to do complex tasks. Of course feel free to prove that I'm
wrong, right now I'm more thinking about possibilities (hence
questions on blog to people to provide input), than settling on
anything.

>
> The obvious benefit is that a C/C++ library need not be modified at
> all in order to benefit, other than not being stripped of debugging
> information during its build. Linux distributions are slowly gaining
> infrastructure that makes this almost zero cost (e.g. the Debian
> python-dbg package).
>
> I've had the DWARF specification sitting on my desktop for about 2
> months now, waiting for me to validate my idea.
>
> I'm wondering if you've considered this approach, and if so, why you
> discarded it.

The answer is simple: we didn't consider it yet :-)

From fijall at gmail.com  Thu Oct 23 14:25:20 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 23 Oct 2008 14:25:20 +0200
Subject: [pypy-dev] C++ library bindings
In-Reply-To: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
References: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
Message-ID: <693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>

Hey.

First sorry for late response, we're kind of busy doing other things
now (ie working on 2.5-compatible release). That doesn't mean we don't
appreciate input about our problems.

On Fri, Oct 17, 2008 at 5:50 AM, Geoffrey Irving <irving at naml.us> wrote:
> Hello,
>
> I posted a response to your blog post on C++ library bindings, and
> wanted to continue the discussion further via email if anyone's
> interested.  I just signed up for the mailing list, so apologies if I
> missed a lot of previous discussion.  I'll say up front that it's
> unlikely that I'll be able to devote any actual coding effort to this,
> so feel free to tell me to get lost if you have plenty of ideas and
> not enough manpower. :)

That's fine. We don't have enough manpower to work on this now, but
knowing what people do in this area is very valuable once we get to
it.

>
> I started out writing C++ bindings using Boost.Python, and was very
> happy with it for a long time.  It's strongest point is the ability to
> wrap libraries that were never designed with python in mind,
> specifically code with poor and inflexible ownership semantics.
> Internally, this means that C++ objects are exposed indirectly through
> a holder object containing either an inline copy of the C++ object or
> any type of pointer holding the object.  Every access to the object
> has to go through runtime dispatch in order to work with any possible
> holder type.  The holder also contains the logic for ownership and
> finalization.  For example, Boost.Python can return a reference to a
> field inside another object, in which case the holder will keep a
> reference to the parent object to keep it alive as long as the field
> reference lives.
>
> The problem with this generality is that it produces a huge amount of
> object code (wrapping a single function in Boost.Python can add 10k to
> the object file), and adds a lot of runtime indirection.
>
> Assuming that one is writing C++ bindings because of speed issues,
> it'd be nice if this extra layer of memory indirection and runtime
> dispatch was exposed to the (eventual) JIT.  In order to do that, pypy
> would have to be capable of handling pointers to raw memory containing
> non-python objects (is already true due to ctypes stuff?)

That's true. PyPy is able to handle pointers to any C place.

> .. with
> separate information about type and ownership.

We don't provide this, since C has no notion of that at all.

> For example, if you
> have bindings for a C++ vector class and a C++ array containing the
> vectors, a "reference" to an individual vector in the array is really
> three different pieces:
>
> 1. The actual pointer to the vector.
> 2. A type structure containing functions to be called with the pointer
> (1) as an argument.
> 3. A list of references to other objects that need to stay alive while
> this reference lives.
>
> If pypy and the JIT ends up able to treat these pieces separately,
> it'd be a significant performance win over libraries wrapped with
> CPython.
>
> The other main source of slowness and complexity in Boost.Python is
> overloading support, but I think that part is fairly straightforward
> to handle in the python level.  All Boost.Python does internally is
> loop over the set of functions registered for a given name, and for
> each one loop over the arguments calling into its converter registry
> to see if the python object can be converted to the C++ type.
>
> As I mentioned in the blog comment, a lot of these issues come up in
> contexts outside C++, like numpy.  Internally numpy represents
> operations like addition as a big list of optimized routines to call
> depending on the stored data type.  Functions in these tables are
> called on raw pointers to memory, which is fundamental since numpy
> arrays can refer to memory inside objects from C++, Fortran, mmap,
> etc.  It'd be really awesome if the type dispatch step could be
> written in python but still call into optimized C code for the final
> arithmetic.

That's the goal. Well, not exactly - point is that you write this code
in Python/RPython and JIT is able to generate efficient assembler out
of it. That's a very far-reaching goal though to have nice integration
between yet-non-existant JIT and yet-non-existant PyPy's numpy :-)

>
> The other major issue is safety: if a lot of overloading and dispatch
> code is going to be written in python, it'd be nice to shield that
> code from segfaults.  I think you can get a long way there just by
> having a consistent scheme for boxing the three components above
> (pointer, type, and reference info), a way to label C function
> pointers with type information, a small RPython layer that did simple
> type-checked calls (with no support for overloading or type
> conversion).  I just wrote a C++ analogue to this last part as a
> minimal replacement for Boost.Python, so I could try to formulate what
> I mean in pseudocode if there's interest.  There'd be some amount of
> duplicate type checking if higher level layers such as overload
> resolution were written in application level python, but that
> duplication should be amenable to elimination by the JIT.

I think for now we're happy with extra overhead. We would like to have
*any* working C++ bindings first and then eventually think about
speeding it up.

>
> That's enough for now.  I'll look forward to the discussion.  Most of
> my uses of python revolve heavily around C++ bindings, so it's
> exciting to see that you're starting to think about it even if it's a
> long way off.

Thank you :)

Cheers,
fijal

From fijall at gmail.com  Thu Oct 23 14:27:34 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 23 Oct 2008 14:27:34 +0200
Subject: [pypy-dev] py2php
In-Reply-To: <34f4097d0810200432y51321e86t4588958e1393f45e@mail.gmail.com>
References: <34f4097d0810200432y51321e86t4588958e1393f45e@mail.gmail.com>
Message-ID: <693bc9ab0810230527m7b5f1418v6ee7c411aeecbf5@mail.gmail.com>

On Mon, Oct 20, 2008 at 1:32 PM, Jurgis Pralgauskis
<jurgis.pralgauskis at gmail.com> wrote:
> Hello,
>
> I'd like to make life easier for writing simple php scripts
> (as I am teaching python and web technologies, but my students (and I)
> complain, that python web hosting is still not widelly affordable).
>
> I imagine, that php is mostly similar to javascript for what's in pypy now
> first I need very basic things: variables, loops, some string_methods,
> echo and read/write to file.
>
> I looked around https://codespeak.net/viewvc/pypy/dist/pypy/translator/js/

This is going to be moved somewhere else as it's not working as we
would expect. Besides, it's translating rpython, not python into JS.

> ps.: another idea:
> can pypy quite easily help to visualize the list (or other structure)
> manipulation for teaching purposes
> it would be kind of watchvariables functionality
> but more the way of https://launchpad.net/algorithms-animator

I think it's easier to do such things in pypy, since you'll write code
that does in in python and not in C. But it's still work - it doesn't
come for free at all. If you want to do it, you can expect to get some
help.

Cheers,
fijal

From cfbolz at gmx.de  Thu Oct 23 14:36:00 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Thu, 23 Oct 2008 14:36:00 +0200
Subject: [pypy-dev] Sprint Discussions: C++ Library Bindings
In-Reply-To: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
References: <98edab2e0810140939l44d5dc83xf2461947790dba9@mail.gmail.com>
Message-ID: <49006FB0.9010004@gmx.de>

Hi David,

David Wilson wrote:
> I just read about this on the PyPy status blog and find myself
> wondering about the strategies you considered before settling on the
> Reflex approach.
> 
> I've had an extension to ctypes in mind for quite some time that
> involves simply leaving the debugging segments present in a shared
> library in order to be able to introspect them and provide type-safe
> access to the code within.
> 
> The idea is simple: implement a DWARF/stabs/etc. interpreter that can
> construct Python types and function proxies to represent what is
> present in the shared library. Initially I wanted this for C, so that
> a Python programmer needn't manually declare structs using quirky
> ctypes class definitions, but I can't see why a (rather more complex)
> implementation couldn't reconstruct C++ classes with their associated
> class/member functions.
> 
> The obvious benefit is that a C/C++ library need not be modified at
> all in order to benefit, other than not being stripped of debugging
> information during its build. Linux distributions are slowly gaining
> infrastructure that makes this almost zero cost (e.g. the Debian
> python-dbg package).
> 
> I've had the DWARF specification sitting on my desktop for about 2
> months now, waiting for me to validate my idea.
> 
> I'm wondering if you've considered this approach, and if so, why you
> discarded it.

One problem of this approach (in addition to what Maciek mentioned) is
that the debug information is quite large. It contains much more
information that what is needed for doing language bindings, e.g. also
detailed information about the inside of functions ("when the PC is
this, these variables are in those registers"). The Reflex information
should be rather small in size, compared to that.

Cheers,

Carl Friedrich

From irving at naml.us  Thu Oct 23 19:30:33 2008
From: irving at naml.us (Geoffrey Irving)
Date: Thu, 23 Oct 2008 10:30:33 -0700
Subject: [pypy-dev] C++ library bindings
In-Reply-To: <693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>
References: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
	<693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>
Message-ID: <7f9d599f0810231030v61693bb1r1cfb10e2e2db908c@mail.gmail.com>

On Thu, Oct 23, 2008 at 5:25 AM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Hey.
>
> First sorry for late response, we're kind of busy doing other things
> now (ie working on 2.5-compatible release). That doesn't mean we don't
> appreciate input about our problems.
>
> On Fri, Oct 17, 2008 at 5:50 AM, Geoffrey Irving <irving at naml.us> wrote:
> <snip>
>
> That's true. PyPy is able to handle pointers to any C place.
>
>> .. with
>>
>> separate information about type and ownership.
>
> We don't provide this, since C has no notion of that at all.

At the lowest level the type is just a hashable identifier object, so
it can probably implemented at the RPython level.  E.g.,

# RPython type-safety layer

class CppObject:
    def __init__(ptr, type):
        self.ptr = ptr # pointer to the actual C++ instance
        self.type = type # represents the C++ type
        self.destructor = type.destructor # function pointer to destructor

    def __traverse__(self):
        ... traverse through list of contained python object pointers ...

    def __del__(self):
        CCall(self.destructor, self.ptr)

class CppFunc:
    def __init__(ptr, resulttype, argtypes):
        self.ptr = ptr
        self.resulttype = resulttype
        self.argtypes = argtypes

    def __call__(self, *args):
        if len(args) != len(self.argtypes):
            raise TypeError(...)
        argptrs = []
        for a,t in zip(args,self.argtypes):
            if not isinstance(a, CppObject) or a.type != t:
                raise TypeError(...)
            argptrs.append(a.ptr)
        resultptr = Alloc(self.resulttype.size)
        try:
            CppCall(self.ptr, resultptr, *argptrs) # assumes specific
calling convention
        except CppException, e: # CppCall would have to generate this
            Dealloc(resultptr)
            raise CppToPythonException(e)
        return CppObject(resultptr, self.resulttype)

If this layer is written in RPython, features like overload resolution
and C++ methods can be written in application-level python without
worring about safety.

>> <snip>
>>
>> As I mentioned in the blog comment, a lot of these issues come up in
>> contexts outside C++, like numpy.  Internally numpy represents
>> operations like addition as a big list of optimized routines to call
>> depending on the stored data type.  Functions in these tables are
>> called on raw pointers to memory, which is fundamental since numpy
>> arrays can refer to memory inside objects from C++, Fortran, mmap,
>> etc.  It'd be really awesome if the type dispatch step could be
>> written in python but still call into optimized C code for the final
>> arithmetic.
>
> That's the goal. Well, not exactly - point is that you write this code
> in Python/RPython and JIT is able to generate efficient assembler out
> of it. That's a very far-reaching goal though to have nice integration
> between yet-non-existant JIT and yet-non-existant PyPy's numpy :-)

Asking the JIT to generate to generate efficient code might be
sufficient in this case, but in terms of this discussion it just
removes numpy as a useful thought experiment towards C++ bindings. :)
Also for maximum speed I doubt the JIT will be able to match custom
code such as BLAS, given that C++ compilers usually don't get there
either.

>> The other major issue is safety: if a lot of overloading and dispatch
>> code is going to be written in python, it'd be nice to shield that
>> code from segfaults.  I think you can get a long way there just by
>> having a consistent scheme for boxing the three components above
>> (pointer, type, and reference info), a way to label C function
>> pointers with type information, a small RPython layer that did simple
>> type-checked calls (with no support for overloading or type
>> conversion).  I just wrote a C++ analogue to this last part as a
>> minimal replacement for Boost.Python, so I could try to formulate what
>> I mean in pseudocode if there's interest.  There'd be some amount of
>> duplicate type checking if higher level layers such as overload
>> resolution were written in application level python, but that
>> duplication should be amenable to elimination by the JIT.
>
> I think for now we're happy with extra overhead. We would like to have
> *any* working C++ bindings first and then eventually think about
> speeding it up.

Another advantage of splitting the code into an RPython type-safety
layer and application-level code is that the latter could be shared
with between pypy and cpython.  I haven't looked at reflex at all, but
in Boost.Python most of the complexity goes into code that could exist
at the application-level.

Geoffrey

From anto.cuni at gmail.com  Mon Oct 27 12:13:20 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 27 Oct 2008 12:13:20 +0100
Subject: [pypy-dev] bugday
Message-ID: <4905A250.5050403@gmail.com>

Hi pypy-dev,

we are trying to organize the first "PyPy Bugday", but it seems it's 
hard to find a proper date for it; so, we've set up a poll on doodle:

http://doodle.com/73tepx6ktg9cyd29

the poll is mainly targeted at core pypy developers, but of course 
everyone can vote.
Please spend two minutes for voting, so that we can pick the best date 
for it.

ciao & a bientot,
Anto & Armin

From arigo at tunes.org  Mon Oct 27 15:09:16 2008
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 27 Oct 2008 15:09:16 +0100
Subject: [pypy-dev] C++ library bindings
In-Reply-To: <7f9d599f0810231030v61693bb1r1cfb10e2e2db908c@mail.gmail.com>
References: <7f9d599f0810162050n42cb0eacpfdafb16ae40d03b0@mail.gmail.com>
	<693bc9ab0810230525y300c73c5o36b515f450473d3d@mail.gmail.com>
	<7f9d599f0810231030v61693bb1r1cfb10e2e2db908c@mail.gmail.com>
Message-ID: <20081027140916.GA30825@code0.codespeak.net>

Hi Geoffrey,

On Thu, Oct 23, 2008 at 10:30:33AM -0700, Geoffrey Irving wrote:
> If this layer is written in RPython, features like overload resolution
> and C++ methods can be written in application-level python without
> worring about safety.

I think that in this area, any worries of safety went out of the window
when CPython officially adopted ctypes.  So it's completely fine if the
application-level dispatching layer has access to unsafe features, as
long as user applications that only use the dispatching layer are
shielded from crashes.


A bientot,

Armin.

From brunogola at gmail.com  Mon Oct 27 15:21:25 2008
From: brunogola at gmail.com (Bruno Gola)
Date: Mon, 27 Oct 2008 12:21:25 -0200
Subject: [pypy-dev] bugday
In-Reply-To: <4905A250.5050403@gmail.com>
References: <4905A250.5050403@gmail.com>
Message-ID: <b27f1a340810270721w471db2e0g4b4d3b008d65d010@mail.gmail.com>

Hi,

On Mon, Oct 27, 2008 at 9:13 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> we are trying to organize the first "PyPy Bugday", but it seems it's
> hard to find a proper date for it; so, we've set up a poll on doodle:

Cool!

Is there any information available about how to help in a PyPy Bugday?

S?o Paulo Python User Group always participate of the Python Bugday
and I'm sure they will be happy to help PyPy.

Cheers,
--
Bruno Fialho Marques Gola <brunogola at gmail.com>
http://www.brunogola.com.br
Cel: (11) 9294-5883

From fijall at gmail.com  Mon Oct 27 15:54:46 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 27 Oct 2008 15:54:46 +0100
Subject: [pypy-dev] bugday
In-Reply-To: <b27f1a340810270721w471db2e0g4b4d3b008d65d010@mail.gmail.com>
References: <4905A250.5050403@gmail.com>
	<b27f1a340810270721w471db2e0g4b4d3b008d65d010@mail.gmail.com>
Message-ID: <693bc9ab0810270754t15e1a198u2a47aeac5051b37c@mail.gmail.com>

On Mon, Oct 27, 2008 at 3:21 PM, Bruno Gola <brunogola at gmail.com> wrote:
> Hi,
>
> On Mon, Oct 27, 2008 at 9:13 AM, Antonio Cuni <anto.cuni at gmail.com> wrote:
>> we are trying to organize the first "PyPy Bugday", but it seems it's
>> hard to find a proper date for it; so, we've set up a poll on doodle:
>
> Cool!
>
> Is there any information available about how to help in a PyPy Bugday?
>
> S?o Paulo Python User Group always participate of the Python Bugday
> and I'm sure they will be happy to help PyPy.
>
> Cheers,
> --
> Bruno Fialho Marques Gola <brunogola at gmail.com>
> http://www.brunogola.com.br
> Cel: (11) 9294-5883

Of course! By fixing bugs :-)

Details of failures are here:
https://codespeak.net/issue/pypy-dev/index and here:
http://codespeak.net:8099/summary

Cheers,
fijal

From p.giarrusso at gmail.com  Tue Oct 28 20:20:19 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Tue, 28 Oct 2008 20:20:19 +0100
Subject: [pypy-dev] Possible student project on PyPy
In-Reply-To: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
References: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
Message-ID: <a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>

Hi, for a student project, we are evaluating the possibility to
experiment with some ideas on PyPy.

Even if it is a student project, we have high expectations (our aim is
to really improve PyPy's speed, if possible). But we are still trying
to choose which features to implement (we started with some nice
ideas, but part of them is already implemented).

We will work for something like 2 months on a separate branch - maybe
we can give readonly access to the source code repository, but write
access would be a problem for our exam, for obvious reasons.

Do you have any possibility to host a development branch for this project?

To let you know who we are, I'll present us:

- I, Paolo 'Blaisorblade' Giarrusso, am a past Linux kernel hacker, I
worked also with Java, C++, and a bit of Python, and I am currently a
graduate student at Aarhus University, Denmark;
- Sigurd is a PhD candidate at Aarhus University, Denmark, currently
working, among other things, on a cryptography research project in
Python (viff.dk).

- our professor is Lars Bak, the lead architect of the Google V8
Javascript engine, on which we implemented various optimizations in
the previous months.

We are obviously open to suggestions, and we have been looking at the
status blog and at various blogs.

It seems that there is still space for improvement in the space of
garbage collectors, as mentioned here:
- http://codespeak.net/pypy/extradoc/talk/osdc2008/paper.txt
working on that could be interesting.

The main idea we wanted to apply to Python, based on a suggestion from
Lars, is the main peculiar V8 idea, the one of hidden classes with
transitions. That allows avoiding dictionary lookups for property
accesses on objects, using instead a Java-like data representation in
memory, without any visible effects on the Python semantics.

While revising this email, we saw that this is already partially
implemented: for what the blog says, the memory savings where done but
not the memory speedups. Also, the blog does not mention whether and
how class transitions are used. Some of the initial ideas below may
already have been implemented, so point out what is there and what can
be interesting.
I've also read the discussion about the JIT, which will be missing in
the next release.

Obviously, this would require partial reworking the core object model of PyPy.

Still, we think that specialization will especially benefit from this,
even because this allows to specialize better also on class types, not
only on primitive types. It is not clear to me how do you handle
changes (like addition of a property) to a single object - is it
possible to specialize the code on the (anonymous) type of this new
object? Can you give me pointers inside the documentation, if this is
already explained? I've read the EU reports about this and partial
compilation, but it is still unclear to us how things work.

Btw, the first thing to do is studying the object model and reflection
capability of Python. Since Lars said that the target of a VM designer
is to allow programmers work in the most pleasant way and use advanced
features without incurring any cost, our development plan should try
to optimize also reflective features as much as possible.

Just to make an example, in Python you can register an handler for
unknown method calls on an object (I don't remember how, but I'm
pretty sure something like this exists). This idea means that our
design should be able to do inline caching by storing a call to this
handler (maybe we'll defer the implementation, but the initial design
should allow doing that IMHO).

Any comments?
--
Paolo Giarrusso

From arigo at tunes.org  Wed Oct 29 11:29:29 2008
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 29 Oct 2008 11:29:29 +0100
Subject: [pypy-dev] Possible student project on PyPy
In-Reply-To: <a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
References: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
	<a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
Message-ID: <20081029102929.GA8041@code0.codespeak.net>

Hi Paolo,

Thanks for the introduction e-mail.  I'd like to answer it more
directly, but maybe I should first give a quick overview of what I think
are possible misunderstandings.

The main issue is that unlike Google V8, in PyPy there is no machine
code generation at all (ignoring the JIT compiler generator for the
first part of this e-mail).  That means that the notion of inline cache
simply doesn't make sense.  Let me explain this in more details.

What we implemented long ago in PyPy is completely similar to the "maps"
of the Self language.  It is a dictionary implementation that stores the
keys in a separate, shared "map" object, and that stores the values in a
flat list.  In PyPy this thing still presents the interface of a
full-featured dictionary, so it supports adding and removing arbitrary
elements, which is done by changing the dictionary's current map.

This saves memory, as you know.  But I don't see how this can provide
much speed-up in a pure interpreter (as opposed to in generated machine
code with inline caches).  It seems to me that whatever you do, you need
to perform a lookup *somewhere* in order to implement the "getattr"
operation.  The lookup can be done directly in the object's dictionary,
or it can be done in the shadow "map" object, but the point is that in
both cases it needs to be done at runtime for every attribute access.

There are probably advanced ways to cache recent lookups, which we
didn't try so far, like attaching to each GETATTR opcode in the Python
bytecode a tiny structure that caches the result of the most recent
lookup performed at that source code position.  I don't know how much
speed-ups this would give; in fact it is rather unclear to me why people
think that dictionary lookups are seriously bad.  Of course they are
more expensive than just reading an item out of a list, but I don't
think it's worse than roughly 3 times slower; this number looks big in
the context of machine code, but insignificant in the context of a
complicated bytecode interpreter.  I'm ready to be proven wrong, of
course.

However, this is only half of the story.  Another very important point
is that the mid-term goal for PyPy is to have again a JIT compiler
generator (it did so in the 1.0 release, but it was far from providing
consistent speed-ups, which is the reason that we are re-experimenting
at the moment).  In the context of our JIT, the inline cache technique
you describe -- as well as the polymorphic inline caches of Java VMs --
are special cases of a generic construct called "promotion" in [1] or
"unlifting" in Psyco [2].  Using this construct, the JIT compiler
generator takes as input our "dictionary-with-maps"-enabled interpreter
and produces as output a JIT compiler that itself emits inline caches
similar to Google V8's.  Another way to put it is that in PyPy the real
purpose of our "maps", besides saving memory, is to be
JIT-generator-friendly, enabling good speed-ups in the machine code
generated by the JIT (and not in the pure interpreter).  Thus, what we
implemented in the interpreter is enough already; the main missing piece
is a (completely generic) JIT generator producing a consistent JIT.
Note that this is not just an abstract wish: such inline caches is what
Psyco emits already; and the automatically generated JIT of PyPy 1.0
definitely did that too.

To repeat a point that we tried to make a lot of times but apparently
keep failing to: in the context of the PyPy JIT there is no point in
thinking deeply about the various semantics of Python specifically, like
reflection, operator overloading, the handlers for sends to unsupported
methods (that's __getattr__ in Python), and so on.  These semantics are
encoded in the interpreter already, where the JIT compiler generator can
see them and use them to produce a JIT that emits reasonably efficient
machine code for a given user program.  The relevant question in this
context is not how to encode some complicated Python semantics in the
machine code; the relevant question is to identify the cases where the
generated JIT emits definitely inefficient code, and how to tweak the
interpreter in order to be more JIT-friendly (e.g. by adding maps into
the interpreter).

In an attempt to make it clearer that thinking about advanced Python
semantics is not useful in this context, the "Python" in the previous
paragraph can be replaced by any language, preferably dynamic, and we
still get a JIT for it (e.g. we did it for Prolog, and we have partial
or in-progress interpreters for JavaScript and Smalltalk and other
languages).

Does the above make any sense?  There are three relatively distinct
areas to work on -- tweaking the interpreter to maximize its speed,
tweaking it to maximize its JIT-friendlyness (better done after the JIT
generator is ready), or helping develop the JIT generator itself; I hope
the above helped clarify a bit this aspect of PyPy.  In any case, we
would definitely be interested in working with you in these areas, or in
the other areas that you mentioned in your e-mail (GC...).  Feel free to
discuss matters further, here or on irc (#pypy on irc.freenode.net).


    [1] http://codespeak.net/pypy/dist/pypy/doc/jit/
    [2] http://psyco.sourceforge.net/doc.html [PEPM'04 paper]


A bientot,

Armin.

From cfbolz at gmx.de  Wed Oct 29 15:40:15 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Wed, 29 Oct 2008 15:40:15 +0100
Subject: [pypy-dev] Possible student project on PyPy
In-Reply-To: <a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
References: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
	<a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
Message-ID: <490875CF.6010303@gmx.de>

Paolo Giarrusso wrote:
> Hi, for a student project, we are evaluating the possibility to
> experiment with some ideas on PyPy.
> 
> Even if it is a student project, we have high expectations (our aim is
> to really improve PyPy's speed, if possible). But we are still trying
> to choose which features to implement (we started with some nice
> ideas, but part of them is already implemented).
> 
> We will work for something like 2 months on a separate branch - maybe
> we can give readonly access to the source code repository, but write
> access would be a problem for our exam, for obvious reasons.
> 
> Do you have any possibility to host a development branch for this project?
> 
> To let you know who we are, I'll present us:
> 
> - I, Paolo 'Blaisorblade' Giarrusso, am a past Linux kernel hacker, I
> worked also with Java, C++, and a bit of Python, and I am currently a
> graduate student at Aarhus University, Denmark;
> - Sigurd is a PhD candidate at Aarhus University, Denmark, currently
> working, among other things, on a cryptography research project in
> Python (viff.dk).
> 
> - our professor is Lars Bak, the lead architect of the Google V8
> Javascript engine, on which we implemented various optimizations in
> the previous months.
> 
> We are obviously open to suggestions, and we have been looking at the
> status blog and at various blogs.
> 
> It seems that there is still space for improvement in the space of
> garbage collectors, as mentioned here:
> - http://codespeak.net/pypy/extradoc/talk/osdc2008/paper.txt
> working on that could be interesting.
> 

Just to go a bit in a different direction than what Armin already wrote:
Yes, we would be very interested in having somebody work on our GCs.
They currently work, but they are not very advanced in many respects. So
if somebody is interested in having a go at them, that would probably be
quite useful for PyPy. I can't really say how interesting it would be
research-wise, though.

Cheers,

Carl Friedrich

From andrewfr_ice at yahoo.com  Fri Oct 31 20:45:00 2008
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Fri, 31 Oct 2008 12:45:00 -0700 (PDT)
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
Message-ID: <361057.6962.qm@web34203.mail.mud.yahoo.com>

Hello Colleagues:

I am not sure if this is the right place to ask....

I am trying to get RPython to work. I get the following problem:

[translation:ERROR]     from compiler import parse, ast, pycodegen
[translation:ERROR]  ImportError: cannot import name parse

I also installed Pylint including the Logilab-astng package. I get the error:
 
no module called common.compat.

I read a post stating the problem may be with an __init__.py file. I am using Xandros on an ASUS eee (similar to Ubuntu). Any help would be appreciated.

Cheers,
Andrew








      

From fijall at gmail.com  Fri Oct 31 20:51:15 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 31 Oct 2008 20:51:15 +0100
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
In-Reply-To: <361057.6962.qm@web34203.mail.mud.yahoo.com>
References: <361057.6962.qm@web34203.mail.mud.yahoo.com>
Message-ID: <693bc9ab0810311251s113cb564i1f51e275546e73c1@mail.gmail.com>

On Fri, Oct 31, 2008 at 8:45 PM, Andrew Francis <andrewfr_ice at yahoo.com> wrote:
> Hello Colleagues:
>
> I am not sure if this is the right place to ask....
>
> I am trying to get RPython to work. I get the following problem:
>
> [translation:ERROR]     from compiler import parse, ast, pycodegen
> [translation:ERROR]  ImportError: cannot import name parse

You cannot use a compiler package when writing rpython code. I'm not
sure what you're trying to do, but if your goal is to translate
existing python program using rpython toolchain, this will not work
(almost certainly).

>
> I also installed Pylint including the Logilab-astng package. I get the error:
>
> no module called common.compat.
>
> I read a post stating the problem may be with an __init__.py file. I am using Xandros on an ASUS eee (similar to Ubuntu). Any help would be appreciated.

I think it's not the right place to ask (at least i have no clue).

From adimasci at gmail.com  Fri Oct 31 22:23:46 2008
From: adimasci at gmail.com (Adrien Di Mascio)
Date: Fri, 31 Oct 2008 22:23:46 +0100
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
In-Reply-To: <693bc9ab0810311251s113cb564i1f51e275546e73c1@mail.gmail.com>
References: <361057.6962.qm@web34203.mail.mud.yahoo.com>
	<693bc9ab0810311251s113cb564i1f51e275546e73c1@mail.gmail.com>
Message-ID: <437de8500810311423v19f49e8fvc47428711ac58da2@mail.gmail.com>

Hi there,

On 10/31/08, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Fri, Oct 31, 2008 at 8:45 PM, Andrew Francis <andrewfr_ice at yahoo.com> wrote:
>  > I also installed Pylint including the Logilab-astng package. I get the error:
>  >
>  > no module called common.compat.
>  >
>  > I read a post stating the problem may be with an __init__.py file. I am using Xandros on an ASUS eee (similar to Ubuntu). Any help would be appreciated.
>
>
> I think it's not the right place to ask (at least i have no clue).

This is indeed not the right place to ask :-). Try
python-projects at logilab.org instead for any pylint related question.
Juste to make a quick answer, your problem is probably just
that you didn't install the logilab.common package
(http://www.logilab.org/project/logilab-common)

Cheers,
Adrien.

From andrewfr_ice at yahoo.com  Sat Nov  1 14:26:14 2008
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Sat, 1 Nov 2008 06:26:14 -0700 (PDT)
Subject: [pypy-dev] Rpython-mode Re: Problems getting Translate.py and
	Pylint to work
In-Reply-To: <437de8500810311423v19f49e8fvc47428711ac58da2@mail.gmail.com>
Message-ID: <53669.23721.qm@web34208.mail.mud.yahoo.com>


Hello Maciej and Adrien:

I am new to PyPy so excuse me for silly questions....

> > I am trying to get RPython to work. I get the
> following problem:

...

> You cannot use a compiler package when writing rpython
> code. I'm not
> sure what you're trying to do, but if your goal is to
> translate
> existing python program using rpython toolchain, this will
> not work
> (almost certainly).

I was testing out RPython by following Christopher Armstrong's tutorial "How to do Something in RPython" (http://radix.twistedmatrix.com/2006/12/how-to-do-something-with-rpython-part-1.html). Have things changed?

> This is indeed not the right place to ask :-). Try
> python-projects at logilab.org instead for any pylint related
> question.
> Juste to make a quick answer, your problem is probably just
> that you didn't install the logilab.common package
> (http://www.logilab.org/project/logilab-common)

Thanks for the hint. I'll ask future questions in the Logilab mailing list. The main reason I asked here is that I believe pylint has the --rpython-mode. I saw the following in one of the pypy mailing list archives.

"it is faster and therefore provides feedback faster than  ``translate.py``
+
+* it does not stop at the first problem it finds, so you can get more
+  feedback on the code in one run"


Cheers,
Andrew



      

From arigo at tunes.org  Sat Nov  1 18:05:45 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 1 Nov 2008 18:05:45 +0100
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
In-Reply-To: <361057.6962.qm@web34203.mail.mud.yahoo.com>
References: <361057.6962.qm@web34203.mail.mud.yahoo.com>
Message-ID: <20081101170545.GA27305@code0.codespeak.net>

Hi Andrew,

On Fri, Oct 31, 2008 at 12:45:00PM -0700, Andrew Francis wrote:
> [translation:ERROR]     from compiler import parse, ast, pycodegen

"from compiler import parse" works for me on all Python versions 2.2 to
2.6.  Either your Python installation is broken, or you have a module
called "compiler.py" that hides the package "compiler" from the stdlib.


Armin

From andrewfr_ice at yahoo.com  Mon Nov  3 21:34:53 2008
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Mon, 3 Nov 2008 12:34:53 -0800 (PST)
Subject: [pypy-dev] Problems getting Translate.py and Pylint to work
In-Reply-To: <20081101170545.GA27305@code0.codespeak.net>
Message-ID: <323239.10478.qm@web34201.mail.mud.yahoo.com>

Hello Armin:

Thanks! Pylint now runs. The problem was I had a compiler package. I also managed to get translate to compile the Christopher Armstrong example. Sweet!

Cheers,
Andrew 


--- On Sat, 11/1/08, Armin Rigo <arigo at tunes.org> wrote:

> From: Armin Rigo <arigo at tunes.org>
> Subject: Re: [pypy-dev] Problems getting Translate.py and Pylint to work
> To: "Andrew Francis" <andrewfr_ice at yahoo.com>
> Cc: pypy-dev at codespeak.net
> Date: Saturday, November 1, 2008, 10:05 AM
> Hi Andrew,
> 
> On Fri, Oct 31, 2008 at 12:45:00PM -0700, Andrew Francis
> wrote:
> > [translation:ERROR]     from compiler import parse,
> ast, pycodegen
> 
> "from compiler import parse" works for me on all
> Python versions 2.2 to
> 2.6.  Either your Python installation is broken, or you
> have a module
> called "compiler.py" that hides the package
> "compiler" from the stdlib.
> 
> 
> Armin


      

From andrewfr_ice at yahoo.com  Mon Nov  3 22:32:04 2008
From: andrewfr_ice at yahoo.com (Andrew Francis)
Date: Mon, 3 Nov 2008 13:32:04 -0800 (PST)
Subject: [pypy-dev] Questions about Moving from Stackless to PyPy
Message-ID: <963585.74543.qm@web34206.mail.mud.yahoo.com>

Hi Folks:

I am really new to PyPy. However I have been using Stackless. I would like to do two things. First convert a pre-existing Python programme, A scaled down WS-BPEL processor to RPython and in the process use the Stackless module. I would want the end-product to be a C programme or a C library. I am interested stuff like 1) Will the Picking still work? 2) How to glue this to Twisted?

The second thing I would like to do if the first part works - is convert WS-BPEL code (WS-BPEL is an XML based programming language) into a C library too (I have write a very crude WS-BPEL to Python translator).

Is what I want to do viable? What are some of things the pitfalls? Thanks in advance.

Cheers,
Andrew


      

From daniel.furrer at gmail.com  Fri Nov 14 15:48:11 2008
From: daniel.furrer at gmail.com (Daniel Furrer)
Date: Fri, 14 Nov 2008 15:48:11 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
Message-ID: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>

[This message has been posted to python-dev as well. Sorry for the
duplication to those who read both.]

Hello everybody,

As part of an advanced compiler design course at our university (ETH
Zurich), we have to implement an optimization (or various thereof).
I've spoken with a couple of people who are, like me, very fascinated by
Python.
So I would just like to ask if somebody has any ideas on what we could do?

Our constraints:
- We are 4 persons, each willing to potentially spend around 4-6 full days
of work.
- We've been working on generating Control Flow Graphs, generating Static
Single Assignment Form (inserting phi-nodes) and removing it again. We've
also implemented techniques like constant folding, copy propagation, Common
Subexpression Elimination etc.

We're looking for:
- Hints as to which python compiler would be best to work on. (The official
Python compiler package seems very nice to work with actually... [and so
does PyPy.])
- If there are any requests in the area of optimization that we could have a
look at

Any feedback would be highly appreciated as well as pointers to specific
bugs in the tracker and other relevant discussions.

Best regards,
Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081114/f786f305/attachment.htm 

From getxsick at gmail.com  Sun Nov 16 02:01:54 2008
From: getxsick at gmail.com (Bartosz SKOWRON)
Date: Sun, 16 Nov 2008 02:01:54 +0100
Subject: [pypy-dev] =?utf-8?q?sprint_in_Wroc=C5=82aw_=28PL=29?=
Message-ID: <77887e110811151701x5c178eedoaadb201b6d6f44c4@mail.gmail.com>

Hello folks!

Fijal asked me to write about possibility of making a sprint in Wroclaw.

There is a possibility to set a sprint at Wroclaw University of
Technology campus for free. But we have a chance to reserve a class
only during winter break. This period is between 30.01 - 22.02. I got
a permission to make a sprint from uni's boss but i have to set the
dates. My proposition is 9.02-15.02 2009. What about you?
Are you interested in this sprint? The town is quite cheap (as whole
Eastern Europe is) and there are a lot of hostels/hotels etc. Also
it's very near to Germany and Czech Rep.

Let me know if you have any questions.

I should reserve a class asap but I need the dates and your opinion :)

cheers!
bart/getxsick

From cfbolz at gmx.de  Mon Nov 17 14:21:53 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 17 Nov 2008 14:21:53 +0100
Subject: [pypy-dev] =?iso-8859-1?q?sprint_in_Wroc=3Faw_=28PL=29?=
In-Reply-To: <77887e110811151701x5c178eedoaadb201b6d6f44c4@mail.gmail.com>
References: <77887e110811151701x5c178eedoaadb201b6d6f44c4@mail.gmail.com>
Message-ID: <49216FF1.4020909@gmx.de>

Hi!

Bartosz SKOWRON wrote:
> Fijal asked me to write about possibility of making a sprint in Wroclaw.
> 
> There is a possibility to set a sprint at Wroclaw University of
> Technology campus for free. But we have a chance to reserve a class
> only during winter break. This period is between 30.01 - 22.02. I got
> a permission to make a sprint from uni's boss but i have to set the
> dates. My proposition is 9.02-15.02 2009. What about you?
> Are you interested in this sprint? The town is quite cheap (as whole
> Eastern Europe is) and there are a lot of hostels/hotels etc. Also
> it's very near to Germany and Czech Rep.
> 
> Let me know if you have any questions.
> 
> I should reserve a class asap but I need the dates and your opinion :)

I would definitely like to have a sprint in Wroclaw! However, for me it
would fit better if it were a week later, since our lecture goes till
the 6th of february and the exams are in the week afterwards. You think
that would be possible too?

Cheers,

Carl Friedrich

From cfbolz at gmx.de  Mon Nov 17 14:26:24 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 17 Nov 2008 14:26:24 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
In-Reply-To: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>
References: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>
Message-ID: <49217100.1020108@gmx.de>

Hi Daniel,

sorry for the late response.

Daniel Furrer wrote:
> As part of an advanced compiler design course at our university (ETH
> Zurich), we have to implement an optimization (or various thereof).
> I've spoken with a couple of people who are, like me, very fascinated by
> Python.
> So I would just like to ask if somebody has any ideas on what we could do?
> 
> Our constraints:
> - We are 4 persons, each willing to potentially spend around 4-6 full
> days of work.
> - We've been working on generating Control Flow Graphs, generating
> Static Single Assignment Form (inserting phi-nodes) and removing it
> again. We've also implemented techniques like constant folding, copy
> propagation, Common Subexpression Elimination etc.
> 
> We're looking for:
> - Hints as to which python compiler would be best to work on. (The
> official Python compiler package seems very nice to work with
> actually... [and so does PyPy.])
> - If there are any requests in the area of optimization that we could
> have a look at
> 
> Any feedback would be highly appreciated as well as pointers to specific
> bugs in the tracker and other relevant discussions.

I guess the problem for "classical" compiler optimizations applied to a
compiler producing Python bytecode is that most of them are potentially
unsafe. E.g. you cannot do CSE in Python, because any expression can
have arbitrary side-effects. Therefore it is very limited which
optimizations can be applied at all.

PyPy's translation toolchain works in SSI as an intermediate
representation (which is a sub-set of SSA). However, many of the
straightforward optimizations (constant-folding, copy-progragation,
inlining, a form of escape analysis) have already been implemented. Some
things are not done yet, like common subexpression elimination. We
didn't bother to implement them yet, because they are not as useful for
PyPy since we target C, and most C compilers can do theses things for us.

Cheers,

Carl Friedrich

From p.giarrusso at gmail.com  Mon Nov 17 14:51:15 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 17 Nov 2008 14:51:15 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
In-Reply-To: <49217100.1020108@gmx.de>
References: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>
	<49217100.1020108@gmx.de>
Message-ID: <a4d0c86c0811170551l144f2cc2mcb30303e542463@mail.gmail.com>

Hi, first I'd like to state I'm not part of the Zurich group, so what
follows are just my 2 cents.

On Mon, Nov 17, 2008 at 14:26, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Hi Daniel,

> Daniel Furrer wrote:
>> We're looking for:
>> - Hints as to which python compiler would be best to work on. (The
>> official Python compiler package seems very nice to work with
>> actually... [and so does PyPy.])

> I guess the problem for "classical" compiler optimizations applied to a
> compiler producing Python bytecode is that most of them are potentially
> unsafe. E.g. you cannot do CSE in Python, because any expression can
> have arbitrary side-effects. Therefore it is very limited which
> optimizations can be applied at all.

I guess that is because an object can define a custom '+' operator,
for instance, right?
But aren't those optimizations possible anyway after specialization
and inlining? And anyway, who said that CSE should be applied to the
Python bytecode? In fact, I think both Java and .NET do it inside the
VM.
Then, the next point is that you do not have yet a JIT. But still, I
think it is possible to specialize bytecode to eliminate dynamic
lookups and enable optimizations such as CSE. But the benefit of
specialized bytecode can be significant, I guess, only if the
interpreter is really fast (either a threaded one, or a code-copying
one). Is the PyPy interpreter threaded?

> PyPy's translation toolchain works in SSI as an intermediate
> representation (which is a sub-set of SSA). However, many of the
> straightforward optimizations (constant-folding, copy-progragation,
> inlining, a form of escape analysis) have already been implemented. Some
> things are not done yet, like common subexpression elimination. We
> didn't bother to implement them yet, because they are not as useful for
> PyPy since we target C, and most C compilers can do theses things for us.

-- 
Paolo Giarrusso

From anto.cuni at gmail.com  Mon Nov 17 15:05:31 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 17 Nov 2008 15:05:31 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
In-Reply-To: <a4d0c86c0811170551l144f2cc2mcb30303e542463@mail.gmail.com>
References: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>	<49217100.1020108@gmx.de>
	<a4d0c86c0811170551l144f2cc2mcb30303e542463@mail.gmail.com>
Message-ID: <49217A2B.1060704@gmail.com>

Paolo Giarrusso wrote:

> specialized bytecode can be significant, I guess, only if the
> interpreter is really fast (either a threaded one, or a code-copying
> one). Is the PyPy interpreter threaded?

sometime ago I tried to measure if/how much we can gain with a threaded 
interpreter. I manually modified the produced C code to make the main 
loop threaded, but we didn't gain anything; I think there are three 
possible reasons:

1) in Python a lot of opcodes are quite complex and time-consuming, so 
the time spent to dispatch to them is a little percentage of the total 
time spent for the execution

2) due to Python's semantics, it's not possible to just jump from one 
opcode to the next, as we need to do a lot of bookkeeping, like 
remembering what was the last line executed, etc. This means that the 
trampolines at the end of each opcode contains a lot code duplication, 
leading to a bigger main loop, with possibly bad effects on the cache 
(didn't measure this, though)

3) it's possible that I did something wrong, so in that case my 
measurements are completely useless :-).  If anyone wants to try again, 
it cannot hurt.

ciao,
Anto

From p.giarrusso at gmail.com  Mon Nov 17 15:32:24 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 17 Nov 2008 15:32:24 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
In-Reply-To: <49217A2B.1060704@gmail.com>
References: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>
	<49217100.1020108@gmx.de>
	<a4d0c86c0811170551l144f2cc2mcb30303e542463@mail.gmail.com>
	<49217A2B.1060704@gmail.com>
Message-ID: <a4d0c86c0811170632q77988fbfqc45c307888a33968@mail.gmail.com>

On Mon, Nov 17, 2008 at 15:05, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Paolo Giarrusso wrote:
>
>> specialized bytecode can be significant, I guess, only if the
>> interpreter is really fast (either a threaded one, or a code-copying
>> one). Is the PyPy interpreter threaded?
>
> sometime ago I tried to measure if/how much we can gain with a threaded
> interpreter. I manually modified the produced C code to make the main loop
> threaded, but we didn't gain anything; I think there are three possible
> reasons:

> 1) in Python a lot of opcodes are quite complex and time-consuming, so the
> time spent to dispatch to them is a little percentage of the total time
> spent for the execution
That's something difficult to believe, I think. Well, it is possible
to profile execution to count mispredictions, to avoid having to think
about it.

Well, since arithmetic ops may involve a method lookup, I understand
what you mean. OTOH, a method lookup means just one dictionary lookup
when first seeing the bytecode (I would save the hashcode inline in
the bytecode after first execution actually, to speed that up), and an
unpredictable indirect branch; making the dispatch branch more
predictable by threading should still have a significant impact.

> 2) due to Python's semantics, it's not possible to just jump from one opcode
> to the next, as we need to do a lot of bookkeeping, like remembering what
> was the last line executed, etc.

This is a more likely culprit.
But... all languages that I know of (Java, compiled languages) just
have a reverse map from bytecode positions to the original line
information, to be used when and if they are needed (i.e. as debug
info, or to unwind the stack when an exception is thrown I guess).
Isn't that possible for Python?
In any

> This means that the trampolines at the end
> of each opcode contains a lot code duplication, leading to a bigger main
> loop, with possibly bad effects on the cache (didn't measure this, though)


> 3) it's possible that I did something wrong, so in that case my measurements
> are completely useless :-).  If anyone wants to try again, it cannot hurt.

Do you have the original code somewhere? I know one has to redo it
anyway even because that's generated code, but still guidance is
helpful.

-- 
Paolo Giarrusso

From p.giarrusso at gmail.com  Mon Nov 17 15:44:50 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 17 Nov 2008 15:44:50 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
In-Reply-To: <a4d0c86c0811170632q77988fbfqc45c307888a33968@mail.gmail.com>
References: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>
	<49217100.1020108@gmx.de>
	<a4d0c86c0811170551l144f2cc2mcb30303e542463@mail.gmail.com>
	<49217A2B.1060704@gmail.com>
	<a4d0c86c0811170632q77988fbfqc45c307888a33968@mail.gmail.com>
Message-ID: <a4d0c86c0811170644t43e6e64fq9d379217d9aa8a5b@mail.gmail.com>

Sorry, I sent the unfinished mail by mistake.

>> 2) due to Python's semantics, it's not possible to just jump from one opcode
>> to the next, as we need to do a lot of bookkeeping, like remembering what
>> was the last line executed, etc.
>
> This is a more likely culprit.
> But... all languages that I know of (Java, compiled languages) just
> have a reverse map from bytecode positions to the original line
> information, to be used when and if they are needed (i.e. as debug
> info, or to unwind the stack when an exception is thrown I guess).
> Isn't that possible for Python?

Additionally, could you make some more examples?

> In any
[the unfinished sentence]
In any case, have you read "The Structure and Performance of E?cient
Interpreters"?, by M. Ertl and D. Gregg? It's a paper that really
matters in practice, for fast interpreters (and if you don't believe
me, believe http://webkit.org/blog/189/announcing-squirrelfish/).

Basically, what it says is that in efficient interpreters, most of the
runtime cost is caused by indirect branches (they go as far as taking
that as a definition of "efficient interpreter"). And that applies
also to Scheme (which, I think, is not less dynamic than Python - any
arith instruction must still check the argument types and choose
between the fast path and throwing an error).

My first impression is that the problem was that the interpreter is
not yet optimized enough, and that comes from various hints:
- You do bookkeeping for the last executed line (unless tables prove
impossible to apply)
- We discussed about removing dictionary lookups for attributes little
time ago, and one PyPy developers said "it doesn't really matter, an
interpreter is so slow anyway".

This is the original quote from Armin Rigo
(http://codespeak.net/pipermail/pypy-dev/2008q4/004862.html):

"I don't know how much
speed-ups this would give; in fact it is rather unclear to me why people
think that dictionary lookups are seriously bad.  Of course they are
more expensive than just reading an item out of a list, but I don't
think it's worse than roughly 3 times slower; this number looks big in
the context of machine code, but insignificant in the context of a
complicated bytecode interpreter.  I'm ready to be proven wrong, of
course."

And anyway, Armin itself suggested there is space for optimizing the
interpreter.
-- 
Paolo Giarrusso

From p.giarrusso at gmail.com  Mon Nov 17 15:49:59 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 17 Nov 2008 15:49:59 +0100
Subject: [pypy-dev] Possible student project on PyPy
In-Reply-To: <490875CF.6010303@gmx.de>
References: <a4d0c86c0810281212u71a2b09ay6d15be678c970346@mail.gmail.com>
	<a4d0c86c0810281220s544c8db5o4338ebf351f36c67@mail.gmail.com>
	<490875CF.6010303@gmx.de>
Message-ID: <a4d0c86c0811170649h1af169cbsf3343f4ee233e07d@mail.gmail.com>

On Wed, Oct 29, 2008 at 15:40, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Paolo Giarrusso wrote:
>> Hi, for a student project, we are evaluating the possibility to
>> experiment with some ideas on PyPy.

I'm sorry to reply so late - basically, after a consultation with the
professors, we were recommended to start out on our own, to really
learn how to write a VM and make it fast, before going on to more
ambitious projects.
The target we have settled on is for now to write a Python bytecode
interpreter with threading and code-copying, and then try to work on
the GC. We'll see if we get any interesting results (but since we are
not aiming to implement the full Python semantics, I guess it will be
pretty easy to be faster than a full-blown interpreter).

Regards
-- 
Paolo Giarrusso

From daniel.furrer at gmail.com  Mon Nov 17 16:04:57 2008
From: daniel.furrer at gmail.com (Daniel Furrer)
Date: Mon, 17 Nov 2008 16:04:57 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
In-Reply-To: <49217100.1020108@gmx.de>
References: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>
	<49217100.1020108@gmx.de>
Message-ID: <bc17039c0811170704r47166614u43077a3327dfa6d2@mail.gmail.com>

Thanks for your answers.

On Mon, Nov 17, 2008 at 2:26 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

> I guess the problem for "classical" compiler optimizations applied to a
> compiler producing Python bytecode is that most of them are potentially
> unsafe. E.g. you cannot do CSE in Python, because any expression can
> have arbitrary side-effects. Therefore it is very limited which
> optimizations can be applied at all.


Well, the side-effects are a problem, so we can not do any method
invocations (unless we would check that the methods are pure). Thinking
about this: it's not even easily possible to optimize the subset of
numerical operations because we don't have any static type information.
woops...



> PyPy's translation toolchain works in SSI as an intermediate
> representation (which is a sub-set of SSA). However, many of the
> straightforward optimizations (constant-folding, copy-progragation,
> inlining, a form of escape analysis) have already been implemented. Some
> things are not done yet, like common subexpression elimination. We
> didn't bother to implement them yet, because they are not as useful for
> PyPy since we target C, and most C compilers can do theses things for us.


That's true. But I would assume they do constant folding and copy
propagation as well then, don't they? (How about PRE?)
So just out of interest: In which area would you start optimizing PyPy?

cheers,
Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081117/bc133cdf/attachment.htm 

From p.giarrusso at gmail.com  Mon Nov 17 16:15:48 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 17 Nov 2008 16:15:48 +0100
Subject: [pypy-dev] compiler optimizations: collecting ideas
In-Reply-To: <bc17039c0811170704r47166614u43077a3327dfa6d2@mail.gmail.com>
References: <bc17039c0811140648s2cd9adfes1a161751b07c99d9@mail.gmail.com>
	<49217100.1020108@gmx.de>
	<bc17039c0811170704r47166614u43077a3327dfa6d2@mail.gmail.com>
Message-ID: <a4d0c86c0811170715v29d05f63h239f102c00face9f@mail.gmail.com>

On Mon, Nov 17, 2008 at 16:04, Daniel Furrer <daniel.furrer at gmail.com> wrote:
> Thanks for your answers.
>
> On Mon, Nov 17, 2008 at 2:26 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
>>
>> I guess the problem for "classical" compiler optimizations applied to a
>> compiler producing Python bytecode is that most of them are potentially
>> unsafe. E.g. you cannot do CSE in Python, because any expression can
>> have arbitrary side-effects. Therefore it is very limited which
>> optimizations can be applied at all.
>
> Well, the side-effects are a problem, so we can not do any method
> invocations (unless we would check that the methods are pure). Thinking
> about this: it's not even easily possible to optimize the subset of
> numerical operations because we don't have any static type information.
> woops...

Indeed, existing work solved this problem in the VM, by means of
runtime specialization (you generate a version of the method for the
commonly used argument types, so that for each specialized version you
have static type information); there is a lot of literature of the
late '80s and early '90s about this, mainly for Smalltalk, but also on
Self.

Some recommended papers on this are:
* A Survey of Adaptive Optimization in Virtual Machines, Matthew
Arnold, Stephen J. Find, David Grove, Michael Hind, and Peter F.
Sweeney, IEEE.
* An Efficient Implementation of SELF, a Dynamically-Typed
Object-Oriented Language Based on Prototypes, Craig Chambers, David
Ungar, and Elgin Lee, 1991 (specific discussion of code
specialization, for languages such as JavaScript and Python).

Regards
-- 
Paolo Giarrusso

From getxsick at gmail.com  Tue Nov 18 23:52:00 2008
From: getxsick at gmail.com (Bartosz SKOWRON)
Date: Tue, 18 Nov 2008 23:52:00 +0100
Subject: [pypy-dev] sprint in Wroc?aw (PL)
In-Reply-To: <49216FF1.4020909@gmx.de>
References: <77887e110811151701x5c178eedoaadb201b6d6f44c4@mail.gmail.com>
	<49216FF1.4020909@gmx.de>
Message-ID: <77887e110811181452p5eae3f7bnb30c1ad8b9c42cae@mail.gmail.com>

On Mon, Nov 17, 2008 at 2:21 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

> I would definitely like to have a sprint in Wroclaw! However, for me it
> would fit better if it were a week later, since our lecture goes till
> the 6th of february and the exams are in the week afterwards. You think
> that would be possible too?

I can't guarentee if it's possible for 9th-15 of februarry neither
16th to 22th...i have to go and check it, but would be better if we
could set initial date, because it's easier to force something focused
than not specified date ;)
So, 9.02 - 15.02 2009 is still ok (we have still winter break then).
Any feedback from others?

From glavoie at gmail.com  Fri Nov 21 04:13:42 2008
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Thu, 20 Nov 2008 22:13:42 -0500
Subject: [pypy-dev] PyPy under FreeBSD 7
Message-ID: <d05df8620811201913l46bd3223r7ccba26c052ed3d8@mail.gmail.com>

Hello,
     I'm trying to translate PyPy into pypy-c on FreeBSD 7.0 but at the
beginning of the translation process I get the following error,

NotImplementedError("don't know how to get the C-level errno!")

from the file "pypy/rpython/lltypesystem/ll2ctypes.py". How can I solve this
problem? It seems it's python-ctypes related. I'm using Python 2.5.3 (from
ports tree) and when I tried to build the port "devel/py-ctypes", it told me
that ctypes was provided with Python 2.5.x.

Thanks!

Gabriel

-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081120/a8d7dc73/attachment.htm 

From fijall at gmail.com  Fri Nov 21 10:22:31 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 21 Nov 2008 10:22:31 +0100
Subject: [pypy-dev] PyPy under FreeBSD 7
In-Reply-To: <d05df8620811201913l46bd3223r7ccba26c052ed3d8@mail.gmail.com>
References: <d05df8620811201913l46bd3223r7ccba26c052ed3d8@mail.gmail.com>
Message-ID: <693bc9ab0811210122qfcb6b04m7996cc2dcd868397@mail.gmail.com>

There is no reasonable, reliable way of getting errno under different
operating systems. That's why we introduced per-system hack at the
very bottom of ll2ctypes that tries to get errno. Can you think about
a way to do it on FreeBSD? There is also nice, generic way to do it on
top of python2.6, if you're up to trying it on it, I can provide the
fix (it's not implemented yet).

Cheers,
fijal

On Fri, Nov 21, 2008 at 4:13 AM, Gabriel Lavoie <glavoie at gmail.com> wrote:
> Hello,
>      I'm trying to translate PyPy into pypy-c on FreeBSD 7.0 but at the
> beginning of the translation process I get the following error,
>
> NotImplementedError("don't know how to get the C-level errno!")
>
> from the file "pypy/rpython/lltypesystem/ll2ctypes.py". How can I solve this
> problem? It seems it's python-ctypes related. I'm using Python 2.5.3 (from
> ports tree) and when I tried to build the port "devel/py-ctypes", it told me
> that ctypes was provided with Python 2.5.x.
>
> Thanks!
>
> Gabriel
>
> --
> Gabriel Lavoie
> glavoie at gmail.com
>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

From anto.cuni at gmail.com  Fri Nov 21 17:04:47 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 21 Nov 2008 17:04:47 +0100
Subject: [pypy-dev] [pypy-svn] r60055 -
	pypy/trunk/pypy/interpreter/pyparser/test
In-Reply-To: <20081121153708.BF41E168440@codespeak.net>
References: <20081121153708.BF41E168440@codespeak.net>
Message-ID: <4926DC1F.7090400@gmail.com>

fijal at codespeak.net wrote:

> +def setup_module(mod):
> +    if sys.version > '2.5':
> +        py.test.skip("Fails on top of cpy 2.5 for messy reasons, investigate")
> +

maybe I just misunderstood what you want to do, but are you sure you 
don't want to also check if we are running on top of cpython or on top 
of pypy-c? Else you will skip the test once we fix our own version number.

ciao,
Anto

From fijall at gmail.com  Fri Nov 21 17:16:28 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 21 Nov 2008 17:16:28 +0100
Subject: [pypy-dev] [pypy-svn] r60055 -
	pypy/trunk/pypy/interpreter/pyparser/test
In-Reply-To: <4926DC1F.7090400@gmail.com>
References: <20081121153708.BF41E168440@codespeak.net>
	<4926DC1F.7090400@gmail.com>
Message-ID: <693bc9ab0811210816u4e2a2b5o8a38b98c31fc07d3@mail.gmail.com>

I'm pretty sure it'll fail on top of pypy-c anyway, so yes, I wanted to :)

On Fri, Nov 21, 2008 at 5:04 PM, Antonio Cuni <anto.cuni at gmail.com> wrote:
> fijal at codespeak.net wrote:
>
>> +def setup_module(mod):
>> +    if sys.version > '2.5':
>> +        py.test.skip("Fails on top of cpy 2.5 for messy reasons,
>> investigate")
>> +
>
> maybe I just misunderstood what you want to do, but are you sure you don't
> want to also check if we are running on top of cpython or on top of pypy-c?
> Else you will skip the test once we fix our own version number.
>
> ciao,
> Anto
>

From tbaldridge at gmail.com  Sat Nov 22 05:21:06 2008
From: tbaldridge at gmail.com (Timothy Baldridge)
Date: Fri, 21 Nov 2008 22:21:06 -0600
Subject: [pypy-dev] PyPy Talks anyone got a copy?
Message-ID: <b33fdb110811212021o1fed6e70q1157b92a4625431e@mail.gmail.com>

Hey, I'm trying to get a copy of the PyPy talks, but there doesn't
seem to be anyone seeding the torrents. Does anyone have a HTTP copy I
could download? If so I could host them on my HTTP server for others
to download.

Timothy

-- 
Two wrights don't make a rong, they make an airplane. Or bicycles.

From holger at merlinux.eu  Sat Nov 22 10:25:23 2008
From: holger at merlinux.eu (holger krekel)
Date: Sat, 22 Nov 2008 10:25:23 +0100
Subject: [pypy-dev] PyPy Talks anyone got a copy?
In-Reply-To: <b33fdb110811212021o1fed6e70q1157b92a4625431e@mail.gmail.com>
References: <b33fdb110811212021o1fed6e70q1157b92a4625431e@mail.gmail.com>
Message-ID: <20081122092523.GZ2219@trillke.net>

Hi Timothy, 

i am currently loading them up to: 

    http://codespeak.net/download/video-pypy/

from which you could copy them to your server. 
(it will still take a while) 

Once that is done i'd make the links in the video-index page
point to your server. Torrent is not neccessary, i think, 
because the talks are 2-3 years old and not requested much. 

thanks & cheers,
holger

On Fri, Nov 21, 2008 at 22:21 -0600, Timothy Baldridge wrote:
> Hey, I'm trying to get a copy of the PyPy talks, but there doesn't
> seem to be anyone seeding the torrents. Does anyone have a HTTP copy I
> could download? If so I could host them on my HTTP server for others
> to download.
> 
> Timothy
> 
> -- 
> Two wrights don't make a rong, they make an airplane. Or bicycles.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 

-- 
collaborative expert contracting: http://merlinux.eu 
PyPy  Python/Compiler tool chain: http://codespeak.net/pypy 
pylib py.test/greenlets/svn APIs: http://pylib.org 

From glavoie at gmail.com  Tue Nov 25 02:32:43 2008
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Mon, 24 Nov 2008 20:32:43 -0500
Subject: [pypy-dev] PyPy under FreeBSD 7
In-Reply-To: <693bc9ab0811210122qfcb6b04m7996cc2dcd868397@mail.gmail.com>
References: <d05df8620811201913l46bd3223r7ccba26c052ed3d8@mail.gmail.com>
	<693bc9ab0811210122qfcb6b04m7996cc2dcd868397@mail.gmail.com>
Message-ID: <d05df8620811241732t3cf5874cod34573943836c8e7@mail.gmail.com>

PyPy is currently building on my FreeBSD 7 with no problem at the moment.
I'll be able to tell more later. I had to modify a few files and create a
new platform to specify some include/lib paths. I'll have to see with you
Marciej on the best way to integrate that. About "errno", FreeBSD 7 does it
the Darwin way, probably to be POSIX compliant. Else, there are a few
includes that have to be added and some lib incudes that have to be removed
in order to pass the platform_check part. As I write these lines, the
translation just failed with the following trace:


[rtyper] specializing: 76200 / 83054 blocks   (91%)
[rtyper] specializing: 79800 / 83054 blocks   (96%)
[rtyper] -=- specialized 83054 blocks -=-
****************%%%%%%%%%%%%%%%%%%%%%************+++++++++......................

..%........................................................+++++++++++++++++++++
...*#%+..............................................++++++
[rtyper] specializing: 83100 / 85101 blocks   (97%)
[Timer] Timings:
[Timer] annotate                       --- 516.9 s
[Timer] rtype_lltype                   --- 443.3 s
[Timer] ==========================================
[Timer] Total:                         --- 960.2 s
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 275, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/translator/driver.py", line 805, in
proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/translator/tool/taskengine.py", line
116, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/translator/driver.py", line 269, in
_do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/translator/driver.py", line 344, in
task_rtype_lltype
[translation:ERROR]     crash_on_first_typeerror=insist)
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py", line 180, in
specialize
[translation:ERROR]     self.specialize_more_blocks()
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py", line 252, in
specialize_more_blocks
[translation:ERROR]     annmixlevel.finish()
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/annlowlevel.py", line 240, in
finish
[translation:ERROR]     self.finish_rtype()
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/annlowlevel.py", line 290, in
finish_rtype
[translation:ERROR]     rtyper.specialize_more_blocks()
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py", line 226, in
specialize_more_blocks
[translation:ERROR]     self.specialize_block(block)
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py", line 347, in
specialize_block
[translation:ERROR]     self.gottypererror(e, block, hop.spaceop, newops)
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py", line 345, in
specialize_block
[translation:ERROR]     self.translate_hl_to_ll(hop, varmapping)
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py", line 474, in
translate_hl_to_ll
[translation:ERROR]     resultvar = hop.dispatch()
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py", line 709, in
dispatch
[translation:ERROR]     return translate_meth(self)
[translation:ERROR]    File "<codegen
/home/wildchild/devel/pypy-dist/pypy/rpython/rtyper.py:558>", line 5, in
translate_op_eq
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rint.py", line 161, in
rtype_eq
[translation:ERROR]     return _rtype_compare_template(hop, 'eq')
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rint.py", line 268, in
_rtype_compare_template
[translation:ERROR]     return hop.genop(repr.opprefix+func, vlist,
resulttype=Bool)
[translation:ERROR]    File
"/home/wildchild/devel/pypy-dist/pypy/rpython/rmodel.py", line 339, in
_get_opprefix
[translation:ERROR]     self.lowleveltype)
[translation:ERROR]  TyperError: arithmetic not supported on <INT>
[translation:ERROR] .. block at 60 with 2 exits(v443)
[translation:ERROR] .. v445 = eq(result_0, v444)
[translation] start debugger...
> /home/wildchild/devel/pypy-dist/pypy/rpython/rmodel.py(339)_get_opprefix()
-> self.lowleveltype)
(Pdb+)


Any idea?

Thanks

Gabriel

2008/11/21 Maciej Fijalkowski <fijall at gmail.com>

> There is no reasonable, reliable way of getting errno under different
> operating systems. That's why we introduced per-system hack at the
> very bottom of ll2ctypes that tries to get errno. Can you think about
> a way to do it on FreeBSD? There is also nice, generic way to do it on
> top of python2.6, if you're up to trying it on it, I can provide the
> fix (it's not implemented yet).
>
> Cheers,
> fijal
>
> On Fri, Nov 21, 2008 at 4:13 AM, Gabriel Lavoie <glavoie at gmail.com> wrote:
> > Hello,
> >      I'm trying to translate PyPy into pypy-c on FreeBSD 7.0 but at the
> > beginning of the translation process I get the following error,
> >
> > NotImplementedError("don't know how to get the C-level errno!")
> >
> > from the file "pypy/rpython/lltypesystem/ll2ctypes.py". How can I solve
> this
> > problem? It seems it's python-ctypes related. I'm using Python 2.5.3
> (from
> > ports tree) and when I tried to build the port "devel/py-ctypes", it told
> me
> > that ctypes was provided with Python 2.5.x.
> >
> > Thanks!
> >
> > Gabriel
> >
> > --
> > Gabriel Lavoie
> > glavoie at gmail.com
> >
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> >
>



-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081124/f461d632/attachment.htm 

From arigo at tunes.org  Tue Nov 25 13:54:38 2008
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 25 Nov 2008 13:54:38 +0100
Subject: [pypy-dev] sprint in Wroc?aw (PL)
In-Reply-To: <77887e110811181452p5eae3f7bnb30c1ad8b9c42cae@mail.gmail.com>
References: <77887e110811151701x5c178eedoaadb201b6d6f44c4@mail.gmail.com>
	<49216FF1.4020909@gmx.de>
	<77887e110811181452p5eae3f7bnb30c1ad8b9c42cae@mail.gmail.com>
Message-ID: <20081125125438.GA28831@code0.codespeak.net>

Hi,

On Tue, Nov 18, 2008 at 11:52:00PM +0100, Bartosz SKOWRON wrote:
> > I would definitely like to have a sprint in Wroclaw! However, for me it
> > would fit better if it were a week later, since our lecture goes till
> > the 6th of february and the exams are in the week afterwards. You think
> > that would be possible too?

> I can't guarentee if it's possible for 9th-15 of februarry neither
> 16th to 22th...i have to go and check it, but would be better if we
> could set initial date, because it's easier to force something focused
> than not specified date ;)
> So, 9.02 - 15.02 2009 is still ok (we have still winter break then).
> Any feedback from others?

That's annoying.  It means I am busy at the University with Carl until
the 13th of Febrary; then the following week I am absent.  This means
that it's only possible for me starting from the 23th of February.


A bientot,

Armin.

From phyo.arkarlwin at gmail.com  Wed Nov 26 12:29:10 2008
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Wed, 26 Nov 2008 17:59:10 +0630
Subject: [pypy-dev] Some Newbie Questions
Message-ID: <bcc23fcc0811260329m46085cc5iebae2be02f4a4919@mail.gmail.com>

Hi Pypy Devs!

I love python so much and I am so exicted by this ambitious project!

What I understand about this Project is to develop a version of python
, which is written in Pure Python .  SO now , i do not have C Skills
but I can Develop , Tweak , Make changes to python Just in Python , Am
I Right? SO In Future , it is possible that Guido himself will be
developing python in PyPy ?

Question : I have to use Python Or  RPython?

Also  By doing so , We can convert Python to Any langauge of Choice ,
which is very interesting.

I have build pypy-c with optimization Level 3 and stackless , whcih is
done flawlessly. I have seen benchmarks on one of the page and now
PyPy is even faster then Normal python 2.5 827 vs 810 ms on Richard
Test, it is very interesting .

Question : Resulting pypy-c is python 2.4.1 , is there python 2.5
Version of pypy-c avaliable? I saw a branch about it before by a GSOC
student but now is that already marged ?(my version svn is 4 days ago)

If that arleady marged , how can i build it to get pypy-c with python
2.5 support?

Another Questions is , If i follow my Python code in RPython , Can i
converty RPythonic Code to C and compile ? SO it is possilbe to
convert Existing Python Projects into C (if they follow Rpython)?

If So , how can it be done ? An example , i will be very appreciate .


Another , i compiled pypy-c in stackless , which is faster in term of
pypy-c's perfomance? normal or stackless?

If theres better performance options please let me know.

Also doing  , what are the extra modules added with --allworkingmodules ?

I am hoping in near future , i can help in this project after enough
understanding .

Regards,

Phyo.

From fijall at gmail.com  Wed Nov 26 12:44:38 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 26 Nov 2008 12:44:38 +0100
Subject: [pypy-dev] Some Newbie Questions
In-Reply-To: <bcc23fcc0811260329m46085cc5iebae2be02f4a4919@mail.gmail.com>
References: <bcc23fcc0811260329m46085cc5iebae2be02f4a4919@mail.gmail.com>
Message-ID: <693bc9ab0811260344y2df77928h9b69d4005c1fac9b@mail.gmail.com>

On Wed, Nov 26, 2008 at 12:29 PM, Phyo Arkar <phyo.arkarlwin at gmail.com> wrote:
> Hi Pypy Devs!
>
> I love python so much and I am so exicted by this ambitious project!
>
> What I understand about this Project is to develop a version of python
> , which is written in Pure Python .  SO now , i do not have C Skills
> but I can Develop , Tweak , Make changes to python Just in Python , Am
> I Right? SO In Future , it is possible that Guido himself will be
> developing python in PyPy ?

That's possible, although far in the future. Look here:
http://moderator.appspot.com/ and search for pypy.

>
> Question : I have to use Python Or  RPython?

Have to use for what? If you want to participate in PyPy project, you
probably need to know a bit about rpython, otherwise no, you don't.
That can be gradual and relatively painless though :) (probably better
than learning C or CPython's C API)

>
> Also  By doing so , We can convert Python to Any langauge of Choice ,
> which is very interesting.

No, only restricted subset of it, RPython.

>
> I have build pypy-c with optimization Level 3 and stackless , whcih is
> done flawlessly. I have seen benchmarks on one of the page and now
> PyPy is even faster then Normal python 2.5 827 vs 810 ms on Richard
> Test, it is very interesting .
>
> Question : Resulting pypy-c is python 2.4.1 , is there python 2.5
> Version of pypy-c avaliable? I saw a branch about it before by a GSOC
> student but now is that already marged ?(my version svn is 4 days ago)

Yes, it's merged into pypy-trunk. Download it from svn from here:
http://codespeak.net/svn/pypy/trunk/

>
> If that arleady marged , how can i build it to get pypy-c with python
> 2.5 support?

Exactly like 2.4, just use trunk.

>
> Another Questions is , If i follow my Python code in RPython , Can i
> converty RPythonic Code to C and compile ? SO it is possilbe to
> convert Existing Python Projects into C (if they follow Rpython)?

The simple answer is: they don't follow RPython. I'm pretty sure about
this. You need to rewrite them from scratch, at least in parts for
that. It's not very helpful.

> Another , i compiled pypy-c in stackless , which is faster in term of
> pypy-c's perfomance? normal or stackless?

normal, although not by much. There is also an experimental option
--gcrootfinder=asmgcc which makes it even faster, but does not
cooperate well with ctypes for example.

> Also doing  , what are the extra modules added with --allworkingmodules ?

You should have list when you start compiling. For linux machine, it's:

    ["_socket", "unicodedata", "mmap", "fcntl",
      "rctime" , "select", "zipimport", "_lsprof",
     "crypt", "signal", "dyngram", "_rawffi", "termios", "zlib",
     "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO",
     "thread", "itertools"]

(copy-pasted from file)

>
> I am hoping in near future , i can help in this project after enough
> understanding .

You're very much welcomed to do so. Come on #pypy on IRC, it's better
then mail sometimes.

>
> Regards,
>
> Phyo.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

Cheers,
fijal

From phyo.arkarlwin at gmail.com  Wed Nov 26 21:47:24 2008
From: phyo.arkarlwin at gmail.com (phyo.arkarlwin at gmail.com)
Date: Thu, 27 Nov 2008 03:17:24 +0630
Subject: [pypy-dev] Some Newbie Questions
In-Reply-To: <693bc9ab0811260344y2df77928h9b69d4005c1fac9b@mail.gmail.com>
References: <bcc23fcc0811260329m46085cc5iebae2be02f4a4919@mail.gmail.com>
	<693bc9ab0811260344y2df77928h9b69d4005c1fac9b@mail.gmail.com>
Message-ID: <bcc23fcc0811261247u306bd5pb0e165198b09af65@mail.gmail.com>

Thank you very much fijal , you have clear me up a lot of confusions.

>That's possible, although far in the future. Look here:
>http://moderator.appspot.com/ and search for pypy.

Very interesting , Do you think it will happen in next release of pypy
, next 2-3 years? As speed  (pypy-c) is already reaching to that of
Cpython .

>Have to use for what? If you want to participate in PyPy project, you
>probably need to know a bit about rpython, otherwise no, you don't.
>That can be gradual and relatively painless though :) (probably better
>han learning C or CPython's C API)

For contributing Pypy and If Guido retired , pypy may be at the level
to be able to continue Python development via pypy  (may be one real
choice).

>The simple answer is: they don't follow RPython. I'm pretty sure about
>this. You need to rewrite them from scratch, at least in parts for
>that. It's not very helpful.

i have some existing project which i would like to turn to C and
compile . All i need to is to follow RPython right ? After that How
can i compile to C ?


Oh one question , what is pypy's JIT for ?

And i will be joining to #pypy now.



On 11/26/08, Maciej Fijalkowski <fijall at gmail.com> wrote:
> On Wed, Nov 26, 2008 at 12:29 PM, Phyo Arkar <phyo.arkarlwin at gmail.com>
> wrote:
>> Hi Pypy Devs!
>>
>> I love python so much and I am so exicted by this ambitious project!
>>
>> What I understand about this Project is to develop a version of python
>> , which is written in Pure Python .  SO now , i do not have C Skills
>> but I can Develop , Tweak , Make changes to python Just in Python , Am
>> I Right? SO In Future , it is possible that Guido himself will be
>> developing python in PyPy ?
>
> That's possible, although far in the future. Look here:
> http://moderator.appspot.com/ and search for pypy.
>
>>
>> Question : I have to use Python Or  RPython?
>
> Have to use for what? If you want to participate in PyPy project, you
> probably need to know a bit about rpython, otherwise no, you don't.
> That can be gradual and relatively painless though :) (probably better
> than learning C or CPython's C API)
>
>>
>> Also  By doing so , We can convert Python to Any langauge of Choice ,
>> which is very interesting.
>
> No, only restricted subset of it, RPython.
>
>>
>> I have build pypy-c with optimization Level 3 and stackless , whcih is
>> done flawlessly. I have seen benchmarks on one of the page and now
>> PyPy is even faster then Normal python 2.5 827 vs 810 ms on Richard
>> Test, it is very interesting .
>>
>> Question : Resulting pypy-c is python 2.4.1 , is there python 2.5
>> Version of pypy-c avaliable? I saw a branch about it before by a GSOC
>> student but now is that already marged ?(my version svn is 4 days ago)
>
> Yes, it's merged into pypy-trunk. Download it from svn from here:
> http://codespeak.net/svn/pypy/trunk/
>
>>
>> If that arleady marged , how can i build it to get pypy-c with python
>> 2.5 support?
>
> Exactly like 2.4, just use trunk.
>
>>
>> Another Questions is , If i follow my Python code in RPython , Can i
>> converty RPythonic Code to C and compile ? SO it is possilbe to
>> convert Existing Python Projects into C (if they follow Rpython)?
>
> The simple answer is: they don't follow RPython. I'm pretty sure about
> this. You need to rewrite them from scratch, at least in parts for
> that. It's not very helpful.
>
>> Another , i compiled pypy-c in stackless , which is faster in term of
>> pypy-c's perfomance? normal or stackless?
>
> normal, although not by much. There is also an experimental option
> --gcrootfinder=asmgcc which makes it even faster, but does not
> cooperate well with ctypes for example.
>
>> Also doing  , what are the extra modules added with --allworkingmodules ?
>
> You should have list when you start compiling. For linux machine, it's:
>
>     ["_socket", "unicodedata", "mmap", "fcntl",
>       "rctime" , "select", "zipimport", "_lsprof",
>      "crypt", "signal", "dyngram", "_rawffi", "termios", "zlib",
>      "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO",
>      "thread", "itertools"]
>
> (copy-pasted from file)
>
>>
>> I am hoping in near future , i can help in this project after enough
>> understanding .
>
> You're very much welcomed to do so. Come on #pypy on IRC, it's better
> then mail sometimes.
>
>>
>> Regards,
>>
>> Phyo.
>> _______________________________________________
>> pypy-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/pypy-dev
>>
>
> Cheers,
> fijal
>

From glavoie at gmail.com  Wed Nov 26 23:20:52 2008
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Wed, 26 Nov 2008 17:20:52 -0500
Subject: [pypy-dev] [patch] Modifications to trunk so it translates under
	FreeBSD 7 64 bits
Message-ID: <d05df8620811261420u391a752alf71426f651498a36@mail.gmail.com>

Hello,
     I made some modifications to trunk so it translates under FreeBSD 7, 64
bits. Here is the diff. I know it's not perfect but at least it builds.

Gabriel

-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081126/56285d5d/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pypy-freebsd7.diff
Type: application/octet-stream
Size: 3690 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20081126/56285d5d/attachment.obj 

From phyo.arkarlwin at gmail.com  Thu Nov 27 18:17:31 2008
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Thu, 27 Nov 2008 23:47:31 +0630
Subject: [pypy-dev] Problem importing csv
Message-ID: <bcc23fcc0811270917t6c730874xc666546ecbe5efdd@mail.gmail.com>

Hello;

I cannot import csv via pypy

Python Python 2.5.2 (pypy 1.0.0 build 60168) on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``this is a perfect channel
topic''
>>>> import csv
Traceback (most recent call last):
  File "<console>", line 1, in <interactive>
  File "/root/python/pypy-trunk/latest/lib-python/2.5.2/csv.py", line 7, in
<module>
    from _csv import Error, __version__, writer, reader, register_dialect, \
ImportError: No module named _csv

is _csv module not developed yet?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081127/bae34a97/attachment.htm 

From glavoie at gmail.com  Fri Nov 28 01:45:00 2008
From: glavoie at gmail.com (Gabriel Lavoie)
Date: Thu, 27 Nov 2008 19:45:00 -0500
Subject: [pypy-dev] [patch] Modifications to trunk so it translates
	under FreeBSD 7 64 bits
In-Reply-To: <d05df8620811261420u391a752alf71426f651498a36@mail.gmail.com>
References: <d05df8620811261420u391a752alf71426f651498a36@mail.gmail.com>
Message-ID: <d05df8620811271645p697e695fqf8aa566f49e5acb5@mail.gmail.com>

Slight modification, the last patch broke build under Linux.

Gabriel

2008/11/26 Gabriel Lavoie <glavoie at gmail.com>

> Hello,
>      I made some modifications to trunk so it translates under FreeBSD 7,
> 64 bits. Here is the diff. I know it's not perfect but at least it builds.
>
> Gabriel
>
> --
> Gabriel Lavoie
> glavoie at gmail.com
>



-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081127/58d4adc0/attachment.htm 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pypy-freebsd7.diff
Url: http://codespeak.net/pipermail/pypy-dev/attachments/20081127/58d4adc0/attachment.diff 

From fijall at gmail.com  Fri Nov 28 02:59:23 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 28 Nov 2008 02:59:23 +0100
Subject: [pypy-dev] corepy - assembler from python
Message-ID: <693bc9ab0811271759sfd08d63w9590e49427ad6d34@mail.gmail.com>

for those of you that don't follow news usually: http://www.corepy.org/
seems worth exploring

Cheers,
fijal

From arigo at tunes.org  Fri Nov 28 14:41:43 2008
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 28 Nov 2008 14:41:43 +0100
Subject: [pypy-dev] Problem importing csv
In-Reply-To: <bcc23fcc0811270917t6c730874xc666546ecbe5efdd@mail.gmail.com>
References: <bcc23fcc0811270917t6c730874xc666546ecbe5efdd@mail.gmail.com>
Message-ID: <20081128134142.GA3190@code0.codespeak.net>

Hi Phyo,

On Thu, Nov 27, 2008 at 11:47:31PM +0630, Phyo Arkar wrote:
> I cannot import csv via pypy

Indeed, this is one of the few standard modules written in C that we
haven't ported yet (_cvs is not in pypy/module/).


A bientot,

Armin.

From arigo at tunes.org  Fri Nov 28 14:49:53 2008
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 28 Nov 2008 14:49:53 +0100
Subject: [pypy-dev] [patch] Modifications to trunk so it translates
	under FreeBSD 7 64 bits
In-Reply-To: <d05df8620811271645p697e695fqf8aa566f49e5acb5@mail.gmail.com>
References: <d05df8620811261420u391a752alf71426f651498a36@mail.gmail.com>
	<d05df8620811271645p697e695fqf8aa566f49e5acb5@mail.gmail.com>
Message-ID: <20081128134952.GA4036@code0.codespeak.net>

Hi Gabriel,

On Thu, Nov 27, 2008 at 07:45:00PM -0500, Gabriel Lavoie wrote:
> >      I made some modifications to trunk so it translates under FreeBSD 7,
> > 64 bits. Here is the diff. I know it's not perfect but at least it builds.

Thanks, applied.


Armin

From fijall at gmail.com  Fri Nov 28 15:52:37 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 28 Nov 2008 15:52:37 +0100
Subject: [pypy-dev] Problem importing csv
In-Reply-To: <20081128134142.GA3190@code0.codespeak.net>
References: <bcc23fcc0811270917t6c730874xc666546ecbe5efdd@mail.gmail.com>
	<20081128134142.GA3190@code0.codespeak.net>
Message-ID: <693bc9ab0811280652g5a8e9bddgcd288e43da73708e@mail.gmail.com>

Or in pypy/lib ...

you can implement it using ctypes, should be super simple :-)

On Fri, Nov 28, 2008 at 2:41 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Phyo,
>
> On Thu, Nov 27, 2008 at 11:47:31PM +0630, Phyo Arkar wrote:
>> I cannot import csv via pypy
>
> Indeed, this is one of the few standard modules written in C that we
> haven't ported yet (_cvs is not in pypy/module/).
>
>
> A bientot,
>
> Armin.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

From cfbolz at gmx.de  Fri Nov 28 16:16:21 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 28 Nov 2008 16:16:21 +0100
Subject: [pypy-dev] Problem importing csv
In-Reply-To: <693bc9ab0811280652g5a8e9bddgcd288e43da73708e@mail.gmail.com>
References: <bcc23fcc0811270917t6c730874xc666546ecbe5efdd@mail.gmail.com>	<20081128134142.GA3190@code0.codespeak.net>
	<693bc9ab0811280652g5a8e9bddgcd288e43da73708e@mail.gmail.com>
Message-ID: <49300B45.5070001@gmx.de>

Maciej Fijalkowski wrote:
> Or in pypy/lib ...
> 
> you can implement it using ctypes, should be super simple :-)

why using ctypes? I don't think cpython's _csv is using any library.
It's just done in C to be faster. So a pure-python implementaiton is fine.

Cheers,

Carl Friedrich

From stephan at transvection.de  Sat Nov 29 11:37:38 2008
From: stephan at transvection.de (Stephan Diehl)
Date: Sat, 29 Nov 2008 11:37:38 +0100
Subject: [pypy-dev] documentation on codespeak
Message-ID: <49311B72.7060403@transvection.de>

Hallo,

I think that the pypy documentation on codespeak is still served from
the 'dist' branch and not the new 'trunk' branch.

cheers, stephan


From hakan at debian.org  Thu Dec  4 10:15:34 2008
From: hakan at debian.org (Hakan Ardo)
Date: Thu, 4 Dec 2008 10:15:34 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
Message-ID: <e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>

Hi,
I've started to play around with the pypy codebase with the intention
to make obj[i] act like obj.__getitem__(i) for rpython objects.  The
approach I tried was to add:

 class __extend__(pairtype(SomeInstance, SomeObject)):
     def getitem((s_array, s_index)):
         s=SomeString()
         s.const="__getitem__"
         p=s_array.getattr(s)
         return p.simple_call(s_index)

and then do something like:

 class __extend__(pairtype(AbstractInstanceRepr, Repr)):
     def rtype_getitem((r_array, r_key), hop):
         hop2=hop.copy()
         ...
         hop2.forced_opname = 'getattr'
         hop2.dispatch()
         hop3=hop.copy()
         ...
         hop3.forced_opname = 'simple_call'
         hop3.dispatch()

But I am having a hard time understanding the rtyper and if this is
the right approach?  Is there anything similar in the code/docs I
could look at to get a better understanding on how to write this?
Would it be a better solution to add an intermediate step between the
annotator and the rtyper that converts
getitem(SomeInstance,SomeObject) into
vx=getattr(SomeInstance,'__getitem__'); simple_call(vx,SomeObject)?

Any suggestions will be appreciated. Thanx!

--
H?kan Ard?

From simon at arrowtheory.com  Fri Dec  5 07:15:36 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Fri, 5 Dec 2008 17:15:36 +1100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
Message-ID: <20081205171536.14185653.simon@arrowtheory.com>

On Thu, 4 Dec 2008 10:15:34 +0100
"Hakan Ardo" <hakan at debian.org> wrote:

> Hi,
> I've started to play around with the pypy codebase with the intention
> to make obj[i] act like obj.__getitem__(i) for rpython objects. 

Woohoo!!

> The
> approach I tried was to add:
> 
>  class __extend__(pairtype(SomeInstance, SomeObject)):
>      def getitem((s_array, s_index)):
>          s=SomeString()
>          s.const="__getitem__"
>          p=s_array.getattr(s)
>          return p.simple_call(s_index)
> 
> and then do something like:
> 
>  class __extend__(pairtype(AbstractInstanceRepr, Repr)):
>      def rtype_getitem((r_array, r_key), hop):
>          hop2=hop.copy()
>          ...
>          hop2.forced_opname = 'getattr'
>          hop2.dispatch()
>          hop3=hop.copy()
>          ...
>          hop3.forced_opname = 'simple_call'
>          hop3.dispatch()

um...

> 
> But I am having a hard time understanding the rtyper and if this is
> the right approach?  Is there anything similar in the code/docs I
> could look at to get a better understanding on how to write this?

I had lots of fun last year writing code in rpython/numpy .
There is plenty of getitem goodness there. Unfortunately it
is probably impenetrable. Um. Perhaps you could revert back to
when the code was sane (but much less functional).

I also had a whack at __str__ for classes but failed horribly.

Keep pestering me and i'm likely to become interested in this stuff again.

Cheering from afar,

Simon.

From arigo at tunes.org  Sun Dec  7 18:04:59 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 7 Dec 2008 18:04:59 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
Message-ID: <20081207170458.GA5599@code0.codespeak.net>

Hi Hakan,

On Thu, Dec 04, 2008 at 10:15:34AM +0100, Hakan Ardo wrote:
> I've started to play around with the pypy codebase with the intention
> to make obj[i] act like obj.__getitem__(i) for rpython objects.  The
> approach I tried was to add:

Calling __getitem__ in this way is not really supported, but here is how
I managed to get it work anyway.  It's only slightly more complicated:

    class __extend__(pairtype(SomeInstance, SomeObject)):
        def getitem((s_array, s_index)):
            # first generate a pseudo call to the helper
            bk = getbookkeeper()
            s_callable = bk.immutablevalue(do_getitem)
            args_s = [s_array, s_index]
            bk.emulate_pbc_call(('instance_getitem', s_array.knowntype),
                                s_callable, args_s)
            # then use your own trick to get the correct result
            s=SomeString()
            s.const="__getitem__"
            p=s_array.getattr(s)
            return p.simple_call(s_index)

    # this is the helper
    def do_getitem(array, key):
        return array.__getitem__(key)
    do_getitem._annspecialcase_ = 'specialize:argtype(0)'
    # ^^^ specialization; not sure I have done it right...    

and then for the code in rclass:

    from pypy.annotation import binaryop
    from pypy.objspace.flow.model import Constant

    class __extend__(pairtype(AbstractInstanceRepr, Repr)):
        def rtype_getitem((r_array, r_key), hop):
            # call the helper do_getitem...
            hop2 = hop.copy()
            bk = r_array.rtyper.annotator.bookkeeper
            v_func = Constant(binaryop.do_getitem)
            s_func = bk.immutablevalue(v_func.value)
            hop2.v_s_insertfirstarg(v_func, s_func)
            hop2.forced_opname = 'simple_call'
            return hop2.dispatch()

I think the _annspecialcase_ should be able to sort out between multiple
unrelated calls to the helper.  The code for the annotator is a bit
bogus, btw, because it emulates a call to the function but also computes
the result explicitly; but I couldn't figure out a better way.


A bientot,

Armin.

From arigo at tunes.org  Sun Dec  7 18:59:32 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 7 Dec 2008 18:59:32 +0100
Subject: [pypy-dev] Sprint dates
Message-ID: <20081207175932.GA11777@code0.codespeak.net>

Hi,

About the February sprint, the proposed dates (mostly the original ones)
are: 7-14th.  After sorting out the Duesseldorf situation, these dates
could be ok too.  Anyone has strong objections?


A bientot,

Armin.

From anto.cuni at gmail.com  Sun Dec  7 23:59:52 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 07 Dec 2008 23:59:52 +0100
Subject: [pypy-dev] Sprint dates
In-Reply-To: <20081207175932.GA11777@code0.codespeak.net>
References: <20081207175932.GA11777@code0.codespeak.net>
Message-ID: <493C5568.5060403@gmail.com>

Armin Rigo wrote:
> Hi,
> 
> About the February sprint, the proposed dates (mostly the original ones)
> are: 7-14th.  After sorting out the Duesseldorf situation, these dates
> could be ok too.  Anyone has strong objections?

the dates should be ok for me, though I don't promise I will come because I'm 
already doing a lot of travels in that period and I'm not sure I feel like 
doing another one... I'll decide later.

ciao,
Anto

From hakan at debian.org  Tue Dec  9 20:14:35 2008
From: hakan at debian.org (Hakan Ardo)
Date: Tue, 9 Dec 2008 20:14:35 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <20081207170458.GA5599@code0.codespeak.net>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
Message-ID: <e7e642fc0812091114sc9c7dexe41a3601b565170e@mail.gmail.com>

Hi,
thanx for all help. For anyone else interested I've placed a few files
on http://hakan.ardoe.net/pypy/ namely:

getitem_support.py    - The suggested implementation
getsetitem_support.py - Generalisation to handle __setitem__ aswell
special_methods.py    - Generalisation to handle several __xxx__ methods
test_getitem.py       - Tests for __getitem__
test_matrix.py        - Tests using __getitem__, __setitem__ and __add__

>    do_getitem._annspecialcase_ = 'specialize:argtype(0)'
>    # ^^^ specialization; not sure I have done it right...
>
> I think the _annspecialcase_ should be able to sort out between multiple
> unrelated calls to the helper.

It seems to be doing it's job. But if I try to apply the same trick to
the __getitem__ method it does not seem to work, e.g. if I try to
compile the code below it only works if I either do a[i] or a[i,j]
calls not a mix of the two.

class arr2d:
    def __init__(self,w,h):
        self.width=w
        self.height=h
        self.data=[i for i in range(w*h)]
    def __getitem__(self,i):
        if isinstance(i,int):
            return self.data[i]
        elif len(i)==2:
            return self.data[i[1]*self.width + i[0]]
        else:
            raise TypeError
    __getitem__._annspecialcase_ = 'specialize:argtype(0)'


-- 
H?kan Ard?

From holger at merlinux.eu  Wed Dec 10 12:41:30 2008
From: holger at merlinux.eu (holger krekel)
Date: Wed, 10 Dec 2008 12:41:30 +0100
Subject: [pypy-dev] Sprint dates
In-Reply-To: <20081207175932.GA11777@code0.codespeak.net>
References: <20081207175932.GA11777@code0.codespeak.net>
Message-ID: <20081210114130.GE2219@trillke.net>

On Sun, Dec 07, 2008 at 18:59 +0100, Armin Rigo wrote:
> Hi,
> 
> About the February sprint, the proposed dates (mostly the original ones)
> are: 7-14th.  After sorting out the Duesseldorf situation, these dates
> could be ok too.  Anyone has strong objections?

i am fine with it. 
holger

From arigo at tunes.org  Thu Dec 11 14:53:47 2008
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 11 Dec 2008 14:53:47 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812091114sc9c7dexe41a3601b565170e@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
	<e7e642fc0812091114sc9c7dexe41a3601b565170e@mail.gmail.com>
Message-ID: <20081211135347.GA6713@code0.codespeak.net>

Hi Hakan,

On Tue, Dec 09, 2008 at 08:14:35PM +0100, Hakan Ardo wrote:
> class arr2d:
>     def __init__(self,w,h):
>         self.width=w
>         self.height=h
>         self.data=[i for i in range(w*h)]
>     def __getitem__(self,i):
>         if isinstance(i,int):
>             return self.data[i]
>         elif len(i)==2:
>             return self.data[i[1]*self.width + i[0]]
>         else:
>             raise TypeError
>     __getitem__._annspecialcase_ = 'specialize:argtype(0)'

That's the wrong annotation.  For this case, it should be
'specialize:argtype(1)' in order to get two versions of __getitem__,
compiled for the two types that can be seen: integers and tuples.


A bientot,

Armin.

From hakan at debian.org  Fri Dec 12 16:49:17 2008
From: hakan at debian.org (Hakan Ardo)
Date: Fri, 12 Dec 2008 16:49:17 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <20081207170458.GA5599@code0.codespeak.net>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
Message-ID: <e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>

On Sun, Dec 7, 2008 at 6:04 PM, Armin Rigo <arigo at tunes.org> wrote:
>
>    class __extend__(pairtype(SomeInstance, SomeObject)):
>        def getitem((s_array, s_index)):
>            # first generate a pseudo call to the helper
>            bk = getbookkeeper()
>            s_callable = bk.immutablevalue(do_getitem)
>            args_s = [s_array, s_index]
>            bk.emulate_pbc_call(('instance_getitem', s_array.knowntype),
>                                s_callable, args_s)
>            # then use your own trick to get the correct result
>            s=SomeString()
>            s.const="__getitem__"
>            p=s_array.getattr(s)
>            return p.simple_call(s_index)
>
> unrelated calls to the helper.  The code for the annotator is a bit
> bogus, btw, because it emulates a call to the function but also computes
> the result explicitly; but I couldn't figure out a better way.

How about instead doing:

    class __extend__(pairtype(SomeInstance, SomeObject)):
        def getitem((s_array, s_index)):
            return call_helper('do_getitem', (s_array, s_index))

    def call_helper(name,s_args):
        bk = getbookkeeper()
        s_callable = bk.immutablevalue(eval(name))
        s_ret=bk.emulate_pbc_call(('instance_'+name,)+tuple([s.knowntype
for s in s_args]),
                                  s_callable, s_args)
        for graph in bk.annotator.pendingblocks.values():
            if graph.name[0:len(name)]==name:
                bk.annotator.notify[graph.returnblock][bk.position_key]=1

        return s_ret;

Is there some way to get hold of the mangled function name of the
created graph? The above code might add too many notifies if there are
several graphs in pendingblocks with a name starting with the name of
the helper. Or is there some better way to get hold of the created
graph object if any was created?

> >     __getitem__._annspecialcase_ = 'specialize:argtype(0)'
>
> That's the wrong annotation.  For this case, it should be
> 'specialize:argtype(1)' in order to get two versions of __getitem__,

Right. Sorry about that.

-- 
H?kan Ard?

From arigo at tunes.org  Fri Dec 12 18:11:47 2008
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 12 Dec 2008 18:11:47 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
	<e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>
Message-ID: <20081212171146.GA9294@code0.codespeak.net>

Hi Hakan,

On Fri, Dec 12, 2008 at 04:49:17PM +0100, Hakan Ardo wrote:
> How about instead doing:
>
>    (...)

Ah, using 'notify' to force a reflow.  Obscure :-/

> Is there some way to get hold of the mangled function name of the
> created graph?

Don't look up graphs by name; the name is only there to get information
about it when printing the graph.  You should probably pass the function
object instead of a string giving the name into your helper.  Then you
can get from the function to the graph(s) with the translator.


Armin

From getxsick at gmail.com  Mon Dec 15 12:53:45 2008
From: getxsick at gmail.com (Bartosz SKOWRON)
Date: Mon, 15 Dec 2008 12:53:45 +0100
Subject: [pypy-dev] Wroclaw (PL) sprint - announcement
Message-ID: <77887e110812150353m63e2f122h9241a392a3aa29e6@mail.gmail.com>

hi,

i would like to announce that I've already reserved a  for upcoming sprint.
dates: 7-14.02.2009 (we have access between 8am - 9 pm but it should
be flexible if needed)
venue: Wroclaw University of Technology, Poland

I've already seen the room and it's pretty nice. Size is similar or a
bit larger than the sprint room from last EuroPython.
The room is dedicated for 30 people so should be enough space there.
Whiteboard and video projector is included.

More informations after x-mass.

cheers!

From hakan at debian.org  Mon Dec 15 20:47:26 2008
From: hakan at debian.org (Hakan Ardo)
Date: Mon, 15 Dec 2008 20:47:26 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <20081212171146.GA9294@code0.codespeak.net>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
	<e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>
	<20081212171146.GA9294@code0.codespeak.net>
Message-ID: <e7e642fc0812151147l663fbb1fs68ae3fcd675805e2@mail.gmail.com>

On Fri, Dec 12, 2008 at 6:11 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi Hakan,
>
> On Fri, Dec 12, 2008 at 04:49:17PM +0100, Hakan Ardo wrote:
>> How about instead doing:
>>
>>    (...)
>
> Ah, using 'notify' to force a reflow.  Obscure :-/

OK, what's the intended use of the notify feature?

The reflow is happening with the previous solution as well. Presumable
because p.simple_call(s_index) gets the getitem opperation registered
as a call site of the __getitem__ method? Maybe a better solution is
to register as a call site of the helper? The following (from
rpython/controllerentry.py) seems to do the trick:

def call_helper(func,s_args):
    bk = getbookkeeper()
    s_callable = bk.immutablevalue(func)
    return bk.emulate_pbc_call(bk.position_key, s_callable, s_args,
                              callback = bk.position_key)


At http://hakan.ardoe.net/pypy/ there is now an implementation of
__add__/__radd__ combination in getsetitem_support.py that calls the
correct method in all cases I could come up with (test_add.py). It
cannot yet handle that the methods return NotImplemented. Would it be
possible to handle that in a similar manner to how None is handled?
That would remove all unneeded tests if the annotator can prove that a
call will always/never return NotImplemented, right?

-- 
H?kan Ard?

From amauryfa at gmail.com  Fri Dec 19 17:50:04 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 19 Dec 2008 17:50:04 +0100
Subject: [pypy-dev] pypy's windows buildbot
In-Reply-To: <693bc9ab0812190751l42d872f8v68926d056567453b@mail.gmail.com>
References: <693bc9ab0812190322h47e55da6q4fcce535dd324df4@mail.gmail.com>
	<e27efe130812190740s22254654t22b1fabf586f38ba@mail.gmail.com>
	<693bc9ab0812190751l42d872f8v68926d056567453b@mail.gmail.com>
Message-ID: <e27efe130812190850t7eaf4c59h97ba3584f8241543@mail.gmail.com>

Maciej Fijalkowski  wrote:
> > pypy-c -S
> >>>> import site
> > [...]
> > RuntimeError: internal error: <RuntimeError object at 0x18f687c>
>
> right, that's a known issue that you cannot do a debug build without
> -O2 at least. I suppose we need to adapt makefiles and such (it sucks,
> doesn't it?).
>
> The reason for that is that we have tons of local vars which are
> temporary and go into registers (but by default, without any
> optimisations, compiler does not remove them from stack space)

I suggest to modify the compiler options in pypy.translator.platform,
on windows first.

- use the /STACK linker option to reserve a stack large enough for
pypy (this affects only .exe, not dlls)
- set the MAX_STACK_SIZE preprocessor symbol used in by
LL_stack_too_big() in src/stack.h,
to a value slightly lower than the one above.

For optimized build, stacksize=512Kb is enough (this is the default
value in src/stack.h)
For debug build stacksize=4Mb works on win32.

I don't know the gcc options very well.
What do you think of adding these two:
        "-DMAX_STACK_SIZE=%d" % (stacksize - 1024)
        "-Wl,--stack,%d" % stacksize

-- 
Amaury Forgeot d'Arc

From p.giarrusso at gmail.com  Sun Dec 21 07:06:33 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sun, 21 Dec 2008 07:06:33 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
Message-ID: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>

Hi all,
after the completion of our student project, I have enough experience
to say something more.
We wrote in C an interpreter for a Python subset and we could make it
much faster than the Python interpreter (50%-60% faster). That was due
to the usage of indirect threading, tagged pointers and unboxed
integers, and a real GC (a copying one, which is enough for the
current benchmarks we have - a generational GC would be more realistic
but we didn't manage to do it).

While we lack some runtime checks, conditional branches are often
almost for free if you have to do an indirect branch soon and likely
flush the pipeline anyway, especially if just one of the possibilities
lays in the fast path. That means that runtime checks for invalid code
have the cost of a normal instruction and not the cost of a
conditional branch (which can be much more expensive).
I obviously do not expect everybody to believe me on this. So, if you
prefer, ignore this "vaporware" and just consider my proposals.

In fact, we are at least 50% faster on anything we can run, but also
on this benchmark, with the usage of unboxed integers and tagged
pointer (we tag pointers with 1 and integers with 0, like V8 does and
SELF did, so you can add integers without untagging them):

def f():
    y = 0
    x = 10000000
    while x:
        x-=1
        y+=3
    return y - 29999873

And since we do overflow checking (by checking EFLAGS in assembly,
even if the code could be improved even more, probably), I don't think
a comparison on this is unfair in any way.

Also, when adding overflow checking we didn't notice any important slowdown.
I think that's enough to say that we are faster also because of design
choices and implementation quality, and the V8 lead developer, Lars
Bak, our professor, had a similar feeling.

On Mon, Nov 17, 2008 at 15:05, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Paolo Giarrusso wrote:

>> specialized bytecode can be significant, I guess, only if the
>> interpreter is really fast (either a threaded one, or a code-copying
>> one). Is the PyPy interpreter threaded?

> sometime ago I tried to measure if/how much we can gain with a threaded
> interpreter. I manually modified the produced C code to make the main loop
> threaded, but we didn't gain anything; I think there are three possible
> reasons:

> 1) in Python a lot of opcodes are quite complex and time-consuming,

That's wrong for a number of reasons - the most common opcodes are
probably loads from the constant pool, and loads and stores to/from
the locals (through LOAD/STORE_FAST). Right now, our hotspot is indeed
the dispatch after the LOAD_FAST opcode.

And those opcodes are quite simple. The hotspot used to be LOAD_CONST,
but to optimize it, I just had to save the pointer to the constant
pool. Instead of accessing current_function->co_consts, I save it into
a local, which can be cached into a register even through function
calls:

        raw_array *current_function_consts = current_function->co_consts;

That made a 4% speedup on our benchmarks. And we didn't write a
microbenchmark for LOAD_CONST. This is the handling code:

                LABEL_BEGIN(LOAD_CONST);
                PUSH(raw_array_skip_header(current_function_consts)[parameter]);
                LABEL_END(LOAD_CONST);

(LABEL_BEGIN/LABEL_END bracket all opcode cases, they expand to labels
for indirect threading and to the dispatch code itself).

After seeing this, you can see why one extra memory load makes a difference.

> so the
> time spent to dispatch to them is a little percentage of the total time
> spent for the execution

That's your problem - threading helps when you spend most of the time
on dispatch, and efficient interpreters get to that point. Threading
helps even Prolog interpreters, like Yap, so it can help even for
Python. Also, according to measurements "The Structure and Performance
of Efficient Interpreters", efficient interpreters tend to execute no
more than 30 instructions per opcode on their RISC machine simulator.
The Python interpreter could still maybe be considered efficient,
since it uses ~30 instructions per opcode, but that's on a CISC
processor (i.e. my laptop), so the numbers are not directly
comparable. I'll maybe measure the ratio, for the interpreters they
benchmark, again on my laptop, to do a  fair comparison.

> 2) due to Python's semantics, it's not possible to just jump from one opcode
> to the next, as we need to do a lot of bookkeeping, like remembering what
> was the last line executed, etc.

No, you don't need that, and not even CPython does it. For exception
handling, just _when an exception is thrown_, you can take the current
opcode index and find the last executed line with a lookup table.
Their source code has a slowpath for tracing support, but that's one
just conditional jump to a slowpath, which is for free in any case if
the predictor guesses it right (and it should).

So, can you make some more examples?

> This means that the trampolines at the end
> of each opcode contains a lot code duplication, leading to a bigger main
> loop, with possibly bad effects on the cache (didn't measure this, though)

If the interpreter loop is able to overflow the Icache, that should be
fought through __builtin_expect first, to give hint for jump
prediction and lay out slowpaths out-of-line.

> 3) it's possible that I did something wrong, so in that case my measurements
> are completely useless :-).  If anyone wants to try again, it cannot hurt.

Well, my plan is first to try, at some point, to implant threading
into the Python interpreter and benchmark the difference - it
shouldn't take long but it has a low priority currently.

Regards
-- 
Paolo Giarrusso

From cfbolz at gmx.de  Sun Dec 21 19:13:04 2008
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Sun, 21 Dec 2008 19:13:04 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
 optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
Message-ID: <494E8730.7050307@gmx.de>

Hi Paolo,

Paolo Giarrusso wrote:
> after the completion of our student project, I have enough experience
> to say something more.
> We wrote in C an interpreter for a Python subset and we could make it
> much faster than the Python interpreter (50%-60% faster). That was due
> to the usage of indirect threading, tagged pointers and unboxed
> integers, and a real GC (a copying one, which is enough for the
> current benchmarks we have - a generational GC would be more realistic
> but we didn't manage to do it).

Interesting, but it sounds like you are comparing apples to oranges.
What sort of subset of Python are you implementing, i.e. what things
don't work? It has been shown time and time again that implementing only
a subset of Python makes it possible to get interesting speedups
compared to CPython. Then, as more and more features are implemented,
the difference gets smaller and smaller. This was true for a number of
Python implementations (e.g. IronPython).

I think to get really meaningful comparisons it would be good to modify
an existing Python implementation and compare that. Yes, I know this can
be a lot of work.

On your actual techniques used I don't have an opinion. I am rather sure
that a copying GC helped performance ? it definitely did for PyPy.
Tagged pointers make PyPy slower, but then, we tag integers with 1, not
with 0. This could be changed, wouldn't even be too much work.

About better implementations of the bytecode dispatch I am unsure. Note
however, that a while ago we did measurements to see how large the
bytecode dispatch overhead is. I don't recall the exact number, but I
think it was below 10%. That means that even if you somehow manage to
reduce that to no overhead at all, you would still only get 10%
performance win.

Cheers,

Carl Friedrich


From p.giarrusso at gmail.com  Sun Dec 21 19:41:54 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sun, 21 Dec 2008 19:41:54 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <494E8730.7050307@gmx.de>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494E8730.7050307@gmx.de>
Message-ID: <a4d0c86c0812211041q6780bae7xc40c63e262fc1ff9@mail.gmail.com>

On Sun, Dec 21, 2008 at 19:13, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:
> Hi Paolo,
>
> Paolo Giarrusso wrote:
>> after the completion of our student project, I have enough experience
>> to say something more.
>> We wrote in C an interpreter for a Python subset and we could make it
>> much faster than the Python interpreter (50%-60% faster). That was due
>> to the usage of indirect threading, tagged pointers and unboxed
>> integers, and a real GC (a copying one, which is enough for the
>> current benchmarks we have - a generational GC would be more realistic
>> but we didn't manage to do it).
>
> Interesting, but it sounds like you are comparing apples to oranges.
> What sort of subset of Python are you implementing, i.e. what things
> don't work? It has been shown time and time again that implementing only
> a subset of Python makes it possible to get interesting speedups
> compared to CPython. Then, as more and more features are implemented,
> the difference gets smaller and smaller. This was true for a number of
> Python implementations (e.g. IronPython).

Yes, I remember - that's part of the motivation of your project, or
something like that. Still, on the example of the arithmetic loop
there couldn't be anything to add I feel; I forgot to note that we
support, beyond overflow checking, operator overloading; i.e. this is
our BINARY_ADD implementation:

                LABEL_BEGIN(BINARY_ADD);
                {
                        tagged_el_t a = POP;
                        tagged_el_t b = POP;
                        if (!(unlikely(is_pointer(a)) ||
unlikely(is_pointer(b)))) {
                                long sum;
                                if (likely(!overflow_checked_add(&sum,
TAGGED_EL_RAW_VALUE(a), TAGGED_EL_RAW_VALUE(b)))) {
                                        PUSH(raw_long_to_tagged_el(sum));
                                } else {
                                        //XXX box to unlimited
precision integers.
                                        PUSH(long_to_tagged_el(INT_MAX));
                                }
                        } else {
                                Object *bp = get_ptr(b);
                                Object *(*add)(Object *, Object *) =
bp->type->add;
                                if (add != NULL) {

PUSH(ptr_to_tagged_el(add(get_ptr(b), get_ptr(a))));
                                } else {
                                        assert(false);
                                }
                        }
                }
                LABEL_END(BINARY_ADD);

As you can see, the slowpaths still need a lot of implementation work
(we don't support __radd__ for instance), but I tried to ensure that
nothing is missing from the fastpath I'm benchmarking. That's why
overflow checking is implemented even if overflow is not handled
properly (and the given example does _not_ overflow).

> I think to get really meaningful comparisons it would be good to modify
> an existing Python implementation and compare that. Yes, I know this can
> be a lot of work.

Yes, I'm aware of that, and that's why I plan to test threading in Python.

> On your actual techniques used I don't have an opinion. I am rather sure
> that a copying GC helped performance ? it definitely did for PyPy.
> Tagged pointers make PyPy slower, but then, we tag integers with 1, not
> with 0. This could be changed, wouldn't even be too much work.

The major problem in changing it is maybe to convert pointers, since
in your representation pointers can be used directly (it depends on
your macros). But since offseting a pointer is directly supported by
the processor, it shouldn't have any major cost (except 1-extra byte
of output, I guess, for each pointer access).

> About better implementations of the bytecode dispatch I am unsure. Note
> however, that a while ago we did measurements to see how large the
> bytecode dispatch overhead is. I don't recall the exact number, but I
> think it was below 10%. That means that even if you somehow manage to
> reduce that to no overhead at all, you would still only get 10%
> performance win.

Sure - but my point, discussed in the mail, and taken from the paper I
mentioned, is that in an efficient interpreter, bytecode dispatch
becomes much more important than the other activities. Don't believe
me, believe that paper (which is quite respected). They report
speedups given by threading up to 2x :-), on efficient interpreters.

Also, Antonio Cuni made just one example of additional overhead for
dispatch: source-line tracking, for exception handling I guess. And as
I explained, as far as I understand that's simply a mistake, even
compared to CPython - so I think there may be other low-hanging fruits
in the interpreter. But I'm still waiting for answers and further
explainations on this point.

Regards
-- 
Paolo Giarrusso

From anto.cuni at gmail.com  Sun Dec 21 23:49:18 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 21 Dec 2008 23:49:18 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
 optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
Message-ID: <494EC7EE.7000205@gmail.com>

Paolo Giarrusso wrote:
> Hi all,

Hi!

> after the completion of our student project, I have enough experience
> to say something more.
> We wrote in C an interpreter for a Python subset and we could make it
> much faster than the Python interpreter (50%-60% faster). That was due
> to the usage of indirect threading, tagged pointers and unboxed
> integers, and a real GC (a copying one, which is enough for the
> current benchmarks we have - a generational GC would be more realistic
> but we didn't manage to do it).

That's interesting but says little about the benefit of threaded 
interpretation itself, as the speedup could be given by the other 
optimizations.  For example, I suspect that for the benchmark you showed most 
of the speedup is because of tagged pointers and the better GC.

Is it possible to make you eval loop non-threaded? Measuring the difference 
with and without indirect threading could give a good hint of how much you 
gain with it.

What kind of bytecode you use? The same as CPython or a custom one? E.g. I 
found that if we want to handle properly the EXTENDED_ARG CPython opcode it is 
necessary to insert a lot of code before jumping to the next opcode.

Moreover, tagging pointer with 1 helps a lot for numerical benchmarks, but it 
is possible that causes a slowdown for other kind of operations.  Do you have 
non-numerical benchmarks? (though I know that it's hard to get fair 
comparison, because the Python object model is complex and it's not easy to 
write a subset of it in a way that is not cheating)

Finally, as Carl said, it would be nice to know which kind of subset it is. 
E.g. does it support exceptions, sys.settrace() and sys._getframe()?


> In fact, we are at least 50% faster on anything we can run, but also
> on this benchmark, with the usage of unboxed integers and tagged
> pointer (we tag pointers with 1 and integers with 0, like V8 does and
> SELF did, so you can add integers without untagging them):
> 
> def f():
>     y = 0
>     x = 10000000
>     while x:
>         x-=1
>         y+=3
>     return y - 29999873
> 
> And since we do overflow checking (by checking EFLAGS in assembly,
> even if the code could be improved even more, probably), I don't think
> a comparison on this is unfair in any way.

is your subset large enough to handle e.g. pystone? What is the result?


>> 1) in Python a lot of opcodes are quite complex and time-consuming,
> 
> That's wrong for a number of reasons - the most common opcodes are
> probably loads from the constant pool, and loads and stores to/from
> the locals (through LOAD/STORE_FAST). Right now, our hotspot is indeed
> the dispatch after the LOAD_FAST opcode.

if you do benchmarks as the one showed above, I agree with you.  If you 
consider real world applications, unfortunately there is more than LOAD_CONST 
and LOAD_FAST: GETATTR, SETATTR, CALL, etc. are all much more time consuming 
than LOAD_{FAST,CONST}

> That's your problem - threading helps when you spend most of the time
> on dispatch, and efficient interpreters get to that point. 

the question is: is it possible for a full python interpreter to be 
"efficient" as you define it?


>> 2) due to Python's semantics, it's not possible to just jump from one opcode
>> to the next, as we need to do a lot of bookkeeping, like remembering what
>> was the last line executed, etc.
> 
> No, you don't need that, and not even CPython does it. For exception
> handling, just _when an exception is thrown_, 
[cut]

Sorry, I made a typo: it is needed to remember the last *bytecode* executed, 
not the last line.  This is necessary to implement properly sys.settrace().

I never mentioned exception handling, that was your (wrong :-)) guess.

> If the interpreter loop is able to overflow the Icache, that should be
> fought through __builtin_expect first, to give hint for jump
> prediction and lay out slowpaths out-of-line.

I think that Armin tried once to use __builtin_expect, but I don't remember 
the outcome.  Armin, what was it?


> Well, my plan is first to try, at some point, to implant threading
> into the Python interpreter and benchmark the difference - it
> shouldn't take long but it has a low priority currently.

That would be cool, tell us when you have done :-).

From anto.cuni at gmail.com  Mon Dec 22 09:41:21 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 22 Dec 2008 09:41:21 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
 optimizations: collecting ideas)
In-Reply-To: <494E8730.7050307@gmx.de>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494E8730.7050307@gmx.de>
Message-ID: <494F52B1.2080802@gmail.com>

Carl Friedrich Bolz wrote:

> About better implementations of the bytecode dispatch I am unsure. Note
> however, that a while ago we did measurements to see how large the
> bytecode dispatch overhead is. I don't recall the exact number, but I
> think it was below 10%. 

I think it's something more.  There is the 'rbench' module that contains 
geninterpreted versions of both richards and pystone; IIRC last time I tried 
they where ~50% faster than they interpreted counterparts, on both pypy-c and 
pypy-cli.

Of course, with geninterp you remove more than just the interpretatiom 
overhead, as e.g. locals are stored on the stack instead that on a frame.

ciao,
Anto

From arigo at tunes.org  Tue Dec 23 12:23:44 2008
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 23 Dec 2008 12:23:44 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812151147l663fbb1fs68ae3fcd675805e2@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
	<e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>
	<20081212171146.GA9294@code0.codespeak.net>
	<e7e642fc0812151147l663fbb1fs68ae3fcd675805e2@mail.gmail.com>
Message-ID: <20081223112344.GA15154@code0.codespeak.net>

Hi Hakan,

On Mon, Dec 15, 2008 at 08:47:26PM +0100, Hakan Ardo wrote:
> cannot yet handle that the methods return NotImplemented. Would it be
> possible to handle that in a similar manner to how None is handled?

Not easily.  The annotation framework of PyPy was never meant to handle
the full Python language, but only a subset reasonable for writing
interpreters.  Anyway, None-or-integer is not supported either, simply
because there is no way to represent that in a single machine word.


A bientot,

Armin.

From p.giarrusso at gmail.com  Tue Dec 23 12:29:01 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Tue, 23 Dec 2008 12:29:01 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <20081223112344.GA15154@code0.codespeak.net>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
	<e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>
	<20081212171146.GA9294@code0.codespeak.net>
	<e7e642fc0812151147l663fbb1fs68ae3fcd675805e2@mail.gmail.com>
	<20081223112344.GA15154@code0.codespeak.net>
Message-ID: <a4d0c86c0812230329qcaf39ddk6cfb686e9b2f73e@mail.gmail.com>

On Tue, Dec 23, 2008 at 12:23, Armin Rigo <arigo at tunes.org> wrote:
> Hi Hakan,
>
> On Mon, Dec 15, 2008 at 08:47:26PM +0100, Hakan Ardo wrote:
>> cannot yet handle that the methods return NotImplemented. Would it be
>> possible to handle that in a similar manner to how None is handled?
>
> Not easily.  The annotation framework of PyPy was never meant to handle
> the full Python language, but only a subset reasonable for writing
> interpreters.  Anyway, None-or-integer is not supported either, simply
> because there is no way to represent that in a single machine word.

There are at least two ways, once you have a singleton (maybe static)
None object around:
- box all integers and use only pointers - the slow one;
- tagged integers/pointers that you already use elsewhere. So integers
of up to 31/63 bits get represented directly, while the other ones are
through pointers.

-- 
Paolo Giarrusso

From p.giarrusso at gmail.com  Tue Dec 23 06:16:32 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Tue, 23 Dec 2008 06:16:32 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <494EC7EE.7000205@gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
Message-ID: <a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>

On Sun, Dec 21, 2008 at 23:49, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Paolo Giarrusso wrote:

Thanks for the interesting mail. It really goes into detail about the
issues I raised, and it does suggest what I needed: stuff needing
support inside the interpreter fast path, if just for unlikely
branches. I started the required coding, but it will take a while
(also because I have little time, given Christmas). But I'm answering
with some preliminary results.

First, I'd answer this question:

>> That's your problem - threading helps when you spend most of the time
>> on dispatch, and efficient interpreters get to that point.

> the question is: is it possible for a full python interpreter to be
> "efficient" as you define it?

Well, my guess is "if Prolog, Scheme and so on can, why can't Python"?

I know this is an interesting question, and I and my colleague, during
the past two months, tried to get an answer. And it wasn't really two
months for two people, I would say that much less time was actually
spent. So for instance we don't have support for modules, exception
handling, and the basic support for object (through Self/V8 style
maps) is not yet complete (but almost).

Then, for a couple of points, it isn't possible. Unboxed integers work
better with the Python 3.x specs - if you have them and you still want
a 32-bit integer, which is boxed, to be an int instead of a long, you
get some problems.

Also, the semantics of Python destructors are too restrictive for a
proper GC, and they can create unreclaimable reference cycles (when
all objects have destructors). In fact, Java, unlike C++, has instead
finalizers - the difference is that you can finalize an object after
finalizing the objects it points to. That's why I prefer not to say
"Python finalizers" like the official docs.
However, for both cases I feel that the semantics are wrong. Even
finalizers are a bad idea - it's interesting that .NET is even more
misguided than Java about them (and that's the opinion of Hans Boehm,
not mine).

So, those semantics fit well only reference counting - and reference
counting, in turn, makes support for multithreading impossible (the
patches for "free threading", without the global interpreter lock,
gave a 2x slowdown I think). Somebody on the python-dev ML realized
that CPython simply gets both things wrong (reference counting and
threading support), but most people went to say "oh, but people
shouldn't use threading anyway, it's too complicated to get right,
they should use multiprocess stuff..." . That was one of the most
(negatively) astonishing things I saw.

Oh, no, the worst was something like "oh, we should allow using more
interpreters in the same process, but we should move lots of statics
to the interpreter context, and it's a lot of work so it's not worth
it". Since I've participated to a community where bigger patches are
committed every week, this looks like plain laziness; it could be
because much less people are paid to work on Python, or maybe because
the general community mood is that one.

In fact, one of the reason I started this project _at all_ so early is
that the Python community clearly shows _ignorance_ about years of VM
research and little attention about VM performance.

== Benchmarks about dispatch overhead ==

And while you don't look like that, the mention of "tracking the last
line executed" seemed quite weird.
And even tracking the last bytecode executed looks weird, even if it
is not maybe. I'm inspecting CPython's Python/ceval.c, and the
overhead for instruction dispatch looks comparable.
The only real problem I'm getting right now is committing the last
bytecode executed to memory. If I store it into a local, I have no
problem at all, if I store it into the interpreter context, it's a
store to memory, so it hurts performance a lot - I'm still wondering
about the right road to go.

Runtime of the while benchmark, with 10x more iterations, user time:

baseline  (now, with tracking of f_lasti inside a local): ~4.6s
add ->f_lasti tracking _into memory_: ~5.6s
disable indirect threading: ~5.6s
both slowdown together: ~7s
Python 2.5.1: ~11.2s

Other benchmarks use fatter opcodes, so the impact is still visible,
but is lower relative to the overall time.
Even if f_lasti must be updated at all the time, could it be stored in
memory just in some occasions? Is it needed during the call to
settrace(), or only to the next opcode after that, i.e. after the
return of the interpreter loop? The second case would be enough.
Storing the variable into memory just for that would be simply
excellent for performance, maybe even in the stock Python interpreter.

Note: all this is on a 64bit box, with 16 registers; we're also faster
on a 32-bit one, and in some cases the difference wrt. Python is
bigger on 32-bit (on the GC stress-test, for instance, due to smaller
pointers to update, and on the recursion test, maybe for the same
reason when using the stack, since the stack element size is 64bit on
64bit machines), while on the others the extra registers (I guess)
give advantage wrt. Python to the 64bit interpreter (obviously, the
comparison is always with Python on the same machine and
architecture). See slide 8 of pres.pdf.

> That's interesting but says little about the benefit of threaded
> interpretation itself, as the speedup could be given by the other
> optimizations.  For example, I suspect that for the benchmark you showed
> most of the speedup is because of tagged pointers and the better GC.

> Is it possible to make you eval loop non-threaded? Measuring the difference
> with and without indirect threading could give a good hint of how much you
> gain with it.

Ok, just done it, the speedup given by indirect threading seems to be
about 18% (see also above). More proper benchmarks are needed though.
And as you say in the other mail, the overhead given by dispatch is
quite more than 50% (maybe). Am I correct in assuming that
"geninterpret"ing _basically_ pastes the opcode handlers together? I
guess with your infrastructure, you can even embed easily the opcode
parameters inside the handlers, it's just a trivial partial evaluation
- I tried to apply code copying of machine code to my interpreter, but
I would have had to keep parameter fetching separate (getting the
output I needed from GCC was not easy - I could make code copying work
just for an early prototype).
About keeping variables into the stack instead that in the frame,
that's even stranger to me, given this argument.

> What kind of bytecode you use? The same as CPython or a custom one? E.g. I
> found that if we want to handle properly the EXTENDED_ARG CPython opcode it
> is necessary to insert a lot of code before jumping to the next opcode.

We handle a subset of the Python's opcodes, but what you say means
that we'll have to handle EXTENDED_ARG properly, because of our
mission. Oh, done, I would say (tests needed though).
I think that "lot of code" can only refer to the requirement of
tracking the last bytecode executed, but I'll have to look at your and
CPython sources for that.

That seems to have an impact of a few percent points (like 4-5% slower
than before), but I have to do real benchmarking for these stuff. I
fear I'll have to calculate confidence interval to give any meaningful
number (I never needed that when comparing us with CPython), because
the impact seems under statistical noise.

However, just having the code laid out the wrong way can make it much
worse. As soon as I wrote it, it almost doubled the runtime of the
arithmetic benchmark (while it had a much lower impact on the other).
It seems it was because GCC decided that HAS_ARG(opcode) had become
unlikely, adding "likely" fixed it back.

In fact, one of the reason we are so fast is that we started with a
small interpreter and tracked down every performance regression as
soon as it happened. We lost half of our performance, at some point,
just because the address of the program_counter local variable
escaped, so that var had to be allocated on the stack and not inside a
register.

== Opcode set ==
We really wanted to have our own one opcode set, but we didn't manage.
In particular, I was refreshed when I read about your LOAD_METHOD
optimization. Also, invoking __init__ is a real mess, Java handles it
much more nicely, and I don't see the point of not using JVM-style
2-step class construction:

new Class
dup <--
<arguments>
invokespecial Class.<init>

Existence of bound methods prevents simply pushing the receiver as the
first argument on the stack; however, I can't see a similar argument
for class construction, even if __init__ can be a bound method.
Generating similar bytecode would avoid the need to rotate the stack.
In Python, one would get something like:

ALLOC_INSTANCE Class // <--- new opcode
DUP_TOP //Copy the value to return
LOAD_METHOD __init__
<arguments>
INVOKE_METHOD
POP_TOP //Remove the None returned by the constructor

However, the dispatch cost of the additional DUP_TOP and POP_TOP might
be a problem. I guess such bytecode will be for sure more efficient to
JIT-compile, but for actual interpretation, benchmarks would be
needed.

> Moreover, tagging pointer with 1 helps a lot for numerical benchmarks, but
> it is possible that causes a slowdown for other kind of operations.  Do you
> have non-numerical benchmarks? (though I know that it's hard to get fair
> comparison, because the Python object model is complex and it's not easy to
> write a subset of it in a way that is not cheating)

I agree with that. The choice was done following the one done by V8
people, and was already tested in the Self system, but I'll answer you
with those tests. We have a couple of recursion test, with a recursive
function and a recursive method (but this one is unfair). And a test
with list access - we are ~50% faster even there.

For the Python object model, we have the opposite problem, because
it's much more code to write and optimize: our INPLACE_ADD is equal to
BINARY_ADD, so given a list l, a loop with "l += [1]" is able to kill
us completely (since for now we create a list each time). Indeed, "l =
l + [1]" is just as slow with our interp, but gets much slower in
Python. Even here we are still faster, but most of the time is spent
adding lists, so we are just 15% faster. I guess we'd get more
advantage if INPLACE_ADD was implemented, and that's not a big deal.

> Finally, as Carl said, it would be nice to know which kind of subset it is.

I just need to find the time to write it down, sorry for the wait. I'm
attaching our final presentation, the report we wrote for the course,
and the _benchmarks_ we have. But basically, the subset we implemented
is _much_ smaller.

> E.g. does it support exceptions, sys.settrace() and sys._getframe()?

Exception handling is not supported, but stack unwind should be
possible without impact on the fastpath.

We didn't know about the other two ones.
I saw code which I guess is for handling settrace in CPython, but
that's on the slowpath, I think I should be able to easily simulate
the impact of that by adding one unlikely conditional jump. Is it
required to track the last bytecode even when tracing is not active,
as asked above?

About _getframe(), I'm quite curious about how it works and how your
support for it works. From the next mail you sent, it seems that you
construct the needed frames during the function calls (and I saw some
documentation about them). Well, in the Smalltalk age, developers
found that constructing the frame object only when reflective access
to the frame is needed is the only reasonable solution, if performance
is important.
_If_ the frame object returned is modifiable, I still think one can
intercept field modifications and modify the original stack to track
the modified object.

In fact, to get the current performance, we don't allocate a locals
dictionary unless needed; and currently it seems that doing it only
when STORE_NAME is invoked gives the correct semantics (and if that
were wrong, computing a NEEDS_LOCAL_DICT flag during compilation would
be trivial). Otherwise, the recursion test would be slower than the
one in Python.

Also, we didn't bother with Unicode, and that's complicated to handle
- even Unicode handling in V8 has problems, and string operations are
slower than in TraceMonkey.

> is your subset large enough to handle e.g. pystone? What is the result?

We didn't have the time to even download pystone.

>>> 1) in Python a lot of opcodes are quite complex and time-consuming,
>>
>> That's wrong for a number of reasons - the most common opcodes are
>> probably loads from the constant pool, and loads and stores to/from
>> the locals (through LOAD/STORE_FAST). Right now, our hotspot is indeed
>> the dispatch after the LOAD_FAST opcode.

> if you do benchmarks as the one showed above, I agree with you.  If you
> consider real world applications, unfortunately there is more than
> LOAD_CONST and LOAD_FAST: GETATTR, SETATTR, CALL, etc. are all much more
> time consuming than LOAD_{FAST,CONST}

Yep, but well, if you want to implement a low-level algorithm in
Python, at some point you'll write such code in an inner loop. But
yes, we've been working also on attribute fetch - we've been
implementing V8 maps just for that (we worked on that for a couple of
days, and I'm fixing the remaining bugs - but it's not really a lot of
code).

For GET/SET_ATTR, I will need to do "inline caching" of lookup results
(even if the term doesn't really apply to an interpreter). Making it
faster than dictionary lookup will be hard however, since our strings
cache their hash code.

>>> 2) due to Python's semantics, it's not possible to just jump from one
>>> opcode
>>> to the next, as we need to do a lot of bookkeeping, like remembering what
>>> was the last line executed, etc.
>>
>> No, you don't need that, and not even CPython does it. For exception
>> handling, just _when an exception is thrown_,

> [cut]

> Sorry, I made a typo: it is needed to remember the last *bytecode* executed,
> not the last line.  This is necessary to implement properly sys.settrace().

> I never mentioned exception handling, that was your (wrong :-)) guess.

Indeed, and I was quite surprised to see something like that. I'm
quite happier now, but see above for comment on its impact.

>> If the interpreter loop is able to overflow the Icache, that should be
>> fought through __builtin_expect first, to give hint for jump
>> prediction and lay out slowpaths out-of-line.

> I think that Armin tried once to use __builtin_expect, but I don't remember
> the outcome.  Armin, what was it?

On suitable benchmarks, for me it's easy to see that the wrong
expectation can give horrible results (not always, but often). And
sometimes GCC does get the wrong result. But once, on a short opcode,
fixing code laid out the wrong way didn't make any important
difference. I guess that to execute one simple opcode one maybe
doesn't manage to fill a 20-stage pipeline before of the next
dispatch, which will flush it. So, if you have 10 or 20 instructions,
it doesn't matter that much - 20 cycles are needed anyway. In that
case, fixing the layout made the same instructions execute faster
probably, but they had to wait more time. Also, I do guess that
dynamic branch prediction could fix the wrong code layout (not
perfectly though - a predicted taken branch costs more than one not
taken).

Note: to give you actual numbers, BINARY_ADD uses 16 assembler
instructions, and then 11 are needed for dispatching, plus other 7 if
the next opcode has a parameter; the slow paths use some more
instructions, in case of EXTENDED_ARG (but they are laid out-of-line).
DUP_TOP is three assembly instructions, POP_TOP is one, and the stack
pointer is kept in RBX.

Note 2: by suitable benchmarks, I don't mean unfair ones. It would be
easy (and unfair) to give hints for the exact code you have in your
benchmark.

>> Well, my plan is first to try, at some point, to implant threading
>> into the Python interpreter and benchmark the difference - it
>> shouldn't take long but it has a low priority currently.

> That would be cool, tell us when you have done :-).

For sure I will let you know, just don't hold your breath on that :-).

Regards
-- 
Paolo Giarrusso
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pres.pdf
Type: application/pdf
Size: 56344 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20081223/c4fcfdf7/attachment-0001.pdf 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: report.pdf.bz2
Type: application/x-bzip2
Size: 105640 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20081223/c4fcfdf7/attachment-0002.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bench.tar.bz2
Type: application/x-bzip2
Size: 2549 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20081223/c4fcfdf7/attachment-0003.bin 

From arigo at tunes.org  Tue Dec 23 15:44:26 2008
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 23 Dec 2008 15:44:26 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <a4d0c86c0812230329qcaf39ddk6cfb686e9b2f73e@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
	<e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>
	<20081212171146.GA9294@code0.codespeak.net>
	<e7e642fc0812151147l663fbb1fs68ae3fcd675805e2@mail.gmail.com>
	<20081223112344.GA15154@code0.codespeak.net>
	<a4d0c86c0812230329qcaf39ddk6cfb686e9b2f73e@mail.gmail.com>
Message-ID: <20081223144426.GA1401@code0.codespeak.net>

Hi Paolo,

On Tue, Dec 23, 2008 at 12:29:01PM +0100, Paolo Giarrusso wrote:
> There are at least two ways, once you have a singleton (maybe static)
> None object around:
> - box all integers and use only pointers - the slow one;
> - tagged integers/pointers that you already use elsewhere. So integers
> of up to 31/63 bits get represented directly, while the other ones are
> through pointers.

Yes, we're using both ways, but for app-level integers, not for regular
RPython-level integers.  That would be a major slow-down.


A bientot,

Armin.

From arigo at tunes.org  Tue Dec 23 16:01:14 2008
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 23 Dec 2008 16:01:14 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
Message-ID: <20081223150113.GA1522@code0.codespeak.net>

Hi Paolo,

Just a quick note as an answer to your long and detailed e-mail (thanks
for it, btw).

On the whole you're making quite some efforts to get Python fast,
starting with a subset of Python and adding feature after feature until
it is more or less complete, while benchmarking at every step. This is
not a new approach: it has been tried before for Python.  Usually, this
kind of project ends up being not used and forgotten, because it's
"only" 80% or 90% compatible but not 99% -- and people care much more,
on average, about 99% compatibility than about 50% performance
improvement.  PyPy on the other hand starts from the path of 99%
compatibility and then tries to improve performance (which started as
10000 times slower... and is now roughly 1.5 or 2 times slower).

Just saying that the approach is completely different...  And I have not
much interest in it -- because you change the language and have to start
again from scratch.  A strong point of PyPy is that you don't have to;
e.g. we have, in addition to the Python interpreter, a JavaScript, a
Smalltalk, etc...


A bientot,

Armin.

From anto.cuni at gmail.com  Tue Dec 23 15:59:37 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 23 Dec 2008 15:59:37 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <a4d0c86c0812230329qcaf39ddk6cfb686e9b2f73e@mail.gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>	<20081207170458.GA5599@code0.codespeak.net>	<e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>	<20081212171146.GA9294@code0.codespeak.net>	<e7e642fc0812151147l663fbb1fs68ae3fcd675805e2@mail.gmail.com>	<20081223112344.GA15154@code0.codespeak.net>
	<a4d0c86c0812230329qcaf39ddk6cfb686e9b2f73e@mail.gmail.com>
Message-ID: <4950FCD9.8050104@gmail.com>

Paolo Giarrusso wrote:

> There are at least two ways, once you have a singleton (maybe static)
> None object around:
> - box all integers and use only pointers - the slow one;
> - tagged integers/pointers that you already use elsewhere. So integers
> of up to 31/63 bits get represented directly, while the other ones are
> through pointers.

I think you are confusing level: here we are talking about RPython, i.e. the 
language which our Python interpreter is implemented in.  Hence, RPython ints 
are really like C ints, and you don't want to manipulate C ints as tagged 
pointer, do you?

ciao,
Anto


From p.giarrusso at gmail.com  Tue Dec 23 17:49:20 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Tue, 23 Dec 2008 17:49:20 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <4950FCD9.8050104@gmail.com>
References: <e7e642fc0812032342m4f36458ere9543b35f538cbe2@mail.gmail.com>
	<e7e642fc0812040115s5cc89daes9e54b1d1ee0280f7@mail.gmail.com>
	<20081207170458.GA5599@code0.codespeak.net>
	<e7e642fc0812120749h494c2d70r344e2beafd6420df@mail.gmail.com>
	<20081212171146.GA9294@code0.codespeak.net>
	<e7e642fc0812151147l663fbb1fs68ae3fcd675805e2@mail.gmail.com>
	<20081223112344.GA15154@code0.codespeak.net>
	<a4d0c86c0812230329qcaf39ddk6cfb686e9b2f73e@mail.gmail.com>
	<4950FCD9.8050104@gmail.com>
Message-ID: <a4d0c86c0812230849n74780a88o46792976eb4c4808@mail.gmail.com>

On Tue, Dec 23, 2008 at 15:59, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Paolo Giarrusso wrote:
>
>> There are at least two ways, once you have a singleton (maybe static)
>> None object around:
>> - box all integers and use only pointers - the slow one;
>> - tagged integers/pointers that you already use elsewhere. So integers
>> of up to 31/63 bits get represented directly, while the other ones are
>> through pointers.
>
> I think you are confusing level: here we are talking about RPython, i.e. the
> language which our Python interpreter is implemented in.  Hence, RPython
> ints are really like C ints, and you don't want to manipulate C ints as
> tagged pointer, do you?

I understood the difference, but writing "there's no way to represent
both of them in a machine word" was a statement that prompted me to
write something - actually, I was thinking to just the return
convention of __add__ and __radd__.
If those method start returning NotImplemented or None, any _sound_
static type analysis won't assign type "int" to them, so it looks (to
me, who ignore the content of RPython, I'm aware of that) that it may
be possible to do this without tagging _all_ integers. And there are
examples of compiled languages with tagged integers (I know at least
of OcaML).
But can you currently live in RPython without anything which could be
a pointer or an integer? Can you have a list like [1, None] in
RPython?

Then I wonder how do you get an omogeneous call interface for all
__add__ methods (i.e. how to force the one returning just integers to
also have type NotImplementedOrInteger). And I also wonder if the
RPython compiler can inline the __add__ call and optimize the tagging
away.

That said, I do not know if what I'm suggesting is implementable in
RPython, or if it would be a good idea. Just my 2 cents, since this
might be what Hakan is looking for.

Regards
-- 
Paolo Giarrusso

From anto.cuni at gmail.com  Wed Dec 24 22:29:33 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Wed, 24 Dec 2008 22:29:33 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
 optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>	
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
Message-ID: <4952A9BD.6060903@gmail.com>

Paolo Giarrusso wrote:

>> the question is: is it possible for a full python interpreter to be
>> "efficient" as you define it?
> 
> Well, my guess is "if Prolog, Scheme and so on can, why can't Python"?

a possible answer is that python is much more complex than prolog; for 
example, in PyPy we also have an rpython implementation of both prolog and 
scheme (though I don't know how much complete is the latter one).

I quickly counted the number of lines for the interpreters, excluding the 
builtin types/functions, and we have 28188 non-empty lines for python, 5376 
for prolog and 1707 for scheme.

I know that the number of lines does not mean anything, but I think it's a 
good hint about the relative complexities of the languages.  I also know that 
being more complex does not necessarily mean that it's impossible to write an 
"efficient" interpreter for it, it's an open question.

Thanks for the interesting email, but unfortunately I don't have time to 
answer right now (xmas is coming :-)), I just drop few quick notes:

> And while you don't look like that, the mention of "tracking the last
> line executed" seemed quite weird.
> And even tracking the last bytecode executed looks weird, even if it
> is not maybe. I'm inspecting CPython's Python/ceval.c, and the
> overhead for instruction dispatch looks comparable.
> The only real problem I'm getting right now is committing the last
> bytecode executed to memory. If I store it into a local, I have no
> problem at all, if I store it into the interpreter context, it's a
> store to memory, so it hurts performance a lot - I'm still wondering
> about the right road to go.

by "tracking the last bytecode executed" I was really referring to the 
equivalent of f_lasti; are you sure you can store it in a local and still 
implement sys.settrace()?

> Ok, just done it, the speedup given by indirect threading seems to be
> about 18% (see also above). More proper benchmarks are needed though.

that's interesting, thanks for having tried. I wonder I should try again with 
indirect threading in pypy soon or later.
Btw, are the sources for your project available somewhere?

> And as you say in the other mail, the overhead given by dispatch is
> quite more than 50% (maybe). 

no, it's less. 50% is the total speedup given by geninterp, which removes 
dispatch overhead but also other things, like storing variables on the stack 
and turning python level flow control into C-level flow control (so e.g. loops 
are expressed as C loops).

> Am I correct in assuming that
> "geninterpret"ing _basically_ pastes the opcode handlers together? I
> guess with your infrastructure, you can even embed easily the opcode
> parameters inside the handlers, it's just a trivial partial evaluation

that's (part of) what our JIT is doing/will do.  But it does much more than 
that, of course.


Merry Christmas to you and all pypyers on the list!

ciao,
Anto

From p.giarrusso at gmail.com  Thu Dec 25 00:42:18 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Thu, 25 Dec 2008 00:42:18 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <20081223150113.GA1522@code0.codespeak.net>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<20081223150113.GA1522@code0.codespeak.net>
Message-ID: <a4d0c86c0812241542r28291965s92ab5c603451d0ab@mail.gmail.com>

Hi Armin,

first, thanks for your answer.

On Tue, Dec 23, 2008 at 16:01, Armin Rigo <arigo at tunes.org> wrote:
> On the whole you're making quite some efforts to get Python fast,
> starting with a subset of Python and adding feature after feature until
> it is more or less complete, while benchmarking at every step. This is
> not a new approach: it has been tried before for Python.  Usually, this
> kind of project ends up being not used and forgotten, because it's
> "only" 80% or 90% compatible but not 99% -- and people care much more,
> on average, about 99% compatibility than about 50% performance
> improvement.  PyPy on the other hand starts from the path of 99%
> compatibility and then tries to improve performance (which started as
> 10000 times slower... and is now roughly 1.5 or 2 times slower).

> Just saying that the approach is completely different...  And I have not
> much interest in it -- because you change the language and have to start
> again from scratch.  A strong point of PyPy is that you don't have to;
> e.g. we have, in addition to the Python interpreter, a JavaScript, a
> Smalltalk, etc...
Never said we're gonna turn this into a full-featured Python
interpreter, and to rewrite all the libraries for it.

So, just a few clarifications:

1) this is a _student project_ which is currently "completed" and has
been handed in, has been written by two students and was our first
interpreter ever (and for one of us, the first really big C project).
I knew that locals are faster than structure fields, but I had
absolutely no idea of why and how much, before starting experimenting
with this.

2) it is intended to be a way to learn how to write it, and a proof of
concept about how Python can be made faster. The first two things I'll
try to optimize are the assignment to ->f_lasti and addition of
indirect threading (even if right now I'd guess an impact around 5%,
if anything, because of refcounting).
If I'll want to try something without refcounting, I'll guess I'd turn
to PyPy, but don't hold your breath for that. The fact that indirect
threading didn't work, that you're 1.5-2x slower than CPython, and
that you store locals in frame objects, they all show that the
abstraction overhead of the interpret is too high. Since you have
different type of frame objects, I guess you might use virtuals to
access them (even though I hope not), or that you have anyhow some
virtuals. And that'd be a problem as well.

3) still, I do believe that working on it was interesting to get
experience about how to optimize an interpreter. And the original idea
was to show that real multithreading (without a global interpreter
lock) cannot be done in Python just because of the big design mistakes
of CPython.

Regards
-- 
Paolo Giarrusso

From p.giarrusso at gmail.com  Fri Dec 26 05:52:06 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Fri, 26 Dec 2008 05:52:06 +0100
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <4952A9BD.6060903@gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<4952A9BD.6060903@gmail.com>
Message-ID: <a4d0c86c0812252052o48bfb765wf483470552332ae3@mail.gmail.com>

Hi!
This time, I'm trying to answer shortly

Is this the geninterp you're talking about?
http://codespeak.net/pypy/dist/pypy/doc/geninterp.html
Is the geninterpreted version RPython code? I'm almost sure, except
for the """NOT_RPYTHON""" doc string in the geninterpreted source
snippet. I guess it's there because the _source_ of it is not RPython
code.

On Wed, Dec 24, 2008 at 22:29, Antonio Cuni <anto.cuni at gmail.com> wrote:
> Paolo Giarrusso wrote:

> I quickly counted the number of lines for the interpreters, excluding the
> builtin types/functions, and we have 28188 non-empty lines for python, 5376
> for prolog and 1707 for scheme.

> I know that the number of lines does not mean anything, but I think it's a
> good hint about the relative complexities of the languages.

Also about the amount of Python-specific optimizations you did :-).

>  I also know
> that being more complex does not necessarily mean that it's impossible to
> write an "efficient" interpreter for it, it's an open question.

The 90-10 rule should apply anyway, but overhead for obscure features
might be a problem.
Well, reflection on the call stack can have a big runtime impact, but
that's also present in Smalltalk as far as I know and that can be
handled as well.
Anyway, if Python developers are not able to implement efficient
multithreading in the interpreter because of the excessive performance
impact and they don't decide to drop refcounting, saying "there's
space for optimizations" looks like a safe bet; the source of the idea
is what I've been taught in the course, but I'm also noticing this by
myself.

> Thanks for the interesting email, but unfortunately I don't have time to
> answer right now (xmas is coming :-)), I just drop few quick notes:

Yeah, for me as well, plus I'm in the last month of my Erasmus study time :-)

>> Ok, just done it, the speedup given by indirect threading seems to be
>> about 18% (see also above). More proper benchmarks are needed though.

> that's interesting, thanks for having tried. I wonder I should try again
> with indirect threading in pypy soon or later.

I would do it together with OProfile benchmarking of indirect branches
and of their mispredictions (see the presentation for the OProfile
commands on the Core 2 processor).

> Btw, are the sources for your project available somewhere?

They'll be sooner or later. There are a few bugs I should fix, and a
few other interesting things to do.
But if you are interested in trying to do benchmarking even if it's a
student project, it's not feature complete, and it's likely buggy, I
might publish it earlier.

>> And as you say in the other mail, the overhead given by dispatch is
>> quite more than 50% (maybe).

> no, it's less.

Yeah, sorry, I remember you wrote geninterp also does other stuff.

> 50% is the total speedup given by geninterp, which removes
> dispatch overhead but also other things, like storing variables on the stack

I wonder why that's not done by your stock interpreter - the CPython
frame object has a pointer to a real stack frame; I'm not sure, but I
guess this can increase stack locality since a 32/64-byte cacheline is
much bigger than a typical stack frame and has space for the operand
stack (and needless to say we store locals on the stack, like JVMs
do).

The right benchmark for this, I guess, would be oprofiling cache
misses on a recursion test like factorial or Fibonacci.

> and turning python level flow control into C-level flow control (so e.g.
> loops are expressed as C loops).

Looking at the geninterpreted code, it's amazing that the RPython
translator can do this. Can it also already specialize the interpreter
for each of the object spaces and save the virtual calls?

== About F_LASTI ==
> by "tracking the last bytecode executed" I was really referring to the
> equivalent of f_lasti; are you sure you can store it in a local and still
> implement sys.settrace()?

Not really, I didn't even start studying its proper semantics, but now
I know it's worth a try and some additional complexity, at least in an
interpreter with GC. If one write to memory has such a horrible
impact, I'm frightened by the possible impact of refcounting; on the
other side, I wouldn't be surprised if saving the f_lasti write had no
impact on CPython.

My current approach would be that if I can identify code paths where
no code can even look at it (and I guess that most simple opcodes are
such paths), I can copy f_lasti to a global structure only in the
other paths; if f_lasti is just passed to the code tracing routine and
it's called only from the interpreter loop, I could even turn it into
a parameter to that routine (it may be faster with a register calling
convention, but anyway IMHO one gets code which is easier to follow).

Actually, I even wonder if I can just set it when tracing is active,
but since that'd be trivial to do, I guess that when you return from a
call to settrace, you discover (without being able to anticipate it)
that now you need to discover the previous opcode, that's why it's not
already fixed. Still, a local can do even for that, or more
complicated algorithms can do as well (basically, the predecessor is
always known at compile time except for jumps, so only jump opcodes
really need to compute f_lasti).

Regards
-- 
Paolo Giarrusso

From hakan at debian.org  Fri Dec 26 11:31:53 2008
From: hakan at debian.org (Hakan Ardo)
Date: Fri, 26 Dec 2008 11:31:53 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
References: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
Message-ID: <e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>

On Tue, Dec 23, 2008 at 12:23 PM, Armin Rigo <arigo at tunes.org> wrote:
> On Mon, Dec 15, 2008 at 08:47:26PM +0100, Hakan Ardo wrote:
>> cannot yet handle that the methods return NotImplemented. Would it be
>> possible to handle that in a similar manner to how None is handled?
>
> Not easily.  The annotation framework of PyPy was never meant to handle
> the full Python language, but only a subset reasonable for writing

OK, that makes sens.

On Tue, Dec 23, 2008 at 5:49 PM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
> And I also wonder if the
> RPython compiler can inline the __add__ call and optimize the tagging
> away.
>
> That said, I do not know if what I'm suggesting is implementable in
> RPython, or if it would be a good idea. Just my 2 cents, since this
> might be what Hakan is looking for.

Yes, in most cases of interest, the annotator can determined whether
the __add__ method always/never return NotImplemented and thus the
test will be removed. If we'll not  be able to handle the most general
case where the choise between __add__ and __radd__ has to be done at
runtime, that's fine.

If I use None instead of NotImplemented I get the behaviour  I want.
The following code does for example compile:

  class mystr(object):
       def __init__(self,s):
           self.s=str(s)
       def __str__(self):
           return self.s
       def __add__(self,other):
           return mystr(self.s+str(other))
       def __radd__(self,other):
           return mystr(str(other)+self.s)
       __add__._annspecialcase_ = 'specialize:argtype(1)'
       __radd__._annspecialcase_ = 'specialize:argtype(1)'

   class pair(object):
       def __init__(self,a,b):
           self.a=a
           self.b=b
       def __str__(self):
           return "(%d,%d)"%(self.a,self.b)
       def __add__(self,other):
           if isinstance(other,pair):
               return pair(self.a+other.a, self.b+other.b)
           else:
               return None
       __add__._annspecialcase_ = 'specialize:argtype(1)'
       __radd__=__add__

   def dotst_notimplemented():
       a=mystr('a')
       b=pair(1,2)
       return (str(b+a), str(a+b))

If I use the following helper to call the method:

   def do_add_radd(lop,rop):
       if isinstance(rop,lop.__class__) and not isinstance(lop,rop.__class__):
           r=rop.__radd__(lop)
           if r is None:
               return lop.__add__(rop)
           return r
       else:
           r=lop.__add__(rop)
           if r is None:
               return rop.__radd__(lop)
           return r

But if I replace None with NotImplemented, it does not compile
anymore. So, can we get the annotater to treat NotImplemented in a
similar manner as it treats None?

--
H?kan Ard?

From jbaker at zyasoft.com  Fri Dec 26 16:19:36 2008
From: jbaker at zyasoft.com (Jim Baker)
Date: Fri, 26 Dec 2008 08:19:36 -0700
Subject: [pypy-dev] Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <a4d0c86c0812252052o48bfb765wf483470552332ae3@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<4952A9BD.6060903@gmail.com>
	<a4d0c86c0812252052o48bfb765wf483470552332ae3@mail.gmail.com>
Message-ID: <d03bb4010812260719l7dde884dm6215137cb9d337d6@mail.gmail.com>

Interesting discussion. Just a note:

in Jython, f_lasti is only used to manage exit/entry points, specifically
for coroutines/generators, so this is not at the level of bytecode
granularity. We also set f_lineno, at the level of Python code of course.
HotSpot apparently optimizes this access nicely in any event. (There are
other problems with supporting call frames, but this is not one of them it
seems.)

Java also offers a debugging interface, which in conjunction with a C++
agent, allows for more fine-grained access to these internals, potentially
with lower overhead. This is something Tobias Ivarsson has been exploring.

- Jim

On Thu, Dec 25, 2008 at 9:52 PM, Paolo Giarrusso <p.giarrusso at gmail.com>wrote:

> Hi!
> This time, I'm trying to answer shortly
>
> Is this the geninterp you're talking about?
> http://codespeak.net/pypy/dist/pypy/doc/geninterp.html
> Is the geninterpreted version RPython code? I'm almost sure, except
> for the """NOT_RPYTHON""" doc string in the geninterpreted source
> snippet. I guess it's there because the _source_ of it is not RPython
> code.
>
> On Wed, Dec 24, 2008 at 22:29, Antonio Cuni <anto.cuni at gmail.com> wrote:
> > Paolo Giarrusso wrote:
>
> > I quickly counted the number of lines for the interpreters, excluding the
> > builtin types/functions, and we have 28188 non-empty lines for python,
> 5376
> > for prolog and 1707 for scheme.
>
> > I know that the number of lines does not mean anything, but I think it's
> a
> > good hint about the relative complexities of the languages.
>
> Also about the amount of Python-specific optimizations you did :-).
>
> >  I also know
> > that being more complex does not necessarily mean that it's impossible to
> > write an "efficient" interpreter for it, it's an open question.
>
> The 90-10 rule should apply anyway, but overhead for obscure features
> might be a problem.
> Well, reflection on the call stack can have a big runtime impact, but
> that's also present in Smalltalk as far as I know and that can be
> handled as well.
> Anyway, if Python developers are not able to implement efficient
> multithreading in the interpreter because of the excessive performance
> impact and they don't decide to drop refcounting, saying "there's
> space for optimizations" looks like a safe bet; the source of the idea
> is what I've been taught in the course, but I'm also noticing this by
> myself.
>
> > Thanks for the interesting email, but unfortunately I don't have time to
> > answer right now (xmas is coming :-)), I just drop few quick notes:
>
> Yeah, for me as well, plus I'm in the last month of my Erasmus study time
> :-)
>
> >> Ok, just done it, the speedup given by indirect threading seems to be
> >> about 18% (see also above). More proper benchmarks are needed though.
>
> > that's interesting, thanks for having tried. I wonder I should try again
> > with indirect threading in pypy soon or later.
>
> I would do it together with OProfile benchmarking of indirect branches
> and of their mispredictions (see the presentation for the OProfile
> commands on the Core 2 processor).
>
> > Btw, are the sources for your project available somewhere?
>
> They'll be sooner or later. There are a few bugs I should fix, and a
> few other interesting things to do.
> But if you are interested in trying to do benchmarking even if it's a
> student project, it's not feature complete, and it's likely buggy, I
> might publish it earlier.
>
> >> And as you say in the other mail, the overhead given by dispatch is
> >> quite more than 50% (maybe).
>
> > no, it's less.
>
> Yeah, sorry, I remember you wrote geninterp also does other stuff.
>
> > 50% is the total speedup given by geninterp, which removes
> > dispatch overhead but also other things, like storing variables on the
> stack
>
> I wonder why that's not done by your stock interpreter - the CPython
> frame object has a pointer to a real stack frame; I'm not sure, but I
> guess this can increase stack locality since a 32/64-byte cacheline is
> much bigger than a typical stack frame and has space for the operand
> stack (and needless to say we store locals on the stack, like JVMs
> do).
>
> The right benchmark for this, I guess, would be oprofiling cache
> misses on a recursion test like factorial or Fibonacci.
>
> > and turning python level flow control into C-level flow control (so e.g.
> > loops are expressed as C loops).
>
> Looking at the geninterpreted code, it's amazing that the RPython
> translator can do this. Can it also already specialize the interpreter
> for each of the object spaces and save the virtual calls?
>
> == About F_LASTI ==
> > by "tracking the last bytecode executed" I was really referring to the
> > equivalent of f_lasti; are you sure you can store it in a local and still
> > implement sys.settrace()?
>
> Not really, I didn't even start studying its proper semantics, but now
> I know it's worth a try and some additional complexity, at least in an
> interpreter with GC. If one write to memory has such a horrible
> impact, I'm frightened by the possible impact of refcounting; on the
> other side, I wouldn't be surprised if saving the f_lasti write had no
> impact on CPython.
>
> My current approach would be that if I can identify code paths where
> no code can even look at it (and I guess that most simple opcodes are
> such paths), I can copy f_lasti to a global structure only in the
> other paths; if f_lasti is just passed to the code tracing routine and
> it's called only from the interpreter loop, I could even turn it into
> a parameter to that routine (it may be faster with a register calling
> convention, but anyway IMHO one gets code which is easier to follow).
>
> Actually, I even wonder if I can just set it when tracing is active,
> but since that'd be trivial to do, I guess that when you return from a
> call to settrace, you discover (without being able to anticipate it)
> that now you need to discover the previous opcode, that's why it's not
> already fixed. Still, a local can do even for that, or more
> complicated algorithms can do as well (basically, the predecessor is
> always known at compile time except for jumps, so only jump opcodes
> really need to compute f_lasti).
>
> Regards
> --
> Paolo Giarrusso
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>



-- 
Jim Baker
jbaker at zyasoft.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081226/eee7b158/attachment.htm 

From arigo at tunes.org  Sat Dec 27 19:45:13 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 27 Dec 2008 19:45:13 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>
References: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
	<e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>
Message-ID: <20081227184512.GA20899@code0.codespeak.net>

Hi,

On Fri, Dec 26, 2008 at 11:31:53AM +0100, Hakan Ardo wrote:
> But if I replace None with NotImplemented, it does not compile
> anymore. So, can we get the annotater to treat NotImplemented in a
> similar manner as it treats None?

You can try, but it's messy.  It's not a problem for the annotator but
for the later RTyper.  None is implemented as a NULL pointer by the
RTyper; I don't know how you would distinguish between None-as-a-NULL
and NotImplemented.  You could try to go for something like ((TYPE*) 1),
but this doesn't work on top of ootype, where you really have only one
NULL value.


A bientot,

Armin.

From p.giarrusso at gmail.com  Sat Dec 27 19:56:51 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Sat, 27 Dec 2008 19:56:51 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <20081227184512.GA20899@code0.codespeak.net>
References: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
	<e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>
	<20081227184512.GA20899@code0.codespeak.net>
Message-ID: <a4d0c86c0812271056u7037395auac8851d51360341d@mail.gmail.com>

On 27/12/2008, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
>
>  On Fri, Dec 26, 2008 at 11:31:53AM +0100, Hakan Ardo wrote:
>  > But if I replace None with NotImplemented, it does not compile
>  > anymore. So, can we get the annotater to treat NotImplemented in a
>  > similar manner as it treats None?

> You can try, but it's messy.  It's not a problem for the annotator but
>  for the later RTyper.  None is implemented as a NULL pointer by the
>  RTyper; I don't know how you would distinguish between None-as-a-NULL
>  and NotImplemented.  You could try to go for something like ((TYPE*) 1),
>  but this doesn't work on top of ootype, where you really have only one
>  NULL value.

I don't know ootype, but why not having a NotImplemented singleton
type, and returning a pointer to its instance? Python has a None
singleton type as well, so it makes sense.

Regards
-- 
Paolo Giarrusso

From arigo at tunes.org  Sat Dec 27 21:51:40 2008
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 27 Dec 2008 21:51:40 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <a4d0c86c0812271056u7037395auac8851d51360341d@mail.gmail.com>
References: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
	<e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>
	<20081227184512.GA20899@code0.codespeak.net>
	<a4d0c86c0812271056u7037395auac8851d51360341d@mail.gmail.com>
Message-ID: <20081227205140.GA31712@code0.codespeak.net>

Hi Paolo,

On Sat, Dec 27, 2008 at 07:56:51PM +0100, Paolo Giarrusso wrote:
> I don't know ootype, but why not having a NotImplemented singleton
> type, and returning a pointer to its instance? Python has a None
> singleton type as well, so it makes sense.

See our doc: http://codespeak.net/pypy/dist/pypy/doc/translation.html
and http://codespeak.net/pypy/dist/pypy/doc/rtyper.html .


A bientot,

Armin.

From jbaker at zyasoft.com  Sat Dec 27 22:08:46 2008
From: jbaker at zyasoft.com (Jim Baker)
Date: Sat, 27 Dec 2008 14:08:46 -0700
Subject: [pypy-dev] Fwd: Threaded interpretation (was: Re: compiler
	optimizations: collecting ideas)
In-Reply-To: <d03bb4010812271308g4daef87md0d4c6e5f0b11e98@mail.gmail.com>
References: <a4d0c86c0812202206l6d05b74fq8b9dbe6b175a2f66@mail.gmail.com>
	<494EC7EE.7000205@gmail.com>
	<a4d0c86c0812222116o458f9fcchd0543c871c6c1af0@mail.gmail.com>
	<4952A9BD.6060903@gmail.com>
	<a4d0c86c0812252052o48bfb765wf483470552332ae3@mail.gmail.com>
	<d03bb4010812260719l7dde884dm6215137cb9d337d6@mail.gmail.com>
	<a4d0c86c0812271158r39865d55xff0336bac3ed6871@mail.gmail.com>
	<d03bb4010812271308g4daef87md0d4c6e5f0b11e98@mail.gmail.com>
Message-ID: <d03bb4010812271308n174a1153mb2445e715f83c60d@mail.gmail.com>

forgot to reply-all

---------- Forwarded message ----------
From: Jim Baker <jbaker at zyasoft.com>
Date: Sat, Dec 27, 2008 at 2:08 PM
Subject: Re: [pypy-dev] Threaded interpretation (was: Re: compiler
optimizations: collecting ideas)
To: Paolo Giarrusso <p.giarrusso at gmail.com>


I'm only speaking of Jython 2.5, since that's what we're working on, but I
believe it was the same for 2.2. (I personally regard 2.5 as more robust, we
certainly test it more extensively, although external interfaces may still
have some change before our forthcoming release.)

We have limited support for tracing in terms of its events interface. It
seems usable enough, although we don't produce quite the same traces.
Fidelity to things we can't readily and efficiently support is not a goal
for Jython. This is especially when they don't bear on running interesting
applications. So we don't support ref counting, a GIL, and certain other
internal details. We have found that when code does rely on these details
that we have been able to push changes into the appropriate projects. Often
the same changes are needed for alternative implementations like PyPy; we
saw this with Django support. However, we do support frame introspection,
the standard Python obj model (including classic classes), and even the
rather mixed up unicode/str model.

Having said that, we do plan to support a Python bytecode (PBC) VM running
in Jython in a future release (possibly 2.5.1). At that point, we may
support last_i at the level of the PBC instruction, just like CPython.
Reasons for supporting PBC include scenarios where we can't dynamically
generate and then load Java bytecode (unsigned applets, for example, or
Android), greenlets (which really needs last_i, although it's possible to
get a subset of greenlet functionality w/o it), and various components like
Jinga2, a templating engines, that directly emit PBC. It's also rather cool
to do so I think.

- Jim


On Sat, Dec 27, 2008 at 12:58 PM, Paolo Giarrusso <p.giarrusso at gmail.com>wrote:

> On 26/12/2008, Jim Baker <jbaker at zyasoft.com> wrote:
> > Interesting discussion. Just a note:
> >
> > in Jython, f_lasti is only used to manage exit/entry points, specifically
> > for coroutines/generators, so this is not at the level of bytecode
> > granularity.
>
> But is it used to support sys._settrace() ? Also, its past usage
> (before Python 2.3) for generators is mentioned in CPython source code
> comments, and last stable Jython release is 2.2.1. What happens in
> 2.5-beta could be more interesting.
>
> Here's the comment from <src_tree>/Python/ceval.c:
>
>           f->f_lasti now refers to the index of the last instruction
>           executed.  You might think this was obvious from the name, but
>           this wasn't always true before 2.3!  PyFrame_New now sets
>           f->f_lasti to -1 (i.e. the index *before* the first instruction)
>           and YIELD_VALUE doesn't fiddle with f_lasti any more.  So this
>           does work.  Promise.
>
> > We also set f_lineno, at the level of Python code of course.
>
> Hmm, CPython is already able to do this only when needed (i.e. when
> calling trace functions), "as of Python 2.3":
>
>    /* As of 2.3 f_lineno is only valid when tracing is active (i.e. when
>       f_trace is set) -- at other times use PyCode_Addr2Line instead. */
>    int f_lineno;               /* Current line number */
>
> > HotSpot apparently optimizes this access nicely in any event. (There are
> > other problems with supporting call frames, but this is not one of them
> it
> > seems.)
>
> Do you mean you inspected the code generated by the Hotspot native
> compiler?
> Hmm, it can't save a write to memory if they are fields of the frame
> object, unless it can prove that a pointer to the object does not
> escape the local function through escape analysis (which has been
> added in Java 1.6 but is said _somewhere_ to notice too few cases).
>
> > Java also offers a debugging interface, which in conjunction with a C++
> > agent, allows for more fine-grained access to these internals,
> potentially
> > with lower overhead. This is something Tobias Ivarsson has been
> exploring.
>
> That sounds interesting, even if strange (and not applicable to
> CPython nor PyPy) - do you want to offer an alternate debug interface
> or to implement settrace through this?
>
> > - Jim
>
> > On Thu, Dec 25, 2008 at 9:52 PM, Paolo Giarrusso <p.giarrusso at gmail.com>
> > wrote:
> > > Hi!
> > > This time, I'm trying to answer shortly
> > >
> > > Is this the geninterp you're talking about?
> > >
> > http://codespeak.net/pypy/dist/pypy/doc/geninterp.html
> > > Is the geninterpreted version RPython code? I'm almost sure, except
> > > for the """NOT_RPYTHON""" doc string in the geninterpreted source
> > > snippet. I guess it's there because the _source_ of it is not RPython
> > > code.
> > >
> > >
> > > On Wed, Dec 24, 2008 at 22:29, Antonio Cuni <anto.cuni at gmail.com>
> wrote:
> > > > Paolo Giarrusso wrote:
> > >
> > >
> > > > I quickly counted the number of lines for the interpreters, excluding
> > the
> > > > builtin types/functions, and we have 28188 non-empty lines for
> python,
> > 5376
> > > > for prolog and 1707 for scheme.
> > >
> > > > I know that the number of lines does not mean anything, but I think
> it's
> > a
> > > > good hint about the relative complexities of the languages.
> > >
> > > Also about the amount of Python-specific optimizations you did :-).
> > >
> > >
> > > >  I also know
> > > > that being more complex does not necessarily mean that it's
> impossible
> > to
> > > > write an "efficient" interpreter for it, it's an open question.
> > >
> > > The 90-10 rule should apply anyway, but overhead for obscure features
> > > might be a problem.
> > > Well, reflection on the call stack can have a big runtime impact, but
> > > that's also present in Smalltalk as far as I know and that can be
> > > handled as well.
> > > Anyway, if Python developers are not able to implement efficient
> > > multithreading in the interpreter because of the excessive performance
> > > impact and they don't decide to drop refcounting, saying "there's
> > > space for optimizations" looks like a safe bet; the source of the idea
> > > is what I've been taught in the course, but I'm also noticing this by
> > > myself.
> > >
> > >
> > > > Thanks for the interesting email, but unfortunately I don't have time
> to
> > > > answer right now (xmas is coming :-)), I just drop few quick notes:
> > >
> > > Yeah, for me as well, plus I'm in the last month of my Erasmus study
> time
> > :-)
> > >
> > >
> > > >> Ok, just done it, the speedup given by indirect threading seems to
> be
> > > >> about 18% (see also above). More proper benchmarks are needed
> though.
> > >
> > > > that's interesting, thanks for having tried. I wonder I should try
> again
> > > > with indirect threading in pypy soon or later.
> > >
> > > I would do it together with OProfile benchmarking of indirect branches
> > > and of their mispredictions (see the presentation for the OProfile
> > > commands on the Core 2 processor).
> > >
> > >
> > > > Btw, are the sources for your project available somewhere?
> > >
> > > They'll be sooner or later. There are a few bugs I should fix, and a
> > > few other interesting things to do.
> > > But if you are interested in trying to do benchmarking even if it's a
> > > student project, it's not feature complete, and it's likely buggy, I
> > > might publish it earlier.
> > >
> > >
> > > >> And as you say in the other mail, the overhead given by dispatch is
> > > >> quite more than 50% (maybe).
> > >
> > > > no, it's less.
> > >
> > > Yeah, sorry, I remember you wrote geninterp also does other stuff.
> > >
> > >
> > > > 50% is the total speedup given by geninterp, which removes
> > > > dispatch overhead but also other things, like storing variables on
> the
> > stack
> > >
> > > I wonder why that's not done by your stock interpreter - the CPython
> > > frame object has a pointer to a real stack frame; I'm not sure, but I
> > > guess this can increase stack locality since a 32/64-byte cacheline is
> > > much bigger than a typical stack frame and has space for the operand
> > > stack (and needless to say we store locals on the stack, like JVMs
> > > do).
> > >
> > > The right benchmark for this, I guess, would be oprofiling cache
> > > misses on a recursion test like factorial or Fibonacci.
> > >
> > >
> > > > and turning python level flow control into C-level flow control (so
> e.g.
> > > > loops are expressed as C loops).
> > >
> > > Looking at the geninterpreted code, it's amazing that the RPython
> > > translator can do this. Can it also already specialize the interpreter
> > > for each of the object spaces and save the virtual calls?
> > >
> > > == About F_LASTI ==
> > >
> > > > by "tracking the last bytecode executed" I was really referring to
> the
> > > > equivalent of f_lasti; are you sure you can store it in a local and
> > still
> > > > implement sys.settrace()?
> > >
> > > Not really, I didn't even start studying its proper semantics, but now
> > > I know it's worth a try and some additional complexity, at least in an
> > > interpreter with GC. If one write to memory has such a horrible
> > > impact, I'm frightened by the possible impact of refcounting; on the
> > > other side, I wouldn't be surprised if saving the f_lasti write had no
> > > impact on CPython.
> > >
> > > My current approach would be that if I can identify code paths where
> > > no code can even look at it (and I guess that most simple opcodes are
> > > such paths), I can copy f_lasti to a global structure only in the
> > > other paths; if f_lasti is just passed to the code tracing routine and
> > > it's called only from the interpreter loop, I could even turn it into
> > > a parameter to that routine (it may be faster with a register calling
> > > convention, but anyway IMHO one gets code which is easier to follow).
> > >
> > > Actually, I even wonder if I can just set it when tracing is active,
> > > but since that'd be trivial to do, I guess that when you return from a
> > > call to settrace, you discover (without being able to anticipate it)
> > > that now you need to discover the previous opcode, that's why it's not
> > > already fixed. Still, a local can do even for that, or more
> > > complicated algorithms can do as well (basically, the predecessor is
> > > always known at compile time except for jumps, so only jump opcodes
> > > really need to compute f_lasti).
> > >
> > > Regards
> > > --
> > > Paolo Giarrusso
> > >
> > >
> > >
> > > _______________________________________________
> > > pypy-dev at codespeak.net
> > >
> > http://codespeak.net/mailman/listinfo/pypy-dev
> > >
>
>
> --
> Paolo Giarrusso
>



-- 
Jim Baker
jbaker at zyasoft.com



-- 
Jim Baker
jbaker at zyasoft.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20081227/cad009f1/attachment-0001.htm 

From hakan at debian.org  Mon Dec 29 16:15:09 2008
From: hakan at debian.org (Hakan Ardo)
Date: Mon, 29 Dec 2008 16:15:09 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <20081227184512.GA20899@code0.codespeak.net>
References: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
	<e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>
	<20081227184512.GA20899@code0.codespeak.net>
Message-ID: <e7e642fc0812290715j352dc105qa9544b5f3dfc2845@mail.gmail.com>

On Sat, Dec 27, 2008 at 7:45 PM, Armin Rigo <arigo at tunes.org> wrote:
>
> You can try, but it's messy.  It's not a problem for the annotator but
> for the later RTyper.  None is implemented as a NULL pointer by the

Atatched is a small patch for the annotator that makes it treat None
and NotImplemented alike. This is all that is needed for most cases as
all NotImplemented are typically removed by the optimisations
performed by the annotator.

At http://hakan.ardoe.net/pypy/ I have placed special_methods.py that
adds support for the methods listed below together with 42 test
including the relevant parts of test_augassign.py and test_binop.py
from the cpython source (somewhat modified to work). The methods
currently supported are:

__str__, __repr__, __len__, __getitem__, __setitem__, __add__,
__mul__, __sub__, __div__, __floordiv__, __mod__, __xor__, __rshift__,
__lshift__, __radd__, __rmul__, __rsub__, __rdiv__, __rfloordiv__,
__rmod__, __rxor__, __rrshift__, __rlshift__, __iadd__, __imul__,
__isub__, __idiv__, __ifloordiv__, __imod__, __ixor__, __irshift__,
__ilshift__


With this implementation, the opperation str(o) calls o.__str__(), but
the opperation "%s"%o does not. I don't know why.

-- 
H?kan Ard?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: annotator.patch
Type: text/x-patch
Size: 1370 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20081229/47b38521/attachment.bin 

From p.giarrusso at gmail.com  Mon Dec 29 20:55:52 2008
From: p.giarrusso at gmail.com (Paolo Giarrusso)
Date: Mon, 29 Dec 2008 20:55:52 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <e7e642fc0812290715j352dc105qa9544b5f3dfc2845@mail.gmail.com>
References: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
	<e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>
	<20081227184512.GA20899@code0.codespeak.net>
	<e7e642fc0812290715j352dc105qa9544b5f3dfc2845@mail.gmail.com>
Message-ID: <a4d0c86c0812291155v370fd3b1se15e0726215f057a@mail.gmail.com>

On Mon, Dec 29, 2008 at 16:15, Hakan Ardo <hakan at debian.org> wrote:
> On Sat, Dec 27, 2008 at 7:45 PM, Armin Rigo <arigo at tunes.org> wrote:
>>
>> You can try, but it's messy.  It's not a problem for the annotator but
>> for the later RTyper.  None is implemented as a NULL pointer by the
>
> Atatched is a small patch for the annotator that makes it treat None
> and NotImplemented alike. This is all that is needed for most cases as
> all NotImplemented are typically removed by the optimisations
> performed by the annotator.
That can be made to work, but if such a method returns None you get
completely different semantics (trying again with something else) from
CPython (which will maybe return a failure, or return None for the
result of such an operation), so you have to restrict the allowed
semantics in RPython to the "most cases" you are referring to.

Basically, I'd propose that in RPython with your patch, those methods
(__add__ etc.) cannot return None (I can't think of a possible use
case for an addition returning None), while I guess NotImplemented can
only be returned by them, and not by any other function or method (do
the current sources ever use NotImplemented?). There must be no
function which can choose to return None or NotImplemented.

Regards
-- 
Paolo Giarrusso

From hakan at debian.org  Mon Dec 29 23:50:37 2008
From: hakan at debian.org (Hakan Ardo)
Date: Mon, 29 Dec 2008 23:50:37 +0100
Subject: [pypy-dev] Support for __getitem__ in rpython?
In-Reply-To: <a4d0c86c0812291155v370fd3b1se15e0726215f057a@mail.gmail.com>
References: <mailman.69602.1230285687.4712.pypy-dev@codespeak.net>
	<e7e642fc0812260231g5e82bb25ub9cf347f8cabc2fe@mail.gmail.com>
	<20081227184512.GA20899@code0.codespeak.net>
	<e7e642fc0812290715j352dc105qa9544b5f3dfc2845@mail.gmail.com>
	<a4d0c86c0812291155v370fd3b1se15e0726215f057a@mail.gmail.com>
Message-ID: <e7e642fc0812291450g1777d4f6n6a390fe4a9582f4a@mail.gmail.com>

On Mon, Dec 29, 2008 at 8:55 PM, Paolo Giarrusso <p.giarrusso at gmail.com> wrote:
>>
>> Atatched is a small patch for the annotator that makes it treat None
>> and NotImplemented alike. This is all that is needed for most cases as
>> all NotImplemented are typically removed by the optimisations
>> performed by the annotator.
> That can be made to work, but if such a method returns None you get
> completely different semantics (trying again with something else) from
> CPython (which will maybe return a failure, or return None for the
> result of such an operation), so you have to restrict the allowed

No, the patch do distinguish between None and NotImplemented. What I
mean is that NotImplemented is treated in a similar manner as to how
None is treated. The following crazy construction do compile and
generate the same result as in cpython ('OK', 'String', 'None',
'None'):

class mystr:
    def __init__(self,s):
        self.s=s
    def __str__(self):
        return self.s
    def __add__(self,other):
        if isinstance(other,mystr):
            return NotImplemented
        s=self.s+other
        if s=='None':
            return None
        else:
            return s
    __add__._annspecialcase_ = 'specialize:argtype(1)'
    def __radd__(self,other):
        return str(other)+self.s
    __radd__._annspecialcase_ = 'specialize:argtype(1)'

def dotst_nonestr():
    s1=mystr('No')+'ne'
    if s1 is None: s1='OK'
    s2=mystr('Str')+'ing'
    s3=mystr('No')+mystr('ne')
    s4='No'+mystr('ne')
    return (s1,s2,s3,s4)

-- 
H?kan Ard?

From hakan at debian.org  Tue Dec 30 12:15:18 2008
From: hakan at debian.org (Hakan Ardo)
Date: Tue, 30 Dec 2008 12:15:18 +0100
Subject: [pypy-dev] String or None segmentation fault
Message-ID: <e7e642fc0812300315o5a2a1f41v1241fbc1efd61fd8@mail.gmail.com>

Hi,
the following code compiles into something that segfaults. Attached is
a patch trying to fix that.

        def nstr(i):
            if i==0: return None
            return str(i)
        def fn(i):
            return (str(nstr(0)), str(nstr(i)))


-- 
H?kan Ard?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nonesegv.patch
Type: text/x-patch
Size: 1611 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20081230/fe2dab3a/attachment.bin 

From anto.cuni at gmail.com  Tue Dec 30 23:37:52 2008
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Tue, 30 Dec 2008 23:37:52 +0100
Subject: [pypy-dev] [Fwd: [Fwd: Re: Threaded interpretation]]
Message-ID: <495AA2C0.9050105@gmail.com>

Hi,
Antoine Pitrou told me that his mail got rejected by the mailing list, so I'm 
forwarding it.

-------- Message transf?r? --------
De: Antoine Pitrou <solipsis at pitrou.net>
?: pypy-dev at codespeak.net
Sujet: Re: Threaded interpretation
Date: Fri, 26 Dec 2008 21:16:36 +0000 (UTC)

Hi people,

By reading this thread I had the idea to write a threaded interpretation patch
for py3k. Speedup on pybench and pystone is 15-20%.
http://bugs.python.org/issue4753

Regards

Antoine.