From rxe at ukshells.co.uk  Sat Jul  1 01:24:16 2006
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Sat, 1 Jul 2006 00:24:16 +0100 (BST)
Subject: [pypy-dev] patch for __getattribute__()
Message-ID: <Pine.LNX.4.62.0607010016280.2028@echo.ukshells.co.uk>


Hi!

The following patch initiated by a chat with Runar Petursson about 
descriptors - increases pypy-c/pypy-llvm by 25% on richards.py.

I couldnt be bothered creating a branch - but if someone wants to sanity 
check it and fix the method repr and check it in, you are more than 
welcome! :-)

Cheers,
Richard

-------------- next part --------------
Index: objspace/descroperation.py
===================================================================
--- objspace/descroperation.py	(revision 29539)
+++ objspace/descroperation.py	(working copy)
@@ -1,7 +1,7 @@
 import operator
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import ObjSpace
-from pypy.interpreter.function import Function, Method
+from pypy.interpreter.function import Function, Method, descr_function_get
 from pypy.interpreter.argument import Arguments
 from pypy.interpreter.typedef import default_identity_hash
 from pypy.tool.sourcetools import compile2, func_with_new_name
@@ -19,6 +19,10 @@
         name = space.str_w(w_name)
         w_descr = space.lookup(w_obj, name)
         if w_descr is not None:
+            # XXX test performance
+            descr = space.interpclass_w(w_descr)
+            if type(descr) is Function:
+                return descr_function_get(space, w_descr, w_obj)
             if space.is_data_descr(w_descr):
                 return space.get(w_descr, w_obj)
         w_value = w_obj.getdictvalue(space, w_name)
Index: objspace/cpy/objspace.py
===================================================================
--- objspace/cpy/objspace.py	(revision 29539)
+++ objspace/cpy/objspace.py	(working copy)
@@ -62,8 +62,8 @@
             return PyString_FromStringAndSize(x, len(x))
         if isinstance(x, float): 
             return PyFloat_FromDouble(x)
-        if isinstance(x, r_uint):
-            return PyLong_FromUnsignedLong(x)
+        #if isinstance(x, r_uint):
+        #    return PyLong_FromUnsignedLong(x)
         # if we arrive here during RTyping, then the problem is *not* the %r
         # in the format string, but it's that someone is calling space.wrap()
         # on a strange object.
Index: interpreter/function.py
===================================================================
--- interpreter/function.py	(revision 29539)
+++ interpreter/function.py	(working copy)
@@ -385,6 +385,16 @@
             w_class = space.type(self.w_instance)
         else:
             w_class = self.w_class
+        if w_class is None:
+            if self.w_instance is None:
+                s = "<unbound method %s>" % (name)
+                return space.wrap(s)
+            else:
+                objrepr = space.str_w(space.repr(self.w_instance))
+                info = 'bound method %s of %s' % (name, objrepr)
+                # info = "method %s of %s object" % (name, typename)
+                return self.w_instance.getrepr(self.space, info)
+
         typename = w_class.getname(self.space, '?')
         if self.w_instance is None:
             s = "<unbound method %s.%s>" % (typename, name)

From pedronis at strakt.com  Sat Jul  1 10:03:25 2006
From: pedronis at strakt.com (Samuele Pedroni)
Date: Sat, 01 Jul 2006 10:03:25 +0200
Subject: [pypy-dev] patch for __getattribute__()
In-Reply-To: <Pine.LNX.4.62.0607010016280.2028@echo.ukshells.co.uk>
References: <Pine.LNX.4.62.0607010016280.2028@echo.ukshells.co.uk>
Message-ID: <44A62C4D.8090300@strakt.com>

Richard Emslie wrote:
> 
> Hi!
> 
> The following patch initiated by a chat with Runar Petursson about 
> descriptors - increases pypy-c/pypy-llvm by 25% on richards.py.

well, it breaks this:

Python 2.4.1 (#2, Mar 31 2005, 00:05:10)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> class C(object):
...    def m(self):
...       return "class"
...
 >>> c=C()
 >>> c.m = lambda: "instance"
 >>>
 >>> c.m()
'instance'
 >>>

vs. patched PyPy

PyPy 0.9.0 in StdObjSpace on top of Python 2.4.1 (startuptime: 4.97 secs)
 >>>> class C(object):
....    def m(self):
....        return "class"
....
 >>>> c=C()
 >>>> c.m = lambda: "instance"
 >>>> c.m()
'class'

functions are not data descriptors, this feature makes increasing the 
performance of method lookup more involved that one would like but
Python semantics expect this kind of overriding to work.

I have added a test about this.


> 
> I couldnt be bothered creating a branch - but if someone wants to sanity 
> check it and fix the method repr and check it in, you are more than 
> welcome! :-)
> 
> Cheers,
> Richard
> 
> 
> ------------------------------------------------------------------------
> 
> Index: objspace/descroperation.py
> ===================================================================
> --- objspace/descroperation.py	(revision 29539)
> +++ objspace/descroperation.py	(working copy)
> @@ -1,7 +1,7 @@
>  import operator
>  from pypy.interpreter.error import OperationError
>  from pypy.interpreter.baseobjspace import ObjSpace
> -from pypy.interpreter.function import Function, Method
> +from pypy.interpreter.function import Function, Method, descr_function_get
>  from pypy.interpreter.argument import Arguments
>  from pypy.interpreter.typedef import default_identity_hash
>  from pypy.tool.sourcetools import compile2, func_with_new_name
> @@ -19,6 +19,10 @@
>          name = space.str_w(w_name)
>          w_descr = space.lookup(w_obj, name)
>          if w_descr is not None:
> +            # XXX test performance
> +            descr = space.interpclass_w(w_descr)
> +            if type(descr) is Function:
> +                return descr_function_get(space, w_descr, w_obj)
>              if space.is_data_descr(w_descr):
>                  return space.get(w_descr, w_obj)
>          w_value = w_obj.getdictvalue(space, w_name)
> Index: objspace/cpy/objspace.py
> ===================================================================
> --- objspace/cpy/objspace.py	(revision 29539)
> +++ objspace/cpy/objspace.py	(working copy)
> @@ -62,8 +62,8 @@
>              return PyString_FromStringAndSize(x, len(x))
>          if isinstance(x, float): 
>              return PyFloat_FromDouble(x)
> -        if isinstance(x, r_uint):
> -            return PyLong_FromUnsignedLong(x)
> +        #if isinstance(x, r_uint):
> +        #    return PyLong_FromUnsignedLong(x)
>          # if we arrive here during RTyping, then the problem is *not* the %r
>          # in the format string, but it's that someone is calling space.wrap()
>          # on a strange object.
> Index: interpreter/function.py
> ===================================================================
> --- interpreter/function.py	(revision 29539)
> +++ interpreter/function.py	(working copy)
> @@ -385,6 +385,16 @@
>              w_class = space.type(self.w_instance)
>          else:
>              w_class = self.w_class
> +        if w_class is None:
> +            if self.w_instance is None:
> +                s = "<unbound method %s>" % (name)
> +                return space.wrap(s)
> +            else:
> +                objrepr = space.str_w(space.repr(self.w_instance))
> +                info = 'bound method %s of %s' % (name, objrepr)
> +                # info = "method %s of %s object" % (name, typename)
> +                return self.w_instance.getrepr(self.space, info)
> +
>          typename = w_class.getname(self.space, '?')
>          if self.w_instance is None:
>              s = "<unbound method %s.%s>" % (typename, name)
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From arigo at tunes.org  Sat Jul  1 14:38:03 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 1 Jul 2006 14:38:03 +0200
Subject: [pypy-dev] [pypy-svn] r29534
	-	pypy/dist/pypy/rpython/ootypesystem/test
In-Reply-To: <44A5735A.6090908@gmail.com>
References: <20060630140821.989F310081@code0.codespeak.net>
	<44A56302.2090808@strakt.com> <44A5735A.6090908@gmail.com>
Message-ID: <20060701123803.GA18685@code0.codespeak.net>

Hi Antonio,

On Fri, Jun 30, 2006 at 08:54:18PM +0200, Antonio Cuni wrote:
> Probably it's safer to store the name in some dictionary.

It's even required, yes: no mutating of the low-level type objects
please :-)  The case of Instance is a bit special in this respect.


A bientot,

Armin

From arigo at tunes.org  Sat Jul  1 21:51:53 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 1 Jul 2006 21:51:53 +0200
Subject: [pypy-dev] [dls2006@hpi.uni-potsdam.de: DLS 2006 -- Author
	Notifications Delayed]
Message-ID: <20060701195153.GA22357@code0.codespeak.net>

Hi all,

Looks like we won't know for another week:


----- Forwarded message from dls2006 at hpi.uni-potsdam.de -----

Date: Sat, 01 Jul 2006 21:16:12 +0200
From: dls2006 at hpi.uni-potsdam.de
To: arigo at tunes.org
Subject: DLS 2006 -- Author Notifications Delayed
Reply-To: dls2006 at hpi.uni-potsdam.de

Dear Authors --

Due to unforeseen circumstances author notifications are delayed, expected to be ready mid of next week.

My apologies for the inconvenience,
Robert

----- End forwarded message -----


Armin

From seojiwon at gmail.com  Tue Jul  4 17:55:40 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Tue, 4 Jul 2006 08:55:40 -0700
Subject: [pypy-dev] question on using ctypes
Message-ID: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>

Hi,

I added a module that uses ctypes under pypy/module/ but I'm getting
ImportError while running translate.py for targetpypystandalone.py

The error message is like,

..... cut ...
[translation:ERROR]     w_value = loader(space)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/interpreter/mixedmodule.py", line 135,
in ifileloader
[translation:ERROR]     d[name] = __import__(name, None, None, [name])
[translation:ERROR]  ImportError': No module named tcc
[translation:ERROR]     .. v0 = getattr(self_0, ('w_dict'))
[translation:ERROR] Processing block:
[translation:ERROR]  block at -1 is a <class
'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in
(pypy.interpreter.mixedmodule:33)MixedModule.MixedModule.getdictvalue
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v1 = getattr(space_0, ('finditem'))
[translation:ERROR]        v0 = getattr(self_0, ('w_dict'))
[translation:ERROR]        v2 = simple_call(v1, v0, w_name_0)
[translation:ERROR]        v3 = getattr(self_0, ('lazy'))
[translation:ERROR]        v4 = is_true(v3)
[translation:ERROR]  --end--

and the part I added is pypy/module/tcc/__init__.py and pypy/module/tcc/tcc.py

### __init__.py

# Package initialisation
from pypy.interpreter.mixedmodule import MixedModule

class Module(MixedModule):
    appleveldefs = {}
    interpleveldefs = {
        'print0':   'tcc.tcc_print0'
    }

### tcc.py
from pypy.interpreter.baseobjspace import ObjSpace
from ctypes import *

libtcc = CDLL('/path/to/lib.so')
print0 = libtcc.TCC_Print
print0.argtypes = [c_char_p]
print0.restype = c_int

def tcc_print0(space, msg):
    print0(msg)
tcc_print0.unwrap_spec = [ObjSpace, str]

What have I done wrong?

Thaniks,

-Jiwon

From amauryfa at gmail.com  Tue Jul  4 18:26:09 2006
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 4 Jul 2006 18:26:09 +0200
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
Message-ID: <e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>

2006/7/4, Jiwon Seo <seojiwon at gmail.com>:
> Hi,
>
> I added a module that uses ctypes under pypy/module/ but I'm getting
> ImportError while running translate.py for targetpypystandalone.py
>
> The error message is like,
>
> ..... cut ...
> [translation:ERROR]     w_value = loader(space)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/interpreter/mixedmodule.py", line 135,
> in ifileloader
> [translation:ERROR]     d[name] = __import__(name, None, None, [name])
> [translation:ERROR]  ImportError': No module named tcc
> [translation:ERROR]     .. v0 = getattr(self_0, ('w_dict'))
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at -1 is a <class
> 'pypy.objspace.flow.flowcontext.SpamBlock'>
> [translation:ERROR]  in
> (pypy.interpreter.mixedmodule:33)MixedModule.MixedModule.getdictvalue
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v1 = getattr(space_0, ('finditem'))
> [translation:ERROR]        v0 = getattr(self_0, ('w_dict'))
> [translation:ERROR]        v2 = simple_call(v1, v0, w_name_0)
> [translation:ERROR]        v3 = getattr(self_0, ('lazy'))
> [translation:ERROR]        v4 = is_true(v3)
> [translation:ERROR]  --end--
>
> and the part I added is pypy/module/tcc/__init__.py and pypy/module/tcc/tcc.py

Hello,

Don't know if it helps, but I have already seen this problem:
sometimes an ImportError can mask another ImportError.
The code in mixedmodule.py looks like this:
    try:
        d[name] = __import__(pkgroot+'.'+name, None, None, [name])
    except ImportError:
        d[name] = __import__(name, None, None, [name])

The first call is supposed to be the good one, but if it fails with an
ImportError, you get the error from the second call.
Is the ctypes module correctly installed?

-- 
Amaury Forgeot d'Arc

From seojiwon at gmail.com  Wed Jul  5 01:43:58 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Tue, 4 Jul 2006 16:43:58 -0700
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
Message-ID: <b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>

> Hello,
>
> Don't know if it helps, but I have already seen this problem:
> sometimes an ImportError can mask another ImportError.
> The code in mixedmodule.py looks like this:
>    try:
>        d[name] = __import__(pkgroot+'.'+name, None, None, [name])
>    except ImportError:
>        d[name] = __import__(name, None, None, [name])
>
> The first call is supposed to be the good one, but if it fails with an
> ImportError, you get the error from the second call.
> Is the ctypes module correctly installed?
>

Yeah, I think so. (I installed in my home directory, and I think I
properly set up PYTHONSTARTUP and did sys.path.append... in the
PYTHONSTARTUP file.
And, 'p name' in the pdb tells me it's tcc modue. So, I think ctypes
might not be related to this. (but I could be wrong)

Any other suggestions?

-Jiwon

From seojiwon at gmail.com  Wed Jul  5 09:25:12 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Wed, 5 Jul 2006 00:25:12 -0700
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
Message-ID: <b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>

On further inspection, it is actually ctypes problem. It seems like
setting sys.path in PYTHONSTARTUP file isn't enough. I guess I should
just install it in system-wide python site-package directory.

Thanks.

On 7/4/06, Jiwon Seo <seojiwon at gmail.com> wrote:
> > Hello,
> >
> > Don't know if it helps, but I have already seen this problem:
> > sometimes an ImportError can mask another ImportError.
> > The code in mixedmodule.py looks like this:
> >    try:
> >        d[name] = __import__(pkgroot+'.'+name, None, None, [name])
> >    except ImportError:
> >        d[name] = __import__(name, None, None, [name])
> >
> > The first call is supposed to be the good one, but if it fails with an
> > ImportError, you get the error from the second call.
> > Is the ctypes module correctly installed?
> >
>
> Yeah, I think so. (I installed in my home directory, and I think I
> properly set up PYTHONSTARTUP and did sys.path.append... in the
> PYTHONSTARTUP file.
> And, 'p name' in the pdb tells me it's tcc modue. So, I think ctypes
> might not be related to this. (but I could be wrong)
>
> Any other suggestions?
>
> -Jiwon
>

From seojiwon at gmail.com  Wed Jul  5 10:16:27 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Wed, 5 Jul 2006 01:16:27 -0700
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
	<b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
Message-ID: <b008462b0607050116l2fe0ac51m10c611b2d61bbccb@mail.gmail.com>

After properly installing ctypes, translate.py now gives this error.

##################################################################

[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "./translate.py", line 332, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 506, in
proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/tool/taskengine.py", line
108, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 195, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 208, in
task_annotate
[translation:ERROR]     s = annotator.build_types(self.entry_point,
self.inputtypes)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 100, in
build_types
[translation:ERROR]     return self.build_graph_types(flowgraph,
inputcells, complete_now=complete_now)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 143, in
build_graph_types
[translation:ERROR]     self.complete()
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 188, in
complete
[translation:ERROR]     self.processblock(graph, block)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 434, in
processblock
[translation:ERROR]     self.flowin(graph, block)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 490, in
flowin
[translation:ERROR]     self.consider_op(block.operations[i])
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 661, in
consider_op
[translation:ERROR]     argcells = [self.binding(a) for a in op.args]
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 238, in
binding
[translation:ERROR]     return self.bookkeeper.immutableconstant(arg)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 297, in
immutableconstant
[translation:ERROR]     return self.immutablevalue(const.value)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 393, in
immutablevalue
[translation:ERROR]     result = SomePBC([self.getdesc(x)])
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 524, in
getdesc
[translation:ERROR]     return self.descs[pyobj]
[translation:ERROR]  TypeError: unhashable type
[translation:ERROR] Processing block:
[translation:ERROR]  block at -1 is a <class
'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in (pypy.module.tcc.tcc:9)tcc_print0
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v107 = simple_call((_FuncPtr TCC_Print), msg_0)
[translation:ERROR]  --end--

############# end of error message ##################################

pypy/module/tcc/__init__.py is like following
# Package initialisation
from pypy.interpreter.mixedmodule import MixedModule

class Module(MixedModule):
    appleveldefs = {}
    interpleveldefs = {
        'print0':   'tcc.tcc_print0'
    }

and pypy/module/tcc/tcc.py is like following

from pypy.interpreter.baseobjspace import ObjSpace
from ctypes import *

libtcc = CDLL('/u/jiwon/proj/less-bus/libLESS.so')
myprint0 = libtcc.TCC_Print
myprint0.argtypes = [c_char_p]
myprint0.restype = c_int

def tcc_print0(space, msg):
    return myprint0(msg)
tcc_print0.unwrap_spec = [ObjSpace, str]


What am I missing here?

-Jiwon

From amauryfa at gmail.com  Wed Jul  5 11:47:00 2006
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 5 Jul 2006 11:47:00 +0200
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <e27efe130607050245y5d268500jd8a47db4593bebdd@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
	<b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
	<b008462b0607050116l2fe0ac51m10c611b2d61bbccb@mail.gmail.com>
	<e27efe130607050245y5d268500jd8a47db4593bebdd@mail.gmail.com>
Message-ID: <e27efe130607050247m9dc6015g545b405cbfde6746@mail.gmail.com>

Hello,

2006/7/5, Jiwon Seo <seojiwon at gmail.com>:
> After properly installing ctypes, translate.py now gives this error.
>
> ##################################################################
>
> [translation:ERROR] Error:
> [translation:ERROR]  Traceback (most recent call last):
> [translation:ERROR]    File "./translate.py", line 332, in main
> [translation:ERROR]     drv.proceed(goals)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 506, in
> proceed
> [translation:ERROR]     return self._execute(goals, task_skip =
> self._maybe_skip())
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/tool/taskengine.py", line
> 108, in _execute
> [translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 195, in _do
> [translation:ERROR]     res = func()
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 208, in
> task_annotate
> [translation:ERROR]     s = annotator.build_types(self.entry_point,
> self.inputtypes)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 100, in
> build_types
> [translation:ERROR]     return self.build_graph_types(flowgraph,
> inputcells, complete_now=complete_now)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 143, in
> build_graph_types
> [translation:ERROR]     self.complete()
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 188, in
> complete
> [translation:ERROR]     self.processblock(graph, block)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 434, in
> processblock
> [translation:ERROR]     self.flowin(graph, block)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 490, in
> flowin
> [translation:ERROR]     self.consider_op(block.operations[i])
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 661, in
> consider_op
> [translation:ERROR]     argcells = [self.binding(a) for a in op.args]
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 238, in
> binding
> [translation:ERROR]     return self.bookkeeper.immutableconstant(arg)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 297, in
> immutableconstant
> [translation:ERROR]     return self.immutablevalue(const.value)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 393, in
> immutablevalue
> [translation:ERROR]     result = SomePBC([self.getdesc(x)])
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 524, in
> getdesc
> [translation:ERROR]     return self.descs[pyobj]
> [translation:ERROR]  TypeError: unhashable type
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at -1 is a <class
> 'pypy.objspace.flow.flowcontext.SpamBlock'>
> [translation:ERROR]  in (pypy.module.tcc.tcc:9)tcc_print0
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v107 = simple_call((_FuncPtr TCC_Print), msg_0)
> [translation:ERROR]  --end--
>
> ############# end of error message ##################################
>
> pypy/module/tcc/__init__.py is like following
> # Package initialisation
> from pypy.interpreter.mixedmodule import MixedModule
>
> class Module(MixedModule):
>     appleveldefs = {}
>     interpleveldefs = {
>         'print0':   'tcc.tcc_print0'
>     }
>
> and pypy/module/tcc/tcc.py is like following
>
> from pypy.interpreter.baseobjspace import ObjSpace
> from ctypes import *
>
> libtcc = CDLL('/u/jiwon/proj/less-bus/libLESS.so')
> myprint0 = libtcc.TCC_Print
> myprint0.argtypes = [c_char_p]
> myprint0.restype = c_int
>
> def tcc_print0(space, msg):
>     return myprint0(msg)
> tcc_print0.unwrap_spec = [ObjSpace, str]
>
>
> What am I missing here?

I think the currently supported way to compile modules using rctypes
is the pypy/bin/compilemodule.py script, and indeed your code
translates well with it.

I suggest (but I did not test) adding this line somewhere in
targetstandalone.py:
    import pypy.rpython.rctypes.implementation
This should register the annotation for calling ctypes functions.


-- 
Amaury Forgeot d'Arc

From stephen at thorne.id.au  Thu Jul  6 08:43:12 2006
From: stephen at thorne.id.au (Stephen Thorne)
Date: Thu, 6 Jul 2006 16:43:12 +1000
Subject: [pypy-dev] Writing os.pipe() using ctypes
In-Reply-To: 0
Message-ID: <20060706064312.29014.218909000.divmod.quotient.24486@ohm>

Hi,

I've been having fun this week trying to get twisted's unit tests running on top of pypy. This has the dual goals of getting twisted to run on a funky platform like pypy, and being able to give pypy a bit of a run around the yard. I have to report this is going "better than expected". I've already used rhymes's fcntl implementation and had to extend the posix module, I intend to do a diff once I get everything sorted.

I have a problem with rpython translation of my implementation of posix.pipe(). I have other ctypes implementations of functions (such as posix.access()) which work okay, but I'm having trouble with this particular case.

man 2 pipe says that I use it like this:

       int pipe(int filedes[2]);

My implementation looks like:

fd_arr = c_int * 2
def pipe(space):
    fds = fd_arr()
    fds[0] = 0
    fds[1] = 0
    if libc.pipe(fds):
        raise makeOSError(space)
    return space.wrap((fds[0], fds[1]))
pipe.unwrap_spec = [ObjSpace]

And the error I get from time {{{python translate.py --text --batch  standalone --thread}}} is:
[translation:ERROR]  Exception': unexpected prebuilt constant: <slot wrapper '__init__' of '_ctypes.Array' objects>
[translation:ERROR]     .. v0 = simple_call((ArrayType c_long_Array_2))
[translation:ERROR] Processing block:
[translation:ERROR]  block at 3 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in (pypy.module.posix.ctypes_posix:43)pipe
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v0 = simple_call((ArrayType c_long_Array_2))
[translation:ERROR]        v1 = setitem(v0, (0), (0))
[translation:ERROR]        v2 = setitem(v0, (1), (0))
[translation:ERROR]        v3 = getattr((<CDLL '/usr/lib/libc...20a89f0>), ('pipe'))
[translation:ERROR]        v4 = simple_call(v3, v0)
[translation:ERROR]        v5 = is_true(v4)
[translation:ERROR]  --end--

I've tried a bunch of ways of initialising the 'fds' object as to not make it a 'prebuilt constant', fds = fd_arr(0,0), creating fd_arr dynamically, fd_arr(zero, zero), etc. And I simply cannot seem to get rid of this exception.

So, any ideas as to how I go about making this work?

Stephen.

From amauryfa at gmail.com  Thu Jul  6 11:43:32 2006
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 6 Jul 2006 11:43:32 +0200
Subject: [pypy-dev] Writing os.pipe() using ctypes
In-Reply-To: <20060706064312.29014.218909000.divmod.quotient.24486@ohm>
References: <20060706064312.29014.218909000.divmod.quotient.24486@ohm>
Message-ID: <e27efe130607060243m5eed761fma545b5c976ac66e9@mail.gmail.com>

Hello,

2006/7/6, Stephen Thorne <stephen at thorne.id.au>:
> Hi,
>
> I've been having fun this week trying to get twisted's unit tests running on top of pypy. This has the dual goals of getting twisted to run on a funky platform like pypy, and being able to give pypy a bit of a run around the yard. I have to report this is going "better than expected". I've already used rhymes's fcntl implementation and had to extend the posix module, I intend to do a diff once I get everything sorted.
>
> I have a problem with rpython translation of my implementation of posix.pipe(). I have other ctypes implementations of functions (such as posix.access()) which work okay, but I'm having trouble with this particular case.
>
> man 2 pipe says that I use it like this:
>
>        int pipe(int filedes[2]);
>
> My implementation looks like:
>
> fd_arr = c_int * 2
> def pipe(space):
>     fds = fd_arr()
>     fds[0] = 0
>     fds[1] = 0
>     if libc.pipe(fds):
>         raise makeOSError(space)
>     return space.wrap((fds[0], fds[1]))
> pipe.unwrap_spec = [ObjSpace]
>
> And the error I get from time {{{python translate.py --text --batch  standalone --thread}}} is:
> [translation:ERROR]  Exception': unexpected prebuilt constant: <slot wrapper '__init__' of '_ctypes.Array' objects>
> [translation:ERROR]     .. v0 = simple_call((ArrayType c_long_Array_2))
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at 3 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
> [translation:ERROR]  in (pypy.module.posix.ctypes_posix:43)pipe
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v0 = simple_call((ArrayType c_long_Array_2))
> [translation:ERROR]        v1 = setitem(v0, (0), (0))
> [translation:ERROR]        v2 = setitem(v0, (1), (0))
> [translation:ERROR]        v3 = getattr((<CDLL '/usr/lib/libc...20a89f0>), ('pipe'))
> [translation:ERROR]        v4 = simple_call(v3, v0)
> [translation:ERROR]        v5 = is_true(v4)
> [translation:ERROR]  --end--
>
> I've tried a bunch of ways of initialising the 'fds' object as to not make it a 'prebuilt constant', fds = fd_arr(0,0), creating fd_arr dynamically, fd_arr(zero, zero), etc. And I simply cannot seem to get rid of this exception.
>
> So, any ideas as to how I go about making this work?
>
> Stephen.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

It seems that translate.py does not "register" annotation for rctypes objects.
In a previous mail I suggested to add
    import pypy.rpython.rctypes.implementation
in pypy/translator/goal/translate.py and translation seems to go further.
(I'm running
    translate.py standalone --usemodules=_demo
after I added your code in demo.py)

Unfortunately the complete translation takes too much memory for the
machine I'm currently working on, so I could not test completely.
And since its a Windows box, I had to remove the libc.pipe call completely...

Hope this helps.
It seems that many people are interested in the new rctypes way to
create modules...

-- 
Amaury Forgeot d'Arc

From arigo at tunes.org  Thu Jul  6 13:13:30 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Jul 2006 13:13:30 +0200
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <e27efe130607050247m9dc6015g545b405cbfde6746@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
	<b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
	<b008462b0607050116l2fe0ac51m10c611b2d61bbccb@mail.gmail.com>
	<e27efe130607050245y5d268500jd8a47db4593bebdd@mail.gmail.com>
	<e27efe130607050247m9dc6015g545b405cbfde6746@mail.gmail.com>
Message-ID: <20060706111330.GA29352@code0.codespeak.net>

Hi,

On Wed, Jul 05, 2006 at 11:47:00AM +0200, Amaury Forgeot d'Arc wrote:
> I suggest (but I did not test) adding this line somewhere in
> targetstandalone.py:
>     import pypy.rpython.rctypes.implementation
> This should register the annotation for calling ctypes functions.

Indeed, this module needs to be imported from somewhere.  I'd recommend
to import it from tcc.py directly, even.  See the end of the following
section:

  http://codespeak.net/pypy/dist/pypy/doc/rctypes.html#restrictions


A bientot,

Armin

From cfbolz at gmx.de  Fri Jul  7 14:03:17 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 7 Jul 2006 14:03:17 +0200
Subject: [pypy-dev] PyPy is faster than CPython (given an evil enough
	benchmark)
Message-ID: <348899050607070503o62fe5edaw134ea2953bb17b92@mail.gmail.com>

Hi all!

Together with the optimized string objects that Armin, Pieter,
Alexander and me have been working on, it is possible to construct a
(very corner-casish) benchmark, which exploits the fast cases of PyPy
in such a way, that it is quite a bit faster CPython. See the
following code:

import time, sys

def test_speed(num):
    t1 = time.time()
    result = ""
    intermediate = []
    for i in range(num):
        result += str(i)
        intermediate.append(result)
    t2 = time.time()
    return t2 - t1, intermediate

t1 = time.time()
for i in range(1, 12):
    speed, intermediate = test_speed(i * 1000)
t2 = time.time()
print t2 - t1


It runs 3.4 times faster on pypy-c with WITHSTRSLICE = WITHSTRJOIN =
True. In addition CPython uses 25 times more memory (which of course
accounts for some of the slowness).

Of course the lesson of this is that benchmarks are worthless :-).

Cheers,

Carl Friedrich

From seojiwon at gmail.com  Sat Jul  8 21:51:56 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Sat, 8 Jul 2006 12:51:56 -0700
Subject: [pypy-dev] Question on pypy exception...
Message-ID: <b008462b0607081251r3a07b15dldfd925ade56fe533@mail.gmail.com>

I'm getting an exception with pypy-c, and the exception occurs right
after pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr function in
pypy_g_PyInterpFrame_dispatch_translated.

1. Is "pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr" same as
LL_thread_fused_releaseacquirelock ? I can only see "#define
LL_thread_fused_releaseacquirelock
pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr", but not the other
way around.

2. Is that function there for forcing context switching?

3. What could happen in
pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr() that can set
(&pypy_g_pypy_translator_c_exceptiontransform_ExcData)->ed_inst_exc_type?
the error message that I'm seeing is like following

propagating implement_6.c:100878 pypy_g_PyInterpFrame_dispatch_translated error
propagating implement_5.c:30508 pypy_g_PyFrame_eval error
propagating implement_5.c:37782 pypy_g_EvalFrame_resume error
propagating implement_39.c:41465 pypy_g_ll_deallocator__Address_68 error
propagating implement.c:27685 pypy_g_funccall_star1 error
propagating implement_39.c:32105 pypy_g_ll_deallocator__Address_14 error
propagating implement_39.c:32158 pypy_g_ll_deallocator__Address_15 error
propagating implement_39.c:32294 pypy_g_ll_deallocator__Address_17 error
propagating implement.c:1336 pypy_g_entry_point error

Thanks!

-Jiwon

From mwh at python.net  Sun Jul  9 13:54:17 2006
From: mwh at python.net (Michael Hudson)
Date: Sun, 09 Jul 2006 12:54:17 +0100
Subject: [pypy-dev] PyPy Post-EuroPython 2006 Sprint Report
Message-ID: <2mejwv3seu.fsf@starship.python.net>

Greetings from an assortment of tired brains at CERN, home to a larger
assortment of hopefully less tired physicist brains.  We are reaching
the end of a succesful and well-attended (24 participants!) sprint.

3.5 days times 24 people is a bit more than two man months of work --
we are going to try to cover all we've achieved, but will undoubtedly
forget some stuff!

Summer of Code student Lawrence and his mentor Arre spent the majority
of the time integrating Lawrence's SoC work -- ctypes implementations
of various CPython extension modules -- into PyPy.  He has implemented
the time, mmap and fcntl modules so far, and the work at this sprint
concentrated on putting the time module into the form required for
PyPy -- making it a mixed module, conforming to the restrictions of
rctypes and fighting the extension compiler.

Brian Sutherland and Sad Rejeb also joined in this play, attempting to
convert some prewritten performance sensitive caching code used in
Zope(!) somewhere to a mixed module suitable for feeding to the
extension compiler.  Unfortunately the resulting code was no faster
than the pure Python version... probably because of the ext-compiler
using refcounting for GC.

Antonio appeared to spend his entire time writing commit messages but
somehow must have written some code in between (with the help of *his*
SoC mentor, Armin) because now all of PyPy's Standard Interpreter can
be rtyped with the ootypesystem.  "All" that remains are the dict
implementation and external function support in the CLI backend before
IronPython has a competitor in the shape of PyPy.Net :-)

Guido (not *that* Guido) being the JavaScript deity he is wrote a JS
tool to instrospect the DOM methods and produce useful information for
Maciej's SoC (we only had three SoC students here...) project.
Unfortunately browsers' JavaScript implementations are famously
eccentric and running his tool popped up the Print dialog box twice
and then crashed Epiphany.  It proved marginally less frustrating to
write a scraper for Mozilla's documentation and fix the broken HTML
he found there.

Aur?lien, Ludovic and part of Armin tried to figure out why the
example of coroutine cloning written for the "What can PyPy do for
you?" talk didn't work.  Assorted mind boggling bugs were found and
fixed, with the aggravating result that it now works fine when run on
the llinterpreter but crashes -- after a while -- when translated to
C.  More wizardry will be required here...

Just before the sprint Maciej attempted to improve the annotator's
error messages and he and Samuele spent some time fixing this new code
(it's very frustrating when your error printing routines crash on
you...).

A varying team of people including but not limited to Pieter, Carl
Friedrich, Maciej and Alexander worked on alternative, hopefully
faster implementations of Python data types, particularly strings and
dictionaries.  We now have efficient representations of slices of
strings and for the result of adding two strings, all completely
transparent to application-level.  For a sufficiently cooked
benchmark, we are now up to seven times faster than CPython (and use
30 times less memory).  On the dictionary side, we stole an
optimization for CPython: assume on creation that a dictionary will
only contain strings keys and switch to an alternative implementation
on the insertion of a non-string key.  This makes pystone and richards
both around 20% faster, some benchmarks up to 40% faster (but also
some a few percent slower).

Alexandre and David started to think about configuration and option
handling, but got distracted when they tried to run a translation on a
64-bit machine (or more specifically, a machine with 64-bit userland),
because it broke in obscure ways.  A few obscure fixes later and it
worked again apart from stackless, and another few fixes fixed that
too.

Michael and Fabrizio worked on the stackless code, "un-inlining" some
of the frame-state saving code with the goal of reducing the size of
the generated code.  This involved taking Fabrizio on a whirlwind tour
of one of the scarier areas of the PyPy codebase, but he survived and
the feature was implemented, although the size reduction was not as
great as had been hoped -- more investigation required (we also need
to build a stacklessgc build to check that).

Michael and Fabrizio then worked on AST- and bytecode-level
optimizations, which mostly involved building an infrastructure to
make modifications of the AST easier.  They implemented a simple
constant folder at the AST level and at the bytecode level an
optimization to remove the temporary tuple in code like "x,y=y,x".

Simon and Eric resurrected and experimented more with LLVM's JIT
interface.  Simon then worked with Ludovic and Arre on implementing
RPython support for numarrays and Eric worked on making the output of
genc compilable with C++.

Guido and Holger implemented the beginning and perhaps the middle but
probably not the end of doctest support in py.test.

After some discussion with Alexandre, David and Holger, Guido and Carl
Friedrich began to work on the rather ugly-looking task of taming
pypy's growing menagerie of options.  They started to implement a
general mechanism for defining a hierarchy of options, handling
dependencies and conflicts among them.  From this you can generate
optparse Options for command line interfaces and will one day be able
to generate a web form to choose among the zoo of pypy's options.
They used the config mechanisms to de-insane the selection of the
various object space variants.  This, for example, makes automated
testing of the above-mentioned variant string and dictionary
implementations possible.

Pieter worked on random modules^W^Wthe random module, porting it to
RPython and implementing some missing functions.

So, after a pre-sprint (for some of us), a conference and a
post-sprint, we are well prepared for the usual post-sprint week long
sleep.  Apart from Carl, he has an exam on Friday.  Oops!

Met vriendelijke groeten,
mwh&cfbolz

-- 
   This proposal, if accepted, will probably mean a heck of a lot of
   work for somebody.  But since I don't want it accepted, I don't
   care.                                   -- Laura Creighton, PEP 666


From arigo at tunes.org  Mon Jul 10 16:12:36 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 10 Jul 2006 16:12:36 +0200
Subject: [pypy-dev] [emil@dmr.ath.cx: Re: [pypy-svn] r29899 -
	pypy/extradoc/talk/ep2006]
Message-ID: <20060710141236.GA27105@code0.codespeak.net>

Hi all,

----- Forwarded message from Emil Mikulic <emil at dmr.ath.cx> -----

Date: Mon, 10 Jul 2006 17:23:05 +1000
From: Emil Mikulic <emil at dmr.ath.cx>
To: arigo at codespeak.net
Subject: Re: [pypy-svn] r29899 - pypy/extradoc/talk/ep2006

On Sun, Jul 09, 2006 at 03:35:03PM +0200, arigo at codespeak.net wrote:
> http://snake.cs.uni-duesseldorf.de/tgtest/

I see:

	Bad Gateway

	The proxy server received an invalid response from an upstream server.
	Apache Server at snake.cs.uni-duesseldorf.de Port 80

=/

----- End forwarded message -----

From faassen at infrae.com  Tue Jul 11 14:14:46 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Tue, 11 Jul 2006 14:14:46 +0200
Subject: [pypy-dev] PyPy is faster than CPython (given an evil enough
 benchmark)
In-Reply-To: <348899050607070503o62fe5edaw134ea2953bb17b92@mail.gmail.com>
References: <348899050607070503o62fe5edaw134ea2953bb17b92@mail.gmail.com>
Message-ID: <44B39636.1030804@infrae.com>

Carl Friedrich Bolz wrote:
[snip]
> Of course the lesson of this is that benchmarks are worthless :-).

Cool! I'll go tell everybody that PyPy is already faster than CPython 
now. :)

The best benchmarks will be large real-world codebases running on PyPy.

Regards,

Martijn


From l.oluyede at gmail.com  Tue Jul 11 18:00:12 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Tue, 11 Jul 2006 18:00:12 +0200
Subject: [pypy-dev] Bug hunting in rctime becomes wtf
Message-ID: <9eebf5740607110900l1ba16f2fu4eb220528c978494@mail.gmail.com>

I had a nasty bug like this one in the compilation process of the rctime module:

[translation:ERROR] Error:
[translation:ERROR]  TyperError: non-constant tuple index
[translation:ERROR] .. block at 34 with 1 exits
[translation:ERROR] .. v0 = getitem(tup_0, ___i_0)

You can reproduce it decommenting asctime function from interp_time.py
and in the __init__.py file of the rctime module.

After some digging I've found the source of the problem in the code
Armin and I have checked-in at the sprint to add the support for
"*args" arguments (see visit_args_w in pypy.objspace.cpy.function). I
tried patching it but I'm not very into cpy objspace and the compiler
mechanism under the hood.

The "wtf" thing came out when I tried to use the file with this
"stupid" patch applied:

rhymes at groove:~/scc/pypy/pypy/objspace/cpy $ svn diff
Index: function.py
===================================================================
--- function.py (revision 29953)
+++ function.py (working copy)
@@ -61,7 +61,7 @@
         tramp.inputargs.append('*' + basename)
         tramp.wrappings.append('%s = []' % (argname,))
         tramp.wrappings.append('for ___i in range(len(%s)):' % (basename,))
-        tramp.wrappings.append('    %s.append(___W_Object(%s[___i]))' % (
+        tramp.wrappings.append('    %s.append(___W_Object(%s[0]))' % (
             argname, basename))
         tramp.passedargs.append(argname)

The thing surprised me is the compilation after the patch goes fine
and (!!) the module works correctly. asctime() does it's dirty job.

So, why does it work?

-- 
Lawrence
http://www.oluyede.org/blog

From stephen at thorne.id.au  Wed Jul 12 06:56:35 2006
From: stephen at thorne.id.au (Stephen Thorne)
Date: Wed, 12 Jul 2006 14:56:35 +1000
Subject: [pypy-dev] PyPy is faster than CPython (given an evil enough
 benchmark)
In-Reply-To: <44B39636.1030804@infrae.com>
Message-ID: <20060712045635.29014.1792869848.divmod.quotient.32800@ohm>

On Tue, 11 Jul 2006 14:14:46 +0200, Martijn Faassen <faassen at infrae.com> wrote:
>Carl Friedrich Bolz wrote:
>[snip]
>> Of course the lesson of this is that benchmarks are worthless :-).
>
>Cool! I'll go tell everybody that PyPy is already faster than CPython
>now. :)
>
>The best benchmarks will be large real-world codebases running on PyPy.

This is precisely the reason I'm putting effort into getting the twisted unit tests to run on top of PyPy. Once we get the tests running on translated pypy, we can add a column to the http://twistedmatrix.com/buildbot/ builder, and get PyPy up as a platform we can run a harsh series of non-trivial unit tests on.

My work on this has stalled currently as there are a couple bugs in translation, but with the work Lawrence is doing on things like the posix module and fcntl, we're well on our way to being able to get this going.

FWIW, i've already managed to get twisted to run its own unit tests on its unit testing engine (trial) using py.py. Once we get fcntl translatable, I hope to report what happens when I run the lot (2764 tests) under pypy-c and 'see what happens'.

250 tests took approximately 11 hours to run.

Stephen.

From hpk at trillke.net  Thu Jul 13 08:02:29 2006
From: hpk at trillke.net (holger krekel)
Date: Thu, 13 Jul 2006 08:02:29 +0200
Subject: [pypy-dev] Security ideas
In-Reply-To: <20060524114728.GA6374@code0.codespeak.net>
References: <20060524114728.GA6374@code0.codespeak.net>
Message-ID: <20060713060229.GR32712@solar.trillke>

Hi Armin, 

On Wed, May 24, 2006 at 13:47 +0200, Armin Rigo wrote:
> On Monday I was at an inspiring seminar about (a specific form of)
> language-level security.  I've collected the PyPy-ification of these
> ideas there:
> 
> http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt

IIUC, in this discussion you argue against an "Proxy-Object space" solution
because a code block may depend on a condition involving secret values. 

In your alternative "annotator" suggestion, you give the following example: 

    def enter_bid(n):
        if n > highest_bid.value:
            highest_bid.value = n 
    enter_bid = secure(enter_bid)

Here the annotator analysis is supposed to prevent a leak of information
from the secret value.  But if the if-branch additionally contains:

            num_bids += 1

don't you run into a branching/code-dependent-on-secret-condition 
problem again?  Would the annotator prevent the manipulation of 
the global 'num_bids'?  Would it need to be a public value? 

Moreover, i have practical concerns: your proposed
scheme requires RPython annotator analysis which implies to
have the PyPy tool chain available and accessible at programming 
time.  Not impossible but also not a use case that we went for 
so far.  Also it is not clear to which target "secure" would compile 
functions to, C or bytecode or ...?

best, 

    holger

From Ben.Young at risk.sungard.com  Fri Jul 14 13:12:59 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Fri, 14 Jul 2006 12:12:59 +0100
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
In-Reply-To: <20060714105155.DC18A10090@code0.codespeak.net>
Message-ID: <OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>

Hi Xoraxax (sorry, I don't know your real name!),

This looks really good! It shows there are still some major (easy) 
performance wins in PyPy.

One question, why would you need any calls to str2object in any of the 
non-mutating methods? Surely the class knows that no non-strings have been 
added, and if __getitem__ is called with a non-string, you could just 
throw KeyError. I guess this is becuase you have to replicate the fact 
that hash is always called, but couldn't you do:

            if space.is_w(w_lookup_type, space.w_str):
                return w_dict.content_str[space.str_w(w_lookup)]
            else:
                w_lookup.__hash__()
                raise KeyError

?

Keep up the good work (to all PyPy's). Things are really starting to look 
good!

Cheers,
Ben

pypy-svn-bounces at codespeak.net wrote on 14/07/2006 11:51:55:

> Author: xoraxax
> Date: Fri Jul 14 12:51:55 2006
> New Revision: 30034
> 
> Modified:
>    pypy/dist/pypy/objspace/std/dictstrobject.py
> Log:
> Speeded up dict creation, added shortcut for look ups of vars with 
> sane hashes, fixed get mm, fixed mutation handling in the iterators.
> (manual merge of changes done in the other branch)
> 
> Modified: pypy/dist/pypy/objspace/std/dictstrobject.py
> 
==============================================================================
> --- pypy/dist/pypy/objspace/std/dictstrobject.py   (original)
> +++ pypy/dist/pypy/objspace/std/dictstrobject.py   Fri Jul 14 12:51:55 
2006
> @@ -1,4 +1,3 @@
> -
>  from pypy.objspace.std.objspace import *
>  from pypy.interpreter import gateway
> 
> @@ -7,52 +6,52 @@
>  class W_DictStrObject(W_Object):
>      from pypy.objspace.std.dicttype import dict_typedef as typedef
> 
> -    def __init__(w_self, space, w_otherdict=None):
> +    def __init__(w_self, space):
>          w_self.space = space
> -        if w_otherdict is None:
> -            w_self.content = None
> -            w_self.content_str = {}
> -        else:
> -            if w_otherdict.content is None:
> -                w_self.content = None
> -                w_self.content_str = w_otherdict.content_str.copy()
> -            else:
> -                w_self.content = w_otherdict.content.copy()
> -                w_self.content_str = None
> +        w_self.content = None
> +        w_self.content_str = {}
> 
>      def initialize_content(w_self, list_pairs_w): # YYY
>          for w_k, w_v in list_pairs_w:
>              w_self.setitem(w_k, w_v)
> 
>      def getitem(w_dict, w_lookup):
> +        """ Helper function, raises rpython exceptions. """
>          space = w_dict.space
>          if w_dict.content is None:
> -            if space.is_w(space.type(w_lookup), space.w_str):
> +            w_lookup_type = space.type(w_lookup)
> +            if space.is_w(w_lookup_type, space.w_str):
>                  return w_dict.content_str[space.str_w(w_lookup)]
>              else:
> +                if w_dict.isSaneHash(w_lookup_type):
> +                    raise KeyError
>                  w_dict.str2object()
>          return w_dict.content[w_lookup]
> 
> -    def get(w_dict, w_lookup):
> +    def get(w_dict, w_lookup, w_default):
> +        """ Helper function, raises rpython exceptions. """
>          space = w_dict.space
>          if w_dict.content is None:
> -            if space.is_w(space.type(w_lookup), space.w_str):
> -                return w_dict.content_str.get(space.str_w(w_lookup), 
None)
> +            w_lookup_type = space.type(w_lookup)
> +            if space.is_w(w_lookup_type, space.w_str):
> +                return w_dict.content_str.get(space.
> str_w(w_lookup), w_default)
>              else:
> +                if w_dict.isSaneHash(w_lookup_type):
> +                    return w_default
>                  w_dict.str2object()
> -        return w_dict.content.get(w_lookup, None)
> +        return w_dict.content.get(w_lookup, w_default)
> 
> 
>      def setitem(w_self, w_k, w_v):
>          space = w_self.space
> -        if w_self.content is None:
> +        if w_self.content is not None:
> +            w_self.content[w_k] = w_v
> +        else:
>              if space.is_w(space.type(w_k), space.w_str):
>                  w_self.content_str[space.str_w(w_k)] = w_v
>              else:
>                  w_self.str2object()
>                  w_self.content[w_k] = w_v
> -        else:
> -            w_self.content[w_k] = w_v
> 
>      set_str_keyed_item = setitem
> 
> @@ -65,7 +64,17 @@
>          for k, w_v in w_self.content_str.items():
>              w_self.content[w_self.space.wrap(k)] = w_v
>          w_self.content_str = None
> - 
> +
> +    def isSaneHash(w_self, w_lookup_type):
> +        """ Handles the case of a non string key lookup.
> +        Types that have a sane hash/eq function should allow us to 
> return True
> +        directly to signal that the key is not in the dict in any case.
> +        XXX The types should provide such a flag. """
> + 
> +        space = w_self.space
> +        # XXX there are much more types
> +        return space.is_w(w_lookup_type, space.w_NoneType) or 
> space.is_w(w_lookup_type, space.w_int)
> +
>      def __repr__(w_self):
>          """ representation for debugging purposes """
>          return "%s(%s,%s)" % (w_self.__class__.__name__, w_self.
> content, w_self.content_str)
> @@ -144,7 +153,11 @@
> 
>  def contains__DictStr_ANY(space, w_dict, w_lookup):
>      if w_dict.content is None:
> -        if not space.is_w(space.type(w_lookup), space.w_str):
> +        w_lookup_type = space.type(w_lookup)
> +        if not space.is_w(w_lookup_type, space.w_str):
> +            if w_dict.isSaneHash(w_lookup_type):
> +                return space.w_False
> +            #foo("degenerated in contains: " + space.
> type(w_lookup).getname(space, '?'))
>              w_dict.str2object()
>          else:
>              return space.newbool(space.str_w(w_lookup) in 
w_dict.content_str)
> @@ -247,7 +260,14 @@
>      return w_res
> 
>  def dict_copy__DictStr(space, w_self):
> -    return W_DictStrObject(space, w_self)
> +    w_new_dict = W_DictStrObject(space)
> +    if w_self.content is None:
> +        w_new_dict.content = None
> +        w_new_dict.content_str = w_self.content_str.copy()
> +    else:
> +        w_new_dict.content = w_self.content.copy()
> +        w_new_dict.content_str = None
> +    return w_new_dict
> 
>  def dict_items__DictStr(space, w_self):
>      if w_self.content is not None:
> @@ -291,9 +311,7 @@
>          w_self.content.clear()
> 
>  def dict_get__DictStr_ANY_ANY(space, w_dict, w_lookup, w_default):
> -    if w_dict.content is None:
> -        return w_dict.content_str.get(space.str_w(w_lookup), w_default)
> -    return w_dict.content.get(w_lookup, w_default)
> +    return w_dict.get(w_lookup, w_default)
> 
>  app = gateway.applevel('''
>      def dictrepr(currently_in_repr, d):
> @@ -348,6 +366,12 @@
>      def return_entry(w_self, w_key, w_value):
>          raise NotImplementedError
> 
> +    def handleMutation(w_self):
> +        space = w_self.space
> +        w_self.len = -1   # Make this error state sticky
> +        raise OperationError(space.w_RuntimeError,
> +                 space.wrap("dictionary changed size during 
iteration"))
> +
>  registerimplementation(W_DictStrIterObject)
> 
>  class W_DictStrIter_Keys(W_DictStrIterObject):
> @@ -435,16 +459,25 @@
>      return w_dictiter
> 
>  def next__DictStrIterObject(space, w_dictiter):
> -    # XXX hum, this is broken currently
> -    if w_dictiter.content_str is not None and w_dictiter.
> w_dictobject.content_str is None:
> -        raise OperationError(space.w_RuntimeError,
> -                space.wrap("you stupid dictionary user - you have 
> just done an operation internally mutating the dictionary"))
> -
> -    if w_dictiter.content is not None:
> +    # iterate over the string dict even if the dictobject's data was 
forced
> +    # to degenerate. just bail out if the obj's dictionary size 
differs.
> +    if (w_dictiter.content_str is not None and w_dictiter.
> w_dictobject.content_str is None
> +        and len(w_dictiter.w_dictobject.content) != w_dictiter.len):
> +        w_dictiter.handleMutation()
> +
> +    if w_dictiter.content_str is not None:
> +        if w_dictiter.len != len(w_dictiter.content_str):
> +            w_dictiter.handleMutation()
> +        # look for the next entry
> +        w_result = w_dictiter.next_entry()
> +        if w_result is not None:
> +            w_dictiter.pos += 1
> +            return w_result
> +        # no more entries
> +        w_dictiter.content_str = None
> +    elif w_dictiter.content is not None:
>          if w_dictiter.len != len(w_dictiter.content):
> -            w_dictiter.len = -1   # Make this error state sticky
> -            raise OperationError(space.w_RuntimeError,
> -                     space.wrap("dictionary changed size during 
iteration"))
> +            w_dictiter.handleMutation()
>          # look for the next entry
>          w_result = w_dictiter.next_entry()
>          if w_result is not None:
> @@ -452,19 +485,6 @@
>              return w_result
>          # no more entries
>          w_dictiter.content = None
> -    else:
> -        if w_dictiter.content_str is not None:
> -            if w_dictiter.len != len(w_dictiter.content_str):
> -                w_dictiter.len = -1   # Make this error state sticky
> -                raise OperationError(space.w_RuntimeError,
> -                         space.wrap("dictionary changed size during
> iteration"))
> -            # look for the next entry
> -            w_result = w_dictiter.next_entry()
> -            if w_result is not None:
> -                w_dictiter.pos += 1
> -                return w_result
> -            # no more entries
> -            w_dictiter.content_str = None
>      raise OperationError(space.w_StopIteration, space.w_None)
> 
>  def len__DictStrIterObject(space, w_dictiter):
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
> 


From hpk at trillke.net  Fri Jul 14 13:40:55 2006
From: hpk at trillke.net (holger krekel)
Date: Fri, 14 Jul 2006 13:40:55 +0200
Subject: [pypy-dev] fixing ireland sprint (21th-27th august)
Message-ID: <20060714114055.GH32712@solar.trillke>

Hi folks, 

i took over trying to organise the Ireland sprint (scheduled
21st - 27th August currently) but i had problems getting feedback 
from the organisers yet.  Therefore, i am not sure about some 
details but i started a draft announcement anyway: 

    extradoc/sprintinfo/ireland-2006/announcement.txt

and also listed people from whom i have heard that they want to come

    extradoc/sprintinfo/ireland-2006/announcement.txt

Please check and add yourself already. 

However, how do we go on: do we just assume that the sprint
works out (it's definitely very likely) and book travels 
arriving 20th evening and leaving 27th or 28th? 

Or do we wait until - say - end next week in order
to be sure?  Anyway, if you book flights already, 
then make sure you have some insurance that you 
can step back and get most of your money back
(that's probably the strategy i am going to follow
myself). 

any comments? 

    holger

From seojiwon at gmail.com  Sat Jul 15 07:17:28 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Fri, 14 Jul 2006 22:17:28 -0700
Subject: [pypy-dev] cpy object space
Message-ID: <b008462b0607142217y3d8b2972sf64534c8d4288058@mail.gmail.com>

pypy/bin/py.py does not work with cpy object space

[llcoolj] bin > ./py.py -o cpy
[cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c
ctypesplatcheck_0.c -o ctypesplatcheck_0.o
[cbuild:execute] cc -pthread /tmp/usession-23/ctypesplatcheck_0.o -lm
-lpthread -o /tmp/usession-23/ctypesplatcheck_0
Traceback (most recent call last):
  File "./py.py", line 204, in ?
    sys.exit(main_(sys.argv))
  File "./py.py", line 82, in main_
    space.setitem(space.sys.w_dict,space.wrap('executable'),space.wrap(argv[0]))
AttributeError: 'CPyObjSpace' object has no attribute 'sys'

Having almost only introductory knowledge of pypy, I thought naively
following might solve the problem

class CPyObjSpace(baseobjspace.ObjSpace):
    def initialize(self):
        self.config.objspace.geninterp = False
        self.wrap_cache = {}
        self.sys = self.getbuiltinmodule("sys") ##### <-- adding just this line


but now it errors that it can't find w_dict attribute in space.sys
(py.py:main_() #82)  - still space.sys is W_Object. Maybe I need to
transform it as proper pypy interpreter module?

From seojiwon at gmail.com  Sat Jul 15 22:26:23 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Sat, 15 Jul 2006 13:26:23 -0700
Subject: [pypy-dev] cpy object space
In-Reply-To: <b008462b0607142217y3d8b2972sf64534c8d4288058@mail.gmail.com>
References: <b008462b0607142217y3d8b2972sf64534c8d4288058@mail.gmail.com>
Message-ID: <b008462b0607151326m1db3196fl81c9408e0a18033f@mail.gmail.com>

Never mind. I just heard that cpy obj space is for mixed module translation.
I thought I'd try thunk obj space with cpy obj space, but I rather not. ;)

-jiwon

On 7/14/06, Jiwon Seo <seojiwon at gmail.com> wrote:
> pypy/bin/py.py does not work with cpy object space
>
> [llcoolj] bin > ./py.py -o cpy
> [cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c
> ctypesplatcheck_0.c -o ctypesplatcheck_0.o
> [cbuild:execute] cc -pthread /tmp/usession-23/ctypesplatcheck_0.o -lm
> -lpthread -o /tmp/usession-23/ctypesplatcheck_0
> Traceback (most recent call last):
>  File "./py.py", line 204, in ?
>    sys.exit(main_(sys.argv))
>  File "./py.py", line 82, in main_
>    space.setitem(space.sys.w_dict,space.wrap('executable'),space.wrap(argv[0]))
> AttributeError: 'CPyObjSpace' object has no attribute 'sys'
>
> Having almost only introductory knowledge of pypy, I thought naively
> following might solve the problem
>
> class CPyObjSpace(baseobjspace.ObjSpace):
>    def initialize(self):
>        self.config.objspace.geninterp = False
>        self.wrap_cache = {}
>        self.sys = self.getbuiltinmodule("sys") ##### <-- adding just this line
>
>
> but now it errors that it can't find w_dict attribute in space.sys
> (py.py:main_() #82)  - still space.sys is W_Object. Maybe I need to
> transform it as proper pypy interpreter module?
>

From 2006a at usenet.alexanderweb.de  Fri Jul 14 18:09:13 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Fri, 14 Jul 2006 18:09:13 +0200
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
References: <20060714105155.DC18A10090@code0.codespeak.net>
	<OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>
Message-ID: <1p6bpe8fz3vkg$.dlg@usenet.alexanderweb.de>

Hi Ben Young,

On Fri, 14 Jul 2006 12:12:59 +0100, Ben.Young at risk.sungard.com wrote:

> This looks really good! It shows there are still some major (easy) 
> performance wins in PyPy.

Yes, I measured 35% speedup on richards/pystone on "the other branch" (that
solely knows this kind of dictionary unlike the current trunk).

> One question, why would you need any calls to str2object in any of the 
> non-mutating methods?

Because every class can define it's own hash/eq functions that might be
considered insane in case of having them return hashes that match the
hashes of a string. That would mean that {"foo": 1}[myClass()] could return
1. Therefore we have to switch the mode while having a shortcut for types
which are known to have sane hashes only.

>                 w_lookup.__hash__()

I didn't even consider that people would write __hash__ methods with
side-effects :)

Kind regards,
Alexander


From 2006a at usenet.alexanderweb.de  Sun Jul 16 23:40:01 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Sun, 16 Jul 2006 23:40:01 +0200
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
References: <20060714105155.DC18A10090@code0.codespeak.net>
	<OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>
Message-ID: <nbhefnx5b0oe.dlg@usenet.alexanderweb.de>

Hi Ben Young,

On Fri, 14 Jul 2006 12:12:59 +0100, Ben.Young at risk.sungard.com wrote:

> This looks really good! It shows there are still some major (easy) 
> performance wins in PyPy.

Yes, I measured 35% speedup on richards/pystone on "the other branch" (that 
solely knows this kind of dictionary unlike the current trunk).

> One question, why would you need any calls to str2object in any of the 
> non-mutating methods?

Because every class can define it's own hash/eq functions that might be 
considered insane in case of having them return hashes that match the 
hashes of a string. That would mean that {"foo": 1}[myClass()] could return 
1. Therefore we have to switch the mode while having a shortcut for types 
which are known to have sane hashes only.

>                 w_lookup.__hash__()

I didn't even consider that people would write __hash__ methods with 
side-effects :)

Kind regards,
Alexander, who has resent this posting 2 days after the initial try - Gmane
problems?


From Ben.Young at risk.sungard.com  Mon Jul 17 09:56:38 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 17 Jul 2006 08:56:38 +0100
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
In-Reply-To: <nbhefnx5b0oe.dlg@usenet.alexanderweb.de>
Message-ID: <OF3239899F.71F77371-ON802571AE.002B5B07-802571AE.002BA2EC@risk.sungard.com>

pypy-dev-bounces at codespeak.net wrote on 16/07/2006 22:40:01:

> Hi Ben Young,

Hi Alexander,

> 
> On Fri, 14 Jul 2006 12:12:59 +0100, Ben.Young at risk.sungard.com wrote:
> 
> > This looks really good! It shows there are still some major (easy) 
> > performance wins in PyPy.
> 
> Yes, I measured 35% speedup on richards/pystone on "the other branch" 
(that 
> solely knows this kind of dictionary unlike the current trunk).
> 
> > One question, why would you need any calls to str2object in any of the 

> > non-mutating methods?
> 
> Because every class can define it's own hash/eq functions that might be 
> considered insane in case of having them return hashes that match the 
> hashes of a string. That would mean that {"foo": 1}[myClass()] could 
return 
> 1. Therefore we have to switch the mode while having a shortcut for 
types 
> which are known to have sane hashes only.

Ouch! Yes, I see the problem now. Roll on parameretised collections :)

> 
> >                 w_lookup.__hash__()
> 
> I didn't even consider that people would write __hash__ methods with 
> side-effects :)
> 
> Kind regards,
> Alexander, who has resent this posting 2 days after the initial try - 
Gmane
> problems?
>

Not me!
 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From arigo at tunes.org  Mon Jul 17 20:02:57 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Jul 2006 20:02:57 +0200
Subject: [pypy-dev] Security ideas
In-Reply-To: <20060713060229.GR32712@solar.trillke>
References: <20060524114728.GA6374@code0.codespeak.net>
	<20060713060229.GR32712@solar.trillke>
Message-ID: <20060717180257.GA18722@code0.codespeak.net>

Hi Holger,

On Thu, Jul 13, 2006 at 08:02:29AM +0200, holger krekel wrote:
>     def enter_bid(n):
>         if n > highest_bid.value:
>             highest_bid.value = n 
>     enter_bid = secure(enter_bid)
> 
> Here the annotator analysis is supposed to prevent a leak of information
> from the secret value.  But if the if-branch additionally contains:
> 
>             num_bids += 1
> 
> don't you run into a branching/code-dependent-on-secret-condition 
> problem again?  Would the annotator prevent the manipulation of 
> the global 'num_bids'?  Would it need to be a public value? 

You can't modify global values in RPython anyway.  But more generally,
yes, the annotator would follow all mutations and propagate security
levels.

> Moreover, i have practical concerns: your proposed
> scheme requires RPython annotator analysis which implies to
> have the PyPy tool chain available and accessible at programming 
> time.  Not impossible but also not a use case that we went for 
> so far.

Well, we haven't considered much at all.  This is just one approach,
which we didn't think of earlier, but there are of course others.

> Also it is not clear to which target "secure" would compile 
> functions to, C or bytecode or ...?

I don't know.  That's not too important for the security aspect;
anything would work, maybe even to the point of not actually compiling
at all but just using the annotator for the security proof (although I'd
regard this as less secure somehow, given the subtle differences between
RPython and full Python).


A bientot,

Armin

From hpk at trillke.net  Tue Jul 18 08:34:21 2006
From: hpk at trillke.net (holger krekel)
Date: Tue, 18 Jul 2006 08:34:21 +0200
Subject: [pypy-dev] Security ideas
In-Reply-To: <20060717180257.GA18722@code0.codespeak.net>
References: <20060524114728.GA6374@code0.codespeak.net>
	<20060713060229.GR32712@solar.trillke>
	<20060717180257.GA18722@code0.codespeak.net>
Message-ID: <20060718063421.GS28134@solar.trillke.net>

Hi Armin, 

On Mon, Jul 17, 2006 at 20:02 +0200, Armin Rigo wrote:
> On Thu, Jul 13, 2006 at 08:02:29AM +0200, holger krekel wrote:
> >     def enter_bid(n):
> >         if n > highest_bid.value:
> >             highest_bid.value = n 
> >     enter_bid = secure(enter_bid)
> > 
> > Here the annotator analysis is supposed to prevent a leak of information
> > from the secret value.  But if the if-branch additionally contains:
> > 
> >             num_bids += 1
> > 
> > don't you run into a branching/code-dependent-on-secret-condition 
> > problem again?  Would the annotator prevent the manipulation of 
> > the global 'num_bids'?  Would it need to be a public value? 
> 
> You can't modify global values in RPython anyway.  But more generally,
> yes, the annotator would follow all mutations and propagate security
> levels.

Hum, sorry for having asked too many questions at once.  Obviously, 
i could have said "whatever.num_bids +=1". 
Anyway, it seems that we cannot avoid the "dependent code block" issue 
but you consider the annotator more suited than an object space 
to deal with it, right? 

Also, from your answers i gather that your focus is more on 
getting something at-or-beyond-state-of-the-art first rather
than to provide something directly practical.  Probably makes
sense. 

best, 

    holger

From elmo13 at jippii.fi  Wed Jul 19 15:11:58 2006
From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=)
Date: Wed, 19 Jul 2006 16:11:58 +0300
Subject: [pypy-dev] translation with --profopt fails
Message-ID: <44BE2F9E.4060306@jippii.fi>

Last lines of output (python translate.py --backend=c --profopt='-c
"from richards import *;main(iterations=1)"' --text --batch
targetpypystandalone.py):

"""

[flowgraph] (pypy.rpython.memory.gctransform:733)ll_finalizer
[flowgraph] (pypy.rpython.memory.gctransform:733)ll_finalizer
[flowgraph] (pypy.rpython.memory.gctransform:733)ll_finalizer
XXX: operation v805 = direct_call((<* fn eq__Long_Long>), (StdObjSpace),
arg0_626, v804) cannot raise, but has exception guarding in graph
(?:1)_mm_eq_longS0_W_LongObject_W_BoolObject
XXX: operation v807 = direct_call((<* fn eq__Long_Long>), (StdObjSpace),
arg0_629, v806) cannot raise, but has exception guarding in graph
(?:1)_mm_eq_longS0_W_LongObject_W_IntObject
XXX: operation v809 = direct_call((<* fn eq__Long_Long>), (StdObjSpace),
arg0_631, v808) cannot raise, but has exception guarding in graph
(?:1)_mm_eq_longS0_W_LongObject_W_LongObject
[c]    28000 nodes  [ array: 3816  boehm rtti: 120  func: 2075  struct:
29285 ]
[c]    29000 nodes  [ array: 4045  boehm rtti: 120  func: 2149  struct:
29980 ]
[c]    30000 nodes  [ array: 4129  boehm rtti: 120  func: 2186  struct:
31192 ]
[c]    31000 nodes  [ array: 4257  boehm rtti: 120  func: 2201  struct:
32700 ]
[c]    32000 nodes  [ array: 4419  boehm rtti: 137  func: 2279  struct:
33433 ]
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 335, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 509, in proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/taskengine.py",
line 108, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 197, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 301, in task_database_c
[translation:ERROR]     database = cbuilder.build_database()
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/genc.py",
line 86, in build_database
[translation:ERROR]     db.complete()
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py",
line 302, in complete
[translation:ERROR]     add_dependencies(node.enum_dependencies())
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py",
line 288, in add_dependencies
[translation:ERROR]     self.get(value)
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py",
line 179, in get
[translation:ERROR]     if obj:   # test if the ptr is non-NULL
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/rpython/lltypesystem/lltype.py",
line 886, in __nonzero__
[translation:ERROR]     return self._obj is not None
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/rpython/lltypesystem/lltype.py",
line 910, in _getobj
[translation:ERROR]     raise RuntimeError("accessing already garbage
collected %r"
[translation:ERROR]  RuntimeError: accessing already garbage collected
<Array of Char >
"""

Just in case no one noticed this already...

Elmo

From leo.soto at gmail.com  Wed Jul 19 19:23:07 2006
From: leo.soto at gmail.com (Leonardo Soto)
Date: Wed, 19 Jul 2006 13:23:07 -0400
Subject: [pypy-dev] Implenting -m command line option
Message-ID: <5a8cc9e00607191023i75f4e6a6r79eda06a610b8944@mail.gmail.com>

Hi all,

I'm new to pypy, and to get myself started I'm trying to fix #145,
that is, implement the -m flag.

So I did search on the web and python sources looking for what exactly
-m means, and PEP-338[1] has the better explanation:

"If the module is found, and is of type PY_SOURCE or PY_COMPILED, then
the command line is effectively reinterpreted from python <options> -m
<module> <args> to python <options> <filename> <args>. This includes
setting sys.argv[0] correctly (some scripts rely on this - Python's
own regrtest.py is one example)."

But, by the way, PEP-338 is about generalizing -m to fully support
import hooks and executing modules inside packages, including a
reference implementation almost entirely written as a python module
(runpy). It has been included recently on CPython 2.5.

So, here is my question:

What "-m behaviour" should be implemented on pypy?

-- 
Leonardo Soto M.

[1] http://www.python.org/dev/peps/pep-0338/

From mwh at python.net  Wed Jul 19 21:59:43 2006
From: mwh at python.net (Michael Hudson)
Date: Wed, 19 Jul 2006 20:59:43 +0100
Subject: [pypy-dev] Implenting -m command line option
References: <5a8cc9e00607191023i75f4e6a6r79eda06a610b8944@mail.gmail.com>
Message-ID: <2md5c11i33.fsf@starship.python.net>

"Leonardo Soto" <leo.soto at gmail.com> writes:

> Hi all,
>
> I'm new to pypy, and to get myself started I'm trying to fix #145,
> that is, implement the -m flag.
>
> So I did search on the web and python sources looking for what exactly
> -m means, and PEP-338[1] has the better explanation:
>
> "If the module is found, and is of type PY_SOURCE or PY_COMPILED, then
> the command line is effectively reinterpreted from python <options> -m
> <module> <args> to python <options> <filename> <args>. This includes
> setting sys.argv[0] correctly (some scripts rely on this - Python's
> own regrtest.py is one example)."
>
> But, by the way, PEP-338 is about generalizing -m to fully support
> import hooks and executing modules inside packages, including a
> reference implementation almost entirely written as a python module
> (runpy). It has been included recently on CPython 2.5.
>
> So, here is my question:
>
> What "-m behaviour" should be implemented on pypy?

To be honest, either would be great.  The runpy-based solution sounds
easier, so maybe that?

Cheers,
mwh

-- 
41. Some programming languages manage to absorb change, but
    withstand progress.
  -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html


From anto.cuni at gmail.com  Thu Jul 20 00:53:50 2006
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 20 Jul 2006 00:53:50 +0200
Subject: [pypy-dev] Tomorrow morning...
Message-ID: <44BEB7FE.3060903@gmail.com>

... I will graduate!! :-) :-) :-)

A big, big 'thank you' to all pypy developers, for having welcomed and 
supported me during last months: it has been very nice to work with you 
(and I hope to continue in future).

I still remember my first post on pypy-dev: I was "Looking for a thesis" 
(cit.) and I was rather confused about pypy architecture: it was only 5 
months ago, but it seems years. Then I started coding and I've never 
stopped :-).

For those interested in, the final version of my thesis is here:
http://codespeak.net/svn/user/antocuni/tesi/Implementing%20Python%20in%20.NET.pdf

ciao Anto

From hpk at trillke.net  Thu Jul 20 07:05:45 2006
From: hpk at trillke.net (holger krekel)
Date: Thu, 20 Jul 2006 07:05:45 +0200
Subject: [pypy-dev] Tomorrow morning...
In-Reply-To: <44BEB7FE.3060903@gmail.com>
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <20060720050545.GC27182@solar.trillke.net>

Hi Antonio!

On Thu, Jul 20, 2006 at 00:53 +0200, Antonio Cuni wrote:
> ... I will graduate!! :-) :-) :-)

Woa! Big congrats! :))
 
> A big, big 'thank you' to all pypy developers, for having welcomed and 
> supported me during last months: it has been very nice to work with you 
> (and I hope to continue in future).

same here with you! 

> I still remember my first post on pypy-dev: I was "Looking for a thesis" 
> (cit.) and I was rather confused about pypy architecture: it was only 5 
> months ago, but it seems years. Then I started coding and I've never 
> stopped :-).
> 
> For those interested in, the final version of my thesis is here:
> http://codespeak.net/svn/user/antocuni/tesi/Implementing%20Python%20in%20.NET.pdf

just printed it and will see to read it sometime in the sun :) 

best & see you, 

    holger

From l.oluyede at gmail.com  Thu Jul 20 09:25:16 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 20 Jul 2006 09:25:16 +0200
Subject: [pypy-dev] Tomorrow morning...
In-Reply-To: <44BEB7FE.3060903@gmail.com>
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <9eebf5740607200025x29d5675flde43353894113e95@mail.gmail.com>

On 7/20/06, Antonio Cuni <anto.cuni at gmail.com> wrote:
> ... I will graduate!! :-) :-) :-)

Congratulations Antonio :-)

-- 
Lawrence
http://www.oluyede.org/blog

From Ben.Young at risk.sungard.com  Thu Jul 20 09:29:07 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Thu, 20 Jul 2006 08:29:07 +0100
Subject: [pypy-dev] .NET build failures
Message-ID: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>

Hi Antocuni,

I ran the .NET build last night and it got a lot further. The errors I get 
now are:

[translation:ERROR] ***** FAILURE *****
[translation:ERROR]
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(66076) : error -- 
Duplicate label: '__check_block_5'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(372039) : error -- 
Duplicate label: '__check_block_5'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821924) : error -- 
Duplicate label: '__check_block_7'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821985) : error -- 
Duplicate label: '__check_block_3'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822900) : error -- 
Duplicate label: '__check_block_7'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822961) : error -- 
Duplicate label: '__check_block_3'

I also notice that you appear to have identifiers with hundreds of 
underscores appended. Is this expected?

[translation:ERROR] Assembled global method 
memo_get_unique_interplevel_subclass_4______________________________________________________________________________
________________________________________________________________________________________________________________________________________________________________
________________________________________________________________________________________________________________________________________________________________
____________________________________

Cheers,
Ben

From mwh at python.net  Thu Jul 20 10:05:14 2006
From: mwh at python.net (Michael Hudson)
Date: Thu, 20 Jul 2006 09:05:14 +0100
Subject: [pypy-dev] Tomorrow morning...
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <2m8xmo1z2d.fsf@starship.python.net>

Antonio Cuni <anto.cuni at gmail.com> writes:

> ... I will graduate!! :-) :-) :-)
>
> A big, big 'thank you' to all pypy developers, for having welcomed and 
> supported me during last months: it has been very nice to work with you 
> (and I hope to continue in future).

No, thank you for all you've done!

Cheers,
mwh

-- 
  C++ is a siren song.  It *looks* like a HLL in which you ought to
  be able to write an application, but it really isn't.
                                       -- Alain Picard, comp.lang.lisp


From mwh at python.net  Thu Jul 20 10:16:21 2006
From: mwh at python.net (Michael Hudson)
Date: Thu, 20 Jul 2006 09:16:21 +0100
Subject: [pypy-dev] return of the return of pypy-sync
Message-ID: <2m4pxc1yju.fsf@starship.python.net>

Once again, I'd like to invite all active PyPy developers to a short
(no more than half an hour) sychronisation meeting in #pypy-sync on
freenode at 17:30 UTC+2 today.

Agenda:

 - status

   pre-prepared 3-line reports of what you did last week, what you
   plan to do next week and what is blocking your progress at this
   time.

   this is the area i see as the most important this week: work has
   begun to spread in many directions, and it's a good idea to have an
   idea who is doing what and where they've got to.

 - work out how to resolve any blockers above

 - summer of pypy

   the summer of pypy program is ready to be announced:

   http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy.html

   there's not too much to discuss, but the "project ideas" page:

   http://codespeak.net/pypy/dist/pypy/doc/independent-project-ideas.html

   could use an update, we can talk about ideas for that.

 - ireland sprint

   again, mostly an information point: our next scheduled sprint in
   Ireland towards the end of August will be announced soon:

   http://codespeak.net/pypy/extradoc/sprintinfo/ireland-2006/announce.html

   so maybe you can start thinking about travel arrangements already.
   it is looking likely that the sprint part will not be attended by a
   large number of locals, so it's likely to be a fairly "core"
   sprint.

See you there!

Cheers,
mwh

-- 
  The "of course, while I have no problem with this at all, it's
  surely too much for a lesser being" flavor of argument always
  rings hollow to me.                       -- Tim Peters, 29 Apr 1998


From Ben.Young at risk.sungard.com  Thu Jul 20 15:08:02 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Thu, 20 Jul 2006 14:08:02 +0100
Subject: [pypy-dev] .NET build failures
In-Reply-To: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>
Message-ID: <OFD704530D.0D116AD9-ON802571B1.0047CBEC-802571B1.00482551@risk.sungard.com>

To reply to myself :)

The problem appears to be in the _check function in opcodes.py. This 
generates a number for the jump at opcode creation time. Therefore, if an 
opcode is used twice in a block then you get the error.

I guess the solution would be to create a "_check" micro instruction that 
increments the count on render. I'm afraid I haven't got time to actually 
do that myself though.

Cheers,
Ben


pypy-dev-bounces at codespeak.net wrote on 20/07/2006 08:29:07:

> Hi Antocuni,
> 
> I ran the .NET build last night and it got a lot further. The errors I 
get 
> now are:
> 
> [translation:ERROR] ***** FAILURE *****
> [translation:ERROR]
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(66076) : error 
-- 
> Duplicate label: '__check_block_5'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(372039) : error 
-- 
> Duplicate label: '__check_block_5'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821924) : error 
-- 
> Duplicate label: '__check_block_7'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821985) : error 
-- 
> Duplicate label: '__check_block_3'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822900) : error 
-- 
> Duplicate label: '__check_block_7'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822961) : error 
-- 
> Duplicate label: '__check_block_3'
> 
> I also notice that you appear to have identifiers with hundreds of 
> underscores appended. Is this expected?
> 
> [translation:ERROR] Assembled global method 
> 
memo_get_unique_interplevel_subclass_4______________________________________________________________________________
> 
________________________________________________________________________________________________________________________________________________________________
> 
________________________________________________________________________________________________________________________________________________________________
> ____________________________________
> 
> Cheers,
> Ben
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From arigo at tunes.org  Thu Jul 20 17:59:12 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 20 Jul 2006 17:59:12 +0200
Subject: [pypy-dev] Tomorrow morning...
In-Reply-To: <44BEB7FE.3060903@gmail.com>
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <20060720155912.GA30518@code0.codespeak.net>

Hi Antonio,

On Thu, Jul 20, 2006 at 12:53:50AM +0200, Antonio Cuni wrote:
> ... I will graduate!! :-) :-) :-)

Congratulation!

> A big, big 'thank you' to all pypy developers, for having welcomed and 
> supported me during last months: it has been very nice to work with you 
> (and I hope to continue in future).

Thanks to you, you are doing quite a great job :-)

Armin

From 2006a at usenet.alexanderweb.de  Fri Jul 21 02:23:10 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Fri, 21 Jul 2006 02:23:10 +0200
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
References: <20060714105155.DC18A10090@code0.codespeak.net>
	<OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>
	<nbhefnx5b0oe.dlg@usenet.alexanderweb.de>
Message-ID: <no0jbpy9l9pv$.dlg@usenet.alexanderweb.de>

On Sun, 16 Jul 2006 23:40:01 +0200, I wrote:

>> This looks really good! It shows there are still some major (easy) 
>> performance wins in PyPy.
> 
> Yes, I measured 35% speedup on richards/pystone on "the other branch" (that 
> solely knows this kind of dictionary unlike the current trunk).

OK, newest benchmarks show a stable 32% speedup at richards.py, even
working on the trunk - you just need to enable the strdict flag like this
in the target:

    config.objspace.std.withstrdict = True

This gives the following factors for pypy-c vs. cpython (measured on
pypy2):
Richards: 2.98x as slow
PyStone: 3.5x as slow

Now I am looking forward to have someone building an LLVM/C/Profile guided
opt binary with strdicts enabled :-)

Kind regards,
Alexander


From hpk at trillke.net  Fri Jul 21 13:20:39 2006
From: hpk at trillke.net (holger krekel)
Date: Fri, 21 Jul 2006 13:20:39 +0200
Subject: [pypy-dev] Summer of PyPy: call for proposals
Message-ID: <20060721112039.GC27182@solar.trillke.net>

Hi to all students! 

Happily, we are able to offer students mentoring and full sprint
participant's funding if we receive a proposal outlining an
interesting project related to PyPy and its development tools.  This
follows up on the "Summer of Code" campaign from Google but is
completely independent from it and also works differently (i.e. there
is no success bounty).

See the full call for details: 

    http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy.html

and here is a list of concrete inspirations for possible work: 

    http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html

There are certainly many more possibilities now that PyPy starts to
get faster, Stackless and custom GCs are integrated, PyPy.NET, Logic
Variables, lazy evaluation and other new features are emerging.  Note
though that PyPy still is pursuing challenging research goals so it
will take some time before it becomes viable in production
environments.

Don't hesitate to drop by on #pypy on freenode or post questions/ideas
to pypy-dev at codespeak net.

A last note: you can propose something at any time, you don't need to
wait for a specific deadline. In fact, our next sprint is nearing in
Ireland, 21st-27th August and if you are quick you may already be
there with us :)

For the PyPy team, 

    Armin Rigo, Holger Krekel, Michael Hudson, Carl Friedrich Bolz, ... 


From bea at changemaker.nu  Fri Jul 21 13:40:42 2006
From: bea at changemaker.nu (Beatrice During)
Date: Fri, 21 Jul 2006 13:40:42 +0200
Subject: [pypy-dev] PyPy sprint Limerick 21-27th of August
Message-ID: <44C0BD3A.3060300@changemaker.nu>

Hi there

The PyPy team goes sprinting in Limerick/Ireland  - are
you interested in participating?


Ireland PyPy sprint 21th-27th August 2006
====================================================

The next PyPy sprint will happen in the nice city of
Limerick in Ireland from 21st till 27th August.
(Most people intend to arrive 20th August).

The main focus of the sprint will be on JIT compiler works,
various optimization works, porting extension modules,
infrastructure works like a build tool for PyPy, or
extended (distributed) testing.

It's also open to new topics.  If you are a student
consider to participate in `Summer of PyPy` in order
get funding for your travels and accomodation.

The sprint is being hosted by University of Limerick
(http://www.ul.ie/) - and is arranged in co-operation
with people from our sister project Calibre (www.calibre.ie).
Our contact at the University is P?r ?gerfalk and Eoin
Oconchuir.

For more information on logistics, facilities etc - please
read the full announcement at:

http://codespeak.net/pypy/extradoc/sprintinfo/ireland-2006/announce.html

For the PyPy team

Bea & Holger


From arigo at tunes.org  Fri Jul 21 17:29:39 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Jul 2006 17:29:39 +0200
Subject: [pypy-dev] translation with --profopt fails
In-Reply-To: <44BE2F9E.4060306@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>
Message-ID: <20060721152939.GB31604@code0.codespeak.net>

Hi Elmo,

On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
> Last lines of output (python translate.py --backend=c --profopt='-c
> "from richards import *;main(iterations=1)"' --text --batch
> targetpypystandalone.py):

I have no clue how --profopt can affect the translation itself.  You
don't get the same error without the --profopt option?


A bientot,

Armin

From elmo13 at jippii.fi  Fri Jul 21 18:07:38 2006
From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=)
Date: Fri, 21 Jul 2006 19:07:38 +0300
Subject: [pypy-dev] translation with --profopt fails
In-Reply-To: <20060721152939.GB31604@code0.codespeak.net>
References: <44BE2F9E.4060306@jippii.fi>
	<20060721152939.GB31604@code0.codespeak.net>
Message-ID: <44C0FBCA.1090605@jippii.fi>

Armin Rigo wrote:
> Hi Elmo,
>
> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>   
>> Last lines of output (python translate.py --backend=c --profopt='-c
>> "from richards import *;main(iterations=1)"' --text --batch
>> targetpypystandalone.py):
>>     
>
> I have no clue how --profopt can affect the translation itself.  You
> don't get the same error without the --profopt option?
>
>
> A bientot,
>
> Armin
>   
Should've tested that, will do as soon as possible. benchmark.html on
snake shows something that says to me that the profopt executable
doesn't exist, which suggests that it wasn't built.

Elmo


From anto.cuni at gmail.com  Fri Jul 21 21:07:26 2006
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 21 Jul 2006 21:07:26 +0200
Subject: [pypy-dev] .NET build failures
In-Reply-To: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>
References: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>
Message-ID: <44C125EE.1000900@gmail.com>

Ben.Young at risk.sungard.com wrote:
> Hi Antocuni,

Hi Ben,

> I ran the .NET build last night and it got a lot further. The errors I get 
> now are:
> 
> [translation:ERROR] ***** FAILURE *****
> [translation:ERROR]
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(66076) : error -- 
> Duplicate label: '__check_block_5'

as you correctly spotted out, the problem was in opcodes._check; I've 
fixed that, thanks for the help.

> I also notice that you appear to have identifiers with hundreds of 
> underscores appended. Is this expected?
> 
> [translation:ERROR] Assembled global method 
> memo_get_unique_interplevel_subclass_4______________________________________________________________________________
> ________________________________________________________________________________________________________________________________________________________________
> ________________________________________________________________________________________________________________________________________________________________
> ____________________________________

Yes, it's expected. Appending underscores it's a way to get unique names 
for methods and classes, though I didn't think to get such a big number 
of underscores. I changed the code and now I append a unique number at 
the end of the name instead of a list of underscores.

ciao Anto


From elmo13 at jippii.fi  Fri Jul 21 22:26:53 2006
From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=)
Date: Fri, 21 Jul 2006 23:26:53 +0300
Subject: [pypy-dev] translation with --profopt fails
In-Reply-To: <44C0FBCA.1090605@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>	<20060721152939.GB31604@code0.codespeak.net>
	<44C0FBCA.1090605@jippii.fi>
Message-ID: <44C1388D.80003@jippii.fi>

Elmo M?ntynen wrote:
> Armin Rigo wrote:
>   
>> Hi Elmo,
>>
>> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>>   
>>     
>>> Last lines of output (python translate.py --backend=c --profopt='-c
>>> "from richards import *;main(iterations=1)"' --text --batch
>>> targetpypystandalone.py):
>>>     
>>>       
>> I have no clue how --profopt can affect the translation itself.  You
>> don't get the same error without the --profopt option?
>>
>>
>> A bientot,
>>
>> Armin
>>   
>>     
> Should've tested that, will do as soon as possible. benchmark.html on
> snake shows something that says to me that the profopt executable
> doesn't exist, which suggests that it wasn't built.
>
> Elmo
>   
Translation without profopt worked now, though there might have been
something that got changed in between, will report after some more builds.

Elmo

From arigo at tunes.org  Fri Jul 21 22:36:13 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Jul 2006 22:36:13 +0200
Subject: [pypy-dev] Failing stackless tests
Message-ID: <20060721203613.GA2825@code0.codespeak.net>

Hi all, hi Aurelien,

The stackless-related tests are all consistently failing nowadays.  Some
have been failing for 4 days and others for 9 days.  See

   http://snake.cs.uni-duesseldorf.de/pypytest/summary.html

Given that you appear to continue to work on it, but without making
tests pass again, I'm a bit worrying exactly about what you are doing?
That doesn't seem too much in line with the "test-driven development"
approach.


A bientot,

Armin

From scott+pypy-dev at scottdial.com  Sat Jul 22 02:31:57 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Fri, 21 Jul 2006 20:31:57 -0400
Subject: [pypy-dev] Newb questions about ext module writing
Message-ID: <44C171FD.60801@scottdial.com>

Greetings everyone,

I'm relatively new to PyPy. I've been aware of it for a while, but never 
really got into poking around at all. But recently I decided this is a 
project that I would like to get involved in. I think at the time I 
planned on doing something big, like addding a JVM backend or something 
like that.. maybe I'll get back to that eventually. I realized I was 
fairly clueless with the codebase and decided to jump on writing a 
module as a type of introductory project.

The independent-project-ideas went out of its way to mention readline, 
so boom, readline is what I had/have chosen to finish. Things were going 
well until I started having to deal with making callback functions.

I have become stuck on the need to setup_readline with some callback 
functions which are capable of accessing the State of the module. I'm 
pretty lost on how to go about getting the state. Within the 
interp_readline, I do the "space.fromcache(State)" deal to get ahold of 
the State, but how do I do this from within the "c_readline" component. 
Perhaps this is completely the wrong way to go about this? My first 
thought is to import the "getstate" function from interp_readline, but I 
can't find anywhere else this is done in the codebase off-hand.

Furthermore, my callback functions from readline will not be passed any 
sort of "state", so how do I get that from within my callback function? 
Again, I thought a poor kludge was to tag reference to "state" onto the 
function during setup_readline.

def setup_readline(state, w_module):
     ...
     cfunc = CFUNCTYPE(POINTER(c_char_p), c_char_p, c_int, c_int)
     c_rl_attempted_completion_function = cfunc(space.flex_complete)
     space.flex_complete.space = space
     ...

def flex_complete(text, start, end):
     from pypy.module.readline.interp_readline import getstate
     getstate(flex_complete).begidx = start
     getstate(flex_complete).endidx = end
     ...

Some ugly stuff like that.. help? :) Thanks.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From len-l at telus.net  Sat Jul 22 03:24:19 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Fri, 21 Jul 2006 18:24:19 -0700
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C171FD.60801@scottdial.com>
Message-ID: <44C11BD3.25178.4E9F16@len-l.telus.net>

On 21 Jul 2006 at 20:31, Scott Dial wrote:

> Furthermore, my callback functions from readline will not be passed any 
> sort of "state", so how do I get that from within my callback function? 
> Again, I thought a poor kludge was to tag reference to "state" onto the 
> function during setup_readline.
> 
> def setup_readline(state, w_module):
>      ...
>      cfunc = CFUNCTYPE(POINTER(c_char_p), c_char_p, c_int, c_int)
>      c_rl_attempted_completion_function = cfunc(space.flex_complete)
>      space.flex_complete.space = space
>      ...
> 
> def flex_complete(text, start, end):
>      from pypy.module.readline.interp_readline import getstate
>      getstate(flex_complete).begidx = start
>      getstate(flex_complete).endidx = end
>      ...
> 
In ctypes that is easy: use a bound method as the callback. Does that 
even work in rctypes?

Lenard Lindstrom
<len-l at telus.net>



From arigo at tunes.org  Sat Jul 22 12:24:42 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 22 Jul 2006 12:24:42 +0200
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C11BD3.25178.4E9F16@len-l.telus.net>
References: <44C171FD.60801@scottdial.com>
	<44C11BD3.25178.4E9F16@len-l.telus.net>
Message-ID: <20060722102442.GA8055@code0.codespeak.net>

Hi,

On Fri, Jul 21, 2006 at 06:24:19PM -0700, Lenard Lindstrom wrote:
> In ctypes that is easy: use a bound method as the callback. Does that 
> even work in rctypes?

I'm afraid not, and it's not obvious how this would be compiled to C
code, given that there is no kind of closure in ANSI C.

I don't see how to do that at all, actually :-(  It's somehow a design
problem of the readline lib, which should in theory always pass an extra
void* "closure" arguments to callbacks...

It might be necessary to use hacks like a single global State instance,
storing the "space" the readline lib was initialized with, instead of
one State per space.


Armin

From scott+pypy-dev at scottdial.com  Sat Jul 22 13:39:27 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Sat, 22 Jul 2006 07:39:27 -0400
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <20060722102442.GA8055@code0.codespeak.net>
References: <44C171FD.60801@scottdial.com>	<44C11BD3.25178.4E9F16@len-l.telus.net>
	<20060722102442.GA8055@code0.codespeak.net>
Message-ID: <44C20E6F.2080608@scottdial.com>

Armin Rigo wrote:
> On Fri, Jul 21, 2006 at 06:24:19PM -0700, Lenard Lindstrom wrote:
>> In ctypes that is easy: use a bound method as the callback. Does that 
>> even work in rctypes?
> 
> I'm afraid not, and it's not obvious how this would be compiled to C
> code, given that there is no kind of closure in ANSI C.
> 
> I don't see how to do that at all, actually :-(  It's somehow a design
> problem of the readline lib, which should in theory always pass an extra
> void* "closure" arguments to callbacks...
> 
> It might be necessary to use hacks like a single global State instance,
> storing the "space" the readline lib was initialized with, instead of
> one State per space.

I am admittedly ignorant of rctypes capabilities, but I can appreciate
the conversion in C code being awkward. Reading around more, I see in
rctypes.implementation where an attempt for CALLBACK_FUNCTYPE was
droppped. Seems like I have bitten off more than I can chew here.

I think you understate the issue by saying it's "a design problem of the
readline lib" because many, many C libraries are going to need callback
functionality. If I have evaluated the project's current state
correctly, then I realize this is more-or-less not the current focus of
development. So, no biggie.

I just figured it would something I could just go through the motions on
to become familiar. I am open to other suggestions. I can comeback to
this when I have a better understanding of everything.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu


From mahs at telcopartners.com  Sat Jul 22 23:17:46 2006
From: mahs at telcopartners.com (Michael Spencer)
Date: Sat, 22 Jul 2006 14:17:46 -0700
Subject: [pypy-dev] 20th Level IP Communicator?
Message-ID: <e9u4la$54j$1@sea.gmane.org>

http://gigaom.com/2006/07/15/sipphone-brings-voice-to-nokia-770-tablet/#comment-125818

I thought you'd like this comment:

"looks like i?m about to fail my saving throw against ?cool gadget temptation?."


From mahs at telcopartners.com  Sat Jul 22 23:21:52 2006
From: mahs at telcopartners.com (Michael Spencer)
Date: Sat, 22 Jul 2006 14:21:52 -0700
Subject: [pypy-dev] 20th Level IP Communicator?
In-Reply-To: <e9u4la$54j$1@sea.gmane.org>
References: <e9u4la$54j$1@sea.gmane.org>
Message-ID: <e9u4sv$54j$2@sea.gmane.org>

Michael Spencer wrote:
> http://gigaom.com/2006/07/15/sipphone-brings-voice-to-nokia-770-tablet/#comment-125818
> 
> I thought you'd like this comment:
> 
> "looks like i?m about to fail my saving throw against ?cool gadget temptation?."
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
Sorry - sent to the wrong address
Michael


From arigo at tunes.org  Sun Jul 23 09:18:38 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 23 Jul 2006 09:18:38 +0200
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C20E6F.2080608@scottdial.com>
References: <44C171FD.60801@scottdial.com>
	<44C11BD3.25178.4E9F16@len-l.telus.net>
	<20060722102442.GA8055@code0.codespeak.net>
	<44C20E6F.2080608@scottdial.com>
Message-ID: <20060723071838.GA29559@code0.codespeak.net>

Hi Scott,

On Sat, Jul 22, 2006 at 07:39:27AM -0400, Scott Dial wrote:
> I am admittedly ignorant of rctypes capabilities, but I can appreciate
> the conversion in C code being awkward. Reading around more, I see in
> rctypes.implementation where an attempt for CALLBACK_FUNCTYPE was
> droppped. Seems like I have bitten off more than I can chew here.

Callbacks by themselves should "almost" work.  We need to re-enable the
commented-out code and try -- this code is lot complexity, some of it
bordering on the hackish side, which turned out not to be needed any
more at some point.  So to ease development of the rest of rctypes I
disabled the tests and commented it out.  But of course callbacks are
clearly needed in the long run.

Was I was complaining about is that there is no C-convertable way to
have e.g. bound methods as callbacks.  You can only use plain functions.
Many C libs have the same "design problem" as readline and don't pass
you a "void*" argument back.  An example of lib that is "correctly
designed" in this respect is the Windows API.

So for now I suppose these cases can be handled by just assuming that
there is only one 'space' around, and storing it in global state.


A bientot,

Armin.

From len-l at telus.net  Sun Jul 23 19:56:06 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Sun, 23 Jul 2006 10:56:06 -0700
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <20060722102442.GA8055@code0.codespeak.net>
References: <44C11BD3.25178.4E9F16@len-l.telus.net>
Message-ID: <44C355C6.11938.7869AE@len-l.telus.net>

On 22 Jul 2006 at 12:24, Armin Rigo wrote:

> Hi,
> 
> On Fri, Jul 21, 2006 at 06:24:19PM -0700, Lenard Lindstrom wrote:
> > In ctypes that is easy: use a bound method as the callback. Does that 
> > even work in rctypes?
> 
> I'm afraid not, and it's not obvious how this would be compiled to C
> code, given that there is no kind of closure in ANSI C.
> 
I just brought it up as it is one more thing to consider when 
converting a ctypes Python module to rctypes. But could rctypes use 
ffi when the callback is a bound method?

Lenard Lindstrom
<len-l at telus.net>



From l.oluyede at gmail.com  Sun Jul 23 19:58:50 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Sun, 23 Jul 2006 19:58:50 +0200
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C355C6.11938.7869AE@len-l.telus.net>
References: <44C11BD3.25178.4E9F16@len-l.telus.net>
	<20060722102442.GA8055@code0.codespeak.net>
	<44C355C6.11938.7869AE@len-l.telus.net>
Message-ID: <9eebf5740607231058w2a2b688bpb5c746c01750ad96@mail.gmail.com>

> I just brought it up as it is one more thing to consider when
> converting a ctypes Python module to rctypes. But could rctypes use
> ffi when the callback is a bound method?

AFAIK one of the reasons of the existence of rctypes and the ext.
compiler is not to use the ffi library...

-- 
Lawrence
http://www.oluyede.org/blog

From elmo13 at jippii.fi  Sun Jul 23 20:43:05 2006
From: elmo13 at jippii.fi (=?windows-1252?Q?Elmo_M=E4ntynen?=)
Date: Sun, 23 Jul 2006 21:43:05 +0300
Subject: [pypy-dev] A bug in cc? (was: "translation with --profopt
	fails")
In-Reply-To: <44C1388D.80003@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>	<20060721152939.GB31604@code0.codespeak.net>	<44C0FBCA.1090605@jippii.fi>
	<44C1388D.80003@jippii.fi>
Message-ID: <44C3C339.8070809@jippii.fi>

Elmo M?ntynen wrote:
> Elmo M?ntynen wrote:
>   
>> Armin Rigo wrote:
>>   
>>     
>>> Hi Elmo,
>>>
>>> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>>>   
>>>     
>>>       
>>>> Last lines of output (python translate.py --backend=c --profopt='-c
>>>> "from richards import *;main(iterations=1)"' --text --batch
>>>> targetpypystandalone.py):
>>>>     
>>>>       
>>>>         
>>> I have no clue how --profopt can affect the translation itself.  You
>>> don't get the same error without the --profopt option?
>>>
>>>
>>> A bientot,
>>>
>>> Armin
>>>   
>>>     
>>>       
>> Should've tested that, will do as soon as possible. benchmark.html on
>> snake shows something that says to me that the profopt executable
>> doesn't exist, which suggests that it wasn't built.
>>
>> Elmo
>>   
>>     
> Translation without profopt worked now, though there might have been
> something that got changed in between, will report after some more builds.
>   
Now I got an internal compiler error from cc:
"""
[cbuild:profopt] Gathering profile data from:
/tmp/usession-11/testing_1/testing_1 -c "from richards import
*;main(iterations=1)"
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c testing_1.c -o
testing_1.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c structimpl.c -o
structimpl.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c nonfuncnodes.c
-o nonfuncnodes.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_1.c -o nonfuncnodes_1.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_2.c -o nonfuncnodes_2.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_3.c -o nonfuncnodes_3.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_4.c -o nonfuncnodes_4.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_5.c -o nonfuncnodes_5.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_6.c -o nonfuncnodes_6.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_7.c -o nonfuncnodes_7.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_8.c -o nonfuncnodes_8.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_9.c -o nonfuncnodes_9.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_10.c -o nonfuncnodes_10.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_11.c -o nonfuncnodes_11.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_12.c -o nonfuncnodes_12.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_13.c -o nonfuncnodes_13.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_14.c -o nonfuncnodes_14.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_15.c -o nonfuncnodes_15.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_16.c -o nonfuncnodes_16.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_17.c -o nonfuncnodes_17.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_18.c -o nonfuncnodes_18.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_19.c -o nonfuncnodes_19.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_20.c -o nonfuncnodes_20.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c implement.c -o
implement.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c implement_1.c
-o implement_1.o
debug: entry point starting
debug: argv -> /tmp/usession-11/testing_1/testing_1
debug: argv -> -c "from richards import *;main(iterations=1)"
unrecognized option '-c "from richards import *;main(iterations=1)"'
usage: /tmp/usession-11/testing_1/testing_1 [options]
Try `/tmp/usession-11/testing_1/testing_1 -h` for more information.
implement_1.c: In function ?pypy_g_get_len_of_range?:
implement_1.c:137183: internal compiler error: Floating point exception
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions,
see <URL:file:///usr/share/doc/gcc-4.0/README.Bugs>.

[translation:ERROR] Error:
[translation:ERROR] Traceback (most recent call last):
[translation:ERROR] File "translate.py", line 349, in main
[translation:ERROR] drv.proceed(goals)
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 508, in proceed
[translation:ERROR] return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/taskengine.py",
line 108, in _execute
[translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 197, in _do
[translation:ERROR] res = func()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 333, in task_compile_c
[translation:ERROR] cbuilder.compile()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/genc.py",
line 229, in compile
[translation:ERROR] compiler.build()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/cbuild.py",
line 323, in build
[translation:ERROR] self._build()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/cbuild.py",
line 353, in _build
[translation:ERROR] extra_preargs=self.compile_extra)
[translation:ERROR] File "/usr/lib/python2.4/distutils/ccompiler.py",
line 699, in compile
[translation:ERROR] self._compile(obj, src, ext, cc_args,
extra_postargs, pp_opts)
[translation:ERROR] File
"/usr/lib/python2.4/distutils/unixccompiler.py", line 115, in _compile
[translation:ERROR] raise CompileError, msg
[translation:ERROR] CompileError: command 'cc' failed with exit status 1
"""
If this really calls for a bugreport, it would be better if some one
more comfortable with the compiling process and the gcc toolset would do
it. Should I send some of the produced files?

Elmo

From elmo13 at jippii.fi  Sun Jul 23 22:15:47 2006
From: elmo13 at jippii.fi (=?windows-1252?Q?Elmo_M=E4ntynen?=)
Date: Sun, 23 Jul 2006 23:15:47 +0300
Subject: [pypy-dev] A bug in cc?
In-Reply-To: <44C3C339.8070809@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>	<20060721152939.GB31604@code0.codespeak.net>	<44C0FBCA.1090605@jippii.fi>	<44C1388D.80003@jippii.fi>
	<44C3C339.8070809@jippii.fi>
Message-ID: <44C3D8F3.4080603@jippii.fi>

Elmo M?ntynen wrote:
> Elmo M?ntynen wrote:
>   
>> Elmo M?ntynen wrote:
>>   
>>     
>>> Armin Rigo wrote:
>>>   
>>>     
>>>       
>>>> Hi Elmo,
>>>>
>>>> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>>>>   
>>>>     
>>>>       
>>>>         
>>>>> Last lines of output (python translate.py --backend=c --profopt='-c
>>>>> "from richards import *;main(iterations=1)"' --text --batch
>>>>> targetpypystandalone.py):
>>>>>     
>>>>>       
>>>>>         
>>>>>           
>>>> I have no clue how --profopt can affect the translation itself.  You
>>>> don't get the same error without the --profopt option?
>>>>
>>>>
>>>> A bientot,
>>>>
>>>> Armin
>>>>   
>>>>     
>>>>       
>>>>         
>>> Should've tested that, will do as soon as possible. benchmark.html on
>>> snake shows something that says to me that the profopt executable
>>> doesn't exist, which suggests that it wasn't built.
>>>
>>> Elmo
>>>   
>>>     
>>>       
>> Translation without profopt worked now, though there might have been
>> something that got changed in between, will report after some more builds.
>>   
>>     
> Now I got an internal compiler error from cc:
> """
> [cbuild:profopt] Gathering profile data from:
> /tmp/usession-11/testing_1/testing_1 -c "from richards import
> *;main(iterations=1)"
>
>   
...
> [translation:ERROR] File
> "/usr/lib/python2.4/distutils/unixccompiler.py", line 115, in _compile
> [translation:ERROR] raise CompileError, msg
> [translation:ERROR] CompileError: command 'cc' failed with exit status 1
> """
> If this really calls for a bugreport, it would be better if some one
> more comfortable with the compiling process and the gcc toolset would do
> it. Should I send some of the produced files?
>   
Just to note that building without profopt works (this time its the same
rev).

Elmo

From len-l at telus.net  Mon Jul 24 06:12:41 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Sun, 23 Jul 2006 21:12:41 -0700
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <9eebf5740607231058w2a2b688bpb5c746c01750ad96@mail.gmail.com>
References: <44C355C6.11938.7869AE@len-l.telus.net>
Message-ID: <44C3E649.11747.B7036B@len-l.telus.net>

On 23 Jul 2006 at 19:58, Lawrence Oluyede wrote:

> > I just brought it up as it is one more thing to consider when
> > converting a ctypes Python module to rctypes. But could rctypes use
> > ffi when the callback is a bound method?
> 
> AFAIK one of the reasons of the existence of rctypes and the ext.
> compiler is not to use the ffi library...
> 
Then perhaps the ext. compiler can achieve the same effect as an ffi 
C closure by creating a stub function to call the bound method, which 
is stored in a thread-local variable.

Lenard Lindstrom
<len-l at telus.net>



From aurelien.campeas at logilab.fr  Mon Jul 24 12:21:52 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 24 Jul 2006 12:21:52 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060721203613.GA2825@code0.codespeak.net>
References: <20060721203613.GA2825@code0.codespeak.net>
Message-ID: <20060724102151.GC15904@crater.logilab.fr>

On Fri, Jul 21, 2006 at 10:36:13PM +0200, Armin Rigo wrote:
> Hi all, hi Aurelien,
> 
> The stackless-related tests are all consistently failing nowadays.  Some
> have been failing for 4 days and others for 9 days.  See
> 
>    http://snake.cs.uni-duesseldorf.de/pypytest/summary.html
> 
> Given that you appear to continue to work on it, but without making
> tests pass again, I'm a bit worrying exactly about what you are
> doing?

First, I did some (superficial, nothing related to the translation
aspect itself) changes that allowed me to build a (somewhat
non-crashing) translated pypy-logic using clonable coroutines. These
changes indeed broke some tests, in ways that I don't
understand. Please note that indeed most failing tests in
/modules/_stackless/test are due to run-time failures, after a
successful translation. Michael put in a fix last week which,
strangely, yielded a working version on ... PPC arch only.

IOW, wizards needed for these cases.

I'll have a look at the applevel stuff in /lib/test2 (the 3 tests pass
all independantly but fail when executed in sequence), since I'm more
likely directly responsible for breakage there.

> That doesn't seem too much in line with the "test-driven development"
> approach.
> 

Yes. Otoh, it seems that I'm currently the only consummer of these
features, so I didn't feel the urge to rush to fix them immediately. I
will look at them today.

Regards,
Aur?lien.

From fijal at genesilico.pl  Mon Jul 24 12:35:05 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Mon, 24 Jul 2006 12:35:05 +0200
Subject: [pypy-dev] JS backend updates
Message-ID: <44C4A259.3010309@genesilico.pl>

There were some updates recently about JS backend.

Recently:

- Added some decorator which might be usefull at declaring server-side 
of AJAX communication
- Polished a little bit backend so now it is able to run richards and 
pystone as soon as you cut off zeroing assumption out of ll_alloc_and_set
- Added some documentation
- Bugfixes and some helper functions (ie StringBuilder) so now 
"aaa".upper() works as expected

Coming work:

- Hiding browser differencies
- more examples & docs
- Maybe some stackless stuff?
- Very hard and usefull point: to get out Python-level traceback out of 
exception handlers.
- Some JS optimisations (assuming that JS compilers themselves cannot 
optimise much if anything at all)
- BnB more playable.


From cfbolz at gmx.de  Mon Jul 24 13:51:02 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 24 Jul 2006 13:51:02 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060724102151.GC15904@crater.logilab.fr>
References: <20060721203613.GA2825@code0.codespeak.net>
	<20060724102151.GC15904@crater.logilab.fr>
Message-ID: <44C4B426.60206@gmx.de>

Hi!

Aur?lien Camp?as wrote:
> On Fri, Jul 21, 2006 at 10:36:13PM +0200, Armin Rigo wrote:
>> Hi all, hi Aurelien,
>>
>> The stackless-related tests are all consistently failing nowadays.  Some
>> have been failing for 4 days and others for 9 days.  See
>>
>>    http://snake.cs.uni-duesseldorf.de/pypytest/summary.html
>>
>> Given that you appear to continue to work on it, but without making
>> tests pass again, I'm a bit worrying exactly about what you are
>> doing?
>
> First, I did some (superficial, nothing related to the translation
> aspect itself) changes that allowed me to build a (somewhat
> non-crashing) translated pypy-logic using clonable coroutines. These
> changes indeed broke some tests, in ways that I don't
> understand. Please note that indeed most failing tests in
> /modules/_stackless/test are due to run-time failures, after a
> successful translation. Michael put in a fix last week which,
> strangely, yielded a working version on ... PPC arch only.
>
> IOW, wizards needed for these cases.
>
> I'll have a look at the applevel stuff in /lib/test2 (the 3 tests pass
> all independantly but fail when executed in sequence), since I'm more
> likely directly responsible for breakage there.

It's just not good to have failing tests checked in. If they fail for
good reasons and will probably work soon again, just skip them with an
appropriate message. Otherwise it becomes impossible for somebody
working in a related area to find out whether his changes were
responsible for the breakage.

>> That doesn't seem too much in line with the "test-driven development"
>> approach.
>>
>
> Yes. Otoh, it seems that I'm currently the only consummer of these
> features, so I didn't feel the urge to rush to fix them immediately. I
> will look at them today.

Well, that's not true. People wanted to try out stackless and couldn't
because it didn't translate. I needed to do benchmarks and couldn't
either. It's just a good general policy to have tranlations working all
the time.

Cheers,

Carl Friedrich

From Ben.Young at risk.sungard.com  Mon Jul 24 14:48:57 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 24 Jul 2006 13:48:57 +0100
Subject: [pypy-dev] Example of .NET filehandles
Message-ID: <OF4BE4E08A.626924BA-ON802571B5.00462B19-802571B5.004666AF@risk.sungard.com>

Hi Antonio,

Here's the example I mentioned. I don't think it's particularly safe, so 
it may be better to keep a map of ints to filestreams that you have 
opened, and use that if possible, and fall back to this if you can't.

Cheers,
Ben

using System;
using System.Collections.Generic;
using System.Text;

namespace posix
{
    class Program
    {
        static void Main(string[] args)
        {
            System.IO.FileStream stream1 = new System.IO.FileStream
(args[0], System.IO.FileMode.Create);
            int handle = 
stream1.SafeFileHandle.DangerousGetHandle().ToInt32();

            System.IO.FileStream stream2 = new System.IO.FileStream(new 
Microsoft.Win32.SafeHandles.SafeFileHandle(new IntPtr(handle), true), 
System.IO.FileAccess.Write);
            stream2.Write(new byte[]{(byte)'H', (byte)'e', (byte)'l', (
byte)'l', (byte)'o'}, 0, 5);
            stream2.Close();
        }
    }
}

From arigo at tunes.org  Mon Jul 24 15:33:59 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Jul 2006 15:33:59 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060724102151.GC15904@crater.logilab.fr>
References: <20060721203613.GA2825@code0.codespeak.net>
	<20060724102151.GC15904@crater.logilab.fr>
Message-ID: <20060724133359.GB12961@code0.codespeak.net>

Hi Aurelien,

On Mon, Jul 24, 2006 at 12:21:52PM +0200, Aur?lien Camp?as wrote:
> First, I did some (superficial, nothing related to the translation
> aspect itself) changes that allowed me to build a (somewhat
> non-crashing) translated pypy-logic using clonable coroutines.

I did not look closely, so I cannot comment, but it seems to me that the
tests and the demo we fixed during the post-EuroPython sprint all
translated correctly, so I'm a bit surprized that you had to hack so
much on the existing stackless code only for translatability reasons.

This said, there were not many tests so far.  That's also why I'm not
happy to see half of them fail nowadays.  The "agreed" development
procedure for PyPy is to write more tests when a bug is identified or
existing code doesn't translate in some context, and then make the test
pass without breaking the previous tests (which are all valid, so again
I'm surprized that you can compile anything at all when half of the
basic tests fail).


A bientot,

Armin

From aurelien.campeas at logilab.fr  Mon Jul 24 15:56:27 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 24 Jul 2006 15:56:27 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060724133359.GB12961@code0.codespeak.net>
References: <20060721203613.GA2825@code0.codespeak.net>
	<20060724102151.GC15904@crater.logilab.fr>
	<20060724133359.GB12961@code0.codespeak.net>
Message-ID: <20060724135627.GB23406@crater.logilab.fr>

On Mon, Jul 24, 2006 at 03:33:59PM +0200, Armin Rigo wrote:
> Hi Aurelien,
> 
> On Mon, Jul 24, 2006 at 12:21:52PM +0200, Aur?lien Camp?as wrote:
> > First, I did some (superficial, nothing related to the translation
> > aspect itself) changes that allowed me to build a (somewhat
> > non-crashing) translated pypy-logic using clonable coroutines.
> 
> I did not look closely, so I cannot comment, but it seems to me that the
> tests and the demo we fixed during the post-EuroPython sprint all
> translated correctly, so I'm a bit surprized that you had to hack so
> much on the existing stackless code only for translatability
> reasons.

svn blame currently shows how incredibly little I had to play with the
code (to break it ... and have it fit my needs, and later make it
translatable again).

Apart from some temporary uncautious changes in interp_clonable from
which one can trace back the first breakage, and that were fixed
since, there is something else going on, that happened some time
after. The current breakage is, I think, unrelated to my
misdoings. This fact is just masked by the insufficient granularity of
http://snake.cs.uni-duesseldorf.de/pypytest/summary.html.

I am tracking every change concerning stackless, now.

Then, again, please consider that the current module/_stackless tests
fail at RUNTIME. They translate perfectly well.

> 
> This said, there were not many tests so far.  That's also why I'm not
> happy to see half of them fail nowadays.  The "agreed" development
> procedure for PyPy is to write more tests when a bug is identified or
> existing code doesn't translate in some context, and then make the test
> pass without breaking the previous tests (which are all valid, so again
> I'm surprized that you can compile anything at all when half of the
> basic tests fail).

I am surprised, too. But I can exhibit the binary :)

Regards,
Aur?lien.

From stephan.diehl at gmx.net  Mon Jul 24 20:00:56 2006
From: stephan.diehl at gmx.net (Stephan Diehl)
Date: Mon, 24 Jul 2006 20:00:56 +0200
Subject: [pypy-dev] JS backend updates
In-Reply-To: <44C4A259.3010309@genesilico.pl>
References: <44C4A259.3010309@genesilico.pl>
Message-ID: <44C50AD8.9080004@gmx.net>

Hi Maciek,

Maciek Fijalkowski wrote:
> There were some updates recently about JS backend.
> 
> Recently:
> 
> - Added some decorator which might be usefull at declaring server-side 
> of AJAX communication
> - Polished a little bit backend so now it is able to run richards and 
> pystone as soon as you cut off zeroing assumption out of ll_alloc_and_set
> - Added some documentation
> - Bugfixes and some helper functions (ie StringBuilder) so now 
> "aaa".upper() works as expected

thanks for the added documentation.
In the meantime, I've been busy with my own JS test application (not 
using pypy yet :-(   ). If you have some time on your hand, you could 
check out http://codespeak.net/svn/user/stephan/hacks/webapp
which, if nothing else, contains a working JsonRpc implementation.
Just run 'python examplewebapp.py' and point your browser to localhost:8000.
The only dependency is on 'simplejson' (MochiKit is included)

Anyway, I'd like to see some widget stuff working, so one could maybe 
integrate / expand the pyjamas stuff or write our own.

Cheers

Stephan

> 
> Coming work:
> 
> - Hiding browser differencies
> - more examples & docs
> - Maybe some stackless stuff?
> - Very hard and usefull point: to get out Python-level traceback out of 
> exception handlers.
> - Some JS optimisations (assuming that JS compilers themselves cannot 
> optimise much if anything at all)
> - BnB more playable.
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
> 


From arigo at tunes.org  Mon Jul 24 20:32:35 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Jul 2006 20:32:35 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060724135627.GB23406@crater.logilab.fr>
References: <20060721203613.GA2825@code0.codespeak.net>
	<20060724102151.GC15904@crater.logilab.fr>
	<20060724133359.GB12961@code0.codespeak.net>
	<20060724135627.GB23406@crater.logilab.fr>
Message-ID: <20060724183235.GA11228@code0.codespeak.net>

Hi Aurelien,

On Mon, Jul 24, 2006 at 03:56:27PM +0200, Aur?lien Camp?as wrote:
> svn blame currently shows how incredibly little I had to play with the
> code

Well, that's hardly an argument, is it?  I know that the code is
obscure, and you're welcome to ask questions here and on IRC.  I also
know from experience that making coroutines translatable can trigger
obscure problems.  It doesn't change the basic issue that breaking tests
without worrying about them is not a good way to go forward...

> The current breakage is, I think, unrelated to my
> misdoings. This fact is just masked by the insufficient granularity of
> http://snake.cs.uni-duesseldorf.de/pypytest/summary.html.

Well, at least one of these tests - test_coroutine_reconstruction - has
been breaking in the same way since your rev 30119.  For the other
tests, knowing exactly what is the cause of the current breakage is
difficult, and that's precisely because the tests were already failing.
This prevents us from easily tracking what breaks what.  For now "naive"
tracking tells me that you broke the tests first...

> Then, again, please consider that the current module/_stackless tests
> fail at RUNTIME. They translate perfectly well.

I don't see how that is an argument, either.

As far as I can tell, the only way out of this kind of mess is to revert
to the first revision before the first failure, and try to re-understand
the diff of that initial breaking revision, debug until we see what the
problem was, and then attempt to fix it in the head.  That's certainly
more work than needed because of the revision-juggling involved.

I'd appreciate it if you could tell us if you're going to do that, or if
there is something out of your scope going on.  I'd like this situation
to be resolved.


A bientot,

Armin

From arigo at tunes.org  Mon Jul 24 22:56:10 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Jul 2006 22:56:10 +0200
Subject: [pypy-dev] A bug in cc? (was: "translation with --profopt
	fails")
In-Reply-To: <44C3C339.8070809@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>
	<20060721152939.GB31604@code0.codespeak.net>
	<44C0FBCA.1090605@jippii.fi> <44C1388D.80003@jippii.fi>
	<44C3C339.8070809@jippii.fi>
Message-ID: <20060724205610.GA25017@code0.codespeak.net>

Hi Elmo,

On Sun, Jul 23, 2006 at 09:43:05PM +0300, Elmo M?ntynen wrote:
> Now I got an internal compiler error from cc:

Couldn't reproduce it...  I ran

  python translate.py --backend=c --profopt='-c "from richards import
  *;main(iterations=1)"' --text --batch targetpypystandalone.py

but everything went fine.  It's a gcc 3.4.5 (Gentoo 3.4.5-r1,
ssp-3.4.5-1.0, pie-8.7.9).  Either it's a bug in a specific gcc version,
or else maybe (just guessing) the floating-point exception is when gcc
computes something with the profile data, maybe timings, in which case
the bug could be hard to reproduce on a different machine.

In any case, if the bug is still there in the latest gcc, yes, I'd
consider reporting it.  PyPy is good to push many limits of its backends
-- e.g. it gave quite a few LLVM bug reports and I wouldn't be surpized
if mono was next :-)


A bientot,

Armin

From scott+pypy-dev at scottdial.com  Mon Jul 24 23:33:09 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Mon, 24 Jul 2006 17:33:09 -0400
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for win32
Message-ID: <44C53C95.4070002@scottdial.com>

I attempted to translate a pypy-c in windows today. I had to mask out 
including sys/mman.h. Afterwards, I was able to succesfully translate.

Index: ll_osdefs.h
===================================================================
--- ll_osdefs.h	(revision 30390)
+++ ll_osdefs.h	(working copy)
@@ -42,7 +42,9 @@

  #include <sys/types.h>

+#if !defined(MS_WINDOWS)
  #include <sys/mman.h>
+#endif

  #ifdef HAVE_STROPTS_H
  #include <stropts.h>

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From anto.cuni at gmail.com  Mon Jul 24 23:41:47 2006
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Mon, 24 Jul 2006 23:41:47 +0200
Subject: [pypy-dev] A bug in cc?
In-Reply-To: <20060724205610.GA25017@code0.codespeak.net>
References: <44BE2F9E.4060306@jippii.fi>	<20060721152939.GB31604@code0.codespeak.net>	<44C0FBCA.1090605@jippii.fi>
	<44C1388D.80003@jippii.fi>	<44C3C339.8070809@jippii.fi>
	<20060724205610.GA25017@code0.codespeak.net>
Message-ID: <44C53E9B.3000605@gmail.com>

Armin Rigo wrote:
> In any case, if the bug is still there in the latest gcc, yes, I'd
> consider reporting it.  PyPy is good to push many limits of its backends
> -- e.g. it gave quite a few LLVM bug reports and I wouldn't be surpized
> if mono was next :-)

mono *will be* the next, definitively :-).

ciao Anto

From elmo13 at jippii.fi  Tue Jul 25 00:01:23 2006
From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=)
Date: Tue, 25 Jul 2006 01:01:23 +0300
Subject: [pypy-dev] A bug in cc?
In-Reply-To: <20060724205610.GA25017@code0.codespeak.net>
References: <44BE2F9E.4060306@jippii.fi>
	<20060721152939.GB31604@code0.codespeak.net>
	<44C0FBCA.1090605@jippii.fi> <44C1388D.80003@jippii.fi>
	<44C3C339.8070809@jippii.fi>
	<20060724205610.GA25017@code0.codespeak.net>
Message-ID: <44C54333.5080701@jippii.fi>

Armin Rigo wrote:
> Hi Elmo,
>
> On Sun, Jul 23, 2006 at 09:43:05PM +0300, Elmo M?ntynen wrote:
>   
>> Now I got an internal compiler error from cc:
>>     
>
> Couldn't reproduce it...  I ran
>
>   python translate.py --backend=c --profopt='-c "from richards import
>   *;main(iterations=1)"' --text --batch targetpypystandalone.py
>
> but everything went fine.  It's a gcc 3.4.5 (Gentoo 3.4.5-r1,
> ssp-3.4.5-1.0, pie-8.7.9).  Either it's a bug in a specific gcc version,
> or else maybe (just guessing) the floating-point exception is when gcc
> computes something with the profile data, maybe timings, in which case
> the bug could be hard to reproduce on a different machine.
>
> In any case, if the bug is still there in the latest gcc, yes, I'd
> consider reporting it.  PyPy is good to push many limits of its backends
> -- e.g. it gave quite a few LLVM bug reports and I wouldn't be surpized
> if mono was next :-)
>
>
> A bientot,
>
> Armin
>   
I have 4.0.4 (debian 4.0.3-4). How should we go about testing on the
latest gcc, or are you doing it right now. I can definitely reproduce
the bug with the that certain file(s). Which of the .gcda and .gcno (or
the rest) files should I send?

Elmo

From l.oluyede at gmail.com  Tue Jul 25 06:45:17 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Tue, 25 Jul 2006 06:45:17 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <44C53C95.4070002@scottdial.com>
References: <44C53C95.4070002@scottdial.com>
Message-ID: <9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>

On 7/24/06, Scott Dial <scott+pypy-dev at scottdial.com> wrote:
> I attempted to translate a pypy-c in windows today. I had to mask out
> including sys/mman.h. Afterwards, I was able to succesfully translate.
>
> Index: ll_osdefs.h
> ===================================================================
> --- ll_osdefs.h (revision 30390)
> +++ ll_osdefs.h (working copy)
> @@ -42,7 +42,9 @@
>
>   #include <sys/types.h>
>
> +#if !defined(MS_WINDOWS)
>   #include <sys/mman.h>
> +#endif
>
>   #ifdef HAVE_STROPTS_H
>   #include <stropts.h>

That was my fault. It has been fixed. Thanks

-- 
Lawrence
http://www.oluyede.org/blog

From aurelien.campeas at logilab.fr  Tue Jul 25 11:09:32 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Tue, 25 Jul 2006 11:09:32 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060724183235.GA11228@code0.codespeak.net>
References: <20060721203613.GA2825@code0.codespeak.net>
	<20060724102151.GC15904@crater.logilab.fr>
	<20060724133359.GB12961@code0.codespeak.net>
	<20060724135627.GB23406@crater.logilab.fr>
	<20060724183235.GA11228@code0.codespeak.net>
Message-ID: <20060725090932.GA2908@crater.logilab.fr>

On Mon, Jul 24, 2006 at 08:32:35PM +0200, Armin Rigo wrote:
> Well, that's hardly an argument, is it?  I know that the code is
> obscure, and you're welcome to ask questions here and on IRC.  I also

Sometimes people are on vacation. I apologize for the nuisance.

Given the time to learn enough of subversion, I'll try to replay
versions up to what caused these failures.

From arigo at tunes.org  Tue Jul 25 11:35:26 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 25 Jul 2006 11:35:26 +0200
Subject: [pypy-dev] A bug in cc?
In-Reply-To: <44C54333.5080701@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>
	<20060721152939.GB31604@code0.codespeak.net>
	<44C0FBCA.1090605@jippii.fi> <44C1388D.80003@jippii.fi>
	<44C3C339.8070809@jippii.fi>
	<20060724205610.GA25017@code0.codespeak.net>
	<44C54333.5080701@jippii.fi>
Message-ID: <20060725093526.GA25834@code0.codespeak.net>

Hi Elmo,

On Tue, Jul 25, 2006 at 01:01:23AM +0300, Elmo M?ntynen wrote:
> I have 4.0.4 (debian 4.0.3-4). How should we go about testing on the
> latest gcc, or are you doing it right now. I can definitely reproduce
> the bug with the that certain file(s). Which of the .gcda and .gcno (or
> the rest) files should I send?

I don't know what these files are, so I can't comment, but you should
look or ask around in the gcc community instead.  Also, I'm sure they
would like to see information related to their latest release, so you
should try with that.  And of course you should try to see if it's a
known bug, maybe even if it's been fixed already...


A bientot,

Armin

From arigo at tunes.org  Tue Jul 25 11:38:45 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 25 Jul 2006 11:38:45 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
Message-ID: <20060725093843.GA26167@code0.codespeak.net>

Hi all,

On Tue, Jul 25, 2006 at 06:45:17AM +0200, Lawrence Oluyede wrote:
> > I attempted to translate a pypy-c in windows today. I had to mask out
> > including sys/mman.h. Afterwards, I was able to succesfully translate.
> 
> That was my fault. It has been fixed. Thanks

Now that we have automated test runs and translations on Linux machines,
we could try to think how we could go about doing the same on Windows
machines.  Very few of us actually have Windows machines around, so it
would be important, otherwise things will break more and more (it's
already a good surprize that a PyPy translation still works after just a
quick fix :-)

"""What is not tested is broken."""


A bientot,

Armin

From stephan.diehl at gmx.net  Tue Jul 25 11:55:36 2006
From: stephan.diehl at gmx.net (Stephan Diehl)
Date: Tue, 25 Jul 2006 11:55:36 +0200
Subject: [pypy-dev] pypy development tasks
Message-ID: <44C5EA98.5070700@gmx.net>

Hi all,

starting next week, I'll be able again to continue working on pypy.
As a part-part-part time worker, I find it a bit difficult to stay on 
top on what's going on (and what needs to be done).

These are the things, I would continue on, if nobody has a better idea:

- write a python module that exposes Coroutine interface (using
   greenlets, I suppose). This would be usefull for further development
   of applevel stackless module. As a sideeffect, the stackless module
   could then be used with CPython.
- continue work on applevel stackless module and documentation.

Other than that, if I should rather do things like adding docstrings or 
do other usefull tasks, don't hesitate to tell me :-)

Cheers

Stephan

From scott+pypy-dev at scottdial.com  Tue Jul 25 13:44:45 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Tue, 25 Jul 2006 07:44:45 -0400
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h
	for	win32
In-Reply-To: <20060725093843.GA26167@code0.codespeak.net>
References: <44C53C95.4070002@scottdial.com>	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
Message-ID: <44C6042D.7090806@scottdial.com>

Armin Rigo wrote:
> Now that we have automated test runs and translations on Linux machines,
> we could try to think how we could go about doing the same on Windows
> machines.  Very few of us actually have Windows machines around, so it
> would be important, otherwise things will break more and more (it's
> already a good surprize that a PyPy translation still works after just a
> quick fix :-)
> 

I actually was pretty suprised too.. Presuming this is an important 
platform to test (*grin*), I can provide a daily testing run for 
Windows. I don't know what the code behind your pypytest is, but I 
presume I could run something similar? Let me know.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From l.oluyede at gmail.com  Tue Jul 25 20:51:09 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Tue, 25 Jul 2006 20:51:09 +0200
Subject: [pypy-dev] extension compiler and Python special methods
Message-ID: <9eebf5740607251151y29f5341l43bc92fe28ff592e@mail.gmail.com>

As u can see in rev. #30543 I commited a series of methods to complete
the mmap module. Those are the special methods (__xyz__) implemented
as regular ones. I thought it was best to have the functionality in
anyway. When and if we'll have that support it will be only a matter
of change methods names to gain compatibility with CPython's code.

If you disagree with me let me know, I'll comment out the code or whatever.

-- 
Lawrence
http://www.oluyede.org/blog

From arigo at tunes.org  Wed Jul 26 11:18:25 2006
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 26 Jul 2006 11:18:25 +0200
Subject: [pypy-dev] about bz2 and more
In-Reply-To: <9eebf5740607260150x3be8d4ta9ee5497886d5c87@mail.gmail.com>
References: <9eebf5740607260150x3be8d4ta9ee5497886d5c87@mail.gmail.com>
Message-ID: <20060726091824.GA26783@code0.codespeak.net>

Hi Lawrence,

On Wed, Jul 26, 2006 at 10:50:40AM +0200, Lawrence Oluyede wrote:
> Bz2 is kinda a beast module doing heavy operations with function
> pointers, buffering and so on. The problem I encountered lies in the
> initialization of the bz2 file object. It has support for buffering,
> universal newlines and more. All this stuff is exposed in CPython via
> file()/open() factory functions. Those are not supported at RPython
> level and I can't write the class at app-level.

I think the first thing do to is support a "raw" interface, that gives
access to unbuffered binary data.  Then for PyPy we can simply plug the
existing buffering and universal-newline code, which is already written
at app-level in pypy/lib/_sio.py.

It means that if you write a class with the correct interface, and
expose it to app-level in a mixed-module, then at least in PyPy you can
write the rest at app-level: in bz2.open(), instantiate the "raw" class
from the interp-level part of the module, then import the _sio module
and build the correct stack of buffering and translating classes on top
of your "raw" class.

See _file.py, _inithelper(), for how to build such a stack from standard
mode/buffering arguments.  Maybe abstracting out some of this code in a
helper function in _sio or _file would be useful.

An example of "raw" class is _sio.DiskFile, which uses os.read()
internally.  For bz2 you'd build a class with the same interface, based
on ctypes calls instead.

IMHO this all shows that writing buffering and universal newlines
explicitly and at app-level is a great idea :-) Essentially, with almost
no more effort we could provide users with a PyPy-specific module that
plugs buffering and translating on top of any "raw" reader/writer that
she provides.


A bientot,

Armin

From Ben.Young at risk.sungard.com  Wed Jul 26 13:34:29 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 26 Jul 2006 12:34:29 +0100
Subject: [pypy-dev] [pypy-svn] r30565 - in pypy/dist/pypy/tool/build: .
	bin builds test
In-Reply-To: <20060726111851.DD6C6100B8@code0.codespeak.net>
Message-ID: <OFA83F9FBD.75F9FF4B-ON802571B7.003F7C6B-802571B7.003F9419@risk.sungard.com>

Hi Guido,

Cool tool! Is this something I would be able to run just in the 
evenings/weekends etc? Will it pick up that fact that I don't have boehm 
automatically?

Cheers,
Ben


pypy-svn-bounces at codespeak.net wrote on 26/07/2006 12:18:51:

> Author: guido
> Date: Wed Jul 26 13:18:25 2006
> New Revision: 30565
> 
> Added:
>    pypy/dist/pypy/tool/build/   (props changed)
>    pypy/dist/pypy/tool/build/README.txt   (contents, props changed)
>    pypy/dist/pypy/tool/build/__init__.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/bin/   (props changed)
>    pypy/dist/pypy/tool/build/bin/client   (contents, props changed)
>    pypy/dist/pypy/tool/build/bin/path.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/bin/server   (contents, props changed)
>    pypy/dist/pypy/tool/build/bin/startcompile   (contents, props 
changed)
>    pypy/dist/pypy/tool/build/builds/
>    pypy/dist/pypy/tool/build/client.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/config.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/conftest.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/execnetconference.py   (contents, props 
changed)
>    pypy/dist/pypy/tool/build/server.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/test/   (props changed)
>    pypy/dist/pypy/tool/build/test/fake.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/test/path.py   (contents, props changed)
>    pypy/dist/pypy/tool/build/test/test.zip   (contents, props changed)
>    pypy/dist/pypy/tool/build/test/test_client.py   (contents, props 
changed)
>    pypy/dist/pypy/tool/build/test/test_pypybuilder.py   (contents, 
> props changed)
>    pypy/dist/pypy/tool/build/test/test_request_storage.py 
> (contents, props changed)
>    pypy/dist/pypy/tool/build/test/test_server.py   (contents, props 
changed)
> Log:
> Added 'pypybuilder', a tool to build a 'build farm' from 
> participating clients,
> the clients register to a server with information about what they can 
build,
> then the server waits for build requests and dispatches to clients. 
Clients
> send back a build when they're done, on which the server sends out 
emails to
> whoever is waiting for the build. When a build is already available, the
> requestor is provided with a path (will be URL in the future) to the 
build,
> if no client is available for a certain request the request is queued 
until
> there is one.
> Worked on this from https://merlinux.de/svn/user/guido/pypybuilder 
before
> checking in here.
> 
> 
> Added: pypy/dist/pypy/tool/build/README.txt
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/README.txt   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,59 @@
> +============
> +PyPyBuilder
> +============
> +
> +What is this?
> +=============
> +
> +PyPyBuilder is an application that allows people to build PyPy 
instances on
> +demand. If you have a nice idle machine connected to the Internet, and 
don't
> +mind us 'borrowing' it every once in a while, you can start up the 
client
> +script (in bin/client) and have the server send compile jobs to your 
machine.
> +If someone requests a build of PyPy that is not already available on 
the PyPy
> +website, and your machine is capable of making such a build, the 
> server may ask
> +your machine to create it. If enough people participate, with diverse 
enough
> +machines, an ad-hoc 'build farm' is created this way.
> +
> +Components
> +==========
> +
> +The application consists of 3 main components: a server component, 
> a client and
> +a small component to start compile jobs, which we'll call 
'startcompile' for
> +now. 
> +
> +The server waits for clients to register, and for compile job requests. 
When
> +clients register, they pass the server information about what 
> compilations they
> +can handle (system info). Then when the 'startcompile' component 
requests a
> +compilation job, the server first checks whether a binary is 
> already available,
> +and if so returns that. 
> +
> +If there isn't one, the server walks through a list of connected 
> clients to see
> +if there is one that can handle the job, and if so tells it to perform 
it. If
> +there's no client to handle the job, it gets queued until there is.
> +
> +Once a build is available, the server will send an email to all 
> email addresses
> +(it could be more than one person asked for the same build at the same 
time!)
> +passed to it by 'startcompile'.
> +
> +Installation
> +============
> +
> +Client
> +------
> +
> +Installing the system should not be required: just run '.
> /bin/client' to start
> +the client. Note that it depends on the `py lib`_.
> +
> +Server
> +------
> +
> +Also for the server there's no real setup required, and again there's a 

> +dependency on the `py lib`_.
> +
> +.. _`py lib`: http://codespeak.net/py
> +
> +More info
> +=========
> +
> +For more information, bug reports, patches, etc., please send an email 
to 
> +guido at merlinux.de.
> 
> Added: pypy/dist/pypy/tool/build/__init__.py
> 
==============================================================================
> 
> Added: pypy/dist/pypy/tool/build/bin/client
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/client   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,58 @@
> +#!/usr/bin/python
> +
> +BUFSIZE = 1024
> +
> +import path
> +import sys
> +import random
> +from pypy.tool.build import config
> +
> +# XXX using random values for testing
> +modules = ['_stackless', '_socket']
> +
> +"""
> +random.shuffle(modules)
> +sysinfo = {
> +    'maxint': random.choice((sys.maxint, (2 ** 63 - 1))),
> +    'use_modules': modules[:random.randrange(len(modules) + 1)],
> +    'byteorder': random.choice(('little', 'big')),
> +}
> +"""
> +
> +sysinfo = {
> +    'maxint': sys.maxint,
> +    'use_modules': ['_stackless', '_socket'],
> +    'byteorder': sys.byteorder,
> +}
> +
> +if __name__ == '__main__':
> +    from py.execnet import SshGateway
> +    from pypy.tool.build.client import init
> +    gw = SshGateway(config.server)
> +    channel = init(gw, sysinfo, path=config.path, port=config.port)
> +    print channel.receive() # welcome message
> +    try:
> +        while 1:
> +            data = channel.receive()
> +            if not isinstance(data, dict): # needs more checks here
> +                raise ValueError(
> +                    'received wrong unexpected data of type %s' % 
> (type(data),)
> +                )
> +            info = data
> +            # XXX we should compile here, using data dict for info
> +            print 'compilation requested for info %r, now faking 
> that' % (info,)
> +            import time; time.sleep(10)
> +
> +            # write the zip to the server in chunks to server
> +            # XXX we're still faking this
> +            zipfp = (path.packagedir / 'test/test.zip').open()
> +            while True:
> +                chunk = zipfp.read(BUFSIZE)
> +                if not chunk:
> +                    break
> +                channel.send(chunk)
> +            channel.send(None) # tell the server we're done
> +            print 'done with compilation, waiting for next'
> +    finally:
> +        channel.close()
> +        gw.exit()
> 
> Added: pypy/dist/pypy/tool/build/bin/path.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/path.py   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,5 @@
> +import py
> +
> +packagedir = py.magic.autopath().dirpath().dirpath()
> +rootpath = packagedir.dirpath().dirpath().dirpath()
> +py.std.sys.path.append(str(rootpath))
> 
> Added: pypy/dist/pypy/tool/build/bin/server
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/server   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,27 @@
> +#!/usr/bin/python
> +
> +import path
> +from pypy.tool.build import config
> +
> +from py.execnet import SshGateway
> +
> +if __name__ == '__main__':
> +    from py.execnet import SshGateway
> +    from pypy.tool.build.server import init
> +
> +    gw = SshGateway(config.server)
> +    channel = init(gw, port=config.port, path=config.path, 
> +                    projectname=config.projectname,
> +                    buildpath=str(config.buildpath),
> +                    mailhost=config.mailhost,
> +                    mailport=config.mailport,
> +                    mailfrom=config.mailfrom)
> +
> +    try:
> +        while 1:
> +            data = channel.receive()
> +            assert isinstance(data, str)
> +            print data
> +    finally:
> +        channel.close()
> +        gw.exit()
> 
> Added: pypy/dist/pypy/tool/build/bin/startcompile
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/bin/startcompile   Wed Jul 26 13:18:25 
2006
> @@ -0,0 +1,57 @@
> +#!/usr/bin/python
> +
> +import path
> +import sys
> +import random
> +from pypy.tool.build import config
> +
> +initcode = """
> +    import sys
> +    sys.path += %r
> + 
> +    from pypy.tool.build import ppbserver
> +    channel.send(ppbserver.compile(%r, %r))
> +    channel.close()
> +"""
> +def init(gw, sysinfo, email, port=12321):
> +    from pypy.tool.build import execnetconference
> +
> +    conference = execnetconference.conference(gw, port, False)
> +    channel = conference.remote_exec(initcode % (config.path, 
> email, sysinfo))
> +    return channel
> +
> +if __name__ == '__main__':
> +    from py.execnet import SshGateway
> +
> +    from optparse import OptionParser
> +    optparser = OptionParser('%prog [options] email')
> +    for args, kwargs in config.options:
> +        optparser.add_option(*args, **kwargs)
> +    optparser.add_option('-r', '--revision', dest='revision', 
> default='trunk',
> +                        help='SVN revision (defaults to "trunk")')
> + 
> +    (options, args) = optparser.parse_args()
> +
> +    if not args or len(args) != 1:
> +        optparser.error('please provide an email address')
> +
> +    sysinfo = dict([(attr, getattr(options, attr)) for attr in 
> dir(options) if
> +                        not attr.startswith('_') and 
> +                        not callable(getattr(options, attr))])
> + 
> +    print 'going to start compile job with info:'
> +    for k, v in sysinfo.items():
> +        print '%s: %r' % (k, v)
> +    print
> +
> +    gw = SshGateway(config.server)
> +    channel = init(gw, sysinfo, args[0], port=config.port)
> +    ispath, data = channel.receive()
> +    if ispath:
> +        print ('a suitable result is already available, you can find it 
'
> +                'at "%s"' % (data,))
> +    else:
> +        print data
> +        print 'you will be mailed once it\'s ready'
> +    channel.close()
> +    gw.exit()
> 
> Added: pypy/dist/pypy/tool/build/client.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/client.py   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,71 @@
> +import time
> +import thread
> +
> +class PPBClient(object):
> +    def __init__(self, channel, sysinfo, testing=False):
> +        self.channel = channel
> +        self.sysinfo = sysinfo
> +        self.busy_on = None
> +        self.testing = testing
> +
> +        from pypy.tool.build import ppbserver
> +        self.server = ppbserver
> +        self.server.register(self)
> + 
> +    def sit_and_wait(self):
> +        """connect to the host and wait for commands"""
> +        self.channel.waitclose()
> +        self.channel.close()
> +
> +    def compile(self, info):
> +        """send a compile job to the client side
> +
> +            this waits until the client is done, and assumes the client 
sends
> +            back the whole binary as a single string (XXX this 
> should change ;)
> +        """
> +        self.busy_on = info
> +        self.channel.send(info)
> +        thread.start_new_thread(self.wait_until_done, (info,))
> +
> +    def wait_until_done(self, info):
> +        buildpath = self.server.get_new_buildpath(info)
> + 
> +        fp = buildpath.zipfile.open('w')
> +        if not self.testing:
> +            try:
> +                while True:
> +                    try:
> +                        chunk = self.channel.receive()
> +                    except EOFError:
> +                        # stop compilation, client has disconnected
> +                        return 
> +                    if chunk is None:
> +                        break
> +                    fp.write(chunk)
> +            finally:
> +                fp.close()
> + 
> +        self.server.compilation_done(info, buildpath)
> +        self.busy_on = None
> +
> +initcode = """
> +    import sys
> +    sys.path += %r
> + 
> +    from pypy.tool.build.client import PPBClient
> +
> +    try:
> +        client = PPBClient(channel, %r, %r)
> +        client.sit_and_wait()
> +    finally:
> +        channel.close()
> +"""
> +def init(gw, sysinfo, path=None, port=12321, testing=False):
> +    from pypy.tool.build import execnetconference
> + 
> +    if path is None:
> +        path = []
> +
> +    conference = execnetconference.conference(gw, port, False)
> +    channel = conference.remote_exec(initcode % (path, sysinfo, 
testing))
> +    return channel
> 
> Added: pypy/dist/pypy/tool/build/config.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/config.py   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,45 @@
> +import py
> +
> +# general settings, used by both server and client
> +server = 'johnnydebris.net'
> +port = 12321
> +path = ['/home/johnny/temp/pypy-dist'] 
> +
> +# option definitions for the startcompile script
> +# for now we have them here, we should probably use pypy's config 
instead 
> +# though...
> +import sys
> +def _use_modules_callback(option, opt_str, value, parser):
> +    parser.values.use_modules = [m.strip() for m in value.split(',') 
> +                                    if m.strip()]
> +
> +def _maxint_callback(option, opt_str, value, parser):
> +    parser.values.maxint = 2 ** (int(value) - 1) - 1
> +
> +options = [
> +    (('-m', '--use-modules'), {'action': 'callback', 'type': 'string',
> +                                'callback': _use_modules_callback,
> +                                'dest': 'use_modules', 'default': [],
> +                                'help': 'select the modules you 
> want to use'}),
> +    (('-i', '--maxint'), {'action': 'callback', 'callback': 
_maxint_callback,
> +                                'default': sys.maxint, 'dest': 
'maxint',
> +                                'type': 'string',
> +                                'help': ('size of an int in bits 
(32/64, '
> +                                            'defaults to 
sys.maxint)')}),
> +    (('-b', '--byteorder'), {'action': 'store', 
> +                                'dest': 'byteorder', 'default': 
> sys.byteorder,
> +                                'nargs': 1,
> +                                'help': ('byte order (little/big, 
defaults '
> +                                            'to sys.byteorder)')}),
> +]
> +
> +# settings for the server
> +projectname = 'pypy'
> +buildpath = '/home/johnny/temp/pypy-dist/pypy/tool/build/builds'
> +mailhost = '127.0.0.1'
> +mailport = 25
> +mailfrom = 'johnny at johnnydebris.net'
> +
> +# settings for the tests
> +testpath = [str(py.magic.autopath().dirpath().dirpath())] 
> +
> 
> Added: pypy/dist/pypy/tool/build/conftest.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/conftest.py   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,20 @@
> +import py
> +from py.__.documentation.conftest import Directory as Dir, DoctestText, 
\
> +                                            ReSTChecker
> +mypath = py.magic.autopath().dirpath()
> +
> +Option = py.test.Config.Option 
> +option = py.test.Config.addoptions("pypybuilder test options", 
> +        Option('', '--functional',
> +               action="store_true", dest="functional", default=False,
> +               help="run pypybuilder functional tests"
> +        ),
> +) 
> +
> +py.test.pypybuilder_option = option
> +
> +class Directory(Dir):
> +    def run(self):
> +        if self.fspath == mypath:
> +            return ['README.txt', 'test']
> +        return super(Directory, self).run()
> 
> Added: pypy/dist/pypy/tool/build/execnetconference.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/execnetconference.py   Wed Jul 26 13:18:25 
2006
> @@ -0,0 +1,126 @@
> +"""
> +An extension to py.execnet to allow multiple programs to exchange 
information
> +via a common server.  The idea is that all programs first open a 
gateway to
> +the same server (e.g. an SshGateway), and then call the conference() 
function
> +with a local TCP port number.  The first program must pass 
is_server=True and
> +the next ones is_server=False: the first program's remote gateway is 
used as
> +shared server for the next ones.
> +
> +For all programs, the conference() call returns a new gateway that is
> +connected to the Python process of this shared server.  Information can
> +be exchanged by passing data around within this Python process.
> +"""
> +import py
> +from py.__.execnet.register import InstallableGateway
> +
> +
> +def conference(gateway, port, is_server='auto'):
> +    if is_server:    # True or 'auto'
> +        channel = gateway.remote_exec(r"""
> +            import thread
> +            from socket import *
> +            s = socket(AF_INET, SOCK_STREAM)
> +            port = channel.receive()
> +            try:
> +                s.bind(('', port))
> +                s.listen(5)
> +            except error:
> +                channel.send(0)
> +            else:
> +                channel.send(1)
> +
> +                def readall(s, n):
> +                    result = ''
> +                    while len(result) < n:
> +                        t = s.read(n-len(result))
> +                        if not t:
> +                            raise EOFError
> +                        result += t
> +                    return result
> +
> +                def handle_connexion(clientsock, address):
> +                    clientfile = clientsock.makefile('r+b',0)
> +                    source = clientfile.readline().rstrip()
> +                    clientfile.close()
> +                    g = {'clientsock' : clientsock, 'address' : 
address}
> +                    source = eval(source)
> +                    if source:
> +                        g = {'clientsock' : clientsock, 'address' : 
address}
> +                        co = compile(source+'\n', source, 'exec')
> +                        exec co in g
> +
> +                while True:
> +                    conn, addr = s.accept()
> +                    if addr[0] == '127.0.0.1':   # else connexion 
refused
> +                        thread.start_new_thread(handle_connexion, 
> (conn, addr))
> +                    del conn
> +        """)
> +        channel.send(port)
> +        ok = channel.receive()
> +        if ok:
> +            return gateway
> +        if is_server == 'auto':
> +            pass   # fall-through and try as a client
> +        else:
> +            raise IOError("cannot listen on port %d (already in 
> use?)" % port)
> +
> +    if 1:   # client
> +        channel = gateway.remote_exec(r"""
> +            import thread
> +            from socket import *
> +            s = socket(AF_INET, SOCK_STREAM)
> +            port = channel.receive()
> +            s.connect(('', port))
> +            channel.send(1)
> +            def receiver(s, channel):
> +                while True:
> +                    data = s.recv(4096)
> +                    #print >> open('LOG','a'), 'backward', repr(data)
> +                    channel.send(data)
> +                    if not data: break
> +            thread.start_new_thread(receiver, (s, channel))
> +            try:
> +                for data in channel:
> +                    #print >> open('LOG','a'), 'forward', repr(data)
> +                    s.sendall(data)
> +            finally:
> +                s.shutdown(1)
> +        """)
> +        channel.send(port)
> +        ok = channel.receive()
> +        assert ok
> +        return InstallableGateway(ConferenceChannelIO(channel))
> +
> +
> +class ConferenceChannelIO:
> +    server_stmt = """
> +io = SocketIO(clientsock)
> +"""
> +
> +    error = (EOFError,)
> +
> +    def __init__(self, channel):
> +        self.channel = channel
> +        self.buffer = ''
> +
> +    def read(self, numbytes):
> +        #print >> open('LOG', 'a'), 'read %d bytes' % numbytes
> +        while len(self.buffer) < numbytes:
> +            t = self.channel.receive()
> +            if not t:
> +                #print >> open('LOG', 'a'), 'EOFError'
> +                raise EOFError
> +            self.buffer += t
> +        buf, self.buffer = self.buffer[:numbytes], 
self.buffer[numbytes:]
> +        #print >> open('LOG', 'a'), '--->', repr(buf)
> +        return buf
> +
> +    def write(self, data):
> +        #print >> open('LOG', 'a'), 'write(%r)' % (data,)
> +        self.channel.send(data)
> +
> +    def close_read(self):
> +        pass
> +
> +    def close_write(self):
> +        self.channel.close()
> 
> Added: pypy/dist/pypy/tool/build/server.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/server.py   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,332 @@
> +import random
> +import time
> +import thread
> +import smtplib
> +import py
> +
> +def issubdict(d1, d2):
> +    """sees whether a dict is a 'subset' of another dict
> + 
> +        dictvalues can be immutable data types and list and dicts of 
> +        immutable data types and lists and ... (recursive)
> +    """
> +    for k, v in d1.iteritems():
> +        if not k in d2:
> +            return False
> +        d2v = d2[k]
> +        if isinstance(v, dict):
> +            if not issubdict(v, d2v):
> +                return False
> +        elif isinstance(v, list):
> +            if not set(v).issubset(set(d2v)):
> +                return False
> +        elif v != d2v:
> +            return False
> +    return True
> +
> +# XXX note that all this should be made thread-safe at some point 
(meaning it
> +# currently isn't)!!
> +
> +class RequestStorage(object):
> +    """simple registry that manages information"""
> +    def __init__(self, info_to_path=[]):
> +        self._id_to_info = {} # id -> info dict
> +        self._id_to_emails = {} # id -> requestor email address
> +        self._id_to_path = {} # id -> filepath
> +
> +        self._last_id = 0
> +        self._id_lock = thread.allocate_lock()
> +
> +        self._build_initial(info_to_path)
> +
> +    def request(self, email, info):
> +        """place a request
> +
> +            this either returns a path to the binary (if it's available 

> +            already) or an id for the info
> +        """
> +        self._normalize(info)
> +        infoid = self.get_info_id(info)
> +        path = self._id_to_path.get(infoid)
> +        if path is not None:
> +            return path
> +        self._id_to_emails.setdefault(infoid, []).append(email)
> + 
> +    def get_info_id(self, info):
> +        """retrieve or create an id for an info dict"""
> +        self._id_lock.acquire()
> +        try:
> +            self._normalize(info)
> +            for k, v in self._id_to_info.iteritems():
> +                if v == info:
> +                    return k
> +            self._last_id += 1
> +            id = self._last_id
> +            self._id_to_info[id] = info
> +            return id
> +        finally:
> +            self._id_lock.release()
> +
> +    def add_build(self, info, path):
> +        """store the data for a build and make it available
> +
> +            returns a list of email addresses for the people that 
should be
> +            warned
> +        """
> +        self._normalize(info)
> +        infoid = self.get_info_id(info)
> +        emails = self._id_to_emails.pop(infoid)
> +        self._id_to_path[infoid] = path
> +        return emails
> +
> +    def _build_initial(self, info_to_path):
> +        """fill the dicts with info about files that are already 
built"""
> +        for info, path in info_to_path:
> +            id = self.get_info_id(info)
> +            self._id_to_path[id] = path
> +
> +    def _normalize(self, info):
> +        for k, v in info.iteritems():
> +            if isinstance(v, list):
> +                v.sort()
> +
> +from py.__.path.local.local import LocalPath
> +class BuildPath(LocalPath):
> +    def _info(self):
> +        info = getattr(self, '_info_value', {})
> +        if info:
> +            return info
> +        infopath = self / 'info.txt'
> +        if not infopath.check():
> +            return {}
> +        for line in infopath.readlines():
> +            line = line.strip()
> +            if not line:
> +                continue
> +            chunks = line.split(':')
> +            key = chunks.pop(0)
> +            value = ':'.join(chunks)
> +            info[key] = eval(value)
> +        self._info_value = info
> +        return info
> +
> +    def _set_info(self, info):
> +        self._info_value = info
> +        infopath = self / 'info.txt'
> +        infopath.ensure()
> +        fp = infopath.open('w')
> +        try:
> +            for key, value in info.iteritems():
> +                fp.write('%s: %r\n' % (key, value))
> +        finally:
> +            fp.close()
> + 
> +    info = property(_info, _set_info)
> +
> +    def _zipfile(self):
> +        return py.path.local(self / 'data.zip')
> +
> +    def _set_zipfile(self, iterable):
> +        # XXX not in use right now...
> +        fp = self._zipfile().open('w')
> +        try:
> +            for chunk in iterable:
> +                fp.write(chunk)
> +        finally:
> +            fp.close()
> +
> +    zipfile = property(_zipfile, _set_zipfile)
> +
> +class PPBServer(object):
> +    retry_interval = 10
> + 
> +    def __init__(self, projname, channel, builddir, mailhost=None, 
> +                    mailport=None, mailfrom=None):
> +        self._projname = projname
> +        self._channel = channel
> +        self._builddir = builddir
> +        self._mailhost = mailhost
> +        self._mailport = mailport
> +        self._mailfrom = mailfrom
> + 
> +        self._buildpath = py.path.local(builddir)
> +        self._clients = []
> +        info_to_path = [(p.info, str(p)) for p in 
> +                        self._get_buildpaths(builddir)]
> +        self._requeststorage = RequestStorage(info_to_path)
> +        self._queued = []
> +
> +        self._queuelock = thread.allocate_lock()
> +        self._namelock = thread.allocate_lock()
> + 
> +    def register(self, client):
> +        self._clients.append(client)
> +        self._channel.send('registered %s with info %r' % (
> +                            client, client.sysinfo))
> +        client.channel.send('welcome')
> +
> +    def compile(self, requester_email, info):
> +        """start a compilation
> +
> +            returns a tuple (ispath, data)
> +
> +            if there's already a build available for info, this will 
return
> +            a tuple (True, path), if not, this will return (False, 
message),
> +            where message describes what is happening with the request 
(is
> +            a build made rightaway, or is there no client available?)
> +
> +            in any case, if the first item of the tuple returned is 
False,
> +            an email will be sent once the build is available
> +        """
> +        path = self._requeststorage.request(requester_email, info)
> +        if path is not None:
> +            self._channel.send('already a build for this info 
available')
> +            return (True, path)
> +        for client in self._clients:
> +            if client.busy_on == info:
> +                self._channel.send('build for %r currently in progress' 
% 
> +                                    (info,))
> +                return (False, 'this build is already in progress')
> +        # we don't have a build for this yet, find a client to compile 
it
> +        if self.run(info):
> +            return (False, 'found a suitable client, going to build')
> +        else:
> +            self._queuelock.acquire()
> +            try:
> +                self._queued.append(info)
> +            finally:
> +                self._queuelock.release()
> +            return (False, 'no suitable client found; your request 
> is queued')
> + 
> +    def run(self, info):
> +        """find a suitable client and run the job if possible"""
> +        # XXX shuffle should be replaced by something smarter obviously 
;)
> +        clients = self._clients[:]
> +        random.shuffle(clients)
> +        rev = info.pop('revision', 'trunk')
> +        for client in clients:
> +            # popping out revision here, going to add later... the 
client 
> +            # should be able to retrieve source code for any revision 
(so
> +            # it doesn't need to match a revision field in 
client.sysinfo)
> +            if client.busy_on or not issubdict(info, client.sysinfo):
> +                continue
> +            else:
> +                info['revision'] = rev
> +                self._channel.send(
> +                    'going to send compile job with info %r to %s' % (
> +                        info, client
> +                    )
> +                )
> +                client.compile(info)
> +                return True
> +        info['revision'] = rev
> +        self._channel.send(
> +            'no suitable client available for compilation with info %r' 
% (
> +                info,
> +            )
> +        )
> +
> +    def serve_forever(self):
> +        """this keeps the script from dying, and re-tries jobs"""
> +        self._channel.send('going to serve')
> +        while 1:
> +            time.sleep(self.retry_interval)
> +            self._cleanup_clients()
> +            self._try_queued()
> +
> +    def get_new_buildpath(self, info):
> +        path = BuildPath(str(self._buildpath / 
self._create_filename()))
> +        path.info = info
> +        return path
> +
> +    def compilation_done(self, info, path):
> +        """client is done with compiling and sends data"""
> +        self._channel.send('compilation done for %r, written to %s' % (
> +                                                                info, 
path))
> +        emails = self._requeststorage.add_build(info, path)
> +        for emailaddr in emails:
> +            self._send_email(emailaddr, info, path)
> +
> +    def _cleanup_clients(self):
> +        self._queuelock.acquire()
> +        try:
> +            clients = self._clients[:]
> +            for client in clients:
> +                if client.channel.isclosed():
> +                    if client.busy_on:
> +                        self._queued.append(client.busy_on)
> +                    self._clients.remove(client)
> +        finally:
> +            self._queuelock.release()
> +
> +    def _try_queued(self):
> +        self._queuelock.acquire()
> +        try:
> +            toremove = []
> +            for info in self._queued:
> +                if self.run(info):
> +                    toremove.append(info)
> +            for info in toremove:
> +                self._queued.remove(info)
> +        finally:
> +            self._queuelock.release()
> +
> +    def _get_buildpaths(self, dirpath):
> +        for p in py.path.local(dirpath).listdir():
> +            yield BuildPath(str(p))
> +
> +    _i = 0
> +    def _create_filename(self):
> +        self._namelock.acquire()
> +        try:
> +            today = time.strftime('%Y%m%d')
> +            buildnames = [p.basename for p in 
> +                            py.path.local(self._buildpath).listdir()]
> +            while True:
> +                name = '%s-%s-%s' % (self._projname, today, self._i)
> +                self._i += 1
> +                if name not in buildnames:
> +                    return name
> +        finally:
> +            self._namelock.release()
> +
> +    def _send_email(self, addr, info, path):
> +        self._channel.send('going to send email to %s' % (addr,))
> +        if self._mailhost is not None:
> +            msg = '\r\n'.join([
> +                'From: %s' % (self._mailfrom,),
> +                'To: %s' % (addr,),
> +                'Subject: %s compilation done' % (self._projname,),
> +                '',
> +                'The compilation you requested is done. You can find it 
at',
> +                str(path),
> +            ])
> +            server = smtplib.SMTP(self._mailhost, self._mailport)
> +            server.set_debuglevel(0)
> +            server.sendmail(self._mailfrom, addr, msg)
> +            server.quit()
> +
> +initcode = """
> +    import sys
> +    sys.path += %r
> +
> +    try:
> +        from pypy.tool.build.server import PPBServer
> +        server = PPBServer(%r, channel, %r, %r, %r, %r)
> +
> +        # make the server available to clients as 
pypy.tool.build.ppbserver
> +        from pypy.tool import build
> +        build.ppbserver = server
> +
> +        server.serve_forever()
> +    finally:
> +        channel.close()
> +"""
> +def init(gw, port=12321, path=[], projectname='pypy', buildpath=None,
> +            mailhost=None, mailport=25, mailfrom=None):
> +    from pypy.tool.build import execnetconference
> +    conference = execnetconference.conference(gw, port, True)
> +    channel = conference.remote_exec(initcode % (path, 
projectname,buildpath,
> +                                                    mailhost, mailport,
> +                                                    mailfrom))
> +    return channel
> 
> Added: pypy/dist/pypy/tool/build/test/fake.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/fake.py   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,54 @@
> +from pypy.tool.build.server import BuildPath
> +
> +class FakeChannel(object):
> +    def __init__(self):
> +        self._buffer = []
> +
> +    def send(self, item):
> +        self._buffer.append(item)
> +
> +    def receive(self):
> +        return self._buffer.pop(0)
> +
> +    def close(self):
> +        pass
> +
> +    def waitclose(self):
> +        pass
> +
> +class FakeClient(object):
> +    def __init__(self, info):
> +        self.channel = FakeChannel()
> +        self.sysinfo = info
> +        self.busy_on = None
> +
> +    def compile(self, info):
> +        info.pop('revision')
> +        for k, v in info.items():
> +            self.channel.send('%s: %r' % (k, v))
> +        self.channel.send(None)
> +        self.busy_on = info
> +
> +class FakeServer(object):
> +    def __init__(self, builddirpath):
> +        builddirpath.ensure(dir=True)
> +        self._channel = FakeChannel()
> +        self._builddirpath = builddirpath
> +        self._clients = []
> +        self._done = []
> +
> +    def register(self, client):
> +        self._clients.append(client)
> +
> +    def compilation_done(self, info, data):
> +        self._done.append((info, data))
> + 
> +    i = 0
> +    def get_new_buildpath(self, info):
> +        name = 'build-%s' % (self.i,)
> +        self.i += 1
> +        bp = BuildPath(str(self._builddirpath / name))
> +        bp.info = info
> +        bp.ensure(dir=1)
> +        return bp
> +
> 
> Added: pypy/dist/pypy/tool/build/test/path.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/path.py   Wed Jul 26 13:18:25 2006
> @@ -0,0 +1,6 @@
> +import py
> +
> +testpath = py.magic.autopath().dirpath()
> +packagepath = testpath.dirpath()
> +rootpath = packagepath.dirpath().dirpath().dirpath()
> +py.std.sys.path.append(str(rootpath))
> 
> Added: pypy/dist/pypy/tool/build/test/test.zip
> 
==============================================================================
> Binary file. No diff available.
> 
> Added: pypy/dist/pypy/tool/build/test/test_client.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_client.py   Wed Jul 26 13:18:25 
2006
> @@ -0,0 +1,43 @@
> +import path
> +from pypy.tool.build import client
> +import py
> +import time
> +from fake import FakeChannel, FakeServer
> +
> +class ClientForTests(client.PPBClient):
> +    def __init__(self, *args, **kwargs):
> +        super(ClientForTests, self).__init__(*args, **kwargs)
> +        self._done = []
> + 
> +def setup_module(mod):
> +    mod.temp = temp = py.test.ensuretemp('pypybuilder-client')
> +    mod.svr = svr = FakeServer(temp)
> +
> +    import pypy.tool.build
> +    pypy.tool.build.ppbserver = svr
> +
> +    mod.c1c = c1c = FakeChannel()
> +    mod.c1 = c1 = ClientForTests(c1c, {'foo': 1, 'bar': [1,2]})
> +    svr.register(c1)
> +
> +    mod.c2c = c2c = FakeChannel()
> +    mod.c2 = c2 = ClientForTests(c2c, {'foo': 2, 'bar': [2,3]})
> +    svr.register(c2)
> +
> +def test_compile():
> +    info = {'foo': 1}
> +    c1.compile(info)
> +    c1.channel.receive()
> +    c1.channel.send('foo bar')
> +    c1.channel.send(None)
> +
> +    # meanwhile the client starts a thread that waits until there's 
data 
> +    # available on its own channel, with our FakeChannel it has 
> data rightaway,
> +    # though (the channel out and in are the same, and we just sent 
'info'
> +    # over the out one)
> +    time.sleep(1) 
> + 
> +    done = svr._done.pop()
> + 
> +    assert done[0] == info
> +    assert done[1] == (temp / 'build-0')
> 
> Added: pypy/dist/pypy/tool/build/test/test_pypybuilder.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_pypybuilder.py   Wed Jul 26 
> 13:18:25 2006
> @@ -0,0 +1,137 @@
> +import path
> +from pypy.tool.build import client, server, execnetconference
> +from pypy.tool.build import config
> +import py
> +
> +# some functional tests (although some of the rest aren't strictly
> +# unit tests either), to run use --functional as an arg to py.test
> +def test_functional_1():
> +    if not py.test.pypybuilder_option.functional:
> +        py.test.skip('skipping functional test, use --functional to run 
it')
> + 
> +    # XXX this one is a bit messy, it's a quick functional test forthe 
whole
> +    # system, but for instance contains time.sleep()s to make sure 
> all threads
> +    # get the time to perform tasks and such... 
> +
> +    sleep_interval = 0.3
> +
> +    # first initialize a server
> +    sgw = py.execnet.PopenGateway()
> +    temppath = py.test.ensuretemp('pypybuilder-functional')
> +    sc = server.init(sgw, port=config.port, path=config.testpath, 
> +                        buildpath=str(temppath))
> +
> +    # give the server some time to wake up
> +    py.std.time.sleep(sleep_interval)
> +
> +    # then two clients, both with different system info
> +    sysinfo1 = {
> +        'foo': 1,
> +        'bar': [1,2],
> +    }
> +    cgw1 = py.execnet.PopenGateway()
> +    cc1 = client.init(cgw1, sysinfo1, port=config.port, testing=True)
> +
> +    sysinfo2 = {
> +        'foo': 2,
> +        'bar': [1],
> +    }
> +    cgw2 = py.execnet.PopenGateway()
> +    cc2 = client.init(cgw2, sysinfo2, port=config.port, testing=True)
> +
> +    # give the clients some time to register themselves
> +    py.std.time.sleep(sleep_interval)
> +
> +    # now we're going to send some compile jobs
> +    code = """
> +        import sys
> +        sys.path += %r
> + 
> +        from pypy.tool.build import ppbserver
> +        channel.send(ppbserver.compile(%r, %r))
> +        channel.close()
> +    """
> +    compgw = py.execnet.PopenGateway()
> +    compconf = execnetconference.conference(compgw, config.port)
> +
> +    # this one should fail because there's no client found for foo = 3
> +    compc = compconf.remote_exec(code % (config.testpath, 
'foo1 at bar.com', 
> +                                            {'foo': 3}))
> + 
> +    # sorry...
> +    py.std.time.sleep(sleep_interval)
> +
> +    ret = compc.receive()
> +    assert not ret[0]
> +    assert ret[1].find('no suitable client found') > -1
> +
> +    # this one should be handled by client 1
> +    compc = compconf.remote_exec(code % (config.testpath, 
'foo2 at bar.com',
> +                                            {'foo': 1, 'bar': [1]}))
> + 
> +    # and another one
> +    py.std.time.sleep(sleep_interval)
> + 
> +    ret = compc.receive()
> +    assert not ret[0]
> +    assert ret[1].find('found a suitable client') > -1
> +
> +    # the messages may take a bit to arrive, too
> +    py.std.time.sleep(sleep_interval)
> +
> +    # client 1 should by now have received the info to build for
> +    cc1.receive() # 'welcome'
> +    ret = cc1.receive() 
> +    assert ret == {'foo': 1, 'bar': [1], 'revision': 'trunk'}
> +
> +    # this should have created a package in the temp dir
> +    assert len(temppath.listdir()) == 1
> +
> +    # now we're going to satisfy the first request by adding a new 
client
> +    sysinfo3 = {'foo': 3}
> +    cgw3 = py.execnet.PopenGateway()
> +    cc3 = client.init(cgw3, sysinfo3, port=config.port, testing=True)
> +
> +    # again a bit of waiting may be desired
> +    py.std.time.sleep(sleep_interval)
> +
> +    # _try_queued() should check whether there are new clients 
available for 
> +    # queued jobs
> +    code = """
> +        import sys, time
> +        sys.path += %r
> + 
> +        from pypy.tool.build import ppbserver
> +        ppbserver._try_queued()
> +        # give the server some time, the clients 'compile' in threads
> +        time.sleep(%s) 
> +        channel.send(ppbserver._requeststorage._id_to_emails)
> +        channel.close()
> +    """
> +    compgw2 = py.execnet.PopenGateway()
> +    compconf2 = execnetconference.conference(compgw2, config.port)
> +
> +    compc2 = compconf2.remote_exec(code % (config.testpath, 
sleep_interval))
> +
> +
> +    # we check whether all emails are now sent, since after adding the 
third
> +    # client, and calling _try_queued(), both jobs should have 
beenprocessed
> +    ret = compc2.receive()
> +    assert ret.values() == []
> +
> +    # this should also have created another package in the temp dir
> +    assert len(temppath.listdir()) == 2
> +
> +    # some cleanup (this should all be in nested try/finallys, blegh)
> +    cc1.close()
> +    cc2.close()
> +    cc3.close()
> +    compc.close()
> +    compc2.close()
> +    sc.close()
> +
> +    cgw1.exit()
> +    cgw2.exit()
> +    compgw.exit()
> +    compgw2.exit()
> +    sgw.exit()
> 
> Added: pypy/dist/pypy/tool/build/test/test_request_storage.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_request_storage.py   Wed Jul
> 26 13:18:25 2006
> @@ -0,0 +1,64 @@
> +import path
> +import py
> +from pypy.tool.build.server import RequestStorage
> +
> +def test_request_storage():
> +    s = RequestStorage()
> +
> +    assert s._id_to_info == {}
> +    assert s._id_to_emails == {}
> +    assert s._id_to_path == {}
> +
> +    info = {'foo': 1}
> +    infoid = s.get_info_id(info)
> + 
> +    path = s.request('foo at bar.com', info)
> +    assert path is None
> +    assert s._id_to_info == {infoid: info}
> +    assert s._id_to_emails == {infoid: ['foo at bar.com']}
> +    assert s._id_to_path == {}
> +
> +    path = s.request('bar at bar.com', info)
> +    assert path is None
> +    assert s._id_to_info == {infoid: info}
> +    assert s._id_to_emails == {infoid: ['foo at bar.com', 'bar at bar.com']}
> +    assert s._id_to_path == {}
> +
> +    emails = s.add_build(info, 'foobar')
> +    assert emails == ['foo at bar.com', 'bar at bar.com']
> +    assert s._id_to_info == {infoid: info}
> +    assert s._id_to_emails == {}
> +    assert s._id_to_path == {infoid: 'foobar'}
> +
> +    info2 = {'foo': 2, 'bar': [1,2]}
> +    infoid2 = s.get_info_id(info2)
> + 
> +    path = s.request('foo at baz.com', info2)
> +    assert path is None
> +    assert s._id_to_info == {infoid: info, infoid2: info2}
> +    assert s._id_to_emails == {infoid2: ['foo at baz.com']}
> +    assert s._id_to_path == {infoid: 'foobar'}
> +
> +    emails = s.add_build(info2, 'foobaz')
> +    assert emails == ['foo at baz.com']
> +    assert s._id_to_info == {infoid: info, infoid2: info2}
> +    assert s._id_to_emails == {}
> +    assert s._id_to_path == {infoid: 'foobar', infoid2: 'foobaz'}
> +
> +    path = s.request('foo at qux.com', info)
> +    assert path == 'foobar'
> +
> +def test__build_initial():
> +    s = RequestStorage([({'foo': 1}, 'foo'), ({'foo': 2}, 'bar'),])
> +
> +    id1 = s.get_info_id({'foo': 1})
> +    id2 = s.get_info_id({'foo': 2})
> +
> +    assert s._id_to_info == {id1: {'foo': 1}, id2: {'foo': 2}}
> +    assert s._id_to_emails == {}
> +    assert s._id_to_path == {id1: 'foo', id2: 'bar'}
> +
> +def test__normalize():
> +    s = RequestStorage()
> +    assert (s._normalize({'foo': ['bar', 'baz']}) == 
> +            s._normalize({'foo': ['baz', 'bar']}))
> 
> Added: pypy/dist/pypy/tool/build/test/test_server.py
> 
==============================================================================
> --- (empty file)
> +++ pypy/dist/pypy/tool/build/test/test_server.py   Wed Jul 26 13:18:25 
2006
> @@ -0,0 +1,132 @@
> +import path
> +from pypy.tool.build import server
> +import py
> +from fake import FakeChannel, FakeClient
> +from pypy.tool.build.server import RequestStorage
> +from pypy.tool.build.server import BuildPath
> +import time
> +
> +def setup_module(mod):
> +    mod.temppath = temppath = py.test.ensuretemp('pypybuilder-server')
> +    mod.svr = server.PPBServer('pypytest', FakeChannel(), 
str(temppath))
> + 
> +    mod.c1 = FakeClient({'foo': 1, 'bar': [1,2]})
> +    mod.svr.register(mod.c1)
> +
> +    mod.c2 = FakeClient({'foo': 2, 'bar': [2,3]})
> +    mod.svr.register(mod.c2)
> +
> +def test_server_issubdict():
> +    from pypy.tool.build.server import issubdict
> +    assert issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'bar': 2, 'baz': 
3})
> +    assert not issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'baz': 3})
> +    assert not issubdict({'foo': 1, 'bar': 3}, {'foo': 1, 'bar': 
2,'baz': 3})
> +    assert issubdict({'foo': [1,2]}, {'foo': [1,2,3]})
> +    assert not issubdict({'foo': [1,2,3]}, {'foo': [1,2]})
> +    assert issubdict({'foo': 1L}, {'foo': 1})
> +    assert issubdict({}, {'foo': 1})
> +    assert issubdict({'foo': [1,2]}, {'foo': [1,2,3,4], 'bar': [1,2]})
> +
> +# XXX: note that the order of the tests matters! the first test reads 
the
> +# information from the channels that was set by the setup_module() 
function,
> +# the rest assumes this information is already read...
> + 
> +def test_register():
> +    assert len(svr._clients) == 2
> +    assert svr._clients[0] == c1
> +    assert svr._clients[1] == c2
> +
> +    assert c1.channel.receive() == 'welcome'
> +    assert c2.channel.receive() == 'welcome'
> +    py.test.raises(IndexError, "c1.channel.receive()")
> +
> +    assert svr._channel.receive().find('registered') > -1
> +    assert svr._channel.receive().find('registered') > -1
> +    py.test.raises(IndexError, 'svr._channel.receive()')
> +
> +def test_compile():
> +    # XXX this relies on the output not changing... quite scary
> +    info = {'foo': 1}
> +    ret = svr.compile('test at domain.com', info)
> +    assert not ret[0]
> +    assert ret[1].find('found a suitable client') > -1
> +    assert svr._channel.receive().find('going to send compile job') > 
-1
> +    assert c1.channel.receive() == 'foo: 1'
> +    assert c1.channel.receive() is None
> +    py.test.raises(IndexError, "c2.channel.receive()")
> +
> +    svr.compile('test at domain.com', {'foo': 3})
> +    assert svr._channel.receive().find('no suitable client available') 
> -1
> +
> +    info = {'bar': [3]}
> +    ret = svr.compile('test at domain.com', info)
> +    assert svr._channel.receive().find('going to send') > -1
> +    assert c2.channel.receive() == 'bar: [3]'
> +    assert c2.channel.receive() is None
> +    py.test.raises(IndexError, "c1.channel.receive()")
> +
> +    info = {'foo': 1}
> +    ret = svr.compile('test at domain.com', info)
> +    assert not ret[0]
> +    assert ret[1].find('this build is already') > -1
> +    assert svr._channel.receive().find('currently in progress') > -1
> +
> +    c1.busy_on = None
> +    bp = BuildPath(str(temppath / 'foo'))
> +    svr.compilation_done(info, bp)
> +    ret = svr.compile('test at domain.com', info)
> +    assert ret[0]
> +    assert isinstance(ret[1], BuildPath)
> +    assert ret[1] == bp
> +    assert svr._channel.receive().find('compilation done for') > -1
> +    for i in range(2):
> +        assert svr._channel.receive().find('going to send email to') > 
-1
> +    assert svr._channel.receive().find('already a build for this info') 
> -1
> + 
> +def test_buildpath():
> +    tempdir = py.test.ensuretemp('pypybuilder-buildpath')
> +    # grmbl... local.__new__ checks for class equality :(
> +    bp = BuildPath(str(tempdir / 'test1')) 
> +    assert not bp.check()
> +    assert bp.info == {}
> +
> +    bp.info = {'foo': 1, 'bar': [1,2]}
> +    assert bp.info == {'foo': 1, 'bar': [1,2]}
> +    assert (sorted((bp / 'info.txt').readlines()) == 
> +            ['bar: [1, 2]\n', 'foo: 1\n'])
> +
> +    assert isinstance(bp.zipfile, py.path.local)
> +    bp.zipfile = ['foo', 'bar', 'baz']
> +    assert bp.zipfile.read() == 'foobarbaz'
> +
> +def test__create_filename():
> +    svr._i = 0 # reset counter
> +    today = time.strftime('%Y%m%d')
> +    name1 = svr._create_filename()
> +    assert name1 == 'pypytest-%s-0' % (today,)
> +    assert svr._create_filename() == ('pypytest-%s-1' % (today,))
> +    bp = BuildPath(str(temppath / ('pypytest-%s-2' % (today,))))
> +    try:
> +        bp.ensure()
> +        assert svr._create_filename() == 'pypytest-%s-3'% (today,)
> +    finally:
> +        bp.remove()
> + 
> +def test_get_new_buildpath():
> +    svr._i = 0
> +    today = time.strftime('%Y%m%d')
> +
> +    path1 = svr.get_new_buildpath({'foo': 'bar'})
> +    try:
> +        assert isinstance(path1, BuildPath)
> +        assert path1.info == {'foo': 'bar'}
> +        assert path1.basename == 'pypytest-%s-0' % (today,)
> +
> +        try:
> +            path2 = svr.get_new_buildpath({'foo': 'baz'})
> +            assert path2.info == {'foo': 'baz'}
> +            assert path2.basename == 'pypytest-%s-1' % (today,)
> +        finally:
> +            path2.remove()
> +    finally:
> +        path1.remove()
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
> 


From guido at merlinux.de  Wed Jul 26 13:47:15 2006
From: guido at merlinux.de (Guido Wesdorp)
Date: Wed, 26 Jul 2006 13:47:15 +0200
Subject: [pypy-dev] [pypy-svn] r30565 - in pypy/dist/pypy/tool/build:
 .	bin builds test
In-Reply-To: <OFA83F9FBD.75F9FF4B-ON802571B7.003F7C6B-802571B7.003F9419@risk.sungard.com>
References: <OFA83F9FBD.75F9FF4B-ON802571B7.003F7C6B-802571B7.003F9419@risk.sungard.com>
Message-ID: <44C75643.6040004@merlinux.de>

Ben.Young at risk.sungard.com wrote:
> Cool tool! Is this something I would be able to run just in the 
> evenings/weekends etc? 
Sure! Do note (I wasn't clear there) that it's a work in progress, and
currently doesn't actually compile yet (it sends out some dummy zip file).

The idea is that a server will at some point be ran on codespeak.net to
which you can connect.

> Will it pick up that fact that I don't have boehm 
> automatically?
>
>   
It should become fairly intelligent, the idea is that the client scans
the system what it's capable of compiling, so I guess that's a yes... ;)
But again, it's a work in progress. Currently in the test data I only
check things like sys.maxint and such, the more advanced stuff will come
later (with the help of someone more initiated into PyPy than I am, I
hope ;).

Cheers,

Guido

From Ben.Young at risk.sungard.com  Wed Jul 26 13:51:59 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 26 Jul 2006 12:51:59 +0100
Subject: [pypy-dev] [pypy-svn] r30565 - in pypy/dist/pypy/tool/build:
 .	bin builds test
In-Reply-To: <44C75643.6040004@merlinux.de>
Message-ID: <OF2DA340EC.CA4FEE5F-ON802571B7.0040E7DA-802571B7.00412EDB@risk.sungard.com>

Hi Guido,

It's alright, I wasn't about to try it! It will be good to know I can help 
out the project, even though I don't really have the time to do any coding 
as such.

Cheers,
Ben

pypy-dev-bounces at codespeak.net wrote on 26/07/2006 12:47:15:

> Ben.Young at risk.sungard.com wrote:
> > Cool tool! Is this something I would be able to run just in the 
> > evenings/weekends etc? 
> Sure! Do note (I wasn't clear there) that it's a work in progress, and
> currently doesn't actually compile yet (it sends out some dummy zip 
file).
> 
> The idea is that a server will at some point be ran on codespeak.net to
> which you can connect.
> 
> > Will it pick up that fact that I don't have boehm 
> > automatically?
> >
> > 
> It should become fairly intelligent, the idea is that the client scans
> the system what it's capable of compiling, so I guess that's a yes... ;)
> But again, it's a work in progress. Currently in the test data I only
> check things like sys.maxint and such, the more advanced stuff will come
> later (with the help of someone more initiated into PyPy than I am, I
> hope ;).
> 
> Cheers,
> 
> Guido
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From guido at merlinux.de  Wed Jul 26 14:01:57 2006
From: guido at merlinux.de (Guido Wesdorp)
Date: Wed, 26 Jul 2006 14:01:57 +0200
Subject: [pypy-dev] [pypy-svn] r30565 - in pypy/dist/pypy/tool/build:
 .	bin builds test
In-Reply-To: <OF2DA340EC.CA4FEE5F-ON802571B7.0040E7DA-802571B7.00412EDB@risk.sungard.com>
References: <OF2DA340EC.CA4FEE5F-ON802571B7.0040E7DA-802571B7.00412EDB@risk.sungard.com>
Message-ID: <44C759B5.6010905@merlinux.de>

Ben.Young at risk.sungard.com wrote:
> It will be good to know I can help 
> out the project, even though I don't really have the time to do any coding 
> as such.
>
>   
Your help will certainly be appreciated! ;)

Keep an eye on this list, I will announce when we can use build clients.

Thanks in advance!

Cheers,

Guido

From jacob at strakt.com  Wed Jul 26 15:57:19 2006
From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Wed, 26 Jul 2006 15:57:19 +0200
Subject: [pypy-dev] Registering Pypy on Freshmeat
Message-ID: <200607261557.20132.jacob@strakt.com>

It just struck me that maybe we should register the Pypy project on Freshmeat. 
It would increase the visibility of the project as well as increase the 
visibility of Python in the more general OpenSource world.

What do people think?

Jacob

From mwh at python.net  Wed Jul 26 17:04:42 2006
From: mwh at python.net (Michael Hudson)
Date: Wed, 26 Jul 2006 16:04:42 +0100
Subject: [pypy-dev] pypy development tasks
References: <44C5EA98.5070700@gmx.net>
Message-ID: <2mslkoxv91.fsf@starship.python.net>

Stephan Diehl <stephan.diehl at gmx.net> writes:

> Hi all,
>
> starting next week, I'll be able again to continue working on pypy.
> As a part-part-part time worker, I find it a bit difficult to stay on 
> top on what's going on (and what needs to be done).
>
> These are the things, I would continue on, if nobody has a better idea:
>
> - write a python module that exposes Coroutine interface (using
>    greenlets, I suppose). This would be usefull for further development
>    of applevel stackless module. As a sideeffect, the stackless module
>    could then be used with CPython.

Erm, don't we have this part already?  Oh, you mean on top of
_CPython_, rather than on top of py.py on top of CPython.  I guess
this would help testing...

> - continue work on applevel stackless module and documentation.

This would be good.  Samuele pointed out some more stackless examples
at:

    http://members.verizon.net/olsongt/stackless/why_stackless.html

It would be nice to add these to the test suite (some of the examples
use pygame, so those need to be changed).

> Other than that, if I should rather do things like adding docstrings or 
> do other usefull tasks, don't hesitate to tell me :-)

We've just had a quick IRC discussion, and we decided that it would be
really nice if you could work on the pre-emptive scheduling mentioned
in the description of WP07; this will require adding a hook to the
interpreter than is called every few hundred bytecodes, but other than
that should mostly be implementable at app level (and the design
stolen from stackless CPython, I guess).

Cheers,
mwh

-- 
  <glyph> I am *not* a PSU agent.
                                                -- from Twisted.Quotes


From leo.soto at gmail.com  Wed Jul 26 23:48:35 2006
From: leo.soto at gmail.com (Leonardo Soto)
Date: Wed, 26 Jul 2006 17:48:35 -0400
Subject: [pypy-dev] Fix for #145 (-m cmdline option)
Message-ID: <5a8cc9e00607261448t7ea3d9bfo9cf567b4124681ed@mail.gmail.com>

Hi,

I've tried to upload the attached patch on roundup, but it gives me a
"Required issue properties title, priority not supplied" error. It
could be a permission error, but the message don't suggest that.

So the patch is here. Comments and corrections are welcomed.

-- 
Leonardo Soto M.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pypy-pep-338.patch
Type: text/x-patch
Size: 28507 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060726/62259ba3/attachment-0001.bin 

From elmo13 at jippii.fi  Thu Jul 27 00:42:48 2006
From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=)
Date: Thu, 27 Jul 2006 01:42:48 +0300
Subject: [pypy-dev] A bug in cc?
In-Reply-To: <20060725093526.GA25834@code0.codespeak.net>
References: <44BE2F9E.4060306@jippii.fi>
	<20060721152939.GB31604@code0.codespeak.net>
	<44C0FBCA.1090605@jippii.fi> <44C1388D.80003@jippii.fi>
	<44C3C339.8070809@jippii.fi>
	<20060724205610.GA25017@code0.codespeak.net>
	<44C54333.5080701@jippii.fi>
	<20060725093526.GA25834@code0.codespeak.net>
Message-ID: <44C7EFE8.10903@jippii.fi>

Armin Rigo wrote:
> Hi Elmo,
>
> On Tue, Jul 25, 2006 at 01:01:23AM +0300, Elmo M?ntynen wrote:
>   
>> I have 4.0.4 (debian 4.0.3-4). How should we go about testing on the
>> latest gcc, or are you doing it right now. I can definitely reproduce
>> the bug with the that certain file(s). Which of the .gcda and .gcno (or
>> the rest) files should I send?
>>     
>
> I don't know what these files are, so I can't comment, but you should
> look or ask around in the gcc community instead.  Also, I'm sure they
> would like to see information related to their latest release, so you
> should try with that.  And of course you should try to see if it's a
> known bug, maybe even if it's been fixed already...
>
>
> A bientot,
>
> Armin
>   
Just to note: I ran into some obstacles, and regenerating the -fprofile
related file produced a non crashing one. Simply, compiling with
profiling works, and so I will drop this.

Elmo

From stephan.diehl at gmx.net  Thu Jul 27 10:33:04 2006
From: stephan.diehl at gmx.net (Stephan Diehl)
Date: Thu, 27 Jul 2006 10:33:04 +0200
Subject: [pypy-dev] pypy development tasks
In-Reply-To: <2mslkoxv91.fsf@starship.python.net>
References: <44C5EA98.5070700@gmx.net> <2mslkoxv91.fsf@starship.python.net>
Message-ID: <44C87A40.9030908@gmx.net>


> 
> Erm, don't we have this part already?  Oh, you mean on top of
> _CPython_, rather than on top of py.py on top of CPython.  I guess
> this would help testing...

yep, that's it: coroutine on top of cpython.

> 
>> - continue work on applevel stackless module and documentation.
> 
> This would be good.  Samuele pointed out some more stackless examples
> at:
> 
>     http://members.verizon.net/olsongt/stackless/why_stackless.html
> 
> It would be nice to add these to the test suite (some of the examples
> use pygame, so those need to be changed).

sounds like a good idea...

> 
>> Other than that, if I should rather do things like adding docstrings or 
>> do other usefull tasks, don't hesitate to tell me :-)
> 
> We've just had a quick IRC discussion, and we decided that it would be
> really nice if you could work on the pre-emptive scheduling mentioned
> in the description of WP07; this will require adding a hook to the
> interpreter than is called every few hundred bytecodes, but other than
> that should mostly be implementable at app level (and the design
> stolen from stackless CPython, I guess).

I'll look into that.

More on it next week.

Cheers

Stephan
> 
> Cheers,
> mwh
> 


From arigo at tunes.org  Thu Jul 27 11:07:29 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 11:07:29 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h
	for	win32
In-Reply-To: <44C6042D.7090806@scottdial.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<44C6042D.7090806@scottdial.com>
Message-ID: <20060727090729.GA25278@code0.codespeak.net>

Hi Scottt,

On Tue, Jul 25, 2006 at 07:44:45AM -0400, Scott Dial wrote:
> I actually was pretty suprised too.. Presuming this is an important 
> platform to test (*grin*), I can provide a daily testing run for 
> Windows. I don't know what the code behind your pypytest is, but I 
> presume I could run something similar? Let me know.

The scripts I use in the following directory:

    http://codespeak.net/svn/user/arigo/hack/misc/htmlconftest/

While some details might be Unix-specific, it shouldn't be too hard to
adapt them to Windows.  (They produce symlinks, for example.)

Put the scripts in a directory, together with a subdir 'html' (initially
empty) which is visible on the web, and a checkout of PyPy in
'pypy-dist'.  The entry point script is autotest.py.


A bientot,

Armin

From arigo at tunes.org  Thu Jul 27 11:17:07 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 11:17:07 +0200
Subject: [pypy-dev] extension compiler and Python special methods
In-Reply-To: <9eebf5740607251151y29f5341l43bc92fe28ff592e@mail.gmail.com>
References: <9eebf5740607251151y29f5341l43bc92fe28ff592e@mail.gmail.com>
Message-ID: <20060727091707.GA27011@code0.codespeak.net>

Hi Lawrence,

On Tue, Jul 25, 2006 at 08:51:09PM +0200, Lawrence Oluyede wrote:
> As u can see in rev. #30543 I commited a series of methods to complete
> the mmap module. Those are the special methods (__xyz__) implemented
> as regular ones. I thought it was best to have the functionality in
> anyway. When and if we'll have that support it will be only a matter
> of change methods names to gain compatibility with CPython's code.
> 
> If you disagree with me let me know, I'll comment out the code or whatever.

That's fine.  Note also that you can use special method *names* in
RPython code, you just have to call them explicitly.  So for example
it's ok to have code like:

    class W_X:
       def __len__(self):
           return 42

       def get_len(self, space):
           return space.wrap(self.__len__())

Or even:

    class PurePythonBase:
        def __len__(self):
            return 42

    class W_X(PurePythonBase):
        def get_len(self, space):
            return space.wrap(self.__len__())


A bientot,

Armin

From l.oluyede at gmail.com  Thu Jul 27 11:17:12 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 27 Jul 2006 11:17:12 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <20060725093843.GA26167@code0.codespeak.net>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
Message-ID: <9eebf5740607270217q38087a07idab3ff95ee8e6@mail.gmail.com>

I think it could also be better to refactor a bit that header. I
placed the header I needed for my extension modules but I realize
that's not the right place for them. We can move out "my" includes in
a separate modules.h and include it in g_prerequisite.h or g_include.h
or whatever.



-- 
Lawrence
http://www.oluyede.org/blog

From l.oluyede at gmail.com  Thu Jul 27 11:20:02 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 27 Jul 2006 11:20:02 +0200
Subject: [pypy-dev] extension compiler and Python special methods
In-Reply-To: <20060727091707.GA27011@code0.codespeak.net>
References: <9eebf5740607251151y29f5341l43bc92fe28ff592e@mail.gmail.com>
	<20060727091707.GA27011@code0.codespeak.net>
Message-ID: <9eebf5740607270220v75a9c68fqb3a862fc89a004de@mail.gmail.com>

> That's fine.  Note also that you can use special method *names* in
> RPython code, you just have to call them explicitly.

Is it some sort of coding style rule? I think is ok but sounds like a
duplication of code to me now...


-- 
Lawrence
http://www.oluyede.org/blog

From arigo at tunes.org  Thu Jul 27 11:55:08 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 11:55:08 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <9eebf5740607270217q38087a07idab3ff95ee8e6@mail.gmail.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<9eebf5740607270217q38087a07idab3ff95ee8e6@mail.gmail.com>
Message-ID: <20060727095508.GB27011@code0.codespeak.net>

Hi Lawrence,

On Thu, Jul 27, 2006 at 11:17:12AM +0200, Lawrence Oluyede wrote:
> I think it could also be better to refactor a bit that header. I
> placed the header I needed for my extension modules but I realize
> that's not the right place for them. We can move out "my" includes in
> a separate modules.h and include it in g_prerequisite.h or g_include.h
> or whatever.

The rctypes-based modules' includes shouldn't go in the static .h files.
They are generated automatically by genc if you use the correct magic:
look for _includes_ in module/readline/c_readline.py.


A bientot,

Armin

From arigo at tunes.org  Thu Jul 27 11:58:25 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 27 Jul 2006 11:58:25 +0200
Subject: [pypy-dev] extension compiler and Python special methods
In-Reply-To: <9eebf5740607270220v75a9c68fqb3a862fc89a004de@mail.gmail.com>
References: <9eebf5740607251151y29f5341l43bc92fe28ff592e@mail.gmail.com>
	<20060727091707.GA27011@code0.codespeak.net>
	<9eebf5740607270220v75a9c68fqb3a862fc89a004de@mail.gmail.com>
Message-ID: <20060727095825.GC27011@code0.codespeak.net>

Hi Lawrence,

On Thu, Jul 27, 2006 at 11:20:02AM +0200, Lawrence Oluyede wrote:
> > That's fine.  Note also that you can use special method *names* in
> > RPython code, you just have to call them explicitly.
> 
> Is it some sort of coding style rule? I think is ok but sounds like a
> duplication of code to me now...

Well, as long as we have to write the space wrappers by hand, but would
also like the "pure" method to be available, there is going to be this
kind of duplication :-(  At some point, the unwrap_spec could be
enhanced to cover much more automatic wrapping and unwrapping; then
we'll be able to use the "pure" method directly, as in:

    class X:
        def __len__(self):
            return 42
        __len__.unwrap_spec = ['self'], 'returns an int'  # or whatever

and directly point to the __len__ method in the TypeDef, getting rid of
all explicit references to 'space' in the simple methods.


A bientot,

Armin.

From l.oluyede at gmail.com  Thu Jul 27 13:06:08 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 27 Jul 2006 13:06:08 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <20060727095508.GB27011@code0.codespeak.net>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<9eebf5740607270217q38087a07idab3ff95ee8e6@mail.gmail.com>
	<20060727095508.GB27011@code0.codespeak.net>
Message-ID: <9eebf5740607270406q43d44e31t3b965c07365529f5@mail.gmail.com>

> The rctypes-based modules' includes shouldn't go in the static .h files.
> They are generated automatically by genc if you use the correct magic:
> look for _includes_ in module/readline/c_readline.py.

I definitely missed that! Thanks.

-- 
Lawrence
http://www.oluyede.org/blog

From l.oluyede at gmail.com  Thu Jul 27 13:21:57 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 27 Jul 2006 13:21:57 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <9eebf5740607270406q43d44e31t3b965c07365529f5@mail.gmail.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<9eebf5740607270217q38087a07idab3ff95ee8e6@mail.gmail.com>
	<20060727095508.GB27011@code0.codespeak.net>
	<9eebf5740607270406q43d44e31t3b965c07365529f5@mail.gmail.com>
Message-ID: <9eebf5740607270421r227820dax7ffc0f1879aa366@mail.gmail.com>

On 7/27/06, Lawrence Oluyede <l.oluyede at gmail.com> wrote:
> > The rctypes-based modules' includes shouldn't go in the static .h files.
> > They are generated automatically by genc if you use the correct magic:
> > look for _includes_ in module/readline/c_readline.py.
>
> I definitely missed that! Thanks.

That doesn't seem to work. I commented out the includes I added to
make fcntl module compile (like sys/ioctl.h for example) and added
them to the _includes_ and this seem to work correctly only with tests
and the generation of ctypesplatcheck files. The headers are not
injected in the generated fcntl.c nor in any of the file it includes.

-- 
Lawrence
http://www.oluyede.org/blog

From l.oluyede at gmail.com  Thu Jul 27 13:32:25 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 27 Jul 2006 13:32:25 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <9eebf5740607270406q43d44e31t3b965c07365529f5@mail.gmail.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<9eebf5740607270217q38087a07idab3ff95ee8e6@mail.gmail.com>
	<20060727095508.GB27011@code0.codespeak.net>
	<9eebf5740607270406q43d44e31t3b965c07365529f5@mail.gmail.com>
Message-ID: <9eebf5740607270432p6791448fm96e0dc26f590261a@mail.gmail.com>

On 7/27/06, Lawrence Oluyede <l.oluyede at gmail.com> wrote:
> > The rctypes-based modules' includes shouldn't go in the static .h files.
> > They are generated automatically by genc if you use the correct magic:
> > look for _includes_ in module/readline/c_readline.py.
>
> I definitely missed that! Thanks.

That doesn't seem to work. I commented out the includes I added to
make fcntl module compile (like sys/ioctl.h for example) and added
them to the _includes_ and this seem to work correctly only with tests
and the generation of ctypesplatcheck files. The headers are not
injected in the generated fcntl.c nor in any of the file it includes.

-- 
Lawrence
http://www.oluyede.org/blog

From fijal at genesilico.pl  Thu Jul 27 15:58:12 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Thu, 27 Jul 2006 15:58:12 +0200
Subject: [pypy-dev] Good news everyone
Message-ID: <44C8C674.7030108@genesilico.pl>

Right now, b'n'b JS client should be almost playable. Speed and key 
granularity is still an issue, but there should be no problems with 
zindex and all other problems. Enjoy.



From radix at twistedmatrix.com  Thu Jul 27 17:08:01 2006
From: radix at twistedmatrix.com (Christopher Armstrong)
Date: Thu, 27 Jul 2006 11:08:01 -0400
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060721203613.GA2825@code0.codespeak.net>
References: <20060721203613.GA2825@code0.codespeak.net>
Message-ID: <60ed19d40607270808k6fa6d24di61ba290264acadb@mail.gmail.com>

On 7/21/06, Armin Rigo <arigo at tunes.org> wrote:
> Hi all, hi Aurelien,
>
> The stackless-related tests are all consistently failing nowadays.  Some
> have been failing for 4 days and others for 9 days.  See
>
>    http://snake.cs.uni-duesseldorf.de/pypytest/summary.html
>
> Given that you appear to continue to work on it, but without making
> tests pass again, I'm a bit worrying exactly about what you are doing?
> That doesn't seem too much in line with the "test-driven development"
> approach.

I'll point out that Twisted has been avoiding issues such as these in
the past 8 months or so by using a process that involves always
branching the repository for developments and always getting a review
before merging to trunk. The administrative overhead has definitely
paid off in much, much higher code and test coverage.

-- 
Christopher Armstrong
International Man of Twistery
http://radix.twistedmatrix.com/
http://twistedmatrix.com/
http://canonical.com/

From 2006a at usenet.alexanderweb.de  Thu Jul 27 19:09:14 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Thu, 27 Jul 2006 19:09:14 +0200
Subject: [pypy-dev] Combining strdicts and LLVM-C-Prof ...
Message-ID: <1ol7ygzoxm0fx.dlg@usenet.alexanderweb.de>

Hi,

today I made some tests of the strdict code on snake:

date                           size codesize    executable                        richards             pystone
Thu Jul 27 20:14:33 2006          -        -    python 2.4.3                        795ms (   1.0x)    40000 (   1.0x)
Thu Jul 27 20:08:02 2006   10425736  2475162    pypy-llvm-30641-x86                2902ms (   3.6x)     9191 (   4.4x)
Thu Jul 27 20:05:53 2006   10474191  2028604    pypy-llvm-30641-c-prof             1795ms (   2.3x)    14619 (   2.7x)
Thu Jul 27 19:58:41 2006   10574247  2128805    pypy-llvm-30641-c                  2193ms (   2.8x)    11441 (   3.5x)
Thu Jul 27 17:37:21 2006   10162160  2208132    pypy-c-30634-profopt               2380ms (   3.0x)    12285 (   3.3x)
Thu Jul 27 16:56:55 2006   10382148  2428236    pypy-c-30634                       2590ms (   3.3x)    10846 (   3.7x)

2.3x is really neat - 1.0x is not far :-)

Kind regards,
Alexander


From scott+pypy-dev at scottdial.com  Fri Jul 28 19:41:11 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Fri, 28 Jul 2006 13:41:11 -0400
Subject: [pypy-dev] small patch for
	translator/c/src/ll_osdefs.h	for	win32
In-Reply-To: <20060727090729.GA25278@code0.codespeak.net>
References: <44C53C95.4070002@scottdial.com>	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>	<20060725093843.GA26167@code0.codespeak.net>	<44C6042D.7090806@scottdial.com>
	<20060727090729.GA25278@code0.codespeak.net>
Message-ID: <44CA4C37.3060709@scottdial.com>

Armin Rigo wrote:
> While some details might be Unix-specific, it shouldn't be too hard to
> adapt them to Windows.  (They produce symlinks, for example.)

I was able to adapt the script just fine. And I ran it once without 
problems (with the script, not pypy ;). But, I realized I made an error 
and forget to setup the build enviroment for compiling. After doing so, 
I find that I get a stalled in module\_stackless\test

Nevertheless, in the future the results will be at 
http://scottdial.com/pypytest/

I will look into patching the misbehaving tests.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From scott+pypy-dev at scottdial.com  Sat Jul 29 06:48:46 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Sat, 29 Jul 2006 00:48:46 -0400
Subject: [pypy-dev] MSVC "too many initializer" problem
Message-ID: <44CAE8AE.4070006@scottdial.com>

Right now, if I were to enter the translatorshell and try to compile any 
of the snippets I will get an the following error:

testing_1.c(2318) : warning C4047: 'initializing' : 'char *' differs in 
levels of indirection from 'int'
testing_1.c(2318) : error C2078: too many initializers

Which refers to this line (for this example):

static globalfunctiondef_t globalfunctiondefs[] = {
	{&pypy_gfunc_sieve_of_eratosthenes, "pypy_gfunc_sieve_of_eratosthenes", 
{"sieve_of_eratosthenes", 
(PyCFunction)pypy_g_pyfn_sieve_of_eratosthenes, 
METH_VARARGS|METH_KEYWORDS, {
83,105,101,118,101,32,111,102,32,69,114,97,116,111,115,116,104,101,110,101,
115,10,32,32,32,32,10,32,32,32,32,84,104,105,115,32,111,110,101,32,
105,115,32,102,114,111,109,32,97,110,32,105,110,102,97,109,111,117,115,32,
98,101,110,99,104,109,97,114,107,44,32,34,84,104,101,32,71,114,101,97,
116,32,67,111,109,112,117,116,101,114,10,32,32,32,32,76,97,110,103,117,
97,103,101,32,83,104,111,111,116,111,117,116,34,46,10,10,32,32,32,32,
85,82,76,32,105,115,58,32,104,116,116,112,58,47,47,119,119,119,46,98,
97,103,108,101,121,46,111,114,103,47,126,100,111,117,103,47,115,104,111,111,
116,111,117,116,47,10,32,32,32,32}}},
	{ NULL }	/* Sentinel */
};

I'm not too familiar with structure initialization in MSVC's compiler, 
but it seems to barf on that one. If anyone has a quick answer, then I 
am all ears, otherwise I'll have to do some research.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From arigo at tunes.org  Sat Jul 29 09:05:49 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 29 Jul 2006 09:05:49 +0200
Subject: [pypy-dev] small patch for
	translator/c/src/ll_osdefs.h	for	win32
In-Reply-To: <44CA4C37.3060709@scottdial.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<44C6042D.7090806@scottdial.com>
	<20060727090729.GA25278@code0.codespeak.net>
	<44CA4C37.3060709@scottdial.com>
Message-ID: <20060729070549.GA22919@code0.codespeak.net>

Hi Scott,

On Fri, Jul 28, 2006 at 01:41:11PM -0400, Scott Dial wrote:
> Nevertheless, in the future the results will be at 
> http://scottdial.com/pypytest/

Thanks a lot!  Can we link to this page from the "Status" section of the
documentation index?


A bientot,

Armin

From arigo at tunes.org  Sat Jul 29 09:08:36 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 29 Jul 2006 09:08:36 +0200
Subject: [pypy-dev] small patch for translator/c/src/ll_osdefs.h for
	win32
In-Reply-To: <9eebf5740607270432p6791448fm96e0dc26f590261a@mail.gmail.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<9eebf5740607270217q38087a07idab3ff95ee8e6@mail.gmail.com>
	<20060727095508.GB27011@code0.codespeak.net>
	<9eebf5740607270406q43d44e31t3b965c07365529f5@mail.gmail.com>
	<9eebf5740607270432p6791448fm96e0dc26f590261a@mail.gmail.com>
Message-ID: <20060729070836.GB22919@code0.codespeak.net>

Hi Lawrence,

On Thu, Jul 27, 2006 at 01:32:25PM +0200, Lawrence Oluyede wrote:
> That doesn't seem to work. I commented out the includes I added to
> make fcntl module compile (like sys/ioctl.h for example) and added
> them to the _includes_ and this seem to work correctly only with tests
> and the generation of ctypesplatcheck files. The headers are not
> injected in the generated fcntl.c nor in any of the file it includes.

Strange: it works with the readline module.  If you run 'compilemodule.py
readline' you can check that /tmp/usession-*/readline/common_header.h
contains '#include <readline/readline.h>'.


A bientot,

Armin.

From arigo at tunes.org  Sat Jul 29 09:21:52 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 29 Jul 2006 09:21:52 +0200
Subject: [pypy-dev] MSVC "too many initializer" problem
In-Reply-To: <44CAE8AE.4070006@scottdial.com>
References: <44CAE8AE.4070006@scottdial.com>
Message-ID: <20060729072152.GC22919@code0.codespeak.net>

Hi Scott,

On Sat, Jul 29, 2006 at 12:48:46AM -0400, Scott Dial wrote:
> Right now, if I were to enter the translatorshell and try to compile any 
> of the snippets I will get an the following error:
> 
> testing_1.c(2318) : warning C4047: 'initializing' : 'char *' differs in 
> levels of indirection from 'int'
> testing_1.c(2318) : error C2078: too many initializers

Indeed, the line in question is not valid C at all.  In gcc it just
produces a lot of warnings and invalid data, but not an actual error :-/

This list of numbers is actually means to be the docstring of the
function, but in this broken syntax gcc uses the first number only and
interprets that number as a char*.  So we get (char*)83 as the C-level
docstring.  You bet that accessing it via the __doc__ attribute of the
built-in function gives a nice segfault...  What is not tested is
broken :-(

I'll try to fix this.


A bientot,

Armin.

From l.oluyede at gmail.com  Sat Jul 29 15:58:45 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Sat, 29 Jul 2006 15:58:45 +0200
Subject: [pypy-dev] make _ssl module compile
Message-ID: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>

I'm trying to feed the _ssl module (feature complete) to the ext.
compiler but I get a strange error I've never had before:

[translation:ERROR] Error:
[translation:ERROR]  AttributeError: 'CallEntry' object has no
attribute 'get_repr'
[translation] start debugger...
> /Users/rhymes/scc/pypy/pypy/rpython/rexternalobj.py(20)rtyper_makerepr()
-> return entry.get_repr(rtyper, self)
(Pdb) self
SomeCTypesObject(knowntype=CFunctionType, ownsmemory=False)
(Pdb) entry
<pypy.rpython.rctypes.afunc.CallEntry object at 0x202fd10>
(Pdb) locals()
{'entry': <pypy.rpython.rctypes.afunc.CallEntry object at 0x202fd10>,
'rtyper': <pypy.rpython.rtyper.RPythonTyper object at 0x172ef50>,
'self': SomeCTypesObject(knowntype=CFunctionType, ownsmemory=False)}
(Pdb) dir(entry)
['__class__', '__delattr__', '__dict__', '__doc__', '__eq__',
'__getattribute__', '__hash__', '__init__', '__metaclass__',
'__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', '__weakref__',
'compute_annotation', 'compute_result_annotation', 'instance',
'specialize_call', 'type']

I took a look into rctypes.afunc and in fact there's no get_repr
(because there's no rfunc.py). What problem is that?

-- 
Lawrence
http://www.oluyede.org/blog

From scott+pypy-dev at scottdial.com  Sat Jul 29 17:27:49 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Sat, 29 Jul 2006 11:27:49 -0400
Subject: [pypy-dev] small patch
	for	translator/c/src/ll_osdefs.h	for	win32
In-Reply-To: <20060729070549.GA22919@code0.codespeak.net>
References: <44C53C95.4070002@scottdial.com>	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>	<20060725093843.GA26167@code0.codespeak.net>	<44C6042D.7090806@scottdial.com>	<20060727090729.GA25278@code0.codespeak.net>	<44CA4C37.3060709@scottdial.com>
	<20060729070549.GA22919@code0.codespeak.net>
Message-ID: <44CB7E75.5010805@scottdial.com>

Armin Rigo wrote:
> On Fri, Jul 28, 2006 at 01:41:11PM -0400, Scott Dial wrote:
>> Nevertheless, in the future the results will be at 
>> http://scottdial.com/pypytest/
> 
> Thanks a lot!  Can we link to this page from the "Status" section of the
> documentation index?
> 

You sure can. I'll do my best to keep it available as much as possible. 
I'll be moving in 3 weeks and there'll be some days of downtime while I 
get my internet hooked up there, but otherwise I intend to keep it 
updated and available.

I'm not sure what time is the best for it to run daily. I've currently 
set it run daily at midnight GMT. Although I may have to change that 
depending on my sleeping behavior.. when it runs probably isn't that big 
of a concern as long it gets run, yah?

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From leo.soto at gmail.com  Sun Jul 30 03:11:14 2006
From: leo.soto at gmail.com (Leonardo Soto)
Date: Sat, 29 Jul 2006 21:11:14 -0400
Subject: [pypy-dev] gateway.applevel and kwargs
Message-ID: <5a8cc9e00607291811k3a0d0902w6e1cc7bccd170308@mail.gmail.com>

Hi all,

I've recently filled the issue 244 (dict.update() doesn't support
keywords argument), and tried to solve it in a naive way, just
modifying the applevel  definition to:

    def update(d, o, **kwargs):
        if hasattr(o, 'keys'):
            for k in o.keys():
                d[k] = o[k]
        else:
            for k,v in o:
                d[k] = v
        for k in kwargs:
            d[k] = kwargs[k]

But it simply doesn't work:

test_dictobject.py[34] .......................FF.........

__________________________________________________________________________________________________________________________________________
__________________________________________ entrypoint:
AppTest_DictObject().test_update_kwargs
___________________________________________

    def test_update_kwargs(self):
        d = {}
E       d.update(foo='bar', baz=1)
>       (application-level) TypeError: update() got 2 unexpected keyword arguments

[/home/leo/src/pypy-dist/pypy/objspace/std/test/None</home/leo/src/pypy-dist/pypy/interpreter/gateway.py:754>:3]
__________________________________________________________________________________________________________________________________________
______________________________________ entrypoint:
AppTest_DictObject().test_update_dict_and_kwargs
______________________________________

    def test_update_dict_and_kwargs(self):
        d = {}
E       d.update({'foo': 'bar'}, baz=1)
>       (application-level) TypeError: update() got an unexpected keyword argument 'baz'

[/home/leo/src/pypy-dist/pypy/objspace/std/test/None</home/leo/src/pypy-dist/pypy/interpreter/gateway.py:754>:3]

Why?. Should it work or is the lack of kwargs support a known limitation?

-- 
Leonardo Soto M.

From arigo at tunes.org  Sun Jul 30 09:10:27 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 30 Jul 2006 09:10:27 +0200
Subject: [pypy-dev] small patch
	for	translator/c/src/ll_osdefs.h	for	win32
In-Reply-To: <44CB7E75.5010805@scottdial.com>
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<44C6042D.7090806@scottdial.com>
	<20060727090729.GA25278@code0.codespeak.net>
	<44CA4C37.3060709@scottdial.com>
	<20060729070549.GA22919@code0.codespeak.net>
	<44CB7E75.5010805@scottdial.com>
Message-ID: <20060730071027.GA21606@code0.codespeak.net>

Hi Scott,

On Sat, Jul 29, 2006 at 11:27:49AM -0400, Scott Dial wrote:
> You sure can.

Thanks!

> I'm not sure what time is the best for it to run daily. I've currently 
> set it run daily at midnight GMT. Although I may have to change that 
> depending on my sleeping behavior.. when it runs probably isn't that big 
> of a concern as long it gets run, yah?

Indeed.  On snake it's in the early morning, just to minimize the risk
that someone else is doing something.


A bientot,

Armin

From scott+pypy-dev at scottdial.com  Tue Aug  1 07:47:11 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Tue, 01 Aug 2006 01:47:11 -0400
Subject: [pypy-dev] small
	patch	for	translator/c/src/ll_osdefs.h	for	win32
In-Reply-To: <44CB7E75.5010805@scottdial.com>
References: <44C53C95.4070002@scottdial.com>	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>	<20060725093843.GA26167@code0.codespeak.net>	<44C6042D.7090806@scottdial.com>	<20060727090729.GA25278@code0.codespeak.net>	<44CA4C37.3060709@scottdial.com>	<20060729070549.GA22919@code0.codespeak.net>
	<44CB7E75.5010805@scottdial.com>
Message-ID: <44CEEADF.4090104@scottdial.com>

Scott Dial wrote:
> I'm not sure what time is the best for it to run daily. I've currently 
> set it run daily at midnight GMT. 

For the record, I have a need for this to happen much later. I have 
moved it to 9am GMT which I realize may be afternoon for some people in 
Europe, but I can't help the Earth's rotation determining my schedule.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From cfbolz at gmx.de  Tue Aug  1 09:44:26 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Tue, 01 Aug 2006 09:44:26 +0200
Subject: [pypy-dev]
	small	patch	for	translator/c/src/ll_osdefs.h	for	win32
In-Reply-To: <44CEEADF.4090104@scottdial.com>
References: <44C53C95.4070002@scottdial.com>	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>	<20060725093843.GA26167@code0.codespeak.net>	<44C6042D.7090806@scottdial.com>	<20060727090729.GA25278@code0.codespeak.net>	<44CA4C37.3060709@scottdial.com>	<20060729070549.GA22919@code0.codespeak.net>	<44CB7E75.5010805@scottdial.com>
	<44CEEADF.4090104@scottdial.com>
Message-ID: <44CF065A.5050101@gmx.de>

Hi Scott!

Scott Dial wrote:
> Scott Dial wrote:
>> I'm not sure what time is the best for it to run daily. I've currently
>> set it run daily at midnight GMT.
>
> For the record, I have a need for this to happen much later. I have
> moved it to 9am GMT which I realize may be afternoon for some people in
> Europe, but I can't help the Earth's rotation determining my schedule.
>

Just for the record, I think we all appreciate very much that you are
running the tests and it does not matter too much _when_ you run them
:-). Thank you!

Cheers,

Carl Friedrich

From mwh at python.net  Tue Aug  1 10:06:44 2006
From: mwh at python.net (Michael Hudson)
Date: Tue, 01 Aug 2006 09:06:44 +0100
Subject: [pypy-dev] small
	patch	for	translator/c/src/ll_osdefs.h	for	win32
References: <44C53C95.4070002@scottdial.com>
	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>
	<20060725093843.GA26167@code0.codespeak.net>
	<44C6042D.7090806@scottdial.com>
	<20060727090729.GA25278@code0.codespeak.net>
	<44CA4C37.3060709@scottdial.com>
	<20060729070549.GA22919@code0.codespeak.net>
	<44CB7E75.5010805@scottdial.com> <44CEEADF.4090104@scottdial.com>
Message-ID: <2mvepcx4kr.fsf@starship.python.net>

Scott Dial <scott+pypy-dev at scottdial.com> writes:

> Scott Dial wrote:
>> I'm not sure what time is the best for it to run daily. I've currently 
>> set it run daily at midnight GMT. 
>
> For the record, I have a need for this to happen much later. I have 
> moved it to 9am GMT which I realize may be afternoon for some people in 
> Europe,

How could 9am _Greenwich_ (a place in the UK) Mean Time be in the
afternoon anywhere in Europe? :)

> but I can't help the Earth's rotation determining my schedule.

Indeed :) As Carl says, thanks a lot for doing this!

Cheers,
mwh

-- 
[1]For those of you who aren't aware "tossing" is a euphamism for,
   well, vigourously rubbing your love pole. You understand?
   Flogging the dolphin.  Stretching the chicken's neck. Waving your
   magic wand.  Basically, wanking.  -- Just another Morfans SDA update


From arigo at tunes.org  Tue Aug  1 13:06:59 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 1 Aug 2006 13:06:59 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
Message-ID: <20060801110659.GA21064@code0.codespeak.net>

Hi Lawrence,

On Sat, Jul 29, 2006 at 03:58:45PM +0200, Lawrence Oluyede wrote:
> [translation:ERROR]  AttributeError: 'CallEntry' object has no
> attribute 'get_repr'

This means that you are passing 'ctypes function' objects around, in
variables.  The only thing you can do with ctypes functions is call them
directly.  For now you have to find a workaround, e.g. pass stub RPython
functions around.  E.g. instead of:

    f1 = mylib.f1
    f2 = mylib.f2

    def invoke(f):
        return f(17)     # 'f' holds a variable ctypes function object

    ... 
       invoke(f1)
    ...
       invoke(f2)

Do:

    f1 = mylib.f1
    f2 = mylib.f2

    def f1stub(x): return f1(x)
    def f2stub(x): return f2(x)

    def invoke(f):
        return f(17)     # 'f' holds a regular RPython function

    ... 
       invoke(f1stub)
    ...
       invoke(f2stub)

(Note: this and a number of other ext-compiler or rctypes limitations
could be removed with a bit of work; don't hesitate to tell us if a
particular workaround seems really awkard in your cases.  I'd love this
kind of input, to know where to put efforts for more support.)


A bientot,

Armin

From arigo at tunes.org  Tue Aug  1 13:41:59 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 1 Aug 2006 13:41:59 +0200
Subject: [pypy-dev] gateway.applevel and kwargs
In-Reply-To: <5a8cc9e00607291811k3a0d0902w6e1cc7bccd170308@mail.gmail.com>
References: <5a8cc9e00607291811k3a0d0902w6e1cc7bccd170308@mail.gmail.com>
Message-ID: <20060801114159.GA23548@code0.codespeak.net>

Hi Leonardo,

On Sat, Jul 29, 2006 at 09:11:14PM -0400, Leonardo Soto wrote:
> I've recently filled the issue 244 (dict.update() doesn't support
> keywords argument), and tried to solve it in a naive way, just
> modifying the applevel  definition to:
> 
>     def update(d, o, **kwargs):
>         (...)

This is a bit messy.  It's an app-level definition but it goes through
the interp-level gateway code via the multimethod logic.  The first
thing to change is the declared signature of the dict_update
multimethod, also in dicttype.py:

    dict_update = SMM('update', 2, defaults=((),), doc="...")

This means "two arguments with a default of () for the second one".
Note that it's where the default comes from, too; not from the
"def update(d, o)" implementation.

Note first a difficulty: if we were implementing this method in pure
Python, we couldn't say  'def update(self, o, **kwargs)'  because there is
a name clash between 'o' and the keywords, preventing an explicit 'o'
keyword from being passed.  To solve this we would have to declare it
'def update(self, *args, **kwargs)'.

The equivalent solution here is:

    dict_update = SMM('update', 1, general__args__=True, doc="...")
                               ^^^^^^^^^^^^^^^^^^^^^^^^

    def update(d, *args, **kwargs):
        if len(args) > 1:
            raise TypeError("update takes at most 1 (non-keyword) argument")
        ...

    dict_update__ANY = app.interphook("update")
                ^^^^

But it doesn't work yet, because I'm just discovering that we have no
support for *args/**kwargs in app.interphook()!


Trying to fix this now...

Armin

From arigo at tunes.org  Tue Aug  1 13:56:33 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 1 Aug 2006 13:56:33 +0200
Subject: [pypy-dev] gateway.applevel and kwargs
In-Reply-To: <20060801114159.GA23548@code0.codespeak.net>
References: <5a8cc9e00607291811k3a0d0902w6e1cc7bccd170308@mail.gmail.com>
	<20060801114159.GA23548@code0.codespeak.net>
Message-ID: <20060801115633.GB23548@code0.codespeak.net>

Hi Leonardo,

On Tue, Aug 01, 2006 at 01:41:59PM +0200, Armin Rigo wrote:
> But it doesn't work yet, because I'm just discovering that we have no
> support for *args/**kwargs in app.interphook()!

Should work now.

Armin

From l.oluyede at gmail.com  Tue Aug  1 15:09:42 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Tue, 1 Aug 2006 15:09:42 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <20060801110659.GA21064@code0.codespeak.net>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
Message-ID: <9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>

Thanks for replying.

> This means that you are passing 'ctypes function' objects around, in
> variables.  The only thing you can do with ctypes functions is call them
> directly.

Wish it was that but as far as I know no behavior of this kind is
present in the code. Isn't there a way to know the name of the ctypes
function troubled from the pdb inspector when the crash happens? From
the CallEntry instance.

-- 
Lawrence
http://www.oluyede.org/blog

From l.oluyede at gmail.com  Tue Aug  1 15:20:53 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Tue, 1 Aug 2006 15:20:53 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
Message-ID: <9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>

> Wish it was that but as far as I know no behavior of this kind is
> present in the code. Isn't there a way to know the name of the ctypes
> function troubled from the pdb inspector when the crash happens? From
> the CallEntry instance.

I dig some more and you were right (as always :))

I have SSLv23_method which returns a pointer to SSL_METHOD (which is a
struct) and seems to dislike it.

-- 
Lawrence
http://www.oluyede.org/blog

From l.oluyede at gmail.com  Tue Aug  1 16:15:22 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Tue, 1 Aug 2006 16:15:22 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
Message-ID: <9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>

> I have SSLv23_method which returns a pointer to SSL_METHOD (which is a
> struct) and seems to dislike it.

I forgot to say that disliking means "not fixable" :(

-- 
Lawrence
http://www.oluyede.org/blog

From arigo at tunes.org  Tue Aug  1 18:18:04 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 1 Aug 2006 18:18:04 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
Message-ID: <20060801161803.GA20629@code0.codespeak.net>

Hi Lawrence,

On Tue, Aug 01, 2006 at 04:15:22PM +0200, Lawrence Oluyede wrote:
> > I have SSLv23_method which returns a pointer to SSL_METHOD (which is a
> > struct) and seems to dislike it.
> 
> I forgot to say that disliking means "not fixable" :(

Ah, oups, of course.  It prevents supporting functions that return other
functions, or structures with function pointers in them.  I was just bit
by the same restriction, when trying to cast a pointer to a function in
order to call it.  So indeed, it should be fixed.

BTW I just added some support for var-sized arrays, so that we can now
write:

    a = (c_int * n)()

to create an array of 'n' integers even if 'n' is not a compile-time
constant.


A bientot,

Armin

From arigo at tunes.org  Tue Aug  1 19:25:28 2006
From: arigo at tunes.org (Armin Rigo)
Date: Tue, 1 Aug 2006 19:25:28 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <20060801161803.GA20629@code0.codespeak.net>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
Message-ID: <20060801172528.GA27594@code0.codespeak.net>

Hi,

On Tue, Aug 01, 2006 at 06:18:04PM +0200, Armin Rigo wrote:
> Ah, oups, of course.  It prevents supporting functions that return other
> functions, or structures with function pointers in them.  I was just bit
> by the same restriction, when trying to cast a pointer to a function in
> order to call it.  So indeed, it should be fixed.

arigo -> off-line holidays, though, so it will have to wait next week :-/


A bientot,

Armin

From scott+pypy-dev at scottdial.com  Wed Aug  2 02:05:34 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Tue, 01 Aug 2006 20:05:34 -0400
Subject: [pypy-dev]
	small	patch	for	translator/c/src/ll_osdefs.h	for	win32
In-Reply-To: <2mvepcx4kr.fsf@starship.python.net>
References: <44C53C95.4070002@scottdial.com>	<9eebf5740607242145j5fe29b20t2b10de9c28747b33@mail.gmail.com>	<20060725093843.GA26167@code0.codespeak.net>	<44C6042D.7090806@scottdial.com>	<20060727090729.GA25278@code0.codespeak.net>	<44CA4C37.3060709@scottdial.com>	<20060729070549.GA22919@code0.codespeak.net>	<44CB7E75.5010805@scottdial.com>
	<44CEEADF.4090104@scottdial.com>
	<2mvepcx4kr.fsf@starship.python.net>
Message-ID: <44CFEC4E.70004@scottdial.com>

Michael Hudson wrote:
> How could 9am _Greenwich_ (a place in the UK) Mean Time be in the
> afternoon anywhere in Europe? :)

*blush* I should avoid writing emails at 2am (localtime). :-)

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From bea at changemaker.nu  Thu Aug  3 15:28:06 2006
From: bea at changemaker.nu (Beatrice During)
Date: Thu, 03 Aug 2006 15:28:06 +0200
Subject: [pypy-dev] Phase 2 report available!
Message-ID: <44D1F9E6.3090805@changemaker.nu>

Hi there

Our report on phase 2 of the project is now finished and is available on:

http://codespeak.net/pypy/dist/pypy/doc/index-report.html

This report covers work achieved during the period September 2005-May 
2006 (and a bit of June - the 0.9 release ;-). Although this report is 
written as part of the EU project we hope that many of you out there 
will enjoy reading it. Looking back to all those long months of 
check-ins and what we all actually accomplished it is not difficult to 
realise that it is the community around PyPy that makes this project 
tick - so thanks everyone out there!

A special thank you to Holger for major part of the technical writing,
Carl Friedrich for the funky metrics (and forever stuck in the swamp of 
ReST/pdf:s) and to Michael - always stuck in the dark hole of language 
reviews!

Cheers

Bea

From Ben.Young at risk.sungard.com  Mon Aug  7 16:35:21 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 7 Aug 2006 15:35:21 +0100
Subject: [pypy-dev] Bytecode counting patch
Message-ID: <OF912636EB.4AA82474-ON802571C3.004FBF20-802571C3.005022FE@risk.sungard.com>

Hi PyPys! 

Here's a patch that counts the number of times the different bytecodes are 
used. I haven't tried doing translation yet as I can't use the work 
machine cycles, but it can't be far off. I've put the counts on the object 
space as this is accessable from most places.

I've also implemented something that should resort the opcodes in the 
switch statement of the bytecode dispatcher according to the statistics I 
found. This may have a slight speed improvement.



Cheers,
Ben
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: diff.txt
Url: http://codespeak.net/pipermail/pypy-dev/attachments/20060807/80741122/attachment-0001.txt 

From cfbolz at gmx.de  Mon Aug  7 16:56:55 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 07 Aug 2006 16:56:55 +0200
Subject: [pypy-dev] Bytecode counting patch
In-Reply-To: <OF912636EB.4AA82474-ON802571C3.004FBF20-802571C3.005022FE@risk.sungard.com>
References: <OF912636EB.4AA82474-ON802571C3.004FBF20-802571C3.005022FE@risk.sungard.com>
Message-ID: <44D754B7.3030007@gmx.de>

Hi Ben!

Ben.Young at risk.sungard.com wrote:
> Here's a patch that counts the number of times the different bytecodes
are
> used. I haven't tried doing translation yet as I can't use the work
> machine cycles, but it can't be far off. I've put the counts on the
object
> space as this is accessable from most places.
>
> I've also implemented something that should resort the opcodes in the
> switch statement of the bytecode dispatcher according to the statistics I
> found. This may have a slight speed improvement.

Looks quite good. The only thing I don't really like is the fact that
the logbytecodes config options is in the global option name space.
Should probably be config.objspace.logbytecodes or even introduce an new
interpreter name space and be config.interpreter.logbytecodes

Cheers,

Carl Friedrich


From Ben.Young at risk.sungard.com  Mon Aug  7 17:00:26 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 7 Aug 2006 16:00:26 +0100
Subject: [pypy-dev] Bytecode counting patch
In-Reply-To: <44D754B7.3030007@gmx.de>
Message-ID: <OFA6F5FA72.5198D39E-ON802571C3.00522F87-802571C3.00526FFB@risk.sungard.com>

Carl Friedrich Bolz <cfbolz at gmx.de> wrote on 07/08/2006 15:56:55:

> Hi Ben!
> 
> Ben.Young at risk.sungard.com wrote:
> > Here's a patch that counts the number of times the different bytecodes
> are
> > used. I haven't tried doing translation yet as I can't use the work
> > machine cycles, but it can't be far off. I've put the counts on the
> object
> > space as this is accessable from most places.
> >
> > I've also implemented something that should resort the opcodes in the
> > switch statement of the bytecode dispatcher according to the 
statistics I
> > found. This may have a slight speed improvement.
> 
> Looks quite good. The only thing I don't really like is the fact that
> the logbytecodes config options is in the global option name space.
> Should probably be config.objspace.logbytecodes or even introduce an new
> interpreter name space and be config.interpreter.logbytecodes
> 

Thanks! I think it should have gone in objspace. It originally was, but 
then I moved the count onto ExecutionContext for a while, so I put it in 
global. Feel free to move it.

The other bit I'm not sure about it the reporting of the results. At the 
momement it's just a hack in py.py, but it should go in objspace.finish. 
However, I couldn't work out how to print from there.

Cheers,
Ben

> Cheers,
> 
> Carl Friedrich
> 
> 


From eric at vanrietpaap.nl  Mon Aug  7 18:35:17 2006
From: eric at vanrietpaap.nl (Eric van Riet Paap)
Date: Mon, 7 Aug 2006 18:35:17 +0200
Subject: [pypy-dev] Bytecode counting patch
In-Reply-To: <OF912636EB.4AA82474-ON802571C3.004FBF20-802571C3.005022FE@risk.sungard.com>
References: <OF912636EB.4AA82474-ON802571C3.004FBF20-802571C3.005022FE@risk.sungard.com>
Message-ID: <9EBBE66F-EFF0-4290-943C-7AA4AFE87F70@vanrietpaap.nl>

Hi Ben,

It's nice to know what opcodes are popular! However, sorting them in  
whatever order within a switch will likely not make a difference.
This switch will be converted to a jumptable, so the compiler will  
order the cases again.

cheers
Eric


On Aug 7, 2006, at 4:35 PM, Ben.Young at risk.sungard.com wrote:

> Hi PyPys!
>
> Here's a patch that counts the number of times the different  
> bytecodes are
> used. I haven't tried doing translation yet as I can't use the work
> machine cycles, but it can't be far off. I've put the counts on the  
> object
> space as this is accessable from most places.
>
> I've also implemented something that should resort the opcodes in the
> switch statement of the bytecode dispatcher according to the  
> statistics I
> found. This may have a slight speed improvement.
>
>
>
> Cheers,
> Ben
> <diff.txt>
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From mwh at python.net  Mon Aug  7 18:50:11 2006
From: mwh at python.net (Michael Hudson)
Date: Mon, 07 Aug 2006 17:50:11 +0100
Subject: [pypy-dev] Bytecode counting patch
References: <OF912636EB.4AA82474-ON802571C3.004FBF20-802571C3.005022FE@risk.sungard.com>
	<9EBBE66F-EFF0-4290-943C-7AA4AFE87F70@vanrietpaap.nl>
Message-ID: <2mmzagv6bg.fsf@starship.python.net>

Eric van Riet Paap <eric at vanrietpaap.nl> writes:

> Hi Ben,
>
> It's nice to know what opcodes are popular! However, sorting them in  
> whatever order within a switch will likely not make a difference.
> This switch will be converted to a jumptable, so the compiler will  
> order the cases again.

Unclear; similar rearrangements made a noticeable difference in
CPython a few years back (I think, anyway).

Cheers,
mwh

-- 
  <datazone> okay, tell me if i am crazy
  <Yosomono> you are
  <datazone> damn                               -- from Twisted.Quotes


From Ben.Young at risk.sungard.com  Tue Aug  8 08:57:55 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Tue, 8 Aug 2006 07:57:55 +0100
Subject: [pypy-dev] Bytecode counting patch
In-Reply-To: <9EBBE66F-EFF0-4290-943C-7AA4AFE87F70@vanrietpaap.nl>
Message-ID: <OF0DF5C058.CD866DEE-ON802571C4.002617F8-802571C4.0026430B@risk.sungard.com>

pypy-dev-bounces at codespeak.net wrote on 07/08/2006 17:35:17:

> Hi Ben,
> 
> It's nice to know what opcodes are popular! However, sorting them in 
> whatever order within a switch will likely not make a difference.
> This switch will be converted to a jumptable, so the compiler will 
> order the cases again.
> 
> cheers
> Eric
> 

Hi Eric,

It probably won't, but there appeared to be some desire for this on irc, 
and this is the kind of project I can fit into my lunch break! I think 
it's worth a try to see if it makes a difference. Unfortunately, I can't 
test it here.

Cheers,
Ben

> 
> On Aug 7, 2006, at 4:35 PM, Ben.Young at risk.sungard.com wrote:
> 
> > Hi PyPys!
> >
> > Here's a patch that counts the number of times the different 
> > bytecodes are
> > used. I haven't tried doing translation yet as I can't use the work
> > machine cycles, but it can't be far off. I've put the counts on the 
> > object
> > space as this is accessable from most places.
> >
> > I've also implemented something that should resort the opcodes in the
> > switch statement of the bytecode dispatcher according to the 
> > statistics I
> > found. This may have a slight speed improvement.
> >
> >
> >
> > Cheers,
> > Ben
> > <diff.txt>
> > _______________________________________________
> > pypy-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/pypy-dev
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From arigo at tunes.org  Wed Aug  9 14:28:04 2006
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 9 Aug 2006 14:28:04 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <20060801172528.GA27594@code0.codespeak.net>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
Message-ID: <20060809122804.GA30370@code0.codespeak.net>

Hi Lawrence,

rctypes should now support variables of "function pointer" types, so
this problem with _ssl and bzip2 should be fixed.

Congrats on finishing the ctypes implementation of all your modules, btw :-)


Armin

From arigo at tunes.org  Wed Aug  9 14:36:40 2006
From: arigo at tunes.org (Armin Rigo)
Date: Wed, 9 Aug 2006 14:36:40 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <20060809122804.GA30370@code0.codespeak.net>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
	<20060809122804.GA30370@code0.codespeak.net>
Message-ID: <20060809123640.GA31312@code0.codespeak.net>

Re-hi,

Ouch.  _ssl is full of ctypes Unions, which rctypes doesn't support at
all...  More hacks in sight.  Maybe we can handle the unions as
structures with a special hint that forces the C code generator to write
'union' instead of 'struct' in their declaration...


A bientot,

Armin

From l.oluyede at gmail.com  Thu Aug 10 15:13:20 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 10 Aug 2006 15:13:20 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <20060809123640.GA31312@code0.codespeak.net>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
	<20060809122804.GA30370@code0.codespeak.net>
	<20060809123640.GA31312@code0.codespeak.net>
Message-ID: <9eebf5740608100613k4f4803c0p14dd09e1efecf533@mail.gmail.com>

> Ouch.  _ssl is full of ctypes Unions, which rctypes doesn't support at
> all...  More hacks in sight.  Maybe we can handle the unions as
> structures with a special hint that forces the C code generator to write
> 'union' instead of 'struct' in their declaration...

So do you mean treat unions as struct at the rctypes level ? That
could be simply making rctypes recognizing "Union" base class as an
alias of Structure?

-- 
Lawrence
http://www.oluyede.org/blog

From arigo at tunes.org  Fri Aug 11 11:32:28 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 11 Aug 2006 11:32:28 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <9eebf5740608100613k4f4803c0p14dd09e1efecf533@mail.gmail.com>
References: <9eebf5740607290658w2dba6c96n38e5ab6d61b5c41b@mail.gmail.com>
	<20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
	<20060809122804.GA30370@code0.codespeak.net>
	<20060809123640.GA31312@code0.codespeak.net>
	<9eebf5740608100613k4f4803c0p14dd09e1efecf533@mail.gmail.com>
Message-ID: <20060811093227.GA28992@code0.codespeak.net>

Hi Lawrence,

On Thu, Aug 10, 2006 at 03:13:20PM +0200, Lawrence Oluyede wrote:
> So do you mean treat unions as struct at the rctypes level ? That
> could be simply making rctypes recognizing "Union" base class as an
> alias of Structure?

Yes.  After all, as far as I can tell, in C a 'union' is exactly the
same as a 'struct' apart from one detail: all fields start at offset 0
in the 'union' case, whereas they are one after the other in the
'struct' case.


A bientot,

Armin

From arigo at tunes.org  Fri Aug 11 12:59:43 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 11 Aug 2006 12:59:43 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <20060811093227.GA28992@code0.codespeak.net>
References: <20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010609t506bacdep7631bd0ff3f74c90@mail.gmail.com>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
	<20060809122804.GA30370@code0.codespeak.net>
	<20060809123640.GA31312@code0.codespeak.net>
	<9eebf5740608100613k4f4803c0p14dd09e1efecf533@mail.gmail.com>
	<20060811093227.GA28992@code0.codespeak.net>
Message-ID: <20060811105943.GA5327@code0.codespeak.net>

Hi,

Ok, unions should now compile correctly.

The next problem with "compilemodule.py _ssl" is that some structs have
no _fields_ declaration (they are opaque).  I guess that we could
consider this case as equivalent to "_fields_ = []", for structs that
are already declared in C headers anyway.  GenC doesn't try to produce a
new declaration for these structures.


A bientot,

Armin

From l.oluyede at gmail.com  Fri Aug 11 14:15:51 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Fri, 11 Aug 2006 14:15:51 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <20060811105943.GA5327@code0.codespeak.net>
References: <20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010620w34abe276m2dca94ea35a7e63e@mail.gmail.com>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
	<20060809122804.GA30370@code0.codespeak.net>
	<20060809123640.GA31312@code0.codespeak.net>
	<9eebf5740608100613k4f4803c0p14dd09e1efecf533@mail.gmail.com>
	<20060811093227.GA28992@code0.codespeak.net>
	<20060811105943.GA5327@code0.codespeak.net>
Message-ID: <9eebf5740608110515n2750bfe5u2bd27a8fbfcc17e3@mail.gmail.com>

> The next problem with "compilemodule.py _ssl" is that some structs have
> no _fields_ declaration (they are opaque).  I guess that we could
> consider this case as equivalent to "_fields_ = []", for structs that
> are already declared in C headers anyway.  GenC doesn't try to produce a
> new declaration for these structures.

I think that could be fixed by the code generator that generates that
fields. I should talk to Thomas Heller to know what he can do about
that. Meanwhile I can fix that manually and see what comes next.

-- 
Lawrence
http://www.oluyede.org/blog

From l.oluyede at gmail.com  Fri Aug 11 14:24:12 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Fri, 11 Aug 2006 14:24:12 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <9eebf5740608110515n2750bfe5u2bd27a8fbfcc17e3@mail.gmail.com>
References: <20060801110659.GA21064@code0.codespeak.net>
	<9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
	<20060809122804.GA30370@code0.codespeak.net>
	<20060809123640.GA31312@code0.codespeak.net>
	<9eebf5740608100613k4f4803c0p14dd09e1efecf533@mail.gmail.com>
	<20060811093227.GA28992@code0.codespeak.net>
	<20060811105943.GA5327@code0.codespeak.net>
	<9eebf5740608110515n2750bfe5u2bd27a8fbfcc17e3@mail.gmail.com>
Message-ID: <9eebf5740608110524u301bd860ucdd8234f1aef4b7a@mail.gmail.com>

> I think that could be fixed by the code generator that generates that
> fields. I should talk to Thomas Heller to know what he can do about
> that. Meanwhile I can fix that manually and see what comes next.

I added empty fields, now I have a:

[flowgraph] (pypy.rpython.lltypesystem.rrange:85)ll_rangeiter
[annrpython] <FunctionGraph of
(pypy.rpython.lltypesystem.rrange:85)ll_rangeiter__Ptr_GcStruct_rangeLlT_rangePtr
at 0x14f0198> -> SomePtr(ll_ptrtype=<* GcStruct range { next: Signed,
stop: Signed }>)
[translation:ERROR] Error:
[translation:ERROR]  KeyError: <weakref at 0x81e0; to 'type' at
0x2e3180 (NoneType)>


-- 
Lawrence
http://www.oluyede.org/blog

From mwh at python.net  Fri Aug 11 17:49:50 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 11 Aug 2006 16:49:50 +0100
Subject: [pypy-dev] fun with dictionaries
Message-ID: <2mk65ftgpt.fsf@starship.python.net>

So part of WP06 is improving the dictionary implementation.  To do
this, it seemed like a good idea to find out what Python code does
with dictionaries, which is what I've been working on this week.

If you activate the "objspace.std.withmultidict" option, make
MEASURE_DICT in pypy/objspace/std/dictmultiobject.py true and build
yourself a pypy-c, you'll find that this pypy-c will create a
dictinfo.txt file that summarizes how every dictionary in the program
has been used.

The benchmark programs I have been using are: pystone, richards,
"rst2html coding-guide.txt" and "translate.py --backendopt
--no-compile --batch --text targetrpystonedalone.py", and the
(compressed) results can be found in:

    http://codespeak.net/~mwh/dictinfo/

The file dictinfos.tar.bz2 contains the dictinfo.txt files created by
the above runs, and the RData files are binary files suitable for
loading into R:

    http://www.r-project.org/

What I'd like to get some input on is stuff like: what aspects of this
data should I analyse?  Is there any data you think I should collect?

Something that I don't measure at all is the order things happen in,
which might be interesting: it's easy to believe many dictionaries go
through a phase of being written to before a longer pahse of being
read from.  But I'm not sure how to measure that...

Cheers,
mwh

-- 
  <glyph> I am *not* a PSU agent.
                                                -- from Twisted.Quotes


From mwh at python.net  Fri Aug 11 18:30:21 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 11 Aug 2006 17:30:21 +0100
Subject: [pypy-dev] fun with dictionaries
References: <2mk65ftgpt.fsf@starship.python.net>
Message-ID: <2mfyg3teua.fsf@starship.python.net>

Michael Hudson <mwh at python.net> writes:

> The benchmark programs I have been using are: pystone, richards,
> "rst2html coding-guide.txt" and "translate.py --backendopt
> --no-compile --batch --text targetrpystonedalone.py", and the
> (compressed) results can be found in:
>
>     http://codespeak.net/~mwh/dictinfo/
>
> The file dictinfos.tar.bz2 contains the dictinfo.txt files created by
> the above runs, and the RData files are binary files suitable for
> loading into R:

Unfortunately, the RData files aren't quite right.  I'll fix them soon
(though it would be sooner if pypy2 reappeared again quickly...).

Cheers,
mwh

-- 
  Screaming 14-year-old boys attempting to prove to each other that
  they are more 3133t than j00.
         -- Reason #8 for quitting slashdot today, from
            http://www.cs.washington.edu/homes/klee/misc/slashdot.html


From mwh at python.net  Fri Aug 11 19:18:26 2006
From: mwh at python.net (Michael Hudson)
Date: Fri, 11 Aug 2006 18:18:26 +0100
Subject: [pypy-dev] fun with dictionaries
References: <2mk65ftgpt.fsf@starship.python.net>
	<2mfyg3teua.fsf@starship.python.net>
Message-ID: <2mbqqrtcm5.fsf@starship.python.net>

Michael Hudson <mwh at python.net> writes:

> Michael Hudson <mwh at python.net> writes:
>
>> The benchmark programs I have been using are: pystone, richards,
>> "rst2html coding-guide.txt" and "translate.py --backendopt
>> --no-compile --batch --text targetrpystonedalone.py", and the
>> (compressed) results can be found in:
>>
>>     http://codespeak.net/~mwh/dictinfo/
>>
>> The file dictinfos.tar.bz2 contains the dictinfo.txt files created by
>> the above runs, and the RData files are binary files suitable for
>> loading into R:
>
> Unfortunately, the RData files aren't quite right.  I'll fix them soon
> (though it would be sooner if pypy2 reappeared again quickly...).

Should be better now.

Cheers,
mwh

-- 
  I've even been known to get Marmite *near* my mouth -- but never
  actually in it yet.  Vegamite is right out.
 UnicodeError: ASCII unpalatable error: vegamite found, ham expected
                                       -- Tim Peters, comp.lang.python


From arigo at tunes.org  Fri Aug 11 19:33:23 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 11 Aug 2006 19:33:23 +0200
Subject: [pypy-dev] make _ssl module compile
In-Reply-To: <9eebf5740608110524u301bd860ucdd8234f1aef4b7a@mail.gmail.com>
References: <9eebf5740608010715j279d31dby9c4173cd4ba17773@mail.gmail.com>
	<20060801161803.GA20629@code0.codespeak.net>
	<20060801172528.GA27594@code0.codespeak.net>
	<20060809122804.GA30370@code0.codespeak.net>
	<20060809123640.GA31312@code0.codespeak.net>
	<9eebf5740608100613k4f4803c0p14dd09e1efecf533@mail.gmail.com>
	<20060811093227.GA28992@code0.codespeak.net>
	<20060811105943.GA5327@code0.codespeak.net>
	<9eebf5740608110515n2750bfe5u2bd27a8fbfcc17e3@mail.gmail.com>
	<9eebf5740608110524u301bd860ucdd8234f1aef4b7a@mail.gmail.com>
Message-ID: <20060811173323.GA14768@code0.codespeak.net>

Hi Lawrence,

On Fri, Aug 11, 2006 at 02:24:12PM +0200, Lawrence Oluyede wrote:
> [translation:ERROR] Error:
> [translation:ERROR]  KeyError: <weakref at 0x81e0; to 'type' at
> 0x2e3180 (NoneType)>

My mistake.  Fixed in rev 31266.


Armin

From santagada at gmail.com  Fri Aug 11 23:59:00 2006
From: santagada at gmail.com (Leonardo Santagada)
Date: Fri, 11 Aug 2006 18:59:00 -0300
Subject: [pypy-dev] Summer of PyPy pre-proposal
Message-ID: <2f2e5f950608111459p4f0d511er7373de5b26394b75@mail.gmail.com>

I would like to enter Summer of Pypy with my proposal to make a
javascript interpreter. I had a chat with some nice people in #pypy
and they thought it was a good idea. So I checked the code and started
looking at what should/can do on 2 months of work ( I don't know if
there is this limitation on time). I am a litle lost with all the code
from pypy and the idea of making another interpreter so I am posting
here my ideas and I am specting your feedback.

What do you think I need to do? define a javascript virtual machine or
do simple direct interpretation of the source?

Should I use the parser defined on pyparsing (this one seems really
nice) or write one manually?

What I am certain is that it need to have an Interpreter so people can
play with it.

I should probably write more, but I want to know what you all think
about this proposal.

ps: I had another idea of translating readline to ctype and make it
work on windows also like the one in ipython and also translating
sqlite so people can try to run their webframeworks on pypy, but the
js interpreter seems more important.

-- 
Leonardo Santagada (http://www.lomohomes.com/retype)

N?o se preocupe com o que os outros v?o fazer. O melhor jeito de
prever o futuro ? inventa-lo.
- Alan Kay

From len-l at telus.net  Sat Aug 12 03:50:31 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Fri, 11 Aug 2006 18:50:31 -0700
Subject: [pypy-dev] Summer of PyPy pre-proposal
In-Reply-To: <2f2e5f950608111459p4f0d511er7373de5b26394b75@mail.gmail.com>
Message-ID: <44DCD177.28255.10875BF@len-l.telus.net>

On 11 Aug 2006 at 18:59, Leonardo Santagada wrote:

> I would like to enter Summer of Pypy with my proposal to make a
> javascript interpreter. I had a chat with some nice people in #pypy
> and they thought it was a good idea. So I checked the code and started
> looking at what should/can do on 2 months of work ( I don't know if
> there is this limitation on time). I am a litle lost with all the code
> from pypy and the idea of making another interpreter so I am posting
> here my ideas and I am specting your feedback.
> 
> What do you think I need to do? define a javascript virtual machine or
> do simple direct interpretation of the source?
> 
Maybe javascript can compile to Python byte code.

Lenard Lindstrom
<len-l at telus.net>



From arigo at tunes.org  Sat Aug 12 10:09:01 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 12 Aug 2006 10:09:01 +0200
Subject: [pypy-dev] Summer of PyPy pre-proposal
In-Reply-To: <44DCD177.28255.10875BF@len-l.telus.net>
References: <2f2e5f950608111459p4f0d511er7373de5b26394b75@mail.gmail.com>
	<44DCD177.28255.10875BF@len-l.telus.net>
Message-ID: <20060812080901.GA29264@code0.codespeak.net>

Hi Lenard,

On Fri, Aug 11, 2006 at 06:50:31PM -0700, Lenard Lindstrom wrote:
> Maybe javascript can compile to Python byte code.

No, that wouldn't work.  The two language's object models are different,
as is always the case between dynamic languages, so you cannot just map
the syntax of one to the bytecode of the other.


Armin

From arigo at tunes.org  Sat Aug 12 10:45:31 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 12 Aug 2006 10:45:31 +0200
Subject: [pypy-dev] Summer of PyPy pre-proposal
In-Reply-To: <2f2e5f950608111459p4f0d511er7373de5b26394b75@mail.gmail.com>
References: <2f2e5f950608111459p4f0d511er7373de5b26394b75@mail.gmail.com>
Message-ID: <20060812084530.GB29264@code0.codespeak.net>

Hi Leonardo,

On Fri, Aug 11, 2006 at 06:59:00PM -0300, Leonardo Santagada wrote:
> I would like to enter Summer of Pypy with my proposal to make a
> javascript interpreter. I had a chat with some nice people in #pypy
> and they thought it was a good idea.

Nice plan :-)

> So I checked the code and started
> looking at what should/can do on 2 months of work ( I don't know if
> there is this limitation on time).

Not necessarily, but the proposal should have a timeline and your
calendar should allow you to put "reasonable" amounts of work on it.

> I am a litle lost with all the code
> from pypy and the idea of making another interpreter so I am posting
> here my ideas and I am specting your feedback.
> 
> What do you think I need to do? define a javascript virtual machine or
> do simple direct interpretation of the source?

I'm not completely clear about the difference between a simple direct
interpreter and a virtual machine.  JavaScript like Python is a language
where the main work is in getting the execution model and the object
model right.  For the object model I mean building the Python classes
that support the various kinds of JavaScript objects -- "kind" as in the
[[Class]] internal property -- with all the other internal properties
and methods; and then making the pre-built objects and prototypes and
functions.  It's ok to leave out some parts of the library of pre-built
objects for the Summer of PyPy's duration.  (This part corresponds to
PyPy's "object spaces".)

For the execution model, as you note, you need to start with an
interpreter.  It would interpret bytecodes that you are free to invent;
I guess they would directly correspond to JavaScript syntax elements.
(What I mean by execution model also includes things like function and
method calls, and how to fit built-in functions.)  (This part is
documented under the name "bytecode interpreter" in PyPy.)

(JavaScript ref.:
http://www.ecma-international.org/publications/standards/Ecma-262.htm)

> Should I use the parser defined on pyparsing (this one seems really
> nice) or write one manually?

I'm not sure about this.  Our tokenizer and parser are definitely meant
to be reusable, but I'm not exactly sure how :-/  I think there is a way
to recompile the tables in pytokenize.py.  The parser loads a grammar
dynamically, so that's simpler.  My 2 cents, though, is that you don't
start the work here; if you know about another JavaScript parser that
produces some data that is easy to feed to Python, it would be a good
start.  (Ultimately, we really need a parser written in RPython, but
it's ok to leave this out of the Summer of PyPy proposal.)


A bientot,

Armin

From len-l at telus.net  Sat Aug 12 20:54:42 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Sat, 12 Aug 2006 11:54:42 -0700
Subject: [pypy-dev] Summer of PyPy pre-proposal
In-Reply-To: <20060812080901.GA29264@code0.codespeak.net>
References: <44DCD177.28255.10875BF@len-l.telus.net>
Message-ID: <44DDC182.899.B56587@len-l.telus.net>

Hi Armin,

I must be missing something here. Isn't Python's object model defined 
by the type objects themselves?  So implement a javascript prototype 
type. Create the necessary javascript root objects and import them 
into the __builtin__ namespace or something. I would expect any 
serious problems would involve identifier scope or exception 
handling, language qualities very much tied to the bytecode 
interpreter framework.

On 12 Aug 2006 at 10:09, Armin Rigo wrote:

> Hi Lenard,
> 
> On Fri, Aug 11, 2006 at 06:50:31PM -0700, Lenard Lindstrom wrote:
> > Maybe javascript can compile to Python byte code.
> 
> No, that wouldn't work.  The two language's object models are different,
> as is always the case between dynamic languages, so you cannot just map
> the syntax of one to the bytecode of the other.
> 

Lenard Lindstrom
<len-l at telus.net>



From scott+pypy-dev at scottdial.com  Sun Aug 13 12:58:36 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Sun, 13 Aug 2006 06:58:36 -0400
Subject: [pypy-dev] Lingering issues w/my pypy tester
Message-ID: <44DF05DC.4090202@scottdial.com>

Whether any of these are "bugs" or not is up to you, but they are 
nuissances for my test runs:

* module._stackless.test.test_choicepoint

http://scottdial.com/pypytest/31094/failed/module._stackless.test.test_choicepoint.py.test_choicepoint.html

This test gets stalled while running a subprocess (testing_1.exe) and I 
must manually kill this process for the testing to continue. I haven't 
been able to identify the problem.

* translator.c.test.test_stackless.TestStackless().test_stack_too_big

http://scottdial.com/pypytest/31094/failed/translator.c.test.test_stackless.py.TestStacklesstest_stack_too_big.html

This test causes the boehm gc library to throw and catch an exception 
"Too many heap sections". I suspect this means that the test has indeed 
failed. However, this assertion from the gc library is modal and 
blocking, so it stalls the test run until human intervention clicks 
"ok". I haven't looked deeply into this one.

Otherwise, none of the failures require human involvement to get things 
going again, but there are plenty of other tests which are erroneous 
failures.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From arigo at tunes.org  Mon Aug 14 13:38:06 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 14 Aug 2006 13:38:06 +0200
Subject: [pypy-dev] Summer of PyPy pre-proposal
In-Reply-To: <44DDC182.899.B56587@len-l.telus.net>
References: <44DCD177.28255.10875BF@len-l.telus.net>
	<44DDC182.899.B56587@len-l.telus.net>
Message-ID: <20060814113806.GB9100@code0.codespeak.net>

Hi Lenard,

On Sat, Aug 12, 2006 at 11:54:42AM -0700, Lenard Lindstrom wrote:
> I must be missing something here. Isn't Python's object model defined 
> by the type objects themselves?

I was thinking about the way all objects work; the type objects only
define methods and properties of their instances, but not *what* a
method or a property is.  In Python, the "object model" contains a lot
of logic about what special method calls are triggered by "a+b", and how
an attribute lookup is performed.  In JavaScript the object model is far
simpler: it is described by a number of internal properties that objects
have, like [[Prototype]], and how these properties are used in various
operations like attribute lookup.

It needs to be done, but that's not necessarily difficult indeed.  It's
possible that other aspects will be more irksome.  Nevertheless, it
means that you can never really reuse the bytecode of one dynamic
language to implement another one, because the "object models" are
typically quite different, in the sense above.  Most bytecodes -
GETATTR, BINARY_ADD, etc. - do things in Python that are completely
unrelated to what the equivalent bytecode would need to do in
JavaScript, even before we consider type objects.


A bientot,

Armin

From arigo at tunes.org  Mon Aug 14 13:42:11 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 14 Aug 2006 13:42:11 +0200
Subject: [pypy-dev] Lingering issues w/my pypy tester
In-Reply-To: <44DF05DC.4090202@scottdial.com>
References: <44DF05DC.4090202@scottdial.com>
Message-ID: <20060814114211.GC9100@code0.codespeak.net>

Hi Scott,

On Sun, Aug 13, 2006 at 06:58:36AM -0400, Scott Dial wrote:
> Whether any of these are "bugs" or not is up to you, but they are 
> nuissances for my test runs:

We'll try to look into them, but for now, feel free to skip them.
(You can probably just drop a py.test.skip("XXX") in the tests in your
automated runner's working copy.)


A bientot,

Armin

From arigo at tunes.org  Thu Aug 17 10:57:11 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 17 Aug 2006 10:57:11 +0200
Subject: [pypy-dev] Self-hosted
Message-ID: <20060817085711.GA17222@code0.codespeak.net>

Hi all,

A few days ago we got our first version of pypy-c compiled with pypy-c.
At least it produced the C code before distutils crashed because of our
incomplete posix module; so you have to say

    cd pypy/translator/goal
    pypy-c --text --source translate.py
    cd /tmp/usession-yourname/testing_1
    make

So now we're self-hosted :-)


A bientot,

Armin

From seojiwon at gmail.com  Fri Aug 18 03:31:13 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Thu, 17 Aug 2006 18:31:13 -0700
Subject: [pypy-dev] Self-hosted
In-Reply-To: <20060817085711.GA17222@code0.codespeak.net>
References: <20060817085711.GA17222@code0.codespeak.net>
Message-ID: <b008462b0608171831x2139fe14hc9560f0eac04c003@mail.gmail.com>

Great!

On 8/17/06, Armin Rigo <arigo at tunes.org> wrote:
> Hi all,
>
> A few days ago we got our first version of pypy-c compiled with pypy-c.
> At least it produced the C code before distutils crashed because of our
> incomplete posix module; so you have to say
>
>     cd pypy/translator/goal
>     pypy-c --text --source translate.py
>     cd /tmp/usession-yourname/testing_1
>     make
>
> So now we're self-hosted :-)
>
>
> A bientot,
>
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

From faassen at infrae.com  Fri Aug 18 10:58:38 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Fri, 18 Aug 2006 10:58:38 +0200
Subject: [pypy-dev] Self-hosted
In-Reply-To: <20060817085711.GA17222@code0.codespeak.net>
References: <20060817085711.GA17222@code0.codespeak.net>
Message-ID: <44E5813E.9000709@infrae.com>

Armin Rigo wrote:
[snip]
> So now we're self-hosted :-)

Awesome! I like little (or bigger) progress reports like this. :)

Regards,

Martijn

From l.oluyede at gmail.com  Sun Aug 20 02:44:57 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Sun, 20 Aug 2006 02:44:57 +0200
Subject: [pypy-dev] Modules on Windows
Message-ID: <9eebf5740608191744w6f553703p846517512236fef5@mail.gmail.com>

I managed to port rctime and mmap on Windows. So there's a little bit
more of pypy on Windows platform. I can't tell you how much pain was
to configure Windows to compile but I'm sure you all can understand
:-)

-- 
Lawrence
http://www.oluyede.org/blog

From arigo at tunes.org  Sun Aug 20 15:49:23 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 20 Aug 2006 15:49:23 +0200
Subject: [pypy-dev] Modules on Windows
In-Reply-To: <9eebf5740608191744w6f553703p846517512236fef5@mail.gmail.com>
References: <9eebf5740608191744w6f553703p846517512236fef5@mail.gmail.com>
Message-ID: <20060820134923.GA17301@code0.codespeak.net>

Hi Lawrence,

On Sun, Aug 20, 2006 at 02:44:57AM +0200, Lawrence Oluyede wrote:
> I managed to port rctime and mmap on Windows. So there's a little bit
> more of pypy on Windows platform. I can't tell you how much pain was
> to configure Windows to compile but I'm sure you all can understand
> :-)

Great news :-)


Armin

From alexandre.fayolle at logilab.fr  Tue Aug 22 16:35:05 2006
From: alexandre.fayolle at logilab.fr (Alexandre Fayolle)
Date: Tue, 22 Aug 2006 16:35:05 +0200
Subject: [pypy-dev] [jinty@web.de: [Zope3-dev] zope.interface using PyPy's
	extension compiler]
Message-ID: <20060822143505.GE4823@crater.logilab.fr>

Hi everyone,

I hope you're having fun in Ireland. 

I don't remember seing the information below on this list, and thought
it could interest people to know of other's efforts to use pypy. 

Cheers, 

----- Forwarded message from Brian Sutherland <jinty at web.de> -----

From: Brian Sutherland <jinty at web.de>
To: Zope3-dev <zope3-dev at zope.org>
Date: Mon, 21 Aug 2006 16:57:06 +0200
Subject: [Zope3-dev] zope.interface using PyPy's extension compiler

At EuroPython I investigated re-implementing a class from zope.interface
in PyPy's Restricted Python. I compiled it to C and then compared it to
the Pure Python implementation and handcrafted C optimization.

Not having touched PyPy before, I found it extreemly interesting. The
report and code I wrote is available here for anyone else interested:
http://lentejasverdes.ath.cx/stuff/pypy/europython_2006/README.html

-- 
Brian Sutherland



----- End forwarded message -----

-- 
Alexandre Fayolle                              LOGILAB, Paris (France)
Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
Informatique scientifique:               http://www.logilab.fr/science
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 481 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060822/2064b6e6/attachment.pgp 

From jinty at web.de  Tue Aug 22 17:05:56 2006
From: jinty at web.de (Brian Sutherland)
Date: Tue, 22 Aug 2006 17:05:56 +0200
Subject: [pypy-dev] [jinty@web.de: [Zope3-dev] zope.interface using
	PyPy's extension compiler]
In-Reply-To: <20060822143505.GE4823@crater.logilab.fr>
References: <20060822143505.GE4823@crater.logilab.fr>
Message-ID: <20060822150556.GC7145@minipas.home>

On Tue, Aug 22, 2006 at 04:35:05PM +0200, Alexandre Fayolle wrote:
> Hi everyone,
> 
> I hope you're having fun in Ireland. 
> 
> I don't remember seing the information below on this list, and thought
> it could interest people to know of other's efforts to use pypy. 

Actually, I was going to post it on this list "Real Soon". But thanks
for doing it for me;)

> Cheers, 
> 
> ----- Forwarded message from Brian Sutherland <jinty at web.de> -----
> 
> From: Brian Sutherland <jinty at web.de>
> To: Zope3-dev <zope3-dev at zope.org>
> Date: Mon, 21 Aug 2006 16:57:06 +0200
> Subject: [Zope3-dev] zope.interface using PyPy's extension compiler
> 
> At EuroPython I investigated re-implementing a class from zope.interface
> in PyPy's Restricted Python. I compiled it to C and then compared it to
> the Pure Python implementation and handcrafted C optimization.
> 
> Not having touched PyPy before, I found it extreemly interesting. The
> report and code I wrote is available here for anyone else interested:
> http://lentejasverdes.ath.cx/stuff/pypy/europython_2006/README.html
> 
> -- 
> Brian Sutherland
> 
> 
> 
> ----- End forwarded message -----
> 
> -- 
> Alexandre Fayolle                              LOGILAB, Paris (France)
> Formations Python, Zope, Plone, Debian:  http://www.logilab.fr/formations
> D?veloppement logiciel sur mesure:       http://www.logilab.fr/services
> Informatique scientifique:               http://www.logilab.fr/science



> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev

-- 
Brian Sutherland

Metropolis - "it's the first movie with a robot. And she's a woman.
              And she's EVIL!!"

From faassen at infrae.com  Wed Aug 23 12:51:15 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Wed, 23 Aug 2006 12:51:15 +0200
Subject: [pypy-dev] practical experiences with PyPy's extension compiler
In-Reply-To: <20060822150556.GC7145@minipas.home>
References: <20060822143505.GE4823@crater.logilab.fr>
	<20060822150556.GC7145@minipas.home>
Message-ID: <44EC3323.90301@infrae.com>

Hey,

As someone who suggested to Brian he try something like this with PyPy, 
I just wanted to highlight this. To make sure the PyPy developer's 
brains don't turn off when they hear Zope, that's not important bit, so 
I changed the title. :) Please all read Brian's entry!

Brian Sutherland wrote:
[snip]

 > At EuroPython I investigated re-implementing a class from
 > zope.interface in PyPy's Restricted Python. I compiled it to C
 > and then compared it to the Pure Python implementation and
 > handcrafted C optimization.
 >
 > Not having touched PyPy before, I found it extremely interesting. The
 > report and code I wrote is available here for anyone else interested:
 > http://lentejasverdes.ath.cx/stuff/pypy/europython_2006/README.html

The point of Brian's experiment from PyPy's perspective is that you get 
feedback on PyPy's utility from a developer who is not primarily engaged 
with the development of programming language technology. I think that's 
good for the PyPy project.

I think the following strategy has a lot of potential getting PyPy 
technology to the "masses":

* write your module in RPython.

* use (R)CTypes if you want to bind to existing C libraries.

* promise performance as this module can compile to a CPython C extension.

* promise maintainability and testability as this module should be able 
to run in straight Python (+ CTypes). This is one of the main areas 
where this strategy would give practical benefits over technology like 
Pyrex.

* and actually all these people are making their extension modules ready 
for PyPy too.

I know that this is your strategy, I'm just repeating it to let the echo 
sink in. And feedback on the experience from people like Brian may give 
vital information on how to improve the experience... In this case, 
Brian's feedback (also personally on irc) indicates to me that the above 
points have not yet been reached entirely:

* writing RPython is rather difficult as you need to keep the various 
levels straight. This unfortunately hurts the maintainability point.

* in this case, the promised performance wasn't delivered, unfortunately.

* as far as I understood it, in this case, the code wasn't directly 
testable in CPython after the rewriting to RPython was done. This hurts 
the testability case.

(my apologies of any of this is wrong; Brian would know better of course)

I'm really excited about the potential here, so I hope this can be the 
start of a longer dialog between PyPy developers and application developers.

Now I'm done highlighting this. :)

Regards,

Martijn

From bea at changemaker.nu  Wed Aug 23 15:33:38 2006
From: bea at changemaker.nu (Beatrice During)
Date: Wed, 23 Aug 2006 15:33:38 +0200
Subject: [pypy-dev] Sprint reports 2003-2006
Message-ID: <44EC5932.5040901@changemaker.nu>

Hi there

We now have a page linking to all sprint reports from PyPy sprints
since the project started 2003 up to present time.

http://codespeak.net/pypy/dist/pypy/doc/sprint-reports.html

There are 22 (!!) sprints during this period and we have reports from 16 
of them (soon - after the ongoing Limerick sprint 17 ;-).

This page should help people to better understand the progress of the 
project - so enjoy a stroll down the PyPy memory lane.

Thanks Michael for helping out with this.

Cheers

Bea

From jacob at strakt.com  Wed Aug 23 18:15:28 2006
From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=)
Date: Wed, 23 Aug 2006 18:15:28 +0200
Subject: [pypy-dev] Sprint reports 2003-2006
In-Reply-To: <44EC5932.5040901@changemaker.nu>
References: <44EC5932.5040901@changemaker.nu>
Message-ID: <200608231815.33714.jacob@strakt.com>

Great work!

I think we should try to reconstruct at least stub reports for the remaining 
sprints. Here are some notes on what happened at the ones I attended. This is 
in no way complete (and fairly Jacob focused, since I remember what I did, 
but have a hazy understanding of what others did):


Gothenburg (May 2003)
===================
The major focus of the sprint was to make the interpreter chain complete. A 
first goal was to make "Hello world" run and this was accomplished about half 
way into the sprint. We then proceded to making a more complex program that 
had operations involving string operations and dictionaries work. This was 
accomplished before the end of the sprint.

http://www.strakt.com/~jacob/pypy/img_0062.jpg

There was a big discussion, mostly involving Armin and Samuele over how to 
handle operator dispatching. This ended in the Multimethod dispatcher that 
mostly still survives in the Standard Object Space.

http://www.strakt.com/~jacob/pypy/img_0064.jpg
http://www.strakt.com/~jacob/pypy/img_0065.jpg
http://www.strakt.com/~jacob/pypy/img_0066.jpg

People I remember being there: Armin, Christian, Samuele, Michael, Alex 
Martelli, Anna Ravenscroft, Laura, Tomek Meka

Europython/Gothenburg (June 2004)
=============================
The sprint happened before the conference, and I'm rather hazy on what 
actually happened since I spent almost all my time on conference 
preparations. I'm rather certain Christian, Holger, Armin and Anders Lehman 
were there. Maybe they can fill in details. (To jog your memories, you were 
sitting in a room with green doors, skylights and no windows in the 
IT-support department).

Vilnius (Nov 2004)
===============
I don't really remember what others did at this sprint, but this is where 
Laura and I wrote the file I/O handling.

People I remember being there: Marius Gedminas, Bob Ippolito, Christian, 
Armin.

Leysin (April 2006)
===============
Armin, Arre and Samuele started with 2 days working on the JIT. When others 
joined the sprint focus changed and there was a bunch of work on fixing 
translation to work with stackless features and other stuff. There was also 
some work done on the constraints package.

People I remember being there:
Armin, Arre, Samuele, Holger, Michael, Alexandre, Aurelien

Europython/Gothenburg (June 2005)
=============================
Apart from the presprint, which is already reoprted, there was an open 
post.-Europython sprint. At the open sprint there were 24 people on the first 
day and activities happened on may fronts. A large group of people focused on 
porting PyPy to support Python2.4. This work was 3/4 finished by the end of 
the sprint. Christian and Jacob worked on support for floats and Jacob did a 
bunch of work on binascii and related modules. I think there was also a bunch 
of work going into annotation, translation and the C and llvm backends. Nick 
worked on his sre Summer of Code project. There was also work being done on 
the AST compiler.

People I remember being there: All the regulars plus Anders Qvist and Marius 
Gedminas.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060823/5cc8359f/attachment-0001.pgp 

From marius at pov.lt  Wed Aug 23 18:25:52 2006
From: marius at pov.lt (Marius Gedminas)
Date: Wed, 23 Aug 2006 19:25:52 +0300
Subject: [pypy-dev] Sprint reports 2003-2006
In-Reply-To: <200608231815.33714.jacob@strakt.com>
References: <44EC5932.5040901@changemaker.nu>
	<200608231815.33714.jacob@strakt.com>
Message-ID: <20060823162552.GA16202@fridge.pov.lt>

On Wed, Aug 23, 2006 at 06:15:28PM +0200, Jacob Hall?n wrote:
> Vilnius (Nov 2004)
> ===============
> I don't really remember what others did at this sprint, but this is where 
> Laura and I wrote the file I/O handling.
> 
> People I remember being there: Marius Gedminas, Bob Ippolito, Christian, 
> Armin.

IIRC everyone was overjoyed when PyPy managed to completely translate
into C and compute the Ultimate Answer to Life, the Universe, and
Everything (def somefunc(): return -6 * -7).

> Europython/Gothenburg (June 2005)
> =============================
> Apart from the presprint, which is already reoprted, there was an open 
> post.-Europython sprint. At the open sprint there were 24 people on the first 
> day and activities happened on may fronts. A large group of people focused on 
> porting PyPy to support Python2.4. This work was 3/4 finished by the end of 
> the sprint. Christian and Jacob worked on support for floats and Jacob did a 
> bunch of work on binascii and related modules. I think there was also a bunch 
> of work going into annotation, translation and the C and llvm backends. Nick 
> worked on his sre Summer of Code project. There was also work being done on 
> the AST compiler.
> 
> People I remember being there: All the regulars plus Anders Qvist and Marius 
> Gedminas.

I was there for one day only, and didn't get anything done.  (I tried to
make py.test run the Zope 3 testsuite, which was not a very PyPy-related
task.)

Marius Gedminas
-- 
If the code and the comments disagree, then both are probably wrong.
                -- Norm Schryer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060823/bb38fffe/attachment.pgp 

From bea at changemaker.nu  Thu Aug 24 12:27:11 2006
From: bea at changemaker.nu (Beatrice During)
Date: Thu, 24 Aug 2006 12:27:11 +0200
Subject: [pypy-dev] Sprint reports 2003-2006
In-Reply-To: <200608231815.33714.jacob@strakt.com>
References: <44EC5932.5040901@changemaker.nu>
	<200608231815.33714.jacob@strakt.com>
Message-ID: <44ED7EFF.5070407@changemaker.nu>

Hi there

Thanks Jacob and Marius for your feedback. I am working on updating the 
sprint report page with this feedback. If anyone has comments and want 
me to add stuff/clear out missing pieces - please email me and I will 
update.

Cheers

Bea

Jacob Hall?n skrev:
> Great work!
> 
> I think we should try to reconstruct at least stub reports for the remaining 
> sprints. Here are some notes on what happened at the ones I attended. This is 
> in no way complete (and fairly Jacob focused, since I remember what I did, 
> but have a hazy understanding of what others did):
> 
> 
> Gothenburg (May 2003)
> ===================
> The major focus of the sprint was to make the interpreter chain complete. A 
> first goal was to make "Hello world" run and this was accomplished about half 
> way into the sprint. We then proceded to making a more complex program that 
> had operations involving string operations and dictionaries work. This was 
> accomplished before the end of the sprint.
> 
> http://www.strakt.com/~jacob/pypy/img_0062.jpg
> 
> There was a big discussion, mostly involving Armin and Samuele over how to 
> handle operator dispatching. This ended in the Multimethod dispatcher that 
> mostly still survives in the Standard Object Space.
> 
> http://www.strakt.com/~jacob/pypy/img_0064.jpg
> http://www.strakt.com/~jacob/pypy/img_0065.jpg
> http://www.strakt.com/~jacob/pypy/img_0066.jpg
> 
> People I remember being there: Armin, Christian, Samuele, Michael, Alex 
> Martelli, Anna Ravenscroft, Laura, Tomek Meka
> 
> Europython/Gothenburg (June 2004)
> =============================
> The sprint happened before the conference, and I'm rather hazy on what 
> actually happened since I spent almost all my time on conference 
> preparations. I'm rather certain Christian, Holger, Armin and Anders Lehman 
> were there. Maybe they can fill in details. (To jog your memories, you were 
> sitting in a room with green doors, skylights and no windows in the 
> IT-support department).
> 
> Vilnius (Nov 2004)
> ===============
> I don't really remember what others did at this sprint, but this is where 
> Laura and I wrote the file I/O handling.
> 
> People I remember being there: Marius Gedminas, Bob Ippolito, Christian, 
> Armin.
> 
> Leysin (April 2006)
> ===============
> Armin, Arre and Samuele started with 2 days working on the JIT. When others 
> joined the sprint focus changed and there was a bunch of work on fixing 
> translation to work with stackless features and other stuff. There was also 
> some work done on the constraints package.
> 
> People I remember being there:
> Armin, Arre, Samuele, Holger, Michael, Alexandre, Aurelien
> 
> Europython/Gothenburg (June 2005)
> =============================
> Apart from the presprint, which is already reoprted, there was an open 
> post.-Europython sprint. At the open sprint there were 24 people on the first 
> day and activities happened on may fronts. A large group of people focused on 
> porting PyPy to support Python2.4. This work was 3/4 finished by the end of 
> the sprint. Christian and Jacob worked on support for floats and Jacob did a 
> bunch of work on binascii and related modules. I think there was also a bunch 
> of work going into annotation, translation and the C and llvm backends. Nick 
> worked on his sre Summer of Code project. There was also work being done on 
> the AST compiler.
> 
> People I remember being there: All the regulars plus Anders Qvist and Marius 
> Gedminas.
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev

From laurent.destriau at gmail.com  Fri Aug 25 22:05:02 2006
From: laurent.destriau at gmail.com (laurent destriau)
Date: Fri, 25 Aug 2006 22:05:02 +0200
Subject: [pypy-dev] Extension compiler and external DLLs
Message-ID: <3bb8d3330608251305k333890blca6215055e854e8a@mail.gmail.com>

Hi,
I would like to use pypy/bin/compilemodule.py to compile some of my code,
which uses OpenGL.
I am wondering why the following does not work (Link error on glEnd) :

    from ctypes import windll

    glEnd = windll.opengl32.glEnd
    glEnd.restype = None

    def DrawSomething(space):
        glEnd()



While this does compile fine:

    from pypy.rpython.rctypes.tool import ctypes_platform
    from pypy.rpython.rctypes.tool.libc import libc
    from ctypes import *

    time_t = ctypes_platform.getsimpletype('time_t', '#include <time.h>',
c_long)
    time = libc.time
    time.argtypes = [POINTER(time_t)]
    time.restype = time_t

    def DrawLine(space):
        time(None)


Any help would be much appreciated.
Thanks,

Laurent Destriau
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20060825/7d343b70/attachment.htm 

From l.oluyede at gmail.com  Fri Aug 25 23:19:48 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Fri, 25 Aug 2006 23:19:48 +0200
Subject: [pypy-dev] Extension compiler and external DLLs
In-Reply-To: <3bb8d3330608251305k333890blca6215055e854e8a@mail.gmail.com>
References: <3bb8d3330608251305k333890blca6215055e854e8a@mail.gmail.com>
Message-ID: <9eebf5740608251419x7ccf2e81gec3e915d78dbdd24@mail.gmail.com>

On 8/25/06, laurent destriau <laurent.destriau at gmail.com> wrote:
> Hi,
>  I would like to use pypy/bin/compilemodule.py to compile some of my code,
> which uses OpenGL.
>  I am wondering why the following does not work (Link error on glEnd) :
>
>      from ctypes import windll
>
>      glEnd = windll.opengl32.glEnd
>      glEnd.restype = None
>
>      def DrawSomething(space):
>          glEnd()
>

You should provide more information about your errors so it could be
easier to help you. Anyway I guess the compiler doesn't find the C
header in which glEnd() is defined. Do you have the headers of the
opengl library on the machine?

ps. do you know there is a opengl-ctypes implementation, right?

-- 
Lawrence
http://www.oluyede.org/blog

From laurent.destriau at gmail.com  Sat Aug 26 01:58:28 2006
From: laurent.destriau at gmail.com (laurent destriau)
Date: Sat, 26 Aug 2006 01:58:28 +0200
Subject: [pypy-dev] Extension compiler and external DLLs
In-Reply-To: <9eebf5740608251419x7ccf2e81gec3e915d78dbdd24@mail.gmail.com>
References: <3bb8d3330608251305k333890blca6215055e854e8a@mail.gmail.com>
	<9eebf5740608251419x7ccf2e81gec3e915d78dbdd24@mail.gmail.com>
Message-ID: <3bb8d3330608251658p4e3cfdbbycedd0c1222237c54@mail.gmail.com>

Thank you for your answer.
Here is some more info:
- I use Visual C++ .Net 2003, which includes the OpenGL header
- I get the following warning: "warning C4013: glEnd not defined ; extern
returning int assumed"
- I get the following error: "error LNK2001: unresolved external symbol
_glEnd"

How can I tell the extension compiler how to find the proper
headers/libraries?

> ps. do you know there is a opengl-ctypes implementation, right?

No, I didn't. Thanks for the tip.

--
Laurent Destriau

On 8/25/06, Lawrence Oluyede <l.oluyede at gmail.com> wrote:
>
> On 8/25/06, laurent destriau <laurent.destriau at gmail.com> wrote:
> > Hi,
> >  I would like to use pypy/bin/compilemodule.py to compile some of my
> code,
> > which uses OpenGL.
> >  I am wondering why the following does not work (Link error on glEnd) :
> >
> >      from ctypes import windll
> >
> >      glEnd = windll.opengl32.glEnd
> >      glEnd.restype = None
> >
> >      def DrawSomething(space):
> >          glEnd()
> >
>
> You should provide more information about your errors so it could be
> easier to help you. Anyway I guess the compiler doesn't find the C
> header in which glEnd() is defined. Do you have the headers of the
> opengl library on the machine?
>
> ps. do you know there is a opengl-ctypes implementation, right?
>
> --
> Lawrence
> http://www.oluyede.org/blog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20060826/b67f2552/attachment.htm 

From l.oluyede at gmail.com  Sat Aug 26 02:02:45 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Sat, 26 Aug 2006 02:02:45 +0200
Subject: [pypy-dev] Extension compiler and external DLLs
In-Reply-To: <3bb8d3330608251658p4e3cfdbbycedd0c1222237c54@mail.gmail.com>
References: <3bb8d3330608251305k333890blca6215055e854e8a@mail.gmail.com>
	<9eebf5740608251419x7ccf2e81gec3e915d78dbdd24@mail.gmail.com>
	<3bb8d3330608251658p4e3cfdbbycedd0c1222237c54@mail.gmail.com>
Message-ID: <9eebf5740608251702u56b89d18n89634649fc6b910@mail.gmail.com>

On 8/26/06, laurent destriau <laurent.destriau at gmail.com> wrote:
> Thank you for your answer.
>  Here is some more info:
>  - I use Visual C++ .Net 2003, which includes the OpenGL header
>  - I get the following warning: "warning C4013: glEnd not defined ; extern
> returning int assumed"
>  - I get the following error: "error LNK2001: unresolved external symbol
> _glEnd"

Mmm that's happened to me too but I'm still looking for the solution

>  How can I tell the extension compiler how to find the proper
> headers/libraries?

Look at the definition of CConfig in this file:
http://codespeak.net/svn/pypy/dist/pypy/module/readline/c_readline.py

I have to admit it didn't work for me...

-- 
Lawrence
http://www.oluyede.org/blog

From laurent.destriau at gmail.com  Sat Aug 26 04:14:11 2006
From: laurent.destriau at gmail.com (laurent destriau)
Date: Sat, 26 Aug 2006 04:14:11 +0200
Subject: [pypy-dev]  Extension compiler and external DLLs
In-Reply-To: <3bb8d3330608251849y58399db1v851e42e41cb650c1@mail.gmail.com>
References: <3bb8d3330608251305k333890blca6215055e854e8a@mail.gmail.com>
	<9eebf5740608251419x7ccf2e81gec3e915d78dbdd24@mail.gmail.com>
	<3bb8d3330608251658p4e3cfdbbycedd0c1222237c54@mail.gmail.com>
	<9eebf5740608251702u56b89d18n89634649fc6b910@mail.gmail.com>
	<3bb8d3330608251849y58399db1v851e42e41cb650c1@mail.gmail.com>
Message-ID: <3bb8d3330608251914l2a6db9a5yd06c4ffece607131@mail.gmail.com>

> I have to admit it didn't work for me...

It did for me:

    from pypy.rpython.rctypes.tool.ctypes_platform import configure, Library

    class CConfig:
        _header_ = ""
        _includes_ = ["windows.h", "gl/gl.h"]
        gl = Library("opengl32")

    gl = configure(CConfig)["gl"]

    glEnd = gl.glEnd

    glEnd.restype = None

    def DrawSomething(space):
        glEnd()

Thanks!

--
Laurent Destriau
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/pypy-dev/attachments/20060826/0f21694c/attachment.htm 

From arigo at tunes.org  Sun Aug 27 19:30:01 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 27 Aug 2006 19:30:01 +0200
Subject: [pypy-dev] Extension compiler and external DLLs
In-Reply-To: <3bb8d3330608251914l2a6db9a5yd06c4ffece607131@mail.gmail.com>
References: <3bb8d3330608251305k333890blca6215055e854e8a@mail.gmail.com>
	<9eebf5740608251419x7ccf2e81gec3e915d78dbdd24@mail.gmail.com>
	<3bb8d3330608251658p4e3cfdbbycedd0c1222237c54@mail.gmail.com>
	<9eebf5740608251702u56b89d18n89634649fc6b910@mail.gmail.com>
	<3bb8d3330608251849y58399db1v851e42e41cb650c1@mail.gmail.com>
	<3bb8d3330608251914l2a6db9a5yd06c4ffece607131@mail.gmail.com>
Message-ID: <20060827173001.GA29486@code0.codespeak.net>

Hi Laurent,

On Sat, Aug 26, 2006 at 04:14:11AM +0200, laurent destriau wrote:
> It did for me:
> 
>    from pypy.rpython.rctypes.tool.ctypes_platform import configure, Library
> 
>    class CConfig:
>        _header_ = ""
>        _includes_ = ["windows.h", "gl/gl.h"]
>        gl = Library("opengl32")

This works because of the Library() object, which tells the extension
compiler to pass the appropriate flags to the C linker.  You can get the
same effect by attaching flags directly to the function instead:

       glEnd = windll.opengl32.glEnd
       glEnd.restype = None
       glEnd.includes = ["windows.h", "gl/gl.h"]
       glEnd.libraries = ["opengl32"]
       # nb. untested code

But using a CConfig is better anyway for other reasons, e.g. to avoid
having to hard-code too many details of the type and structure
declarations.


A bientot,

Armin.

From hpk at trillke.net  Tue Aug 29 15:48:06 2006
From: hpk at trillke.net (holger krekel)
Date: Tue, 29 Aug 2006 15:48:06 +0200
Subject: [pypy-dev] pypy-sync thursday 4pm UTC+2
Message-ID: <20060829134806.GK21969@solar.trillke.net>

Hi folks,

at the nice and small Ireland sprint we discussed apart from
the weather a bit future sprint planning.  Next Thursday 4pm UTC+2,
31st August, there is now a pypy-sync meeting scheduled. 

Concrete initial topics: 

* rough sprint planning till March 2007

* concrete sprint planning beginning November, HHU/Duesseldorf? 

* security prototype: when to work on it and the actual
  workshop timing with the IBM guy(s), suggestion is
  January/February 

feel free to append topics, also at the meeting, but
the meeting is limited to 30 minutes from my moderation side. 

best, 

    holger

From Ben.Young at risk.sungard.com  Wed Aug 30 09:48:12 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 30 Aug 2006 08:48:12 +0100
Subject: [pypy-dev] Fw: [Python-3000] Premature optimization and all that
Message-ID: <OF75CE8C48.8C0DC7D3-ON802571DA.002A6C42-802571DA.002AD982@risk.sungard.com>

Hi PyPy'rs!

I'm sure you will have already seen this email, but I thought it was quite 
interesting, as it may mean that Python 3000 is where PyPy can become 
faster than CPython! Given a years development time at the current rate of 
progress.

I think PyPy is about 2.5 times slower than CPython at the moment isn't 
it?

Cheers,
Ben

----- Forwarded by Ben Young/Infinity on 30/08/2006 08:43 -----

python-3000-bounces+python=theyoungfamily.co.uk at python.org wrote on 
29/08/2006 22:51:17:

> Over lunch with Neal we came upon the topic of optimization and Python 
3000.
> 
> It is our strong opinion that in this stage of the Py3k project we
> should focus on getting the new language spec and implementation
> feature-complete, without worrying much about optimizations.
> 
> We're doing major feature-level surgery, e.g. int/long unification,
> str/unicode unification, range/xrange unification, keys() views, and
> many others. Keeping everything working is hard work in and of itself;
> having to keep it as fast as it was through all the transformations
> just makes it that much harder.
> 
> if Python 3.0 alpha 1 is twice as slow as 2.5, that's okay with me; we
> will have another year to do performance measurements and add new
> optimizations in the ramp-up for 3.0 final. Even if 3.0 final is a bit
> slower than 2.5 it doesn't bother me too  much; we can continue the
> tweaks during the 3.1 and 3.2 development cycle.
> 
> Note: I'm note advicating wholesale proactive *removal* of
> optimizations. However, I'm allowing new features to slow down
> performance temporarily while we get all the features in place. I
> expect that the optimization possibilities and needs will be different
> than for 2.x, since some of the fundamental data types will be so
> different.
> 
> In particular, I hope that Martin's int/long unification code can land
> ASAP; it's much better to have this feature landed in the p3yk branch,
> where everyone can bang on it easily, and learn how this affects user
> code, even if it makes everything twice as slow. This seems much
> preferable over having it languish in a separate branch until it's
> perfect.
> 
> -- 
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-3000 mailing list
> Python-3000 at python.org
> http://mail.python.org/mailman/listinfo/python-3000
> Unsubscribe: http://mail.python.
> org/mailman/options/python-3000/python%40theyoungfamily.co.uk
> 


From haggai.eran at gmail.com  Wed Aug 30 10:53:56 2006
From: haggai.eran at gmail.com (Haggai Eran)
Date: Wed, 30 Aug 2006 11:53:56 +0300
Subject: [pypy-dev] compilemodule.py not working
Message-ID: <e3f813180608300153x1170ad55jddb3c0972b2f22a7@mail.gmail.com>

Hi
I cannot get the compilemodule to work. It seems that strings returned
from PyString_FromStringAndSize are unwrapped automatically, causing
ctypes.byref to fail.

Traceback (most recent call last):
  File "../bin/compilemodule.py", line 12, in ?
    main(sys.argv)
  File "/usr/src/pypy-dist/pypy/rpython/rctypes/tool/compilemodule.py",
line 78, in main
    c_ext_module = compilemodule(argv[1], interactive=True)
  File "/usr/src/pypy-dist/pypy/rpython/rctypes/tool/compilemodule.py",
line 26, in compilemodule
    module = ModuleClass(space, space.wrap(modname))
  File "/usr/src/pypy-dist/pypy/module/thread/__init__.py", line 25, in __init__
    MixedModule.__init__(self, space, *args)
  File "/usr/src/pypy-dist/pypy/interpreter/mixedmodule.py", line 18,
in __init__
    Module.__init__(self, space, w_name)
  File "/usr/src/pypy-dist/pypy/interpreter/module.py", line 17, in __init__
    space.setitem(w_dict, space.new_interned_str('__name__'), w_name)
  File "/usr/src/pypy-dist/pypy/objspace/cpy/objspace.py", line 171,
in new_interned_str
    PyString_InternInPlace(byref(w_s))
TypeError: byref() argument must be a ctypes instance, not 'str'


-- 
Haggai Eran

From haggai.eran at gmail.com  Thu Aug 31 12:35:13 2006
From: haggai.eran at gmail.com (Haggai Eran)
Date: Thu, 31 Aug 2006 13:35:13 +0300
Subject: [pypy-dev] compilemodule.py not working
In-Reply-To: <e3f813180608300153x1170ad55jddb3c0972b2f22a7@mail.gmail.com>
References: <e3f813180608300153x1170ad55jddb3c0972b2f22a7@mail.gmail.com>
Message-ID: <e3f813180608310335j7ee8675dpcf98ac756c8bb1cd@mail.gmail.com>

Silly me: I had an older version of ctypes.
Version 0.9.9 and 1.0.0 don't have this problem.

Haggai

On 8/30/06, Haggai Eran <haggai.eran at gmail.com> wrote:
> Hi
> I cannot get the compilemodule to work. It seems that strings returned
> from PyString_FromStringAndSize are unwrapped automatically, causing
> ctypes.byref to fail.
>
> Traceback (most recent call last):
>   File "../bin/compilemodule.py", line 12, in ?
>     main(sys.argv)
>   File "/usr/src/pypy-dist/pypy/rpython/rctypes/tool/compilemodule.py",
> line 78, in main
>     c_ext_module = compilemodule(argv[1], interactive=True)
>   File "/usr/src/pypy-dist/pypy/rpython/rctypes/tool/compilemodule.py",
> line 26, in compilemodule
>     module = ModuleClass(space, space.wrap(modname))
>   File "/usr/src/pypy-dist/pypy/module/thread/__init__.py", line 25, in __init__
>     MixedModule.__init__(self, space, *args)
>   File "/usr/src/pypy-dist/pypy/interpreter/mixedmodule.py", line 18,
> in __init__
>     Module.__init__(self, space, w_name)
>   File "/usr/src/pypy-dist/pypy/interpreter/module.py", line 17, in __init__
>     space.setitem(w_dict, space.new_interned_str('__name__'), w_name)
>   File "/usr/src/pypy-dist/pypy/objspace/cpy/objspace.py", line 171,
> in new_interned_str
>     PyString_InternInPlace(byref(w_s))
> TypeError: byref() argument must be a ctypes instance, not 'str'
>
>
> --
> Haggai Eran
>


-- 
Haggai Eran

From anto.cuni at gmail.com  Sun Sep  3 12:51:30 2006
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Sun, 03 Sep 2006 12:51:30 +0200
Subject: [pypy-dev] Reference letters for a PhD application
Message-ID: <44FAB3B2.5060809@gmail.com>

Hi all,

as you might know I've not yet decided what to do in the future when my 
experience at HHU will end; one of the possibilities is to take a PhD 
here in Genoa, so I'm trying to fill up the form for the application.

To participate I need one to three reference letters stating what are my 
capacity especially in the research fields. I was wondering if someone 
that has an official position in a company/university is willing to 
write such a letter (hoping it will contain a good evaluation :-).

The template for the letters is here:
http://www.disi.unige.it/dottorato/AMMISSIONE/adm-rules/RecommendationLetter.html

Thanks a lot for the help
ciao Anto

From Ben.Young at risk.sungard.com  Mon Sep  4 12:59:48 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 4 Sep 2006 11:59:48 +0100
Subject: [pypy-dev] Non dynalloc framework roots
Message-ID: <OF758AFD9E.44453AE9-ON802571DF.003BEAE3-802571DF.003C683F@risk.sungard.com>

Hi PyPyr's

This is mainly for Carl or Michael as it concerns the framework gc, but I 
was just interested how easy it would be to track the roots without doing 
any dynamic memory allocation. I came up with the following code, which I 
am interested to see  how easily it could be translated to llpython or 
whatever the lowest level is called. 

If someone can give me some hints, I may even have a go at implementing it 
myself!

Cheers,
Ben

#include "assert.h"

// Code

typedef struct _GCFuncNode
{
    int count;
    void*** pointers;
    struct _GCFuncNode* prev;
} GCFuncNode;

GCFuncNode* GC_top_node;

void GC_push_func_node(GCFuncNode* node, int count, void*** pointers)
{
    GCFuncNode* old_top;

    node->count = count;
    node->pointers = pointers;
    old_top = GC_top_node;
    GC_top_node = node;
    node->prev = old_top;
}

void GC_pop_func_node(void)
{
    GC_top_node = GC_top_node->prev;
}

typedef struct _GCRootItr
{
    GCFuncNode* cur_node;
    int cur_ptr;
} GCRootItr;

void GC_init_root_itr(GCRootItr* itr)
{
    itr->cur_node = GC_top_node;
    itr->cur_ptr = 0;
}

void** GC_itr_cur(GCRootItr* itr)
{
    if(itr->cur_node)
    {
        return itr->cur_node->pointers[itr->cur_ptr];
    }
    else 
    {
        return (void**)0;
    }
}

void GC_itr_next(GCRootItr* itr)
{ 
    assert(itr->cur_ptr < itr->cur_node->count);

    // Find next non-null pointer
    while(1)
    {
        itr->cur_ptr++;
        if(itr->cur_ptr == itr->cur_node->count) 
        {
            itr->cur_node = itr->cur_node->prev;
            itr->cur_ptr = 0;
            return;
        }

        if(*itr->cur_node->pointers[itr->cur_ptr] != (void*)0)
        {
            return;
        }
    }
}

// Example

typedef void* PyObjPtr;

void example_func_2(PyObjPtr a, PyObjPtr b, int x, int y);

void example_func_1(int x, int y)
{
    PyObjPtr a = 0, b = 0, c = 0;

    // GC code
    GCFuncNode node;
    void** GC_local_vars[] = {&a, &b, &c};
    GC_push_func_node(&node, sizeof(GC_local_vars)/sizeof(void*), 
GC_local_vars);
    // end GC code

    // Do stuff with a, b, c

    example_func_2(a, b, 3, y);

    // GC code
    GC_pop_func_node();
    // end GC code
}

void example_func_2(PyObjPtr a, PyObjPtr b, int x, int y)
{
    PyObjPtr i = 0, j = 0;

    // GC code
    GCFuncNode node;
    void** GC_local_vars[] = {&i, &j};
    GC_push_func_node(&node, sizeof(GC_local_vars)/sizeof(void*), 
GC_local_vars);
    // end GC code

    // GC code
    GC_pop_func_node();
    // end GC code
}


int main(int argc, char* argv[])
{
        return 0;
}


From arigo at tunes.org  Fri Sep  8 13:41:27 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 8 Sep 2006 13:41:27 +0200
Subject: [pypy-dev] PyPy JIT frontend/backend
Message-ID: <20060908114127.GB8257@code0.codespeak.net>

Hi all,

The JIT work finally reached an important step in the past week: there
is an RPython interface between the JIT frontend and any possible
machine-code-level backend.  The interface (called RGenOp and To Be
Documented Soon, Tm) has two implementations so far:

* codegen/llgraph/rgenop.py that produces flow graphs again (don't look,
  it's quite obscure to do that while still being RPython enough).  It
  is for testing only, so far.

* codegen/i386/ri386genop.py that produces i386 ("IA32", more precisely)
  machine code into mmap-ed memory blocks.

There are some basic tests about using the interface in
i386/test/test_ri386genop.py.  The tests there have the following
structure:

* def make_xxx(): this is an example RPython function that calls the
  rgenop interface to generate some simple code.

* def test_xxx_interpret(): tries to run make_xxx() with the
  codegen/llgraph implementation, and then llinterprets the produced
  graph.

* def test_xxx_direct(): tries to run make_xxx() with the codegen/i386
  implementation, and then executes the machine code.  Gets nice
  segfaults if the machine code is wrong.

* def test_xxx_compile(): compiles the make_xxx() function into a
  stand-alone executable (via the normal genc route).  This is where we
  check that our ri386genop implementation is really RPython code only.

A hint about the "token" thingy: there are many rgenop.xxxToken() static
methods that are here basically because we cannot dynamically handle in
RPython code objects like low-level types.  So a "token" is whatever
RPython value that the back-end needs in order to perform a specific
operation on that type.  Each variant of token can be anything - it's
returned by rgenop only to be passed back to it - but they must be
RPython values.  For example, 'fieldToken(T, name)' makes a token that
is passed back to methods like genop_getfield().  For the i386 backend,
the fieldToken is simply an integer: the offset of the field in the
structure T.  This is used by genop_getfield() to encode the offset in
the machine instruction.  In this way, genop_getfield() is a completely
RPython-friendly function, and only fieldToken() needs to be special (a
memo static method).

More in the above-promized documentation :-)


A bientot,

Armin


From arigo at tunes.org  Sat Sep  9 11:56:27 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 9 Sep 2006 11:56:27 +0200
Subject: [pypy-dev] Non dynalloc framework roots
In-Reply-To: <OF758AFD9E.44453AE9-ON802571DF.003BEAE3-802571DF.003C683F@risk.sungard.com>
References: <OF758AFD9E.44453AE9-ON802571DF.003BEAE3-802571DF.003C683F@risk.sungard.com>
Message-ID: <20060909095627.GA29130@code0.codespeak.net>

On Mon, Sep 04, 2006 at 11:59:48AM +0100, Ben.Young at risk.sungard.com wrote:
> This is mainly for Carl or Michael as it concerns the framework gc, but I 
> was just interested how easy it would be to track the roots without doing 
> any dynamic memory allocation. I came up with the following code, which I 
> am interested to see  how easily it could be translated to llpython or 
> whatever the lowest level is called. 

>     // GC code
>     GCFuncNode node;
>     void** GC_local_vars[] = {&a, &b, &c};
>     GC_push_func_node(&node, sizeof(GC_local_vars)/sizeof(void*), 
>                       GC_local_vars);
>     // end GC code

Yes, it makes sense.  As usual it's not completely clear if it is better
or worse than our current approach.  One thing I fear is that taking the
address of all the local variables is going to kill all gcc
optimizations on them.  A more annoying problem is that - anyway - we
cannot take addresses of local variables in our flow graphs, so far.
There is not even a notion of "all the local variables" of a graph; only
of a block.  It would require quite some tweaking.  If would be nice to
try out...

An intermediate solution between what you suggest and what we do now
would be to still use purely-local variables (i.e. no taking pointers to
them) and save them around calls instead of around the whole function,
but instead of saving to a global stack, save in a precomputed
graph-local Array type with enough items for the maximum number of
variables that need to be saved.  Then, instead of saving/restoring the
roots in the global root stack, they would be saved/restored into the
local Array.

I guess the node and GC_local_vars could both be packed in a single
local var-sized Struct, with a 'prev' pointer, and an Array at the end.
In term of flow graphs, to obtain local variables of Struct type, we use
a 'stack' flavor of malloc:

   v = flavored_malloc(S, 'stack')

It gives you a pointer to an otherwise-invisible local variable of type
S.  In other words in the C code it becomes:

   struct S invisible;
   struct S* v = &invisible;   /* constant pointer used for all accesses */

I'm not sure any more if we support 'stack'-flavored mallocs of
var-sized structures and arrays, but that should be easy to fix.

The meat of the work is to extend the class FrameworkGCTransformer in
pypy.rpython.memory.gctransform.  See StacklessFrameworkGCTransformer
for an example of providing an alternate StackRootIterator.  Its
push_root/pop_root static methods should save/restore a local variable
into/from some place where the StackRootIterator instance can find them
later.  (For now, pop_root can be a no-op as long as we can't support
moving GCs... but hopefully that is going to change soon, thanks to
mwh).


A bientot,

Armin

From arigo at tunes.org  Mon Sep 11 19:03:27 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 11 Sep 2006 19:03:27 +0200
Subject: [pypy-dev] DLS 2006 Call for Participation
Message-ID: <20060911170327.GA19754@code0.codespeak.net>

Hi all,

Here is the call for participation for the DLS conference.  We have a
30-minutes slot there.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Dynamic Languages Symposium 2006

** Call for Participation **

Portland, Oregon, United States, October 23, 2006

http://www.dcl.hpi.uni-potsdam.de/dls2006/
http://www.oopsla.org/2006/program/program/dynamic_languages_symposium.html

Part of OOPSLA 2006, being held October 22-26 in historic Portland,
Oregon (USA). You can learn all about OOPSLA at www.oopsla.org, and
download the Advance Program PDF at
http://www.oopsla.org/2006//program/oopsla_06_advance_program.pdf

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

** Dynamic Languages Symposium Program **

8:30 - 9:30 Invited Talk 1

    Openness and simplicity in dynamic systems implementation
    Ian Piumarta

9:30 - 10:00 Break

10:00 - 11:30 Research Papers 1

    PyPy's Approach to Virtual Machine Construction
    Armin Rigo and Samuele Pedroni

    Runtime Synthesis of High-Performance Code from Scripting Languages
    Christopher Mueller and Andrew Lumsdaine

    Interlanguage Migration: From Scripts to Programs
    Sam Tobin-Hochstadt and Matthias Felleisen

11:30 - 13:00 Break

13:00 - 14:00 Invited Talk 2

    Perl 6
    Audrey Tang

14:00 - 14:30 Break

14:30 - 16:00 Research Papers 2

    Hop, a Language for Programming the Web 2.0
    Manuel Serrano, Erick Gallesio, and Florian Loitsch

    Ambient References: Addressing Objects in Mobile Networks
    Tom Van Cutsem, Jessie Dedecker, Stijn Mostinckx,
    Elisa Gonzalez Boix, Theo D'Hondt, and Wolfgang De Meuter

    Hardware Transactional Memory Support for Lightweight Dynamic
    Language Evolution
    Nicholas Riley and Craig Zilles

16:00 - 16:15 Short Break

16:15 - 17:15 Invited Talk 3

    Data Refactoring for Amateurs
    Avi Bryant

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

** Dynamic Languages Symposium Invited Talks **

Openness and simplicity in dynamic systems implementation Ian Piumarta

The talk will describe a basis for constructing systems (programming
languages, environments and applications) in which users can be
encouraged to adapt the characteristics of the system to match their
needs (rather than the other way round). Such systems can be evolved
from a pair of abstractions for state (objects communicating by
messaging) and behaviour (first-class functions) that are mutually
supporting: objects form structures representing symbolic expressions
that fully describe the message sequencing and sending that are needed
to implement objects. The result is extreme late-binding (nothing in
the system is immune from dynamic modification) and extreme simplicity
(each abstraction can be written down in a handful of lines of
mathematics, and only slightly more lines of code).

Ian Piumarta is a computer scientist at Viewpoints Research
Institute. He spends much of his time designing and building systems
whose implementations are maximally open, reflexive, dynamically
self-describing and understandable. He can be contacted at squeakland
dot org.

Perl 6 Audrey Tang

Perl is a general-purpose language, known for its vast number of
freely available libraries. The Perl 6 project was started to improve
the language's support for multi-paradigmatic programming, while
retaining compatibility with the existing code base. This talk
discusses how Perl 6 attempts to reconcile various competing paradigms
in the field of programming language design, such as static
vs. dynamic typechecking, nominal vs. structural subtyping, prototype
vs. class-based objects, and lazy vs. eager evaluation. Moreover, this
talk also covers the design and development of Pugs, a self-hosting
Perl 6 implementation bootstrapped from Haskell, targeting multiple
runtime environments, including Perl 5, JavaScript and Parrot.

Audrey Tang is a Taiwanese free software programmer, best known for
initiating and leading the Pugs project, a joint effort from Haskell
and Perl communities to implement the Perl 6 language. She is also
known for internationalization and localization contributions to
several Free Software programs, including SVK, Kwiki, Request Tracker
and Slash, as well as heading Traditional Chinese translation efforts
for various Open Source-related books. On the CPAN, Tang initiated
over 100 Perl projects, including the popular Perl Archive Toolkit
(PAR), a cross-platform packaging and deployment tool for Perl 5. She
is also responsible for setting up smoke test and digital signature
systems for CPAN. Tang is a high school dropout and a vocal proponent
for autodidacticism and individualist anarchism.

Data Refactoring for Amateurs Avi Bryant

Agile software development methodologies such as Extreme Programming
advocate iterative design via incremental, test-driven code extension
and automated refactorings. When the goal is to allow non-developers
to build their own solutions, even in a limited way, this approach to
incrementality becomes even more important -- non-developers generally
have even less of the design experience necessary to make reasonable
decisions up front, and need real use and concrete examples to guide
their decisions. Dabble DB is a commercial data management tool aimed
at casual business users. It encourages users to evolve data models
slowly over time, starting with untyped and de-normalized models and
proceeding to more sophisticated models only as the need becomes
apparent. We introduce a set of data refactorings designed to support
this usage pattern, and show selected examples of their real-world
use.

Avi Bryant is the co-CEO of Smallthought Systems Inc., a Vancouver
startup focused on web-based collaboration tools. He is the author and
maintainer of the Seaside web application framework, and is active in
the open source Squeak Smalltalk community.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Dynamic Languages Symposium is part of OOPSLA, the premier gathering
of professionals from industry and academia, all sharing their
experiences with today's object technologies and its offshoots.
OOPSLA appeals to practitioners, researchers, students, educators, and
managers, all of whom discover a wealth of information and the chance
to meet others with similar interests and varied experiences and
knowledge.

You can mold your own OOPSLA experience, attending your choices of
technical papers, practitioner reports, expert panels, demonstrations,
essays, lightning talks, formal and informal educator symposia,
workshops, and diverse tutorials and certificate courses from
world-class experts. The popular Onward! track presents out-of-the-box
thinking at the frontiers of computing. Posters discuss late-breaking
results, culminating in the ACM Student Research Competition. Try your
hand at solving the DesignFest(R) design challenge. And of course there
are plenty of social opportunities for mingling and professional
networking.

Go to the web (www.oopsla.org) today to reserve your place at OOPSLA
'06. See you in Portland!

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

From mwh at python.net  Mon Sep 11 23:19:11 2006
From: mwh at python.net (Michael Hudson)
Date: Mon, 11 Sep 2006 22:19:11 +0100
Subject: [pypy-dev] DLS 2006 Call for Participation
References: <20060911170327.GA19754@code0.codespeak.net>
Message-ID: <2mwt8aksog.fsf@starship.python.net>

Armin Rigo <arigo at tunes.org> writes:

>     Runtime Synthesis of High-Performance Code from Scripting Languages
>     Christopher Mueller and Andrew Lumsdaine

This is (a) interesting (b) potentially stealable :-) and (c) looks
frighteningly like translator/asm/ppcgen ... I don't know whether it's
more surprising that they don't seem to know what I did or that I
didn't know about that :-)

Cheers,
mwh

-- 
  : exploding like a turd
  Never had that happen to me, I have to admit.  They do that
  often in your world?              -- Eric The Read & Dave Brown, asr


From mwh at python.net  Tue Sep 12 09:12:51 2006
From: mwh at python.net (Michael Hudson)
Date: Tue, 12 Sep 2006 08:12:51 +0100
Subject: [pypy-dev] DLS 2006 Call for Participation
In-Reply-To: <4505E377.5080502@scottdial.com>
References: <20060911170327.GA19754@code0.codespeak.net>
	<2mwt8aksog.fsf@starship.python.net>
	<4505E377.5080502@scottdial.com>
Message-ID: <1B078E8F-A125-4824-ADEA-56E766D2F52D@python.net>


On 11 Sep 2006, at 23:30, Scott Dial wrote:

> Michael Hudson wrote:
>> Armin Rigo <arigo at tunes.org> writes:
>>>     Runtime Synthesis of High-Performance Code from Scripting  
>>> Languages
>>>     Christopher Mueller and Andrew Lumsdaine
>> This is (a) interesting (b) potentially stealable :-) and (c) looks
>> frighteningly like translator/asm/ppcgen ... I don't know whether  
>> it's
>> more surprising that they don't seem to know what I did or that I
>> didn't know about that :-)
>
> (a) yes,
> (b) not at the moment (restrictive license)

Yes, I saw that, but I get the impression that we could probably beg :)

> (c) hmm..
>
> And as for not being aware of it, well, AFAICT, this is very  
> recently demonstrated at SciPy 2006 (Aug 17th), and I think you can  
> be forgiven for not having heard of it before because I don't think  
> anyone else had either.

Oh right.  Then they should at least have heard of PPY: http:// 
www.python.net/crew/mwh/hacks/PPY.html

It would at least have saved some time typing numbers out of the back  
of the PPC architecture manual...

> You can look at it all here: http://www.osl.iu.edu/~chemuell/new/ 
> sp.php

Yeah, it was that that finally convinced me that it _wasn't_ based on  
my code :-)

It also look like it might be easier to convert into RPython than my  
code.

> I haven't met Chris, but I feel like this is one of those "small  
> world" moments since he literally works in the same building as me.

Heh :)

Cheers,
mwh


From Ben.Young at risk.sungard.com  Wed Sep 13 11:22:06 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Wed, 13 Sep 2006 10:22:06 +0100
Subject: [pypy-dev] Non dynalloc framework roots
In-Reply-To: <20060909095627.GA29130@code0.codespeak.net>
Message-ID: <OF601CCEDB.1ACAA0AB-ON802571E8.0032EE33-802571E8.00337636@risk.sungard.com>

Hi Armin,

Thanks for the reply!

pypy-dev-bounces at codespeak.net wrote on 09/09/2006 10:56:27:

> On Mon, Sep 04, 2006 at 11:59:48AM +0100, Ben.Young at risk.sungard.com 
wrote:
> > This is mainly for Carl or Michael as it concerns the framework gc, 
but I 
> > was just interested how easy it would be to track the roots without 
doing 
> > any dynamic memory allocation. I came up with the following code, 
which I 
> > am interested to see  how easily it could be translated to llpython or 

> > whatever the lowest level is called. 
> 
> >     // GC code
> >     GCFuncNode node;
> >     void** GC_local_vars[] = {&a, &b, &c};
> >     GC_push_func_node(&node, sizeof(GC_local_vars)/sizeof(void*), 
> >                       GC_local_vars);
> >     // end GC code
> 
> Yes, it makes sense.  As usual it's not completely clear if it is better
> or worse than our current approach.  One thing I fear is that taking the
> address of all the local variables is going to kill all gcc
> optimizations on them.  A more annoying problem is that - anyway - we
> cannot take addresses of local variables in our flow graphs, so far.
> There is not even a notion of "all the local variables" of a graph; only
> of a block.  It would require quite some tweaking.  If would be nice to
> try out...
> 

Hmm, I hadn't thought about killing GC optimizations. I guess the only way 
to find out it do it and see what happens!

> An intermediate solution between what you suggest and what we do now
> would be to still use purely-local variables (i.e. no taking pointers to
> them) and save them around calls instead of around the whole function,
> but instead of saving to a global stack, save in a precomputed
> graph-local Array type with enough items for the maximum number of
> variables that need to be saved.  Then, instead of saving/restoring the
> roots in the global root stack, they would be saved/restored into the
> local Array.
> 
> I guess the node and GC_local_vars could both be packed in a single
> local var-sized Struct, with a 'prev' pointer, and an Array at the end.
> In term of flow graphs, to obtain local variables of Struct type, we use
> a 'stack' flavor of malloc:
> 
>    v = flavored_malloc(S, 'stack')
> 
> It gives you a pointer to an otherwise-invisible local variable of type
> S.  In other words in the C code it becomes:
> 
>    struct S invisible;
>    struct S* v = &invisible;   /* constant pointer used for all accesses 
*/
> 
> I'm not sure any more if we support 'stack'-flavored mallocs of
> var-sized structures and arrays, but that should be easy to fix.
> 
> The meat of the work is to extend the class FrameworkGCTransformer in
> pypy.rpython.memory.gctransform.  See StacklessFrameworkGCTransformer
> for an example of providing an alternate StackRootIterator.  Its
> push_root/pop_root static methods should save/restore a local variable
> into/from some place where the StackRootIterator instance can find them
> later.  (For now, pop_root can be a no-op as long as we can't support
> moving GCs... but hopefully that is going to change soon, thanks to
> mwh).
> 

I think I understand. If I get some time I'll have a play with some code. 
I'm currently looking at how annotation works to see how easy it would be 
to get generator support in rpython aswell. It would allow me to fix all 
the iter... methods in the multidict code easily.

Cheers,
Ben

> 
> A bientot,
> 
> Armin
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From arigo at tunes.org  Mon Sep 18 09:54:28 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 18 Sep 2006 09:54:28 +0200
Subject: [pypy-dev] schedule()
Message-ID: <20060918075428.GA19879@code0.codespeak.net>

Hi Aurelien,

It seems that the logic objspace has had now for a long time a
schedule() function, instead of just doing the preemptive scheduling
automtically.  As we've always said, it is very easy to do the
preemptive scheduling bit, but as you never followed-up on that, I guess
I have to ask again.  Do you want me to add a hook in the logic objspace
that calls schedule() every Nth bytecode instruction?


A bientot,

Armin

From aurelien.campeas at logilab.fr  Mon Sep 18 10:47:26 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 18 Sep 2006 10:47:26 +0200
Subject: [pypy-dev] schedule()
In-Reply-To: <20060918075428.GA19879@code0.codespeak.net>
References: <20060918075428.GA19879@code0.codespeak.net>
Message-ID: <20060918084726.GB31821@crater.logilab.fr>

Hi Armin,

On Mon, Sep 18, 2006 at 09:54:28AM +0200, Armin Rigo wrote:
> Hi Aurelien,
> 
> It seems that the logic objspace has had now for a long time a
> schedule() function, instead of just doing the preemptive scheduling
> automtically.  As we've always said, it is very easy to do the
> preemptive scheduling bit, but as you never followed-up on that, I
> guess

Well, it's nice to know. I didn't ask further because it seemed not
'very easy' nor high-priority (from my pov) at the time (a few monthes
ago).

> I have to ask again.  Do you want me to add a hook in the logic objspace
> that calls schedule() every Nth bytecode instruction?

Sure. 

Thanks,
Aur?lien.

From cfbolz at gmx.de  Mon Sep 18 20:01:08 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 18 Sep 2006 20:01:08 +0200
Subject: [pypy-dev] PyPy Sprint Announcement, Duesseldorf 30 Oct - 5 Nov
Message-ID: <450EDEE4.2020601@gmx.de>

Hi all!

The next PyPy sprint will be held in the Computer Science department of
Heinrich-Heine Universitaet Duesseldorf from the 30th of October to the
5th of November 2006.

Topics and goals
----------------

The topics of the sprints are not fixed yet.  We will progress on the
subjects that we are currently working on, while giving a special
priority to any topic that "non-core" people find interesting.  There
are many topics that could fit both category :-)  Here are some
examples:

* Just-In-Time work.  Two sub-topics:

  - write and/or optimize a machine-code backend (we have 386 only so
    far)
  - work on turning simple interpreters into JIT compilers (we cannot do
    this for the whole of the PyPy interpreter yet, we're getting there
    small step by small step).

* Optimization of core Python data types, making full
  use of PyPy's flexible architecture and python-implemented (and then
  translated) type system.  (We have already various dict and str
  implementations.)

* "Next-step stuff" that will requires some thinking and design:

  - distribution (where a single program runs on multiple machines)
  - persistence (save an "image" of a running program, or a part of it)
  - security (in many possible senses of the word)

* Working on py.test testing tool:

  - py.test recently grew some distribution features which are still
    rough around the edges and could use improvement

  - there are some more ideas for features of py.test around, like
    adding profiling capabilities (and more)

* Work on the PyPy build tool: There are some plans to provide a tool
  that allows one to flexibly configure PyPy and to also request builds
  from a set of build servers. If there is interest there could be work
  in this area.

* and as always, there is the topic of implementing or completing core
  extension modules (e.g. socket...).  This is hacking with a mix of
  ctypes and RPython.


Location
--------

The sprint will (probably) take place in a seminar room of the geography
department (which is getting assimilated by the cs department and is
below it).  It is in the building 25.12 of the university campus. For
travel instructions see

    http://stups.cs.uni-duesseldorf.de/anreise/esbahn.php

Registration
------------

If you'd like to come, please subscribe to the `pypy-sprint mailing
list`_ and drop a note about your interests and post any questions.
More organisational information will be send to that list.  We'll keep a
list of `people`_ which we'll update (which you can do so yourself if
you have codespeak commit rights).

.. _`pypy-sprint mailing list`:
http://codespeak.net/mailman/listinfo/pypy-sprint
.. _`people`:
http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2006b/people.html



Cheers,

Carl Friedrich Bolz & the PyPy team