From rxe at ukshells.co.uk  Sat Jul  1 01:24:16 2006
From: rxe at ukshells.co.uk (Richard Emslie)
Date: Sat, 1 Jul 2006 00:24:16 +0100 (BST)
Subject: [pypy-dev] patch for __getattribute__()
Message-ID: <Pine.LNX.4.62.0607010016280.2028@echo.ukshells.co.uk>


Hi!

The following patch initiated by a chat with Runar Petursson about 
descriptors - increases pypy-c/pypy-llvm by 25% on richards.py.

I couldnt be bothered creating a branch - but if someone wants to sanity 
check it and fix the method repr and check it in, you are more than 
welcome! :-)

Cheers,
Richard

-------------- next part --------------
Index: objspace/descroperation.py
===================================================================
--- objspace/descroperation.py	(revision 29539)
+++ objspace/descroperation.py	(working copy)
@@ -1,7 +1,7 @@
 import operator
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import ObjSpace
-from pypy.interpreter.function import Function, Method
+from pypy.interpreter.function import Function, Method, descr_function_get
 from pypy.interpreter.argument import Arguments
 from pypy.interpreter.typedef import default_identity_hash
 from pypy.tool.sourcetools import compile2, func_with_new_name
@@ -19,6 +19,10 @@
         name = space.str_w(w_name)
         w_descr = space.lookup(w_obj, name)
         if w_descr is not None:
+            # XXX test performance
+            descr = space.interpclass_w(w_descr)
+            if type(descr) is Function:
+                return descr_function_get(space, w_descr, w_obj)
             if space.is_data_descr(w_descr):
                 return space.get(w_descr, w_obj)
         w_value = w_obj.getdictvalue(space, w_name)
Index: objspace/cpy/objspace.py
===================================================================
--- objspace/cpy/objspace.py	(revision 29539)
+++ objspace/cpy/objspace.py	(working copy)
@@ -62,8 +62,8 @@
             return PyString_FromStringAndSize(x, len(x))
         if isinstance(x, float): 
             return PyFloat_FromDouble(x)
-        if isinstance(x, r_uint):
-            return PyLong_FromUnsignedLong(x)
+        #if isinstance(x, r_uint):
+        #    return PyLong_FromUnsignedLong(x)
         # if we arrive here during RTyping, then the problem is *not* the %r
         # in the format string, but it's that someone is calling space.wrap()
         # on a strange object.
Index: interpreter/function.py
===================================================================
--- interpreter/function.py	(revision 29539)
+++ interpreter/function.py	(working copy)
@@ -385,6 +385,16 @@
             w_class = space.type(self.w_instance)
         else:
             w_class = self.w_class
+        if w_class is None:
+            if self.w_instance is None:
+                s = "<unbound method %s>" % (name)
+                return space.wrap(s)
+            else:
+                objrepr = space.str_w(space.repr(self.w_instance))
+                info = 'bound method %s of %s' % (name, objrepr)
+                # info = "method %s of %s object" % (name, typename)
+                return self.w_instance.getrepr(self.space, info)
+
         typename = w_class.getname(self.space, '?')
         if self.w_instance is None:
             s = "<unbound method %s.%s>" % (typename, name)

From pedronis at strakt.com  Sat Jul  1 10:03:25 2006
From: pedronis at strakt.com (Samuele Pedroni)
Date: Sat, 01 Jul 2006 10:03:25 +0200
Subject: [pypy-dev] patch for __getattribute__()
In-Reply-To: <Pine.LNX.4.62.0607010016280.2028@echo.ukshells.co.uk>
References: <Pine.LNX.4.62.0607010016280.2028@echo.ukshells.co.uk>
Message-ID: <44A62C4D.8090300@strakt.com>

Richard Emslie wrote:
> 
> Hi!
> 
> The following patch initiated by a chat with Runar Petursson about 
> descriptors - increases pypy-c/pypy-llvm by 25% on richards.py.

well, it breaks this:

Python 2.4.1 (#2, Mar 31 2005, 00:05:10)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> class C(object):
...    def m(self):
...       return "class"
...
 >>> c=C()
 >>> c.m = lambda: "instance"
 >>>
 >>> c.m()
'instance'
 >>>

vs. patched PyPy

PyPy 0.9.0 in StdObjSpace on top of Python 2.4.1 (startuptime: 4.97 secs)
 >>>> class C(object):
....    def m(self):
....        return "class"
....
 >>>> c=C()
 >>>> c.m = lambda: "instance"
 >>>> c.m()
'class'

functions are not data descriptors, this feature makes increasing the 
performance of method lookup more involved that one would like but
Python semantics expect this kind of overriding to work.

I have added a test about this.


> 
> I couldnt be bothered creating a branch - but if someone wants to sanity 
> check it and fix the method repr and check it in, you are more than 
> welcome! :-)
> 
> Cheers,
> Richard
> 
> 
> ------------------------------------------------------------------------
> 
> Index: objspace/descroperation.py
> ===================================================================
> --- objspace/descroperation.py	(revision 29539)
> +++ objspace/descroperation.py	(working copy)
> @@ -1,7 +1,7 @@
>  import operator
>  from pypy.interpreter.error import OperationError
>  from pypy.interpreter.baseobjspace import ObjSpace
> -from pypy.interpreter.function import Function, Method
> +from pypy.interpreter.function import Function, Method, descr_function_get
>  from pypy.interpreter.argument import Arguments
>  from pypy.interpreter.typedef import default_identity_hash
>  from pypy.tool.sourcetools import compile2, func_with_new_name
> @@ -19,6 +19,10 @@
>          name = space.str_w(w_name)
>          w_descr = space.lookup(w_obj, name)
>          if w_descr is not None:
> +            # XXX test performance
> +            descr = space.interpclass_w(w_descr)
> +            if type(descr) is Function:
> +                return descr_function_get(space, w_descr, w_obj)
>              if space.is_data_descr(w_descr):
>                  return space.get(w_descr, w_obj)
>          w_value = w_obj.getdictvalue(space, w_name)
> Index: objspace/cpy/objspace.py
> ===================================================================
> --- objspace/cpy/objspace.py	(revision 29539)
> +++ objspace/cpy/objspace.py	(working copy)
> @@ -62,8 +62,8 @@
>              return PyString_FromStringAndSize(x, len(x))
>          if isinstance(x, float): 
>              return PyFloat_FromDouble(x)
> -        if isinstance(x, r_uint):
> -            return PyLong_FromUnsignedLong(x)
> +        #if isinstance(x, r_uint):
> +        #    return PyLong_FromUnsignedLong(x)
>          # if we arrive here during RTyping, then the problem is *not* the %r
>          # in the format string, but it's that someone is calling space.wrap()
>          # on a strange object.
> Index: interpreter/function.py
> ===================================================================
> --- interpreter/function.py	(revision 29539)
> +++ interpreter/function.py	(working copy)
> @@ -385,6 +385,16 @@
>              w_class = space.type(self.w_instance)
>          else:
>              w_class = self.w_class
> +        if w_class is None:
> +            if self.w_instance is None:
> +                s = "<unbound method %s>" % (name)
> +                return space.wrap(s)
> +            else:
> +                objrepr = space.str_w(space.repr(self.w_instance))
> +                info = 'bound method %s of %s' % (name, objrepr)
> +                # info = "method %s of %s object" % (name, typename)
> +                return self.w_instance.getrepr(self.space, info)
> +
>          typename = w_class.getname(self.space, '?')
>          if self.w_instance is None:
>              s = "<unbound method %s.%s>" % (typename, name)
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev


From arigo at tunes.org  Sat Jul  1 14:38:03 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 1 Jul 2006 14:38:03 +0200
Subject: [pypy-dev] [pypy-svn] r29534
	-	pypy/dist/pypy/rpython/ootypesystem/test
In-Reply-To: <44A5735A.6090908@gmail.com>
References: <20060630140821.989F310081@code0.codespeak.net>
	<44A56302.2090808@strakt.com> <44A5735A.6090908@gmail.com>
Message-ID: <20060701123803.GA18685@code0.codespeak.net>

Hi Antonio,

On Fri, Jun 30, 2006 at 08:54:18PM +0200, Antonio Cuni wrote:
> Probably it's safer to store the name in some dictionary.

It's even required, yes: no mutating of the low-level type objects
please :-)  The case of Instance is a bit special in this respect.


A bientot,

Armin

From arigo at tunes.org  Sat Jul  1 21:51:53 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 1 Jul 2006 21:51:53 +0200
Subject: [pypy-dev] [dls2006@hpi.uni-potsdam.de: DLS 2006 -- Author
	Notifications Delayed]
Message-ID: <20060701195153.GA22357@code0.codespeak.net>

Hi all,

Looks like we won't know for another week:


----- Forwarded message from dls2006 at hpi.uni-potsdam.de -----

Date: Sat, 01 Jul 2006 21:16:12 +0200
From: dls2006 at hpi.uni-potsdam.de
To: arigo at tunes.org
Subject: DLS 2006 -- Author Notifications Delayed
Reply-To: dls2006 at hpi.uni-potsdam.de

Dear Authors --

Due to unforeseen circumstances author notifications are delayed, expected to be ready mid of next week.

My apologies for the inconvenience,
Robert

----- End forwarded message -----


Armin

From seojiwon at gmail.com  Tue Jul  4 17:55:40 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Tue, 4 Jul 2006 08:55:40 -0700
Subject: [pypy-dev] question on using ctypes
Message-ID: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>

Hi,

I added a module that uses ctypes under pypy/module/ but I'm getting
ImportError while running translate.py for targetpypystandalone.py

The error message is like,

..... cut ...
[translation:ERROR]     w_value = loader(space)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/interpreter/mixedmodule.py", line 135,
in ifileloader
[translation:ERROR]     d[name] = __import__(name, None, None, [name])
[translation:ERROR]  ImportError': No module named tcc
[translation:ERROR]     .. v0 = getattr(self_0, ('w_dict'))
[translation:ERROR] Processing block:
[translation:ERROR]  block at -1 is a <class
'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in
(pypy.interpreter.mixedmodule:33)MixedModule.MixedModule.getdictvalue
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v1 = getattr(space_0, ('finditem'))
[translation:ERROR]        v0 = getattr(self_0, ('w_dict'))
[translation:ERROR]        v2 = simple_call(v1, v0, w_name_0)
[translation:ERROR]        v3 = getattr(self_0, ('lazy'))
[translation:ERROR]        v4 = is_true(v3)
[translation:ERROR]  --end--

and the part I added is pypy/module/tcc/__init__.py and pypy/module/tcc/tcc.py

### __init__.py

# Package initialisation
from pypy.interpreter.mixedmodule import MixedModule

class Module(MixedModule):
    appleveldefs = {}
    interpleveldefs = {
        'print0':   'tcc.tcc_print0'
    }

### tcc.py
from pypy.interpreter.baseobjspace import ObjSpace
from ctypes import *

libtcc = CDLL('/path/to/lib.so')
print0 = libtcc.TCC_Print
print0.argtypes = [c_char_p]
print0.restype = c_int

def tcc_print0(space, msg):
    print0(msg)
tcc_print0.unwrap_spec = [ObjSpace, str]

What have I done wrong?

Thaniks,

-Jiwon

From amauryfa at gmail.com  Tue Jul  4 18:26:09 2006
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 4 Jul 2006 18:26:09 +0200
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
Message-ID: <e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>

2006/7/4, Jiwon Seo <seojiwon at gmail.com>:
> Hi,
>
> I added a module that uses ctypes under pypy/module/ but I'm getting
> ImportError while running translate.py for targetpypystandalone.py
>
> The error message is like,
>
> ..... cut ...
> [translation:ERROR]     w_value = loader(space)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/interpreter/mixedmodule.py", line 135,
> in ifileloader
> [translation:ERROR]     d[name] = __import__(name, None, None, [name])
> [translation:ERROR]  ImportError': No module named tcc
> [translation:ERROR]     .. v0 = getattr(self_0, ('w_dict'))
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at -1 is a <class
> 'pypy.objspace.flow.flowcontext.SpamBlock'>
> [translation:ERROR]  in
> (pypy.interpreter.mixedmodule:33)MixedModule.MixedModule.getdictvalue
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v1 = getattr(space_0, ('finditem'))
> [translation:ERROR]        v0 = getattr(self_0, ('w_dict'))
> [translation:ERROR]        v2 = simple_call(v1, v0, w_name_0)
> [translation:ERROR]        v3 = getattr(self_0, ('lazy'))
> [translation:ERROR]        v4 = is_true(v3)
> [translation:ERROR]  --end--
>
> and the part I added is pypy/module/tcc/__init__.py and pypy/module/tcc/tcc.py

Hello,

Don't know if it helps, but I have already seen this problem:
sometimes an ImportError can mask another ImportError.
The code in mixedmodule.py looks like this:
    try:
        d[name] = __import__(pkgroot+'.'+name, None, None, [name])
    except ImportError:
        d[name] = __import__(name, None, None, [name])

The first call is supposed to be the good one, but if it fails with an
ImportError, you get the error from the second call.
Is the ctypes module correctly installed?

-- 
Amaury Forgeot d'Arc

From seojiwon at gmail.com  Wed Jul  5 01:43:58 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Tue, 4 Jul 2006 16:43:58 -0700
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
Message-ID: <b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>

> Hello,
>
> Don't know if it helps, but I have already seen this problem:
> sometimes an ImportError can mask another ImportError.
> The code in mixedmodule.py looks like this:
>    try:
>        d[name] = __import__(pkgroot+'.'+name, None, None, [name])
>    except ImportError:
>        d[name] = __import__(name, None, None, [name])
>
> The first call is supposed to be the good one, but if it fails with an
> ImportError, you get the error from the second call.
> Is the ctypes module correctly installed?
>

Yeah, I think so. (I installed in my home directory, and I think I
properly set up PYTHONSTARTUP and did sys.path.append... in the
PYTHONSTARTUP file.
And, 'p name' in the pdb tells me it's tcc modue. So, I think ctypes
might not be related to this. (but I could be wrong)

Any other suggestions?

-Jiwon

From seojiwon at gmail.com  Wed Jul  5 09:25:12 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Wed, 5 Jul 2006 00:25:12 -0700
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
Message-ID: <b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>

On further inspection, it is actually ctypes problem. It seems like
setting sys.path in PYTHONSTARTUP file isn't enough. I guess I should
just install it in system-wide python site-package directory.

Thanks.

On 7/4/06, Jiwon Seo <seojiwon at gmail.com> wrote:
> > Hello,
> >
> > Don't know if it helps, but I have already seen this problem:
> > sometimes an ImportError can mask another ImportError.
> > The code in mixedmodule.py looks like this:
> >    try:
> >        d[name] = __import__(pkgroot+'.'+name, None, None, [name])
> >    except ImportError:
> >        d[name] = __import__(name, None, None, [name])
> >
> > The first call is supposed to be the good one, but if it fails with an
> > ImportError, you get the error from the second call.
> > Is the ctypes module correctly installed?
> >
>
> Yeah, I think so. (I installed in my home directory, and I think I
> properly set up PYTHONSTARTUP and did sys.path.append... in the
> PYTHONSTARTUP file.
> And, 'p name' in the pdb tells me it's tcc modue. So, I think ctypes
> might not be related to this. (but I could be wrong)
>
> Any other suggestions?
>
> -Jiwon
>

From seojiwon at gmail.com  Wed Jul  5 10:16:27 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Wed, 5 Jul 2006 01:16:27 -0700
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
	<b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
Message-ID: <b008462b0607050116l2fe0ac51m10c611b2d61bbccb@mail.gmail.com>

After properly installing ctypes, translate.py now gives this error.

##################################################################

[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "./translate.py", line 332, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 506, in
proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/tool/taskengine.py", line
108, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 195, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 208, in
task_annotate
[translation:ERROR]     s = annotator.build_types(self.entry_point,
self.inputtypes)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 100, in
build_types
[translation:ERROR]     return self.build_graph_types(flowgraph,
inputcells, complete_now=complete_now)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 143, in
build_graph_types
[translation:ERROR]     self.complete()
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 188, in
complete
[translation:ERROR]     self.processblock(graph, block)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 434, in
processblock
[translation:ERROR]     self.flowin(graph, block)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 490, in
flowin
[translation:ERROR]     self.consider_op(block.operations[i])
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 661, in
consider_op
[translation:ERROR]     argcells = [self.binding(a) for a in op.args]
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 238, in
binding
[translation:ERROR]     return self.bookkeeper.immutableconstant(arg)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 297, in
immutableconstant
[translation:ERROR]     return self.immutablevalue(const.value)
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 393, in
immutablevalue
[translation:ERROR]     result = SomePBC([self.getdesc(x)])
[translation:ERROR]    File
"/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 524, in
getdesc
[translation:ERROR]     return self.descs[pyobj]
[translation:ERROR]  TypeError: unhashable type
[translation:ERROR] Processing block:
[translation:ERROR]  block at -1 is a <class
'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in (pypy.module.tcc.tcc:9)tcc_print0
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v107 = simple_call((_FuncPtr TCC_Print), msg_0)
[translation:ERROR]  --end--

############# end of error message ##################################

pypy/module/tcc/__init__.py is like following
# Package initialisation
from pypy.interpreter.mixedmodule import MixedModule

class Module(MixedModule):
    appleveldefs = {}
    interpleveldefs = {
        'print0':   'tcc.tcc_print0'
    }

and pypy/module/tcc/tcc.py is like following

from pypy.interpreter.baseobjspace import ObjSpace
from ctypes import *

libtcc = CDLL('/u/jiwon/proj/less-bus/libLESS.so')
myprint0 = libtcc.TCC_Print
myprint0.argtypes = [c_char_p]
myprint0.restype = c_int

def tcc_print0(space, msg):
    return myprint0(msg)
tcc_print0.unwrap_spec = [ObjSpace, str]


What am I missing here?

-Jiwon

From amauryfa at gmail.com  Wed Jul  5 11:47:00 2006
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 5 Jul 2006 11:47:00 +0200
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <e27efe130607050245y5d268500jd8a47db4593bebdd@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
	<b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
	<b008462b0607050116l2fe0ac51m10c611b2d61bbccb@mail.gmail.com>
	<e27efe130607050245y5d268500jd8a47db4593bebdd@mail.gmail.com>
Message-ID: <e27efe130607050247m9dc6015g545b405cbfde6746@mail.gmail.com>

Hello,

2006/7/5, Jiwon Seo <seojiwon at gmail.com>:
> After properly installing ctypes, translate.py now gives this error.
>
> ##################################################################
>
> [translation:ERROR] Error:
> [translation:ERROR]  Traceback (most recent call last):
> [translation:ERROR]    File "./translate.py", line 332, in main
> [translation:ERROR]     drv.proceed(goals)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 506, in
> proceed
> [translation:ERROR]     return self._execute(goals, task_skip =
> self._maybe_skip())
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/tool/taskengine.py", line
> 108, in _execute
> [translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 195, in _do
> [translation:ERROR]     res = func()
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/translator/driver.py", line 208, in
> task_annotate
> [translation:ERROR]     s = annotator.build_types(self.entry_point,
> self.inputtypes)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 100, in
> build_types
> [translation:ERROR]     return self.build_graph_types(flowgraph,
> inputcells, complete_now=complete_now)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 143, in
> build_graph_types
> [translation:ERROR]     self.complete()
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 188, in
> complete
> [translation:ERROR]     self.processblock(graph, block)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 434, in
> processblock
> [translation:ERROR]     self.flowin(graph, block)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 490, in
> flowin
> [translation:ERROR]     self.consider_op(block.operations[i])
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 661, in
> consider_op
> [translation:ERROR]     argcells = [self.binding(a) for a in op.args]
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/annrpython.py", line 238, in
> binding
> [translation:ERROR]     return self.bookkeeper.immutableconstant(arg)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 297, in
> immutableconstant
> [translation:ERROR]     return self.immutablevalue(const.value)
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 393, in
> immutablevalue
> [translation:ERROR]     result = SomePBC([self.getdesc(x)])
> [translation:ERROR]    File
> "/u/jiwon/proj/pypy-dist/pypy/annotation/bookkeeper.py", line 524, in
> getdesc
> [translation:ERROR]     return self.descs[pyobj]
> [translation:ERROR]  TypeError: unhashable type
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at -1 is a <class
> 'pypy.objspace.flow.flowcontext.SpamBlock'>
> [translation:ERROR]  in (pypy.module.tcc.tcc:9)tcc_print0
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v107 = simple_call((_FuncPtr TCC_Print), msg_0)
> [translation:ERROR]  --end--
>
> ############# end of error message ##################################
>
> pypy/module/tcc/__init__.py is like following
> # Package initialisation
> from pypy.interpreter.mixedmodule import MixedModule
>
> class Module(MixedModule):
>     appleveldefs = {}
>     interpleveldefs = {
>         'print0':   'tcc.tcc_print0'
>     }
>
> and pypy/module/tcc/tcc.py is like following
>
> from pypy.interpreter.baseobjspace import ObjSpace
> from ctypes import *
>
> libtcc = CDLL('/u/jiwon/proj/less-bus/libLESS.so')
> myprint0 = libtcc.TCC_Print
> myprint0.argtypes = [c_char_p]
> myprint0.restype = c_int
>
> def tcc_print0(space, msg):
>     return myprint0(msg)
> tcc_print0.unwrap_spec = [ObjSpace, str]
>
>
> What am I missing here?

I think the currently supported way to compile modules using rctypes
is the pypy/bin/compilemodule.py script, and indeed your code
translates well with it.

I suggest (but I did not test) adding this line somewhere in
targetstandalone.py:
    import pypy.rpython.rctypes.implementation
This should register the annotation for calling ctypes functions.


-- 
Amaury Forgeot d'Arc

From stephen at thorne.id.au  Thu Jul  6 08:43:12 2006
From: stephen at thorne.id.au (Stephen Thorne)
Date: Thu, 6 Jul 2006 16:43:12 +1000
Subject: [pypy-dev] Writing os.pipe() using ctypes
In-Reply-To: 0
Message-ID: <20060706064312.29014.218909000.divmod.quotient.24486@ohm>

Hi,

I've been having fun this week trying to get twisted's unit tests running on top of pypy. This has the dual goals of getting twisted to run on a funky platform like pypy, and being able to give pypy a bit of a run around the yard. I have to report this is going "better than expected". I've already used rhymes's fcntl implementation and had to extend the posix module, I intend to do a diff once I get everything sorted.

I have a problem with rpython translation of my implementation of posix.pipe(). I have other ctypes implementations of functions (such as posix.access()) which work okay, but I'm having trouble with this particular case.

man 2 pipe says that I use it like this:

       int pipe(int filedes[2]);

My implementation looks like:

fd_arr = c_int * 2
def pipe(space):
    fds = fd_arr()
    fds[0] = 0
    fds[1] = 0
    if libc.pipe(fds):
        raise makeOSError(space)
    return space.wrap((fds[0], fds[1]))
pipe.unwrap_spec = [ObjSpace]

And the error I get from time {{{python translate.py --text --batch  standalone --thread}}} is:
[translation:ERROR]  Exception': unexpected prebuilt constant: <slot wrapper '__init__' of '_ctypes.Array' objects>
[translation:ERROR]     .. v0 = simple_call((ArrayType c_long_Array_2))
[translation:ERROR] Processing block:
[translation:ERROR]  block at 3 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
[translation:ERROR]  in (pypy.module.posix.ctypes_posix:43)pipe
[translation:ERROR]  containing the following operations:
[translation:ERROR]        v0 = simple_call((ArrayType c_long_Array_2))
[translation:ERROR]        v1 = setitem(v0, (0), (0))
[translation:ERROR]        v2 = setitem(v0, (1), (0))
[translation:ERROR]        v3 = getattr((<CDLL '/usr/lib/libc...20a89f0>), ('pipe'))
[translation:ERROR]        v4 = simple_call(v3, v0)
[translation:ERROR]        v5 = is_true(v4)
[translation:ERROR]  --end--

I've tried a bunch of ways of initialising the 'fds' object as to not make it a 'prebuilt constant', fds = fd_arr(0,0), creating fd_arr dynamically, fd_arr(zero, zero), etc. And I simply cannot seem to get rid of this exception.

So, any ideas as to how I go about making this work?

Stephen.

From amauryfa at gmail.com  Thu Jul  6 11:43:32 2006
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 6 Jul 2006 11:43:32 +0200
Subject: [pypy-dev] Writing os.pipe() using ctypes
In-Reply-To: <20060706064312.29014.218909000.divmod.quotient.24486@ohm>
References: <20060706064312.29014.218909000.divmod.quotient.24486@ohm>
Message-ID: <e27efe130607060243m5eed761fma545b5c976ac66e9@mail.gmail.com>

Hello,

2006/7/6, Stephen Thorne <stephen at thorne.id.au>:
> Hi,
>
> I've been having fun this week trying to get twisted's unit tests running on top of pypy. This has the dual goals of getting twisted to run on a funky platform like pypy, and being able to give pypy a bit of a run around the yard. I have to report this is going "better than expected". I've already used rhymes's fcntl implementation and had to extend the posix module, I intend to do a diff once I get everything sorted.
>
> I have a problem with rpython translation of my implementation of posix.pipe(). I have other ctypes implementations of functions (such as posix.access()) which work okay, but I'm having trouble with this particular case.
>
> man 2 pipe says that I use it like this:
>
>        int pipe(int filedes[2]);
>
> My implementation looks like:
>
> fd_arr = c_int * 2
> def pipe(space):
>     fds = fd_arr()
>     fds[0] = 0
>     fds[1] = 0
>     if libc.pipe(fds):
>         raise makeOSError(space)
>     return space.wrap((fds[0], fds[1]))
> pipe.unwrap_spec = [ObjSpace]
>
> And the error I get from time {{{python translate.py --text --batch  standalone --thread}}} is:
> [translation:ERROR]  Exception': unexpected prebuilt constant: <slot wrapper '__init__' of '_ctypes.Array' objects>
> [translation:ERROR]     .. v0 = simple_call((ArrayType c_long_Array_2))
> [translation:ERROR] Processing block:
> [translation:ERROR]  block at 3 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'>
> [translation:ERROR]  in (pypy.module.posix.ctypes_posix:43)pipe
> [translation:ERROR]  containing the following operations:
> [translation:ERROR]        v0 = simple_call((ArrayType c_long_Array_2))
> [translation:ERROR]        v1 = setitem(v0, (0), (0))
> [translation:ERROR]        v2 = setitem(v0, (1), (0))
> [translation:ERROR]        v3 = getattr((<CDLL '/usr/lib/libc...20a89f0>), ('pipe'))
> [translation:ERROR]        v4 = simple_call(v3, v0)
> [translation:ERROR]        v5 = is_true(v4)
> [translation:ERROR]  --end--
>
> I've tried a bunch of ways of initialising the 'fds' object as to not make it a 'prebuilt constant', fds = fd_arr(0,0), creating fd_arr dynamically, fd_arr(zero, zero), etc. And I simply cannot seem to get rid of this exception.
>
> So, any ideas as to how I go about making this work?
>
> Stephen.
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>

It seems that translate.py does not "register" annotation for rctypes objects.
In a previous mail I suggested to add
    import pypy.rpython.rctypes.implementation
in pypy/translator/goal/translate.py and translation seems to go further.
(I'm running
    translate.py standalone --usemodules=_demo
after I added your code in demo.py)

Unfortunately the complete translation takes too much memory for the
machine I'm currently working on, so I could not test completely.
And since its a Windows box, I had to remove the libc.pipe call completely...

Hope this helps.
It seems that many people are interested in the new rctypes way to
create modules...

-- 
Amaury Forgeot d'Arc

From arigo at tunes.org  Thu Jul  6 13:13:30 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 6 Jul 2006 13:13:30 +0200
Subject: [pypy-dev] question on using ctypes
In-Reply-To: <e27efe130607050247m9dc6015g545b405cbfde6746@mail.gmail.com>
References: <b008462b0607040855l798a8c2dodd66255066216f72@mail.gmail.com>
	<e27efe130607040926n370069b2uc02bf388c191dd61@mail.gmail.com>
	<b008462b0607041643t6cc3d1cel125092de786a0f6f@mail.gmail.com>
	<b008462b0607050025w723fb306nf8c8c83dd618c77c@mail.gmail.com>
	<b008462b0607050116l2fe0ac51m10c611b2d61bbccb@mail.gmail.com>
	<e27efe130607050245y5d268500jd8a47db4593bebdd@mail.gmail.com>
	<e27efe130607050247m9dc6015g545b405cbfde6746@mail.gmail.com>
Message-ID: <20060706111330.GA29352@code0.codespeak.net>

Hi,

On Wed, Jul 05, 2006 at 11:47:00AM +0200, Amaury Forgeot d'Arc wrote:
> I suggest (but I did not test) adding this line somewhere in
> targetstandalone.py:
>     import pypy.rpython.rctypes.implementation
> This should register the annotation for calling ctypes functions.

Indeed, this module needs to be imported from somewhere.  I'd recommend
to import it from tcc.py directly, even.  See the end of the following
section:

  http://codespeak.net/pypy/dist/pypy/doc/rctypes.html#restrictions


A bientot,

Armin

From cfbolz at gmx.de  Fri Jul  7 14:03:17 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 7 Jul 2006 14:03:17 +0200
Subject: [pypy-dev] PyPy is faster than CPython (given an evil enough
	benchmark)
Message-ID: <348899050607070503o62fe5edaw134ea2953bb17b92@mail.gmail.com>

Hi all!

Together with the optimized string objects that Armin, Pieter,
Alexander and me have been working on, it is possible to construct a
(very corner-casish) benchmark, which exploits the fast cases of PyPy
in such a way, that it is quite a bit faster CPython. See the
following code:

import time, sys

def test_speed(num):
    t1 = time.time()
    result = ""
    intermediate = []
    for i in range(num):
        result += str(i)
        intermediate.append(result)
    t2 = time.time()
    return t2 - t1, intermediate

t1 = time.time()
for i in range(1, 12):
    speed, intermediate = test_speed(i * 1000)
t2 = time.time()
print t2 - t1


It runs 3.4 times faster on pypy-c with WITHSTRSLICE = WITHSTRJOIN =
True. In addition CPython uses 25 times more memory (which of course
accounts for some of the slowness).

Of course the lesson of this is that benchmarks are worthless :-).

Cheers,

Carl Friedrich

From seojiwon at gmail.com  Sat Jul  8 21:51:56 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Sat, 8 Jul 2006 12:51:56 -0700
Subject: [pypy-dev] Question on pypy exception...
Message-ID: <b008462b0607081251r3a07b15dldfd925ade56fe533@mail.gmail.com>

I'm getting an exception with pypy-c, and the exception occurs right
after pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr function in
pypy_g_PyInterpFrame_dispatch_translated.

1. Is "pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr" same as
LL_thread_fused_releaseacquirelock ? I can only see "#define
LL_thread_fused_releaseacquirelock
pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr", but not the other
way around.

2. Is that function there for forcing context switching?

3. What could happen in
pypy_g_ll_fused_releaseacquirelock__ThreadLockPtr() that can set
(&pypy_g_pypy_translator_c_exceptiontransform_ExcData)->ed_inst_exc_type?
the error message that I'm seeing is like following

propagating implement_6.c:100878 pypy_g_PyInterpFrame_dispatch_translated error
propagating implement_5.c:30508 pypy_g_PyFrame_eval error
propagating implement_5.c:37782 pypy_g_EvalFrame_resume error
propagating implement_39.c:41465 pypy_g_ll_deallocator__Address_68 error
propagating implement.c:27685 pypy_g_funccall_star1 error
propagating implement_39.c:32105 pypy_g_ll_deallocator__Address_14 error
propagating implement_39.c:32158 pypy_g_ll_deallocator__Address_15 error
propagating implement_39.c:32294 pypy_g_ll_deallocator__Address_17 error
propagating implement.c:1336 pypy_g_entry_point error

Thanks!

-Jiwon

From mwh at python.net  Sun Jul  9 13:54:17 2006
From: mwh at python.net (Michael Hudson)
Date: Sun, 09 Jul 2006 12:54:17 +0100
Subject: [pypy-dev] PyPy Post-EuroPython 2006 Sprint Report
Message-ID: <2mejwv3seu.fsf@starship.python.net>

Greetings from an assortment of tired brains at CERN, home to a larger
assortment of hopefully less tired physicist brains.  We are reaching
the end of a succesful and well-attended (24 participants!) sprint.

3.5 days times 24 people is a bit more than two man months of work --
we are going to try to cover all we've achieved, but will undoubtedly
forget some stuff!

Summer of Code student Lawrence and his mentor Arre spent the majority
of the time integrating Lawrence's SoC work -- ctypes implementations
of various CPython extension modules -- into PyPy.  He has implemented
the time, mmap and fcntl modules so far, and the work at this sprint
concentrated on putting the time module into the form required for
PyPy -- making it a mixed module, conforming to the restrictions of
rctypes and fighting the extension compiler.

Brian Sutherland and Sad Rejeb also joined in this play, attempting to
convert some prewritten performance sensitive caching code used in
Zope(!) somewhere to a mixed module suitable for feeding to the
extension compiler.  Unfortunately the resulting code was no faster
than the pure Python version... probably because of the ext-compiler
using refcounting for GC.

Antonio appeared to spend his entire time writing commit messages but
somehow must have written some code in between (with the help of *his*
SoC mentor, Armin) because now all of PyPy's Standard Interpreter can
be rtyped with the ootypesystem.  "All" that remains are the dict
implementation and external function support in the CLI backend before
IronPython has a competitor in the shape of PyPy.Net :-)

Guido (not *that* Guido) being the JavaScript deity he is wrote a JS
tool to instrospect the DOM methods and produce useful information for
Maciej's SoC (we only had three SoC students here...) project.
Unfortunately browsers' JavaScript implementations are famously
eccentric and running his tool popped up the Print dialog box twice
and then crashed Epiphany.  It proved marginally less frustrating to
write a scraper for Mozilla's documentation and fix the broken HTML
he found there.

Aur?lien, Ludovic and part of Armin tried to figure out why the
example of coroutine cloning written for the "What can PyPy do for
you?" talk didn't work.  Assorted mind boggling bugs were found and
fixed, with the aggravating result that it now works fine when run on
the llinterpreter but crashes -- after a while -- when translated to
C.  More wizardry will be required here...

Just before the sprint Maciej attempted to improve the annotator's
error messages and he and Samuele spent some time fixing this new code
(it's very frustrating when your error printing routines crash on
you...).

A varying team of people including but not limited to Pieter, Carl
Friedrich, Maciej and Alexander worked on alternative, hopefully
faster implementations of Python data types, particularly strings and
dictionaries.  We now have efficient representations of slices of
strings and for the result of adding two strings, all completely
transparent to application-level.  For a sufficiently cooked
benchmark, we are now up to seven times faster than CPython (and use
30 times less memory).  On the dictionary side, we stole an
optimization for CPython: assume on creation that a dictionary will
only contain strings keys and switch to an alternative implementation
on the insertion of a non-string key.  This makes pystone and richards
both around 20% faster, some benchmarks up to 40% faster (but also
some a few percent slower).

Alexandre and David started to think about configuration and option
handling, but got distracted when they tried to run a translation on a
64-bit machine (or more specifically, a machine with 64-bit userland),
because it broke in obscure ways.  A few obscure fixes later and it
worked again apart from stackless, and another few fixes fixed that
too.

Michael and Fabrizio worked on the stackless code, "un-inlining" some
of the frame-state saving code with the goal of reducing the size of
the generated code.  This involved taking Fabrizio on a whirlwind tour
of one of the scarier areas of the PyPy codebase, but he survived and
the feature was implemented, although the size reduction was not as
great as had been hoped -- more investigation required (we also need
to build a stacklessgc build to check that).

Michael and Fabrizio then worked on AST- and bytecode-level
optimizations, which mostly involved building an infrastructure to
make modifications of the AST easier.  They implemented a simple
constant folder at the AST level and at the bytecode level an
optimization to remove the temporary tuple in code like "x,y=y,x".

Simon and Eric resurrected and experimented more with LLVM's JIT
interface.  Simon then worked with Ludovic and Arre on implementing
RPython support for numarrays and Eric worked on making the output of
genc compilable with C++.

Guido and Holger implemented the beginning and perhaps the middle but
probably not the end of doctest support in py.test.

After some discussion with Alexandre, David and Holger, Guido and Carl
Friedrich began to work on the rather ugly-looking task of taming
pypy's growing menagerie of options.  They started to implement a
general mechanism for defining a hierarchy of options, handling
dependencies and conflicts among them.  From this you can generate
optparse Options for command line interfaces and will one day be able
to generate a web form to choose among the zoo of pypy's options.
They used the config mechanisms to de-insane the selection of the
various object space variants.  This, for example, makes automated
testing of the above-mentioned variant string and dictionary
implementations possible.

Pieter worked on random modules^W^Wthe random module, porting it to
RPython and implementing some missing functions.

So, after a pre-sprint (for some of us), a conference and a
post-sprint, we are well prepared for the usual post-sprint week long
sleep.  Apart from Carl, he has an exam on Friday.  Oops!

Met vriendelijke groeten,
mwh&cfbolz

-- 
   This proposal, if accepted, will probably mean a heck of a lot of
   work for somebody.  But since I don't want it accepted, I don't
   care.                                   -- Laura Creighton, PEP 666


From arigo at tunes.org  Mon Jul 10 16:12:36 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 10 Jul 2006 16:12:36 +0200
Subject: [pypy-dev] [emil@dmr.ath.cx: Re: [pypy-svn] r29899 -
	pypy/extradoc/talk/ep2006]
Message-ID: <20060710141236.GA27105@code0.codespeak.net>

Hi all,

----- Forwarded message from Emil Mikulic <emil at dmr.ath.cx> -----

Date: Mon, 10 Jul 2006 17:23:05 +1000
From: Emil Mikulic <emil at dmr.ath.cx>
To: arigo at codespeak.net
Subject: Re: [pypy-svn] r29899 - pypy/extradoc/talk/ep2006

On Sun, Jul 09, 2006 at 03:35:03PM +0200, arigo at codespeak.net wrote:
> http://snake.cs.uni-duesseldorf.de/tgtest/

I see:

	Bad Gateway

	The proxy server received an invalid response from an upstream server.
	Apache Server at snake.cs.uni-duesseldorf.de Port 80

=/

----- End forwarded message -----

From faassen at infrae.com  Tue Jul 11 14:14:46 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Tue, 11 Jul 2006 14:14:46 +0200
Subject: [pypy-dev] PyPy is faster than CPython (given an evil enough
 benchmark)
In-Reply-To: <348899050607070503o62fe5edaw134ea2953bb17b92@mail.gmail.com>
References: <348899050607070503o62fe5edaw134ea2953bb17b92@mail.gmail.com>
Message-ID: <44B39636.1030804@infrae.com>

Carl Friedrich Bolz wrote:
[snip]
> Of course the lesson of this is that benchmarks are worthless :-).

Cool! I'll go tell everybody that PyPy is already faster than CPython 
now. :)

The best benchmarks will be large real-world codebases running on PyPy.

Regards,

Martijn


From l.oluyede at gmail.com  Tue Jul 11 18:00:12 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Tue, 11 Jul 2006 18:00:12 +0200
Subject: [pypy-dev] Bug hunting in rctime becomes wtf
Message-ID: <9eebf5740607110900l1ba16f2fu4eb220528c978494@mail.gmail.com>

I had a nasty bug like this one in the compilation process of the rctime module:

[translation:ERROR] Error:
[translation:ERROR]  TyperError: non-constant tuple index
[translation:ERROR] .. block at 34 with 1 exits
[translation:ERROR] .. v0 = getitem(tup_0, ___i_0)

You can reproduce it decommenting asctime function from interp_time.py
and in the __init__.py file of the rctime module.

After some digging I've found the source of the problem in the code
Armin and I have checked-in at the sprint to add the support for
"*args" arguments (see visit_args_w in pypy.objspace.cpy.function). I
tried patching it but I'm not very into cpy objspace and the compiler
mechanism under the hood.

The "wtf" thing came out when I tried to use the file with this
"stupid" patch applied:

rhymes at groove:~/scc/pypy/pypy/objspace/cpy $ svn diff
Index: function.py
===================================================================
--- function.py (revision 29953)
+++ function.py (working copy)
@@ -61,7 +61,7 @@
         tramp.inputargs.append('*' + basename)
         tramp.wrappings.append('%s = []' % (argname,))
         tramp.wrappings.append('for ___i in range(len(%s)):' % (basename,))
-        tramp.wrappings.append('    %s.append(___W_Object(%s[___i]))' % (
+        tramp.wrappings.append('    %s.append(___W_Object(%s[0]))' % (
             argname, basename))
         tramp.passedargs.append(argname)

The thing surprised me is the compilation after the patch goes fine
and (!!) the module works correctly. asctime() does it's dirty job.

So, why does it work?

-- 
Lawrence
http://www.oluyede.org/blog

From stephen at thorne.id.au  Wed Jul 12 06:56:35 2006
From: stephen at thorne.id.au (Stephen Thorne)
Date: Wed, 12 Jul 2006 14:56:35 +1000
Subject: [pypy-dev] PyPy is faster than CPython (given an evil enough
 benchmark)
In-Reply-To: <44B39636.1030804@infrae.com>
Message-ID: <20060712045635.29014.1792869848.divmod.quotient.32800@ohm>

On Tue, 11 Jul 2006 14:14:46 +0200, Martijn Faassen <faassen at infrae.com> wrote:
>Carl Friedrich Bolz wrote:
>[snip]
>> Of course the lesson of this is that benchmarks are worthless :-).
>
>Cool! I'll go tell everybody that PyPy is already faster than CPython
>now. :)
>
>The best benchmarks will be large real-world codebases running on PyPy.

This is precisely the reason I'm putting effort into getting the twisted unit tests to run on top of PyPy. Once we get the tests running on translated pypy, we can add a column to the http://twistedmatrix.com/buildbot/ builder, and get PyPy up as a platform we can run a harsh series of non-trivial unit tests on.

My work on this has stalled currently as there are a couple bugs in translation, but with the work Lawrence is doing on things like the posix module and fcntl, we're well on our way to being able to get this going.

FWIW, i've already managed to get twisted to run its own unit tests on its unit testing engine (trial) using py.py. Once we get fcntl translatable, I hope to report what happens when I run the lot (2764 tests) under pypy-c and 'see what happens'.

250 tests took approximately 11 hours to run.

Stephen.

From hpk at trillke.net  Thu Jul 13 08:02:29 2006
From: hpk at trillke.net (holger krekel)
Date: Thu, 13 Jul 2006 08:02:29 +0200
Subject: [pypy-dev] Security ideas
In-Reply-To: <20060524114728.GA6374@code0.codespeak.net>
References: <20060524114728.GA6374@code0.codespeak.net>
Message-ID: <20060713060229.GR32712@solar.trillke>

Hi Armin, 

On Wed, May 24, 2006 at 13:47 +0200, Armin Rigo wrote:
> On Monday I was at an inspiring seminar about (a specific form of)
> language-level security.  I've collected the PyPy-ification of these
> ideas there:
> 
> http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt

IIUC, in this discussion you argue against an "Proxy-Object space" solution
because a code block may depend on a condition involving secret values. 

In your alternative "annotator" suggestion, you give the following example: 

    def enter_bid(n):
        if n > highest_bid.value:
            highest_bid.value = n 
    enter_bid = secure(enter_bid)

Here the annotator analysis is supposed to prevent a leak of information
from the secret value.  But if the if-branch additionally contains:

            num_bids += 1

don't you run into a branching/code-dependent-on-secret-condition 
problem again?  Would the annotator prevent the manipulation of 
the global 'num_bids'?  Would it need to be a public value? 

Moreover, i have practical concerns: your proposed
scheme requires RPython annotator analysis which implies to
have the PyPy tool chain available and accessible at programming 
time.  Not impossible but also not a use case that we went for 
so far.  Also it is not clear to which target "secure" would compile 
functions to, C or bytecode or ...?

best, 

    holger

From Ben.Young at risk.sungard.com  Fri Jul 14 13:12:59 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Fri, 14 Jul 2006 12:12:59 +0100
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
In-Reply-To: <20060714105155.DC18A10090@code0.codespeak.net>
Message-ID: <OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>

Hi Xoraxax (sorry, I don't know your real name!),

This looks really good! It shows there are still some major (easy) 
performance wins in PyPy.

One question, why would you need any calls to str2object in any of the 
non-mutating methods? Surely the class knows that no non-strings have been 
added, and if __getitem__ is called with a non-string, you could just 
throw KeyError. I guess this is becuase you have to replicate the fact 
that hash is always called, but couldn't you do:

            if space.is_w(w_lookup_type, space.w_str):
                return w_dict.content_str[space.str_w(w_lookup)]
            else:
                w_lookup.__hash__()
                raise KeyError

?

Keep up the good work (to all PyPy's). Things are really starting to look 
good!

Cheers,
Ben

pypy-svn-bounces at codespeak.net wrote on 14/07/2006 11:51:55:

> Author: xoraxax
> Date: Fri Jul 14 12:51:55 2006
> New Revision: 30034
> 
> Modified:
>    pypy/dist/pypy/objspace/std/dictstrobject.py
> Log:
> Speeded up dict creation, added shortcut for look ups of vars with 
> sane hashes, fixed get mm, fixed mutation handling in the iterators.
> (manual merge of changes done in the other branch)
> 
> Modified: pypy/dist/pypy/objspace/std/dictstrobject.py
> 
==============================================================================
> --- pypy/dist/pypy/objspace/std/dictstrobject.py   (original)
> +++ pypy/dist/pypy/objspace/std/dictstrobject.py   Fri Jul 14 12:51:55 
2006
> @@ -1,4 +1,3 @@
> -
>  from pypy.objspace.std.objspace import *
>  from pypy.interpreter import gateway
> 
> @@ -7,52 +6,52 @@
>  class W_DictStrObject(W_Object):
>      from pypy.objspace.std.dicttype import dict_typedef as typedef
> 
> -    def __init__(w_self, space, w_otherdict=None):
> +    def __init__(w_self, space):
>          w_self.space = space
> -        if w_otherdict is None:
> -            w_self.content = None
> -            w_self.content_str = {}
> -        else:
> -            if w_otherdict.content is None:
> -                w_self.content = None
> -                w_self.content_str = w_otherdict.content_str.copy()
> -            else:
> -                w_self.content = w_otherdict.content.copy()
> -                w_self.content_str = None
> +        w_self.content = None
> +        w_self.content_str = {}
> 
>      def initialize_content(w_self, list_pairs_w): # YYY
>          for w_k, w_v in list_pairs_w:
>              w_self.setitem(w_k, w_v)
> 
>      def getitem(w_dict, w_lookup):
> +        """ Helper function, raises rpython exceptions. """
>          space = w_dict.space
>          if w_dict.content is None:
> -            if space.is_w(space.type(w_lookup), space.w_str):
> +            w_lookup_type = space.type(w_lookup)
> +            if space.is_w(w_lookup_type, space.w_str):
>                  return w_dict.content_str[space.str_w(w_lookup)]
>              else:
> +                if w_dict.isSaneHash(w_lookup_type):
> +                    raise KeyError
>                  w_dict.str2object()
>          return w_dict.content[w_lookup]
> 
> -    def get(w_dict, w_lookup):
> +    def get(w_dict, w_lookup, w_default):
> +        """ Helper function, raises rpython exceptions. """
>          space = w_dict.space
>          if w_dict.content is None:
> -            if space.is_w(space.type(w_lookup), space.w_str):
> -                return w_dict.content_str.get(space.str_w(w_lookup), 
None)
> +            w_lookup_type = space.type(w_lookup)
> +            if space.is_w(w_lookup_type, space.w_str):
> +                return w_dict.content_str.get(space.
> str_w(w_lookup), w_default)
>              else:
> +                if w_dict.isSaneHash(w_lookup_type):
> +                    return w_default
>                  w_dict.str2object()
> -        return w_dict.content.get(w_lookup, None)
> +        return w_dict.content.get(w_lookup, w_default)
> 
> 
>      def setitem(w_self, w_k, w_v):
>          space = w_self.space
> -        if w_self.content is None:
> +        if w_self.content is not None:
> +            w_self.content[w_k] = w_v
> +        else:
>              if space.is_w(space.type(w_k), space.w_str):
>                  w_self.content_str[space.str_w(w_k)] = w_v
>              else:
>                  w_self.str2object()
>                  w_self.content[w_k] = w_v
> -        else:
> -            w_self.content[w_k] = w_v
> 
>      set_str_keyed_item = setitem
> 
> @@ -65,7 +64,17 @@
>          for k, w_v in w_self.content_str.items():
>              w_self.content[w_self.space.wrap(k)] = w_v
>          w_self.content_str = None
> - 
> +
> +    def isSaneHash(w_self, w_lookup_type):
> +        """ Handles the case of a non string key lookup.
> +        Types that have a sane hash/eq function should allow us to 
> return True
> +        directly to signal that the key is not in the dict in any case.
> +        XXX The types should provide such a flag. """
> + 
> +        space = w_self.space
> +        # XXX there are much more types
> +        return space.is_w(w_lookup_type, space.w_NoneType) or 
> space.is_w(w_lookup_type, space.w_int)
> +
>      def __repr__(w_self):
>          """ representation for debugging purposes """
>          return "%s(%s,%s)" % (w_self.__class__.__name__, w_self.
> content, w_self.content_str)
> @@ -144,7 +153,11 @@
> 
>  def contains__DictStr_ANY(space, w_dict, w_lookup):
>      if w_dict.content is None:
> -        if not space.is_w(space.type(w_lookup), space.w_str):
> +        w_lookup_type = space.type(w_lookup)
> +        if not space.is_w(w_lookup_type, space.w_str):
> +            if w_dict.isSaneHash(w_lookup_type):
> +                return space.w_False
> +            #foo("degenerated in contains: " + space.
> type(w_lookup).getname(space, '?'))
>              w_dict.str2object()
>          else:
>              return space.newbool(space.str_w(w_lookup) in 
w_dict.content_str)
> @@ -247,7 +260,14 @@
>      return w_res
> 
>  def dict_copy__DictStr(space, w_self):
> -    return W_DictStrObject(space, w_self)
> +    w_new_dict = W_DictStrObject(space)
> +    if w_self.content is None:
> +        w_new_dict.content = None
> +        w_new_dict.content_str = w_self.content_str.copy()
> +    else:
> +        w_new_dict.content = w_self.content.copy()
> +        w_new_dict.content_str = None
> +    return w_new_dict
> 
>  def dict_items__DictStr(space, w_self):
>      if w_self.content is not None:
> @@ -291,9 +311,7 @@
>          w_self.content.clear()
> 
>  def dict_get__DictStr_ANY_ANY(space, w_dict, w_lookup, w_default):
> -    if w_dict.content is None:
> -        return w_dict.content_str.get(space.str_w(w_lookup), w_default)
> -    return w_dict.content.get(w_lookup, w_default)
> +    return w_dict.get(w_lookup, w_default)
> 
>  app = gateway.applevel('''
>      def dictrepr(currently_in_repr, d):
> @@ -348,6 +366,12 @@
>      def return_entry(w_self, w_key, w_value):
>          raise NotImplementedError
> 
> +    def handleMutation(w_self):
> +        space = w_self.space
> +        w_self.len = -1   # Make this error state sticky
> +        raise OperationError(space.w_RuntimeError,
> +                 space.wrap("dictionary changed size during 
iteration"))
> +
>  registerimplementation(W_DictStrIterObject)
> 
>  class W_DictStrIter_Keys(W_DictStrIterObject):
> @@ -435,16 +459,25 @@
>      return w_dictiter
> 
>  def next__DictStrIterObject(space, w_dictiter):
> -    # XXX hum, this is broken currently
> -    if w_dictiter.content_str is not None and w_dictiter.
> w_dictobject.content_str is None:
> -        raise OperationError(space.w_RuntimeError,
> -                space.wrap("you stupid dictionary user - you have 
> just done an operation internally mutating the dictionary"))
> -
> -    if w_dictiter.content is not None:
> +    # iterate over the string dict even if the dictobject's data was 
forced
> +    # to degenerate. just bail out if the obj's dictionary size 
differs.
> +    if (w_dictiter.content_str is not None and w_dictiter.
> w_dictobject.content_str is None
> +        and len(w_dictiter.w_dictobject.content) != w_dictiter.len):
> +        w_dictiter.handleMutation()
> +
> +    if w_dictiter.content_str is not None:
> +        if w_dictiter.len != len(w_dictiter.content_str):
> +            w_dictiter.handleMutation()
> +        # look for the next entry
> +        w_result = w_dictiter.next_entry()
> +        if w_result is not None:
> +            w_dictiter.pos += 1
> +            return w_result
> +        # no more entries
> +        w_dictiter.content_str = None
> +    elif w_dictiter.content is not None:
>          if w_dictiter.len != len(w_dictiter.content):
> -            w_dictiter.len = -1   # Make this error state sticky
> -            raise OperationError(space.w_RuntimeError,
> -                     space.wrap("dictionary changed size during 
iteration"))
> +            w_dictiter.handleMutation()
>          # look for the next entry
>          w_result = w_dictiter.next_entry()
>          if w_result is not None:
> @@ -452,19 +485,6 @@
>              return w_result
>          # no more entries
>          w_dictiter.content = None
> -    else:
> -        if w_dictiter.content_str is not None:
> -            if w_dictiter.len != len(w_dictiter.content_str):
> -                w_dictiter.len = -1   # Make this error state sticky
> -                raise OperationError(space.w_RuntimeError,
> -                         space.wrap("dictionary changed size during
> iteration"))
> -            # look for the next entry
> -            w_result = w_dictiter.next_entry()
> -            if w_result is not None:
> -                w_dictiter.pos += 1
> -                return w_result
> -            # no more entries
> -            w_dictiter.content_str = None
>      raise OperationError(space.w_StopIteration, space.w_None)
> 
>  def len__DictStrIterObject(space, w_dictiter):
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
> 


From hpk at trillke.net  Fri Jul 14 13:40:55 2006
From: hpk at trillke.net (holger krekel)
Date: Fri, 14 Jul 2006 13:40:55 +0200
Subject: [pypy-dev] fixing ireland sprint (21th-27th august)
Message-ID: <20060714114055.GH32712@solar.trillke>

Hi folks, 

i took over trying to organise the Ireland sprint (scheduled
21st - 27th August currently) but i had problems getting feedback 
from the organisers yet.  Therefore, i am not sure about some 
details but i started a draft announcement anyway: 

    extradoc/sprintinfo/ireland-2006/announcement.txt

and also listed people from whom i have heard that they want to come

    extradoc/sprintinfo/ireland-2006/announcement.txt

Please check and add yourself already. 

However, how do we go on: do we just assume that the sprint
works out (it's definitely very likely) and book travels 
arriving 20th evening and leaving 27th or 28th? 

Or do we wait until - say - end next week in order
to be sure?  Anyway, if you book flights already, 
then make sure you have some insurance that you 
can step back and get most of your money back
(that's probably the strategy i am going to follow
myself). 

any comments? 

    holger

From seojiwon at gmail.com  Sat Jul 15 07:17:28 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Fri, 14 Jul 2006 22:17:28 -0700
Subject: [pypy-dev] cpy object space
Message-ID: <b008462b0607142217y3d8b2972sf64534c8d4288058@mail.gmail.com>

pypy/bin/py.py does not work with cpy object space

[llcoolj] bin > ./py.py -o cpy
[cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c
ctypesplatcheck_0.c -o ctypesplatcheck_0.o
[cbuild:execute] cc -pthread /tmp/usession-23/ctypesplatcheck_0.o -lm
-lpthread -o /tmp/usession-23/ctypesplatcheck_0
Traceback (most recent call last):
  File "./py.py", line 204, in ?
    sys.exit(main_(sys.argv))
  File "./py.py", line 82, in main_
    space.setitem(space.sys.w_dict,space.wrap('executable'),space.wrap(argv[0]))
AttributeError: 'CPyObjSpace' object has no attribute 'sys'

Having almost only introductory knowledge of pypy, I thought naively
following might solve the problem

class CPyObjSpace(baseobjspace.ObjSpace):
    def initialize(self):
        self.config.objspace.geninterp = False
        self.wrap_cache = {}
        self.sys = self.getbuiltinmodule("sys") ##### <-- adding just this line


but now it errors that it can't find w_dict attribute in space.sys
(py.py:main_() #82)  - still space.sys is W_Object. Maybe I need to
transform it as proper pypy interpreter module?

From seojiwon at gmail.com  Sat Jul 15 22:26:23 2006
From: seojiwon at gmail.com (Jiwon Seo)
Date: Sat, 15 Jul 2006 13:26:23 -0700
Subject: [pypy-dev] cpy object space
In-Reply-To: <b008462b0607142217y3d8b2972sf64534c8d4288058@mail.gmail.com>
References: <b008462b0607142217y3d8b2972sf64534c8d4288058@mail.gmail.com>
Message-ID: <b008462b0607151326m1db3196fl81c9408e0a18033f@mail.gmail.com>

Never mind. I just heard that cpy obj space is for mixed module translation.
I thought I'd try thunk obj space with cpy obj space, but I rather not. ;)

-jiwon

On 7/14/06, Jiwon Seo <seojiwon at gmail.com> wrote:
> pypy/bin/py.py does not work with cpy object space
>
> [llcoolj] bin > ./py.py -o cpy
> [cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c
> ctypesplatcheck_0.c -o ctypesplatcheck_0.o
> [cbuild:execute] cc -pthread /tmp/usession-23/ctypesplatcheck_0.o -lm
> -lpthread -o /tmp/usession-23/ctypesplatcheck_0
> Traceback (most recent call last):
>  File "./py.py", line 204, in ?
>    sys.exit(main_(sys.argv))
>  File "./py.py", line 82, in main_
>    space.setitem(space.sys.w_dict,space.wrap('executable'),space.wrap(argv[0]))
> AttributeError: 'CPyObjSpace' object has no attribute 'sys'
>
> Having almost only introductory knowledge of pypy, I thought naively
> following might solve the problem
>
> class CPyObjSpace(baseobjspace.ObjSpace):
>    def initialize(self):
>        self.config.objspace.geninterp = False
>        self.wrap_cache = {}
>        self.sys = self.getbuiltinmodule("sys") ##### <-- adding just this line
>
>
> but now it errors that it can't find w_dict attribute in space.sys
> (py.py:main_() #82)  - still space.sys is W_Object. Maybe I need to
> transform it as proper pypy interpreter module?
>

From 2006a at usenet.alexanderweb.de  Fri Jul 14 18:09:13 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Fri, 14 Jul 2006 18:09:13 +0200
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
References: <20060714105155.DC18A10090@code0.codespeak.net>
	<OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>
Message-ID: <1p6bpe8fz3vkg$.dlg@usenet.alexanderweb.de>

Hi Ben Young,

On Fri, 14 Jul 2006 12:12:59 +0100, Ben.Young at risk.sungard.com wrote:

> This looks really good! It shows there are still some major (easy) 
> performance wins in PyPy.

Yes, I measured 35% speedup on richards/pystone on "the other branch" (that
solely knows this kind of dictionary unlike the current trunk).

> One question, why would you need any calls to str2object in any of the 
> non-mutating methods?

Because every class can define it's own hash/eq functions that might be
considered insane in case of having them return hashes that match the
hashes of a string. That would mean that {"foo": 1}[myClass()] could return
1. Therefore we have to switch the mode while having a shortcut for types
which are known to have sane hashes only.

>                 w_lookup.__hash__()

I didn't even consider that people would write __hash__ methods with
side-effects :)

Kind regards,
Alexander


From 2006a at usenet.alexanderweb.de  Sun Jul 16 23:40:01 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Sun, 16 Jul 2006 23:40:01 +0200
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
References: <20060714105155.DC18A10090@code0.codespeak.net>
	<OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>
Message-ID: <nbhefnx5b0oe.dlg@usenet.alexanderweb.de>

Hi Ben Young,

On Fri, 14 Jul 2006 12:12:59 +0100, Ben.Young at risk.sungard.com wrote:

> This looks really good! It shows there are still some major (easy) 
> performance wins in PyPy.

Yes, I measured 35% speedup on richards/pystone on "the other branch" (that 
solely knows this kind of dictionary unlike the current trunk).

> One question, why would you need any calls to str2object in any of the 
> non-mutating methods?

Because every class can define it's own hash/eq functions that might be 
considered insane in case of having them return hashes that match the 
hashes of a string. That would mean that {"foo": 1}[myClass()] could return 
1. Therefore we have to switch the mode while having a shortcut for types 
which are known to have sane hashes only.

>                 w_lookup.__hash__()

I didn't even consider that people would write __hash__ methods with 
side-effects :)

Kind regards,
Alexander, who has resent this posting 2 days after the initial try - Gmane
problems?


From Ben.Young at risk.sungard.com  Mon Jul 17 09:56:38 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 17 Jul 2006 08:56:38 +0100
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
In-Reply-To: <nbhefnx5b0oe.dlg@usenet.alexanderweb.de>
Message-ID: <OF3239899F.71F77371-ON802571AE.002B5B07-802571AE.002BA2EC@risk.sungard.com>

pypy-dev-bounces at codespeak.net wrote on 16/07/2006 22:40:01:

> Hi Ben Young,

Hi Alexander,

> 
> On Fri, 14 Jul 2006 12:12:59 +0100, Ben.Young at risk.sungard.com wrote:
> 
> > This looks really good! It shows there are still some major (easy) 
> > performance wins in PyPy.
> 
> Yes, I measured 35% speedup on richards/pystone on "the other branch" 
(that 
> solely knows this kind of dictionary unlike the current trunk).
> 
> > One question, why would you need any calls to str2object in any of the 

> > non-mutating methods?
> 
> Because every class can define it's own hash/eq functions that might be 
> considered insane in case of having them return hashes that match the 
> hashes of a string. That would mean that {"foo": 1}[myClass()] could 
return 
> 1. Therefore we have to switch the mode while having a shortcut for 
types 
> which are known to have sane hashes only.

Ouch! Yes, I see the problem now. Roll on parameretised collections :)

> 
> >                 w_lookup.__hash__()
> 
> I didn't even consider that people would write __hash__ methods with 
> side-effects :)
> 
> Kind regards,
> Alexander, who has resent this posting 2 days after the initial try - 
Gmane
> problems?
>

Not me!
 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From arigo at tunes.org  Mon Jul 17 20:02:57 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 17 Jul 2006 20:02:57 +0200
Subject: [pypy-dev] Security ideas
In-Reply-To: <20060713060229.GR32712@solar.trillke>
References: <20060524114728.GA6374@code0.codespeak.net>
	<20060713060229.GR32712@solar.trillke>
Message-ID: <20060717180257.GA18722@code0.codespeak.net>

Hi Holger,

On Thu, Jul 13, 2006 at 08:02:29AM +0200, holger krekel wrote:
>     def enter_bid(n):
>         if n > highest_bid.value:
>             highest_bid.value = n 
>     enter_bid = secure(enter_bid)
> 
> Here the annotator analysis is supposed to prevent a leak of information
> from the secret value.  But if the if-branch additionally contains:
> 
>             num_bids += 1
> 
> don't you run into a branching/code-dependent-on-secret-condition 
> problem again?  Would the annotator prevent the manipulation of 
> the global 'num_bids'?  Would it need to be a public value? 

You can't modify global values in RPython anyway.  But more generally,
yes, the annotator would follow all mutations and propagate security
levels.

> Moreover, i have practical concerns: your proposed
> scheme requires RPython annotator analysis which implies to
> have the PyPy tool chain available and accessible at programming 
> time.  Not impossible but also not a use case that we went for 
> so far.

Well, we haven't considered much at all.  This is just one approach,
which we didn't think of earlier, but there are of course others.

> Also it is not clear to which target "secure" would compile 
> functions to, C or bytecode or ...?

I don't know.  That's not too important for the security aspect;
anything would work, maybe even to the point of not actually compiling
at all but just using the annotator for the security proof (although I'd
regard this as less secure somehow, given the subtle differences between
RPython and full Python).


A bientot,

Armin

From hpk at trillke.net  Tue Jul 18 08:34:21 2006
From: hpk at trillke.net (holger krekel)
Date: Tue, 18 Jul 2006 08:34:21 +0200
Subject: [pypy-dev] Security ideas
In-Reply-To: <20060717180257.GA18722@code0.codespeak.net>
References: <20060524114728.GA6374@code0.codespeak.net>
	<20060713060229.GR32712@solar.trillke>
	<20060717180257.GA18722@code0.codespeak.net>
Message-ID: <20060718063421.GS28134@solar.trillke.net>

Hi Armin, 

On Mon, Jul 17, 2006 at 20:02 +0200, Armin Rigo wrote:
> On Thu, Jul 13, 2006 at 08:02:29AM +0200, holger krekel wrote:
> >     def enter_bid(n):
> >         if n > highest_bid.value:
> >             highest_bid.value = n 
> >     enter_bid = secure(enter_bid)
> > 
> > Here the annotator analysis is supposed to prevent a leak of information
> > from the secret value.  But if the if-branch additionally contains:
> > 
> >             num_bids += 1
> > 
> > don't you run into a branching/code-dependent-on-secret-condition 
> > problem again?  Would the annotator prevent the manipulation of 
> > the global 'num_bids'?  Would it need to be a public value? 
> 
> You can't modify global values in RPython anyway.  But more generally,
> yes, the annotator would follow all mutations and propagate security
> levels.

Hum, sorry for having asked too many questions at once.  Obviously, 
i could have said "whatever.num_bids +=1". 
Anyway, it seems that we cannot avoid the "dependent code block" issue 
but you consider the annotator more suited than an object space 
to deal with it, right? 

Also, from your answers i gather that your focus is more on 
getting something at-or-beyond-state-of-the-art first rather
than to provide something directly practical.  Probably makes
sense. 

best, 

    holger

From elmo13 at jippii.fi  Wed Jul 19 15:11:58 2006
From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=)
Date: Wed, 19 Jul 2006 16:11:58 +0300
Subject: [pypy-dev] translation with --profopt fails
Message-ID: <44BE2F9E.4060306@jippii.fi>

Last lines of output (python translate.py --backend=c --profopt='-c
"from richards import *;main(iterations=1)"' --text --batch
targetpypystandalone.py):

"""

[flowgraph] (pypy.rpython.memory.gctransform:733)ll_finalizer
[flowgraph] (pypy.rpython.memory.gctransform:733)ll_finalizer
[flowgraph] (pypy.rpython.memory.gctransform:733)ll_finalizer
XXX: operation v805 = direct_call((<* fn eq__Long_Long>), (StdObjSpace),
arg0_626, v804) cannot raise, but has exception guarding in graph
(?:1)_mm_eq_longS0_W_LongObject_W_BoolObject
XXX: operation v807 = direct_call((<* fn eq__Long_Long>), (StdObjSpace),
arg0_629, v806) cannot raise, but has exception guarding in graph
(?:1)_mm_eq_longS0_W_LongObject_W_IntObject
XXX: operation v809 = direct_call((<* fn eq__Long_Long>), (StdObjSpace),
arg0_631, v808) cannot raise, but has exception guarding in graph
(?:1)_mm_eq_longS0_W_LongObject_W_LongObject
[c]    28000 nodes  [ array: 3816  boehm rtti: 120  func: 2075  struct:
29285 ]
[c]    29000 nodes  [ array: 4045  boehm rtti: 120  func: 2149  struct:
29980 ]
[c]    30000 nodes  [ array: 4129  boehm rtti: 120  func: 2186  struct:
31192 ]
[c]    31000 nodes  [ array: 4257  boehm rtti: 120  func: 2201  struct:
32700 ]
[c]    32000 nodes  [ array: 4419  boehm rtti: 137  func: 2279  struct:
33433 ]
[translation:ERROR] Error:
[translation:ERROR]  Traceback (most recent call last):
[translation:ERROR]    File "translate.py", line 335, in main
[translation:ERROR]     drv.proceed(goals)
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 509, in proceed
[translation:ERROR]     return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/taskengine.py",
line 108, in _execute
[translation:ERROR]     res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 197, in _do
[translation:ERROR]     res = func()
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 301, in task_database_c
[translation:ERROR]     database = cbuilder.build_database()
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/genc.py",
line 86, in build_database
[translation:ERROR]     db.complete()
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py",
line 302, in complete
[translation:ERROR]     add_dependencies(node.enum_dependencies())
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py",
line 288, in add_dependencies
[translation:ERROR]     self.get(value)
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py",
line 179, in get
[translation:ERROR]     if obj:   # test if the ptr is non-NULL
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/rpython/lltypesystem/lltype.py",
line 886, in __nonzero__
[translation:ERROR]     return self._obj is not None
[translation:ERROR]    File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/rpython/lltypesystem/lltype.py",
line 910, in _getobj
[translation:ERROR]     raise RuntimeError("accessing already garbage
collected %r"
[translation:ERROR]  RuntimeError: accessing already garbage collected
<Array of Char >
"""

Just in case no one noticed this already...

Elmo

From leo.soto at gmail.com  Wed Jul 19 19:23:07 2006
From: leo.soto at gmail.com (Leonardo Soto)
Date: Wed, 19 Jul 2006 13:23:07 -0400
Subject: [pypy-dev] Implenting -m command line option
Message-ID: <5a8cc9e00607191023i75f4e6a6r79eda06a610b8944@mail.gmail.com>

Hi all,

I'm new to pypy, and to get myself started I'm trying to fix #145,
that is, implement the -m flag.

So I did search on the web and python sources looking for what exactly
-m means, and PEP-338[1] has the better explanation:

"If the module is found, and is of type PY_SOURCE or PY_COMPILED, then
the command line is effectively reinterpreted from python <options> -m
<module> <args> to python <options> <filename> <args>. This includes
setting sys.argv[0] correctly (some scripts rely on this - Python's
own regrtest.py is one example)."

But, by the way, PEP-338 is about generalizing -m to fully support
import hooks and executing modules inside packages, including a
reference implementation almost entirely written as a python module
(runpy). It has been included recently on CPython 2.5.

So, here is my question:

What "-m behaviour" should be implemented on pypy?

-- 
Leonardo Soto M.

[1] http://www.python.org/dev/peps/pep-0338/

From mwh at python.net  Wed Jul 19 21:59:43 2006
From: mwh at python.net (Michael Hudson)
Date: Wed, 19 Jul 2006 20:59:43 +0100
Subject: [pypy-dev] Implenting -m command line option
References: <5a8cc9e00607191023i75f4e6a6r79eda06a610b8944@mail.gmail.com>
Message-ID: <2md5c11i33.fsf@starship.python.net>

"Leonardo Soto" <leo.soto at gmail.com> writes:

> Hi all,
>
> I'm new to pypy, and to get myself started I'm trying to fix #145,
> that is, implement the -m flag.
>
> So I did search on the web and python sources looking for what exactly
> -m means, and PEP-338[1] has the better explanation:
>
> "If the module is found, and is of type PY_SOURCE or PY_COMPILED, then
> the command line is effectively reinterpreted from python <options> -m
> <module> <args> to python <options> <filename> <args>. This includes
> setting sys.argv[0] correctly (some scripts rely on this - Python's
> own regrtest.py is one example)."
>
> But, by the way, PEP-338 is about generalizing -m to fully support
> import hooks and executing modules inside packages, including a
> reference implementation almost entirely written as a python module
> (runpy). It has been included recently on CPython 2.5.
>
> So, here is my question:
>
> What "-m behaviour" should be implemented on pypy?

To be honest, either would be great.  The runpy-based solution sounds
easier, so maybe that?

Cheers,
mwh

-- 
41. Some programming languages manage to absorb change, but
    withstand progress.
  -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html


From anto.cuni at gmail.com  Thu Jul 20 00:53:50 2006
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Thu, 20 Jul 2006 00:53:50 +0200
Subject: [pypy-dev] Tomorrow morning...
Message-ID: <44BEB7FE.3060903@gmail.com>

... I will graduate!! :-) :-) :-)

A big, big 'thank you' to all pypy developers, for having welcomed and 
supported me during last months: it has been very nice to work with you 
(and I hope to continue in future).

I still remember my first post on pypy-dev: I was "Looking for a thesis" 
(cit.) and I was rather confused about pypy architecture: it was only 5 
months ago, but it seems years. Then I started coding and I've never 
stopped :-).

For those interested in, the final version of my thesis is here:
http://codespeak.net/svn/user/antocuni/tesi/Implementing%20Python%20in%20.NET.pdf

ciao Anto

From hpk at trillke.net  Thu Jul 20 07:05:45 2006
From: hpk at trillke.net (holger krekel)
Date: Thu, 20 Jul 2006 07:05:45 +0200
Subject: [pypy-dev] Tomorrow morning...
In-Reply-To: <44BEB7FE.3060903@gmail.com>
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <20060720050545.GC27182@solar.trillke.net>

Hi Antonio!

On Thu, Jul 20, 2006 at 00:53 +0200, Antonio Cuni wrote:
> ... I will graduate!! :-) :-) :-)

Woa! Big congrats! :))
 
> A big, big 'thank you' to all pypy developers, for having welcomed and 
> supported me during last months: it has been very nice to work with you 
> (and I hope to continue in future).

same here with you! 

> I still remember my first post on pypy-dev: I was "Looking for a thesis" 
> (cit.) and I was rather confused about pypy architecture: it was only 5 
> months ago, but it seems years. Then I started coding and I've never 
> stopped :-).
> 
> For those interested in, the final version of my thesis is here:
> http://codespeak.net/svn/user/antocuni/tesi/Implementing%20Python%20in%20.NET.pdf

just printed it and will see to read it sometime in the sun :) 

best & see you, 

    holger

From l.oluyede at gmail.com  Thu Jul 20 09:25:16 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Thu, 20 Jul 2006 09:25:16 +0200
Subject: [pypy-dev] Tomorrow morning...
In-Reply-To: <44BEB7FE.3060903@gmail.com>
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <9eebf5740607200025x29d5675flde43353894113e95@mail.gmail.com>

On 7/20/06, Antonio Cuni <anto.cuni at gmail.com> wrote:
> ... I will graduate!! :-) :-) :-)

Congratulations Antonio :-)

-- 
Lawrence
http://www.oluyede.org/blog

From Ben.Young at risk.sungard.com  Thu Jul 20 09:29:07 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Thu, 20 Jul 2006 08:29:07 +0100
Subject: [pypy-dev] .NET build failures
Message-ID: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>

Hi Antocuni,

I ran the .NET build last night and it got a lot further. The errors I get 
now are:

[translation:ERROR] ***** FAILURE *****
[translation:ERROR]
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(66076) : error -- 
Duplicate label: '__check_block_5'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(372039) : error -- 
Duplicate label: '__check_block_5'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821924) : error -- 
Duplicate label: '__check_block_7'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821985) : error -- 
Duplicate label: '__check_block_3'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822900) : error -- 
Duplicate label: '__check_block_7'
[translation:ERROR] 
c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822961) : error -- 
Duplicate label: '__check_block_3'

I also notice that you appear to have identifiers with hundreds of 
underscores appended. Is this expected?

[translation:ERROR] Assembled global method 
memo_get_unique_interplevel_subclass_4______________________________________________________________________________
________________________________________________________________________________________________________________________________________________________________
________________________________________________________________________________________________________________________________________________________________
____________________________________

Cheers,
Ben

From mwh at python.net  Thu Jul 20 10:05:14 2006
From: mwh at python.net (Michael Hudson)
Date: Thu, 20 Jul 2006 09:05:14 +0100
Subject: [pypy-dev] Tomorrow morning...
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <2m8xmo1z2d.fsf@starship.python.net>

Antonio Cuni <anto.cuni at gmail.com> writes:

> ... I will graduate!! :-) :-) :-)
>
> A big, big 'thank you' to all pypy developers, for having welcomed and 
> supported me during last months: it has been very nice to work with you 
> (and I hope to continue in future).

No, thank you for all you've done!

Cheers,
mwh

-- 
  C++ is a siren song.  It *looks* like a HLL in which you ought to
  be able to write an application, but it really isn't.
                                       -- Alain Picard, comp.lang.lisp


From mwh at python.net  Thu Jul 20 10:16:21 2006
From: mwh at python.net (Michael Hudson)
Date: Thu, 20 Jul 2006 09:16:21 +0100
Subject: [pypy-dev] return of the return of pypy-sync
Message-ID: <2m4pxc1yju.fsf@starship.python.net>

Once again, I'd like to invite all active PyPy developers to a short
(no more than half an hour) sychronisation meeting in #pypy-sync on
freenode at 17:30 UTC+2 today.

Agenda:

 - status

   pre-prepared 3-line reports of what you did last week, what you
   plan to do next week and what is blocking your progress at this
   time.

   this is the area i see as the most important this week: work has
   begun to spread in many directions, and it's a good idea to have an
   idea who is doing what and where they've got to.

 - work out how to resolve any blockers above

 - summer of pypy

   the summer of pypy program is ready to be announced:

   http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy.html

   there's not too much to discuss, but the "project ideas" page:

   http://codespeak.net/pypy/dist/pypy/doc/independent-project-ideas.html

   could use an update, we can talk about ideas for that.

 - ireland sprint

   again, mostly an information point: our next scheduled sprint in
   Ireland towards the end of August will be announced soon:

   http://codespeak.net/pypy/extradoc/sprintinfo/ireland-2006/announce.html

   so maybe you can start thinking about travel arrangements already.
   it is looking likely that the sprint part will not be attended by a
   large number of locals, so it's likely to be a fairly "core"
   sprint.

See you there!

Cheers,
mwh

-- 
  The "of course, while I have no problem with this at all, it's
  surely too much for a lesser being" flavor of argument always
  rings hollow to me.                       -- Tim Peters, 29 Apr 1998


From Ben.Young at risk.sungard.com  Thu Jul 20 15:08:02 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Thu, 20 Jul 2006 14:08:02 +0100
Subject: [pypy-dev] .NET build failures
In-Reply-To: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>
Message-ID: <OFD704530D.0D116AD9-ON802571B1.0047CBEC-802571B1.00482551@risk.sungard.com>

To reply to myself :)

The problem appears to be in the _check function in opcodes.py. This 
generates a number for the jump at opcode creation time. Therefore, if an 
opcode is used twice in a block then you get the error.

I guess the solution would be to create a "_check" micro instruction that 
increments the count on render. I'm afraid I haven't got time to actually 
do that myself though.

Cheers,
Ben


pypy-dev-bounces at codespeak.net wrote on 20/07/2006 08:29:07:

> Hi Antocuni,
> 
> I ran the .NET build last night and it got a lot further. The errors I 
get 
> now are:
> 
> [translation:ERROR] ***** FAILURE *****
> [translation:ERROR]
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(66076) : error 
-- 
> Duplicate label: '__check_block_5'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(372039) : error 
-- 
> Duplicate label: '__check_block_5'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821924) : error 
-- 
> Duplicate label: '__check_block_7'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(821985) : error 
-- 
> Duplicate label: '__check_block_3'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822900) : error 
-- 
> Duplicate label: '__check_block_7'
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(822961) : error 
-- 
> Duplicate label: '__check_block_3'
> 
> I also notice that you appear to have identifiers with hundreds of 
> underscores appended. Is this expected?
> 
> [translation:ERROR] Assembled global method 
> 
memo_get_unique_interplevel_subclass_4______________________________________________________________________________
> 
________________________________________________________________________________________________________________________________________________________________
> 
________________________________________________________________________________________________________________________________________________________________
> ____________________________________
> 
> Cheers,
> Ben
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 


From arigo at tunes.org  Thu Jul 20 17:59:12 2006
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 20 Jul 2006 17:59:12 +0200
Subject: [pypy-dev] Tomorrow morning...
In-Reply-To: <44BEB7FE.3060903@gmail.com>
References: <44BEB7FE.3060903@gmail.com>
Message-ID: <20060720155912.GA30518@code0.codespeak.net>

Hi Antonio,

On Thu, Jul 20, 2006 at 12:53:50AM +0200, Antonio Cuni wrote:
> ... I will graduate!! :-) :-) :-)

Congratulation!

> A big, big 'thank you' to all pypy developers, for having welcomed and 
> supported me during last months: it has been very nice to work with you 
> (and I hope to continue in future).

Thanks to you, you are doing quite a great job :-)

Armin

From 2006a at usenet.alexanderweb.de  Fri Jul 21 02:23:10 2006
From: 2006a at usenet.alexanderweb.de (Alexander Schremmer)
Date: Fri, 21 Jul 2006 02:23:10 +0200
Subject: [pypy-dev] [pypy-svn] r30034 - pypy/dist/pypy/objspace/std
References: <20060714105155.DC18A10090@code0.codespeak.net>
	<OFED1A6F46.5AACC121-ON802571AB.003D1B50-802571AB.003D9CD1@risk.sungard.com>
	<nbhefnx5b0oe.dlg@usenet.alexanderweb.de>
Message-ID: <no0jbpy9l9pv$.dlg@usenet.alexanderweb.de>

On Sun, 16 Jul 2006 23:40:01 +0200, I wrote:

>> This looks really good! It shows there are still some major (easy) 
>> performance wins in PyPy.
> 
> Yes, I measured 35% speedup on richards/pystone on "the other branch" (that 
> solely knows this kind of dictionary unlike the current trunk).

OK, newest benchmarks show a stable 32% speedup at richards.py, even
working on the trunk - you just need to enable the strdict flag like this
in the target:

    config.objspace.std.withstrdict = True

This gives the following factors for pypy-c vs. cpython (measured on
pypy2):
Richards: 2.98x as slow
PyStone: 3.5x as slow

Now I am looking forward to have someone building an LLVM/C/Profile guided
opt binary with strdicts enabled :-)

Kind regards,
Alexander


From hpk at trillke.net  Fri Jul 21 13:20:39 2006
From: hpk at trillke.net (holger krekel)
Date: Fri, 21 Jul 2006 13:20:39 +0200
Subject: [pypy-dev] Summer of PyPy: call for proposals
Message-ID: <20060721112039.GC27182@solar.trillke.net>

Hi to all students! 

Happily, we are able to offer students mentoring and full sprint
participant's funding if we receive a proposal outlining an
interesting project related to PyPy and its development tools.  This
follows up on the "Summer of Code" campaign from Google but is
completely independent from it and also works differently (i.e. there
is no success bounty).

See the full call for details: 

    http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy.html

and here is a list of concrete inspirations for possible work: 

    http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html

There are certainly many more possibilities now that PyPy starts to
get faster, Stackless and custom GCs are integrated, PyPy.NET, Logic
Variables, lazy evaluation and other new features are emerging.  Note
though that PyPy still is pursuing challenging research goals so it
will take some time before it becomes viable in production
environments.

Don't hesitate to drop by on #pypy on freenode or post questions/ideas
to pypy-dev at codespeak net.

A last note: you can propose something at any time, you don't need to
wait for a specific deadline. In fact, our next sprint is nearing in
Ireland, 21st-27th August and if you are quick you may already be
there with us :)

For the PyPy team, 

    Armin Rigo, Holger Krekel, Michael Hudson, Carl Friedrich Bolz, ... 


From bea at changemaker.nu  Fri Jul 21 13:40:42 2006
From: bea at changemaker.nu (Beatrice During)
Date: Fri, 21 Jul 2006 13:40:42 +0200
Subject: [pypy-dev] PyPy sprint Limerick 21-27th of August
Message-ID: <44C0BD3A.3060300@changemaker.nu>

Hi there

The PyPy team goes sprinting in Limerick/Ireland  - are
you interested in participating?


Ireland PyPy sprint 21th-27th August 2006
====================================================

The next PyPy sprint will happen in the nice city of
Limerick in Ireland from 21st till 27th August.
(Most people intend to arrive 20th August).

The main focus of the sprint will be on JIT compiler works,
various optimization works, porting extension modules,
infrastructure works like a build tool for PyPy, or
extended (distributed) testing.

It's also open to new topics.  If you are a student
consider to participate in `Summer of PyPy` in order
get funding for your travels and accomodation.

The sprint is being hosted by University of Limerick
(http://www.ul.ie/) - and is arranged in co-operation
with people from our sister project Calibre (www.calibre.ie).
Our contact at the University is P?r ?gerfalk and Eoin
Oconchuir.

For more information on logistics, facilities etc - please
read the full announcement at:

http://codespeak.net/pypy/extradoc/sprintinfo/ireland-2006/announce.html

For the PyPy team

Bea & Holger


From arigo at tunes.org  Fri Jul 21 17:29:39 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Jul 2006 17:29:39 +0200
Subject: [pypy-dev] translation with --profopt fails
In-Reply-To: <44BE2F9E.4060306@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>
Message-ID: <20060721152939.GB31604@code0.codespeak.net>

Hi Elmo,

On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
> Last lines of output (python translate.py --backend=c --profopt='-c
> "from richards import *;main(iterations=1)"' --text --batch
> targetpypystandalone.py):

I have no clue how --profopt can affect the translation itself.  You
don't get the same error without the --profopt option?


A bientot,

Armin

From elmo13 at jippii.fi  Fri Jul 21 18:07:38 2006
From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=)
Date: Fri, 21 Jul 2006 19:07:38 +0300
Subject: [pypy-dev] translation with --profopt fails
In-Reply-To: <20060721152939.GB31604@code0.codespeak.net>
References: <44BE2F9E.4060306@jippii.fi>
	<20060721152939.GB31604@code0.codespeak.net>
Message-ID: <44C0FBCA.1090605@jippii.fi>

Armin Rigo wrote:
> Hi Elmo,
>
> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>   
>> Last lines of output (python translate.py --backend=c --profopt='-c
>> "from richards import *;main(iterations=1)"' --text --batch
>> targetpypystandalone.py):
>>     
>
> I have no clue how --profopt can affect the translation itself.  You
> don't get the same error without the --profopt option?
>
>
> A bientot,
>
> Armin
>   
Should've tested that, will do as soon as possible. benchmark.html on
snake shows something that says to me that the profopt executable
doesn't exist, which suggests that it wasn't built.

Elmo


From anto.cuni at gmail.com  Fri Jul 21 21:07:26 2006
From: anto.cuni at gmail.com (Antonio Cuni)
Date: Fri, 21 Jul 2006 21:07:26 +0200
Subject: [pypy-dev] .NET build failures
In-Reply-To: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>
References: <OFFD8947AE.9DE95F27-ON802571B1.00289D40-802571B1.00291DAE@risk.sungard.com>
Message-ID: <44C125EE.1000900@gmail.com>

Ben.Young at risk.sungard.com wrote:
> Hi Antocuni,

Hi Ben,

> I ran the .NET build last night and it got a lot further. The errors I get 
> now are:
> 
> [translation:ERROR] ***** FAILURE *****
> [translation:ERROR]
> [translation:ERROR] 
> c:\docume~1\ben~1.you\locals~1\temp\usession-16\main.il(66076) : error -- 
> Duplicate label: '__check_block_5'

as you correctly spotted out, the problem was in opcodes._check; I've 
fixed that, thanks for the help.

> I also notice that you appear to have identifiers with hundreds of 
> underscores appended. Is this expected?
> 
> [translation:ERROR] Assembled global method 
> memo_get_unique_interplevel_subclass_4______________________________________________________________________________
> ________________________________________________________________________________________________________________________________________________________________
> ________________________________________________________________________________________________________________________________________________________________
> ____________________________________

Yes, it's expected. Appending underscores it's a way to get unique names 
for methods and classes, though I didn't think to get such a big number 
of underscores. I changed the code and now I append a unique number at 
the end of the name instead of a list of underscores.

ciao Anto


From elmo13 at jippii.fi  Fri Jul 21 22:26:53 2006
From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=)
Date: Fri, 21 Jul 2006 23:26:53 +0300
Subject: [pypy-dev] translation with --profopt fails
In-Reply-To: <44C0FBCA.1090605@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>	<20060721152939.GB31604@code0.codespeak.net>
	<44C0FBCA.1090605@jippii.fi>
Message-ID: <44C1388D.80003@jippii.fi>

Elmo M?ntynen wrote:
> Armin Rigo wrote:
>   
>> Hi Elmo,
>>
>> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>>   
>>     
>>> Last lines of output (python translate.py --backend=c --profopt='-c
>>> "from richards import *;main(iterations=1)"' --text --batch
>>> targetpypystandalone.py):
>>>     
>>>       
>> I have no clue how --profopt can affect the translation itself.  You
>> don't get the same error without the --profopt option?
>>
>>
>> A bientot,
>>
>> Armin
>>   
>>     
> Should've tested that, will do as soon as possible. benchmark.html on
> snake shows something that says to me that the profopt executable
> doesn't exist, which suggests that it wasn't built.
>
> Elmo
>   
Translation without profopt worked now, though there might have been
something that got changed in between, will report after some more builds.

Elmo

From arigo at tunes.org  Fri Jul 21 22:36:13 2006
From: arigo at tunes.org (Armin Rigo)
Date: Fri, 21 Jul 2006 22:36:13 +0200
Subject: [pypy-dev] Failing stackless tests
Message-ID: <20060721203613.GA2825@code0.codespeak.net>

Hi all, hi Aurelien,

The stackless-related tests are all consistently failing nowadays.  Some
have been failing for 4 days and others for 9 days.  See

   http://snake.cs.uni-duesseldorf.de/pypytest/summary.html

Given that you appear to continue to work on it, but without making
tests pass again, I'm a bit worrying exactly about what you are doing?
That doesn't seem too much in line with the "test-driven development"
approach.


A bientot,

Armin

From scott+pypy-dev at scottdial.com  Sat Jul 22 02:31:57 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Fri, 21 Jul 2006 20:31:57 -0400
Subject: [pypy-dev] Newb questions about ext module writing
Message-ID: <44C171FD.60801@scottdial.com>

Greetings everyone,

I'm relatively new to PyPy. I've been aware of it for a while, but never 
really got into poking around at all. But recently I decided this is a 
project that I would like to get involved in. I think at the time I 
planned on doing something big, like addding a JVM backend or something 
like that.. maybe I'll get back to that eventually. I realized I was 
fairly clueless with the codebase and decided to jump on writing a 
module as a type of introductory project.

The independent-project-ideas went out of its way to mention readline, 
so boom, readline is what I had/have chosen to finish. Things were going 
well until I started having to deal with making callback functions.

I have become stuck on the need to setup_readline with some callback 
functions which are capable of accessing the State of the module. I'm 
pretty lost on how to go about getting the state. Within the 
interp_readline, I do the "space.fromcache(State)" deal to get ahold of 
the State, but how do I do this from within the "c_readline" component. 
Perhaps this is completely the wrong way to go about this? My first 
thought is to import the "getstate" function from interp_readline, but I 
can't find anywhere else this is done in the codebase off-hand.

Furthermore, my callback functions from readline will not be passed any 
sort of "state", so how do I get that from within my callback function? 
Again, I thought a poor kludge was to tag reference to "state" onto the 
function during setup_readline.

def setup_readline(state, w_module):
     ...
     cfunc = CFUNCTYPE(POINTER(c_char_p), c_char_p, c_int, c_int)
     c_rl_attempted_completion_function = cfunc(space.flex_complete)
     space.flex_complete.space = space
     ...

def flex_complete(text, start, end):
     from pypy.module.readline.interp_readline import getstate
     getstate(flex_complete).begidx = start
     getstate(flex_complete).endidx = end
     ...

Some ugly stuff like that.. help? :) Thanks.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu

From len-l at telus.net  Sat Jul 22 03:24:19 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Fri, 21 Jul 2006 18:24:19 -0700
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C171FD.60801@scottdial.com>
Message-ID: <44C11BD3.25178.4E9F16@len-l.telus.net>

On 21 Jul 2006 at 20:31, Scott Dial wrote:

> Furthermore, my callback functions from readline will not be passed any 
> sort of "state", so how do I get that from within my callback function? 
> Again, I thought a poor kludge was to tag reference to "state" onto the 
> function during setup_readline.
> 
> def setup_readline(state, w_module):
>      ...
>      cfunc = CFUNCTYPE(POINTER(c_char_p), c_char_p, c_int, c_int)
>      c_rl_attempted_completion_function = cfunc(space.flex_complete)
>      space.flex_complete.space = space
>      ...
> 
> def flex_complete(text, start, end):
>      from pypy.module.readline.interp_readline import getstate
>      getstate(flex_complete).begidx = start
>      getstate(flex_complete).endidx = end
>      ...
> 
In ctypes that is easy: use a bound method as the callback. Does that 
even work in rctypes?

Lenard Lindstrom
<len-l at telus.net>



From arigo at tunes.org  Sat Jul 22 12:24:42 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sat, 22 Jul 2006 12:24:42 +0200
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C11BD3.25178.4E9F16@len-l.telus.net>
References: <44C171FD.60801@scottdial.com>
	<44C11BD3.25178.4E9F16@len-l.telus.net>
Message-ID: <20060722102442.GA8055@code0.codespeak.net>

Hi,

On Fri, Jul 21, 2006 at 06:24:19PM -0700, Lenard Lindstrom wrote:
> In ctypes that is easy: use a bound method as the callback. Does that 
> even work in rctypes?

I'm afraid not, and it's not obvious how this would be compiled to C
code, given that there is no kind of closure in ANSI C.

I don't see how to do that at all, actually :-(  It's somehow a design
problem of the readline lib, which should in theory always pass an extra
void* "closure" arguments to callbacks...

It might be necessary to use hacks like a single global State instance,
storing the "space" the readline lib was initialized with, instead of
one State per space.


Armin

From scott+pypy-dev at scottdial.com  Sat Jul 22 13:39:27 2006
From: scott+pypy-dev at scottdial.com (Scott Dial)
Date: Sat, 22 Jul 2006 07:39:27 -0400
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <20060722102442.GA8055@code0.codespeak.net>
References: <44C171FD.60801@scottdial.com>	<44C11BD3.25178.4E9F16@len-l.telus.net>
	<20060722102442.GA8055@code0.codespeak.net>
Message-ID: <44C20E6F.2080608@scottdial.com>

Armin Rigo wrote:
> On Fri, Jul 21, 2006 at 06:24:19PM -0700, Lenard Lindstrom wrote:
>> In ctypes that is easy: use a bound method as the callback. Does that 
>> even work in rctypes?
> 
> I'm afraid not, and it's not obvious how this would be compiled to C
> code, given that there is no kind of closure in ANSI C.
> 
> I don't see how to do that at all, actually :-(  It's somehow a design
> problem of the readline lib, which should in theory always pass an extra
> void* "closure" arguments to callbacks...
> 
> It might be necessary to use hacks like a single global State instance,
> storing the "space" the readline lib was initialized with, instead of
> one State per space.

I am admittedly ignorant of rctypes capabilities, but I can appreciate
the conversion in C code being awkward. Reading around more, I see in
rctypes.implementation where an attempt for CALLBACK_FUNCTYPE was
droppped. Seems like I have bitten off more than I can chew here.

I think you understate the issue by saying it's "a design problem of the
readline lib" because many, many C libraries are going to need callback
functionality. If I have evaluated the project's current state
correctly, then I realize this is more-or-less not the current focus of
development. So, no biggie.

I just figured it would something I could just go through the motions on
to become familiar. I am open to other suggestions. I can comeback to
this when I have a better understanding of everything.

-- 
Scott Dial
scott at scottdial.com
scodial at indiana.edu


From mahs at telcopartners.com  Sat Jul 22 23:17:46 2006
From: mahs at telcopartners.com (Michael Spencer)
Date: Sat, 22 Jul 2006 14:17:46 -0700
Subject: [pypy-dev] 20th Level IP Communicator?
Message-ID: <e9u4la$54j$1@sea.gmane.org>

http://gigaom.com/2006/07/15/sipphone-brings-voice-to-nokia-770-tablet/#comment-125818

I thought you'd like this comment:

"looks like i?m about to fail my saving throw against ?cool gadget temptation?."


From mahs at telcopartners.com  Sat Jul 22 23:21:52 2006
From: mahs at telcopartners.com (Michael Spencer)
Date: Sat, 22 Jul 2006 14:21:52 -0700
Subject: [pypy-dev] 20th Level IP Communicator?
In-Reply-To: <e9u4la$54j$1@sea.gmane.org>
References: <e9u4la$54j$1@sea.gmane.org>
Message-ID: <e9u4sv$54j$2@sea.gmane.org>

Michael Spencer wrote:
> http://gigaom.com/2006/07/15/sipphone-brings-voice-to-nokia-770-tablet/#comment-125818
> 
> I thought you'd like this comment:
> 
> "looks like i?m about to fail my saving throw against ?cool gadget temptation?."
> 
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
> 
Sorry - sent to the wrong address
Michael


From arigo at tunes.org  Sun Jul 23 09:18:38 2006
From: arigo at tunes.org (Armin Rigo)
Date: Sun, 23 Jul 2006 09:18:38 +0200
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C20E6F.2080608@scottdial.com>
References: <44C171FD.60801@scottdial.com>
	<44C11BD3.25178.4E9F16@len-l.telus.net>
	<20060722102442.GA8055@code0.codespeak.net>
	<44C20E6F.2080608@scottdial.com>
Message-ID: <20060723071838.GA29559@code0.codespeak.net>

Hi Scott,

On Sat, Jul 22, 2006 at 07:39:27AM -0400, Scott Dial wrote:
> I am admittedly ignorant of rctypes capabilities, but I can appreciate
> the conversion in C code being awkward. Reading around more, I see in
> rctypes.implementation where an attempt for CALLBACK_FUNCTYPE was
> droppped. Seems like I have bitten off more than I can chew here.

Callbacks by themselves should "almost" work.  We need to re-enable the
commented-out code and try -- this code is lot complexity, some of it
bordering on the hackish side, which turned out not to be needed any
more at some point.  So to ease development of the rest of rctypes I
disabled the tests and commented it out.  But of course callbacks are
clearly needed in the long run.

Was I was complaining about is that there is no C-convertable way to
have e.g. bound methods as callbacks.  You can only use plain functions.
Many C libs have the same "design problem" as readline and don't pass
you a "void*" argument back.  An example of lib that is "correctly
designed" in this respect is the Windows API.

So for now I suppose these cases can be handled by just assuming that
there is only one 'space' around, and storing it in global state.


A bientot,

Armin.

From len-l at telus.net  Sun Jul 23 19:56:06 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Sun, 23 Jul 2006 10:56:06 -0700
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <20060722102442.GA8055@code0.codespeak.net>
References: <44C11BD3.25178.4E9F16@len-l.telus.net>
Message-ID: <44C355C6.11938.7869AE@len-l.telus.net>

On 22 Jul 2006 at 12:24, Armin Rigo wrote:

> Hi,
> 
> On Fri, Jul 21, 2006 at 06:24:19PM -0700, Lenard Lindstrom wrote:
> > In ctypes that is easy: use a bound method as the callback. Does that 
> > even work in rctypes?
> 
> I'm afraid not, and it's not obvious how this would be compiled to C
> code, given that there is no kind of closure in ANSI C.
> 
I just brought it up as it is one more thing to consider when 
converting a ctypes Python module to rctypes. But could rctypes use 
ffi when the callback is a bound method?

Lenard Lindstrom
<len-l at telus.net>



From l.oluyede at gmail.com  Sun Jul 23 19:58:50 2006
From: l.oluyede at gmail.com (Lawrence Oluyede)
Date: Sun, 23 Jul 2006 19:58:50 +0200
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <44C355C6.11938.7869AE@len-l.telus.net>
References: <44C11BD3.25178.4E9F16@len-l.telus.net>
	<20060722102442.GA8055@code0.codespeak.net>
	<44C355C6.11938.7869AE@len-l.telus.net>
Message-ID: <9eebf5740607231058w2a2b688bpb5c746c01750ad96@mail.gmail.com>

> I just brought it up as it is one more thing to consider when
> converting a ctypes Python module to rctypes. But could rctypes use
> ffi when the callback is a bound method?

AFAIK one of the reasons of the existence of rctypes and the ext.
compiler is not to use the ffi library...

-- 
Lawrence
http://www.oluyede.org/blog

From elmo13 at jippii.fi  Sun Jul 23 20:43:05 2006
From: elmo13 at jippii.fi (=?windows-1252?Q?Elmo_M=E4ntynen?=)
Date: Sun, 23 Jul 2006 21:43:05 +0300
Subject: [pypy-dev] A bug in cc? (was: "translation with --profopt
	fails")
In-Reply-To: <44C1388D.80003@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>	<20060721152939.GB31604@code0.codespeak.net>	<44C0FBCA.1090605@jippii.fi>
	<44C1388D.80003@jippii.fi>
Message-ID: <44C3C339.8070809@jippii.fi>

Elmo M?ntynen wrote:
> Elmo M?ntynen wrote:
>   
>> Armin Rigo wrote:
>>   
>>     
>>> Hi Elmo,
>>>
>>> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>>>   
>>>     
>>>       
>>>> Last lines of output (python translate.py --backend=c --profopt='-c
>>>> "from richards import *;main(iterations=1)"' --text --batch
>>>> targetpypystandalone.py):
>>>>     
>>>>       
>>>>         
>>> I have no clue how --profopt can affect the translation itself.  You
>>> don't get the same error without the --profopt option?
>>>
>>>
>>> A bientot,
>>>
>>> Armin
>>>   
>>>     
>>>       
>> Should've tested that, will do as soon as possible. benchmark.html on
>> snake shows something that says to me that the profopt executable
>> doesn't exist, which suggests that it wasn't built.
>>
>> Elmo
>>   
>>     
> Translation without profopt worked now, though there might have been
> something that got changed in between, will report after some more builds.
>   
Now I got an internal compiler error from cc:
"""
[cbuild:profopt] Gathering profile data from:
/tmp/usession-11/testing_1/testing_1 -c "from richards import
*;main(iterations=1)"
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c testing_1.c -o
testing_1.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c structimpl.c -o
structimpl.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c nonfuncnodes.c
-o nonfuncnodes.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_1.c -o nonfuncnodes_1.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_2.c -o nonfuncnodes_2.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_3.c -o nonfuncnodes_3.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_4.c -o nonfuncnodes_4.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_5.c -o nonfuncnodes_5.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_6.c -o nonfuncnodes_6.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_7.c -o nonfuncnodes_7.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_8.c -o nonfuncnodes_8.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_9.c -o nonfuncnodes_9.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_10.c -o nonfuncnodes_10.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_11.c -o nonfuncnodes_11.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_12.c -o nonfuncnodes_12.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_13.c -o nonfuncnodes_13.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_14.c -o nonfuncnodes_14.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_15.c -o nonfuncnodes_15.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_16.c -o nonfuncnodes_16.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_17.c -o nonfuncnodes_17.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_18.c -o nonfuncnodes_18.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_19.c -o nonfuncnodes_19.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c
nonfuncnodes_20.c -o nonfuncnodes_20.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c implement.c -o
implement.o
[cbuild:execute] cc -O2 -pthread -fprofile-use
-I/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c
-I/usr/include/python2.4 -I/tmp/usession-11/testing_1 -c implement_1.c
-o implement_1.o
debug: entry point starting
debug: argv -> /tmp/usession-11/testing_1/testing_1
debug: argv -> -c "from richards import *;main(iterations=1)"
unrecognized option '-c "from richards import *;main(iterations=1)"'
usage: /tmp/usession-11/testing_1/testing_1 [options]
Try `/tmp/usession-11/testing_1/testing_1 -h` for more information.
implement_1.c: In function ?pypy_g_get_len_of_range?:
implement_1.c:137183: internal compiler error: Floating point exception
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:http://gcc.gnu.org/bugs.html> for instructions.
For Debian GNU/Linux specific bug reporting instructions,
see <URL:file:///usr/share/doc/gcc-4.0/README.Bugs>.

[translation:ERROR] Error:
[translation:ERROR] Traceback (most recent call last):
[translation:ERROR] File "translate.py", line 349, in main
[translation:ERROR] drv.proceed(goals)
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 508, in proceed
[translation:ERROR] return self._execute(goals, task_skip =
self._maybe_skip())
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/taskengine.py",
line 108, in _execute
[translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds)
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 197, in _do
[translation:ERROR] res = func()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py",
line 333, in task_compile_c
[translation:ERROR] cbuilder.compile()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/genc.py",
line 229, in compile
[translation:ERROR] compiler.build()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/cbuild.py",
line 323, in build
[translation:ERROR] self._build()
[translation:ERROR] File
"/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/cbuild.py",
line 353, in _build
[translation:ERROR] extra_preargs=self.compile_extra)
[translation:ERROR] File "/usr/lib/python2.4/distutils/ccompiler.py",
line 699, in compile
[translation:ERROR] self._compile(obj, src, ext, cc_args,
extra_postargs, pp_opts)
[translation:ERROR] File
"/usr/lib/python2.4/distutils/unixccompiler.py", line 115, in _compile
[translation:ERROR] raise CompileError, msg
[translation:ERROR] CompileError: command 'cc' failed with exit status 1
"""
If this really calls for a bugreport, it would be better if some one
more comfortable with the compiling process and the gcc toolset would do
it. Should I send some of the produced files?

Elmo

From elmo13 at jippii.fi  Sun Jul 23 22:15:47 2006
From: elmo13 at jippii.fi (=?windows-1252?Q?Elmo_M=E4ntynen?=)
Date: Sun, 23 Jul 2006 23:15:47 +0300
Subject: [pypy-dev] A bug in cc?
In-Reply-To: <44C3C339.8070809@jippii.fi>
References: <44BE2F9E.4060306@jippii.fi>	<20060721152939.GB31604@code0.codespeak.net>	<44C0FBCA.1090605@jippii.fi>	<44C1388D.80003@jippii.fi>
	<44C3C339.8070809@jippii.fi>
Message-ID: <44C3D8F3.4080603@jippii.fi>

Elmo M?ntynen wrote:
> Elmo M?ntynen wrote:
>   
>> Elmo M?ntynen wrote:
>>   
>>     
>>> Armin Rigo wrote:
>>>   
>>>     
>>>       
>>>> Hi Elmo,
>>>>
>>>> On Wed, Jul 19, 2006 at 04:11:58PM +0300, Elmo M??ntynen wrote:
>>>>   
>>>>     
>>>>       
>>>>         
>>>>> Last lines of output (python translate.py --backend=c --profopt='-c
>>>>> "from richards import *;main(iterations=1)"' --text --batch
>>>>> targetpypystandalone.py):
>>>>>     
>>>>>       
>>>>>         
>>>>>           
>>>> I have no clue how --profopt can affect the translation itself.  You
>>>> don't get the same error without the --profopt option?
>>>>
>>>>
>>>> A bientot,
>>>>
>>>> Armin
>>>>   
>>>>     
>>>>       
>>>>         
>>> Should've tested that, will do as soon as possible. benchmark.html on
>>> snake shows something that says to me that the profopt executable
>>> doesn't exist, which suggests that it wasn't built.
>>>
>>> Elmo
>>>   
>>>     
>>>       
>> Translation without profopt worked now, though there might have been
>> something that got changed in between, will report after some more builds.
>>   
>>     
> Now I got an internal compiler error from cc:
> """
> [cbuild:profopt] Gathering profile data from:
> /tmp/usession-11/testing_1/testing_1 -c "from richards import
> *;main(iterations=1)"
>
>   
...
> [translation:ERROR] File
> "/usr/lib/python2.4/distutils/unixccompiler.py", line 115, in _compile
> [translation:ERROR] raise CompileError, msg
> [translation:ERROR] CompileError: command 'cc' failed with exit status 1
> """
> If this really calls for a bugreport, it would be better if some one
> more comfortable with the compiling process and the gcc toolset would do
> it. Should I send some of the produced files?
>   
Just to note that building without profopt works (this time its the same
rev).

Elmo

From len-l at telus.net  Mon Jul 24 06:12:41 2006
From: len-l at telus.net (Lenard Lindstrom)
Date: Sun, 23 Jul 2006 21:12:41 -0700
Subject: [pypy-dev] Newb questions about ext module writing
In-Reply-To: <9eebf5740607231058w2a2b688bpb5c746c01750ad96@mail.gmail.com>
References: <44C355C6.11938.7869AE@len-l.telus.net>
Message-ID: <44C3E649.11747.B7036B@len-l.telus.net>

On 23 Jul 2006 at 19:58, Lawrence Oluyede wrote:

> > I just brought it up as it is one more thing to consider when
> > converting a ctypes Python module to rctypes. But could rctypes use
> > ffi when the callback is a bound method?
> 
> AFAIK one of the reasons of the existence of rctypes and the ext.
> compiler is not to use the ffi library...
> 
Then perhaps the ext. compiler can achieve the same effect as an ffi 
C closure by creating a stub function to call the bound method, which 
is stored in a thread-local variable.

Lenard Lindstrom
<len-l at telus.net>



From aurelien.campeas at logilab.fr  Mon Jul 24 12:21:52 2006
From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=)
Date: Mon, 24 Jul 2006 12:21:52 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060721203613.GA2825@code0.codespeak.net>
References: <20060721203613.GA2825@code0.codespeak.net>
Message-ID: <20060724102151.GC15904@crater.logilab.fr>

On Fri, Jul 21, 2006 at 10:36:13PM +0200, Armin Rigo wrote:
> Hi all, hi Aurelien,
> 
> The stackless-related tests are all consistently failing nowadays.  Some
> have been failing for 4 days and others for 9 days.  See
> 
>    http://snake.cs.uni-duesseldorf.de/pypytest/summary.html
> 
> Given that you appear to continue to work on it, but without making
> tests pass again, I'm a bit worrying exactly about what you are
> doing?

First, I did some (superficial, nothing related to the translation
aspect itself) changes that allowed me to build a (somewhat
non-crashing) translated pypy-logic using clonable coroutines. These
changes indeed broke some tests, in ways that I don't
understand. Please note that indeed most failing tests in
/modules/_stackless/test are due to run-time failures, after a
successful translation. Michael put in a fix last week which,
strangely, yielded a working version on ... PPC arch only.

IOW, wizards needed for these cases.

I'll have a look at the applevel stuff in /lib/test2 (the 3 tests pass
all independantly but fail when executed in sequence), since I'm more
likely directly responsible for breakage there.

> That doesn't seem too much in line with the "test-driven development"
> approach.
> 

Yes. Otoh, it seems that I'm currently the only consummer of these
features, so I didn't feel the urge to rush to fix them immediately. I
will look at them today.

Regards,
Aur?lien.

From fijal at genesilico.pl  Mon Jul 24 12:35:05 2006
From: fijal at genesilico.pl (Maciek Fijalkowski)
Date: Mon, 24 Jul 2006 12:35:05 +0200
Subject: [pypy-dev] JS backend updates
Message-ID: <44C4A259.3010309@genesilico.pl>

There were some updates recently about JS backend.

Recently:

- Added some decorator which might be usefull at declaring server-side 
of AJAX communication
- Polished a little bit backend so now it is able to run richards and 
pystone as soon as you cut off zeroing assumption out of ll_alloc_and_set
- Added some documentation
- Bugfixes and some helper functions (ie StringBuilder) so now 
"aaa".upper() works as expected

Coming work:

- Hiding browser differencies
- more examples & docs
- Maybe some stackless stuff?
- Very hard and usefull point: to get out Python-level traceback out of 
exception handlers.
- Some JS optimisations (assuming that JS compilers themselves cannot 
optimise much if anything at all)
- BnB more playable.


From cfbolz at gmx.de  Mon Jul 24 13:51:02 2006
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 24 Jul 2006 13:51:02 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060724102151.GC15904@crater.logilab.fr>
References: <20060721203613.GA2825@code0.codespeak.net>
	<20060724102151.GC15904@crater.logilab.fr>
Message-ID: <44C4B426.60206@gmx.de>

Hi!

Aur?lien Camp?as wrote:
> On Fri, Jul 21, 2006 at 10:36:13PM +0200, Armin Rigo wrote:
>> Hi all, hi Aurelien,
>>
>> The stackless-related tests are all consistently failing nowadays.  Some
>> have been failing for 4 days and others for 9 days.  See
>>
>>    http://snake.cs.uni-duesseldorf.de/pypytest/summary.html
>>
>> Given that you appear to continue to work on it, but without making
>> tests pass again, I'm a bit worrying exactly about what you are
>> doing?
>
> First, I did some (superficial, nothing related to the translation
> aspect itself) changes that allowed me to build a (somewhat
> non-crashing) translated pypy-logic using clonable coroutines. These
> changes indeed broke some tests, in ways that I don't
> understand. Please note that indeed most failing tests in
> /modules/_stackless/test are due to run-time failures, after a
> successful translation. Michael put in a fix last week which,
> strangely, yielded a working version on ... PPC arch only.
>
> IOW, wizards needed for these cases.
>
> I'll have a look at the applevel stuff in /lib/test2 (the 3 tests pass
> all independantly but fail when executed in sequence), since I'm more
> likely directly responsible for breakage there.

It's just not good to have failing tests checked in. If they fail for
good reasons and will probably work soon again, just skip them with an
appropriate message. Otherwise it becomes impossible for somebody
working in a related area to find out whether his changes were
responsible for the breakage.

>> That doesn't seem too much in line with the "test-driven development"
>> approach.
>>
>
> Yes. Otoh, it seems that I'm currently the only consummer of these
> features, so I didn't feel the urge to rush to fix them immediately. I
> will look at them today.

Well, that's not true. People wanted to try out stackless and couldn't
because it didn't translate. I needed to do benchmarks and couldn't
either. It's just a good general policy to have tranlations working all
the time.

Cheers,

Carl Friedrich

From Ben.Young at risk.sungard.com  Mon Jul 24 14:48:57 2006
From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com)
Date: Mon, 24 Jul 2006 13:48:57 +0100
Subject: [pypy-dev] Example of .NET filehandles
Message-ID: <OF4BE4E08A.626924BA-ON802571B5.00462B19-802571B5.004666AF@risk.sungard.com>

Hi Antonio,

Here's the example I mentioned. I don't think it's particularly safe, so 
it may be better to keep a map of ints to filestreams that you have 
opened, and use that if possible, and fall back to this if you can't.

Cheers,
Ben

using System;
using System.Collections.Generic;
using System.Text;

namespace posix
{
    class Program
    {
        static void Main(string[] args)
        {
            System.IO.FileStream stream1 = new System.IO.FileStream
(args[0], System.IO.FileMode.Create);
            int handle = 
stream1.SafeFileHandle.DangerousGetHandle().ToInt32();

            System.IO.FileStream stream2 = new System.IO.FileStream(new 
Microsoft.Win32.SafeHandles.SafeFileHandle(new IntPtr(handle), true), 
System.IO.FileAccess.Write);
            stream2.Write(new byte[]{(byte)'H', (byte)'e', (byte)'l', (
byte)'l', (byte)'o'}, 0, 5);
            stream2.Close();
        }
    }
}

From arigo at tunes.org  Mon Jul 24 15:33:59 2006
From: arigo at tunes.org (Armin Rigo)
Date: Mon, 24 Jul 2006 15:33:59 +0200
Subject: [pypy-dev] Failing stackless tests
In-Reply-To: <20060724102151.GC15904@crater.logilab.fr>
References: <20060721203613.GA2825@code0.codespeak.net>
	<20060724102151.GC15904@crater.logilab.fr>
Message-ID: <20060724133359.GB12961@code0.codespeak.net>

Hi Aurelien,

On Mon, Jul 24, 2006 at 12:21:52PM +0200, Aur?lien Camp?as wrote:
> First, I did some (superficial, nothing related to the translation
> aspect itself) changes that allowed me to build a (somewhat
> non-crashing) translated pypy-logic using clonable coroutines.

I did not look closely, so I cannot comment, but it seems to me that the
tests and the demo we fixed during the post-EuroPython sprint all
translated correctly, so I'm a bit surprized that you had to hack so
much on the existing stackless code only for translatability reasons.

This said, there were not many tests so far.  That's also why I'm not
happy to see half of them fail nowadays.  The "agreed" development
procedure for PyPy is to write more tests when a bug is identified or
existing code doesn't translate in some context, and then make the test
pass without breaking the previous tests (which are all valid, so again
I'm surprized that you can compile anything at all when half of the
basic tests fail).


A bientot,

Armin

From aurelien.campeas at logilab.fr  Mon Jul 24 15:56:27 2006
From: aurelien.camp