From greg.ewing at canterbury.ac.nz  Thu May  1 00:21:25 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 01 May 2008 10:21:25 +1200
Subject: [Cython] [Pyrex] pyrex problems
In-Reply-To: <5533c3c80804211621p39aec737lb31a2dd76cc5f2b5@mail.gmail.com>
References: <5533c3c80804211527o32568558sea7961803c79711c@mail.gmail.com>
	<85e81ba30804211612i3e38bebw5a74825815def96d@mail.gmail.com>
	<5533c3c80804211621p39aec737lb31a2dd76cc5f2b5@mail.gmail.com>
Message-ID: <4818F0E5.70101@canterbury.ac.nz>

On Mon, Apr 21, 2008 at 3:27 PM, Marco Zanger <marcozanger at gmail.com
<mailto:marcozanger at gmail.com>> wrote:

>  ctypedef struct cGCoptimization "GCoptimization":
>    EnergyType expansionIter "expansion"(int max_num_iterations)
>

You need to declare the member functions as function
pointers, e.g.

   ctypedef struct cGCoptimization "GCoptimization":
     EnergyType (*expansionIter "expansion")(int max_num_iterations)

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu May  1 00:24:37 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 01 May 2008 10:24:37 +1200
Subject: [Cython] pxi or pxd for numpy?
In-Reply-To: <96de71860804241417i4ffa5b2bm7b663b3c4f34b273@mail.gmail.com>
References: <6ce0ac130804241316p56006278y8319e49db10eb7be@mail.gmail.com>
	<96de71860804241417i4ffa5b2bm7b663b3c4f34b273@mail.gmail.com>
Message-ID: <4818F1A5.3080907@canterbury.ac.nz>

On Thu, Apr 24, 2008 at 1:16 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:

> The only way I can get all this to work is the rename numpy.pxi ->
> c_numpy.pxd and use cimport.  Then all works well.  But, this seems to
> go against the recommendation that pxd files should not be used for
> this purpose.

What recommendation are you talking about? Using a pxd file
to provide a namespace is fine as far as I'm concerned.
It's one of the things they were invented for.

-- 
Greg

From ellisonbg.net at gmail.com  Thu May  1 00:52:56 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 30 Apr 2008 16:52:56 -0600
Subject: [Cython] pxi or pxd for numpy?
In-Reply-To: <4818F1A5.3080907@canterbury.ac.nz>
References: <6ce0ac130804241316p56006278y8319e49db10eb7be@mail.gmail.com>
	<96de71860804241417i4ffa5b2bm7b663b3c4f34b273@mail.gmail.com>
	<4818F1A5.3080907@canterbury.ac.nz>
Message-ID: <6ce0ac130804301552y62186885i208b0b262452dd9c@mail.gmail.com>

>  What recommendation are you talking about? Using a pxd file
>  to provide a namespace is fine as far as I'm concerned.
>  It's one of the things they were invented for.

Previously, (on this list and at an in person meeting of Sage devs)
some people were saying that .pxi files should be used for this
purpose.  Because of these discussions, numpy.pxd was renamed
numpy.pxi, but now this looks like a bad decision.

>  --
>  Greg
>

From greg.ewing at canterbury.ac.nz  Thu May  1 01:08:36 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 01 May 2008 11:08:36 +1200
Subject: [Cython] pxi or pxd for numpy?
In-Reply-To: <6ce0ac130804301552y62186885i208b0b262452dd9c@mail.gmail.com>
References: <6ce0ac130804241316p56006278y8319e49db10eb7be@mail.gmail.com>
	<96de71860804241417i4ffa5b2bm7b663b3c4f34b273@mail.gmail.com>
	<4818F1A5.3080907@canterbury.ac.nz>
	<6ce0ac130804301552y62186885i208b0b262452dd9c@mail.gmail.com>
Message-ID: <4818FBF4.4010808@canterbury.ac.nz>

Brian Granger wrote:

> Previously, (on this list and at an in person meeting of Sage devs)
> some people were saying that .pxi files should be used for this
> purpose.  Because of these discussions, numpy.pxd was renamed
> numpy.pxi, but now this looks like a bad decision.

Indeed, I would say the opposite. You should almost
*never* use a .pxi file for anything.

The reason is the same reason that "import *" is almost
always a bad idea. When you cimport a name, it's easy
to find out where it came from, but if you get it by
including a .pxi, it can be a lot harder.

The 'include' statement is only present in Pyrex mostly
for historical reasons. If I had invented 'cimport'
earlier, it might never have existed at all.

-- 
Greg

From wstein at gmail.com  Thu May  1 01:15:59 2008
From: wstein at gmail.com (William Stein)
Date: Wed, 30 Apr 2008 16:15:59 -0700
Subject: [Cython] pxi or pxd for numpy?
In-Reply-To: <6ce0ac130804301552y62186885i208b0b262452dd9c@mail.gmail.com>
References: <6ce0ac130804241316p56006278y8319e49db10eb7be@mail.gmail.com>
	<96de71860804241417i4ffa5b2bm7b663b3c4f34b273@mail.gmail.com>
	<4818F1A5.3080907@canterbury.ac.nz>
	<6ce0ac130804301552y62186885i208b0b262452dd9c@mail.gmail.com>
Message-ID: <85e81ba30804301615t95e5d1excd279c0d2ca93aec@mail.gmail.com>

On Wed, Apr 30, 2008 at 3:52 PM, Brian Granger <ellisonbg.net at gmail.com> wrote:
> >  What recommendation are you talking about? Using a pxd file
>  >  to provide a namespace is fine as far as I'm concerned.
>  >  It's one of the things they were invented for.
>
>  Previously, (on this list and at an in person meeting of Sage devs)
>  some people were saying that .pxi files should be used for this
>  purpose.  Because of these discussions, numpy.pxd was renamed
>  numpy.pxi, but now this looks like a bad decision.
>

It's me.     I gave them some advice about using pxi versus
pxd files at Sage Days 8, but I did not actually *look* at numpy.pxd
before giving said advice.    Clearly I was wrong.   Sorry for
any confusion caused by my offhand comment.

 -- William

From robertwb at math.washington.edu  Thu May  1 03:17:31 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 30 Apr 2008 18:17:31 -0700
Subject: [Cython] __getattribute__
In-Reply-To: <20080430004208.GJ15181@tilt>
References: <20080430004208.GJ15181@tilt>
Message-ID: <5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>

On Apr 29, 2008, at 5:42 PM, Peter Todd wrote:

> Is there a __getattribute__ work-alike in Cython?
>
> Essentially I need direct control over an objects tp_getattro and
> tp_setattro slots to implement a wrapper class. Specificly
> wrapped.__class__ should go to the wrapped objects class attribute,  
> not
> the wrapping objects __class__ attribute.
>
> __getattr__ outputs C-source that includes a call to
> PyObject_GenericGetAttr first, and won't run my code if that call
> succeeds.
>
> Thanks,
>
> Peter

Not that I'm aware of, though I'd imagine that implementing  
__getattribute__ (if it exists) as being called at the top of this  
function would be fairly easy to do. One would want to match Python  
symantics exactly.

- Robert

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080430/e5d1c3c1/attachment-0001.pgp 

From robertwb at math.washington.edu  Thu May  1 03:49:09 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 30 Apr 2008 18:49:09 -0700
Subject: [Cython] Porting the Docs
In-Reply-To: <20080430153918.GA6561@giton>
References: <20080426000805.GA5117@basestar> <20080430153918.GA6561@giton>
Message-ID: <AD59527C-2783-4979-BDD5-42101EE10B23@math.washington.edu>

On Apr 30, 2008, at 8:39 AM, Gabriel Gellner wrote:

> So I have finished the first rough mockup, which mainly consists of  
> getting
> the markup correct. There are still some rough edges, mainly with  
> tables, and
> some of the latex output, but in the spirit of release early  
> release often ;-)
>
> Check out the html at:
> http://www.mudskipper.ca/cython-doc
>
> Get the pdf at:
> http://www.mudskipper.ca/cython.pdf
>
> And finally get the source at:
> http://www.mudskipper.ca/cython_doc.tar.gz
> or
> http://www.mudskipper.ca/cython_doc.zip
>
> Again I would appreciate any comments on if I am screwing up  
> authorship. Down
> the road I think it would be good to attribute everything to the  
> 'Cython Doc
> Team' and have a page that lists contributers. To do this we should  
> make the
> license and authorship on the wiki more explicit. Tell me what you  
> think? I am
> no lawyer, and certainly don't want to piss anyone off who has put  
> the hard
> work into documenting either cython or pyrex.
>
> My plan for the next steps (in order of importance, any comments):
> - Make a PyGments lexer for cython/pyrex so we get nice color coding.
> - Fix up the latex style file so that boxes are not messed up when  
> we have
>   code examples.
> - Put on my writing hat, and do an overhaul of the structure of the  
> docs so
>   that it is faster to navigate. I will be using the python doc  
> structure as a
>   reference.
> - Get some simple howto's written.
> - Thinking if there is an easy way to test the cython code in the  
> docs so that
>   I can ensure accuracy.

Thanks again for putting these all together in a nice form. While  
browsing I noticed some inaccuracies (often just dealing with the  
differences between Cython and Pyrex). Should I send them directly to  
you, or should we set up some revision control (need not be  
centralized).

- Robert


From pete at petertodd.org  Thu May  1 03:41:07 2008
From: pete at petertodd.org (Peter Todd)
Date: Wed, 30 Apr 2008 21:41:07 -0400
Subject: [Cython] __getattribute__
In-Reply-To: <5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
Message-ID: <20080501014107.GK15181@tilt>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, Apr 30, 2008 at 06:17:31PM -0700, Robert Bradshaw wrote:
> On Apr 29, 2008, at 5:42 PM, Peter Todd wrote:
> 
> >Is there a __getattribute__ work-alike in Cython?
> >
> >Essentially I need direct control over an objects tp_getattro and
> >tp_setattro slots to implement a wrapper class. Specificly
> >wrapped.__class__ should go to the wrapped objects class attribute,  
> >not
> >the wrapping objects __class__ attribute.
> >
> >__getattr__ outputs C-source that includes a call to
> >PyObject_GenericGetAttr first, and won't run my code if that call
> >succeeds.
> >
> >Thanks,
> >
> >Peter
> 
> Not that I'm aware of, though I'd imagine that implementing  
> __getattribute__ (if it exists) as being called at the top of this  
> function would be fairly easy to do. One would want to match Python  
> symantics exactly.

Thanks for the reply.

I've been looking at the Cython source myself, and it looks doable, I'll
put some effort into implementing it. From what I can see I'd be
updating generate_getattro_function() to do a scope.lookup_here() on
'__getattribute__' first, if that succeeds, output code to call it,
otherwise go on to the existing code. With the additional complication
that if __getattribute__ is defined, as well as __getattr__, the latter
must be called if the former raises AttributeError.


Another question is I noticed that the latest cython-devel code emits
warnings on usage of __new__, saying to use __cinit__ instead. I take it
there is planned work to make __new__ more like Python's __new__ and
have it create the object as well? Again, that's another feature I could
really use.


FWIW I'm writing a electrical design automation library called Tuke.
Currently I have some C extension API code written to wrap objects in
contexts, but would like to re-write that in Cython as well as implement
a whole lot of other basic functionality.

http://github.com/retep/tuke/tree/master

- -- 
http://petertodd.org 'peter'[:-1]@petertodd.org 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIGR+z3bMhDbI9xWQRAjUsAKCx7XeM/+KRs9vCA3+XUuundIfF0wCfRJcf
nH3ByGq6MPaRUktCxWho5KE=
=oz0w
-----END PGP SIGNATURE-----

From robertwb at math.washington.edu  Thu May  1 04:12:42 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 30 Apr 2008 19:12:42 -0700
Subject: [Cython] __getattribute__
In-Reply-To: <20080501014107.GK15181@tilt>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
	<20080501014107.GK15181@tilt>
Message-ID: <8367C02B-B674-44BB-AF04-4472EE8AABED@math.washington.edu>

On Apr 30, 2008, at 6:41 PM, Peter Todd wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Wed, Apr 30, 2008 at 06:17:31PM -0700, Robert Bradshaw wrote:
>> On Apr 29, 2008, at 5:42 PM, Peter Todd wrote:
>>
>>> Is there a __getattribute__ work-alike in Cython?
>>>
>>> Essentially I need direct control over an objects tp_getattro and
>>> tp_setattro slots to implement a wrapper class. Specificly
>>> wrapped.__class__ should go to the wrapped objects class attribute,
>>> not
>>> the wrapping objects __class__ attribute.
>>>
>>> __getattr__ outputs C-source that includes a call to
>>> PyObject_GenericGetAttr first, and won't run my code if that call
>>> succeeds.
>>>
>>> Thanks,
>>>
>>> Peter
>>
>> Not that I'm aware of, though I'd imagine that implementing
>> __getattribute__ (if it exists) as being called at the top of this
>> function would be fairly easy to do. One would want to match Python
>> symantics exactly.
>
> Thanks for the reply.
>
> I've been looking at the Cython source myself, and it looks doable,  
> I'll
> put some effort into implementing it.

Great!

> From what I can see I'd be
> updating generate_getattro_function() to do a scope.lookup_here() on
> '__getattribute__' first, if that succeeds, output code to call it,
> otherwise go on to the existing code.

Sounds like a good plan. Is __getattribute__ inherited?

>
> With the additional complication
> that if __getattribute__ is defined, as well as __getattr__, the  
> latter
> must be called if the former raises AttributeError.

Yep, it should go onto the existing code (or call the super/default  
getattribute) on an error, even if __getattr__ is not defined, to  
match the specs.

> Another question is I noticed that the latest cython-devel code emits
> warnings on usage of __new__, saying to use __cinit__ instead. I  
> take it
> there is planned work to make __new__ more like Python's __new__ and
> have it create the object as well? Again, that's another feature I  
> could
> really use.

Eventually we may implement this (I looked into it some, but it  
wasn't as straightforward as I had hoped), but using __cinit__ now is  
good to avoid the confusion.

> FWIW I'm writing a electrical design automation library called Tuke.
> Currently I have some C extension API code written to wrap objects in
> contexts, but would like to re-write that in Cython as well as  
> implement
> a whole lot of other basic functionality.
>
> http://github.com/retep/tuke/tree/master

Cool. Should I add this to http://wiki.cython.org/projects ?

- Robert

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080430/38c2d1ac/attachment.pgp 

From robertwb at math.washington.edu  Thu May  1 04:16:11 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 30 Apr 2008 19:16:11 -0700
Subject: [Cython] a problem with declaring and initializing <type>
	(*pointer)[N] = NULL
In-Reply-To: <e7ba66e40804300822q4e85fab3lcfde08b67ded0e5d@mail.gmail.com>
References: <e7ba66e40804300822q4e85fab3lcfde08b67ded0e5d@mail.gmail.com>
Message-ID: <EFD5E69C-E05C-4165-9A80-4C77AAF50F07@math.washington.edu>

That is really strange, no idea why.

On Apr 30, 2008, at 8:22 AM, Lisandro Dalcin wrote:
> Just pulled from cython-devel repo, this does not work (neither  
> before the pull)
>
> cdef int (*iranges)[3] = NULL
>
> but this indeed work
>
> cdef int (*iranges)[3]
> iranges = NULL
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From pete at petertodd.org  Thu May  1 05:16:03 2008
From: pete at petertodd.org (Peter Todd)
Date: Wed, 30 Apr 2008 23:16:03 -0400
Subject: [Cython] __getattribute__
In-Reply-To: <8367C02B-B674-44BB-AF04-4472EE8AABED@math.washington.edu>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
	<20080501014107.GK15181@tilt>
	<8367C02B-B674-44BB-AF04-4472EE8AABED@math.washington.edu>
Message-ID: <20080501031603.GL15181@tilt>

On Wed, Apr 30, 2008 at 07:12:42PM -0700, Robert Bradshaw wrote:
> >From what I can see I'd be
> >updating generate_getattro_function() to do a scope.lookup_here() on
> >'__getattribute__' first, if that succeeds, output code to call it,
> >otherwise go on to the existing code.
> 
> Sounds like a good plan. Is __getattribute__ inherited?

Yes, and when inherited, it overrides __getattr__:

>>> class a(object):
...     def __getattribute__(self,n):
...             print 'a getattribute',n
...             raise AttributeError
... 
>>> class b(a):
...     def __getattr__(self,n):
...             print 'b getattr',n
...             return None
... 
>>> o = b()
>>> print o.asdf
a getattribute asdf
b getattr asdf
None
>>> 

But note that b.__getattr__ is called in the end. If b defines __setattr__, it
is called for any setattr as well. 

The Python C extension API docs also state that tp_getattro and tp_setattro are
"inherited by subtypes together with tp_getattr: a subtype inherits both
tp_getattr and tp_getattro from its base type when the subtype's tp_getattr and
tp_getattro are both NULL."

I think this will cause a conflict, in the following situation:

cdef class a:
    def __getattribute__(self,n):
        return n

cdef class b(a):
    def __setattr__(self,n,v):
	print 'b setattr',n,v

Evaluating b().asdf will not evaluate to 'asdf', rather an
AttributeError will be raised. Why? Because PyObject_GenericGetAttr()
doesn't look in the base class for attributes at all. And tp_getattro
will for type b will end up pointing to PyObject_GenericGetAttr due to
the above rule of only inhereting if *both* tp_getattro and tp_setattro
are not set. From what I can see in the slot_table definition
tp_getattro and tp_setattro are handled independently, and setting
setattr will leave getattro set to 0

> >
> >With the additional complication
> >that if __getattribute__ is defined, as well as __getattr__, the  
> >latter
> >must be called if the former raises AttributeError.
> 
> Yep, it should go onto the existing code (or call the super/default  
> getattribute) on an error, even if __getattr__ is not defined, to  
> match the specs.

Will do.

> >Another question is I noticed that the latest cython-devel code emits
> >warnings on usage of __new__, saying to use __cinit__ instead. I  
> >take it
> >there is planned work to make __new__ more like Python's __new__ and
> >have it create the object as well? Again, that's another feature I  
> >could
> >really use.
> 
> Eventually we may implement this (I looked into it some, but it  
> wasn't as straightforward as I had hoped), but using __cinit__ now is  
> good to avoid the confusion.

Thanks, I'll take a look at it as well.

> >FWIW I'm writing a electrical design automation library called Tuke.
> >Currently I have some C extension API code written to wrap objects in
> >contexts, but would like to re-write that in Cython as well as  
> >implement
> >a whole lot of other basic functionality.
> >
> >http://github.com/retep/tuke/tree/master
> 
> Cool. Should I add this to http://wiki.cython.org/projects ?

Sure, although, it may still be another two or three weeks before they
is any actual Cython code in the above tree. ;)

-- 
http://petertodd.org 'peter'[:-1]@petertodd.org 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080430/f62fafeb/attachment.pgp 

From robertwb at math.washington.edu  Thu May  1 07:32:10 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 30 Apr 2008 22:32:10 -0700
Subject: [Cython] Cython 0.9.6.14 release candidate
Message-ID: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>

Up at http://www.cython.org/Cython-0.9.6.14.tar.gz (or pull from the  
devel branch). If no problems are found, this will be the release. (I  
have already tested Sage).

- Robert


From greg.ewing at canterbury.ac.nz  Thu May  1 07:44:36 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 01 May 2008 17:44:36 +1200
Subject: [Cython] __getattribute__
In-Reply-To: <20080501014107.GK15181@tilt>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
	<20080501014107.GK15181@tilt>
Message-ID: <481958C4.2070205@canterbury.ac.nz>

Peter Todd wrote:
> Another question is I noticed that the latest cython-devel code emits
> warnings on usage of __new__, saying to use __cinit__ instead. I take it
> there is planned work to make __new__ more like Python's __new__ and
> have it create the object as well? Again, that's another feature I could
> really use.

That's the idea. I'm not sure exactly what I'm going to
do with __new__ yet, but I want to free up the name so
I can do something better with it later.

-- 
Greg

From stefan_ml at behnel.de  Thu May  1 07:54:52 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 01 May 2008 07:54:52 +0200
Subject: [Cython] __getattribute__
In-Reply-To: <20080501031603.GL15181@tilt>
References: <20080430004208.GJ15181@tilt>	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>	<20080501014107.GK15181@tilt>	<8367C02B-B674-44BB-AF04-4472EE8AABED@math.washington.edu>
	<20080501031603.GL15181@tilt>
Message-ID: <48195B2C.6060301@behnel.de>

Hi,


Peter Todd wrote:
>>> From what I can see I'd be
>>> updating generate_getattro_function() to do a scope.lookup_here() on
>>> '__getattribute__' first, if that succeeds, output code to call it,
>>> otherwise go on to the existing code.

Please see the code in Python's typeobject.c, functions "slot_tp_getattr_hook"
and "slot_tp_getattro" (essentially a fast path), and the comment above them.
The code Cython generates must behave like these two.


> On Wed, Apr 30, 2008 at 07:12:42PM -0700, Robert Bradshaw wrote:
>> Sounds like a good plan. Is __getattribute__ inherited?

Python is already doing a couple of things here for us. There is some code in
"inherit_slots" (typeobject.c) that inherits the slots, but only if both get*
functions are set to NULL, or set* functions respectively:

        if (type->tp_getattr == NULL && type->tp_getattro == NULL) {
                type->tp_getattr = base->tp_getattr;
                type->tp_getattro = base->tp_getattro;
        }
        if (type->tp_setattr == NULL && type->tp_setattro == NULL) {
                type->tp_setattr = base->tp_setattr;
                type->tp_setattro = base->tp_setattro;
        }

Again, Cython must behave alike here.

Stefan

From robertwb at math.washington.edu  Thu May  1 13:01:09 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 1 May 2008 04:01:09 -0700
Subject: [Cython] [Pyrex]   Python 3
In-Reply-To: <fv8a8h$g1b$1@ger.gmane.org>
References: <tkrat.f8570a2e9b3f66cf@skynet.be>	<15D9902A-0C11-4939-95C4-86C50B5B816F@math.washington.edu>	<4816BB88.6000708@canterbury.ac.nz>
	<4816C6FC.2050103@behnel.de>	<4816CC06.1040706@canterbury.ac.nz>
	<4816F091.1000505@cheimes.de> <4817A2D1.9010802@canterbury.ac.nz>
	<fv8a8h$g1b$1@ger.gmane.org>
Message-ID: <D5054997-98BC-44AC-A1F0-E2698188DA8D@math.washington.edu>


On Apr 29, 2008, at 4:16 PM, Christian Heimes wrote:
> Greg Ewing schrieb:
>> Thanks, I didn't know that.
>>
>> So it seems that we could, if it were considered desirable,
>> have an automatic cast from unicode to char *. But the encoding
>> would *have* to be utf8 -- anything else would require memory
>> allocation.
>
> You're welcome! :)
>
> The UTF-8 default encoding is hard coded in Python 3.0. IMHO it's the
> most sensible encoding for users from the Western world. Asian users
> would probably prefer UTF-16 but that's a waste of memory for the  
> rest.

Yes. Even ignoring memory concerns, UTF-8 plays much nicer with unix/ 
other tools that are not unicode-aware.

> In my opinion wchar_t support is much more important than casting
> PyUnicode objects to char*. Especially Windows developers need wchar_t
> for the wide Windows API. Python 2.6 and 3.0 have dropped support for
> the Windows 9x/ME/NT series. Only 2k SP4 and newer are supported.
> wchar_t support is an important step for the poor souls ... err  
> Windows
> developers. ;)

I'm curious what you mean by wchar_t support--one can typedef wchar_t  
to be an integer type. Or perhaps you mean something more...

- Robert


From robertwb at math.washington.edu  Thu May  1 13:02:00 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 1 May 2008 04:02:00 -0700
Subject: [Cython] MultiCores/AMD open source tools
In-Reply-To: <dd3c0ce40804291726j60809287k1ec176240e97f090@mail.gmail.com>
References: <dd3c0ce40804291712i18da1bech8fa7a7df7241a72e@mail.gmail.com>
	<dd3c0ce40804291726j60809287k1ec176240e97f090@mail.gmail.com>
Message-ID: <1C5D2BE2-66BB-4362-9244-332B7B1A7461@math.washington.edu>

There's the issue of the GIL, but if you don't touch the interpreter  
you can do threading just as in C.

On Apr 29, 2008, at 5:26 PM, Gottfried F. Zojer wrote:
> From: Gottfried F. Zojer <goenzoy at gmail.com>
> Date: Apr 30, 2008 2:12 AM
> Subject: MultiCores/AMD open source tools
> To: cython-dev at codespeak.net
>
>
> Hallo,
>
>  Just wondering if someone is using cython in combination with any  
> of AMD
>  open-source projects like framewave or have any other thoughts to use
>  cython/python for parallel programming or Multi Core programming.
>  Any feedback welcome
>
>  Rgds Gottfried
>
>
>  www.5152.eu
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ndbecker2 at gmail.com  Thu May  1 13:56:44 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 01 May 2008 07:56:44 -0400
Subject: [Cython] Cython 0.9.6.14 release candidate
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>
Message-ID: <fvcb5t$bs1$1@ger.gmane.org>

Robert Bradshaw wrote:

> Up at http://www.cython.org/Cython-0.9.6.14.tar.gz (or pull from the
> devel branch). If no problems are found, this will be the release. (I
> have already tested Sage).
> 
> - Robert

I did
hg clone http://hg.cython.org/cython/
python setup.py build && sudo python setup.py install
python runtests.py:

Ran 317 tests in 52.931s

FAILED (failures=3)


From bblais at gmail.com  Thu May  1 15:32:33 2008
From: bblais at gmail.com (Brian Blais)
Date: Thu, 1 May 2008 09:32:33 -0400
Subject: [Cython] access to numpy functions?
Message-ID: <F425677E-4B0F-4634-BF27-0B2138304B9B@gmail.com>

Hello,

I have been using Pyrex/Cython for a couple years now, interfacing  
with the numpy ndarray objects.  I was wondering recently if there is  
direct access to some of the functions of numpy, like dot products,  
addition and subtraction, etc... without going through the python  
API?  If I have two objects in Cython/Pyrex, like:


c_numpy.ndarray x,y

and I want to do a dot product, or add the two arrays, is there  
access to the numpy C-version of this?  I know I could write my own  
for the dot product, but I'd rather not reinvent the wheel when  
possible.



			thanks,


				Brian Blais

From dagss at student.matnat.uio.no  Thu May  1 16:39:44 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 01 May 2008 16:39:44 +0200
Subject: [Cython] access to numpy functions?
In-Reply-To: <F425677E-4B0F-4634-BF27-0B2138304B9B@gmail.com>
References: <F425677E-4B0F-4634-BF27-0B2138304B9B@gmail.com>
Message-ID: <4819D630.9060602@student.matnat.uio.no>


> API?  If I have two objects in Cython/Pyrex, like:
>
>
> c_numpy.ndarray x,y
>
> and I want to do a dot product, or add the two arrays, is there  
> access to the numpy C-version of this?  I know I could write my own  
> for the dot product, but I'd rather not reinvent the wheel when  
> possible.
>   

First off, you can use the Python functions directly, like this (Cython 
syntax):

---
import numpy
cimport c_numpy

cdef c_numpy.ndarray a = ..., b = ...
cdef c_numpy.ndarray c = numpy.dot(a, b)
---

As for calling the NumPy C functions directly: The a place to start is 
looking seems to be core/src/multiarraymodule.c. (You may want to ask in 
the numpy mailing list as well though. I'm not very familiar with NumPy 
internals.)

My first impression is that the implementations (say, 
PyArray_InnerProduct for instance) seem to be a) very generic (any 
number of dimensions/strides etc. and b) coded using the Python/C API 
directly, so there is no "inner layer" to communicate with. I.e., 
arguments to functions are Python objects.

So you're probably just as well off by calling the functions through the 
Python API like stated above.

If that is not fast enough, there doesn't seem to be much to be gained 
by "direct calls" to the C functions; and a new implementation in Cython 
seems in order. The main reason is that to make this faster you will 
want to make use of additional information -- if you know the number of 
dimensions (or even their size) and datatype at compile-time, you can 
make some optimizations that functions in the NumPy library will never 
be able to do. (For most if not all cases I'm guessing that we're 
talking very small gains if any gains here; though if you tend to 
multiply really small arrays a lot (like doing coordinate 
transformation...) then I suppose you must write your own code.)

-- 
Dag Sverre


From stefan_ml at behnel.de  Thu May  1 18:52:46 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 01 May 2008 18:52:46 +0200
Subject: [Cython] Cython 0.9.6.14 release candidate
In-Reply-To: <fvcb5t$bs1$1@ger.gmane.org>
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>
	<fvcb5t$bs1$1@ger.gmane.org>
Message-ID: <4819F55E.5010909@behnel.de>

Hi,

Neal Becker wrote:
> Robert Bradshaw wrote:
>> Up at http://www.cython.org/Cython-0.9.6.14.tar.gz (or pull from the
>> devel branch). If no problems are found, this will be the release. (I
>> have already tested Sage).
>>
>> - Robert
> 
> I did
> hg clone http://hg.cython.org/cython/

That's the wrong repository. As Robert said, the RC is in the devel branch:

    http://hg.cython.org/cython-devel/

Stefan

From ndbecker2 at gmail.com  Thu May  1 19:07:24 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 01 May 2008 13:07:24 -0400
Subject: [Cython] Cython 0.9.6.14 release candidate
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>
	<fvcb5t$bs1$1@ger.gmane.org> <4819F55E.5010909@behnel.de>
Message-ID: <fvctcc$9ed$1@ger.gmane.org>

Stefan Behnel wrote:

> Hi,
> 
> Neal Becker wrote:
>> Robert Bradshaw wrote:
>>> Up at http://www.cython.org/Cython-0.9.6.14.tar.gz (or pull from the
>>> devel branch). If no problems are found, this will be the release. (I
>>> have already tested Sage).
>>>
>>> - Robert
>> 
>> I did
>> hg clone http://hg.cython.org/cython/
> 
> That's the wrong repository. As Robert said, the RC is in the devel
> branch:
> 
>     http://hg.cython.org/cython-devel/
> 
> Stefan

Thanks, now I got:

Traceback (most recent call last):
  File "/usr/lib64/python2.5/doctest.py", line 2112, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for r_vree_1
  File "/home/nbecker/cython-devel/BUILD/r_vree_1.so", line 190, in r_vree_1

----------------------------------------------------------------------
File "/home/nbecker/cython-devel/BUILD/r_vree_1.so", line 203, in r_vree_1
Failed example:
    test(sys.maxint + 1)
Expected:
    2147483648L
Got:
    9223372036854775808L
----------------------------------------------------------------------
File "/home/nbecker/cython-devel/BUILD/r_vree_1.so", line 205, in r_vree_1
Failed example:
    test(sys.maxint * 2 + 1)
Expected:
    4294967295L
Got:
    18446744073709551615L


----------------------------------------------------------------------
Ran 213 tests in 53.887s

FAILED (failures=1)

These are probably harmless 32/64 bit errors?  Tests should be fixed?


From stefan_ml at behnel.de  Thu May  1 19:20:55 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 01 May 2008 19:20:55 +0200
Subject: [Cython] Cython 0.9.6.14 release candidate
In-Reply-To: <fvctcc$9ed$1@ger.gmane.org>
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>	<fvcb5t$bs1$1@ger.gmane.org>
	<4819F55E.5010909@behnel.de> <fvctcc$9ed$1@ger.gmane.org>
Message-ID: <4819FBF7.9030901@behnel.de>

Hi,

Neal Becker wrote:
> Traceback (most recent call last):
>   File "/usr/lib64/python2.5/doctest.py", line 2112, in runTest
>     raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for r_vree_1
>   File "/home/nbecker/cython-devel/BUILD/r_vree_1.so", line 190, in r_vree_1
> 
> ----------------------------------------------------------------------
> File "/home/nbecker/cython-devel/BUILD/r_vree_1.so", line 203, in r_vree_1
> Failed example:
>     test(sys.maxint + 1)
> Expected:
>     2147483648L
> Got:
>     9223372036854775808L
> ----------------------------------------------------------------------
> File "/home/nbecker/cython-devel/BUILD/r_vree_1.so", line 205, in r_vree_1
> Failed example:
>     test(sys.maxint * 2 + 1)
> Expected:
>     4294967295L
> Got:
>     18446744073709551615L
> 
> 
> ----------------------------------------------------------------------
> Ran 213 tests in 53.887s
> 
> FAILED (failures=1)
> 
> These are probably harmless 32/64 bit errors?  Tests should be fixed?

Sure, looks like the test is broken here. What is sys.maxint on a 64 bit platform?

Could you dig into this and try to provide a fix?

Stefan


From robertwb at math.washington.edu  Thu May  1 19:48:54 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 1 May 2008 10:48:54 -0700
Subject: [Cython] access to numpy functions?
In-Reply-To: <4819D630.9060602@student.matnat.uio.no>
References: <F425677E-4B0F-4634-BF27-0B2138304B9B@gmail.com>
	<4819D630.9060602@student.matnat.uio.no>
Message-ID: <65DA8D65-87AE-4D1D-9777-D843BD2B00C1@math.washington.edu>

On May 1, 2008, at 7:39 AM, Dag Sverre Seljebotn wrote:

>
>> API?  If I have two objects in Cython/Pyrex, like:
>>
>>
>> c_numpy.ndarray x,y
>>
>> and I want to do a dot product, or add the two arrays, is there
>> access to the numpy C-version of this?  I know I could write my own
>> for the dot product, but I'd rather not reinvent the wheel when
>> possible.
>>
>
> First off, you can use the Python functions directly, like this  
> (Cython
> syntax):
>
> ---
> import numpy
> cimport c_numpy
>
> cdef c_numpy.ndarray a = ..., b = ...
> cdef c_numpy.ndarray c = numpy.dot(a, b)
> ---
>
> As for calling the NumPy C functions directly: The a place to start is
> looking seems to be core/src/multiarraymodule.c. (You may want to  
> ask in
> the numpy mailing list as well though. I'm not very familiar with  
> NumPy
> internals.)
>
> My first impression is that the implementations (say,
> PyArray_InnerProduct for instance) seem to be a) very generic (any
> number of dimensions/strides etc. and b) coded using the Python/C API
> directly, so there is no "inner layer" to communicate with. I.e.,
> arguments to functions are Python objects.
>
> So you're probably just as well off by calling the functions  
> through the
> Python API like stated above.
>
> If that is not fast enough, there doesn't seem to be much to be gained
> by "direct calls" to the C functions; and a new implementation in  
> Cython
> seems in order.

And fortunately this is another GSoC project.

> The main reason is that to make this faster you will
> want to make use of additional information -- if you know the  
> number of
> dimensions (or even their size) and datatype at compile-time, you can
> make some optimizations that functions in the NumPy library will never
> be able to do. (For most if not all cases I'm guessing that we're
> talking very small gains if any gains here; though if you tend to
> multiply really small arrays a lot (like doing coordinate
> transformation...) then I suppose you must write your own code.)
>
> -- 
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ndbecker2 at gmail.com  Thu May  1 19:59:39 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 01 May 2008 13:59:39 -0400
Subject: [Cython] Cython 0.9.6.14 release candidate
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>	<fvcb5t$bs1$1@ger.gmane.org>
	<4819F55E.5010909@behnel.de> <fvctcc$9ed$1@ger.gmane.org>
	<4819FBF7.9030901@behnel.de>
Message-ID: <fvd0eb$kbh$2@ger.gmane.org>

Stefan Behnel wrote:

> Hi,
> 
> Neal Becker wrote:
...
>> These are probably harmless 32/64 bit errors?  Tests should be fixed?
> 
> Sure, looks like the test is broken here. What is sys.maxint on a 64 bit
> platform?
> 

python -c 'import sys; print sys.maxint'
9223372036854775807

This is x86_64


From stefan_ml at behnel.de  Thu May  1 20:10:38 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 01 May 2008 20:10:38 +0200
Subject: [Cython] Cython 0.9.6.14 release candidate
In-Reply-To: <fvd0eb$kbh$2@ger.gmane.org>
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>	<fvcb5t$bs1$1@ger.gmane.org>	<4819F55E.5010909@behnel.de>
	<fvctcc$9ed$1@ger.gmane.org>	<4819FBF7.9030901@behnel.de>
	<fvd0eb$kbh$2@ger.gmane.org>
Message-ID: <481A079E.8030705@behnel.de>

Hi,

Neal Becker wrote:
> Stefan Behnel wrote:
>> Neal Becker wrote:
> ...
>>> These are probably harmless 32/64 bit errors?  Tests should be fixed?
>> Sure, looks like the test is broken here. What is sys.maxint on a 64 bit
>> platform?
> 
> python -c 'import sys; print sys.maxint'
> 9223372036854775807
> 
> This is x86_64

I pushed a fix for the test case. Could you try it?

Thanks,
Stefan


From ndbecker2 at gmail.com  Thu May  1 20:46:42 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 01 May 2008 14:46:42 -0400
Subject: [Cython] Cython 0.9.6.14 release candidate
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>	<fvcb5t$bs1$1@ger.gmane.org>	<4819F55E.5010909@behnel.de>
	<fvctcc$9ed$1@ger.gmane.org>	<4819FBF7.9030901@behnel.de>
	<fvd0eb$kbh$2@ger.gmane.org> <481A079E.8030705@behnel.de>
Message-ID: <fvd36i$tsb$1@ger.gmane.org>

Stefan Behnel wrote:

> Hi,
> 
> Neal Becker wrote:
>> Stefan Behnel wrote:
>>> Neal Becker wrote:
>> ...
>>>> These are probably harmless 32/64 bit errors?  Tests should be fixed?
>>> Sure, looks like the test is broken here. What is sys.maxint on a 64 bit
>>> platform?
>> 
>> python -c 'import sys; print sys.maxint'
>> 9223372036854775807
>> 
>> This is x86_64
> 
> I pushed a fix for the test case. Could you try it?
> 
> Thanks,
> Stefan
Ran 213 tests in 22.508s

OK


From kirr at landau.phys.spbu.ru  Fri May  2 07:58:44 2008
From: kirr at landau.phys.spbu.ru (Kirill Smelkov)
Date: Fri, 2 May 2008 09:58:44 +0400
Subject: [Cython] Cython 0.9.6.14 release candidate
In-Reply-To: <fvd36i$tsb$1@ger.gmane.org>
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>
	<fvcb5t$bs1$1@ger.gmane.org> <4819F55E.5010909@behnel.de>
	<fvctcc$9ed$1@ger.gmane.org> <4819FBF7.9030901@behnel.de>
	<fvd0eb$kbh$2@ger.gmane.org> <481A079E.8030705@behnel.de>
	<fvd36i$tsb$1@ger.gmane.org>
Message-ID: <20080502055844.GA10755@evo>

On Thu, May 01, 2008 at 02:46:42PM -0400, Neal Becker wrote:
> Stefan Behnel wrote:
> 
> > Hi,
> > 
> > Neal Becker wrote:
> >> Stefan Behnel wrote:
> >>> Neal Becker wrote:
> >> ...
> >>>> These are probably harmless 32/64 bit errors?  Tests should be fixed?
> >>> Sure, looks like the test is broken here. What is sys.maxint on a 64 bit
> >>> platform?
> >> 
> >> python -c 'import sys; print sys.maxint'
> >> 9223372036854775807
> >> 
> >> This is x86_64
> > 
> > I pushed a fix for the test case. Could you try it?
> > 
> > Thanks,
> > Stefan
> Ran 213 tests in 22.508s

I still have 3 failures with latest cython-devel (b869698d6f22)

Attached is a log of 'python runtests.py' run on Debian testing.

-- 
    ????? ????????, ??????.
-------------- next part --------------
Doctest: notinop ... ok
Doctest: ishimoto2 ... ok
Doctest: classkwonlyargs ... ok
Doctest: assert ... ok
Doctest: kwargproblems ... ok
Doctest: ct_IF ... ok
Doctest: behnel3 ... ok
Doctest: r_jiba1 ... ok
Doctest: extkwonlyargs ... ok
Doctest: if ... ok
Doctest: subop ... ok
Doctest: dict ... ok
Doctest: ishimoto3 ... ok
Doctest: addressof ... ok
Doctest: baas3 ... ok
Doctest: r_docstrings ... ok
Doctest: r_hordijk1 ... ok
Doctest: unicodeliterals ... ok
Doctest: tuple ... ok
Doctest: powop ... ok
Doctest: compiledef ... ok
Doctest: print ... ok
Doctest: r_mang1 ... ok
Doctest: jarausch1 ... ok
Doctest: unop ... ok
Doctest: bishop2 ... ok
Doctest: cstruct ... ok
Doctest: unicodeliteralslatin1 ... ok
Doctest: pynumop ... ok
Doctest: isnonebool ... ok
Doctest: exarkun ... ok
Doctest: extinstantiate ... ok
Doctest: unpacklistcomp ... ok
Doctest: r_starargs ... ok
Doctest: kostyrka2 ... ok
Doctest: cfuncdef ... ok
Doctest: attr ... ok
modop.c:252: warning: ?__pyx_f_5modop_modptr? defined but not used
Doctest: modop ... ok
Doctest: bishop1 ... ok
Doctest: r_spamtype ... ok
Doctest: wundram1 ... ok
Doctest: kostyrka ... ok
Doctest: varargcall ... ok
Doctest: pyintop ... ok
Doctest: include ... ok
Doctest: classpass ... ok
Doctest: addloop ... ok
Doctest: extinherit ... ok
Doctest: cunion ... ok
Doctest: cvardef ... ok
Doctest: pinard7 ... ok
Doctest: append ... ok
Doctest: boolop ... ok
Doctest: r_pythonapi ... ok
Doctest: pass ... ok
Doctest: slice3 ... ok
Doctest: unpack ... ok
Doctest: pyextattrref ... ok
Doctest: pylistsubtype ... ok
Doctest: ass2global ... ok
Doctest: pycmp ... ok
Doctest: r_starargcall ... ok
Doctest: new_style_exceptions ... FAIL
Doctest: r_argdefault ... ok
Doctest: extstarargs ... ok
Doctest: kwonlyargs ... ok
Doctest: pinard8 ... ok
Doctest: rodriguez_1 ... ok
Doctest: r_extstarargs ... ok
Doctest: cstringmul ... ok
Doctest: multass ... ok
Doctest: r_barbieri1 ... ok
Doctest: list ... ok
Doctest: r_mitch_chapman_2 ... ok
Doctest: slice2 ... ok
Doctest: backquote ... ok
Doctest: r_toofewargs ... ok
Doctest: strconstinclass ... ok
Doctest: extclasspass ... ok
Doctest: pinard6 ... ok
Doctest: tuplereassign ... FAIL
Doctest: r_starargsonly ... ok
Doctest: dietachmayer1 ... ok
Doctest: lepage_1 ... ok
Doctest: exttype ... ok
Doctest: r_pyclassdefault ... ok
Doctest: r_addint ... ok
Doctest: unicodeliteralsdefault ... ok
Doctest: r_extcomplex2 ... ok
Doctest: return ... ok
Doctest: behnel2 ... ok
Doctest: concatcstrings ... ok
Doctest: getattr3call ... ok
Doctest: unicodefunction ... ok
Doctest: pinard5 ... ok
Doctest: r_primes ... ok
Doctest: r_pyclass ... ok
Doctest: r_print ... ok
Doctest: ass2local ... ok
Doctest: r_vree_1 ... ok
Doctest: simpcall ... ok
Doctest: modbody ... ok
Doctest: altet2 ... ok
Doctest: r_mcintyre1 ... ok
Doctest: specialfloat ... ok
Doctest: addop ... ok
Doctest: r_forloop ... ok
Doctest: strfunction ... ok
Doctest: cintop ... ok
Doctest: ref2local ... ok
Doctest: literals ... ok
Doctest: ct_DEF ... ok
Doctest: extlen ... ok
Doctest: sizeof ... ok
Doctest: inop ... ok
Doctest: starargs ... ok
Doctest: behnel1 ... ok
compiling undefinedname ... ok
compiling encoding ... ok
compiling void_as_arg ... ok
compiling arrayptrcompat ... arrayptrcompat.c:123: warning: ?__pyx_f_14arrayptrcompat_f? defined but not used
ok
compiling extsetattr ... ok
compiling ctypedef ... ok
compiling extpropertydoc ... ok
compiling extimportedsubtype ... ok
compiling extsetitem ... ok
compiling cargdef ... ok
compiling enumintcompat ... enumintcompat.c:135: warning: ?__pyx_f_13enumintcompat_f? defined but not used
ok
compiling omittedargnames ... ok
compiling import ... ok
compiling callingconvention ... ok
compiling lepage_2 ... ok
compiling extpropertyget ... ok
compiling builtin ... ok
compiling formfeed ... ok
compiling emptytry ... emptytry.c:111: warning: ?__pyx_f_8emptytry_f? defined but not used
ok
compiling extcmethcall ... extcmethcall.c:179: warning: ?__pyx_f_12extcmethcall_tomato? defined but not used
ok
compiling extdelslice ... ok
compiling ewing4 ... ewing4.c:110: warning: ?__pyx_f_6ewing4_f? defined but not used
ok
compiling extindex ... ok
compiling huss2 ... huss2.c:125: warning: ?__pyx_f_5huss2_f? defined but not used
ok
compiling declarations ... declarations.c:125: warning: ?__pyx_f_12declarations_f? defined but not used
declarations.c:146: warning: ?__pyx_f_12declarations_g? defined but not used
ok
compiling extdelitem ... ok
compiling excvaldecl ... excvaldecl.c:116: warning: ?__pyx_f_10excvaldecl_spam? defined but not used
excvaldecl.c:131: warning: ?__pyx_f_10excvaldecl_eggs? defined but not used
excvaldecl.c:146: warning: ?__pyx_f_10excvaldecl_grail? defined but not used
excvaldecl.c:161: warning: ?__pyx_f_10excvaldecl_tomato? defined but not used
excvaldecl.c:176: warning: ?__pyx_f_10excvaldecl_brian? defined but not used
excvaldecl.c:190: warning: ?__pyx_f_10excvaldecl_silly? defined but not used
ok
compiling ctypedefenum ... ok
compiling tryexcept ... ok
compiling for ... ok
compiling specmethextarg ... ok
compiling while ... while.c: In function ?__pyx_pf_5while_f?:
while.c:197: warning: ?__pyx_v_i? is used uninitialized in this function
ok
compiling jiba3 ... ok
compiling varargdecl ... varargdecl.c:110: warning: ?__pyx_f_10varargdecl_grail? defined but not used
ok
compiling extsetslice ... ok
compiling belchenko1 ... belchenko1.c:113: warning: ?__pyx_f_10belchenko1__is_aligned? defined but not used
ok
compiling ewing6 ... ewing6.c:207: warning: ?__pyx_f_6ewing6_f? defined but not used
ok
compiling jiba4 ... ok
compiling nogil ... nogil.c:114: warning: ?__pyx_f_5nogil_f? defined but not used
ok
compiling extargdefault ... ok
compiling typecast ... typecast.c:113: warning: ?__pyx_f_8typecast_f? defined but not used
ok
compiling coventry1 ... ok
compiling fromimport ... ok
compiling cstructreturn ... cstructreturn.c:124: warning: ?__pyx_f_13cstructreturn_f? defined but not used
ok
compiling globvardef ... ok
compiling ctypedefclass ... ok
compiling ia_cdefblock ... ia_cdefblock.c:157: warning: ?__pyx_f_12ia_cdefblock_priv_f? defined but not used
ok
compiling coercetovoidptr ... coercetovoidptr.c:111: warning: ?__pyx_f_15coercetovoidptr_f? defined but not used
ok
compiling withgil ... withgil.c:114: warning: ?__pyx_f_7withgil_f? defined but not used
withgil.c:140: warning: ?__pyx_f_7withgil_g? defined but not used
ok
compiling declandimpl ... ok
compiling ishimoto1 ... ok
compiling specmethargdefault ... ok
compiling funcptr ... ok
compiling globalonly ... ok
compiling extdelattr ... ok
compiling pyclass ... ok
compiling tryfinally ... tryfinally.c: In function ?__pyx_pf_10tryfinally_f?:
tryfinally.c:213: warning: ?__pyx_exc_type? is used uninitialized in this function
tryfinally.c:214: warning: ?__pyx_exc_value? is used uninitialized in this function
tryfinally.c:215: warning: ?__pyx_exc_tb? is used uninitialized in this function
tryfinally.c:223: warning: ?__pyx_exc_lineno? is used uninitialized in this function
tryfinally.c:311: warning: ?__pyx_exc_tb? is used uninitialized in this function
tryfinally.c:311: warning: ?__pyx_exc_value? is used uninitialized in this function
tryfinally.c:311: warning: ?__pyx_exc_type? is used uninitialized in this function
tryfinally.c:312: warning: ?__pyx_exc_lineno? is used uninitialized in this function
ok
compiling doda1 ... doda1.c:127: warning: ?__pyx_f_5doda1_foo? defined but not used
ok
compiling extgetattr ... ok
compiling johnson2 ... ok
compiling cenum ... cenum.c:129: warning: ?__pyx_f_5cenum_eggs? defined but not used
ok
compiling coercearraytoptr ... coercearraytoptr.c:126: warning: ?__pyx_f_16coercearraytoptr_eggs? defined but not used
ok
compiling kleckner1 ... kleckner1.c:126: warning: ?__pyx_f_9kleckner1_g? defined but not used
ok
compiling exthash ... ok
compiling extern ... extern.c:131: warning: ?__pyx_f_6extern_grail? defined but not used
ok
compiling pinard4 ... ok
compiling hinsen2 ... ok
compiling ewing9 ... ok
compiling longunsigned ... ok
compiling extinheritset ... ok
compiling ewing7 ... ok
compiling ishimoto4 ... ishimoto4.c:110: warning: ?__pyx_f_9ishimoto4_f? defined but not used
ok
compiling khavkine1 ... khavkine1.c:126: warning: ?__pyx_f_9khavkine1_f? defined but not used
ok
compiling a_capi ... ok
compiling ctypedefunion ... ok
compiling extpropertydel ... ok
compiling withnogil ... withnogil.c:113: warning: ?__pyx_f_9withnogil_f? defined but not used
withnogil.c:145: warning: ?__pyx_f_9withnogil_g? defined but not used
ok
compiling extimported ... ok
compiling ctypedefstruct ... ok
compiling constexpr ... ok
compiling ewing8 ... ok
compiling signedtypes ... ok
compiling nonctypedefclass ... ok
compiling cdefexternfromstar ... ok
compiling complexbasetype ... complexbasetype.c:115: warning: ?__pyx_f_15complexbasetype_brian? defined but not used
ok
compiling classmethargdefault ... ok
compiling cforfromloop ... ok
compiling extpropertyset ... ok
compiling ewing3 ... ok
compiling behnel4 ... behnel4.c:123: warning: ?__pyx_f_7behnel4_f? defined but not used
ok
compiling specmethdocstring ... specmethdocstring.c:131: warning: ?__pyx_doc_17specmethdocstring_1C___init__? defined but not used
specmethdocstring.c:155: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___get__? defined but not used
specmethdocstring.c:171: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___set__? defined but not used
ok
compiling extforward ... ok
compiling cverylongtypes ... ok
compiling argdefault ... ok
compiling nullptr ... ok
compiling extinheritdel ... ok

======================================================================
FAIL: Doctest: new_style_exceptions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.4/doctest.py", line 2157, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for new_style_exceptions
  File "/home/kirr/src/tools/cython/cython-devel/BUILD/new_style_exceptions.so", line unknown line number, in new_style_exceptions

----------------------------------------------------------------------
File "/home/kirr/src/tools/cython/cython-devel/BUILD/new_style_exceptions.so", line ?, in new_style_exceptions
Failed example:
    test(Exception('hi'))
Expected:
    Raising: Exception('hi',)
    Caught: <type 'exceptions.Exception'> Exception('hi',)
Got:
    Raising: <exceptions.Exception instance at 0xb78ad84c>
    Caught: <class exceptions.Exception at 0xb7d8c26c> <exceptions.Exception instance at 0xb78ad84c>


======================================================================
FAIL: Doctest: tuplereassign
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.4/doctest.py", line 2157, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for tuplereassign
  File "/home/kirr/src/tools/cython/cython-devel/BUILD/tuplereassign.so", line 23, in tuplereassign

----------------------------------------------------------------------
File "/home/kirr/src/tools/cython/cython-devel/BUILD/tuplereassign.so", line 31, in tuplereassign
Failed example:
    testnonsense()
Expected:
    Traceback (most recent call last):
    TypeError: 'int' object is not iterable
Got:
    Traceback (most recent call last):
      File "/usr/lib/python2.4/doctest.py", line 1248, in __run
        compileflags, 1) in test.globs
      File "<doctest tuplereassign[3]>", line 1, in ?
        testnonsense()
      File "tuplereassign.pyx", line 26, in tuplereassign.testnonsense (tuplereassign.c:%u)
    TypeError: iteration over non-sequence


----------------------------------------------------------------------
Ran 213 tests in 289.050s

FAILED (failures=2)
Creating lexicon...
Done (0.21 seconds)
Pickling lexicon...
Done (0.02 seconds)

From stefan_ml at behnel.de  Fri May  2 09:13:10 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 02 May 2008 09:13:10 +0200
Subject: [Cython] Cython 0.9.6.14 release candidate
In-Reply-To: <20080502055844.GA10755@evo>
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>	<fvcb5t$bs1$1@ger.gmane.org>
	<4819F55E.5010909@behnel.de>	<fvctcc$9ed$1@ger.gmane.org>
	<4819FBF7.9030901@behnel.de>	<fvd0eb$kbh$2@ger.gmane.org>
	<481A079E.8030705@behnel.de>	<fvd36i$tsb$1@ger.gmane.org>
	<20080502055844.GA10755@evo>
Message-ID: <481ABF06.7020707@behnel.de>

Hi,

Kirill Smelkov wrote:
> I still have 3 failures with latest cython-devel (b869698d6f22)
> Attached is a log of 'python runtests.py' run on Debian testing.

Thanks. That's running against Python 2.4, while the two tests were depending
on Python 2.5 specifics (an exception message and new style exception types).
I relaxed them a bit to run on Py2.3-2.5.

The test runner script also prints the versions of Cython and Python now, to
put some more context into the test log.

Stefan


From kirr at landau.phys.spbu.ru  Fri May  2 09:46:38 2008
From: kirr at landau.phys.spbu.ru (Kirill Smelkov)
Date: Fri, 2 May 2008 11:46:38 +0400
Subject: [Cython] Cython 0.9.6.14 release candidate
In-Reply-To: <481ABF06.7020707@behnel.de>
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>
	<fvcb5t$bs1$1@ger.gmane.org> <4819F55E.5010909@behnel.de>
	<fvctcc$9ed$1@ger.gmane.org> <4819FBF7.9030901@behnel.de>
	<fvd0eb$kbh$2@ger.gmane.org> <481A079E.8030705@behnel.de>
	<fvd36i$tsb$1@ger.gmane.org> <20080502055844.GA10755@evo>
	<481ABF06.7020707@behnel.de>
Message-ID: <20080502074637.GB10755@evo>

Hi Stefan,

On Fri, May 02, 2008 at 09:13:10AM +0200, Stefan Behnel wrote:
> Hi,
> 
> Kirill Smelkov wrote:
> > I still have 3 failures with latest cython-devel (b869698d6f22)
> > Attached is a log of 'python runtests.py' run on Debian testing.
> 
> Thanks. That's running against Python 2.4, while the two tests were depending
> on Python 2.5 specifics (an exception message and new style exception types).
> I relaxed them a bit to run on Py2.3-2.5.
> 
> The test runner script also prints the versions of Cython and Python now, to
> put some more context into the test log.

kirr at evo:~/src/tools/cython/cython-devel$ python runtests.py 
Running tests against Cython 0.9.6.13.1
Python 2.4.5 (#2, Mar 12 2008, 00:15:51) 
[GCC 4.2.3 (Debian 4.2.3-2)]

...

Ran 213 tests in 243.547s

OK

----

Thanks!

-- 
    ????? ????????, ??????.

From robertwb at math.washington.edu  Fri May  2 10:38:42 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 2 May 2008 01:38:42 -0700
Subject: [Cython] Cython 0.9.6.14 released
Message-ID: <463FD591-1904-4CD0-A67B-0EBB62AB6A4E@math.washington.edu>

Cython 0.9.6.14 is out. This is mostly a bugfix release, however  
there are several other improvements, notably:

- Source code encoding support (PEP 263) and UTF-8 default source  
encoding (PEP 3120) (Stefan Behnel)
- New command line option -w to change the working directory when  
running Cython (Gary Furnish)
- L.append(x) now optimized if L a (runtime) list (Robert Bradshaw)
- Cdef variables may be declared python builtin types (CEP 507),  
though there is much more potential for optimization (Robert Bradshaw)
- Enums declared "public" will get exported to the (python- 
accessible) module namespace (Robert Bradshaw)
- Correct special float values (Christian Heimes/Stefan Behnel)

Thanks also to all those who submitted good bug reports as well.

- Robert


From robertwb at math.washington.edu  Fri May  2 10:40:11 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 2 May 2008 01:40:11 -0700
Subject: [Cython] Cython 0.9.6.14 release candidate
In-Reply-To: <20080502074637.GB10755@evo>
References: <AB81C4C8-C77B-49C2-BE77-E2483811ADAD@math.washington.edu>
	<fvcb5t$bs1$1@ger.gmane.org> <4819F55E.5010909@behnel.de>
	<fvctcc$9ed$1@ger.gmane.org> <4819FBF7.9030901@behnel.de>
	<fvd0eb$kbh$2@ger.gmane.org> <481A079E.8030705@behnel.de>
	<fvd36i$tsb$1@ger.gmane.org> <20080502055844.GA10755@evo>
	<481ABF06.7020707@behnel.de> <20080502074637.GB10755@evo>
Message-ID: <CE708A39-DD18-405C-A8DB-A70AA0C46ABC@math.washington.edu>

On May 2, 2008, at 12:46 AM, Kirill Smelkov wrote:

> Hi Stefan,
>
> On Fri, May 02, 2008 at 09:13:10AM +0200, Stefan Behnel wrote:
>> Hi,
>>
>> Kirill Smelkov wrote:
>>> I still have 3 failures with latest cython-devel (b869698d6f22)
>>> Attached is a log of 'python runtests.py' run on Debian testing.
>>
>> Thanks. That's running against Python 2.4, while the two tests  
>> were depending
>> on Python 2.5 specifics (an exception message and new style  
>> exception types).
>> I relaxed them a bit to run on Py2.3-2.5.
>>
>> The test runner script also prints the versions of Cython and  
>> Python now, to
>> put some more context into the test log.
>
> kirr at evo:~/src/tools/cython/cython-devel$ python runtests.py
> Running tests against Cython 0.9.6.13.1
> Python 2.4.5 (#2, Mar 12 2008, 00:15:51)
> [GCC 4.2.3 (Debian 4.2.3-2)]
>
> ...
>
> Ran 213 tests in 243.547s
>
> OK
>
> ----
>
> Thanks!

Thanks for fixing this. It didn't make it into Cython 0.9.6.14 (which  
I pushed out this morning) but at least it's a bug in the testing  
code rather than in the compiler itself, and these changes will  
certainly be in the next release.

- Robert


From dagss at student.matnat.uio.no  Fri May  2 13:22:35 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 02 May 2008 13:22:35 +0200
Subject: [Cython] Some small phase refactorings
Message-ID: <481AF97B.8030305@student.matnat.uio.no>

This isn't much, but it is a start.

Phase refactoring seems like it will become a bit difficult and I think 
it is better left for dev1 if possible; I need to work with someone who 
know the source more intimately. It appears that some parts must almost 
be "rebuilt" or at least control flow heavily altered between 
statements, it is all about keeping the changes to a minimum... the good 
news is that a certain amount of spaghetti will disappear through the 
process (you know -- the things that break in a different part of the 
code when you change another part in the code; while not necesarrily the 
root of all evil, it does make it harder to get to know a codebase).

But I have patch for some trivial stuff at the "top" of the call chain 
that at least is a good point for discussion (if somebody else cares how 
this is done). It can be found here:

http://wiki.cython.org/DagSverreSeljebotn/patches?action=AttachFile&do=get&target=phaserefactoring1.diff

and succeeds the testcases; I consider it ready for inclusion.

Basically, what it does is turn this (psuedo-code describing overall 
structure, you won't find this anywhere):

for each function in module:
  construct_function_scope
  analyse_control_flow
  analyse_declarations
  analyse_expressions
  generate_code

into

for each function in module: construct_function_scope
for each function in module: analyse_control_flow
for each function in module: analyse_declarations
for each function in module: analyse_expressions
for each function in module: generate_code

However, each of these are in turn run as recursive calls just like 
before. In particular, analyse_expressions is still one phase and not 
easily seperateable (it should be split in three I think: analyse types, 
coercion, allocate temps).

Also, these are all "function-level"; on module-level things happen like 
before, i.e. analyse_declarations in the module level is run at another 
time (and for the nearest future it should remain this way; just 
consider module-level and function-level analysis different phases).

The new code is implemented using visitor transforms. One could argue 
about this, but I do think it leads to pretty neat code. The third 
for-loop in the psuedo-code above is this implemented like this:

Cython/Compiler/Transforms/Analysis.py [is this source structure ok?]:

class AnalyseFunctionBodyDeclarations(VisitorTransform):
    def pre_FuncDefNode(self, node):
        node.body.analyse_declarations(node.scope)
        return False # do not recurse beyond first function level

Cython/Compiler/ModuleNode.py:

class ModuleNode:
    def generate_c_code(self, env, options, result):
        ...
        import Cython.Compiler.Transforms.Analysis as Analysis
        ...
        Analysis.AnalyseFunctionBodyDeclarations()(self, env=env)
        ...
        self.body.generate_function_definitions(env, code, 
options.transforms)
        ...


The only non-trivial part is that the function scope is constructed by a 
tree transform that adds the "scope" and "scopenode" attributes in the 
following way: The FuncDefNodes and ModuleNodes gets their "scope" 
attribute set to the object that was previously passed around everywhere 
as the "env" parameter (this is then read back in order to pass the 
"env" correctly). All nodes also have their "scopenode" attribute set to 
the node that caused creation of the enclosing scope. I.e., rather than 
relying on a passed in env passed recursively from the function/module, 
it is possible for nodes to look up self.scopenode.scope. This will aid 
seperation of phases.

-- 
Dag Sverre


From dagss at student.matnat.uio.no  Fri May  2 13:38:53 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 02 May 2008 13:38:53 +0200
Subject: [Cython] Visitor patterns and Python 2.4
Message-ID: <481AFD4D.4050401@student.matnat.uio.no>

Are there any real reasons for leaving the Cython compiler (not talking 
about generated or supported code of course) at Python 2.3, rather than 
a small bump to 2.4? Reason: I'd like decorators.

The rationale: Notice that parse tree visitors can currently be written 
like this:

class AnalyseControlFlow(VisitorTransform):
    def pre_FuncDefNode(self, node):
        node.body.analyse_control_flow(node.scope)
        return False # do not recurse beyond first function level


The VisitorTransform parent implements a tree iteration/recursion and 
calls "pre_X" which is only allowed to visit and stop recursion at that 
point, "process_X" which has full control (can replace or remove node, 
do various infix processing -- if "process_X" is implemented, pre and 
post are not called) and "post_X" (which only visits)).

VisitorTransform basically looks up node.__class__.__name__ to figure 
out the name of the function to call, and uses introspection. Things are 
cached.

There are two disadvantages to this though:
- Relying on node.__class__.__name__ is a bit fragile. Multiple; and 
simply changing a class. It would be better to use a real reference

- If using multiple instances of a Transform class, each instance have 
to rebuild their call table cache. (This is not a real issue though, I 
don't expect such cases to come up.) Using a metaclass, the call table 
could however be done at class definition time which is "nicer". This is 
not a good reason.

If bumping the version to 2.4, decorators can solve this. OTOH, I'd much 
rather do with function names rather than "manually calling decorators". 
The example above could then look like this:

class AnalyseControlFlow(VisitorTransform):
    @pre(FuncDefNode)
    def call_analyse_control_flow(self, node):
        node.body.analyse_control_flow(node.scope)
        return False # do not recurse beyond first function level

This has the advantage that the class is directly references rather than 
being embedded in a string, so that name clashes, class renaming etc 
works correctly. Like this:

from Nodes import FuncDefNode
funcdef = FuncDefNode

class AnalyseControlFlow(VisitorTransform):
    @pre(funcdef)
    def call_analyse_control_flow(self, node):
        ...

Note that the standard visitor pattern (implementing accept on each and 
every node) was shot down by Fabrizio earlier (in private communication) 
as wasting code lines and unpythonic, and I guess I agree, so I 
implemented his suggestion instead. (The only reason I can see is speed 
when running under Cython compilation, if one uses "cdef" and 
compilation binding so that a dictionary lookups happen anyway. Not the 
thing to worry about at this stage IMO.)

-- 
Dag Sverre


From stefan_ml at behnel.de  Sat May  3 07:53:30 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 03 May 2008 07:53:30 +0200
Subject: [Cython] Some small phase refactorings
In-Reply-To: <481AF97B.8030305@student.matnat.uio.no>
References: <481AF97B.8030305@student.matnat.uio.no>
Message-ID: <481BFDDA.8060007@behnel.de>

Hi,

since no-one replied so far (and since I think public code-review is
important), here I go. Lines starting with ### are mine.

In general, I'm +0 for the change and -0 for the patch. I think, if we use
transformations, there should be a good way to subscribe them to what they
work on. We brought up the idea of a path language, but maybe a subscription
to an AST class and just calling the transform to let it return either the
unchanged input or a modified AST subtree might already be enough.


diff -r 3c924a0594ba Cython/Compiler/ModuleNode.py
--- a/Cython/Compiler/ModuleNode.py	Fri May 02 10:22:20 2008 +0200
+++ b/Cython/Compiler/ModuleNode.py	Fri May 02 11:42:17 2008 +0200
@@ -33,7 +33,7 @@ class ModuleNode(Nodes.Node, Nodes.Block
     #  module_temp_cname    string
     #  full_module_name     string

-    children_attrs = ["body"]
+    child_attrs = ["body"]

     def analyse_declarations(self, env):
         if Options.embed_pos_in_docstring:

### This looks like a separate fix to me.


+        import Cython.Compiler.Transforms.Analysis as Analysis
+        Analysis.CreateFunctionScope()(self, env=env)
+        Analysis.AnalyseControlFlow()(self, env=env)
+        Analysis.AnalyseFunctionBodyDeclarations()(self, env=env)
+        Analysis.AnalyseFunctionBodyExpressions()(self, env=env)
+        options.transforms.run('after_function_analysis', self, global_env=env)
+

### These look like functions, they should follow PEP 8 naming. Then again,
### why aren't they functions?


+    def get_visitfunc(self, prefix, cls):
+        mname = prefix + cls.__name__
+        m = self.visitmethods[prefix].get(mname)
+        if m is None:
+            # Must resolve, try entire hierarchy
+            for cls in inspect.getmro(cls):
+                m = getattr(self, prefix + cls.__name__, None)
+                if m is not None:
+                    break
+            if m is None: raise RuntimeError("Not a Node descendant: " + node)
+            self.visitmethods[prefix][mname] = m
+        return m

### I'm not convinced of this one. I understand why you do it, but I believe
### that using the class itself rather than dispatching on a string would help
### here (I saw your decorator proposal - still thinking about it, but might
### be worth doing).


@@ -110,7 +174,6 @@ class TransformSet(dict):
     def run(self, name, node, **options):
         assert name in self
         for transform in self[name]:
-            transform.initialize(phase=name, **options)
-            transform.process_node(node, "(root)")
+            transform(node, phase=name, **options)

### I like this part.


From stefan_ml at behnel.de  Sat May  3 07:56:42 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 03 May 2008 07:56:42 +0200
Subject: [Cython] Visitor patterns and Python 2.4
In-Reply-To: <481AFD4D.4050401@student.matnat.uio.no>
References: <481AFD4D.4050401@student.matnat.uio.no>
Message-ID: <481BFE9A.4080600@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Are there any real reasons for leaving the Cython compiler (not talking 
> about generated or supported code of course) at Python 2.3, rather than 
> a small bump to 2.4? Reason: I'd like decorators.
> 
> The rationale: Notice that parse tree visitors can currently be written 
> like this:
> 
> class AnalyseControlFlow(VisitorTransform):
>     def pre_FuncDefNode(self, node):
>         node.body.analyse_control_flow(node.scope)
>         return False # do not recurse beyond first function level

If this way of doing it is accepted, I'm actually for accepting 2.4 code. I
don't know any important platform that doesn't come with at least Py2.4. And
note that most people won't even need to run Cython themselves, even if they
use software implemented in Cython.

Stefan

From dagss at student.matnat.uio.no  Sat May  3 10:27:16 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 3 May 2008 10:27:16 +0200 (CEST)
Subject: [Cython] Some small phase refactorings
In-Reply-To: <481BFDDA.8060007@behnel.de>
References: <481AF97B.8030305@student.matnat.uio.no>
	<481BFDDA.8060007@behnel.de>
Message-ID: <1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>

Thanks for the feedback.

> -    children_attrs = ["body"]
> +    child_attrs = ["body"]
>
>
> ### This looks like a separate fix to me.

Indeed (sorry).

>
> +        import Cython.Compiler.Transforms.Analysis as Analysis
> +        Analysis.CreateFunctionScope()(self, env=env)
> +        Analysis.AnalyseControlFlow()(self, env=env)
> +        Analysis.AnalyseFunctionBodyDeclarations()(self, env=env)
> +        Analysis.AnalyseFunctionBodyExpressions()(self, env=env)
> +        options.transforms.run('after_function_analysis', self,
> global_env=env)
> +
>
> ### These look like functions, they should follow PEP 8 naming. Then
> again,
> ### why aren't they functions?

Note that this code will almost certainly be moved again and rewritten at
some later point (they can't really belong to ModuleNode); but more
refactoring must happen before they can be moved to their proper location
and they serve a good purpose where they are for now though.

They have to be classes as they are transform objects with member methods,
some state etc. etc. But thinking about it one could have functions like
this in Analysis.py:

def analyse_function_body_declarations(tree, **opts):
    return AnalyseFunctionBodyDeclarations()(tree, **opts)

if that helps. The reason I started using the __call__ is that I think in
time one can treat these as functions, like this:

pipeline = [
   f,
   g,
   AnalyseFunctionBodyDeclarations(),
   Coercions(),
]
for transform in pipeline:
   tree = transform(tree)

(ie, basically saying that "pipeline = f <ring> g <ring> h"...)


> ### I'm not convinced of this one. I understand why you do it, but I
> believe
> ### that using the class itself rather than dispatching on a string would
> help
> ### here (I saw your decorator proposal - still thinking about it, but
> might
> ### be worth doing).

If not, then the classical visitor pattern might put you at ease?:

class FuncDefNode:
    def accept(self, visitor):
        visitor.visit_FuncDefNode(self)

However, that's a lot of extra trivial code to add (this would have to be
added to _all_ classes), and when one is using Python anyway I'd like to
avoid pretending I'm writing Java... :-)

I hope we can start using Python 2.4, then I'll implement a
decorator/metaclass solution instead.


In conclusion, I'd like to mention that I really think the important thing
here is to consider the "grand, large-scale" features of the patch. I
didn't polish the details in any way, because I think that what is
important here is the changes they make possible in the application
structure. How the visitors look like can be changed entirely in
Visitor.py and Analysis.py without interfering with existing code; while
the phase refactoring is going to intrude everywhere and make changes all
over the place, so the form the phase refactoring will take is the
important point.

(OTOH; I guess it is a good time to think about the details as well so
that when the 1000 line coercion refactoring patch should be written one
knows what to do in the details...)

(I've asked myself as to whether it is all worth it BTW. It is a heavy and
non-fun task. But I'm still convinced there's absolutely no way around it
if the feature-set of Cython is to grow significantly in any way. And
realistically, it can be done in two or three days with for instance me,
you and Robert working together... So this might be too early to talk
about it; but I end up working on it anyway because it is effectively a
blocker for me and I cannot get anywhere with my GSoC stuff until it is
done :-) )

Dag Sverre


From kirr at mns.spb.ru  Sat May  3 18:21:36 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sat, 03 May 2008 20:21:36 +0400
Subject: [Cython] [PATCH 2 of 3] Fix cimport example in "Sharing
	Declarations" section
In-Reply-To: <patchbomb.1209831694@tugrik2.mns.mnsspb.ru>
Message-ID: <a81645475dc33f1916f6.1209831696@tugrik2.mns.mnsspb.ru>

# HG changeset patch
# User Kirill Smelkov <kirr at mns.spb.ru>
# Date 1209831534 -14400
# Node ID a81645475dc33f1916f6dc7ee5ec8f52141b20e8
# Parent  0745e2b1f904c2ab8b7c70218d59bd933cbbe3ab
Fix cimport example in "Sharing Declarations" section

- we need to define struct variables with 'cdef'
- there is no -> in Cython/Pyrex

diff --git a/docs/sharing_declarations.rst b/docs/sharing_declarations.rst
--- a/docs/sharing_declarations.rst
+++ b/docs/sharing_declarations.rst
@@ -82,9 +82,9 @@ uses it.
         d.filler = dishes.sausage
 
     def serve():
-        spamdish d
+        cdef spamdish d
         prepare(&d)
-        print "%d oz spam, filler no. %d" % (d->oz_of_spam, d->otherstuff)
+        print "%d oz spam, filler no. %d" % (d.oz_of_spam, d.otherstuff)
                                
 It is important to understand that the :keyword:`cimport` statement can only
 be used to import C data types, C functions and variables, and extension

From kirr at mns.spb.ru  Sat May  3 18:21:37 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sat, 03 May 2008 20:21:37 +0400
Subject: [Cython] [PATCH 3 of 3] Reminder to put the table about styles of
	struct, union and enum declarations
In-Reply-To: <patchbomb.1209831694@tugrik2.mns.mnsspb.ru>
Message-ID: <630bbffdd62e0e82c551.1209831697@tugrik2.mns.mnsspb.ru>

# HG changeset patch
# User Kirill Smelkov <kirr at mns.spb.ru>
# Date 1209831643 -14400
# Node ID 630bbffdd62e0e82c551334cbf4579b751859248
# Parent  a81645475dc33f1916f6dc7ee5ec8f52141b20e8
Reminder to put the table about styles of struct, union and enum declarations

diff --git a/docs/external_C_code.rst b/docs/external_C_code.rst
--- a/docs/external_C_code.rst
+++ b/docs/external_C_code.rst
@@ -158,6 +158,8 @@ header file, and the corresponding Cytho
 header file, and the corresponding Cython declaration that you should put in
 the ``cdef extern`` from block. Struct declarations are used as an example; the
 same applies equally to union and enum declarations.
+
+**TODO** put table here...
 
 Note that in all the cases below, you refer to the type in Cython code simply
 as :ctype:`Foo`, not ``struct Foo``.

From kirr at mns.spb.ru  Sat May  3 18:21:35 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sat, 03 May 2008 20:21:35 +0400
Subject: [Cython] [PATCH 1 of 3] Fix typos in documentation here and there
In-Reply-To: <patchbomb.1209831694@tugrik2.mns.mnsspb.ru>
Message-ID: <0745e2b1f904c2ab8b7c.1209831695@tugrik2.mns.mnsspb.ru>

# HG changeset patch
# User Kirill Smelkov <kirr at mns.spb.ru>
# Date 1209831403 -14400
# Node ID 0745e2b1f904c2ab8b7c70218d59bd933cbbe3ab
# Parent  bc0d87c4a2953e32beed52bdb461bba99a7adda5
Fix typos in documentation here and there

diff --git a/docs/extension_types.rst b/docs/extension_types.rst
--- a/docs/extension_types.rst
+++ b/docs/extension_types.rst
@@ -209,25 +209,26 @@ time it is written to, returns the list 
 time it is written to, returns the list when it is read, and empties the list
 when it is deleted.::
  
-    #cheesy.pyx 	Test input
+    # cheesy.pyx
     cdef class CheeseShop:
 
-    cdef object cheeses
+        cdef object cheeses
 
-    def __cinit__(self):
-        self.cheeses = []
+        def __cinit__(self):
+            self.cheeses = []
 
-    property cheese:
+        property cheese:
 
-        def __get__(self):
-            return "We don't have: %s" % self.cheeses
+            def __get__(self):
+                return "We don't have: %s" % self.cheeses
 
-        def __set__(self, value):
-            self.cheeses.append(value)
+            def __set__(self, value):
+                self.cheeses.append(value)
 
-        def __del__(self):
-            del self.cheeses[:]
+            def __del__(self):
+                del self.cheeses[:]
 
+    # Test input
     from cheesy import CheeseShop
 
     shop = CheeseShop()
@@ -242,7 +243,7 @@ when it is deleted.::
     del shop.cheese
     print shop.cheese
 
-    #Test output
+    # Test output
     We don't have: []
     We don't have: ['camembert']
     We don't have: ['camembert', 'cheddar']
@@ -280,18 +281,17 @@ functions, C methods are declared using 
 :keyword:`def`. C methods are "virtual", and may be overridden in derived
 extension types.::
 
-    pets.pyx
-            Output
+    # pets.pyx
     cdef class Parrot:
 
-    cdef void describe(self):
-        print "This parrot is resting."
+        cdef void describe(self):
+            print "This parrot is resting."
 
     cdef class Norwegian(Parrot):
 
-    cdef void describe(self):
-        Parrot.describe(self)
-        print "Lovely plumage!"
+        cdef void describe(self):
+            Parrot.describe(self)
+            print "Lovely plumage!"
 
 
     cdef Parrot p1, p2
@@ -301,7 +301,9 @@ extension types.::
     p1.describe()
     print "p2:"
     p2.describe()
-            p1:
+
+    # Output
+    p1:
     This parrot is resting.
     p2:
     This parrot is resting.
diff --git a/docs/external_C_code.rst b/docs/external_C_code.rst
--- a/docs/external_C_code.rst
+++ b/docs/external_C_code.rst
@@ -293,18 +293,16 @@ Any public C type or extension type decl
 Any public C type or extension type declarations in the Cython module are also
 made available when you include :file:`modulename_api.h`.::
 
-    delorean.pyx
-
-    marty.c
-
+    # delorean.pyx
     cdef public struct Vehicle:
-    int speed
-    float power
+        int speed
+        float power
 
     cdef api void activate(Vehicle *v):
         if v.speed >= 88 and v.power >= 1.21:
             print "Time travel achieved"
 
+    # marty.c
     #include "delorean_api.h"
 
     Vehicle car;
diff --git a/docs/language_basics.rst b/docs/language_basics.rst
--- a/docs/language_basics.rst
+++ b/docs/language_basics.rst
@@ -327,7 +327,7 @@ Error return values
 Error return values
 -------------------
 
-If you don't do anything special, a function declared with :keyword`cdef` that
+If you don't do anything special, a function declared with :keyword:`cdef` that
 does not return a Python object has no way of reporting Python exceptions to
 its caller. If an exception is detected in such a function, a warning message
 is printed and the exception is ignored.
@@ -520,7 +520,7 @@ statements, combined using any of the Py
 statements, combined using any of the Python expression syntax.
 
 The following compile-time names are predefined, corresponding to the values
-returned by :func:``os.uname``.
+returned by :func:`os.uname`.
 
     UNAME_SYSNAME, UNAME_NODENAME, UNAME_RELEASE,
     UNAME_VERSION, UNAME_MACHINE
diff --git a/docs/pyrex_differences.rst b/docs/pyrex_differences.rst
--- a/docs/pyrex_differences.rst
+++ b/docs/pyrex_differences.rst
@@ -108,7 +108,7 @@ python. One can declare variables and re
 :ctype:`bint` type.  For example::
 
     cdef int i = x
-    Cdef bint b = x
+    cdef bint b = x
 
 The first conversion would happen via ``x.__int__()`` whereas the second would
 happen via ``x.__nonzero__()``. (Actually, if ``x`` is the python object
@@ -156,7 +156,7 @@ method on the class directly, e.g.::
 
     cdef class A:
         cpdef foo(self):
-        pass
+            pass
 
     x = A()
     x.foo()  # will check to see if overridden
@@ -197,10 +197,10 @@ It does not stop one from casting where 
 It does not stop one from casting where there is no conversion (though it will
 emit a warning). If one really wants the address, cast to a ``void *`` first.
 
-As in Pyrex ``<MyExtensionType>x`` will cast ``x`` to type <ctype>`MyExtensionType` without any
+As in Pyrex ``<MyExtensionType>x`` will cast ``x`` to type :ctype:`MyExtensionType` without any
 type checking. Cython supports the syntax ``<MyExtensionType?>`` to do the cast
 with type checking (i.e. it will throw an error if ``x`` is not a (subclass of)
-<ctype>`MyExtensionType`. 
+:ctype:`MyExtensionType`. 
 
 Optional arguments in cdef/cpdef functions
 ------------------------------------------
diff --git a/docs/sharing_declarations.rst b/docs/sharing_declarations.rst
--- a/docs/sharing_declarations.rst
+++ b/docs/sharing_declarations.rst
@@ -16,7 +16,12 @@ modules, and an implementation file with
 modules, and an implementation file with a ``.pyx`` suffix, containing
 everything else. When a module wants to use something declared in another
 module's definition file, it imports it using the :keyword:`cimport`
-statement.  What a Definition File contains A definition file can contain:
+statement.
+
+What a Definition File contains
+-------------------------------
+
+A definition file can contain:
 
 * Any kind of C type declaration.
 * extern C function or variable declarations.
@@ -139,11 +144,11 @@ C functions defined at the top level of 
 :keyword:`cimport` by putting headers for them in the ``.pxd`` file, for
 example,:
 
-:file:`volume.pxd`:
+:file:`volume.pxd`::
+
+    cdef float cube(float)
 
 :file:`spammery.pyx`::
-
-    cdef float cube(float)
 
     from volume cimport cube
 
@@ -185,17 +190,22 @@ Here is an example of a module which def
 Here is an example of a module which defines and exports an extension type,
 and another module which uses it.::
  
-    Shrubbing.pxd 	Shrubbing.pyx
+    # Shrubbing.pxd
     cdef class Shrubbery:
         cdef int width
-        cdef int length 	cdef class Shrubbery:
+        cdef int length
+        
+    # Shrubbing.pyx
+    cdef class Shrubbery:
         def __new__(self, int w, int l):
             self.width = w
             self.length = l
 
     def standard_shrubbery():
         return Shrubbery(3, 7)
-    Landscaping.pyx
+
+
+    # Landscaping.pyx
     cimport Shrubbing
     import Shrubbing
 

From kirr at mns.spb.ru  Sat May  3 18:21:34 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sat, 03 May 2008 20:21:34 +0400
Subject: [Cython] [PATCH 0 of 3] Various fixes to documentation
Message-ID: <patchbomb.1209831694@tugrik2.mns.mnsspb.ru>

Hi,

While studying (again!) Cython/Pyrex, I've fixed few bits in the documentation.

Three patches attached, please apply.

Thanks!

From ggellner at uoguelph.ca  Sat May  3 19:53:48 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Sat, 3 May 2008 13:53:48 -0400
Subject: [Cython] [PATCH 0 of 3] Various fixes to documentation
In-Reply-To: <patchbomb.1209831694@tugrik2.mns.mnsspb.ru>
References: <patchbomb.1209831694@tugrik2.mns.mnsspb.ru>
Message-ID: <20080503175348.GA13881@basestar>

On Sat, May 03, 2008 at 08:21:34PM +0400, Kirill Smelkov wrote:
> Hi,
> 
> While studying (again!) Cython/Pyrex, I've fixed few bits in the documentation.
> 
> Three patches attached, please apply.
> 
Thanks for all the changes! I have applied them.

For future reference the source is now at:
http://hg.cython.org/cython-docs
(I have not pushed the new file upstream yet, but I will do it as soon as I
can. You can see the changes in the website and pdf).

Gabriel

From pete at petertodd.org  Sun May  4 07:15:32 2008
From: pete at petertodd.org (Peter Todd)
Date: Sun, 4 May 2008 01:15:32 -0400
Subject: [Cython] __getattribute__
In-Reply-To: <5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
Message-ID: <20080504051532.GA7720@tilt>

On Wed, Apr 30, 2008 at 06:17:31PM -0700, Robert Bradshaw wrote:
> On Apr 29, 2008, at 5:42 PM, Peter Todd wrote:
> 
> >Is there a __getattribute__ work-alike in Cython?
> >
> >Essentially I need direct control over an objects tp_getattro and
> >tp_setattro slots to implement a wrapper class. Specificly
> >wrapped.__class__ should go to the wrapped objects class attribute,  
> >not
> >the wrapping objects __class__ attribute.
> >
> >__getattr__ outputs C-source that includes a call to
> >PyObject_GenericGetAttr first, and won't run my code if that call
> >succeeds.
> >
> >Thanks,
> >
> >Peter
> 
> Not that I'm aware of, though I'd imagine that implementing  
> __getattribute__ (if it exists) as being called at the top of this  
> function would be fairly easy to do. One would want to match Python  
> symantics exactly.

Here's my first patch. This correctly implements __getattribute__ and
__getattr__ in the single class case. FWIW I also have a mercurial tree
if it'd be better to pull from it than apply patches.

I'm working on making subclasses behave correctly, I've got test cases
written up showing where things fail, but no solutions to that written
yet. The slot_tp_getattro stuff Stefan mentioned is useful though.

-- 
http://petertodd.org 'peter'[:-1]@petertodd.org 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: __getattribute__.patch
Type: text/x-diff
Size: 4799 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080504/376ae4c6/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080504/376ae4c6/attachment.pgp 

From kirr at mns.spb.ru  Sun May  4 10:20:56 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 04 May 2008 12:20:56 +0400
Subject: [Cython] [PATCH 2 of 2] Initial .hgignore
In-Reply-To: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
Message-ID: <aaa56370d9fb26ca99e2.1209889256@tugrik2.mns.mnsspb.ru>

# HG changeset patch
# User Kirill Smelkov <kirr at mns.spb.ru>
# Date 1209814507 -14400
# Node ID aaa56370d9fb26ca99e2aa774e373b7383ffd685
# Parent  2fe2f4f4e58972b2a3eb106964cd68db12d88e0e
Initial .hgignore

We ignore *.pyc, vim swap files, build results under BUILD/ and Lexicon.pickle

This is handy, because otherwise, say after runtest.py run, 'hg status' shows
lots of unrelated info, thus lowering signal-to-noise ratio.

diff --git a/.hgignore b/.hgignore
new file mode 100644
--- /dev/null
+++ b/.hgignore
@@ -0,0 +1,8 @@
+syntax: glob
+
+*.pyc
+*.swp
+
+Cython/Compiler/Lexicon.pickle
+BUILD/
+

From kirr at mns.spb.ru  Sun May  4 10:20:54 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 04 May 2008 12:20:54 +0400
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using cython
	in-tree
Message-ID: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>

Hi,

Here is a couple of handy patches for the case when cython is used "in-tree",
i.e. when no installation is done.

- First we need to 'chmod a+x' relevant files in bin/ ,
- and second, it would be convenient to finally create reasonable .hgignore

Please apply.

From kirr at mns.spb.ru  Sun May  4 10:20:55 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 04 May 2008 12:20:55 +0400
Subject: [Cython] [PATCH 1 of 2] chmod a+x bin/*
In-Reply-To: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
Message-ID: <2fe2f4f4e58972b2a3eb.1209889255@tugrik2.mns.mnsspb.ru>

# HG changeset patch
# User Kirill Smelkov <kirr at mns.spb.ru>
# Date 1209814181 -14400
# Node ID 2fe2f4f4e58972b2a3eb106964cd68db12d88e0e
# Parent  3c924a0594baee13d595e66de8577fcd1691ee7c
chmod a+x bin/*

This is handy when using cython right from development tree

diff --git a/bin/cython b/bin/cython
old mode 100644
new mode 100755
diff --git a/bin/update_references b/bin/update_references
old mode 100644
new mode 100755

From kirr at mns.spb.ru  Sun May  4 11:12:23 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 4 May 2008 13:12:23 +0400
Subject: [Cython] [PATCH 0 of 3] Various fixes to documentation
In-Reply-To: <20080503175348.GA13881@basestar>
References: <patchbomb.1209831694@tugrik2.mns.mnsspb.ru>
	<20080503175348.GA13881@basestar>
Message-ID: <200805041312.23099.kirr@mns.spb.ru>

? ????????? ?? ??????? 03 ??? 2008 Gabriel Gellner ???????(a):
> On Sat, May 03, 2008 at 08:21:34PM +0400, Kirill Smelkov wrote:
> > Hi,
> > 
> > While studying (again!) Cython/Pyrex, I've fixed few bits in the documentation.
> > 
> > Three patches attached, please apply.
> > 
> Thanks for all the changes! I have applied them.

Thanks!

Kirill.

From stefan_ml at behnel.de  Sun May  4 11:18:27 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 04 May 2008 11:18:27 +0200
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
 cython	in-tree
In-Reply-To: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
Message-ID: <481D7F63.5030402@behnel.de>

Hi,

Kirill Smelkov wrote:
> Here is a couple of handy patches for the case when cython is used "in-tree",
> i.e. when no installation is done.
> 
> - First we need to 'chmod a+x' relevant files in bin/ ,
> - and second, it would be convenient to finally create reasonable .hgignore

Thanks, your patches are appreciated. Still, for the next time, could you send
a bundle of changesets (hg bundle) instead of one mail per patch? Although
this list is called "cython-dev", it's a somewhat general purpose list for
Cython, so many people who read it do not care about small fixes and changes.

Thanks,
Stefan

From kirr at mns.spb.ru  Sun May  4 12:08:03 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 4 May 2008 14:08:03 +0400
Subject: [Cython]
	=?utf-8?q?=5BPATCH_0_of_2=5D_A_couple_of_handy_patches_w?=
	=?utf-8?q?hen_using_cython=09in-tree?=
In-Reply-To: <481D7F63.5030402@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481D7F63.5030402@behnel.de>
Message-ID: <200805041408.03746.kirr@mns.spb.ru>

Hi,

? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
> Hi,
> 
> Kirill Smelkov wrote:
> > Here is a couple of handy patches for the case when cython is used "in-tree",
> > i.e. when no installation is done.
> > 
> > - First we need to 'chmod a+x' relevant files in bin/ ,
> > - and second, it would be convenient to finally create reasonable .hgignore
> 
> Thanks, your patches are appreciated. Still, for the next time, could you send
> a bundle of changesets (hg bundle) instead of one mail per patch? Although
> this list is called "cython-dev", it's a somewhat general purpose list for
> Cython, so many people who read it do not care about small fixes and changes.

Stefan, I see your points.

I used 'hg email --outgoing' to send my patches, so if i want it to be sent as
bundle, I'd use 'hg email --bundle'.

I see one problem with this approach though: bundle is a binary file, and if
someone would like to comment on the patch itself - he'll need to manually
paste relevant chunks into reply mail.

I understand this list is general purpose, so maybe it would be good idea to
create separate list for patches? e.g. cython-patches at codespeak.net

Then sending patches there in plain text would be ok for all:

- readable patches right in email client
- ready-to apply patches right from email client  (one button in mutt)
- easy to comment on patches

This is common technique, and for example we do this in sympy-patches:

http://groups.google.com/group/sympy-patches/

What do you think?

Kirill.

From gfurnish at gfurnish.net  Sun May  4 13:22:46 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 4 May 2008 05:22:46 -0600
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
	cython in-tree
In-Reply-To: <200805041408.03746.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481D7F63.5030402@behnel.de> <200805041408.03746.kirr@mns.spb.ru>
Message-ID: <8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>

I strongly disagree.  Bundles are horrible for small changes.  Text
patches are easy to examine without using hg.  I should not have to
modify my repo to inspect a 3 line patch.

On Sun, May 4, 2008 at 4:08 AM, Kirill Smelkov <kirr at mns.spb.ru> wrote:
> Hi,
>
>  ? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
>
> > Hi,
>  >
>  > Kirill Smelkov wrote:
>  > > Here is a couple of handy patches for the case when cython is used "in-tree",
>  > > i.e. when no installation is done.
>  > >
>  > > - First we need to 'chmod a+x' relevant files in bin/ ,
>  > > - and second, it would be convenient to finally create reasonable .hgignore
>  >
>  > Thanks, your patches are appreciated. Still, for the next time, could you send
>  > a bundle of changesets (hg bundle) instead of one mail per patch? Although
>  > this list is called "cython-dev", it's a somewhat general purpose list for
>  > Cython, so many people who read it do not care about small fixes and changes.
>
>  Stefan, I see your points.
>
>  I used 'hg email --outgoing' to send my patches, so if i want it to be sent as
>  bundle, I'd use 'hg email --bundle'.
>
>  I see one problem with this approach though: bundle is a binary file, and if
>  someone would like to comment on the patch itself - he'll need to manually
>  paste relevant chunks into reply mail.
>
>  I understand this list is general purpose, so maybe it would be good idea to
>  create separate list for patches? e.g. cython-patches at codespeak.net
>
>  Then sending patches there in plain text would be ok for all:
>
>  - readable patches right in email client
>  - ready-to apply patches right from email client  (one button in mutt)
>  - easy to comment on patches
>
>  This is common technique, and for example we do this in sympy-patches:
>
>  http://groups.google.com/group/sympy-patches/
>
>  What do you think?
>
>  Kirill.
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Sun May  4 13:34:09 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 04 May 2008 13:34:09 +0200
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
 cython in-tree
In-Reply-To: <8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	<481D7F63.5030402@behnel.de>
	<200805041408.03746.kirr@mns.spb.ru>
	<8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>
Message-ID: <481D9F31.7040800@behnel.de>

Hi,

Gary Furnish top-posted:
> I strongly disagree.  Bundles are horrible for small changes.  Text
> patches are easy to examine without using hg.  I should not have to
> modify my repo to inspect a 3 line patch.

Ok, what do you suggest then?

Stefan


From gfurnish at gfurnish.net  Sun May  4 13:51:28 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 4 May 2008 05:51:28 -0600
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
	cython in-tree
In-Reply-To: <481D9F31.7040800@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481D7F63.5030402@behnel.de> <200805041408.03746.kirr@mns.spb.ru>
	<8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>
	<481D9F31.7040800@behnel.de>
Message-ID: <8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>

I agree with the suggestion to have a cython-patches mailing list,
either on google or codespeak.

On Sun, May 4, 2008 at 5:34 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>  Gary Furnish top-posted:
>
> > I strongly disagree.  Bundles are horrible for small changes.  Text
>  > patches are easy to examine without using hg.  I should not have to
>  > modify my repo to inspect a 3 line patch.
>
>  Ok, what do you suggest then?
>
>  Stefan
>
>
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Sun May  4 14:27:09 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 04 May 2008 14:27:09 +0200
Subject: [Cython] code review
In-Reply-To: <8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	<481D7F63.5030402@behnel.de>
	<200805041408.03746.kirr@mns.spb.ru>	<8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>	<481D9F31.7040800@behnel.de>
	<8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>
Message-ID: <481DAB9D.1070003@behnel.de>

Hi,

Gary Furnish top-posted again:
> I agree with the suggestion to have a cython-patches mailing list,
> either on google or codespeak.

I can ask on codespeak. However, wouldn't a tool like this be better:

http://codereview.appspot.com/

Guido pronounced he'd be open-sourcing it tomorrow (Apache 2 license).

http://comments.gmane.org/gmane.comp.python.python-3000.devel/13110

We could see how much work it would be to get it running with hg, and if it's
doable, maybe Robert and William could host it on their site, next to the hg
repositories?

Stefan


From gfurnish at gfurnish.net  Sun May  4 14:38:56 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 4 May 2008 06:38:56 -0600
Subject: [Cython] code review
In-Reply-To: <481DAB9D.1070003@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481D7F63.5030402@behnel.de> <200805041408.03746.kirr@mns.spb.ru>
	<8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>
	<481D9F31.7040800@behnel.de>
	<8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>
	<481DAB9D.1070003@behnel.de>
Message-ID: <8f8f8530805040538o5ae2be44w333b7974e6894b44@mail.gmail.com>

Why not just use trac?  The interface is much nicer, and we could move
away from the slow and unfriendly launchpad at the same time (and we
could easily host it too).  It is not integrated with svn like
codereview is, but it does have the online patch viewer functionality,
and using trac saves us from having to port some other solution to hg.

--Gary

On Sun, May 4, 2008 at 6:27 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>  Gary Furnish top-posted again:
>  > I agree with the suggestion to have a cython-patches mailing list,
>  > either on google or codespeak.
>
>  I can ask on codespeak. However, wouldn't a tool like this be better:
>
>  http://codereview.appspot.com/
>
>  Guido pronounced he'd be open-sourcing it tomorrow (Apache 2 license).
>
>  http://comments.gmane.org/gmane.comp.python.python-3000.devel/13110
>
>  We could see how much work it would be to get it running with hg, and if it's
>  doable, maybe Robert and William could host it on their site, next to the hg
>  repositories?
>
>  Stefan
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>

From kirr at mns.spb.ru  Sun May  4 14:55:31 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 4 May 2008 16:55:31 +0400
Subject: [Cython] code review
In-Reply-To: <481DAB9D.1070003@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>
	<481DAB9D.1070003@behnel.de>
Message-ID: <200805041655.31562.kirr@mns.spb.ru>

? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
> Hi,
> 
> Gary Furnish top-posted again:
> > I agree with the suggestion to have a cython-patches mailing list,
> > either on google or codespeak.
> 
> I can ask on codespeak. However, wouldn't a tool like this be better:
> 
> http://codereview.appspot.com/
> 
> Guido pronounced he'd be open-sourcing it tomorrow (Apache 2 license).
> 
> http://comments.gmane.org/gmane.comp.python.python-3000.devel/13110

I'm looking forward for this!

Actually, for sympy-patches we already started to think about setting up,
say roundup instance somewhere, sitting and watching for mails on
sympy-patches. Then every review request will get into our issues tracker,
and every mail reply gets into tracker too, and also the whole thing could
be managed then from issues interface too.

I'm +1 for everything that would help manage reviews, but I think it should
integrate pretty well with plain old email and text editor.

Roundup has good email integration, and I hope Guido's tool do too.

Looking forward!
 
> We could see how much work it would be to get it running with hg, and if it's
> doable, maybe Robert and William could host it on their site, next to the hg
> repositories?

In SymPy we are too interested in such things, so the effort could be joined.

Kirill.

From stefan_ml at behnel.de  Sun May  4 14:58:38 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 04 May 2008 14:58:38 +0200
Subject: [Cython] code review
In-Reply-To: <8f8f8530805040538o5ae2be44w333b7974e6894b44@mail.gmail.com>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	<481D7F63.5030402@behnel.de>
	<200805041408.03746.kirr@mns.spb.ru>	<8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>	<481D9F31.7040800@behnel.de>	<8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>	<481DAB9D.1070003@behnel.de>
	<8f8f8530805040538o5ae2be44w333b7974e6894b44@mail.gmail.com>
Message-ID: <481DB2FE.70309@behnel.de>

Hi,

Gary Furnish continues to top-post:
> Why not just use trac?  The interface is much nicer [...] (and we
> could easily host it too).  It is not integrated with svn like
> codereview is, but it does have the online patch viewer functionality,

Hmm, I never used that. Would you know an example to look at? Is it just a
patch "viewer" or can you add comments? And if so, at what granularity? Per
patch? Per line?


> and using trac saves us from having to port some other solution to hg.

that is definitely a plus.


> the slow and unfriendly launchpad

Interesting. Why do you think so?

Stefan


From kirr at mns.spb.ru  Sun May  4 16:24:45 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 4 May 2008 18:24:45 +0400
Subject: [Cython]
	=?utf-8?q?=5BPATCH_0_of_2=5D_A_couple_of_handy_patches_w?=
	=?utf-8?q?hen_using_cython=09in-tree?=
In-Reply-To: <481D7F63.5030402@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481D7F63.5030402@behnel.de>
Message-ID: <200805041824.45735.kirr@mns.spb.ru>

Hi Stefan,

? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
> Hi,
> 
> Kirill Smelkov wrote:
> > Here is a couple of handy patches for the case when cython is used "in-tree",
> > i.e. when no installation is done.
> > 
> > - First we need to 'chmod a+x' relevant files in bin/ ,
> > - and second, it would be convenient to finally create reasonable .hgignore
> 
> Thanks, your patches are appreciated.

I look here

http://hg.cython.org/cython-devel/

and still can't see they were applied.

Does it mean I have to do them another way, or am I missing something?

Thanks,
Kirill.




From gfurnish at gfurnish.net  Sun May  4 16:46:20 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 4 May 2008 08:46:20 -0600
Subject: [Cython] code review
In-Reply-To: <481DB2FE.70309@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481D7F63.5030402@behnel.de> <200805041408.03746.kirr@mns.spb.ru>
	<8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>
	<481D9F31.7040800@behnel.de>
	<8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>
	<481DAB9D.1070003@behnel.de>
	<8f8f8530805040538o5ae2be44w333b7974e6894b44@mail.gmail.com>
	<481DB2FE.70309@behnel.de>
Message-ID: <8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>

The canonical example of a project that uses trac is Sage:
http://trac.sagemath.org/sage_trac
It has good email integration (notify on ticket modification, etc,
although it is supposed to be even better in the next version).
Normally we attach patches to a ticket and then just comment them in
the ticket associated with the patch.  This associates the bug/feature
ticket with the patch (as opposed to needing two systems if you went
with something like codeview + launchpad).  Compared to trac,
launchpad is slow for webpage response time.  It is also slow on
ticket creation time (I can create a ticket for a given item in maybe
30 seconds on one page in trac, whereas launchpad has so much
complexity it is significantly more time consuming, requires a
multipage creation step, etc).

--Gary

On Sun, May 4, 2008 at 6:58 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>  Gary Furnish continues to top-post:
>
> > Why not just use trac?  The interface is much nicer [...] (and we
>
> > could easily host it too).  It is not integrated with svn like
>  > codereview is, but it does have the online patch viewer functionality,
>
>  Hmm, I never used that. Would you know an example to look at? Is it just a
>  patch "viewer" or can you add comments? And if so, at what granularity? Per
>  patch? Per line?
>
>
>
>  > and using trac saves us from having to port some other solution to hg.
>
>  that is definitely a plus.
>
>
>
>  > the slow and unfriendly launchpad
>
>  Interesting. Why do you think so?
>
>  Stefan
>
>

From kirr at mns.spb.ru  Sun May  4 17:16:49 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 4 May 2008 19:16:49 +0400
Subject: [Cython] code review
In-Reply-To: <8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481DB2FE.70309@behnel.de>
	<8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>
Message-ID: <200805041916.49167.kirr@mns.spb.ru>

? ????????? ?? ??????????? 04 ??? 2008 Gary Furnish ???????(a):
> The canonical example of a project that uses trac is Sage:
> http://trac.sagemath.org/sage_trac
> It has good email integration (notify on ticket modification, etc,
> although it is supposed to be even better in the next version).

I'd like to clarify what I mean saying "good email integration":

  Good email integration is two-way

That is:

-> it is possible to affect state of the issues, add comments to patch review,
   etc... by sending mail.
<- you get notification mails, when someone changes something through web
   interface, or another way.


Personally, I think having the first entry is important - a lot of tasks could
be done via plain emails, and at least some people are more productive with
keyboard & text editor (compared to clicking with mouse) :)

Does Trac have two-way email integration?



> Normally we attach patches to a ticket and then just comment them in
> the ticket associated with the patch.  This associates the bug/feature
> ticket with the patch (as opposed to needing two systems if you went
> with something like codeview + launchpad).  Compared to trac,
> launchpad is slow for webpage response time.  It is also slow on
> ticket creation time (I can create a ticket for a given item in maybe
> 30 seconds on one page in trac, whereas launchpad has so much
> complexity it is significantly more time consuming, requires a
> multipage creation step, etc).

Gary, All, imagine you could create a patch issue with plain 'hg email',
or create new issue with just sending mail to special address.

Isn't this cool!?

Roundup has this now and it works. Also, although Roundup is not so shiny,
it was choosen as the tracker for Python itself:

http://bugs.python.org/

What do you think?

From stefan_ml at behnel.de  Sun May  4 17:36:19 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 04 May 2008 17:36:19 +0200
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
 cython in-tree
In-Reply-To: <200805041824.45735.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481D7F63.5030402@behnel.de> <200805041824.45735.kirr@mns.spb.ru>
Message-ID: <481DD7F3.6040401@behnel.de>

Hi,

Kirill Smelkov wrote:
> Hi Stefan,
> 
> ? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
>> Hi,
>>
>> Kirill Smelkov wrote:
>>> Here is a couple of handy patches for the case when cython is used "in-tree",
>>> i.e. when no installation is done.
>>>
>>> - First we need to 'chmod a+x' relevant files in bin/ ,
>>> - and second, it would be convenient to finally create reasonable .hgignore
>> Thanks, your patches are appreciated.
> 
> I look here
> 
> http://hg.cython.org/cython-devel/
> 
> and still can't see they were applied.

See? That's one of the advantages of a bundle. It allows you to send a
complete set of changes against the current trunk, so that things don't get lost.

Anything else missing?

Stefan


From gfurnish at gfurnish.net  Sun May  4 17:37:28 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 4 May 2008 09:37:28 -0600
Subject: [Cython] code review
In-Reply-To: <200805041916.49167.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<481DB2FE.70309@behnel.de>
	<8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>
	<200805041916.49167.kirr@mns.spb.ru>
Message-ID: <8f8f8530805040837k6cd312f4u47e64337b3dbea20@mail.gmail.com>

Trac doesn't have two way notification emails, only outgoing (as far
as I know).  Roundup looks neat (especially the two way email
integration), but I'm not happy about the lack of a good interface for
viewing patches compared to trac's (Example:
http://trac.sagemath.org/sage_trac/attachment/ticket/3025/9609.patch
), as roundup just displays the raw text.

On Sun, May 4, 2008 at 9:16 AM, Kirill Smelkov <kirr at mns.spb.ru> wrote:
> ? ????????? ?? ??????????? 04 ??? 2008 Gary Furnish ???????(a):
>
> > The canonical example of a project that uses trac is Sage:
>  > http://trac.sagemath.org/sage_trac
>  > It has good email integration (notify on ticket modification, etc,
>  > although it is supposed to be even better in the next version).
>
>  I'd like to clarify what I mean saying "good email integration":
>
>   Good email integration is two-way
>
>  That is:
>
>  -> it is possible to affect state of the issues, add comments to patch review,
>    etc... by sending mail.
>  <- you get notification mails, when someone changes something through web
>    interface, or another way.
>
>
>  Personally, I think having the first entry is important - a lot of tasks could
>  be done via plain emails, and at least some people are more productive with
>  keyboard & text editor (compared to clicking with mouse) :)
>
>  Does Trac have two-way email integration?
>
>
>
>
>  > Normally we attach patches to a ticket and then just comment them in
>  > the ticket associated with the patch.  This associates the bug/feature
>  > ticket with the patch (as opposed to needing two systems if you went
>  > with something like codeview + launchpad).  Compared to trac,
>  > launchpad is slow for webpage response time.  It is also slow on
>  > ticket creation time (I can create a ticket for a given item in maybe
>  > 30 seconds on one page in trac, whereas launchpad has so much
>  > complexity it is significantly more time consuming, requires a
>  > multipage creation step, etc).
>
>  Gary, All, imagine you could create a patch issue with plain 'hg email',
>  or create new issue with just sending mail to special address.
>
>  Isn't this cool!?
>
>  Roundup has this now and it works. Also, although Roundup is not so shiny,
>  it was choosen as the tracker for Python itself:
>
>  http://bugs.python.org/
>
>  What do you think?
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Sun May  4 17:41:38 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 04 May 2008 17:41:38 +0200
Subject: [Cython] code review
In-Reply-To: <8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	
	<481D7F63.5030402@behnel.de> <200805041408.03746.kirr@mns.spb.ru>	
	<8f8f8530805040422i1a7fce66h94086394bb5cf41b@mail.gmail.com>	
	<481D9F31.7040800@behnel.de>	
	<8f8f8530805040451t3dfbea2cx670948fd8a8725b2@mail.gmail.com>	
	<481DAB9D.1070003@behnel.de>	
	<8f8f8530805040538o5ae2be44w333b7974e6894b44@mail.gmail.com>	
	<481DB2FE.70309@behnel.de>
	<8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>
Message-ID: <481DD932.4010407@behnel.de>

Hi,

Gary Furnish consistently continues to top-post:
> The canonical example of a project that uses trac is Sage:
> http://trac.sagemath.org/sage_trac

obviously :)


> Normally we attach patches to a ticket and then just comment them in
> the ticket associated with the patch.  This associates the bug/feature
> ticket with the patch (as opposed to needing two systems if you went
> with something like codeview + launchpad).

Launchpad can do these things, but it lacks hg integration.


> Compared to trac,
> launchpad is slow for webpage response time.

I rarely had a problem with that. Maybe we're just using it from different
time zones.


> It is also slow on
> ticket creation time (I can create a ticket for a given item in maybe
> 30 seconds on one page in trac, whereas launchpad has so much
> complexity it is significantly more time consuming, requires a
> multipage creation step, etc).

I can't second that either. In launchpad, you write a subject and on submit it
will tell you (a goodie I really like) if it found any possible matches
amongst the existing tickets. You can then look at them or decide to write a
new ticket. I found that totally straight forward (even the first time I used
it) and very helpful for unexperienced users.

Stefan


From stefan_ml at behnel.de  Sun May  4 17:43:20 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 04 May 2008 17:43:20 +0200
Subject: [Cython] code review
In-Reply-To: <200805041916.49167.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	<481DB2FE.70309@behnel.de>	<8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>
	<200805041916.49167.kirr@mns.spb.ru>
Message-ID: <481DD998.8000204@behnel.de>

Hi,

Kirill Smelkov wrote:
> I'd like to clarify what I mean saying "good email integration":
> 
>   Good email integration is two-way
> 
> That is:
> 
> -> it is possible to affect state of the issues, add comments to patch review,
>    etc... by sending mail.
> <- you get notification mails, when someone changes something through web
>    interface, or another way.

Launchpad also has that. You can just reply to the bug notification email and
it will show up in the ticket comments.

Stefan


From mabshoff at googlemail.com  Sun May  4 17:06:10 2008
From: mabshoff at googlemail.com (Michael Abshoff)
Date: Sun, 04 May 2008 17:06:10 +0200
Subject: [Cython] code review
In-Reply-To: <200805041916.49167.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	<481DB2FE.70309@behnel.de>	<8f8f8530805040746q3da11076l44c0709531f3a2bb@mail.gmail.com>
	<200805041916.49167.kirr@mns.spb.ru>
Message-ID: <481DD0E2.9090207@googlemail.com>

Kirill Smelkov wrote:
> ? ????????? ?? ??????????? 04 ??? 2008 Gary Furnish ???????(a):
>   
>> The canonical example of a project that uses trac is Sage:
>> http://trac.sagemath.org/sage_trac
>> It has good email integration (notify on ticket modification, etc,
>> although it is supposed to be even better in the next version).
>>     
>
> I'd like to clarify what I mean saying "good email integration":
>
>   Good email integration is two-way
>
> That is:
>
> -> it is possible to affect state of the issues, add comments to patch review,
>    etc... by sending mail.
>   

Nope.

> <- you get notification mails, when someone changes something through web
>    interface, or another way.
>
>   
Yes.

> Personally, I think having the first entry is important - a lot of tasks could
> be done via plain emails, and at least some people are more productive with
> keyboard & text editor (compared to clicking with mouse) :)
>
>   
Well, some people like the Debian bugtracker which does let you submit 
item email, some people don't. Trac does not offer that in the default 
config and I don't think it is a good idea to do that. Trac offers some 
subset of wiki tags which make it preferable to use only the 
webinterface. The Debian bug tracker reads like an email list, so if you 
want that there is no advantage to switching to Trac.

> Does Trac have two-way email integration?
>   
Nope. It might be something available at trac-hacks, but last time I 
checked I didn't see anything.

>> Normally we attach patches to a ticket and then just comment them in
>> the ticket associated with the patch.  This associates the bug/feature
>> ticket with the patch (as opposed to needing two systems if you went
>> with something like codeview + launchpad).  Compared to trac,
>> launchpad is slow for webpage response time.  It is also slow on
>> ticket creation time (I can create a ticket for a given item in maybe
>> 30 seconds on one page in trac, whereas launchpad has so much
>> complexity it is significantly more time consuming, requires a
>> multipage creation step, etc).
>>     
>
> Gary, All, imagine you could create a patch issue with plain 'hg email',
> or create new issue with just sending mail to special address.
>
> Isn't this cool!?
>   

No. I don't really see the benefit there by just firing off an email. 
Our work flows are probably very different and I am no mouse pusher, but 
I prefer Trac and its workflow. I was skeptical initially, but since I 
spend a lot of time daily with Trac putting releases together for Sage I 
am quite sold on its work flow. I generally dislie the "submit bug 
report by email" work flow. We have sage-devel for the discussion bit 
and if we really have an issue we open a ticket.

We run 0.10.x, which is the current stable release. It has a couple 
issues, but those are mostly getting sorted out in 0.11. The main point 
of running your own trac install is that you own your data and do not 
depend on some other org to do things for you. Robert Bradshaw wrote an 
hg inspection module to let you check out patches and bundles online. 
There is also svn integration per default and some optional code to 
watch commits in an hg repo.

> Roundup has this now and it works. Also, although Roundup is not so shiny,
> it was choosen as the tracker for Python itself:
>
>   
Sure, but you need Django and a couple other things. Trac is self 
contained and has next to no dependencies, i.e. the default db is 
sqlite. It scales well for Sage, so I would assume it will work well for 
Cython which deals with a lot fewer patches on average compared to Sage.

> http://bugs.python.org/
>
> What do you think?
>   
:)

Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>   


From kirr at mns.spb.ru  Sun May  4 18:10:43 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 4 May 2008 20:10:43 +0400
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
	cython in-tree
In-Reply-To: <481DD7F3.6040401@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805041824.45735.kirr@mns.spb.ru> <481DD7F3.6040401@behnel.de>
Message-ID: <200805042010.43445.kirr@mns.spb.ru>

? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
> Hi,
> 
> Kirill Smelkov wrote:
> > Hi Stefan,
> > 
> > ? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
> >> Hi,
> >>
> >> Kirill Smelkov wrote:
> >>> Here is a couple of handy patches for the case when cython is used "in-tree",
> >>> i.e. when no installation is done.
> >>>
> >>> - First we need to 'chmod a+x' relevant files in bin/ ,
> >>> - and second, it would be convenient to finally create reasonable .hgignore
> >> Thanks, your patches are appreciated.
> > 
> > I look here
> > 
> > http://hg.cython.org/cython-devel/
> > 
> > and still can't see they were applied.
> 
> See? That's one of the advantages of a bundle. It allows you to send a
> complete set of changes against the current trunk, so that things don't get lost.

No :)

I see that bundle v.s. text patches is another things.

You import bundle, and you apply patches.

Actually, applying patches is more convenient to me, since some of the patches could
be good, and some don't. that's why i'd like to apply only some, and with bundle you
have applied them all.

Just look at how things are done in Linux, Mercurial, etc -- they send plain text
patches.

> Anything else missing?

Everything is in, thanks!

I (minorly) wonder why you changed patch description in 'chmod a+x bin/*' patch

Kirill.

From kirr at mns.spb.ru  Sun May  4 19:08:24 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sun, 4 May 2008 21:08:24 +0400
Subject: [Cython] code review
In-Reply-To: <481DD0E2.9090207@googlemail.com>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805041916.49167.kirr@mns.spb.ru>
	<481DD0E2.9090207@googlemail.com>
Message-ID: <200805042108.24619.kirr@mns.spb.ru>

Hi, All

? ????????? ?? ??????????? 04 ??? 2008 Gary Furnish ???????(a):
> Trac doesn't have two way notification emails, only outgoing (as far
> as I know).  Roundup looks neat (especially the two way email
> integration), but I'm not happy about the lack of a good interface for
> viewing patches compared to trac's (Example:
> http://trac.sagemath.org/sage_trac/attachment/ticket/3025/9609.patch
> ), as roundup just displays the raw text.

Gary, I see your points.
Having patch viewer is indeed useful, like having two-way emails are useful too.

? ????????? ?? ??????????? 04 ??? 2008 Stefan Behnel ???????(a):
> Kirill Smelkov wrote:
> > I'd like to clarify what I mean saying "good email integration":
> > 
> >   Good email integration is two-way
> > 
> > That is:
> > 
> > -> it is possible to affect state of the issues, add comments to patch review,
> >    etc... by sending mail.
> > <- you get notification mails, when someone changes something through web
> >    interface, or another way.
> 
> Launchpad also has that. You can just reply to the bug notification email and
> it will show up in the ticket comments.

Then this is a plus for Launchpad.

From the other way is it possible to export all stored data from launchpad into
some interoperable format, e.g. csv?


? ????????? ?? ??????????? 04 ??? 2008 Michael Abshoff ???????(a):
> Kirill Smelkov wrote:
> > ? ????????? ?? ??????????? 04 ??? 2008 Gary Furnish ???????(a):
> >   
> >> The canonical example of a project that uses trac is Sage:
> >> http://trac.sagemath.org/sage_trac
> >> It has good email integration (notify on ticket modification, etc,
> >> although it is supposed to be even better in the next version).
> >>     
> >
> > I'd like to clarify what I mean saying "good email integration":
> >
> >   Good email integration is two-way
> >
> > That is:
> >
> > -> it is possible to affect state of the issues, add comments to patch review,
> >    etc... by sending mail.
> >   
> 
> Nope.

What do you mean here? You don't agree with my understanding of email integration?
Could you please clarify?


> > <- you get notification mails, when someone changes something through web
> >    interface, or another way.
> >
> >   
> Yes.
> 
> > Personally, I think having the first entry is important - a lot of tasks could
> > be done via plain emails, and at least some people are more productive with
> > keyboard & text editor (compared to clicking with mouse) :)
> >
> >   
> Well, some people like the Debian bugtracker which does let you submit 
> item email, some people don't. Trac does not offer that in the default 
> config and I don't think it is a good idea to do that. Trac offers some 
> subset of wiki tags which make it preferable to use only the 
> webinterface. The Debian bug tracker reads like an email list, so if you 
> want that there is no advantage to switching to Trac.

I don't want to be forced to only use either one of web or email. I *do*
want to use web interface often, but sometimes it is more convenient to reply
via email.

And when it comes to patch review, I think it is imho handy to use
full-featured editor, when commenting on a patch.

Maybe at least for me...

> > Does Trac have two-way email integration?
> >   
> Nope. It might be something available at trac-hacks, but last time I 
> checked I didn't see anything.
> 
> >> Normally we attach patches to a ticket and then just comment them in
> >> the ticket associated with the patch.  This associates the bug/feature
> >> ticket with the patch (as opposed to needing two systems if you went
> >> with something like codeview + launchpad).  Compared to trac,
> >> launchpad is slow for webpage response time.  It is also slow on
> >> ticket creation time (I can create a ticket for a given item in maybe
> >> 30 seconds on one page in trac, whereas launchpad has so much
> >> complexity it is significantly more time consuming, requires a
> >> multipage creation step, etc).
> >>     
> >
> > Gary, All, imagine you could create a patch issue with plain 'hg email',
> > or create new issue with just sending mail to special address.
> >
> > Isn't this cool!?
> >   
> 
> No. I don't really see the benefit there by just firing off an email. 
> Our work flows are probably very different and I am no mouse pusher, but 
> I prefer Trac and its workflow. I was skeptical initially, but since I 
> spend a lot of time daily with Trac putting releases together for Sage I 
> am quite sold on its work flow. I generally dislie the "submit bug 
> report by email" work flow. We have sage-devel for the discussion bit 
> and if we really have an issue we open a ticket.

Ok, I agree emails on usual issues maybe not so handy, and creating issue
via email may have its own drawbacks.

But my initial context was about patch reviews, and then I think
it is handy to do it in fully-featured text editor.

e.g. OP sends a mid-size patch, and reviewer comments, e.g.:

http://groups.google.com/group/sympy-patches/msg/5a23e70f64b1b776
http://groups.google.com/group/sympy-patches/msg/aa5a9ce17b9165bc
http://groups.google.com/group/sympy-patches/msg/0b5107e7885f6dcb

So maybe mailing bug reports is not a good idea, but submitting patches via
'hg email' seems (at least to me) to be good, because sending them is one
button, and importing them is one button, and reviewing them is one button
+ text edits + "y" in the end.

I agree this is not ideal, and could be improved, e.g. thus made text edits
could go into patch issue, and when viewing the diff there would be added
comments from various people.

But you know, nothing is ideal, and the scheme I'm talking about works ok in
a lot of places.

> We run 0.10.x, which is the current stable release. It has a couple 
> issues, but those are mostly getting sorted out in 0.11. The main point 
> of running your own trac install is that you own your data and do not 
> depend on some other org to do things for you.

+1 about managing your own data by yourself.

> Robert Bradshaw wrote an  
> hg inspection module to let you check out patches and bundles online. 

Do you mean importing patches from trac with 'hg somecommand'?

This would be handy.

And also, if there were another 'hg somecommand2' moddeled after 'hg email'
to send patches to trac, it would be handy^2.

Btw, how do I check myself how it works?

> There is also svn integration per default and some optional code to 
> watch commits in an hg repo.


> > Roundup has this now and it works. Also, although Roundup is not so shiny,
> > it was choosen as the tracker for Python itself:
> >
> >   
> Sure, but you need Django and a couple other things. Trac is self 

No, Django is not needed to run roundup.

> contained and has next to no dependencies, i.e. the default db is 
> sqlite. It scales well for Sage, so I would assume it will work well for 
> Cython which deals with a lot fewer patches on average compared to Sage.

Actually there are very little dependencies for roundup - it can even use
plain files as DB (not to depend on anything), or optionally sqlite and
other dbs.

I'd say roundup depends only on Python and optionally on sqlite, pytz.
And also you need to setup exim so that incoming emails work.

> 
> > http://bugs.python.org/
> >
> > What do you think?
> >   
> :)

:)

Guys, please don't get me wrong.

I'm not saying "let's use roundup, period." For amount of patches I observe I
think plain cython-patches would do, and then, if needed, we could use any
tool which would do the job.

Only myself I observed that doing things in text editor is handy, thus I
value tools that provide this ability.

I really hope Guido's codereview would be good though and helpful and well
designed.

Let's settle on something practical,
Kirill.

From pete at petertodd.org  Mon May  5 05:37:49 2008
From: pete at petertodd.org (Peter Todd)
Date: Sun, 4 May 2008 23:37:49 -0400
Subject: [Cython] __getattribute__
In-Reply-To: <20080504051532.GA7720@tilt>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
	<20080504051532.GA7720@tilt>
Message-ID: <20080505033749.GF4087@tilt>

On Sun, May 04, 2008 at 01:15:32AM -0400, Peter Todd wrote:
> Here's my first patch. This correctly implements __getattribute__ and
> __getattr__ in the single class case. FWIW I also have a mercurial tree
> if it'd be better to pull from it than apply patches.
> 
> I'm working on making subclasses behave correctly, I've got test cases
> written up showing where things fail, but no solutions to that written
> yet. The slot_tp_getattro stuff Stefan mentioned is useful though.

__getattr(ibute)__ support is now working with subclasses. The semantics
should match Python exactly if my test cases are correct. All that has
changed from the previous patch is that at compile time base classes are
checked for __getattr(ibute)__ methods and those methods are used if
found.

Attached is an hg bundle of the two commits.

-- 
http://petertodd.org 'peter'[:-1]@petertodd.org 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: getattr_support.hg
Type: application/octet-stream
Size: 2517 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080504/e2e831c7/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080504/e2e831c7/attachment.pgp 

From robertwb at math.washington.edu  Mon May  5 20:06:16 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 5 May 2008 11:06:16 -0700
Subject: [Cython] code review
In-Reply-To: <200805042108.24619.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805041916.49167.kirr@mns.spb.ru>
	<481DD0E2.9090207@googlemail.com>
	<200805042108.24619.kirr@mns.spb.ru>
Message-ID: <52199062-9690-4C1F-8E5A-40195520E386@math.washington.edu>

> Guys, please don't get me wrong.
>
> I'm not saying "let's use roundup, period." For amount of patches I  
> observe I
> think plain cython-patches would do, and then, if needed, we could  
> use any
> tool which would do the job.
>
> Only myself I observed that doing things in text editor is handy,  
> thus I
> value tools that provide this ability.
>
> I really hope Guido's codereview would be good though and helpful  
> and well
> designed.
>
> Let's settle on something practical,


I agree that we need something practical. I have also been frustrated  
with the speed (both page load time and having to navigate multiple  
pages to do simple tasks) with launchpad, which is why I don't use it  
as much as I probably should. I would also much prefer to self-host  
our own bug tracker.

I'd also vote for trac, but I guess that's no surprise because I'm  
also part of the Sage team (as are a lot of other potential Cython  
developers). The workflow of using trac to manage bugs and feature  
requests works well. Also, cython.org is hosted from the same server  
as sagemath, so we know that trac will always run well on it as it is  
being used for the Sage project. I wrote a simple bundle inspector  
for trac, it has good support for outgoing email notifications, and  
when the UW math department started using Trac for their help system  
I wrote a simple plugin that lets it accept incoming tickets as well  
if people want this. (One big problem with this is that one then has  
to deal with the issue of spam, so I would suggest only ticket  
modification, not creation, be allowed via email if we decide to do  
this.) If people like this suggestion I can go ahead and set up a  
trac server.

On the questions of patches vs. bundles, for small changes,  
especially if there aren't interdependencies, I prefer patches. Once  
you get beyond a couple of changesets bundles become easier to deal  
with, especially to make sure nothing slips between the cracks (this  
is less of an issue if we start using a good ticketing system as  
discussed above). Note that one can use "hg incoming -p" to see the  
contents of a bundle without applying it.

- Robert


From robertwb at math.washington.edu  Mon May  5 20:30:30 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 5 May 2008 11:30:30 -0700
Subject: [Cython] [Pyrex] newbie list processing question
In-Reply-To: <481F4CF3.1020204@cc.gatech.edu>
References: <481F4CF3.1020204@cc.gatech.edu>
Message-ID: <0F507576-A9F6-43CA-8FF7-B744C1A6796B@math.washington.edu>

On May 5, 2008, at 11:07 AM, Daniel Ashbrook wrote:

> So I've done a lot of Google searching and haven't found an answer to
> this question, or at least one that I understand.
>
> I'm trying to write some fairly simple list-processing code using  
> pyrex.
> As a simple example, let's say I wanted to translate this:
>
> def addOne(l):
>    assert(isinstance(l, list))
>    for i in xrange(l):
>      l[i] += 1
>
> or even this:
>
> def addOne(l):
>    return [i+1 for i in l]
>
> to fast-running pyrex code.

This last implementation of addOne should work as is in Cython, and  
will be nearly optimal (assuming your CPU has reasonable branch  
prediction). However, if you are manipulating word-sized integers,  
using C arrays will give you a manyfold over python arithmetic.

> How can I actually access the list properly?
> (I should point out that in my real problem, I actually need random
> access to the list.) I've seen some talk of using  
> PyObject_AsWriteBuffer
> to do it, but the post was related to array objects, which I'd rather
> not use unless I have to. I've been trying to figure it out using
> PyList_GetItem/SetItem and so on, but it seems extremely  
> convoluted, like:
>
> PyList_SetItem(l,i,PyInt_FromLong(PyInt_AsLong(PyList_GetItem(l,i)) 
> +1))
>
> Surely there is another way! (Plus that line doesn't work - trying to
> use "for i from" gives me errors about making i an integer from a
> pointer without a cast.)

Did you do

cdef int i
for i from 0 <= i < len(L):
     ...

>
> Any tips? If the person who maintains documentation for pyrex is
> reading, I'd love to see a simple list-processing example included.

In Cython, if one writes

L[i]

where i is a cdef int, then it checks to see at runtime if L is a  
list and accesses its elements via a macro. Otherwise one can use  
PyList_SetItem and friends, but as you have noticed that is  
cumbersome (as well ahs being hard to read).

- Robert


From robertwb at math.washington.edu  Mon May  5 21:19:38 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 5 May 2008 12:19:38 -0700
Subject: [Cython] [Pyrex] newbie list processing question
In-Reply-To: <481F55E8.20603@cc.gatech.edu>
References: <481F4CF3.1020204@cc.gatech.edu>
	<0F507576-A9F6-43CA-8FF7-B744C1A6796B@math.washington.edu>
	<481F55E8.20603@cc.gatech.edu>
Message-ID: <DDAC9887-A5C3-49B1-AD48-EA3EA34C99B2@math.washington.edu>

On May 5, 2008, at 11:46 AM, Daniel Ashbrook wrote:
> Robert Bradshaw wrote:
>>> def addOne(l):
>>>    return [i+1 for i in l]
> >> ...
>> This last implementation of addOne should work as is in Cython,  
>> and will be nearly optimal (assuming your CPU has reasonable  
>> branch prediction). However, if you are manipulating word-sized  
>> integers, using C arrays will give you a manyfold over python  
>> arithmetic.
>
> So in the real code, I'm actually doing float math. And I'll be  
> wanting to return my results in a list object. There will be many  
> thousands of float results; what's the best way to deal with that?  
> Use a C-specific data structure to store the results then turn it  
> into a list somehow?

I would do all my computations with C doubles, and convert to a list  
only at the very end. I would also seriously consider using NumPy  
arrays http://numpy.scipy.org/ . There is a fair amount of support  
for NumPy/Cython integration, and it's only going to get better this  
summer (two SoC projects).

>> Did you do
>> cdef int i
>> for i from 0 <= i < len(L):
>>     ...
>
> Ah, I missed the "cdef int i" part of it.

Yeah. One thing you can do is use the -a option, which will spit out  
an "annotated" html file that will catch stuff like this (bright  
yellow lines means there's a lot of unnecessary conversion going on,  
usually indicating that a variable wasn't cdef'd.

>> In Cython, if one writes
>> L[i]
>> where i is a cdef int, then it checks to see at runtime if L is a  
>> list and accesses its elements via a macro. Otherwise one can use  
>> PyList_SetItem and friends, but as you have noticed that is  
>> cumbersome (as well ahs being hard to read).
>
> Oh ho! That works very nicely; I'll include code to help others in  
> the future:
>
>   cdef int i
>   for i from 0 <= i < len(l):
>     l[i] = l[i] + 1
>
>
> Thanks for the help!

No problem.

- Robert


From stefan_ml at behnel.de  Mon May  5 08:16:11 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 05 May 2008 08:16:11 +0200
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
 cython in-tree
In-Reply-To: <200805042010.43445.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805041824.45735.kirr@mns.spb.ru> <481DD7F3.6040401@behnel.de>
	<200805042010.43445.kirr@mns.spb.ru>
Message-ID: <481EA62B.6000405@behnel.de>

Hi,

Kirill Smelkov wrote:
> I (minorly) wonder why you changed patch description in 'chmod a+x bin/*' patch

Because it's more work for me to save an email to a file and import it as
patch, than to apply the trivial change myself and commit it. :)

Stefan



From stefan_ml at behnel.de  Mon May  5 21:35:58 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 05 May 2008 21:35:58 +0200
Subject: [Cython] code review
In-Reply-To: <52199062-9690-4C1F-8E5A-40195520E386@math.washington.edu>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	<200805041916.49167.kirr@mns.spb.ru>	<481DD0E2.9090207@googlemail.com>	<200805042108.24619.kirr@mns.spb.ru>
	<52199062-9690-4C1F-8E5A-40195520E386@math.washington.edu>
Message-ID: <481F619E.50908@behnel.de>

Hi Robert,

Robert Bradshaw wrote:
> I'd also vote for trac,
> [...] If people like this suggestion I can go ahead and set up a
> trac server.

ok, I didn't hear any major reason not to use trac so far, so if you can
handle the setup and hosting, I think trac is the best solution.

Stefan

From greg.ewing at canterbury.ac.nz  Tue May  6 03:51:16 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 06 May 2008 13:51:16 +1200
Subject: [Cython] __getattribute__
In-Reply-To: <20080505033749.GF4087@tilt>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
	<20080504051532.GA7720@tilt> <20080505033749.GF4087@tilt>
Message-ID: <481FB994.9030102@canterbury.ac.nz>

Peter Todd wrote:

> __getattr(ibute)__ support is now working with subclasses.
> 
> Attached is an hg bundle of the two commits.

Can someone send me these as plain text? I may want to
incorporate them into Pyrex as well.

Thanks,
Greg

From robertwb at math.washington.edu  Tue May  6 07:30:12 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 5 May 2008 22:30:12 -0700
Subject: [Cython] __getattribute__
In-Reply-To: <481FB994.9030102@canterbury.ac.nz>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
	<20080504051532.GA7720@tilt> <20080505033749.GF4087@tilt>
	<481FB994.9030102@canterbury.ac.nz>
Message-ID: <F58AEB7D-4390-4468-8CF2-88876A8E4905@math.washington.edu>

On May 5, 2008, at 6:51 PM, Greg Ewing wrote:
> Peter Todd wrote:
>
>> __getattr(ibute)__ support is now working with subclasses.
>>
>> Attached is an hg bundle of the two commits.
>
> Can someone send me these as plain text? I may want to
> incorporate them into Pyrex as well.

Sure. Here you go. There are two patches in this file.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: getattr_support.patch
Type: application/octet-stream
Size: 10506 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080505/fa7b3961/attachment.obj 

From robertwb at math.washington.edu  Tue May  6 08:59:23 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 5 May 2008 23:59:23 -0700
Subject: [Cython] code review
In-Reply-To: <481F619E.50908@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>	<200805041916.49167.kirr@mns.spb.ru>	<481DD0E2.9090207@googlemail.com>	<200805042108.24619.kirr@mns.spb.ru>
	<52199062-9690-4C1F-8E5A-40195520E386@math.washington.edu>
	<481F619E.50908@behnel.de>
Message-ID: <832C57CB-F230-4A14-B520-150FBDD7B5A9@math.washington.edu>

On May 5, 2008, at 12:35 PM, Stefan Behnel wrote:
> Hi Robert,
>
> Robert Bradshaw wrote:
>> I'd also vote for trac,
>> [...] If people like this suggestion I can go ahead and set up a
>> trac server.
>
> ok, I didn't hear any major reason not to use trac so far, so if  
> you can
> handle the setup and hosting, I think trac is the best solution.

Up at http://trac.cython.org/cython_trac/

From kirr at mns.spb.ru  Tue May  6 09:02:50 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Tue, 6 May 2008 11:02:50 +0400
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
	cython in-tree
In-Reply-To: <481EA62B.6000405@behnel.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805042010.43445.kirr@mns.spb.ru> <481EA62B.6000405@behnel.de>
Message-ID: <200805061102.50657.kirr@mns.spb.ru>

Hi,

? ????????? ?? ??????????? 05 ??? 2008 Stefan Behnel ???????(a):
> Hi,
> 
> Kirill Smelkov wrote:
> > I (minorly) wonder why you changed patch description in 'chmod a+x bin/*' patch
> 
> Because it's more work for me to save an email to a file and import it as
> patch, than to apply the trivial change myself and commit it. :)

Stefan, most mail clients could be simply setup to do patch imports with one keypress.

For example in mutt, I use something like this:

  macro pager A "<pipe-entry>(umask 022; hg -R ~/src/sympy/sympy import -)"

And, for git there is a git-am to apply series of patches from mailbox.

So I think we should review our patch applying practices :)

Kirill.


From ondrej at certik.cz  Tue May  6 13:02:41 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 6 May 2008 13:02:41 +0200
Subject: [Cython] [PATCH 0 of 2] A couple of handy patches when using
	cython in-tree
In-Reply-To: <200805061102.50657.kirr@mns.spb.ru>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805042010.43445.kirr@mns.spb.ru> <481EA62B.6000405@behnel.de>
	<200805061102.50657.kirr@mns.spb.ru>
Message-ID: <85b5c3130805060402i50449ef5nc6c1d11aac54f173@mail.gmail.com>

On Tue, May 6, 2008 at 9:02 AM, Kirill Smelkov <kirr at mns.spb.ru> wrote:
> Hi,
>
> ? ????????? ?? ??????????? 05 ??? 2008 Stefan Behnel ???????(a):
>> Hi,
>>
>> Kirill Smelkov wrote:
>> > I (minorly) wonder why you changed patch description in 'chmod a+x bin/*' patch
>>
>> Because it's more work for me to save an email to a file and import it as
>> patch, than to apply the trivial change myself and commit it. :)
>
> Stefan, most mail clients could be simply setup to do patch imports with one keypress.
>
> For example in mutt, I use something like this:
>
>  macro pager A "<pipe-entry>(umask 022; hg -R ~/src/sympy/sympy import -)"
>
> And, for git there is a git-am to apply series of patches from mailbox.
>
> So I think we should review our patch applying practices :)

BTW, I need to try this as well and write it to our tutorial at
docs.sympy.org. So far I also use the "manual" method and so it sucks,
exactly as Stefan said. :)

Ondrej

From stefan_ml at behnel.de  Tue May  6 19:09:29 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 6 May 2008 19:09:29 +0200 (CEST)
Subject: [Cython] code review
In-Reply-To: <832C57CB-F230-4A14-B520-150FBDD7B5A9@math.washington.edu>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805041916.49167.kirr@mns.spb.ru> <481DD0E2.9090207@googlemail.com>
	<200805042108.24619.kirr@mns.spb.ru>
	<52199062-9690-4C1F-8E5A-40195520E386@math.washington.edu>
	<481F619E.50908@behnel.de>
	<832C57CB-F230-4A14-B520-150FBDD7B5A9@math.washington.edu>
Message-ID: <43251.194.114.62.38.1210093769.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> Up at http://trac.cython.org/cython_trac/

Mighty cool! :)

Let's see how people get along with it.

Stefan


From ellisonbg.net at gmail.com  Tue May  6 19:19:27 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 6 May 2008 11:19:27 -0600
Subject: [Cython] non cdef'd class in a pyx file
Message-ID: <6ce0ac130805061019s179c398ocdd4c47fdc22d40c@mail.gmail.com>

Hi,

I am wondering if there are any issues with declaring a regular python
class in a pyx file.  Specifically, we need to define an exception
like this:

class FooError(Exception):
    ...

I have done this before and it seems to work, but are there any
subtleties or things to be aware of?

Thanks

Brian

From robertwb at math.washington.edu  Tue May  6 19:25:06 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 6 May 2008 10:25:06 -0700
Subject: [Cython] non cdef'd class in a pyx file
In-Reply-To: <6ce0ac130805061019s179c398ocdd4c47fdc22d40c@mail.gmail.com>
References: <6ce0ac130805061019s179c398ocdd4c47fdc22d40c@mail.gmail.com>
Message-ID: <CBBAF582-0779-4D66-98A4-CC0A9C903FC7@math.washington.edu>

This should work just fine.

On May 6, 2008, at 10:19 AM, Brian Granger wrote:
> Hi,
>
> I am wondering if there are any issues with declaring a regular python
> class in a pyx file.  Specifically, we need to define an exception
> like this:
>
> class FooError(Exception):
>     ...
>
> I have done this before and it seems to work, but are there any
> subtleties or things to be aware of?
>
> Thanks
>
> Brian
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ellisonbg.net at gmail.com  Tue May  6 19:47:48 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 6 May 2008 11:47:48 -0600
Subject: [Cython] non cdef'd class in a pyx file
In-Reply-To: <CBBAF582-0779-4D66-98A4-CC0A9C903FC7@math.washington.edu>
References: <6ce0ac130805061019s179c398ocdd4c47fdc22d40c@mail.gmail.com>
	<CBBAF582-0779-4D66-98A4-CC0A9C903FC7@math.washington.edu>
Message-ID: <6ce0ac130805061047n136bd362ya93b3342923cac6c@mail.gmail.com>

Thanks!

On Tue, May 6, 2008 at 11:25 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> This should work just fine.
>
>
>
>  On May 6, 2008, at 10:19 AM, Brian Granger wrote:
>  > Hi,
>  >
>  > I am wondering if there are any issues with declaring a regular python
>  > class in a pyx file.  Specifically, we need to define an exception
>  > like this:
>  >
>  > class FooError(Exception):
>  >     ...
>  >
>  > I have done this before and it seems to work, but are there any
>  > subtleties or things to be aware of?
>  >
>  > Thanks
>  >
>  > Brian
>  > _______________________________________________
>  > Cython-dev mailing list
>  > Cython-dev at codespeak.net
>  > http://codespeak.net/mailman/listinfo/cython-dev
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>

From mistobaan at gmail.com  Tue May  6 19:56:16 2008
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Tue, 6 May 2008 10:56:16 -0700
Subject: [Cython] code review
In-Reply-To: <43251.194.114.62.38.1210093769.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <patchbomb.1209889254@tugrik2.mns.mnsspb.ru>
	<200805041916.49167.kirr@mns.spb.ru> <481DD0E2.9090207@googlemail.com>
	<200805042108.24619.kirr@mns.spb.ru>
	<52199062-9690-4C1F-8E5A-40195520E386@math.washington.edu>
	<481F619E.50908@behnel.de>
	<832C57CB-F230-4A14-B520-150FBDD7B5A9@math.washington.edu>
	<43251.194.114.62.38.1210093769.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e437f2f0805061056m51cefd4arb2d4d81c0d790804@mail.gmail.com>

>  > Up at http://trac.cython.org/cython_trac/

Really Good.


Fabrizio

From robertwb at math.washington.edu  Wed May  7 00:21:25 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 6 May 2008 15:21:25 -0700
Subject: [Cython] Visitor patterns and Python 2.4
In-Reply-To: <481BFE9A.4080600@behnel.de>
References: <481AFD4D.4050401@student.matnat.uio.no>
	<481BFE9A.4080600@behnel.de>
Message-ID: <F23CCCB8-24E1-4B09-B2B9-A56A17DFC4C8@math.washington.edu>

On May 2, 2008, at 10:56 PM, Stefan Behnel wrote:

> Hi,
>
> Dag Sverre Seljebotn wrote:
>> Are there any real reasons for leaving the Cython compiler (not  
>> talking
>> about generated or supported code of course) at Python 2.3, rather  
>> than
>> a small bump to 2.4? Reason: I'd like decorators.
>>
>> The rationale: Notice that parse tree visitors can currently be  
>> written
>> like this:
>>
>> class AnalyseControlFlow(VisitorTransform):
>>     def pre_FuncDefNode(self, node):
>>         node.body.analyse_control_flow(node.scope)
>>         return False # do not recurse beyond first function level
>
> If this way of doing it is accepted, I'm actually for accepting 2.4  
> code. I
> don't know any important platform that doesn't come with at least  
> Py2.4. And
> note that most people won't even need to run Cython themselves,  
> even if they
> use software implemented in Cython.

We ship our own Python (2.5.2) with Sage, but I would rather stick  
with 2.3 compatibility. You might not consider it "important" but my  
computer (OS X 10.4, yet I know 10.5 is out but I haven't found the  
time to upgrade yet) has 2.3 by default, as did OS X 10.3.

I very much dislike using __class__.__name__ to decide what functions  
to call, but it doesn't look like you need decorators to get around  
this, do you?

- Robert






From greg.ewing at canterbury.ac.nz  Wed May  7 01:42:52 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 07 May 2008 11:42:52 +1200
Subject: [Cython] non cdef'd class in a pyx file
In-Reply-To: <6ce0ac130805061019s179c398ocdd4c47fdc22d40c@mail.gmail.com>
References: <6ce0ac130805061019s179c398ocdd4c47fdc22d40c@mail.gmail.com>
Message-ID: <4820ECFC.7010006@canterbury.ac.nz>

Brian Granger wrote:
> I am wondering if there are any issues with declaring a regular python
> class in a pyx file.

No, that should work pretty much the same as it does
in Python.

There are a couple of very minor differences -- it creates
the class before executing the class body rather than after,
and it wraps methods in unbound method objects before putting
them in the class (because they're C-implemented functions
rather than Python ones, so the usual descriptor magic
doesn't get done). But you probably won't notice any of this
except in very rare circumstances.

-- 
Greg

From robertwb at math.washington.edu  Wed May  7 03:03:03 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 6 May 2008 18:03:03 -0700
Subject: [Cython] Some small phase refactorings
In-Reply-To: <1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>
References: <481AF97B.8030305@student.matnat.uio.no>
	<481BFDDA.8060007@behnel.de>
	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>
Message-ID: <452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>

On May 3, 2008, at 1:27 AM, Dag Sverre Seljebotn wrote:

>>
>> +        import Cython.Compiler.Transforms.Analysis as Analysis
>> +        Analysis.CreateFunctionScope()(self, env=env)
>> +        Analysis.AnalyseControlFlow()(self, env=env)
>> +        Analysis.AnalyseFunctionBodyDeclarations()(self, env=env)
>> +        Analysis.AnalyseFunctionBodyExpressions()(self, env=env)
>> +        options.transforms.run('after_function_analysis', self,
>> global_env=env)
>> +
>>
>> ### These look like functions, they should follow PEP 8 naming. Then
>> again,
>> ### why aren't they functions?
>
> Note that this code will almost certainly be moved again and  
> rewritten at
> some later point (they can't really belong to ModuleNode); but more
> refactoring must happen before they can be moved to their proper  
> location
> and they serve a good purpose where they are for now though.
>
> They have to be classes as they are transform objects with member  
> methods,
> some state etc. etc. But thinking about it one could have functions  
> like
> this in Analysis.py:
>
> def analyse_function_body_declarations(tree, **opts):
>     return AnalyseFunctionBodyDeclarations()(tree, **opts)
>
> if that helps. The reason I started using the __call__ is that I  
> think in
> time one can treat these as functions, like this:
>
> pipeline = [
>    f,
>    g,
>    AnalyseFunctionBodyDeclarations(),
>    Coercions(),
> ]
> for transform in pipeline:
>    tree = transform(tree)
>
> (ie, basically saying that "pipeline = f <ring> g <ring> h"...)
>
>
>> ### I'm not convinced of this one. I understand why you do it, but I
>> believe
>> ### that using the class itself rather than dispatching on a  
>> string would
>> help
>> ### here (I saw your decorator proposal - still thinking about it,  
>> but
>> might
>> ### be worth doing).
>
> If not, then the classical visitor pattern might put you at ease?:
>
> class FuncDefNode:
>     def accept(self, visitor):
>         visitor.visit_FuncDefNode(self)
>
> However, that's a lot of extra trivial code to add (this would have  
> to be
> added to _all_ classes), and when one is using Python anyway I'd  
> like to
> avoid pretending I'm writing Java... :-)

I think one could put a generic accept function in the Node class  
that would dispatch based on class name (or some class attribute) if  
one wanted, rather than having to write it for every class.

> I hope we can start using Python 2.4, then I'll implement a
> decorator/metaclass solution instead.
>
>
> In conclusion, I'd like to mention that I really think the  
> important thing
> here is to consider the "grand, large-scale" features of the patch. I
> didn't polish the details in any way, because I think that what is
> important here is the changes they make possible in the application
> structure. How the visitors look like can be changed entirely in
> Visitor.py and Analysis.py without interfering with existing code;  
> while
> the phase refactoring is going to intrude everywhere and make  
> changes all
> over the place, so the form the phase refactoring will take is the
> important point.
>
> (OTOH; I guess it is a good time to think about the details as well so
> that when the 1000 line coercion refactoring patch should be  
> written one
> knows what to do in the details...)
>
> (I've asked myself as to whether it is all worth it BTW. It is a  
> heavy and
> non-fun task. But I'm still convinced there's absolutely no way  
> around it
> if the feature-set of Cython is to grow significantly in any way. And
> realistically, it can be done in two or three days with for  
> instance me,
> you and Robert working together... So this might be too early to talk
> about it; but I end up working on it anyway because it is  
> effectively a
> blocker for me and I cannot get anywhere with my GSoC stuff until  
> it is
> done :-) )

I think this is good to talk about now. Basically, you want to change  
Cython to use the "visitor pattern" rather than the recursive pattern  
that it currently uses. This has pros and cons, but I think this  
could be a good thing, and you are certainly convinced that its  
needed to get started on the GSoC stuff. Something doesn't sit right  
about the patch though, but I can't quite put my finger on it. I  
think that it should happen at a higher level than inside the module  
node (i.e. ModuleNode should be "visited" rather than initiate the  
visitors.) As for the "phase shift" between function bodies and the  
ambient code, the ambient analysis phase needs to happen before  
entering function bodies, but I think it could be done immediately  
after at the tail end of the same phase (so all declarations would be  
analyzed before any times need to be looked at) which may make things  
cleaner.

Probably http://wiki.cython.org/enhancements/treevisitors should be  
fleshed out with more specific details and a plan.

- Robert


From mistobaan at gmail.com  Wed May  7 04:49:10 2008
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Tue, 6 May 2008 19:49:10 -0700
Subject: [Cython] Some small phase refactorings
In-Reply-To: <452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
References: <481AF97B.8030305@student.matnat.uio.no>
	<481BFDDA.8060007@behnel.de>
	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>
	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
Message-ID: <e437f2f0805061949r2d769614v680802df983971bc@mail.gmail.com>

> (i.e. ModuleNode should be "visited" rather than initiate the
>  visitors.)

Absolutely.

Fabrizio
-------------
Luck favors the prepared mind. (Pasteur)

From dalcinl at gmail.com  Wed May  7 06:16:03 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 7 May 2008 01:16:03 -0300
Subject: [Cython] WARNING!! cython, classmethod's and Python 2.6
Message-ID: <e7ba66e40805062116m2e08447ye56e37783ce370f8@mail.gmail.com>

Python 2.6 introduces a new feature, a cache for method in C-level
Python types. I believe this will severely interfere with the way
Cython implements 'classmethod' hackery.

I was getting extrange segfaults or very strange behavior (when
accessing a classmethod, then I actually got other method) in my new
mpi4py. Then I started to look at the whole beast, and I believe I've
found the fix. But then I do not kwnow how we could modify Cython for
implementing this, so I'll show and example of what needs to be done.
Look at the very end the 'else' of the pasted generated code, I've
manually added it!!!

  /* "/u/dalcinl/Devel/Cython/tmp/cls.pyx":3
 * cdef class Foo:
 *     def bar(cls): pass
 *     bar = classmethod(bar)             # <<<<<<<<<<<<<<
 *
 */
  __pyx_1 = __Pyx_GetName((PyObject *)__pyx_ptype_3cls_Foo,
__pyx_n_bar); if (unlikely(!__pyx_1)) { /* tb stuff */}
  __pyx_2 = __Pyx_Method_ClassMethod(__pyx_1); if (unlikely(!__pyx_2))
{ /* tb stuff*/}
  Py_DECREF(__pyx_1); __pyx_1 = 0;
  if (PyDict_SetItem((PyObject *)__pyx_ptype_3cls_Foo->tp_dict,
__pyx_n_bar, __pyx_2) < 0) { /* tb stuff*/}
 /* AND  NOW THE FIX !!!!! */
  else {
    (&__pyx_type_3cls_Foo)->tp_flags &= ~Py_TPFLAGS_VALID_VERSION_TAG;
  }

This way, in the next _PyType_Lookup, Python will ignore the method
cache and do a normal lookup and then update the cache.

This seems to solve all the nasty problems I was experienced. Of
course, I'm not completely sure if my fix is the best one. You will
surelly need to ask at Python-Dev with the hope the guy that
implemented this very, very clever hackery on type objects can give
some directions for Cython.

Finally, there is a public function PyType_ClearCache(), but this
function invalidates all the caches starting from the base
'PyBaseObject_Type' type object and recursing on all subclasses.
Perhaps this would be the right way to go, but this would means that
every time you import a cython extension module, you have a full
traversal of type objects clearing caches, and all this because of a
'classmethod' in a cdef class...

Comments?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed May  7 08:30:47 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 07 May 2008 08:30:47 +0200
Subject: [Cython] Visitor patterns and Python 2.4
In-Reply-To: <F23CCCB8-24E1-4B09-B2B9-A56A17DFC4C8@math.washington.edu>
References: <481AFD4D.4050401@student.matnat.uio.no>	<481BFE9A.4080600@behnel.de>
	<F23CCCB8-24E1-4B09-B2B9-A56A17DFC4C8@math.washington.edu>
Message-ID: <48214C97.2030302@student.matnat.uio.no>


> We ship our own Python (2.5.2) with Sage, but I would rather stick  
> with 2.3 compatibility. You might not consider it "important" but my  
> computer (OS X 10.4, yet I know 10.5 is out but I haven't found the  
> time to upgrade yet) has 2.3 by default, as did OS X 10.3.
>   
It is certainly an argument I didn't consider.

Still... Python 2.4 was released 3 1/2 years ago, and I think Cython 
users are going to be power users who will know how to upgrade their Python.

> I very much dislike using __class__.__name__ to decide what functions  
> to call, but it doesn't look like you need decorators to get around  
> this, do you?
>   
Not having decorators makes the overhead of not using the function name 
much greater. I.e. compare:

class MyVisitor(VisitorTransform):
    @pre(cls=FuncDefNode)
    def create_function_scope(self, node):
        ...

(or similar) with something like (changing the match function slightly)

class MyVisitor(VisitorTransform):
    def create_function_scope(self, node):
        ...
    create_function_scope = pre(cls=FuncDefNode, create_function_scope)


The latter one is so heavy to write that I'd rather want to simply do

class MyVisitor(VisitorTransform):
    def pre_FuncDefNode(self, node):
         ....

however I don't think this is as nice. Yes, I know, it is the standard 
visitor pattern, but I think that pattern is the way it is because it is 
made for less powerful languages than Python; using a real class 
references seems better than having (using __name__ or using a 
convention or in some way add extra, additional information to care about).

Also using decorator opens the way for other types of matches (though I 
won't propose a full XPath match again, perhaps simple "attr" matching, 
i.e. to which attribute and parent does the node belong to:

@pre(parent_cls=FuncDefNode, attr="body")
...

-- 
Dag Sverre


From wstein at gmail.com  Wed May  7 08:42:23 2008
From: wstein at gmail.com (William Stein)
Date: Tue, 6 May 2008 23:42:23 -0700
Subject: [Cython] Visitor patterns and Python 2.4
In-Reply-To: <48214C97.2030302@student.matnat.uio.no>
References: <481AFD4D.4050401@student.matnat.uio.no>
	<481BFE9A.4080600@behnel.de>
	<F23CCCB8-24E1-4B09-B2B9-A56A17DFC4C8@math.washington.edu>
	<48214C97.2030302@student.matnat.uio.no>
Message-ID: <85e81ba30805062342g9a125d4m2b48a0d4d987f728@mail.gmail.com>

On Tue, May 6, 2008 at 11:30 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>
>  > We ship our own Python (2.5.2) with Sage, but I would rather stick
>  > with 2.3 compatibility. You might not consider it "important" but my
>  > computer (OS X 10.4, yet I know 10.5 is out but I haven't found the
>  > time to upgrade yet) has 2.3 by default, as did OS X 10.3.
>  >
>  It is certainly an argument I didn't consider.
>
>  Still... Python 2.4 was released 3 1/2 years ago, and I think Cython
>  users are going to be power users who will know how to upgrade their Python.
>
>
>  > I very much dislike using __class__.__name__ to decide what functions
>  > to call, but it doesn't look like you need decorators to get around
>  > this, do you?
>  >
>  Not having decorators makes the overhead of not using the function name
>  much greater. I.e. compare:
>
>  class MyVisitor(VisitorTransform):
>     @pre(cls=FuncDefNode)
>     def create_function_scope(self, node):
>         ...
>
>  (or similar) with something like (changing the match function slightly)
>
>  class MyVisitor(VisitorTransform):
>     def create_function_scope(self, node):
>         ...
>     create_function_scope = pre(cls=FuncDefNode, create_function_scope)
>
>
>  The latter one is so heavy to write that I'd rather want to simply do
>
>  class MyVisitor(VisitorTransform):
>
>     def pre_FuncDefNode(self, node):
>          ....
>
>  however I don't think this is as nice. Yes, I know, it is the standard
>  visitor pattern, but I think that pattern is the way it is because it is
>  made for less powerful languages than Python; using a real class
>  references seems better than having (using __name__ or using a
>  convention or in some way add extra, additional information to care about).
>
>  Also using decorator opens the way for other types of matches (though I
>  won't propose a full XPath match again, perhaps simple "attr" matching,
>  i.e. to which attribute and parent does the node belong to:
>
>  @pre(parent_cls=FuncDefNode, attr="body")
>  ...
>

Two comments:

(1) I personally think Python 2.3 is pretty old at this point, and we
shouldn't worry
too much about supporting it.  I.e., I buy your "Cython users are power
users" argument.

(2) If you use decorators then you'll have to implement them in Cython
so that compiling Cython itself using Cython will work.   That seems
like good motivation.

 -- William

From dagss at student.matnat.uio.no  Wed May  7 08:46:25 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 07 May 2008 08:46:25 +0200
Subject: [Cython] Some small phase refactorings
In-Reply-To: <452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
References: <481AF97B.8030305@student.matnat.uio.no>	<481BFDDA.8060007@behnel.de>	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>
	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
Message-ID: <48215041.7070805@student.matnat.uio.no>


> I think one could put a generic accept function in the Node class  
> that would dispatch based on class name (or some class attribute) if  
> one wanted, rather than having to write it for every class.
>   
Yes, simply move the function. You spoke against using the class name in 
the other thread though. I suppose I could use an attribute; still I 
like decorators much better.

> I think this is good to talk about now. Basically, you want to change  
> Cython to use the "visitor pattern" rather than the recursive pattern  
> that it currently uses. This has pros and cons, but I think this  
> could be a good thing, and you are certainly convinced that its  
> needed to get started on the GSoC stuff. Something doesn't sit right  
> about the patch though, but I can't quite put my finger on it. I  
> think that it should happen at a higher level than inside the module  
> node (i.e. ModuleNode should be "visited" rather than initiate the  
> visitors.) As for the "phase shift" between function bodies and the  
>   
But of course! I'll repeat what I said to Stefan (the context is the 
codelines in ModuleNode.py):

---

Note that this code will almost certainly be moved again and  
rewritten at
some later point (they can't really belong to ModuleNode); but more
refactoring must happen before they can be moved to their proper  
location
and they serve a good purpose where they are for now though.

---

It's a rather big knot, and one has to begin to untangle it somewhere. I 
chose in the middle of ModuleNode.

One could make a (perhaps more natural) decision to start untangle it at 
the top (i.e. in Main.py); however two things spoke against it:

- I wanted to jump right in and have a feel for the most difficult phase 
seperation we were going to transform. If I started at the top, I would 
have to refactor ModuleNode before going on, and that would take some 
time yet still be a (comparatively) easy task -- and I wouldn't get time 
there and then for challenging refactoring the main tree and see what 
the real issues were. So I wouldn't gain any real new knowledge from 
starting "at the right place" (which was needed at this stage, in order 
to start thinking about this refactoring).

-  It will also take some longer to see the immediate gains one gets 
from this starting from the top, but that is fine...

Of course, different needs drive Cython patch inclusion than what I need 
to investigage; and I'm fine with the patch not being included at this 
stage.


-- 
Dag Sverre


From dagss at student.matnat.uio.no  Wed May  7 08:48:38 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 07 May 2008 08:48:38 +0200
Subject: [Cython] Some small phase refactorings
In-Reply-To: <48215041.7070805@student.matnat.uio.no>
References: <481AF97B.8030305@student.matnat.uio.no>	<481BFDDA.8060007@behnel.de>	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
	<48215041.7070805@student.matnat.uio.no>
Message-ID: <482150C6.9070704@student.matnat.uio.no>

An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080507/565d0d72/attachment.htm 

From dagss at student.matnat.uio.no  Wed May  7 08:51:59 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 07 May 2008 08:51:59 +0200
Subject: [Cython] Visitor patterns and Python 2.4
In-Reply-To: <85e81ba30805062342g9a125d4m2b48a0d4d987f728@mail.gmail.com>
References: <481AFD4D.4050401@student.matnat.uio.no>	<481BFE9A.4080600@behnel.de>	<F23CCCB8-24E1-4B09-B2B9-A56A17DFC4C8@math.washington.edu>	<48214C97.2030302@student.matnat.uio.no>
	<85e81ba30805062342g9a125d4m2b48a0d4d987f728@mail.gmail.com>
Message-ID: <4821518F.6050406@student.matnat.uio.no>


> (2) If you use decorators then you'll have to implement them in Cython
> so that compiling Cython itself using Cython will work.   That seems
> like good motivation.
>   
And not too hard either. The funny thing is that I'd really want to have 
decorators available for writing such support :-)

Which brings up the question of a parser...will raise it in a different 
thread.

-- 
Dag Sverre


From dagss at student.matnat.uio.no  Wed May  7 09:00:21 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 07 May 2008 09:00:21 +0200
Subject: [Cython] PyPy parser
Message-ID: <48215385.50207@student.matnat.uio.no>

We all wish Fabrizio had got his SoC. The question is though, what do we 
do now?

I was thinking about if it is scaled down a lot and one only considers 
the PyPy parser and a parser consumer that builds the existing Pyrex 
tree (drop-in replacement of Scanning.py and Parsing.py). How much work 
would that be?

It seems wasteful to put any more work into the Cython parser when PyPy 
is already maintaining one, and there needs to be a few changes in the 
parser this summer (type arguments, probably with statements and 
decorators) so it seems like a good time.

Not volunteering to take it on though (I'll have a GSoC to care for) but 
I'd like to know if there are any plans in this area (or if we can make 
one).

-- 
Dag Sverre


From stefan_ml at behnel.de  Wed May  7 09:51:41 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 7 May 2008 09:51:41 +0200 (CEST)
Subject: [Cython] PyPy parser
In-Reply-To: <48215385.50207@student.matnat.uio.no>
References: <48215385.50207@student.matnat.uio.no>
Message-ID: <10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> We all wish Fabrizio had got his SoC. The question is though, what do we
> do now?
>
> I was thinking about if it is scaled down a lot and one only considers
> the PyPy parser and a parser consumer that builds the existing Pyrex
> tree (drop-in replacement of Scanning.py and Parsing.py). How much work
> would that be?

Implied question: how much work is it to keep in sync with the PyPy parser
development? We'd have to extend the parser to understand C types and
'cdef', so the first thing to check is if there's a way to do that without
having to patch into the parser too heavily. So, even in the long term,
the PyPy parser does not come for free.

We're actually lucky we have a couple of compiler tests by now, but the
first thing to do before starting such a project is growing the test
suite. We should consider including the test suites of Python and PyPy,
but that still wouldn't give us enough tests for the stuff that is most
important here: the C type integration, which is neither a part of Python
nor PyPy.

To some extent, the question of parser rewriting reminds me of this:

http://www.joelonsoftware.com/articles/fog0000000069.html

Doesn't "if it ain't broken, don't fix it" apply here?

Stefan


From dagss at student.matnat.uio.no  Wed May  7 10:01:35 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 07 May 2008 10:01:35 +0200
Subject: [Cython] PyPy parser
In-Reply-To: <10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <482161DF.9010108@student.matnat.uio.no>


> We're actually lucky we have a couple of compiler tests by now, but the
> first thing to do before starting such a project is growing the test
> suite. We should consider including the test suites of Python and PyPy,
> but that still wouldn't give us enough tests for the stuff that is most
> important here: the C type integration, which is neither a part of Python
> nor PyPy.
>
> To some extent, the question of parser rewriting reminds me of this:
>
> http://www.joelonsoftware.com/articles/fog0000000069.html
>
> Doesn't "if it ain't broken, don't fix it" apply here?
>   
Yes, I think you're right :-)

-- 
Dag Sverre


From mistobaan at gmail.com  Wed May  7 17:08:34 2008
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Wed, 7 May 2008 08:08:34 -0700
Subject: [Cython] PyPy parser
In-Reply-To: <482161DF.9010108@student.matnat.uio.no>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
Message-ID: <e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>

>  > Doesn't "if it ain't broken, don't fix it" apply here?
The fact is that it actually breaks with Python 3.0 :)

I see three solutions:
1)
to copy main parts of pypy's parser and put them inside cython
 * Pro:
         more easily to modify and clean
         specialized for cython purpose
 * Cons:
        duplicate code among libraries

2)
main parser logic from the pypy repository
compatibility layer inside the cython repository

* Pro: No code duplication
* Cons: Cython dependent of another library

3) Tell to pypy's guys to create a python parser library for python
from pypy's parser
     and both cython and python use it.
    They will have python 3k support as well as cython

I will try to do my best in working on this parser.

What do you think ?

Fabrizio
--------------------------
Luck favors the prepared mind. (Pasteur)

From robertwb at math.washington.edu  Wed May  7 20:01:05 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 7 May 2008 11:01:05 -0700
Subject: [Cython] PyPy parser
In-Reply-To: <e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
	<e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
Message-ID: <AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>

On May 7, 2008, at 8:08 AM, Fabrizio Milo aka misto wrote:

>>> Doesn't "if it ain't broken, don't fix it" apply here?
> The fact is that it actually breaks with Python 3.0 :)

Yes, this is true. Actually modifying the parser to accept, for  
example, decorators will be easy. With statements are already  
implemented at the parser level, as are inner functions (just have to  
remove the line disallowing them). Parameterized types shouldn't be  
too hard to do either. For about every 2.x deficiency I can think of  
modifying the parser is going to be by far the easy part of the job.  
I am not aware of any huge changes in 3.0, but there may be some.

If we decide to replace the parser, I would like to see one that's  
automatically generated from a grammar file (rather than hand coded)  
and lends itself well to optimization when we compile cython with  
itself (as parsing is a huge portion of the total runtime).

> I see three solutions:
> 1)
> to copy main parts of pypy's parser and put them inside cython
>  * Pro:
>          more easily to modify and clean
>          specialized for cython purpose
>  * Cons:
>         duplicate code among libraries

This seems like the best idea, depending on how much code we are  
talking about here, and also as long as there aren't any license issues.

> 2)
> main parser logic from the pypy repository
> compatibility layer inside the cython repository
>
> * Pro: No code duplication
> * Cons: Cython dependent of another library

I really don't want to increase dependancies, and I have the feeling  
that the "compatibility layer" wouldn't be very clean.

> 3) Tell to pypy's guys to create a python parser library for python
> from pypy's parser
>      and both cython and python use it.
>     They will have python 3k support as well as cython

I'm not sure why they would to this for us, and would we rely on them  
to do it repeatedly as the language changes?

> I will try to do my best in working on this parser.
>
> What do you think ?
>
> Fabrizio

I don't think the discussion would be complete without at least  
looking at the possibility of using the parser that comes with Python  
itself.

- Robert


From mistobaan at gmail.com  Thu May  8 00:37:05 2008
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Wed, 7 May 2008 15:37:05 -0700
Subject: [Cython] PyPy parser
In-Reply-To: <AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
	<e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
	<AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>
Message-ID: <e437f2f0805071537n676db015x8e4decc6dafb3a1@mail.gmail.com>

>  I don't think the discussion would be complete without at least
>  looking at the possibility of using the parser that comes with Python
>  itself.

The one written in C ? and extending it with Cython itself ?
not sure to understand :|

Fabrizio

From jek-gmane at kleckner.net  Thu May  8 03:07:26 2008
From: jek-gmane at kleckner.net (Jim Kleckner)
Date: Wed, 07 May 2008 18:07:26 -0700
Subject: [Cython] Offtopic: Good Python IDEs?
In-Reply-To: <47FF20EC.4040803@behnel.de>
References: <47FE4418.40703@student.matnat.uio.no> <47FF20EC.4040803@behnel.de>
Message-ID: <fvtjok$73f$1@ger.gmane.org>

Stefan Behnel wrote:
...
> For debugging, I mostly use print and unit tests for Python and print, tests
> and valgrind for Cython, so I can't really comment on the debugging
> environments (which actually *are* available for emacs).

Stefan, do you have some tips for Valgrind?
Do you have a suppression file that you like?

I ran valgrind again and found a sea of messages with a small number
of suspicious ones.
Two of them have the form:
  __pyx_4 = PyObject_Call(__pyx_12, __pyx_empty_tuple, NULL); if 
(unlikely(!__pyx_4)) {
which suggests that the NULL is fooling Valgrind in some way.

This page might be a good place to put such things as suppression files:
  http://wiki.cython.org/UsingValgrindToDebug


From jek-gmane at kleckner.net  Thu May  8 03:12:57 2008
From: jek-gmane at kleckner.net (Jim Kleckner)
Date: Wed, 07 May 2008 18:12:57 -0700
Subject: [Cython] Offtopic: Good Python IDEs?
In-Reply-To: <fvtjok$73f$1@ger.gmane.org>
References: <47FE4418.40703@student.matnat.uio.no>
	<47FF20EC.4040803@behnel.de> <fvtjok$73f$1@ger.gmane.org>
Message-ID: <fvtk30$73f$2@ger.gmane.org>

Jim Kleckner wrote:
> Stefan Behnel wrote:
> ...
>> For debugging, I mostly use print and unit tests for Python and print, tests
>> and valgrind for Cython, so I can't really comment on the debugging
>> environments (which actually *are* available for emacs).
> 
> Stefan, do you have some tips for Valgrind?
> Do you have a suppression file that you like?
> 
> I ran valgrind again and found a sea of messages with a small number
> of suspicious ones.
> Two of them have the form:
>   __pyx_4 = PyObject_Call(__pyx_12, __pyx_empty_tuple, NULL); if 
> (unlikely(!__pyx_4)) {
> which suggests that the NULL is fooling Valgrind in some way.
> 
> This page might be a good place to put such things as suppression files:
>   http://wiki.cython.org/UsingValgrindToDebug
> 

This page explains some of the PyMalloc issues:
  http://svn.python.org/projects/python/trunk/Misc/README.valgrind


From stefan_ml at behnel.de  Thu May  8 07:46:29 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 08 May 2008 07:46:29 +0200
Subject: [Cython] Debugging with valgrind
In-Reply-To: <fvtk30$73f$2@ger.gmane.org>
References: <47FE4418.40703@student.matnat.uio.no>	<47FF20EC.4040803@behnel.de>
	<fvtjok$73f$1@ger.gmane.org> <fvtk30$73f$2@ger.gmane.org>
Message-ID: <482293B5.3080802@behnel.de>

Hi,

Jim Kleckner wrote:
> Jim Kleckner wrote:
>> Stefan Behnel wrote:
>> ...
>>> For debugging, I mostly use print and unit tests for Python and print, tests
>>> and valgrind for Cython, so I can't really comment on the debugging
>>> environments (which actually *are* available for emacs).
>> Stefan, do you have some tips for Valgrind?
>> Do you have a suppression file that you like?
>>
>> I ran valgrind again and found a sea of messages with a small number
>> of suspicious ones.
>> Two of them have the form:
>>   __pyx_4 = PyObject_Call(__pyx_12, __pyx_empty_tuple, NULL); if 
>> (unlikely(!__pyx_4)) {
>> which suggests that the NULL is fooling Valgrind in some way.
>>
>> This page might be a good place to put such things as suppression files:
>>   http://wiki.cython.org/UsingValgrindToDebug
>>
> 
> This page explains some of the PyMalloc issues:
>   http://svn.python.org/projects/python/trunk/Misc/README.valgrind

Yes, that's where we took the suppression file from that we use for lxml.

http://codespeak.net/svn/lxml/trunk/valgrind-python.supp

Stefan


From dalcinl at gmail.com  Thu May  8 08:05:39 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 03:05:39 -0300
Subject: [Cython] working patch for generating code targeting Py 2.6 and Py
	3.0
Message-ID: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>

The last four hours (3:00 AM right now here at Argentina) I've been
working in a patch for enabling Cython generate code working with
Python 2.6 and Python 3.0.

Until now, the generated code (at least for the full mpi4py project)
compiles and link fine with no errors.

However, I have a big  problems I do not know how to 'fix' in Cython.
It is related to unbound methods disapearing in Py3K. Then, normal
python classes does not work, but the cdef ones are fine.

Is there any interest on this to go mainstream? I was very
conservative about the PyString/PyUnicode issue. The right one is used
in a place-by-place base. Of course, because of this, I have to pass
'bytes' to MPI, and I get 'bytes' from the C calls.

Finally, I'm completelly sure that I've not fixed all the relevant
parts, but this is IMHO a good starting point.




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu May  8 08:56:19 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 08 May 2008 08:56:19 +0200
Subject: [Cython] working patch for generating code targeting Py 2.6 and
 Py	3.0
In-Reply-To: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
Message-ID: <4822A413.8060902@behnel.de>

Hi,

Lisandro Dalcin wrote:
> The last four hours (3:00 AM right now here at Argentina) I've been
> working in a patch for enabling Cython generate code working with
> Python 2.6 and Python 3.0.
> 
> Until now, the generated code (at least for the full mpi4py project)
> compiles and link fine with no errors.
> 
> However, I have a big  problems I do not know how to 'fix' in Cython.
> It is related to unbound methods disapearing in Py3K. Then, normal
> python classes does not work, but the cdef ones are fine.

These are the kind of things that we planned to address at the dev1 workshop.
The C-API of Python is expected to stabilize next month, I don't know how
stable these parts currently are.


> Is there any interest on this to go mainstream?

Sure, totally!


> I was very
> conservative about the PyString/PyUnicode issue. The right one is used
> in a place-by-place base. Of course, because of this, I have to pass
> 'bytes' to MPI, and I get 'bytes' from the C calls.

Yes, I think this is the right way to deal with it. Python2 was very lax in
terms of semantics here, so the two have to be separated on a case-by-case basis.


> Finally, I'm completelly sure that I've not fixed all the relevant
> parts, but this is IMHO a good starting point.

Is it one big patch or did you/can you split it up?

As this is potentially a big change, trac is the wrong place to discuss it. We
should put up an official Py3 branch that people can actively work on without
impacting the main trunk, so that we can merge working stuff gradually. Can
you send a bundle against cython-devel to me and Robert for now? He can set it up.

Stefan

From dalcinl at gmail.com  Thu May  8 17:33:55 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 12:33:55 -0300
Subject: [Cython] Fwd: working patch for generating code targeting Py 2.6
	and Py 3.0
In-Reply-To: <4822A413.8060902@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
Message-ID: <e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>

Well, I send here a patch obtained with 'hg diff'. Of course, before
seeting up a new repo with this patch as a base, look at it for
possible pitfalls regarding to style. Additionally, take care of the
comments below.

- The PyNumberMethods stuff is the hard part. Some of them changed it
name (nb_nonzero -> nb_bool) in py3k. I did not managed this name
change yet. Other slots are gone, I basically added a flag for manage
this. The realy anoying part is that some slots are UNUSED, but they
are still there in the struct. Could you ask for clarifications in
Python-Dev about all this crap? IMHO, all sloots should be keept, or
all unused in py3k should be removed.

- PyMember_New changed, I've tried to emulate this with a macro, but
normal classes are not working anyway. PyClass_New is gone, I've
emulated it with a call to 'type' (but in C, of course). Perhaps the
class dict should be filled before calling type(name, bases, dict)??
But then I do not know how to hack cython to do that...

- Finally, in the very clever part of traceback hackery, the call to
Py_CodeNew receives 'filenames' created with PyUnicode_AsString. I do
not know what the C preprocessor uses for enconding __FILE__ macro, it
its always ASCII, then all is fine, if not, the filesystem encoding
should be taken into account.

Regards,


On 5/8/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>
>  Lisandro Dalcin wrote:
>  > The last four hours (3:00 AM right now here at Argentina) I've been
>  > working in a patch for enabling Cython generate code working with
>  > Python 2.6 and Python 3.0.
>  >
>  > Until now, the generated code (at least for the full mpi4py project)
>  > compiles and link fine with no errors.
>  >
>  > However, I have a big  problems I do not know how to 'fix' in Cython.
>  > It is related to unbound methods disapearing in Py3K. Then, normal
>  > python classes does not work, but the cdef ones are fine.
>
>
> These are the kind of things that we planned to address at the dev1 workshop.
>  The C-API of Python is expected to stabilize next month, I don't know how
>  stable these parts currently are.
>
>
>
>  > Is there any interest on this to go mainstream?
>
>
> Sure, totally!
>
>
>
>  > I was very
>  > conservative about the PyString/PyUnicode issue. The right one is used
>  > in a place-by-place base. Of course, because of this, I have to pass
>  > 'bytes' to MPI, and I get 'bytes' from the C calls.
>
>
> Yes, I think this is the right way to deal with it. Python2 was very lax in
>  terms of semantics here, so the two have to be separated on a case-by-case basis.
>
>
>
>  > Finally, I'm completelly sure that I've not fixed all the relevant
>  > parts, but this is IMHO a good starting point.
>
>
> Is it one big patch or did you/can you split it up?
>
>  As this is potentially a big change, trac is the wrong place to discuss it. We
>  should put up an official Py3 branch that people can actively work on without
>  impacting the main trunk, so that we can merge working stuff gradually. Can
>  you send a bundle against cython-devel to me and Robert for now? He can set it up.
>
>  Stefan
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cython-py3k.patch
Type: application/octet-stream
Size: 21182 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080508/c5896734/attachment-0001.obj 

From robertwb at math.washington.edu  Thu May  8 17:37:43 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 08:37:43 -0700
Subject: [Cython] working patch for generating code targeting Py 2.6 and
	Py	3.0
In-Reply-To: <4822A413.8060902@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
Message-ID: <800C6DA2-4F48-4BE2-9647-623E09146A63@math.washington.edu>

On May 7, 2008, at 11:56 PM, Stefan Behnel wrote:

> Hi,
>
> Lisandro Dalcin wrote:
>> The last four hours (3:00 AM right now here at Argentina) I've been
>> working in a patch for enabling Cython generate code working with
>> Python 2.6 and Python 3.0.
>>
>> Until now, the generated code (at least for the full mpi4py project)
>> compiles and link fine with no errors.
>>
>> However, I have a big  problems I do not know how to 'fix' in Cython.
>> It is related to unbound methods disapearing in Py3K. Then, normal
>> python classes does not work, but the cdef ones are fine.
>
> These are the kind of things that we planned to address at the dev1  
> workshop.
> The C-API of Python is expected to stabilize next month, I don't  
> know how
> stable these parts currently are.
>
>
>> Is there any interest on this to go mainstream?
>
> Sure, totally!

Yes, we're very interested! Is it backwards compatible too?

>> I was very
>> conservative about the PyString/PyUnicode issue. The right one is  
>> used
>> in a place-by-place base. Of course, because of this, I have to pass
>> 'bytes' to MPI, and I get 'bytes' from the C calls.
>
> Yes, I think this is the right way to deal with it. Python2 was  
> very lax in
> terms of semantics here, so the two have to be separated on a case- 
> by-case basis.
>
>
>> Finally, I'm completelly sure that I've not fixed all the relevant
>> parts, but this is IMHO a good starting point.
>
> Is it one big patch or did you/can you split it up?
>
> As this is potentially a big change, trac is the wrong place to  
> discuss it. We
> should put up an official Py3 branch that people can actively work  
> on without
> impacting the main trunk, so that we can merge working stuff  
> gradually. Can
> you send a bundle against cython-devel to me and Robert for now? He  
> can set it up.

Trac is a good place to put the patches and status though. Are you  
sure we want a separate branch for this, as we want to retain Py2  
compatibility as we work on this.

- Robert



From robertwb at math.washington.edu  Thu May  8 17:55:49 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 08:55:49 -0700
Subject: [Cython] Fwd: working patch for generating code targeting Py
	2.6 and Py 3.0
In-Reply-To: <e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
Message-ID: <D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>

On May 8, 2008, at 8:33 AM, Lisandro Dalcin wrote:

> Well, I send here a patch obtained with 'hg diff'.

Thanks. I briefly glanced at it and it looks good. What you probably  
want to do is commit and then export the patch (that way it will have  
some history attached, and you'll get credit for it.)

FYI, I am preparing for my general exams that I take in a couple of  
weeks, so won't have tons of time to put into Cython in the near  
future, but this looks like really good work.

> Of course, before seeting up a new repo with this patch as a base,  
> look at it for
> possible pitfalls regarding to style. Additionally, take care of the
> comments below.

One remark I had is that perhaps more care  needs to be done with  
"PyInt_CheckExact" as in Py2 this implies that the size fits into a  
long, so it's not quite the same as PyLong_CheckExact.

> - The PyNumberMethods stuff is the hard part. Some of them changed it
> name (nb_nonzero -> nb_bool) in py3k. I did not managed this name
> change yet. Other slots are gone, I basically added a flag for manage
> this. The realy anoying part is that some slots are UNUSED, but they
> are still there in the struct. Could you ask for clarifications in
> Python-Dev about all this crap? IMHO, all sloots should be keept, or
> all unused in py3k should be removed.

Could you elaborate a bit more on what you mean by "unused?" If  
they're still in the struct, I'd think we would need to keep them so  
it has the right size/layout, right?

> - PyMember_New changed, I've tried to emulate this with a macro, but
> normal classes are not working anyway. PyClass_New is gone, I've
> emulated it with a call to 'type' (but in C, of course). Perhaps the
> class dict should be filled before calling type(name, bases, dict)??
> But then I do not know how to hack cython to do that...

Filling the dict ahead of time might actually work better...one would  
execute the body in a temporary scope and then construct the class.  
This would bring things more in line with how Python does things.

> - Finally, in the very clever part of traceback hackery, the call to
> Py_CodeNew receives 'filenames' created with PyUnicode_AsString. I do
> not know what the C preprocessor uses for enconding __FILE__ macro, it
> its always ASCII, then all is fine, if not, the filesystem encoding
> should be taken into account.

I actually have no idea on this one--maybe there's some system call  
that can give this information? Stefan would probably know better  
than me.

>
> Regards,
>
>
> On 5/8/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>>
>>  Lisandro Dalcin wrote:
>>> The last four hours (3:00 AM right now here at Argentina) I've been
>>> working in a patch for enabling Cython generate code working with
>>> Python 2.6 and Python 3.0.
>>>
>>> Until now, the generated code (at least for the full mpi4py project)
>>> compiles and link fine with no errors.
>>>
>>> However, I have a big  problems I do not know how to 'fix' in  
>>> Cython.
>>> It is related to unbound methods disapearing in Py3K. Then, normal
>>> python classes does not work, but the cdef ones are fine.
>>
>>
>> These are the kind of things that we planned to address at the  
>> dev1 workshop.
>>  The C-API of Python is expected to stabilize next month, I don't  
>> know how
>>  stable these parts currently are.
>>
>>
>>
>>> Is there any interest on this to go mainstream?
>>
>>
>> Sure, totally!
>>
>>
>>
>>> I was very
>>> conservative about the PyString/PyUnicode issue. The right one is  
>>> used
>>> in a place-by-place base. Of course, because of this, I have to pass
>>> 'bytes' to MPI, and I get 'bytes' from the C calls.
>>
>>
>> Yes, I think this is the right way to deal with it. Python2 was  
>> very lax in
>>  terms of semantics here, so the two have to be separated on a  
>> case-by-case basis.
>>
>>
>>
>>> Finally, I'm completelly sure that I've not fixed all the relevant
>>> parts, but this is IMHO a good starting point.
>>
>>
>> Is it one big patch or did you/can you split it up?
>>
>>  As this is potentially a big change, trac is the wrong place to  
>> discuss it. We
>>  should put up an official Py3 branch that people can actively  
>> work on without
>>  impacting the main trunk, so that we can merge working stuff  
>> gradually. Can
>>  you send a bundle against cython-devel to me and Robert for now?  
>> He can set it up.
>>
>>  Stefan
>>  _______________________________________________
>>  Cython-dev mailing list
>>  Cython-dev at codespeak.net
>>  http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594<cython- 
> py3k.patch>_______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Thu May  8 17:57:09 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 08 May 2008 17:57:09 +0200
Subject: [Cython] working patch for generating code targeting Py 2.6 and
 Py 3.0
In-Reply-To: <800C6DA2-4F48-4BE2-9647-623E09146A63@math.washington.edu>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<800C6DA2-4F48-4BE2-9647-623E09146A63@math.washington.edu>
Message-ID: <482322D5.3020600@behnel.de>

Hi,

Robert Bradshaw wrote:
> Trac is a good place to put the patches and status though.

Sure.


> Are you sure
> we want a separate branch for this, as we want to retain Py2
> compatibility as we work on this.

I do not expect this to be finished with a single patch, so there will be
things that may break somewhere in between and there may be things that
require larger refactorings. Keeping cython-devel somewhat stable and having a
separate branch where things can become unstable along the way helps us in
keeping up the ability to continue the normal development without being
directly impacted by Py3 work.

The idea is to merge things over step-by-step when they prove to be correct
and "stable enough".

Stefan


From robertwb at math.washington.edu  Thu May  8 18:00:07 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 09:00:07 -0700
Subject: [Cython] PyPy parser
In-Reply-To: <e437f2f0805071537n676db015x8e4decc6dafb3a1@mail.gmail.com>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
	<e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
	<AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>
	<e437f2f0805071537n676db015x8e4decc6dafb3a1@mail.gmail.com>
Message-ID: <743600FF-F1BC-48F4-BDC6-EC89A99396BA@math.washington.edu>

On May 7, 2008, at 3:37 PM, Fabrizio Milo aka misto wrote:

>>  I don't think the discussion would be complete without at least
>>  looking at the possibility of using the parser that comes with  
>> Python
>>  itself.
>
> The one written in C ? and extending it with Cython itself ?
> not sure to understand :|

There are some python parsing files as well, see, for example, src/ 
Lib/compiler in the Python sources. We could perhaps even use some  
stuff (classes, ast vistor, etc) from the compiler modules. On the  
other hand, we don't want to rely to heavily on the specific version  
of Python the user has.

A simple grammar -> parser module like you've talked about before  
seems ideal, and would probably be fairly compact.

- Robert


From robertwb at math.washington.edu  Thu May  8 18:01:39 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 09:01:39 -0700
Subject: [Cython] Debugging with valgrind
In-Reply-To: <482293B5.3080802@behnel.de>
References: <47FE4418.40703@student.matnat.uio.no>	<47FF20EC.4040803@behnel.de>
	<fvtjok$73f$1@ger.gmane.org> <fvtk30$73f$2@ger.gmane.org>
	<482293B5.3080802@behnel.de>
Message-ID: <5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>

On May 7, 2008, at 10:46 PM, Stefan Behnel wrote:
> Hi,
>
> Jim Kleckner wrote:
>> Jim Kleckner wrote:
>>> Stefan Behnel wrote:
>>> ...
>>>> For debugging, I mostly use print and unit tests for Python and  
>>>> print, tests
>>>> and valgrind for Cython, so I can't really comment on the debugging
>>>> environments (which actually *are* available for emacs).
>>> Stefan, do you have some tips for Valgrind?
>>> Do you have a suppression file that you like?
>>>
>>> I ran valgrind again and found a sea of messages with a small number
>>> of suspicious ones.
>>> Two of them have the form:
>>>   __pyx_4 = PyObject_Call(__pyx_12, __pyx_empty_tuple, NULL); if
>>> (unlikely(!__pyx_4)) {
>>> which suggests that the NULL is fooling Valgrind in some way.
>>>
>>> This page might be a good place to put such things as suppression  
>>> files:
>>>   http://wiki.cython.org/UsingValgrindToDebug
>>>
>>
>> This page explains some of the PyMalloc issues:
>>   http://svn.python.org/projects/python/trunk/Misc/README.valgrind
>
> Yes, that's where we took the suppression file from that we use for  
> lxml.
>
> http://codespeak.net/svn/lxml/trunk/valgrind-python.supp

I bet turning off string interning, etc. would help clean up lots of  
the noise too.

- Robert


From robertwb at math.washington.edu  Thu May  8 18:08:43 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 09:08:43 -0700
Subject: [Cython] Abandon support for Python 2.3?
Message-ID: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>

As mentioned in another thread, we are considering requiring Python  
2.4 or greater to run the Cython compiler, mostly so we can use  
decorators in the compiler code. Is anyone still using Cython with  
this version of Python?

- Robert


From mabshoff at googlemail.com  Thu May  8 17:30:41 2008
From: mabshoff at googlemail.com (Michael Abshoff)
Date: Thu, 08 May 2008 17:30:41 +0200
Subject: [Cython] Debugging with valgrind
In-Reply-To: <5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
References: <47FE4418.40703@student.matnat.uio.no>	<47FF20EC.4040803@behnel.de>	<fvtjok$73f$1@ger.gmane.org>
	<fvtk30$73f$2@ger.gmane.org>	<482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
Message-ID: <48231CA1.5080702@googlemail.com>

Robert Bradshaw wrote:
> On May 7, 2008, at 10:46 PM, Stefan Behnel wrote:
>   

Hi,

>> Yes, that's where we took the suppression file from that we use for  
>> lxml.
>>
>> http://codespeak.net/svn/lxml/trunk/valgrind-python.supp
>>     
>
>   
I am actually pretty paranoid about using suppression files and I do 
prefer recompiling python "--without-pymalloc". But that is likely to be 
something most people will not do since they do not build their own 
python interpreter from sources.

> I bet turning off string interning, etc. would help clean up lots of  
> the noise too.
>   
Hmm, I have seen a lot of noise with Cython 0.9.6.14 even when I set

generate_cleanup_code = 3

In Sage about 100 doctests segfault at exit [out of 900] with that 
setting on, nearly all of them involved matrix[2].pyx somehow IIRC. 
Something for dev1 maybe? It is my understanding that this isn't the 
default of Cython, but more which classes import other classes and in 
which order.

Anyhow: How do I turn off "string interning"?
> - Robert
>   
Cheers,

Michael

> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>   


From dalcinl at gmail.com  Thu May  8 18:22:25 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 13:22:25 -0300
Subject: [Cython] Debugging with valgrind
In-Reply-To: <48231CA1.5080702@googlemail.com>
References: <47FE4418.40703@student.matnat.uio.no> <47FF20EC.4040803@behnel.de>
	<fvtjok$73f$1@ger.gmane.org> <fvtk30$73f$2@ger.gmane.org>
	<482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
Message-ID: <e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>

I'm using Python 2.5.1 and run valgrind with the
Misc/valgrind-python.supp inside the Python source distribution, and I
do not get any warnings, even if I activate cleanup code. Of course,
my project is not cimporting anything.


On 5/8/08, Michael Abshoff <mabshoff at googlemail.com> wrote:
> Robert Bradshaw wrote:
>  > On May 7, 2008, at 10:46 PM, Stefan Behnel wrote:
>  >
>
>  Hi,
>
>
> >> Yes, that's where we took the suppression file from that we use for
>  >> lxml.
>  >>
>  >> http://codespeak.net/svn/lxml/trunk/valgrind-python.supp
>  >>
>  >
>  >
>
> I am actually pretty paranoid about using suppression files and I do
>  prefer recompiling python "--without-pymalloc". But that is likely to be
>  something most people will not do since they do not build their own
>  python interpreter from sources.
>
>
>  > I bet turning off string interning, etc. would help clean up lots of
>  > the noise too.
>  >
>
> Hmm, I have seen a lot of noise with Cython 0.9.6.14 even when I set
>
>  generate_cleanup_code = 3
>
>  In Sage about 100 doctests segfault at exit [out of 900] with that
>  setting on, nearly all of them involved matrix[2].pyx somehow IIRC.
>  Something for dev1 maybe? It is my understanding that this isn't the
>  default of Cython, but more which classes import other classes and in
>  which order.
>
>  Anyhow: How do I turn off "string interning"?
>  > - Robert
>  >
>  Cheers,
>
>
>  Michael
>
>
>  > _______________________________________________
>  > Cython-dev mailing list
>  > Cython-dev at codespeak.net
>  > http://codespeak.net/mailman/listinfo/cython-dev
>  >
>  >
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu May  8 18:26:01 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 09:26:01 -0700
Subject: [Cython] Debugging with valgrind
In-Reply-To: <48231CA1.5080702@googlemail.com>
References: <47FE4418.40703@student.matnat.uio.no>	<47FF20EC.4040803@behnel.de>	<fvtjok$73f$1@ger.gmane.org>
	<fvtk30$73f$2@ger.gmane.org>	<482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
Message-ID: <2742AEAA-F1CF-404B-9190-69284823B8D5@math.washington.edu>

On May 8, 2008, at 8:30 AM, Michael Abshoff wrote:

> Robert Bradshaw wrote:
>> On May 7, 2008, at 10:46 PM, Stefan Behnel wrote:
>>
>
> Hi,
>
>>> Yes, that's where we took the suppression file from that we use for
>>> lxml.
>>>
>>> http://codespeak.net/svn/lxml/trunk/valgrind-python.supp
>>>
>>
>>
> I am actually pretty paranoid about using suppression files and I do
> prefer recompiling python "--without-pymalloc". But that is likely  
> to be
> something most people will not do since they do not build their own
> python interpreter from sources.
>
>> I bet turning off string interning, etc. would help clean up lots of
>> the noise too.
>>
> Hmm, I have seen a lot of noise with Cython 0.9.6.14 even when I set
>
> generate_cleanup_code = 3
>
> In Sage about 100 doctests segfault at exit [out of 900] with that
> setting on, nearly all of them involved matrix[2].pyx somehow IIRC.
> Something for dev1 maybe? It is my understanding that this isn't the
> default of Cython, but more which classes import other classes and in
> which order.

Cleanup is inherently dangerous... Suppose one has two modules a and  
b, with classes A and B respectively, both of which have non-empty  
__del__ or __dealloc__ methods. Then if a holds an instance of B and  
b holds an instance of A (think parent-element) then it is impossible  
to cleanup a or b after the other module has been deallocated.

> Anyhow: How do I turn off "string interning"?

It's in Compiler/Options.py. This may some segfaults go away too, but  
slows things down.

- Robert


From robertwb at math.washington.edu  Thu May  8 18:33:49 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 09:33:49 -0700
Subject: [Cython] Some small phase refactorings
In-Reply-To: <48215041.7070805@student.matnat.uio.no>
References: <481AF97B.8030305@student.matnat.uio.no>	<481BFDDA.8060007@behnel.de>	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>
	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
	<48215041.7070805@student.matnat.uio.no>
Message-ID: <5303982D-51B8-4D6F-B363-BDC8C9729892@math.washington.edu>

On May 6, 2008, at 11:46 PM, Dag Sverre Seljebotn wrote:

>> I think one could put a generic accept function in the Node class
>> that would dispatch based on class name (or some class attribute) if
>> one wanted, rather than having to write it for every class.
>>
> Yes, simply move the function. You spoke against using the class  
> name in
> the other thread though. I suppose I could use an attribute; still I
> like decorators much better.

Yes, I do to. I'm not totally dead set against them either. I just  
sent out a (much more explicit) email on this note, if it doesn't get  
any replies than I guess that's an indication. I am actually a bit  
worried about how the use of decorators will impact the ability of  
Cython to compile itself into ultra-efficient C.

>> I think this is good to talk about now. Basically, you want to change
>> Cython to use the "visitor pattern" rather than the recursive pattern
>> that it currently uses. This has pros and cons, but I think this
>> could be a good thing, and you are certainly convinced that its
>> needed to get started on the GSoC stuff. Something doesn't sit right
>> about the patch though, but I can't quite put my finger on it. I
>> think that it should happen at a higher level than inside the module
>> node (i.e. ModuleNode should be "visited" rather than initiate the
>> visitors.) As for the "phase shift" between function bodies and the
>>
> But of course! I'll repeat what I said to Stefan (the context is the
> codelines in ModuleNode.py):
>
> ---
>
> Note that this code will almost certainly be moved again and
> rewritten at
> some later point (they can't really belong to ModuleNode); but more
> refactoring must happen before they can be moved to their proper
> location
> and they serve a good purpose where they are for now though.
>
> ---
>
> It's a rather big knot, and one has to begin to untangle it  
> somewhere. I
> chose in the middle of ModuleNode.
>
> One could make a (perhaps more natural) decision to start untangle  
> it at
> the top (i.e. in Main.py); however two things spoke against it:
>
> - I wanted to jump right in and have a feel for the most difficult  
> phase
> seperation we were going to transform. If I started at the top, I  
> would
> have to refactor ModuleNode before going on, and that would take some
> time yet still be a (comparatively) easy task -- and I wouldn't get  
> time
> there and then for challenging refactoring the main tree and see what
> the real issues were. So I wouldn't gain any real new knowledge from
> starting "at the right place" (which was needed at this stage, in  
> order
> to start thinking about this refactoring).
>
> -  It will also take some longer to see the immediate gains one gets
> from this starting from the top, but that is fine...

You're reasoning makes sense here, as long as it doesn't make it more  
difficult to move things up to the top at the end.

> Of course, different needs drive Cython patch inclusion than what I  
> need
> to investigage; and I'm fine with the patch not being included at this
> stage.


It just didn't seem to add anything except another level of  
indirection, but that will of course eventually be needed. There is a  
lot of re-factoring that will need to be done. I've just got other  
obligations (general exam) to be able to put much time/though into  
this for the next two weeks or so :-(.

- Robert


From robertwb at math.washington.edu  Thu May  8 18:39:07 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 09:39:07 -0700
Subject: [Cython] __getattribute__
In-Reply-To: <20080505033749.GF4087@tilt>
References: <20080430004208.GJ15181@tilt>
	<5EB02A25-EE20-4FAB-A780-19DEC5EFA16A@math.washington.edu>
	<20080504051532.GA7720@tilt> <20080505033749.GF4087@tilt>
Message-ID: <F9F75765-8B64-447B-A76E-BCF9386723F6@math.washington.edu>

Thanks! It works great for me.

On May 4, 2008, at 8:37 PM, Peter Todd wrote:
> On Sun, May 04, 2008 at 01:15:32AM -0400, Peter Todd wrote:
>> Here's my first patch. This correctly implements __getattribute__ and
>> __getattr__ in the single class case. FWIW I also have a mercurial  
>> tree
>> if it'd be better to pull from it than apply patches.
>>
>> I'm working on making subclasses behave correctly, I've got test  
>> cases
>> written up showing where things fail, but no solutions to that  
>> written
>> yet. The slot_tp_getattro stuff Stefan mentioned is useful though.
>
> __getattr(ibute)__ support is now working with subclasses. The  
> semantics
> should match Python exactly if my test cases are correct. All that has
> changed from the previous patch is that at compile time base  
> classes are
> checked for __getattr(ibute)__ methods and those methods are used if
> found.
>
> Attached is an hg bundle of the two commits.
>
> -- 
> http://petertodd.org 'peter'[:-1]@petertodd.org  
> <getattr_support.hg>_______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080508/46ab919d/attachment.pgp 

From robert.kern at gmail.com  Thu May  8 18:58:16 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 8 May 2008 11:58:16 -0500
Subject: [Cython] Abandon support for Python 2.3?
In-Reply-To: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>
References: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>
Message-ID: <3d375d730805080958q65d7cd39v22d9f1cd86e1ba5d@mail.gmail.com>

On Thu, May 8, 2008 at 11:08 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> As mentioned in another thread, we are considering requiring Python
> 2.4 or greater to run the Cython compiler, mostly so we can use
> decorators in the compiler code. Is anyone still using Cython with
> this version of Python?

We have been considering moving from Pyrex to Cython for part of
numpy. numpy will be remaining with Python 2.3 for some time.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From dagss at student.matnat.uio.no  Thu May  8 19:17:10 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 May 2008 19:17:10 +0200
Subject: [Cython] A high-level discussion of the Cython internals
Message-ID: <48233596.5020204@student.matnat.uio.no>

This is in response to this statement by Robert:


-- 

I think this is good to talk about now. Basically, you want to change  
Cython to use the "visitor pattern" rather than the recursive pattern  
that it currently uses. This has pros and cons, but I think this  
could be a good thing, and you are certainly convinced that its  
needed to get started on the GSoC stuff.

--

So, leaving the patch and implementation strategies aside for now I'm 
going to attempt discussing this from the mountain top. (If a lot of 
this seems trivial, all the better -- it can then start to serve as 
documentation for a direction, rather than my "private" efforts :-). If 
not, it will be good to discuss it.)

I honestly believe that any time spent discussing and implementing this 
now can repay itself many times later on in saved developer hours and a 
lower learning curve for new developers. If people agree with me in how 
important this is, I think it would be good to perhaps spend some time 
at the beginning of dev1 talking and discussing this (with some slides 
etc.).

Here we go:

Assertion 1: Development of Cython has a potential for becoming stagnant 
(if it hasn't already) quickly.

My (subjective) felling is that the codebase has a somewhat high 
learning curve. If one wants to add a fundamentally new features (as 
opposed to quick fixes, which do currently happen very quickly -- but 
think type inference!) it looks like one has to be very careful to keep 
many different effects of one's changes in the head at the same time.


Assertion 2: The reason for this state can at least in some part be 
attributed to some confusion as to what the current code tree *is*.

I believe that the reason that the codebase can be practical in the 
current state is tightly connected with how similar C and CPython is 
with Python as a language. The design, so to speak, arose in an 
historical happy accident, because the input and output are almost 1:1 
related. A Python function and a C function loosely correspond, and so 
FuncDefNode can serve the role as both the parsed element and the 
element due for C serialization. However there is no reason to assume 
that the 1:1 correspondance holds everywhere; and I think there is a 
tendency that "ugly hacks" already tend to arise in situations where the 
correspondance in program structure diverges more. (To consider how this 
might become worse, think about a "method template node".)

If this was only a theoretically founded critisism one might simply 
dismiss it, but I believe the problems in assertion 1 comes from this 
very fact, and that they can be solved by dealing with this conceptual 
problem.


The proposed solution: Work towards "The ideal solution" below in tiny, 
incremental steps. All *new* major features are implemented with the 
ideal solution in mind, and old code refactored as far as is needed for 
that to happen, but one doesn't start rewriting working and bugfixed 
code just for the sake of abstract purity.


The ideal solution: A strict pipeline-based approach where the "data" 
(one or more trees, graphs or similar) is in different stages, with 
different phases transforming one into the other. I.e, always try to fit 
what one does into the following example scheme:

data stage:string of code -> phase:parsing ->
data stage:syntax tree -> phase:expand with statements ->
data stage:tree without with statements -> phase:create scopes ->
data stage:tree with scopes -> phase:analyse types ->
data stage:tree with ".type" attributes -> ...
...
data stage:tree closely related to *structure* of C source -> phase:output

Whenever something doesn't quit fit in the scheme of naive input/output, 
split it up into more phases until it does :-)

(Digression: Closely related to C structure above means, for instance, 
no inner functions. No with statements. No untyped variables. That kind 
of thing. Classes may be included though, since one can create a 1:1 
correspondance between C code using the CPython API and a class; so it 
is not about language "features" as such but some deeper structure.)

Note that all phases does not need to be visitors! For instance, the 
output-to-C phase will stay a recursive process for the foreseeable 
future. This is a way of thinking, not a recipe for implementation. (And 
indeed, this way of thinking is already followed to a degree -- but not 
everywhere, and we get problems at the spots where it is not.)

A very important side-effect of this is that it lends itself to 
"implementing complex Cython statements as more simple Cython 
statements". (Though ideally "more simple Cython statements" can also be 
"intermediate simpler instructions" that bridges the gap between Cython 
and C in a more fine-grained way in order to give more code reuse than 
there is today. But I digress.)

(Implementation notes: This does not necesarrily mean that the classes 
of the nodes changes between phases; one does whatever is most 
convenient. For instance, if one has T2 = expand_with_nodes(T1), then T1 
is allowed to keep WithStatementNode, T2 is however not, but otherwise 
the trees are the same. In the same way, if T1 = type_analysis(T0), then 
ExprNodes in T0 is not allowed to have the type attribute, while 
ExprNodes in T1 are required to have them. This can be a documentation 
matter or enforced (in a number of ways), that issue is better left for 
later.).


Final words: This might seem trivial. It is not. The reason is that 
currently, almost wherever you try to implement two simple phases f and 
g one ends up having the result of f depend on what is done in g for 
some nodes, and the result of g depend on f for other nodes. And so 
neither f(g(T)) nor g(f(T)) can be made to work.

The point is, this way of changing how one thinks mostly affects the 
compiler design as such, more than it is a specific style of coding. 
Visitors happen to be the most natural way to implement this, but are 
not the main point. I'd be happy (or, happier than I am today) with a 
cleanly defined phase-based recursive process.

-- 
Dag Sverre


From stefan_ml at behnel.de  Thu May  8 19:27:15 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 08 May 2008 19:27:15 +0200
Subject: [Cython] Abandon support for Python 2.3?
In-Reply-To: <3d375d730805080958q65d7cd39v22d9f1cd86e1ba5d@mail.gmail.com>
References: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>
	<3d375d730805080958q65d7cd39v22d9f1cd86e1ba5d@mail.gmail.com>
Message-ID: <482337F3.8000105@behnel.de>

Hi,

Robert Kern wrote:
> On Thu, May 8, 2008 at 11:08 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> As mentioned in another thread, we are considering requiring Python
>> 2.4 or greater to run the Cython compiler, mostly so we can use
>> decorators in the compiler code. Is anyone still using Cython with
>> this version of Python?
> 
> We have been considering moving from Pyrex to Cython for part of
> numpy. numpy will be remaining with Python 2.3 for some time.

what Robert meant was: the compiler itself would no longer run on Py2.3, the
generated code would still support it. So you would only need Py2.4 or later
if you changed the code, not to compile and run the generated sources.

Would that really impact numpy?

Stefan

From robert.kern at gmail.com  Thu May  8 19:30:27 2008
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 8 May 2008 12:30:27 -0500
Subject: [Cython] Abandon support for Python 2.3?
In-Reply-To: <482337F3.8000105@behnel.de>
References: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>
	<3d375d730805080958q65d7cd39v22d9f1cd86e1ba5d@mail.gmail.com>
	<482337F3.8000105@behnel.de>
Message-ID: <3d375d730805081030w35c0c532xbc6eb9fb44f3c755@mail.gmail.com>

On Thu, May 8, 2008 at 12:27 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Robert Kern wrote:
>> On Thu, May 8, 2008 at 11:08 AM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>> As mentioned in another thread, we are considering requiring Python
>>> 2.4 or greater to run the Cython compiler, mostly so we can use
>>> decorators in the compiler code. Is anyone still using Cython with
>>> this version of Python?
>>
>> We have been considering moving from Pyrex to Cython for part of
>> numpy. numpy will be remaining with Python 2.3 for some time.
>
> what Robert meant was: the compiler itself would no longer run on Py2.3, the
> generated code would still support it. So you would only need Py2.4 or later
> if you changed the code, not to compile and run the generated sources.
>
> Would that really impact numpy?

I do not like being unable to develop with the platform we're deploying on.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco

From robertwb at math.washington.edu  Thu May  8 19:42:53 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 10:42:53 -0700
Subject: [Cython] Abandon support for Python 2.3?
In-Reply-To: <3d375d730805081030w35c0c532xbc6eb9fb44f3c755@mail.gmail.com>
References: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>
	<3d375d730805080958q65d7cd39v22d9f1cd86e1ba5d@mail.gmail.com>
	<482337F3.8000105@behnel.de>
	<3d375d730805081030w35c0c532xbc6eb9fb44f3c755@mail.gmail.com>
Message-ID: <F090844F-0C56-48F8-B7D8-1A809CA08141@math.washington.edu>

On May 8, 2008, at 10:30 AM, Robert Kern wrote:
> On Thu, May 8, 2008 at 12:27 PM, Stefan Behnel  
> <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> Robert Kern wrote:
>>> On Thu, May 8, 2008 at 11:08 AM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>>> As mentioned in another thread, we are considering requiring Python
>>>> 2.4 or greater to run the Cython compiler, mostly so we can use
>>>> decorators in the compiler code. Is anyone still using Cython with
>>>> this version of Python?
>>>
>>> We have been considering moving from Pyrex to Cython for part of
>>> numpy. numpy will be remaining with Python 2.3 for some time.

Numpy and numpy-using code are very important targets for Cython.

>> what Robert meant was: the compiler itself would no longer run on  
>> Py2.3, the
>> generated code would still support it. So you would only need  
>> Py2.4 or later
>> if you changed the code, not to compile and run the generated  
>> sources.
>>
>> Would that really impact numpy?
>
> I do not like being unable to develop with the platform we're  
> deploying on.

This is a valid point. Also, as we work (via Dag's GSoC project for  
example) to make it easier to use numpy effectively from Cython, it  
does seem kind of odd to say "you need 2.3 to use NumPy, but 2.4 to  
use NumPy+Cython."

I was finally starting to be convinced that moving to Py2.4 might be  
OK, but if it impacts our relation with SciPy then that is too high  
of a price to pay. Py2.4 features were suggested for convenience, not  
necessity.

- Robert


From dagss at student.matnat.uio.no  Thu May  8 19:50:26 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 May 2008 19:50:26 +0200
Subject: [Cython] Some small phase refactorings
In-Reply-To: <5303982D-51B8-4D6F-B363-BDC8C9729892@math.washington.edu>
References: <481AF97B.8030305@student.matnat.uio.no>	<481BFDDA.8060007@behnel.de>	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>	<48215041.7070805@student.matnat.uio.no>
	<5303982D-51B8-4D6F-B363-BDC8C9729892@math.washington.edu>
Message-ID: <48233D62.7000200@student.matnat.uio.no>


> any replies than I guess that's an indication. I am actually a bit  
> worried about how the use of decorators will impact the ability of  
> Cython to compile itself into ultra-efficient C.
>   
Efficiency doesn't bother me at all, and I have reasons! What the 
decorators would do (does, in prototype code) is stuff the class name 
and function into a dictionary at *class construction time* (or, without 
metaclass support, at object construction time), i.e. a O(1) overhead 
that is negligible even when running in the Python interpreter. During 
tree traversal, it's just a dict lookup on class id and a dispatch to 
the resulting function.

If one were generating C++ objects then the classical visitor pattern 
would be faster because real vtables are faster than dict lookups, 
however Cython polymorphism is (if I'm not wrong, haven't looked at this 
closely) dict-based anyway so it shouldn't make a difference.

Here's another way to use 2.3 that might be acceptable:

class WithStatementHandler(VisitorTransform):
    def handle_with(self, node): ...

    matches = [
        class_match(WithStatementNode, handle_with)
    ]


It would end up as a dict of class -> function like the other approach 
(but written like a list of objects because I'd like to leave a way open 
up for other types of matches though).

> It just didn't seem to add anything except another level of  
> indirection, but that will of course eventually be needed. There is a  
> lot of re-factoring that will need to be done. I've just got other  
> obligations (general exam) to be able to put much time/though into  
> this for the next two weeks or so :-(.
>   
What it added was the capability to plug in new transforms after type 
analysis, but before generation, on a module-wide basis, rather than 
having the transform run once for each function.

I didn't make that very clear; I suppose because you're right, it's real 
value is in another layer of indirection -- but one that will be needed 
and is useful during refactoring, and can allow "in-production" 
refactoring, rather than applying it all at once.

" The only problem that cannot be solved by another layer of indirection 
is too many layers of indirection " :-)

-- 
Dag Sverre


From dagss at student.matnat.uio.no  Thu May  8 19:52:41 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 08 May 2008 19:52:41 +0200
Subject: [Cython] Abandon support for Python 2.3?
In-Reply-To: <F090844F-0C56-48F8-B7D8-1A809CA08141@math.washington.edu>
References: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>	<3d375d730805080958q65d7cd39v22d9f1cd86e1ba5d@mail.gmail.com>	<482337F3.8000105@behnel.de>	<3d375d730805081030w35c0c532xbc6eb9fb44f3c755@mail.gmail.com>
	<F090844F-0C56-48F8-B7D8-1A809CA08141@math.washington.edu>
Message-ID: <48233DE9.90800@student.matnat.uio.no>


> This is a valid point. Also, as we work (via Dag's GSoC project for  
> example) to make it easier to use numpy effectively from Cython, it  
> does seem kind of odd to say "you need 2.3 to use NumPy, but 2.4 to  
> use NumPy+Cython."
I agree, this seems to pretty much override all my arguments.

-- 
Dag Sverre


From dalcinl at gmail.com  Thu May  8 20:01:51 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 15:01:51 -0300
Subject: [Cython] working patch for generating code targeting Py 2.6 and
	Py 3.0
In-Reply-To: <482322D5.3020600@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<800C6DA2-4F48-4BE2-9647-623E09146A63@math.washington.edu>
	<482322D5.3020600@behnel.de>
Message-ID: <e7ba66e40805081101u5fa2dce8ubdc37e1f87c4fb55@mail.gmail.com>

On 5/8/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> I do not expect this to be finished with a single patch, so there will be
>  things that may break somewhere in between and there may be things that
>  require larger refactorings.

Indeed. Although my patch is (all all future must be IMHO) backward
compatible at the C level, the best would be to have a new repo.


Keeping cython-devel somewhat stable and having a
>  separate branch where things can become unstable along the way helps us in
>  keeping up the ability to continue the normal development without being
>  directly impacted by Py3 work.
>
>  The idea is to merge things over step-by-step when they prove to be correct
>  and "stable enough".
>
>
>  Stefan
>
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu May  8 20:02:06 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 11:02:06 -0700
Subject: [Cython] A high-level discussion of the Cython internals
In-Reply-To: <48233596.5020204@student.matnat.uio.no>
References: <48233596.5020204@student.matnat.uio.no>
Message-ID: <E789DBAA-F559-41BA-A749-54D047DAA4E6@math.washington.edu>

On May 8, 2008, at 10:17 AM, Dag Sverre Seljebotn wrote:

> This is in response to this statement by Robert:
>
>
> --  
>
> I think this is good to talk about now. Basically, you want to change
> Cython to use the "visitor pattern" rather than the recursive pattern
> that it currently uses. This has pros and cons, but I think this
> could be a good thing, and you are certainly convinced that its
> needed to get started on the GSoC stuff.
>
> --
>
> So, leaving the patch and implementation strategies aside for now I'm
> going to attempt discussing this from the mountain top. (If a lot of
> this seems trivial, all the better -- it can then start to serve as
> documentation for a direction, rather than my "private"  
> efforts :-). If
> not, it will be good to discuss it.)
>
> I honestly believe that any time spent discussing and implementing  
> this
> now can repay itself many times later on in saved developer hours  
> and a
> lower learning curve for new developers. If people agree with me in  
> how
> important this is, I think it would be good to perhaps spend some time
> at the beginning of dev1 talking and discussing this (with some slides
> etc.).

+1

>
> Here we go:
>
> Assertion 1: Development of Cython has a potential for becoming  
> stagnant
> (if it hasn't already) quickly.

In the past 18 months or so since I started first hacking on the  
code, development has increased at an exponential rate, far from  
becoming stagnant. This does not mitigate your point though--we need  
to have this discussion to minimize brittle interdependence, lessen  
the learning curve, and take things to the next level.

> My (subjective) felling is that the codebase has a somewhat high
> learning curve. If one wants to add a fundamentally new features (as
> opposed to quick fixes, which do currently happen very quickly -- but
> think type inference!) it looks like one has to be very careful to  
> keep
> many different effects of one's changes in the head at the same time.
>
>
> Assertion 2: The reason for this state can at least in some part be
> attributed to some confusion as to what the current code tree *is*.
>
> I believe that the reason that the codebase can be practical in the
> current state is tightly connected with how similar C and CPython is
> with Python as a language. The design, so to speak, arose in an
> historical happy accident, because the input and output are almost 1:1
> related. A Python function and a C function loosely correspond, and so
> FuncDefNode can serve the role as both the parsed element and the
> element due for C serialization. However there is no reason to assume
> that the 1:1 correspondance holds everywhere; and I think there is a
> tendency that "ugly hacks" already tend to arise in situations  
> where the
> correspondance in program structure diverges more. (To consider how  
> this
> might become worse, think about a "method template node".)
>
> If this was only a theoretically founded critisism one might simply
> dismiss it, but I believe the problems in assertion 1 comes from this
> very fact, and that they can be solved by dealing with this conceptual
> problem.
>
>
> The proposed solution: Work towards "The ideal solution" below in  
> tiny,
> incremental steps. All *new* major features are implemented with the
> ideal solution in mind, and old code refactored as far as is needed  
> for
> that to happen, but one doesn't start rewriting working and bugfixed
> code just for the sake of abstract purity.
>
>
> The ideal solution: A strict pipeline-based approach where the "data"
> (one or more trees, graphs or similar) is in different stages, with
> different phases transforming one into the other. I.e, always try  
> to fit
> what one does into the following example scheme:
>
> data stage:string of code -> phase:parsing ->
> data stage:syntax tree -> phase:expand with statements ->
> data stage:tree without with statements -> phase:create scopes ->
> data stage:tree with scopes -> phase:analyse types ->
> data stage:tree with ".type" attributes -> ...
> ...
> data stage:tree closely related to *structure* of C source ->  
> phase:output
>
> Whenever something doesn't quit fit in the scheme of naive input/ 
> output,
> split it up into more phases until it does :-)
>
> (Digression: Closely related to C structure above means, for instance,
> no inner functions. No with statements. No untyped variables. That  
> kind
> of thing. Classes may be included though, since one can create a 1:1
> correspondance between C code using the CPython API and a class; so it
> is not about language "features" as such but some deeper structure.)
>
> Note that all phases does not need to be visitors! For instance, the
> output-to-C phase will stay a recursive process for the foreseeable
> future. This is a way of thinking, not a recipe for implementation.  
> (And
> indeed, this way of thinking is already followed to a degree -- but  
> not
> everywhere, and we get problems at the spots where it is not.)
>
> A very important side-effect of this is that it lends itself to
> "implementing complex Cython statements as more simple Cython
> statements". (Though ideally "more simple Cython statements" can  
> also be
> "intermediate simpler instructions" that bridges the gap between  
> Cython
> and C in a more fine-grained way in order to give more code reuse than
> there is today. But I digress.)
>
> (Implementation notes: This does not necesarrily mean that the classes
> of the nodes changes between phases; one does whatever is most
> convenient. For instance, if one has T2 = expand_with_nodes(T1),  
> then T1
> is allowed to keep WithStatementNode, T2 is however not, but otherwise
> the trees are the same. In the same way, if T1 = type_analysis(T0),  
> then
> ExprNodes in T0 is not allowed to have the type attribute, while
> ExprNodes in T1 are required to have them. This can be a documentation
> matter or enforced (in a number of ways), that issue is better left  
> for
> later.).
>
>
> Final words: This might seem trivial. It is not. The reason is that
> currently, almost wherever you try to implement two simple phases f  
> and
> g one ends up having the result of f depend on what is done in g for
> some nodes, and the result of g depend on f for other nodes. And so
> neither f(g(T)) nor g(f(T)) can be made to work.
>
> The point is, this way of changing how one thinks mostly affects the
> compiler design as such, more than it is a specific style of coding.
> Visitors happen to be the most natural way to implement this, but are
> not the main point. I'd be happy (or, happier than I am today) with a
> cleanly defined phase-based recursive process.

+1 to this whole idea. The visitor pattern certainly has the  
advantage that one can add phases without changing every node (which  
is perhaps why, e.g., the "analyze" phase does so many things that  
could be logically separated).

- Robert


From dalcinl at gmail.com  Thu May  8 20:12:12 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 15:12:12 -0300
Subject: [Cython] Fwd: working patch for generating code targeting Py
	2.6 and Py 3.0
In-Reply-To: <D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
Message-ID: <e7ba66e40805081112h431c2a44rb6190a5c6db64b00@mail.gmail.com>

On 5/8/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> Thanks. I briefly glanced at it and it looks good. What you probably
>  want to do is commit and then export the patch (that way it will have
>  some history attached, and you'll get credit for it.)

OK. But perhaps better... It's really hard for you to clone
cython-devel and name it cython-devel-py3k, and give me commit access
to only that repo?

> One remark I had is that perhaps more care  needs to be done with
>  "PyInt_CheckExact" as in Py2 this implies that the size fits into a
>  long, so it's not quite the same as PyLong_CheckExact.

Good catch!. I believe there is PyLong_FitsInLong or something similar
to this. I have to check...

> IMHO, all sloots should be keept, or
>  > all unused in py3k should be removed.
>
> Could you elaborate a bit more on what you mean by "unused?" If
>  they're still in the struct, I'd think we would need to keep them so
>  it has the right size/layout, right?

I mean that Python 3.0 sources are maintaining some slots in the
struct for no good reason... for example, nb_hex and nb_oct. IMHO,
they should be removed in core Python... But if they are not removed,
of course we still have to fill the struct.

> PyClass_New is gone, I've
>
> Filling the dict ahead of time might actually work better...

Just a question: Why Cython does not implement old-style classes this
way for Python2.X ? Filling a dict with PyCFunction's and next calling
PyClass_New does not work (I've never tried)?

>
> I actually have no idea on this one--maybe there's some system call
>  that can give this information? Stefan would probably know better
>  than me.

Anyway this is not really important at this point. In the worst
scenario, Python would not be able to find a source file for
displaying traceback lines in a fancy way (like the one in IPython),
but this is not a real problem.


>
>
>  >
>  > Regards,
>  >
>  >
>  > On 5/8/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>  >> Hi,
>  >>
>  >>
>  >>  Lisandro Dalcin wrote:
>  >>> The last four hours (3:00 AM right now here at Argentina) I've been
>  >>> working in a patch for enabling Cython generate code working with
>  >>> Python 2.6 and Python 3.0.
>  >>>
>  >>> Until now, the generated code (at least for the full mpi4py project)
>  >>> compiles and link fine with no errors.
>  >>>
>  >>> However, I have a big  problems I do not know how to 'fix' in
>  >>> Cython.
>  >>> It is related to unbound methods disapearing in Py3K. Then, normal
>  >>> python classes does not work, but the cdef ones are fine.
>  >>
>  >>
>  >> These are the kind of things that we planned to address at the
>  >> dev1 workshop.
>  >>  The C-API of Python is expected to stabilize next month, I don't
>  >> know how
>  >>  stable these parts currently are.
>  >>
>  >>
>  >>
>  >>> Is there any interest on this to go mainstream?
>  >>
>  >>
>  >> Sure, totally!
>  >>
>  >>
>  >>
>  >>> I was very
>  >>> conservative about the PyString/PyUnicode issue. The right one is
>  >>> used
>  >>> in a place-by-place base. Of course, because of this, I have to pass
>  >>> 'bytes' to MPI, and I get 'bytes' from the C calls.
>  >>
>  >>
>  >> Yes, I think this is the right way to deal with it. Python2 was
>  >> very lax in
>  >>  terms of semantics here, so the two have to be separated on a
>  >> case-by-case basis.
>  >>
>  >>
>  >>
>  >>> Finally, I'm completelly sure that I've not fixed all the relevant
>  >>> parts, but this is IMHO a good starting point.
>  >>
>  >>
>  >> Is it one big patch or did you/can you split it up?
>  >>
>  >>  As this is potentially a big change, trac is the wrong place to
>  >> discuss it. We
>  >>  should put up an official Py3 branch that people can actively
>  >> work on without
>  >>  impacting the main trunk, so that we can merge working stuff
>  >> gradually. Can
>  >>  you send a bundle against cython-devel to me and Robert for now?
>  >> He can set it up.
>  >>
>  >>  Stefan
>  >>  _______________________________________________
>  >>  Cython-dev mailing list
>  >>  Cython-dev at codespeak.net
>  >>  http://codespeak.net/mailman/listinfo/cython-dev
>  >>
>  >
>  >
>  > --
>  > Lisandro Dalc?n
>  > ---------------
>  > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>  > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>  > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>  > PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>
> > Tel/Fax: +54-(0)342-451.1594<cython-
>  > py3k.patch>_______________________________________________
>
> > Cython-dev mailing list
>  > Cython-dev at codespeak.net
>  > http://codespeak.net/mailman/listinfo/cython-dev
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu May  8 20:20:25 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 11:20:25 -0700
Subject: [Cython] Fwd: working patch for generating code targeting Py
	2.6 and Py 3.0
In-Reply-To: <e7ba66e40805081112h431c2a44rb6190a5c6db64b00@mail.gmail.com>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<e7ba66e40805081112h431c2a44rb6190a5c6db64b00@mail.gmail.com>
Message-ID: <2A65A78D-FAE9-47FB-940B-2DE0E0D2E6A6@math.washington.edu>

On May 8, 2008, at 11:12 AM, Lisandro Dalcin wrote:

> On 5/8/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> Thanks. I briefly glanced at it and it looks good. What you probably
>>  want to do is commit and then export the patch (that way it will  
>> have
>>  some history attached, and you'll get credit for it.)
>
> OK. But perhaps better... It's really hard for you to clone
> cython-devel and name it cython-devel-py3k, and give me commit access
> to only that repo?

Sure. I'll just need a htpasswd file so you can authenticate. (This  
will add you as a user to trac too.)

>> One remark I had is that perhaps more care  needs to be done with
>>  "PyInt_CheckExact" as in Py2 this implies that the size fits into a
>>  long, so it's not quite the same as PyLong_CheckExact.
>
> Good catch!. I believe there is PyLong_FitsInLong or something similar
> to this. I have to check...

Perhaps this should be handled at a higher level--i.e. we should see  
where in the source we're using PyInt_CheckExact, etc. and do  
something else there so we don't need the #defines...

>> IMHO, all sloots should be keept, or
>>> all unused in py3k should be removed.
>>
>> Could you elaborate a bit more on what you mean by "unused?" If
>>  they're still in the struct, I'd think we would need to keep them so
>>  it has the right size/layout, right?
>
> I mean that Python 3.0 sources are maintaining some slots in the
> struct for no good reason... for example, nb_hex and nb_oct. IMHO,
> they should be removed in core Python... But if they are not removed,
> of course we still have to fill the struct.
>
>> PyClass_New is gone, I've
>>
>> Filling the dict ahead of time might actually work better...
>
> Just a question: Why Cython does not implement old-style classes this
> way for Python2.X ? Filling a dict with PyCFunction's and next calling
> PyClass_New does not work (I've never tried)?

No idea, this is how Greg wrote it.

>
>>
>> I actually have no idea on this one--maybe there's some system call
>>  that can give this information? Stefan would probably know better
>>  than me.
>
> Anyway this is not really important at this point. In the worst
> scenario, Python would not be able to find a source file for
> displaying traceback lines in a fancy way (like the one in IPython),
> but this is not a real problem.

Having good tracebacks is important to me at least, hopefully we'll  
find a way to do it.

- Robert


From dalcinl at gmail.com  Thu May  8 20:33:31 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 15:33:31 -0300
Subject: [Cython] A high-level discussion of the Cython internals
In-Reply-To: <48233596.5020204@student.matnat.uio.no>
References: <48233596.5020204@student.matnat.uio.no>
Message-ID: <e7ba66e40805081133o3d271249je503e05a9d612431@mail.gmail.com>

Definitely +1 on all your comments ...

Being a new cython user with a rather good knowledge of Python at the
C level, and having a serious project (at least, I believe it is
serious) already working in a couple of weeks, the really BIG, BIG
problem I have with Cython is that it is almost impossible for me to
learn and follow the internals in order to help Cython improve. Any
step in this direction would be very well welcome.

And Robert is right, Cython is not stagnat, but I have the feel that
all power users are depending too much on core developers. For
example, after trying to add Py3K compat, I cannot have any idea of
the goodness of this work, how much has to be done, or what is broken.
And that should definitelly not happen. The more understandable the
code base is, less work for core developers to review contributions
from other 'trusted' developers. But at this point, I do not even
trust on myself when hacking inside Cython.


On 5/8/08, Dag Sverre Seljebotn <dagss at student.matnat.uio.no> wrote:
> This is in response to this statement by Robert:
>
>
>  --
>
>  I think this is good to talk about now. Basically, you want to change
>  Cython to use the "visitor pattern" rather than the recursive pattern
>  that it currently uses. This has pros and cons, but I think this
>  could be a good thing, and you are certainly convinced that its
>  needed to get started on the GSoC stuff.
>
>  --
>
>  So, leaving the patch and implementation strategies aside for now I'm
>  going to attempt discussing this from the mountain top. (If a lot of
>  this seems trivial, all the better -- it can then start to serve as
>  documentation for a direction, rather than my "private" efforts :-). If
>  not, it will be good to discuss it.)
>
>  I honestly believe that any time spent discussing and implementing this
>  now can repay itself many times later on in saved developer hours and a
>  lower learning curve for new developers. If people agree with me in how
>  important this is, I think it would be good to perhaps spend some time
>  at the beginning of dev1 talking and discussing this (with some slides
>  etc.).
>
>  Here we go:
>
>  Assertion 1: Development of Cython has a potential for becoming stagnant
>  (if it hasn't already) quickly.
>
>  My (subjective) felling is that the codebase has a somewhat high
>  learning curve. If one wants to add a fundamentally new features (as
>  opposed to quick fixes, which do currently happen very quickly -- but
>  think type inference!) it looks like one has to be very careful to keep
>  many different effects of one's changes in the head at the same time.
>
>
>  Assertion 2: The reason for this state can at least in some part be
>  attributed to some confusion as to what the current code tree *is*.
>
>  I believe that the reason that the codebase can be practical in the
>  current state is tightly connected with how similar C and CPython is
>  with Python as a language. The design, so to speak, arose in an
>  historical happy accident, because the input and output are almost 1:1
>  related. A Python function and a C function loosely correspond, and so
>  FuncDefNode can serve the role as both the parsed element and the
>  element due for C serialization. However there is no reason to assume
>  that the 1:1 correspondance holds everywhere; and I think there is a
>  tendency that "ugly hacks" already tend to arise in situations where the
>  correspondance in program structure diverges more. (To consider how this
>  might become worse, think about a "method template node".)
>
>  If this was only a theoretically founded critisism one might simply
>  dismiss it, but I believe the problems in assertion 1 comes from this
>  very fact, and that they can be solved by dealing with this conceptual
>  problem.
>
>
>  The proposed solution: Work towards "The ideal solution" below in tiny,
>  incremental steps. All *new* major features are implemented with the
>  ideal solution in mind, and old code refactored as far as is needed for
>  that to happen, but one doesn't start rewriting working and bugfixed
>  code just for the sake of abstract purity.
>
>
>  The ideal solution: A strict pipeline-based approach where the "data"
>  (one or more trees, graphs or similar) is in different stages, with
>  different phases transforming one into the other. I.e, always try to fit
>  what one does into the following example scheme:
>
>  data stage:string of code -> phase:parsing ->
>  data stage:syntax tree -> phase:expand with statements ->
>  data stage:tree without with statements -> phase:create scopes ->
>  data stage:tree with scopes -> phase:analyse types ->
>  data stage:tree with ".type" attributes -> ...
>  ...
>  data stage:tree closely related to *structure* of C source -> phase:output
>
>  Whenever something doesn't quit fit in the scheme of naive input/output,
>  split it up into more phases until it does :-)
>
>  (Digression: Closely related to C structure above means, for instance,
>  no inner functions. No with statements. No untyped variables. That kind
>  of thing. Classes may be included though, since one can create a 1:1
>  correspondance between C code using the CPython API and a class; so it
>  is not about language "features" as such but some deeper structure.)
>
>  Note that all phases does not need to be visitors! For instance, the
>  output-to-C phase will stay a recursive process for the foreseeable
>  future. This is a way of thinking, not a recipe for implementation. (And
>  indeed, this way of thinking is already followed to a degree -- but not
>  everywhere, and we get problems at the spots where it is not.)
>
>  A very important side-effect of this is that it lends itself to
>  "implementing complex Cython statements as more simple Cython
>  statements". (Though ideally "more simple Cython statements" can also be
>  "intermediate simpler instructions" that bridges the gap between Cython
>  and C in a more fine-grained way in order to give more code reuse than
>  there is today. But I digress.)
>
>  (Implementation notes: This does not necesarrily mean that the classes
>  of the nodes changes between phases; one does whatever is most
>  convenient. For instance, if one has T2 = expand_with_nodes(T1), then T1
>  is allowed to keep WithStatementNode, T2 is however not, but otherwise
>  the trees are the same. In the same way, if T1 = type_analysis(T0), then
>  ExprNodes in T0 is not allowed to have the type attribute, while
>  ExprNodes in T1 are required to have them. This can be a documentation
>  matter or enforced (in a number of ways), that issue is better left for
>  later.).
>
>
>  Final words: This might seem trivial. It is not. The reason is that
>  currently, almost wherever you try to implement two simple phases f and
>  g one ends up having the result of f depend on what is done in g for
>  some nodes, and the result of g depend on f for other nodes. And so
>  neither f(g(T)) nor g(f(T)) can be made to work.
>
>  The point is, this way of changing how one thinks mostly affects the
>  compiler design as such, more than it is a specific style of coding.
>  Visitors happen to be the most natural way to implement this, but are
>  not the main point. I'd be happy (or, happier than I am today) with a
>  cleanly defined phase-based recursive process.
>
>
>  --
>  Dag Sverre
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May  8 22:35:30 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 17:35:30 -0300
Subject: [Cython] Fwd: working patch for generating code targeting Py
	2.6 and Py 3.0
In-Reply-To: <2A65A78D-FAE9-47FB-940B-2DE0E0D2E6A6@math.washington.edu>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<e7ba66e40805081112h431c2a44rb6190a5c6db64b00@mail.gmail.com>
	<2A65A78D-FAE9-47FB-940B-2DE0E0D2E6A6@math.washington.edu>
Message-ID: <e7ba66e40805081335l552aace7saae56297c8a8021b@mail.gmail.com>

On 5/8/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On May 8, 2008, at 11:12 AM, Lisandro Dalcin wrote:
> Sure. I'll just need a htpasswd file so you can authenticate. (This
>  will add you as a user to trac too.)

The you will send me the password, right?

>
>  >>  "PyInt_CheckExact" as in Py2 this implies that the size fits into a
>  >>  long, so it's not quite the same as PyLong_CheckExact.
>
> Perhaps this should be handled at a higher level--i.e. we should see
>  where in the source we're using PyInt_CheckExact, etc. and do
>  something else there so we don't need the #defines...

You are right. But then I do not know how to do that right now, so
this is for the near future...

>  > Just a question: Why Cython does not implement old-style classes this
>  > way for Python2.X ? Filling a dict with PyCFunction's and next calling
>  > PyClass_New does not work (I've never tried)?
>
>
> No idea, this is how Greg wrote it.

Well, could you ask Greg to post some comment about this? This way, I
can follow the right path trying to fix this for the Py3K case...

> Having good tracebacks is important to me at least, hopefully we'll
>  find a way to do it.

BTW, I've tried to install the source files alongside the generated
extensions modules, but IPython does not show the traceback. Could you
give me a tip about this?  Please note that my *.so extension module
is inside a dir with a __init__.py file, then it is actually a module
inside a package...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu May  8 22:46:39 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 13:46:39 -0700
Subject: [Cython] Fwd: working patch for generating code targeting Py
	2.6 and Py 3.0
In-Reply-To: <e7ba66e40805081335l552aace7saae56297c8a8021b@mail.gmail.com>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<e7ba66e40805081112h431c2a44rb6190a5c6db64b00@mail.gmail.com>
	<2A65A78D-FAE9-47FB-940B-2DE0E0D2E6A6@math.washington.edu>
	<e7ba66e40805081335l552aace7saae56297c8a8021b@mail.gmail.com>
Message-ID: <A6218D42-32AE-4561-BC61-36F62E6CA209@math.washington.edu>

On May 8, 2008, at 1:35 PM, Lisandro Dalcin wrote:

> On 5/8/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On May 8, 2008, at 11:12 AM, Lisandro Dalcin wrote:
>> Sure. I'll just need a htpasswd file so you can authenticate. (This
>>  will add you as a user to trac too.)
>
> The you will send me the password, right?

An htpasswd file is a file that contains a username and (hashed)  
password. You can do

     htpasswd -c -s somefile username

(available on most systems) and it will prompt for a password to  
associate with username and store the sha1 has. Then send me the  
somefile it created and I'll add it to the list. See man htpasswd for  
more details.

>
>>
>>>>  "PyInt_CheckExact" as in Py2 this implies that the size fits  
>>>> into a
>>>>  long, so it's not quite the same as PyLong_CheckExact.
>>
>> Perhaps this should be handled at a higher level--i.e. we should see
>>  where in the source we're using PyInt_CheckExact, etc. and do
>>  something else there so we don't need the #defines...
>
> You are right. But then I do not know how to do that right now, so
> this is for the near future...
>
>>> Just a question: Why Cython does not implement old-style classes  
>>> this
>>> way for Python2.X ? Filling a dict with PyCFunction's and next  
>>> calling
>>> PyClass_New does not work (I've never tried)?
>>
>>
>> No idea, this is how Greg wrote it.
>
> Well, could you ask Greg to post some comment about this? This way, I
> can follow the right path trying to fix this for the Py3K case...

I believe on this list now :-)

>> Having good tracebacks is important to me at least, hopefully we'll
>>  find a way to do it.
>
> BTW, I've tried to install the source files alongside the generated
> extensions modules, but IPython does not show the traceback. Could you
> give me a tip about this?  Please note that my *.so extension module
> is inside a dir with a __init__.py file, then it is actually a module
> inside a package...


Yeah, I'm not sure what mechanism IPython uses to look up source  
files, but something needs to be fixed there (either on our side or  
theirs). You can set it to not give context in which case it will  
give a traceback.

- Robert



From dalcinl at gmail.com  Thu May  8 22:48:50 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 8 May 2008 17:48:50 -0300
Subject: [Cython] Abandon support for Python 2.3?
In-Reply-To: <48233DE9.90800@student.matnat.uio.no>
References: <47797395-6F61-44F6-B232-8C3F9CC751E4@math.washington.edu>
	<3d375d730805080958q65d7cd39v22d9f1cd86e1ba5d@mail.gmail.com>
	<482337F3.8000105@behnel.de>
	<3d375d730805081030w35c0c532xbc6eb9fb44f3c755@mail.gmail.com>
	<F090844F-0C56-48F8-B7D8-1A809CA08141@math.washington.edu>
	<48233DE9.90800@student.matnat.uio.no>
Message-ID: <e7ba66e40805081348r28aeb5b9h3e8decfff15af858@mail.gmail.com>

Dag, suposse the only 2.4 feature you really need is @decorators,
then, Do you believe it would be too much hard to also implement a
code conversion tool transforming the cython 2.4 sources to valid 2.3
code?

On 5/8/08, Dag Sverre Seljebotn <dagss at student.matnat.uio.no> wrote:
>
>  > This is a valid point. Also, as we work (via Dag's GSoC project for
>  > example) to make it easier to use numpy effectively from Cython, it
>  > does seem kind of odd to say "you need 2.3 to use NumPy, but 2.4 to
>  > use NumPy+Cython."
>
> I agree, this seems to pretty much override all my arguments.

I'm not still convinced... as long as cython is used as a command line
tool, you do not need cython depend on 2.3. Of couse, if you can
develop a code conversion tool for decorators, then go on with 2.4!!!


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Fri May  9 03:55:40 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 09 May 2008 13:55:40 +1200
Subject: [Cython] Fwd: working patch for generating code targeting Py
 2.6 and Py 3.0
In-Reply-To: <D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
Message-ID: <4823AF1C.5090900@canterbury.ac.nz>

Robert Bradshaw wrote:
> Filling the dict ahead of time might actually work better...one would  
> execute the body in a temporary scope and then construct the class.  
> This would bring things more in line with how Python does things.

That's probably the right thing to do. I can't remember
exactly, but I think the main reason I created the class
first was so that I could use the same code to access a
name as I was already for module variables. If you can
remember somehow that you need to use dict access instead
of attribute access, I don't think there should be any
problem.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri May  9 04:01:25 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 09 May 2008 14:01:25 +1200
Subject: [Cython] PyPy parser
In-Reply-To: <743600FF-F1BC-48F4-BDC6-EC89A99396BA@math.washington.edu>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
	<e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
	<AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>
	<e437f2f0805071537n676db015x8e4decc6dafb3a1@mail.gmail.com>
	<743600FF-F1BC-48F4-BDC6-EC89A99396BA@math.washington.edu>
Message-ID: <4823B075.5000903@canterbury.ac.nz>

Robert Bradshaw wrote:

> A simple grammar -> parser module like you've talked about before  
> seems ideal, and would probably be fairly compact.

Before diving into this, there are a couple of things
you should take into account:

1. A parser generator of the kind used by CPython is
probably not capable of parsing C declarations without
doing some hackery somewhere.

2. There is computation embedded in the current Pyrex
parser in various places that you would need to track
down and move somewhere else.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri May  9 04:13:21 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 09 May 2008 14:13:21 +1200
Subject: [Cython] __getattribute__
In-Reply-To: <e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>
References: <47FE4418.40703@student.matnat.uio.no> <47FF20EC.4040803@behnel.de>
	<fvtjok$73f$1@ger.gmane.org> <fvtk30$73f$2@ger.gmane.org>
	<482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
	<e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>
Message-ID: <4823B341.8070209@canterbury.ac.nz>

I've been thinking some more about __getattribute__, and I think
that rather than trying to emulate the Python semantics, the
right thing to do is simply to expose the C type slot directly.

This would be in keeping with the way the other type slots
are handled. The philosophy is that when defining an extension
type, you should have as much control and as little overhead
in the way as possible.

The only places where I've done things differently are where
there isn't a directly corresponding C slot, such as __getattr__.
Since __getattribute__ corresponds most closely to tp_getattr,
it should just fill it directly.

-- 
Greg

From robertwb at math.washington.edu  Fri May  9 04:25:21 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 19:25:21 -0700
Subject: [Cython] PyPy parser
In-Reply-To: <4823B075.5000903@canterbury.ac.nz>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
	<e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
	<AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>
	<e437f2f0805071537n676db015x8e4decc6dafb3a1@mail.gmail.com>
	<743600FF-F1BC-48F4-BDC6-EC89A99396BA@math.washington.edu>
	<4823B075.5000903@canterbury.ac.nz>
Message-ID: <EE908415-ACC3-4AB7-B1D9-15057628CDA0@math.washington.edu>

On May 8, 2008, at 7:01 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> A simple grammar -> parser module like you've talked about before
>> seems ideal, and would probably be fairly compact.
>
> Before diving into this, there are a couple of things
> you should take into account:
>
> 1. A parser generator of the kind used by CPython is
> probably not capable of parsing C declarations without
> doing some hackery somewhere.

I think the C declarations are representable by a generative grammar,  
so this method could work.

> 2. There is computation embedded in the current Pyrex
> parser in various places that you would need to track
> down and move somewhere else.

Yes, I've seen this too.

I think most people who have proposed replacing the parser have  
underestimated the complexity and subtleties of parsing Pyrex/Cython  
compared to pure Python. On the surface it is very close to Python,  
but those parts that differ (e.g. type declaration) can be very  
complicated. That being said I think it can be done (I don't want to  
discourage anyone, there are some very capable people on this list)  
and with Python 3.0 on the horizon significant changes will need to  
be made one way or another. Other than that the current parser works  
very well.

- Robert


From robertwb at math.washington.edu  Fri May  9 04:30:03 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 8 May 2008 19:30:03 -0700
Subject: [Cython] __getattribute__
In-Reply-To: <4823B341.8070209@canterbury.ac.nz>
References: <47FE4418.40703@student.matnat.uio.no>
	<47FF20EC.4040803@behnel.de> <fvtjok$73f$1@ger.gmane.org>
	<fvtk30$73f$2@ger.gmane.org> <482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
	<e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>
	<4823B341.8070209@canterbury.ac.nz>
Message-ID: <9E4E3C4E-DD03-4536-8328-838CB4A19676@math.washington.edu>

On May 8, 2008, at 7:13 PM, Greg Ewing wrote:

> I've been thinking some more about __getattribute__, and I think
> that rather than trying to emulate the Python semantics, the
> right thing to do is simply to expose the C type slot directly.
>
> This would be in keeping with the way the other type slots
> are handled. The philosophy is that when defining an extension
> type, you should have as much control and as little overhead
> in the way as possible.

I think this is a philosophical difference between Cython and Pyrex-- 
in Cython we want to stick to Python semantics as closely as possible  
so the user doesn't even have to think about whether they're writing  
an extension class or not. Eventually one should be able to take  
any .py file, run Cython on it, and have it behave exactly the same.  
If this is not the case than it is a bug. (I'm not sure exactly what  
to do about arithmetic though--that might be one exception.)

> The only places where I've done things differently are where
> there isn't a directly corresponding C slot, such as __getattr__.
> Since __getattribute__ corresponds most closely to tp_getattr,
> it should just fill it directly.

In terms of efficiency, __getattribute__ is called at the top, and on  
success tp_getattr returns immediately, so it should be just about as  
fast.

- Robert


From greg.ewing at canterbury.ac.nz  Fri May  9 05:24:28 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 09 May 2008 15:24:28 +1200
Subject: [Cython] __getattribute__
In-Reply-To: <9E4E3C4E-DD03-4536-8328-838CB4A19676@math.washington.edu>
References: <47FE4418.40703@student.matnat.uio.no> <47FF20EC.4040803@behnel.de>
	<fvtjok$73f$1@ger.gmane.org> <fvtk30$73f$2@ger.gmane.org>
	<482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
	<e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>
	<4823B341.8070209@canterbury.ac.nz>
	<9E4E3C4E-DD03-4536-8328-838CB4A19676@math.washington.edu>
Message-ID: <4823C3EC.4080306@canterbury.ac.nz>

Robert Bradshaw wrote:

> I think this is a philosophical difference between Cython and Pyrex-- 
> ... Eventually one should be able to take  
> any .py file, run Cython on it, and have it behave exactly the same.

Actually, it would, because a pure Python file isn't
going to contain any extension classes, and whatever
else is done, __getattribute__ on a Python class should
follow Python semantics.

But this is certainly a difference in goals between
Pyrex and Cython. The goal of Pyrex is not to compile
Python efficiently, but to enable efficient bridging
between Python and C.

> In terms of efficiency, __getattribute__ is called at the top, and on  
> success tp_getattr returns immediately, so it should be just about as  
> fast.

To my mind, the semantic difference is actually a
feature -- it lets you completely control what it
means to look up an attribute of your object.

A compromise might be to find another name for the
method that directly fills tp_getattr. But then
we'd have *three* variants of attribute-getting
methods to keep straight... not sure if that would
be an improvement or not.

-- 
Greg


From pete at petertodd.org  Fri May  9 05:26:12 2008
From: pete at petertodd.org (Peter Todd)
Date: Thu, 8 May 2008 23:26:12 -0400
Subject: [Cython] __getattribute__
In-Reply-To: <4823B341.8070209@canterbury.ac.nz>
References: <47FE4418.40703@student.matnat.uio.no>
	<47FF20EC.4040803@behnel.de> <fvtjok$73f$1@ger.gmane.org>
	<fvtk30$73f$2@ger.gmane.org> <482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
	<e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>
	<4823B341.8070209@canterbury.ac.nz>
Message-ID: <20080509032612.GB27761@tilt>

On Fri, May 09, 2008 at 02:13:21PM +1200, Greg Ewing wrote:
> I've been thinking some more about __getattribute__, and I think
> that rather than trying to emulate the Python semantics, the
> right thing to do is simply to expose the C type slot directly.
> 
> This would be in keeping with the way the other type slots
> are handled. The philosophy is that when defining an extension
> type, you should have as much control and as little overhead
> in the way as possible.
> 
> The only places where I've done things differently are where
> there isn't a directly corresponding C slot, such as __getattr__.
> Since __getattribute__ corresponds most closely to tp_getattr,
> it should just fill it directly.

If no __getattr__ is defined, __getattribute__ is essentially a
tp_getattr slot filler:

static PyObject *__pyx_tp_getattro_3foo_foo(PyObject *o, PyObject *n) {
    PyObject *v = __pyx_pf_3foo_3foo___getattribute__(o, n);
    return v;
}

One level of indirection, but that's just an artifact of the current
implementation that the compiler might very well optimize out anyway.

-- 
http://petertodd.org 'peter'[:-1]@petertodd.org 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080508/e816f6fe/attachment.pgp 

From greg.ewing at canterbury.ac.nz  Fri May  9 05:33:03 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 09 May 2008 15:33:03 +1200
Subject: [Cython] __getattribute__
In-Reply-To: <20080509032612.GB27761@tilt>
References: <47FE4418.40703@student.matnat.uio.no> <47FF20EC.4040803@behnel.de>
	<fvtjok$73f$1@ger.gmane.org> <fvtk30$73f$2@ger.gmane.org>
	<482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
	<e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>
	<4823B341.8070209@canterbury.ac.nz> <20080509032612.GB27761@tilt>
Message-ID: <4823C5EF.6050700@canterbury.ac.nz>

Peter Todd wrote:

> If no __getattr__ is defined, __getattribute__ is essentially a
> tp_getattr slot filler:
> 
> static PyObject *__pyx_tp_getattro_3foo_foo(PyObject *o, PyObject *n) {
>     PyObject *v = __pyx_pf_3foo_3foo___getattribute__(o, n);
>     return v;
> }

What if a __getattr__ is inherited from a base class?

What if this class doesn't define a __getattr__, but
another class that inherits from it does, and doesn't
override __getattribute__?

-- 
Greg

From pete at petertodd.org  Fri May  9 05:51:05 2008
From: pete at petertodd.org (Peter Todd)
Date: Thu, 8 May 2008 23:51:05 -0400
Subject: [Cython] __getattribute__
In-Reply-To: <4823C5EF.6050700@canterbury.ac.nz>
References: <47FF20EC.4040803@behnel.de> <fvtjok$73f$1@ger.gmane.org>
	<fvtk30$73f$2@ger.gmane.org> <482293B5.3080802@behnel.de>
	<5447846C-35D8-4E52-98BA-AB0F0082E311@math.washington.edu>
	<48231CA1.5080702@googlemail.com>
	<e7ba66e40805080922w26a4acf4uad349c440c2826db@mail.gmail.com>
	<4823B341.8070209@canterbury.ac.nz> <20080509032612.GB27761@tilt>
	<4823C5EF.6050700@canterbury.ac.nz>
Message-ID: <20080509035105.GC27761@tilt>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, May 09, 2008 at 03:33:03PM +1200, Greg Ewing wrote:
> Peter Todd wrote:
> 
> > If no __getattr__ is defined, __getattribute__ is essentially a
> > tp_getattr slot filler:
> > 
> > static PyObject *__pyx_tp_getattro_3foo_foo(PyObject *o, PyObject *n) {
> >     PyObject *v = __pyx_pf_3foo_3foo___getattribute__(o, n);
> >     return v;
> > }
> 
> What if a __getattr__ is inherited from a base class?
> 
> What if this class doesn't define a __getattr__, but
> another class that inherits from it does, and doesn't
> override __getattribute__?

My simplistic idea of the world breaks? :)

No, you are totally right. A __tp_getattro__ special method could be
defined, it'd be about ~15 lines of code I think, and another test case,
including error messages if __tp_getattro__ and either __getattribute__
or __getattr__ are defined. I'd be happy to do it, but I think it'd be
better to think about other cases like it first.

- -- 
http://petertodd.org 'peter'[:-1]@petertodd.org 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFII8op3bMhDbI9xWQRAsSIAJ95BRIQSBxXtvfabgqSniajkY7FcACfcehX
tLxL2KN/OG4gPtjqL4qvFIs=
=uRg9
-----END PGP SIGNATURE-----

From stefan_ml at behnel.de  Fri May  9 10:25:14 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 10:25:14 +0200
Subject: [Cython] Fwd: working patch for generating code targeting Py
 2.6 and Py 3.0
In-Reply-To: <D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>	<4822A413.8060902@behnel.de>	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
Message-ID: <48240A6A.7040700@behnel.de>

Hi,

skipping through the patch, the changes seem to make sense in general. I'll
have to look through it a bit more thorough, but not today.

There are still a couple of places where you mix up strings (i.e. "bytes") and
unicode, like this:

------------------------
@@ -4309,7 +4336,11 @@ static int __Pyx_InitStrings(__Pyx_Strin
         if (t->is_unicode) {
             *t->p = PyUnicode_DecodeUTF8(t->s, t->n - 1, NULL);
         } else {
+            #if PY_MAJOR_VERSION < 3
             *t->p = PyString_FromStringAndSize(t->s, t->n - 1);
+            #else
+            *t->p = PyUnicode_FromStringAndSize(t->s, t->n - 1);
+            #endif
         }
         if (!*t->p)
             return -1;
------------------------

The original code already handles unicode strings properly. For Py3 source
code, this has to be adapted in the Cython parser, not in the generated code
(i.e. parse "abc" as if it was u"abc" and b"abc" like "abc").

For current Cython code, this should just work unchanged - assuming you import
the string compatibility header file that #defines PyString_* as PyBytes_*.
Although I might want to see plain PyBytes_* calls generated here - not sure yet.


Robert Bradshaw wrote:
> On May 8, 2008, at 8:33 AM, Lisandro Dalcin wrote:
>> - Finally, in the very clever part of traceback hackery, the call to
>> Py_CodeNew receives 'filenames' created with PyUnicode_AsString. I do
>> not know what the C preprocessor uses for enconding __FILE__ macro, it
>> its always ASCII, then all is fine, if not, the filesystem encoding
>> should be taken into account.
> 
> I actually have no idea on this one--maybe there's some system call  
> that can give this information? Stefan would probably know better  
> than me.

Not sure either. I expect __FILE__ to be in the platform specific filesystem
encoding, so decoding to unicode can easily fail. We should store the UTF-8
encoded filename in the file so that it can be read instead of __FILE__.

Stefan

From stefan_ml at behnel.de  Fri May  9 11:21:07 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 11:21:07 +0200
Subject: [Cython] Fwd: working patch for generating code targeting Py
 2.6 and Py 3.0
In-Reply-To: <48240A6A.7040700@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>	<4822A413.8060902@behnel.de>	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<48240A6A.7040700@behnel.de>
Message-ID: <48241783.5080002@behnel.de>

Hi again,

Stefan Behnel wrote:
> ------------------------
> @@ -4309,7 +4336,11 @@ static int __Pyx_InitStrings(__Pyx_Strin
>          if (t->is_unicode) {
>              *t->p = PyUnicode_DecodeUTF8(t->s, t->n - 1, NULL);
>          } else {
> +            #if PY_MAJOR_VERSION < 3
>              *t->p = PyString_FromStringAndSize(t->s, t->n - 1);
> +            #else
> +            *t->p = PyUnicode_FromStringAndSize(t->s, t->n - 1);
> +            #endif
>          }
>          if (!*t->p)
>              return -1;
> ------------------------

I think this should read

        if (t->is_unicode) {
            #if PY_MAJOR_VERSION < 3
            *t->p = PyUnicode_DecodeUTF8(t->s, t->n - 1, NULL);
            #else
            *t->p = PyUnicode_FromStringAndSize(t->s, t->n - 1);
            #endif
        } else {
            *t->p = PyString_FromStringAndSize(t->s, t->n - 1);
        }

Also this:

------------------------
@@ -4289,7 +4312,11 @@ static int __Pyx_InternStrings(__Pyx_Int
 ""","""
 static int __Pyx_InternStrings(__Pyx_InternTabEntry *t) {
     while (t->p) {
+        #if PY_MAJOR_VERSION < 3
         *t->p = PyString_InternFromString(t->s);
+        #else
+        *t->p = PyUnicode_InternFromString(t->s);
+        #endif
         if (!*t->p)
             return -1;
         ++t;
------------------------

The thing here is that we currently do not intern unicode strings at all, so
this must continue to return byte strings.

The actualy problem should be fixed in the compiler, which should know how to
distinguish byte strings from unicode strings in its interned string
dictionary, and generate similar code as for the normal string table (i.e.
with a "unicode" flag). See the add_py_string() method in Symtab.py for a start.

I noticed that cython-devel-py3 is up, but I'll wait for Lisandro to commit
his patch before I start working on it.

Stefan

From robertwb at math.washington.edu  Fri May  9 11:53:58 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 9 May 2008 02:53:58 -0700
Subject: [Cython] [Pyrex] Callbacks
In-Reply-To: <66a8c45d0805090242lb47b651p388945a78e329ff7@mail.gmail.com>
References: <66a8c45d0805090242lb47b651p388945a78e329ff7@mail.gmail.com>
Message-ID: <86D7288D-E9D9-4EC4-836D-B37524C135E7@math.washington.edu>

On May 9, 2008, at 2:42 AM, Daniele Pianu wrote:

[...]

> Is there a way to solve the problem without importing anything from  
> Python.h?

No, unless you want to cache of the func/data manually somewhere  
else. Also note that this may cause a memory leak, as the reference  
count will never be decremented.

- Robert


From dagss at student.matnat.uio.no  Fri May  9 12:13:24 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 09 May 2008 12:13:24 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
Message-ID: <482423C4.9040005@student.matnat.uio.no>

I think there's a fundamental flaw to my NumPy proposal, which need 
correction. I've thought about it as "polishing the access to the 
extension type", so that after

cdef object x = numpy.zeros([3,3])
cdef numpy.ndarray y = x

y will behave efficiently when treated like a Python object.

There are problems to this way of thinking though: "print y.strides" 
will not work as it is a pointer-array, while "print x.strides" will 
work as it is a tuple (one has to do "print (<object>y).strides", which 
is not going to fly with for the users of my NumPy project). On the 
other hand, one has to remember to do "print y.shape" for tuple-access 
but "y.dimensions[i]" for speedy, non-Python access. (And this 
difference in behaviour comes entirely from strides having a name-clash 
while shape/dimensions do not).

So while my approach has been to make the y variable act "more like a 
numpy array", I now think this is flawed. Optimizations through typing 
and access to extension structs should probably instead be treated as 
fundamentally different things, and the typical NumPy user shouldn't 
deal with a reference to the extension struct even if typing for speed 
is wanted.

I'll now propose a solution for this. It has been proposed before in a 
different form (pxd shadowing etc.); I hope I succeed better now.

I think that what is wanted here is to /speed up how NumPy objects are 
accessed/, and the extension type only comes into it peripherally. So 
I'd like a new syntax for modifying the compile-time behaviour of how 
objects are treated. It could look something like the following.

I'm calling the keyword "compiletimefeatures" but that is for lack of a 
better word, also I use cython_ndarray to avoid namespace clashes (have 
ideas for allowing "ndarray" directly but I'd like to leave that out of 
the discussion for now).

Anyway, numpy.ndarray is an extension type like before, while 
cython_ndarray is a new "type specifier" providing extra compile-time 
optimizations to the variable that carries its type. numpy.pxd:

cdef extern from "numpy/arrayobject.h":
    ctypedef class numpy.ndarray [object PyArrayObject]:
        cdef char *data
        cdef int nd
        cdef Py_intptr_t *dimensions
        cdef Py_intptr_t *strides
        ...


compiletimefeatures cython_ndarray:
    def __applyfeatures__(self):
        if not isinstance(self, numpy.ndarray):
            raise TypeError(...)

    @property
    cdef shape(self):
        cdef numpy.ndarray imp = self
        return (imp.dimensions[i] for idx in range(imp.nd))
       
    # See my gsoc project for __getitem__, just replace "self" with "imp"
    ...


User code:

cdef cython_ndarray y = x
print y.shape

So, what happens:

 * cython_ndarray is registered as a new "cdef-able" type.

 * When assigning something to y, a runtime call to __applyfeatures__ is 
automatically added (i.e. principially the same thing that happens with 
extension type declarations, but more specific and versatile)

 * When y is operated on, it is first checked whether cython_array 
contains any compile time optimizations. If so, they are called (using y 
as the "self", like a class, however "self" is a Python object!)

 * End result: Rather than having to hope for some good luck in the 
namespace resolution towards the extension type struct to provide the 
speedup, one can be explicit about it.

Note: The "optimizations" provided above will only be faster if Cython 
gets quite sophisticated unrolling/optimization. While I'd like to have 
this, a more declarative approach might be more realistic. So something 
like:

compiletimefeatures cython_ndarray:
    carray_or_tuple shape(x): extension_type(x).shape

Sorry, couldn't think about a good declarative syntax now :-)

Note that this is instead of the plans to allow inlineable code in 
extension type declarations. Extension type declarations wouldn't have 
to be touched at all then.

Flame away :-) (yes, I see that this could be confusing to OOP. However 
it is no worse than the current situation, one cannot really override 
extension type struct items either.)


-- 
Dag Sverre


From stefan_ml at behnel.de  Fri May  9 13:50:15 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 13:50:15 +0200
Subject: [Cython] Fwd: working patch for generating code targeting Py
 2.6 and Py 3.0
In-Reply-To: <48240A6A.7040700@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>	<4822A413.8060902@behnel.de>	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<48240A6A.7040700@behnel.de>
Message-ID: <48243A77.9040008@behnel.de>

Hi,

Stefan Behnel wrote:
> For current Cython code, this should just work unchanged - assuming you import
> the string compatibility header file that #defines PyString_* as PyBytes_*.

sorry for the confusion: there is no special header involved. The current API
in Py3.0a5 uses these names:

    bytearray -> PyBytes_*
    bytes     -> PyString_*
    str       -> PyUnicode_*

Described here

http://www.python.org/dev/peps/pep-3137/

with a summary here:

http://www.python.org/dev/peps/pep-3137/#summary

We should eventually special case bytearray(b'') literals, although that is
not strictly required. It's more important to actually support these types and
the coercion between them first. ;)

Stefan

From greg.ewing at canterbury.ac.nz  Fri May  9 13:57:21 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 09 May 2008 23:57:21 +1200
Subject: [Cython] ANN: Pyrex 0.9.7
Message-ID: <48243C21.2020207@canterbury.ac.nz>

Pyrex 0.9.7 is now available:

   http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/

Highlights of this version:

* I have streamlined the integer for-loop syntax. Instead
   of the loop variable redundantly appearing in two places,
   it's now just

     for x < i < y:
       ...

* If you declare a variable as a list or dict, then calls
   to some of its methods will be compiled into type-specific
   Python API calls instead of generic ones.

* Most built-in constants are referenced directly instead of
   via dict lookup.

* There are two new builtin functions, typecheck() and
   issubtype(), for checking the types of arguments more safely
   (since isinstance and issubclass can be fooled).


What is Pyrex?
--------------

Pyrex is a language for writing Python extension modules.
It lets you freely mix operations on Python and C data, with
all Python reference counting and error checking handled
automatically.

From greg.ewing at canterbury.ac.nz  Fri May  9 14:20:04 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 10 May 2008 00:20:04 +1200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <482423C4.9040005@student.matnat.uio.no>
References: <482423C4.9040005@student.matnat.uio.no>
Message-ID: <48244174.10506@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:
> And this 
> difference in behaviour comes entirely from strides having a name-clash 
> while shape/dimensions do not.

I missed the beginning of this, so I'm not sure exactly
what problem is being discussed here, but I suspect you're
making all this much more complicated than it needs to be.

If a name clash is the only problem, it should be easily
resolvable by using a C name declaration to give the
offending attribute a different name in the Pyrex/Cython code.

-- 
Greg

From dagss at student.matnat.uio.no  Fri May  9 14:52:06 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 09 May 2008 14:52:06 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <48244174.10506@canterbury.ac.nz>
References: <482423C4.9040005@student.matnat.uio.no>
	<48244174.10506@canterbury.ac.nz>
Message-ID: <482448F6.2050600@student.matnat.uio.no>

An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080509/bed2e4bd/attachment.htm 

From stefan_ml at behnel.de  Fri May  9 17:21:52 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 17:21:52 +0200
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <48243C21.2020207@canterbury.ac.nz>
References: <48243C21.2020207@canterbury.ac.nz>
Message-ID: <48246C10.6050703@behnel.de>

Hi Greg,

Greg Ewing wrote:
 * I have streamlined the integer for-loop syntax. Instead
>    of the loop variable redundantly appearing in two places,
>    it's now just
> 
>      for x < i < y:
>        ...

while this is shorter, it also breaks existing code - although I assume you
left the old syntax in?

Also, I'm not sure this is more readable if x is non trivial.


> * If you declare a variable as a list or dict, then calls
>    to some of its methods will be compiled into type-specific
>    Python API calls instead of generic ones.

Cython has a bit of that also. This would be something to bring back in line.


> * Most built-in constants are referenced directly instead of
>    via dict lookup.

Again, something like that is in Cython as well, but I'm not up-to-date with
the details. Probably works different...


> * There are two new builtin functions, typecheck() and
>    issubtype(), for checking the types of arguments more safely
>    (since isinstance and issubclass can be fooled).

Could you elaborate on the actual problem here?


I also noticed you have a couple of new test cases. Could you imagine using
our doctest based test runner as well? That way, we could at least share the
test suites and see more easily which problems apply to both code bases.

http://hg.cython.org/cython-devel/file/tip/runtests.py

Stefan

From dalcinl at gmail.com  Fri May  9 18:20:50 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 9 May 2008 13:20:50 -0300
Subject: [Cython] Fwd: working patch for generating code targeting Py
	2.6 and Py 3.0
In-Reply-To: <48243A77.9040008@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<48240A6A.7040700@behnel.de> <48243A77.9040008@behnel.de>
Message-ID: <e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>

Well, at this point I'm a bit confused... The unicode stuff is the
weakest point in my knowledge of Python internals...

Anyway, I will not be able to work on this in the next two weeks, so
if anyone wants to go on and give a try, just start applying the patch
to 'cython-devel-py3' repo (I'm not concerned at all about credits for
the original patch ;-) ) in order this does not stagnate. At any
point, I can contribute testing against my project, just ask me for
help.

Finally, IMHO Cython pyx-level and generated C-level code should work
with any Python version. So, if in a pyx file I wrote "abc", then, by
default, it should be a ascii string for Py2 and a unicode one for
Py3. And Cython should grow a new special 'char' type (like it has for
bint) to specify ascii string. Then, at the C-level, and depending on
Python major version, the proper conversion py->c or c->py can be
managed. If not, wrapping legacy C libs and targeting Py2 and Py3 is
going to be a nightmare.


On 5/9/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>
>  Stefan Behnel wrote:
>  > For current Cython code, this should just work unchanged - assuming you import
>  > the string compatibility header file that #defines PyString_* as PyBytes_*.
>
>
> sorry for the confusion: there is no special header involved. The current API
>  in Py3.0a5 uses these names:
>
>     bytearray -> PyBytes_*
>     bytes     -> PyString_*
>     str       -> PyUnicode_*
>
>  Described here
>
>  http://www.python.org/dev/peps/pep-3137/
>
>  with a summary here:
>
>  http://www.python.org/dev/peps/pep-3137/#summary
>
>  We should eventually special case bytearray(b'') literals, although that is
>  not strictly required. It's more important to actually support these types and
>  the coercion between them first. ;)
>
>
>  Stefan
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri May  9 19:12:24 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 19:12:24 +0200
Subject: [Cython] Fwd: working patch for generating code targeting Py
 2.6 and Py 3.0
In-Reply-To: <e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>	<4822A413.8060902@behnel.de>	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>	<48240A6A.7040700@behnel.de>
	<48243A77.9040008@behnel.de>
	<e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>
Message-ID: <482485F8.5080508@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Well, at this point I'm a bit confused... The unicode stuff is the
> weakest point in my knowledge of Python internals...

:) that's just because Unicode is not exactly trivial - especially if you have
to start thinking about bytes at some point.


> Anyway, I will not be able to work on this in the next two weeks, so
> if anyone wants to go on and give a try, just start applying the patch
> to 'cython-devel-py3' repo (I'm not concerned at all about credits for
> the original patch ;-) ) in order this does not stagnate.

done.


> At any
> point, I can contribute testing against my project, just ask me for
> help.

some test cases would be greatly appreciated. :)


> Finally, IMHO Cython pyx-level and generated C-level code should work
> with any Python version.

But your code shouldn't change semantics when you change the runtime
environment. Otherwise it would become completely impossible to write portable
code.


> So, if in a pyx file I wrote "abc", then, by
> default, it should be a ascii string for Py2 and a unicode one for
> Py3.

Not quite right. If you did that in a .py file, that would be the case. If you
do that in Cython, a literal must not change its type, because it compiles to C.


> And Cython should grow a new special 'char' type (like it has for
> bint) to specify ascii string.

Why is that? Two things: what's special about an ASCII string compared to a
byte string in general? And: why aren't "bytes" and "bytearray" enough?


> Then, at the C-level, and depending on
> Python major version, the proper conversion py->c or c->py can be
> managed. If not, wrapping legacy C libs and targeting Py2 and Py3 is
> going to be a nightmare.

It's a nightmare if your code changes behaviour. I think there should be a
command line switch "-3" for compiling plain Py3 code at some point (as
opposed to normal Py2 code) or maybe that should be determined by the runtime
environment of the *compiler*, but the Cython language itself should stay
compatible to Py2.6 for now.

Stefan


From stefan_ml at behnel.de  Fri May  9 19:51:14 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 19:51:14 +0200
Subject: [Cython] Long strings
Message-ID: <48248F12.7070803@behnel.de>

Hi,

I've seen this problem already a while ago: MSVC 2003 has a problem with
strings that are longer than 2KB. Honestly, a compiler that chokes on 2048
tiny bytes!

This can obviously affect string data in the source code, which I worked
around back then by splitting long strings up and concatenating them at
runtime. However, now I have the problem that a function in lxml has a
docstring(!) that is too long, so there is not much I can do outside Cython
(except for cutting down the documentation, but that would be stupid, right?).

The obvious way to work around this (may I say it?) buggy compiler is to do
the splitting in Cython and reconstruct the complete string at module
initialisation time. Essentially, this Python code

    a = "... loads ... of ... bytes ... "

would map to the (pre-built) equivalent of this:

    a = "... loads" + " ... of " + "... bytes " + "... "

Before I start working on a hack like this, are there other ideas how to deal
with this problem?

Thanks for any ideas,
Stefan

From dagss at student.matnat.uio.no  Fri May  9 20:11:28 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 09 May 2008 20:11:28 +0200
Subject: [Cython] Long strings
In-Reply-To: <48248F12.7070803@behnel.de>
References: <48248F12.7070803@behnel.de>
Message-ID: <482493D0.7070705@student.matnat.uio.no>


> I've seen this problem already a while ago: MSVC 2003 has a problem with
> strings that are longer than 2KB. Honestly, a compiler that chokes on 2048
> tiny bytes!
>   
Hehe, sounds familiar. I remember struggling with getting VC6 to compile 
template code -- nice to know they kept up their "quality" since then...

> Before I start working on a hack like this, are there other ideas how to deal
> with this problem?
>   
I know very, very little about post-2000 Microsoft stuff. But at least 
in VC6 it was possible (and common) to seperately link in binary 
resource files which was accessed through the Windows API. One could 
have a VC-mode where a ".res"-file is dumped containing strings, and 
these could then be compiled in, and looked up run-time. This should be 
very slightly faster as you get your data linked into the executable in 
one block (don't know what the .NET-equivalent would be though); or at 
least, any overhead is O(1) with respect to string length (unlike your 
approach, which is O(N) -- though probably not noticeable anyway).

Your approach sounds a lot saner though :-) Would be nice if you would 
code it up as a transform, so that we had some transforms to start 
building a pipeline structure around and some real code to work out 
issues with the approach. It is an ideal transform situation -- in goes 
string nodes, out goes addition nodes and string nodes; right after the 
parsing phase which is available now.

In fact, if you don't find a better solution, I'll volunteer to do the 
implementation, since it sounds so incredibly ideal for a simple 
transform demonstration (and it is always nicer to have something real 
than something contrived).

Dag Sverre

From stefan_ml at behnel.de  Fri May  9 20:18:44 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 20:18:44 +0200
Subject: [Cython] Long strings
In-Reply-To: <482493D0.7070705@student.matnat.uio.no>
References: <48248F12.7070803@behnel.de>
	<482493D0.7070705@student.matnat.uio.no>
Message-ID: <48249584.2070507@behnel.de>

Hi Dag,

Dag Sverre Seljebotn wrote:
>> Before I start working on a hack like this, are there other ideas how to deal
>> with this problem?
> 
> Would be nice if you would code it up as a transform

Nope. I actually envisioned to code it right into __Pyx_InitStrings(). We
could also write two different versions of that function and write the more
complex one into the .c file only if we detect that it's needed.

Stefan


From dagss at student.matnat.uio.no  Fri May  9 20:20:37 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 09 May 2008 20:20:37 +0200
Subject: [Cython] Long strings
In-Reply-To: <482493D0.7070705@student.matnat.uio.no>
References: <48248F12.7070803@behnel.de>
	<482493D0.7070705@student.matnat.uio.no>
Message-ID: <482495F5.3020900@student.matnat.uio.no>

Dag Sverre wrote:  

> I know very, very little about post-2000 Microsoft stuff. But at least 
> in VC6 it was possible (and common) to seperately link in binary 
> resource files which was accessed through the Windows API. One could 
>   
OK, looked it up, apparently the old approach is still valid when 
compiling "non-managed" code. Basically you output an RC file, which 
must then be compiled and linked in using VC tools. For .NET-code you'd 
use "resource bundles" or something like that.


Dag Sverre

From dalcinl at gmail.com  Fri May  9 20:26:26 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 9 May 2008 15:26:26 -0300
Subject: [Cython] Fwd: working patch for generating code targeting Py
	2.6 and Py 3.0
In-Reply-To: <482485F8.5080508@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<4822A413.8060902@behnel.de>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<48240A6A.7040700@behnel.de> <48243A77.9040008@behnel.de>
	<e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>
	<482485F8.5080508@behnel.de>
Message-ID: <e7ba66e40805091126x7476e91et1cdb83a2d167e4e7@mail.gmail.com>

You see!! All your observations just seems to indicate I do not
actually undestand the whole thing. Please, if you have the time, can
you clarify me a bit this:

I have this class Info (something like a dictionary ) wrapping some
MPI calls  for getting and setting (key,value) pairs (they have to be
ascii, 8-bits strings in order MPI understands them)


cdef class Info:

    # ...

    def Get(self, char key[]):
        """
        Retrieve the value associated with a key
        """
        cdef char value[MPI_MAX_INFO_VAL]
        cdef int flag = 0
        CHKERR( MPI_Info_get(self.ob_mpi, key, maxlen, value, &flag) )
        if not flag: return (None, False)
        else:        return (value, True)

    def Set(self, char key[], char value[]):
        """
        Add the (key,value) pair to info, and overrides the value if a
        value for the same key was previously set
        """
        CHKERR( MPI_Info_set(self.ob_mpi, key, value) )


So then all your comments actually means that I should not take any
special action in Cython for wrapping this, and then

* - If I run this in Python 2, I just do: info.Set("key", ''value")

* - If I run this in Python 3, I should do: info.Set(b"key", b"value")

In that case, suppose now that a user running on Python 3 does the following:

>>> info.Set("key", "value") .

This is broken, because MPI will not recognize the key or the value,
as they are not C plain char arrays containing null-terminated ascii
8-bits string.

Then, how should I modify my *.pyx code to detect this and generate an
error/warning, or even try to coerce the input to ascii 8-bits if the
input is 'unicode', or pass it unchanged if the input is 'bytes'? And
all this working both in 2.3/2.4/2.5/2.6 and 3.0??


On 5/9/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>
>  Lisandro Dalcin wrote:
>  > Well, at this point I'm a bit confused... The unicode stuff is the
>  > weakest point in my knowledge of Python internals...
>
>
> :) that's just because Unicode is not exactly trivial - especially if you have
>  to start thinking about bytes at some point.
>
>
>
>  > Anyway, I will not be able to work on this in the next two weeks, so
>  > if anyone wants to go on and give a try, just start applying the patch
>  > to 'cython-devel-py3' repo (I'm not concerned at all about credits for
>  > the original patch ;-) ) in order this does not stagnate.
>
>
> done.
>
>
>
>  > At any
>  > point, I can contribute testing against my project, just ask me for
>  > help.
>
>
> some test cases would be greatly appreciated. :)
>
>
>
>  > Finally, IMHO Cython pyx-level and generated C-level code should work
>  > with any Python version.
>
>
> But your code shouldn't change semantics when you change the runtime
>  environment. Otherwise it would become completely impossible to write portable
>  code.
>
>
>
>  > So, if in a pyx file I wrote "abc", then, by
>  > default, it should be a ascii string for Py2 and a unicode one for
>  > Py3.
>
>
> Not quite right. If you did that in a .py file, that would be the case. If you
>  do that in Cython, a literal must not change its type, because it compiles to C.
>
>
>
>  > And Cython should grow a new special 'char' type (like it has for
>  > bint) to specify ascii string.
>
>
> Why is that? Two things: what's special about an ASCII string compared to a
>  byte string in general? And: why aren't "bytes" and "bytearray" enough?
>
>
>
>  > Then, at the C-level, and depending on
>  > Python major version, the proper conversion py->c or c->py can be
>  > managed. If not, wrapping legacy C libs and targeting Py2 and Py3 is
>  > going to be a nightmare.
>
>
> It's a nightmare if your code changes behaviour. I think there should be a
>  command line switch "-3" for compiling plain Py3 code at some point (as
>  opposed to normal Py2 code) or maybe that should be determined by the runtime
>  environment of the *compiler*, but the Cython language itself should stay
>  compatible to Py2.6 for now.
>
>
>  Stefan
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri May  9 21:30:38 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 21:30:38 +0200
Subject: [Cython] How to deal with byte strings and unicode strings at
 the API level
In-Reply-To: <e7ba66e40805091126x7476e91et1cdb83a2d167e4e7@mail.gmail.com>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>	<4822A413.8060902@behnel.de>	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>	<48240A6A.7040700@behnel.de>
	<48243A77.9040008@behnel.de>	<e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>	<482485F8.5080508@behnel.de>
	<e7ba66e40805091126x7476e91et1cdb83a2d167e4e7@mail.gmail.com>
Message-ID: <4824A65E.90007@behnel.de>

Hi,

Lisandro Dalcin wrote:
> I have this class Info (something like a dictionary ) wrapping some
> MPI calls  for getting and setting (key,value) pairs (they have to be
> ascii, 8-bits strings in order MPI understands them)

:) first problem: ASCII is 7-bit, not 8-bit.


> cdef class Info:
>     def Get(self, char key[]):
>         [...]
>     def Set(self, char key[], char value[]):
>         [...]

Here, you define your API as taking byte strings as input. Fair enough.

However, in your example, you do not verify that your input is really ASCII
encoded, so you allow users to pass 8-bit strings without any warning.


> So then all your comments actually means that I should not take any
> special action in Cython for wrapping this, and then
> 
> * - If I run this in Python 2, I just do: info.Set("key", ''value")

This works, as plain string literals in Python are byte strings. Also,
conversion between plain ASCII byte strings and unicode strings happens
automatically in Python2 (the infamous UnicodeDecodeError on print).


> * - If I run this in Python 3, I should do: info.Set(b"key", b"value")

Again, this works as you pass byte strings, as enforced by your API.


> In that case, suppose now that a user running on Python 3 does the following:
> 
>>>> info.Set("key", "value") .
> 
> This is broken,

It's not broken, it's just incorrect API usage. In a way, it's like doing this:

   >>> info.Set(-999, 123456789)

and, guess what: Python will raise a TypeError for this!


> because MPI will not recognize the key or the value,
> as they are not C plain char arrays containing null-terminated ascii
> 8-bits string.
>
> Then, how should I modify my *.pyx code to detect this and generate an
> error/warning,

Once the code is in place, Cython will generate a TypeError for you, just like
Py3 itself does when attempting automatic conversion between unicode strings
and bytes objects.


> or even try to coerce the input to ascii 8-bits if the input is 'unicode'

No, that's one of the problems why there is a lot of broken code in Python2:
"works on my machine, so it can't be broken, can it?"


> or pass it unchanged if the input is 'bytes'?

That will work, as a bytes object (which is actually a PyStringObject in both
Py2 and Py3) is compatible with a char*.

However, imagine this line in a Python2 source file:

    info.Set("???", "??????")

or this line in a Python3 source file:

    info.Set(b"???", b"??????")

What will be the byte sequence that you get in your char[] for key and value?

Well, it depends on the source code encoding, which you can declare at the
beginning of your source file. If, for example, it's "UTF-8", you will get a
UTF-8 encoded byte sequence, which is 6 bytes long for key and 12 bytes long
for value. If, on the other hand, it's "iso-8859-1", you will get a 3-byte
sequence for key and a 6-byte sequence for value. Same code, looks exactly the
same in an editor, but results in completely different input to your methods.

How is your code going to deal with this? How will it even know what happens?


> And all this working both in 2.3/2.4/2.5/2.6 and 3.0??

If byte strings is what your API deals with, this will continue to work across
all of those versions. You can use the 2to3 tool to convert your Python2 code
to Python3. It works quite well and will change this

    info.Set("key", "value")

into this

    info.Set(b"key", b"value")

for you (you can even call the tool from your setup.py).

If, however, what you actually want is text input (i.e. unicode characters),
you can fix your API like this (probably plus some more input checking):

  cdef class Info:
      def Get(self, key):
          key = key.encode("ASCII") # or whatever encoding you use internally
          [...]
      def Set(self, key, value):
          key   = key.encode("ASCII")   # or whatever encoding you use
          value = value.encode("ASCII") # or whatever encoding you use
          [...]

That way, users can call your API in Python2 like this:

    info.Set(u"key", u"value")

and the 2to3 tool will convert this to

    info.Set("key", "value")

for them, which (again) will continue to work, and (now the cool thing) your
methods will always receive the input in the expected encoding, and your users
will get an encoding exception if they pass non-ASCII strings. :)

So all you have to take care of is what your actual API is: bytes? characters?

Does this make things a bit clearer?

Stefan

From stefan_ml at behnel.de  Fri May  9 21:48:16 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 09 May 2008 21:48:16 +0200
Subject: [Cython] How to deal with byte strings and unicode strings at
 the API level
In-Reply-To: <4824A65E.90007@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>	<4822A413.8060902@behnel.de>	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>	<48240A6A.7040700@behnel.de>	<48243A77.9040008@behnel.de>	<e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>	<482485F8.5080508@behnel.de>	<e7ba66e40805091126x7476e91et1cdb83a2d167e4e7@mail.gmail.com>
	<4824A65E.90007@behnel.de>
Message-ID: <4824AA80.3040803@behnel.de>

Hi again,

Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>> or even try to coerce the input to ascii 8-bits if the input is 'unicode'
> 
> No, that's one of the problems why there is a lot of broken code in Python2:

Sorry, I misread your sentence here. You were asking how to fix your code and
I thought you meant Cython should do it for you (which it can't and also
shouldn't...)

If your input is unicode characters, then yes, encoding them to a well defined
byte sequence is the right thing to do.

Stefan

From robertwb at math.washington.edu  Fri May  9 22:07:21 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 9 May 2008 13:07:21 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <482423C4.9040005@student.matnat.uio.no>
References: <482423C4.9040005@student.matnat.uio.no>
Message-ID: <9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>

On May 9, 2008, at 3:13 AM, Dag Sverre Seljebotn wrote:

> I think there's a fundamental flaw to my NumPy proposal, which need
> correction. I've thought about it as "polishing the access to the
> extension type", so that after
>
> cdef object x = numpy.zeros([3,3])
> cdef numpy.ndarray y = x
>
> y will behave efficiently when treated like a Python object.
>
> There are problems to this way of thinking though: "print y.strides"
> will not work as it is a pointer-array, while "print x.strides" will
> work as it is a tuple (one has to do "print (<object>y).strides",  
> which
> is not going to fly with for the users of my NumPy project).

I think the correct way to handle this (if we want to) is to make  
y.strides transform into (<object>y).strides when coerced to a  
PyObject (assuming that y.strides is not a type we already know how  
to turn into an object).

> On the
> other hand, one has to remember to do "print y.shape" for tuple-access
> but "y.dimensions[i]" for speedy, non-Python access. (And this
> difference in behaviour comes entirely from strides having a name- 
> clash
> while shape/dimensions do not).

This is a numpy api issue, nothing to do with Cython.

>
> So while my approach has been to make the y variable act "more like a
> numpy array", I now think this is flawed. Optimizations through typing
> and access to extension structs should probably instead be treated as
> fundamentally different things, and the typical NumPy user shouldn't
> deal with a reference to the extension struct even if typing for speed
> is wanted.
>
> I'll now propose a solution for this. It has been proposed before in a
> different form (pxd shadowing etc.); I hope I succeed better now.
>
> I think that what is wanted here is to /speed up how NumPy objects are
> accessed/, and the extension type only comes into it peripherally. So
> I'd like a new syntax for modifying the compile-time behaviour of how
> objects are treated. It could look something like the following.
>
> I'm calling the keyword "compiletimefeatures" but that is for lack  
> of a
> better word, also I use cython_ndarray to avoid namespace clashes  
> (have
> ideas for allowing "ndarray" directly but I'd like to leave that  
> out of
> the discussion for now).
>
> Anyway, numpy.ndarray is an extension type like before, while
> cython_ndarray is a new "type specifier" providing extra compile-time
> optimizations to the variable that carries its type. numpy.pxd:

[...]

> Flame away :-) (yes, I see that this could be confusing to OOP.  
> However
> it is no worse than the current situation, one cannot really override
> extension type struct items either.)

You asked for it... I think this is a very bad idea. The "compile  
time features" of a type belong to the type, and putting them in some  
overlay (with a different name, that one has to know about) seems  
counterintuitive. And I really don't see any advantages (assuming the  
name clashing can be solved as above).

Here is a very simple prototype of what I think could be done in the  
pxd:

---------a.pxd----------
cdef class A:
     cdef int len
     cdef int* data

     cdef inline [final?] int __getitem__(A a, int i):
         """
         Note that subtypes can't override this.
         """
         if i < 0 or i > a.len:
             raise IndexError
         return data[i]

---------b.pyx--------

from a cimport A
cdef A(len=10) a = a([1,2,3,4,5,6,7,8,9,10]) # I'll leave the init  
function to your imagination
print a[9] # the code from __getitem__ gets inlined here, and since  
len is known the a.len is resolved to 10 at compile time.

(Here a.len tries to do a lookup first on the compile time type of a,  
and that failing the runtime type of a. The compile time types need  
not be struct members, but if they're not then they must be specified  
because the "runtime" lookup would fail.)

- Robert


From dalcinl at gmail.com  Fri May  9 22:50:23 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 9 May 2008 17:50:23 -0300
Subject: [Cython] How to deal with byte strings and unicode strings at
	the API level
In-Reply-To: <4824AA80.3040803@behnel.de>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>
	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>
	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>
	<48240A6A.7040700@behnel.de> <48243A77.9040008@behnel.de>
	<e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>
	<482485F8.5080508@behnel.de>
	<e7ba66e40805091126x7476e91et1cdb83a2d167e4e7@mail.gmail.com>
	<4824A65E.90007@behnel.de> <4824AA80.3040803@behnel.de>
Message-ID: <e7ba66e40805091350t58ffdff5i698415824653f4c9@mail.gmail.com>

Stefan, many, many thanks for your explanations. I believe I've
started to understand the whole beast. Please clarify me this:

Suppose I write this method in a pyx file:

def foo(char value[]): pass

and next call it (in a Py3 runtime env) in this two ways:

1-  foo(b"abc")

2-  foo("abc")

then (1) should be just fine, Cython should pass the raw C data as it
comes. But in the case of (2), will/should Cython (or at the Python C
API level) generate an error?

If the answer is NO, does it make sense to extend Cython with a C
'pseudotype' (ala bint), let call it 'bchar' to make the generated C
code being strict about the type of the Python-level argument?.




On 5/9/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi again,
>
>  Stefan Behnel wrote:
>
> > Lisandro Dalcin wrote:
>  >> or even try to coerce the input to ascii 8-bits if the input is 'unicode'
>  >
>  > No, that's one of the problems why there is a lot of broken code in Python2:
>
>
> Sorry, I misread your sentence here. You were asking how to fix your code and
>  I thought you meant Cython should do it for you (which it can't and also
>  shouldn't...)
>
>  If your input is unicode characters, then yes, encoding them to a well defined
>  byte sequence is the right thing to do.
>
>
>  Stefan
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From curiousjan at gmail.com  Fri May  9 23:43:17 2008
From: curiousjan at gmail.com (Jan Strube)
Date: Fri, 9 May 2008 14:43:17 -0700
Subject: [Cython] cython add vs __add__
Message-ID: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>

Hi List,
sorry for the crosspost, I have also asked the question here https:// 
answers.launchpad.net/cython/+question/31930, but probably the list  
is a better place to ask.


In the new version of Cython I am excited to report that overloading  
operators seems to work just fine. This is great, because in the high  
energy physics community, people think it's a good idea to abuse the  
() operator for iterators to mean next (among other nonsense).

So I have a short class in C++, the code is below, and as posted,  
everything works fine.
However, when I try to change "def add(self)" to "def __add__(self)"  
in myTest I get complaints of the sort

   File "test.py", line 37, in <module>
     print test + 3
   File "stdMatrix.pyx", line 67, in stdMatrix.myTest.__add__  
(stdMatrix.cpp:452)
     return self.thisPtr.add(<int>other)
AttributeError: 'stdMatrix.myTest' object has no attribute 'thisPtr'

I find this quite surprising, and could use some enlightenment. I can  
easily enough work around this, but this is a bug, no?

Thanks for cython and best of luck on your sprint.
Best,
     Jan

------------ class_operators.cc ---------------
class TEST
{
public:
     int x;
     TEST(int y) {x=y;}
     int operator() () {x = 17; return x;}
     int operator+(int y) {return x+y;}
     int minimize() {return 25;}
};
----------------------------------------------------------

------stdMatrix.pyx----------------------------------------
cdef extern from "class_operators.cc":
     ctypedef struct monkey "TEST":
         int x
         int (*minimize)()
         int add "operator+"(int y)
         int call "operator()"()
     monkey* new_Test "new TEST"(int y)
     void del_Test "delete"(monkey* x)

cdef class myTest:
     cdef monkey *thisPtr
     def __cinit__(self, int y):
         self.thisPtr = new_Test(y)
     def __dealloc__(self):
         del_Test(self.thisPtr)
     def min(self):
         return self.thisPtr.minimize()
     def plus(self, other):
         return self.thisPtr.add(other)
     def call(self):
         return self.thisPtr.call()
--------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080509/6fa4b0d0/attachment.htm 

From dalcinl at gmail.com  Sat May 10 00:59:01 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 9 May 2008 19:59:01 -0300
Subject: [Cython] cython add vs __add__
In-Reply-To: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>
References: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>
Message-ID: <e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>

I believe you are right, it seems a bug, I'v experienced this in the
past, and there and used this fixes

cdef class myTest
    def __add__(myTest self, other): # note declaration: myTest self
        return self.thisPtr.add(other)

You should go on with this, It should even work when this get fixed
(in the case this is actually a bug and I'm not missing somethig)

So rename your 'plus' and 'call' to standard __add__ and __call__ with
the declaration trick above, and all should just work.

Regards,



On 5/9/08, Jan Strube <curiousjan at gmail.com> wrote:
>
> Hi List,
> sorry for the crosspost, I have also asked the question here
> https://answers.launchpad.net/cython/+question/31930, but
> probably the list is a better place to ask.
>
>
> In the new version of Cython I am excited to report that overloading
> operators seems to work just fine. This is great, because in the high energy
> physics community, people think it's a good idea to abuse the () operator
> for iterators to mean next (among other nonsense).
>
> So I have a short class in C++, the code is below, and as posted, everything
> works fine.
> However, when I try to change "def add(self)" to "def __add__(self)" in
> myTest I get complaints of the sort
>
>   File "test.py", line 37, in <module>
>     print test + 3
>   File "stdMatrix.pyx", line 67, in stdMatrix.myTest.__add__
> (stdMatrix.cpp:452)
>     return self.thisPtr.add(<int>other)
> AttributeError: 'stdMatrix.myTest' object has no attribute 'thisPtr'
>
> I find this quite surprising, and could use some enlightenment. I can easily
> enough work around this, but this is a bug, no?
>
> Thanks for cython and best of luck on your sprint.
> Best,
>     Jan
>
> ------------ class_operators.cc ---------------
> class TEST
> {
> public:
>     int x;
>     TEST(int y) {x=y;}
>     int operator() () {x = 17; return x;}
>     int operator+(int y) {return x+y;}
>     int minimize() {return 25;}
> };
> ----------------------------------------------------------
>
> ------stdMatrix.pyx----------------------------------------
> cdef extern from "class_operators.cc":
>     ctypedef struct monkey "TEST":
>         int x
>         int (*minimize)()
>         int add "operator+"(int y)
>         int call "operator()"()
>     monkey* new_Test "new TEST"(int y)
>     void del_Test "delete"(monkey* x)
>
> cdef class myTest:
>     cdef monkey *thisPtr
>     def __cinit__(self, int y):
>         self.thisPtr = new_Test(y)
>     def __dealloc__(self):
>         del_Test(self.thisPtr)
>     def min(self):
>         return self.thisPtr.minimize()
>     def plus(self, other):
>         return self.thisPtr.add(other)
>     def call(self):
>         return self.thisPtr.call()
> --------------------------------------------------------------------
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From curiousjan at gmail.com  Sat May 10 01:17:23 2008
From: curiousjan at gmail.com (Jan Strube)
Date: Fri, 9 May 2008 16:17:23 -0700
Subject: [Cython] cython add vs __add__
In-Reply-To: <e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>
References: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>
	<e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>
Message-ID: <76513339-9B13-435A-8F56-27DD69EEA841@gmail.com>

Very interesting.
Thanks for the tip.

Cheers,
     Jan


On May 09, 2008, at 3:59 PM, Lisandro Dalcin wrote:

> I believe you are right, it seems a bug, I'v experienced this in the
> past, and there and used this fixes
>
> cdef class myTest
>     def __add__(myTest self, other): # note declaration: myTest self
>         return self.thisPtr.add(other)
>
> You should go on with this, It should even work when this get fixed
> (in the case this is actually a bug and I'm not missing somethig)
>
> So rename your 'plus' and 'call' to standard __add__ and __call__ with
> the declaration trick above, and all should just work.
>
> Regards,
>
>
>
> On 5/9/08, Jan Strube <curiousjan at gmail.com> wrote:
>>
>> Hi List,
>> sorry for the crosspost, I have also asked the question here
>> https://answers.launchpad.net/cython/+question/31930, but
>> probably the list is a better place to ask.
>>
>>
>> In the new version of Cython I am excited to report that overloading
>> operators seems to work just fine. This is great, because in the  
>> high energy
>> physics community, people think it's a good idea to abuse the ()  
>> operator
>> for iterators to mean next (among other nonsense).
>>
>> So I have a short class in C++, the code is below, and as posted,  
>> everything
>> works fine.
>> However, when I try to change "def add(self)" to "def __add__ 
>> (self)" in
>> myTest I get complaints of the sort
>>
>>   File "test.py", line 37, in <module>
>>     print test + 3
>>   File "stdMatrix.pyx", line 67, in stdMatrix.myTest.__add__
>> (stdMatrix.cpp:452)
>>     return self.thisPtr.add(<int>other)
>> AttributeError: 'stdMatrix.myTest' object has no attribute 'thisPtr'
>>
>> I find this quite surprising, and could use some enlightenment. I  
>> can easily
>> enough work around this, but this is a bug, no?
>>
>> Thanks for cython and best of luck on your sprint.
>> Best,
>>     Jan
>>
>> ------------ class_operators.cc ---------------
>> class TEST
>> {
>> public:
>>     int x;
>>     TEST(int y) {x=y;}
>>     int operator() () {x = 17; return x;}
>>     int operator+(int y) {return x+y;}
>>     int minimize() {return 25;}
>> };
>> ----------------------------------------------------------
>>
>> ------stdMatrix.pyx----------------------------------------
>> cdef extern from "class_operators.cc":
>>     ctypedef struct monkey "TEST":
>>         int x
>>         int (*minimize)()
>>         int add "operator+"(int y)
>>         int call "operator()"()
>>     monkey* new_Test "new TEST"(int y)
>>     void del_Test "delete"(monkey* x)
>>
>> cdef class myTest:
>>     cdef monkey *thisPtr
>>     def __cinit__(self, int y):
>>         self.thisPtr = new_Test(y)
>>     def __dealloc__(self):
>>         del_Test(self.thisPtr)
>>     def min(self):
>>         return self.thisPtr.minimize()
>>     def plus(self, other):
>>         return self.thisPtr.add(other)
>>     def call(self):
>>         return self.thisPtr.call()
>> --------------------------------------------------------------------
>>
>>
>> _______________________________________________
>>  Cython-dev mailing list
>>  Cython-dev at codespeak.net
>>  http://codespeak.net/mailman/listinfo/cython-dev
>>
>>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594


From greg.ewing at canterbury.ac.nz  Sat May 10 03:12:04 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 10 May 2008 13:12:04 +1200
Subject: [Cython] PyPy parser
In-Reply-To: <EE908415-ACC3-4AB7-B1D9-15057628CDA0@math.washington.edu>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
	<e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
	<AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>
	<e437f2f0805071537n676db015x8e4decc6dafb3a1@mail.gmail.com>
	<743600FF-F1BC-48F4-BDC6-EC89A99396BA@math.washington.edu>
	<4823B075.5000903@canterbury.ac.nz>
	<EE908415-ACC3-4AB7-B1D9-15057628CDA0@math.washington.edu>
Message-ID: <4824F664.2000807@canterbury.ac.nz>

Robert Bradshaw wrote:
> I think the C declarations are representable by a generative grammar,  
> so this method could work.

It's not the grammar that's the issue, it's the parsing
algorithm. To do it with an LL(1) parser, you need to
be able to tell whether you're looking at a type name.
The current parser does this by remembering all the
names it's seen being declared as types, and this affects
the decisions it makes later on. It would be hard to do
this without being able to embed arbitrary actions in
the grammar rules.

There are also other places where it uses contextual
information. For example, it passes down a flag indicating
whether a statement is inside a cdef extern block, and
if so, it treats everything as though it had 'cdef' in
front of it.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat May 10 04:31:30 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 10 May 2008 14:31:30 +1200
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <48246C10.6050703@behnel.de>
References: <48243C21.2020207@canterbury.ac.nz> <48246C10.6050703@behnel.de>
Message-ID: <48250902.2020804@canterbury.ac.nz>

Stefan Behnel wrote:

>>     for x < i < y:
>>       ...
> 
> while this is shorter, it also breaks existing code - although I assume you
> left the old syntax in?

Yes, it will be there for a while, although it will
produce a deprecation warning. Ultimately I would like
there to be One Way to do it.

>>* There are two new builtin functions, typecheck() and
>>   issubtype(), for checking the types of arguments more safely
>>   (since isinstance and issubclass can be fooled).
> 
> Could you elaborate on the actual problem here?

It's possible for an object to lie about its class
to isinstance() via its  __class__ attribute. This is
not acceptable when you need to ensure that the object
has a particular C layout.

> Could you imagine using our doctest based test runner as well?

To do what, exactly?

-- 
Greg


From greg.ewing at canterbury.ac.nz  Sat May 10 05:04:01 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 10 May 2008 15:04:01 +1200
Subject: [Cython] cython add vs __add__
In-Reply-To: <e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>
References: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>
	<e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>
Message-ID: <482510A1.7090806@canterbury.ac.nz>

Lisandro Dalcin wrote:
> I believe you are right, it seems a bug,
> 
> cdef class myTest
>     def __add__(myTest self, other): # note declaration: myTest self
>         return self.thisPtr.add(other)

It's not a bug. With extension types, the first argument
of operator methods isn't necessarily self, so it's not
automatically typed as such.

See the section on Special Methods of Extension Types in
the docs for a full explanation.

-- 
Greg

From robertwb at math.washington.edu  Sat May 10 05:54:29 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 9 May 2008 20:54:29 -0700
Subject: [Cython] PyPy parser
In-Reply-To: <4824F664.2000807@canterbury.ac.nz>
References: <48215385.50207@student.matnat.uio.no>
	<10619.194.114.62.39.1210146701.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<482161DF.9010108@student.matnat.uio.no>
	<e437f2f0805070808k528b56b6ta1ce2b29e20c8eff@mail.gmail.com>
	<AACFBB6E-0892-41C1-8500-5503055841EF@math.washington.edu>
	<e437f2f0805071537n676db015x8e4decc6dafb3a1@mail.gmail.com>
	<743600FF-F1BC-48F4-BDC6-EC89A99396BA@math.washington.edu>
	<4823B075.5000903@canterbury.ac.nz>
	<EE908415-ACC3-4AB7-B1D9-15057628CDA0@math.washington.edu>
	<4824F664.2000807@canterbury.ac.nz>
Message-ID: <BF901EE2-0F74-4FB6-AAE5-F698E0D0F1CC@math.washington.edu>

On May 9, 2008, at 6:12 PM, Greg Ewing wrote:
> Robert Bradshaw wrote:
>> I think the C declarations are representable by a generative grammar,
>> so this method could work.
>
> It's not the grammar that's the issue, it's the parsing
> algorithm. To do it with an LL(1) parser, you need to
> be able to tell whether you're looking at a type name.
> The current parser does this by remembering all the
> names it's seen being declared as types, and this affects
> the decisions it makes later on. It would be hard to do
> this without being able to embed arbitrary actions in
> the grammar rules.

Yes, determining whether or not a identifier is a type or not would  
happen at a higher level. This needs to be dealt with anyways to  
handle parameterized types.

> There are also other places where it uses contextual
> information. For example, it passes down a flag indicating
> whether a statement is inside a cdef extern block, and
> if so, it treats everything as though it had 'cdef' in
> front of it.


This isn't a blocker, it would be written into the grammar in a more  
direct way.

- Robert


From stefan_ml at behnel.de  Sat May 10 07:32:34 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 10 May 2008 07:32:34 +0200
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <48250902.2020804@canterbury.ac.nz>
References: <48243C21.2020207@canterbury.ac.nz> <48246C10.6050703@behnel.de>
	<48250902.2020804@canterbury.ac.nz>
Message-ID: <48253372.9060802@behnel.de>

Hi Greg,

Greg Ewing wrote:
> Stefan Behnel wrote:
>>> * There are two new builtin functions, typecheck() and
>>>   issubtype(), for checking the types of arguments more safely
>>>   (since isinstance and issubclass can be fooled).
>>
>> Could you elaborate on the actual problem here?
> 
> It's possible for an object to lie about its class
> to isinstance() via its  __class__ attribute. This is
> not acceptable when you need to ensure that the object
> has a particular C layout.

Wouldn't special casing a type test for builtins and extension classes be
better here than adding new functions? I mean, we already had that with
getattr3(), where the right thing to do would have been to fix builtin functions.


>> Could you imagine using our doctest based test runner as well?
> 
> To do what, exactly?

To share a common test suite. Currently, you have a test suite that is mostly
based on comparing the C source, while our test suite is mostly based on
building the module and running a doctest against it. The sources generated by
Pyrex and Cython look very different, so your approach is not portable, but
our test suite can easily be used for both and the tests that are supported by
both compilers must obviously yield the same results. So if you added doctests
to your test modules (as we do for ours), you could still validate the
resulting source code for Pyrex, but we could both join forces and benefit
from a growing test suite.

This would allow us to see if a bug that gets fixed in one compiler must also
be fixed in the other, and it would also make it easier to keep the feature
set of both compilers somewhat aligned.

Stefan

From stefan_ml at behnel.de  Sat May 10 07:36:15 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 10 May 2008 07:36:15 +0200
Subject: [Cython] How to deal with byte strings and unicode strings at
 the API level
In-Reply-To: <e7ba66e40805091350t58ffdff5i698415824653f4c9@mail.gmail.com>
References: <e7ba66e40805072305o5f264ee9m5741cb5a1dabc336@mail.gmail.com>	<e7ba66e40805080833w4eedfa28ia0b509c5c8ef91ac@mail.gmail.com>	<D2F0D019-945D-40F9-8F2D-2504599AF539@math.washington.edu>	<48240A6A.7040700@behnel.de>
	<48243A77.9040008@behnel.de>	<e7ba66e40805090920i47601229v1f70f0fd74f1b5e6@mail.gmail.com>	<482485F8.5080508@behnel.de>	<e7ba66e40805091126x7476e91et1cdb83a2d167e4e7@mail.gmail.com>	<4824A65E.90007@behnel.de>
	<4824AA80.3040803@behnel.de>
	<e7ba66e40805091350t58ffdff5i698415824653f4c9@mail.gmail.com>
Message-ID: <4825344F.20201@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Stefan, many, many thanks for your explanations. I believe I've
> started to understand the whole beast. Please clarify me this:
> 
> Suppose I write this method in a pyx file:
> 
> def foo(char value[]): pass
> 
> and next call it (in a Py3 runtime env) in this two ways:
> 
> 1-  foo(b"abc")
> 
> 2-  foo("abc")
> 
> then (1) should be just fine, Cython should pass the raw C data as it
> comes. But in the case of (2), will/should Cython (or at the Python C
> API level) generate an error?

Yes, you will get a TypeError for (2). The Python3 equivalent is this:

    def foo(value): value = bytes(value)

Stefan

From dagss at student.matnat.uio.no  Sat May 10 12:14:44 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 10 May 2008 12:14:44 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
Message-ID: <48257594.7080709@student.matnat.uio.no>

An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080510/a3c1f2a8/attachment.htm 

From stefan_ml at behnel.de  Sat May 10 14:34:43 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 10 May 2008 14:34:43 +0200
Subject: [Cython] [Pyrex] ANN: Pyrex 0.9.7
In-Reply-To: <205284A8-3452-4FEE-BDCD-D5DECAD61CE9@bryant.edu>
References: <48243C21.2020207@canterbury.ac.nz>
	<205284A8-3452-4FEE-BDCD-D5DECAD61CE9@bryant.edu>
Message-ID: <48259663.3070009@behnel.de>

Hi,

Brian Blais wrote:
> In python, I am used to syntax:
> 
> for var in stuff:
> 
> where my eye finds "var" and then "stuff", so I first find out the
> relevant variable, and then the values it will take.
> 
> in old pyrex syntax,
> 
> for var in begin<= var < end:
> 
> my eye finds "var" and then "begin" and "end", so I first find out the
> relevant variable, and then the values it will take.

Actually, in the old Pyrex syntax, it was

   for var *from* begin <= var < end:

but since now we have

>  for begin <= var < end:

would it be that bad to always use

    for var in ...

instead and just distinguish between

    for var in something:

and

    for var in begin <= var < end:

?

Then again, there even is a switch in Cython that allows you to convert

    for var in range(begin, end):

to

    for var in begin <= var < end:

so maybe the whole discussion is somewhat pointless anyway. I don't quite see
why we shouldn't just always convert

    for var in range(begin, end):

to

    for var in begin <= var < end:

*iff* var is cdef-ed as a C integer type. According to Robert, there's a
difference if the loop overflows, but that case is almost certainly a
programming error when var is a C type, and there is no such thing as Python
compatibility for C variables anyway.

Opinions?

Stefan

From dagss at student.matnat.uio.no  Sat May 10 14:59:15 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 10 May 2008 14:59:15 +0200
Subject: [Cython] (repost in plain text) Python type optimizations (NumPy
	GSoC-related)
In-Reply-To: <48257594.7080709@student.matnat.uio.no>
References: <482423C4.9040005@student.matnat.uio.no>	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<48257594.7080709@student.matnat.uio.no>
Message-ID: <48259C23.9020904@student.matnat.uio.no>

[Sorry about the HTML, have no idea what happened. Posting again in a 
proper format.]

Thanks for the feedback, I have no problems seeing weaknesses in my
suggestion, but I think there are real issues here that you don't
address either.

 >> On the
 >> other hand, one has to remember to do "print y.shape" for tuple-access
 >> but "y.dimensions[i]" for speedy, non-Python access. (And this
 >> difference in behaviour comes entirely from strides having a name-
 >> clash
 >> while shape/dimensions do not).
 >>
 >
 > This is a numpy api issue, nothing to do with Cython.
 >
Sure. Remember however: Which API are we talking about here, when we say
"NumPy API"? When using the extension type, that is essentially the C API.

Basically, this seem to imply:

a) The NumPy community has to provide a stable API for their extension
type (i.e. stable API in header files)
b) ...that lies close to the Python API

What I'd like is to have support for a thin layer in the pxd file so
that one is able to emulate/reimplement parts of the _Python_ API. Then,
that pxd file can be shipped with NumPy, and if changes are made to the
C API/extension type then just change and ship a new pxd as well; client
code remains the same.

This is so that code written in Python towards the Python NumPy API can
simply be typed and compiled. The "NumPy C API" (which is what is really
used when one declares extension types etc.) is however really a
different API to NumPy, and so one cannot in general assume that one
will ever be able to simply "type and compile" Python code towards it.

My point is that we would like this to be attractive for current Python
users, using the current Python API. The less they have to know about
the C API the better, and then relying on "fudging" the C API to "almost
look like" the Python API doesn't seem to cut it.

Ahhh.... We might have slightly different goals here? That is, can you
agree with this statement?:

* Your goal is making writing Cython code towards the NumPy "C API"
(extension type etc.) a little more convenient
* My goal is to start with the Python API and simply speed it up.

Then that is perhaps a fundamental question that should be resolved
first. And I do not think they can coincide, that is discussable as well.

On to your suggestion: In your prototype you use the "cdef class". Sure,
for classes implemented in Cython the C API and Python API coincide, but
that is not so for other extension types. Do you mean to imply that
NumPy ndarrays will have to be reimplemented entirely in Cython in order
to generate effective code for then? I don't think that will happen (nor
should it be necesarry).

If one changes your example slightly and putting the __getitem__ in a
cdef struct for an extension type then you end up "putting them in some
overlay", since the real implementation is in another module (written in
C). I just wanted to make the overlay more explicit, or if it is not
explicit in the syntax as such, at least we need to know that that is
what we're doing.

BTW, as I said, the "different name thing" was something I hoped to get
rid of, it was just to make it clear what was happpening. Sorry, that
was a bit confusing.


-- 
Dag Sverre


From bblais at gmail.com  Sat May 10 16:09:13 2008
From: bblais at gmail.com (Brian Blais)
Date: Sat, 10 May 2008 10:09:13 -0400
Subject: [Cython] [Pyrex] ANN: Pyrex 0.9.7
In-Reply-To: <48259663.3070009@behnel.de>
References: <48243C21.2020207@canterbury.ac.nz>
	<205284A8-3452-4FEE-BDCD-D5DECAD61CE9@bryant.edu>
	<48259663.3070009@behnel.de>
Message-ID: <69731ED6-129E-454F-AE7C-2F22652B3DBF@gmail.com>

On May 10, 2008, at May 10:8:34 AM, Stefan Behnel wrote:

> Brian Blais wrote:
>> In python, I am used to syntax:
>>
>> for var in stuff:
>>
>> where my eye finds "var" and then "stuff", so I first find out the
>> relevant variable, and then the values it will take.
>>
>> in old pyrex syntax,
>>
>> for var in begin<= var < end:
>>
>> my eye finds "var" and then "begin" and "end", so I first find out  
>> the
>> relevant variable, and then the values it will take.
>
> Actually, in the old Pyrex syntax, it was
>
>    for var *from* begin <= var < end:

ooops, my bad.  :)


>
> so maybe the whole discussion is somewhat pointless anyway. I don't  
> quite see
> why we shouldn't just always convert
>
>     for var in range(begin, end):
>
> to
>
>     for var in begin <= var < end:
>
> *iff* var is cdef-ed as a C integer type. According to Robert,  
> there's a
> difference if the loop overflows, but that case is almost certainly a
> programming error when var is a C type, and there is no such thing  
> as Python
> compatibility for C variables anyway.
>

that would be great!


			bb

From stefan_ml at behnel.de  Sat May 10 16:23:55 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 10 May 2008 16:23:55 +0200
Subject: [Cython] String interning and Python 3
Message-ID: <4825AFFB.6040305@behnel.de>

Hi,

I'm wondering how to continue the support for this feature given the fact that
identifiers are Unicode strings in Py3. We currently only intern byte strings
that look like Python identifiers, so in Py3, they simply no longer look like
identifiers, as they are not Unicode strings.

I can see four ways how to deal with this:

1) drop string interning completely

2) disable string interning in Py3 and use normally created byte strings instead

3) keep separate sets of identifier-like byte strings and unicode strings in
the compiler and write them into the C file. Then, depending on the Python
version, either intern the byte strings or the unicode strings, and create the
other set as un-interned strings.

4) keep the information if a string should be interned for all strings we deal
with (bytes and unicode), remove the intern tab and merge it with the general
string tab by adding an additional field "intern". Then __Pyx_InitStrings()
would create the strings differently depending on the compile time Python
version, i.e., it would intern Unicode identifiers in Py3 and byte string
identifiers in Py2, and create everything else as normal strings.

Personally, I favour 4) - although I could live with 1) - but since I'm not
quite sure what the original intention of string interning was (saving
memory?), I'd like to hear other opinions first.

Stefan

From languitar at semipol.de  Sat May 10 18:47:30 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Sat, 10 May 2008 18:47:30 +0200
Subject: [Cython] cimport'ing module with cdef public declarations
Message-ID: <4825D1A2.5000900@semipol.de>

Hi again,

a few weeks ago I reported a problem[1] about cimport'ing a module that
contains cdef public declarations. I could manage that problem that time
but now I'm facing the same problem again. Does anyone know a solution
to this?

[1] http://codespeak.net/pipermail/cython-dev/2008-April/000582.html

Thanks
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080510/49480ebd/attachment.pgp 

From robertwb at math.washington.edu  Sat May 10 19:47:16 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 10 May 2008 10:47:16 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <48257594.7080709@student.matnat.uio.no>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<48257594.7080709@student.matnat.uio.no>
Message-ID: <5DB2E374-D355-4A6E-A2EC-9382AC4E5556@math.washington.edu>

On May 10, 2008, at 3:14 AM, Dag Sverre Seljebotn wrote:

> Thanks for the feedback, I have no problems seeing weaknesses in my  
> suggestion, but I think there are real issues here that you don't  
> address either.
>
>>> On the other hand, one has to remember to do "print y.shape" for  
>>> tuple-access but "y.dimensions[i]" for speedy, non-Python access.  
>>> (And this difference in behaviour comes entirely from strides  
>>> having a name- clash while shape/dimensions do not).
>> This is a numpy api issue, nothing to do with Cython.
> Sure. Remember however: Which API are we talking about here, when  
> we say "NumPy API"? When using the extension type, that is  
> essentially the C API.

Yes, this I was talking about the NumPy C API being different from  
the NumPy Py API.

> Basically, this seem to imply:
>
> a) The NumPy community has to provide a stable API for their  
> extension type (i.e. stable API in header files)
> b) ...that lies close to the Python API
>
> What I'd like is to have support for a thin layer in the pxd file  
> so that one is able to emulate/reimplement parts of the _Python_  
> API. Then, that pxd file can be shipped with NumPy, and if changes  
> are made to the C API/extension type then just change and ship a  
> new pxd as well; client code remains the same.
>
> This is so that code written in Python towards the Python NumPy API  
> can simply be typed and compiled. The "NumPy C API" (which is what  
> is really used when one declares extension types etc.) is however  
> really a different API to NumPy, and so one cannot in general  
> assume that one will ever be able to simply "type and compile"  
> Python code towards it.
>
> My point is that we would like this to be attractive for current  
> Python users, using the current Python API. The less they have to  
> know about the C API the better, and then relying on "fudging" the  
> C API to "almost look like" the Python API doesn't seem to cut it.
>
> Ahhh.... We might have slightly different goals here? That is, can  
> you agree with this statement?:
>
> * Your goal is making writing Cython code towards the NumPy "C  
> API" (extension type etc.) a little more convenient
> * My goal is to start with the Python API and simply speed it up.
>
> Then that is perhaps a fundamental question that should be resolved  
> first. And I do not think they can coincide, that is discussable as  
> well.

I think both goals are important, and can coincide. Our target  
audience is SciPy developers as well as SciPy users.

> On to your suggestion: In your prototype you use the "cdef class".  
> Sure, for classes implemented in Cython the C API and Python API  
> coincide, but that is not so for other extension types. Do you mean  
> to imply that NumPy ndarrays will have to be reimplemented entirely  
> in Cython in order to generate effective code for then? I don't  
> think that will happen (nor should it be necesarry).

Incidentally, some of the classes may be implemented in Cython (there  
is a relevant GSoC for the SciPy folks on this), but that is  
completely besides the point. Looking at my example, I had a.pxd and  
b.pyx that used a.A. If A was implemented in Cython or pure C it made  
no difference.

> If one changes your example slightly and putting the __getitem__ in  
> a cdef struct for an extension type then you end up "putting them  
> in some overlay", since the real implementation is in another  
> module (written in C). I just wanted to make the overlay more  
> explicit, or if it is not explicit in the syntax as such, at least  
> we need to know that that is what we're doing.

Yes, any actual code in a pxd file is an "overlay" as you call it.  
But semantically it belongs to the class, and should be used whenever  
that class is used. Code in a .pyx file is compiled to the module.  
Code in a .pxd file is only used to help compile other modules.

> BTW, as I said, the "different name thing" was something I hoped to  
> get rid of, it was just to make it clear what was happpening.  
> Sorry, that was a bit confusing.

No problem. BTW, did you have any comments about my proposal for how  
use parameterize types?

- Robert


From roed at math.harvard.edu  Sat May  3 09:27:14 2008
From: roed at math.harvard.edu (David Roe)
Date: Sat, 3 May 2008 03:27:14 -0400
Subject: [Cython] Comments on Documentation
Message-ID: <5cddadeb0805030027qe47e3e1y311c242ad2456680@mail.gmail.com>

Hey guys,
I saw the post about the documentation at
http://www.mudskipper.ca/cython-doc/ on sage-devel, and have a few
typos to point out.  It looks awesome though.

1) Indentation problem
(http://www.mudskipper.ca/cython-doc/docs/extension_types.html):

cdef Shrubbery another_shrubbery(Shrubbery sh1):
    cdef Shrubbery sh2
    sh2 = Shrubbery()
    sh2.width = sh1.width
sh2.height = sh1.height
return sh2

2) Another indentation error
(http://www.mudskipper.ca/cython-doc/docs/extension_types.html):
cdef class CheeseShop:

cdef object cheeses

def __cinit__(self):
    self.cheeses = []

property cheese:

    def __get__(self):
        return "We don't have: %s" % self.cheeses

    def __set__(self, value):
        self.cheeses.append(value)

    def __del__(self):
        del self.cheeses[:]

3) Again (http://www.mudskipper.ca/cython-doc/docs/extension_types.html):
cdef class Parrot:

cdef void describe(self):
    print "This parrot is resting."

cdef class Norwegian(Parrot):

cdef void describe(self):
    Parrot.describe(self)
    print "Lovely plumage!"

4) This should be a header?
(http://www.mudskipper.ca/cython-doc/docs/sharing_declarations.html):
What a Definition File contains A definition file can contain:

5) A linebreak error in
(http://www.mudskipper.ca/cython-doc/docs/sharing_declarations.html):
"cdef float cube(float)" should go before "spammery.pyx"

6) The Sharing extension types example is confusing because the file
names are mixed into the code.  They should be split (and in different
colors as in the "Using cimport to resolve naming conflicts" section).

7) The highlighting on "The cdef extern from clause does three
things:" is wrong
(http://www.mudskipper.ca/cython-doc/docs/external_C_code.html)

8) There's a table missing in the "Styles of struct, union and enum
declaration" section
(http://www.mudskipper.ca/cython-doc/docs/external_C_code.html)

9) In "Declaring a function as callable without the GIL," I think
there should be a with before the nogil in the final example.  The
last sentence may have a typo as well.  I could be wrong though: I
don't actually know the syntax you're explaining.

10) Capitalization problem
(http://www.mudskipper.ca/cython-doc/docs/pyrex_differences.html)
Cdef bint b = x

David

From kirr at mns.spb.ru  Sat May  3 13:38:07 2008
From: kirr at mns.spb.ru (Kirill Smelkov)
Date: Sat, 03 May 2008 15:38:07 +0400
Subject: [Cython] [PATCH 0 of 2] Handy patches for those,
	running Cython in-tree
Message-ID: <patchbomb.1209814687@tugrik2.mns.mnsspb.ru>

Hi,

Two handy patches to run cython in-tree, i.e. without installation at all.

Please apply.

From anjiro at cc.gatech.edu  Mon May  5 20:46:00 2008
From: anjiro at cc.gatech.edu (Daniel Ashbrook)
Date: Mon, 05 May 2008 14:46:00 -0400
Subject: [Cython] [Pyrex] newbie list processing question
In-Reply-To: <0F507576-A9F6-43CA-8FF7-B744C1A6796B@math.washington.edu>
References: <481F4CF3.1020204@cc.gatech.edu>
	<0F507576-A9F6-43CA-8FF7-B744C1A6796B@math.washington.edu>
Message-ID: <481F55E8.20603@cc.gatech.edu>

Robert Bradshaw wrote:
>> def addOne(l):
>>    return [i+1 for i in l]
 >> ...
> 
> This last implementation of addOne should work as is in Cython, and will 
> be nearly optimal (assuming your CPU has reasonable branch prediction). 
> However, if you are manipulating word-sized integers, using C arrays 
> will give you a manyfold over python arithmetic.

So in the real code, I'm actually doing float math. And I'll be wanting 
to return my results in a list object. There will be many thousands of 
float results; what's the best way to deal with that? Use a C-specific 
data structure to store the results then turn it into a list somehow?

> Did you do
> 
> cdef int i
> for i from 0 <= i < len(L):
>     ...

Ah, I missed the "cdef int i" part of it.

> In Cython, if one writes
> 
> L[i]
> 
> where i is a cdef int, then it checks to see at runtime if L is a list 
> and accesses its elements via a macro. Otherwise one can use 
> PyList_SetItem and friends, but as you have noticed that is cumbersome 
> (as well ahs being hard to read).

Oh ho! That works very nicely; I'll include code to help others in the 
future:

   cdef int i
   for i from 0 <= i < len(l):
     l[i] = l[i] + 1


Thanks for the help!


dan

From greg at cosc.canterbury.ac.nz  Fri May  9 11:07:21 2008
From: greg at cosc.canterbury.ac.nz (greg)
Date: Fri, 09 May 2008 21:07:21 +1200
Subject: [Cython] ANN: Pyrex 0.9.7
Message-ID: <48241449.5040309@cosc.canterbury.ac.nz>

Pyrex 0.9.7 is now available:

   http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/

Highlights of this version:

* I have streamlined the integer for-loop syntax. Instead
   of the loop variable redundantly appearing in two places,
   it's now just

     for x < i < y:
       ...

* If you declare a variable as a list or dict, then calls
   to some of its methods will be compiled into type-specific
   Python API calls instead of generic ones.

* Most built-in constants are referenced directly instead of
   via dict lookup.

* There are two new builtin functions, typecheck() and
   issubtype(), for checking the types of arguments more safely
   (since isinstance and issubclass can be fooled).


What is Pyrex?
--------------

Pyrex is a language for writing Python extension modules.
It lets you freely mix operations on Python and C data, with
all Python reference counting and error checking handled
automatically.

From bblais at bryant.edu  Sat May 10 13:16:26 2008
From: bblais at bryant.edu (Brian Blais)
Date: Sat, 10 May 2008 07:16:26 -0400
Subject: [Cython] [Pyrex] ANN: Pyrex 0.9.7
In-Reply-To: <48243C21.2020207@canterbury.ac.nz>
References: <48243C21.2020207@canterbury.ac.nz>
Message-ID: <205284A8-3452-4FEE-BDCD-D5DECAD61CE9@bryant.edu>

On May 9, 2008, at May 9:7:57 AM, Greg Ewing wrote:

>
> * I have streamlined the integer for-loop syntax. Instead
>    of the loop variable redundantly appearing in two places,
>    it's now just
>
>      for x < i < y:
>        ...
>
>

Not that my opinion matters a whole lot, but I prefer the redundant  
syntax.  This new syntax solves a problem, and introduces a problem  
which (in my opinion) is worse.  In python, I am used to syntax:

for var in stuff:

where my eye finds "var" and then "stuff", so I first find out the  
relevant variable, and then the values it will take.

in old pyrex syntax,

for var in begin<= var < end:

my eye finds "var" and then "begin" and "end", so I first find out  
the relevant variable, and then the values it will take.

In the new syntax,

for begin <= var < end:

my eye finds some values of var, then the relevant variable, then the  
end.  It changes the logic of the for-loop organization in my head,  
and as I read.  It makes it less like Python, and thus more jarring  
to go between the two.

Personally, I never minded the redundancy (how much extra typing is  
it really?)

I find breaking readability, and a parallel with existing python  
syntax to be far far worse.  I hope you keep the two syntaxes, and  
see what the response is when developers actually use it for a  
while.  My gut feeling is that the new syntax will not be favored,  
but I could very well be wrong.


		Just my 2c


				thanks,


					Brian Blais


-- 
Brian Blais
bblais at bryant.edu
http://web.bryant.edu/~bblais



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080510/1c331c88/attachment-0001.htm 

From jek-gmane at kleckner.net  Sat May 10 22:27:30 2008
From: jek-gmane at kleckner.net (Jim Kleckner)
Date: Sat, 10 May 2008 13:27:30 -0700
Subject: [Cython] [Pyrex] ANN: Pyrex 0.9.7
In-Reply-To: <48259663.3070009@behnel.de>
References: <48243C21.2020207@canterbury.ac.nz>	<205284A8-3452-4FEE-BDCD-D5DECAD61CE9@bryant.edu>
	<48259663.3070009@behnel.de>
Message-ID: <g050fj$b2$1@ger.gmane.org>

Stefan Behnel wrote:
> Brian Blais wrote:
>> In python, I am used to syntax:
>>
>> for var in stuff:
>>
>> where my eye finds "var" and then "stuff", so I first find out the
>> relevant variable, and then the values it will take.

+1 to the value of seeing the "var" first and being more consistent with 
Python (being more Pythonic).


From dalcinl at gmail.com  Sat May 10 23:06:51 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 10 May 2008 18:06:51 -0300
Subject: [Cython] String interning and Python 3
In-Reply-To: <4825AFFB.6040305@behnel.de>
References: <4825AFFB.6040305@behnel.de>
Message-ID: <e7ba66e40805101406n468ef323i4aa49b7b39c8cb62@mail.gmail.com>

Stefan , If  (4) is doable with no much effort, then I believe this is
the right way.

As you said, one of the points of string interning is saving memory.
But there is also another very important benefit. Look at this for
Python 2.6 sources in stringobject.c

int
_PyString_Eq(PyObject *o1, PyObject *o2)
{
	PyStringObject *a = (PyStringObject*) o1;
	PyStringObject *b = (PyStringObject*) o2;
        return Py_SIZE(a) == Py_SIZE(b)
          && *a->ob_sval == *b->ob_sval
          && memcmp(a->ob_sval, b->ob_sval, Py_SIZE(a)) == 0;
}


As you can see the line with '*a->ob_sval == *b->ob_sval' provides a
fast path for string equality. So if o1 and o2 are the same (interned)
strings then with a pointer comparison you avoid at all the memcmp
call. And this is very, very important in dictionary lookups to make
that operation faster in the case the keys are strings.



On 5/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>  I'm wondering how to continue the support for this feature given the fact that
>  identifiers are Unicode strings in Py3. We currently only intern byte strings
>  that look like Python identifiers, so in Py3, they simply no longer look like
>  identifiers, as they are not Unicode strings.
>
>  I can see four ways how to deal with this:
>
>  1) drop string interning completely
>
>  2) disable string interning in Py3 and use normally created byte strings instead
>
>  3) keep separate sets of identifier-like byte strings and unicode strings in
>  the compiler and write them into the C file. Then, depending on the Python
>  version, either intern the byte strings or the unicode strings, and create the
>  other set as un-interned strings.
>
>  4) keep the information if a string should be interned for all strings we deal
>  with (bytes and unicode), remove the intern tab and merge it with the general
>  string tab by adding an additional field "intern". Then __Pyx_InitStrings()
>  would create the strings differently depending on the compile time Python
>  version, i.e., it would intern Unicode identifiers in Py3 and byte string
>  identifiers in Py2, and create everything else as normal strings.
>
>  Personally, I favour 4) - although I could live with 1) - but since I'm not
>  quite sure what the original intention of string interning was (saving
>  memory?), I'd like to hear other opinions first.
>
>  Stefan
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sat May 10 23:30:51 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 10 May 2008 18:30:51 -0300
Subject: [Cython] cython add vs __add__
In-Reply-To: <482510A1.7090806@canterbury.ac.nz>
References: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>
	<e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>
	<482510A1.7090806@canterbury.ac.nz>
Message-ID: <e7ba66e40805101430t3d39e54dx490d1dade90eaca5@mail.gmail.com>

Gre, the (Cython) docs say this

Special Method Table
---------------------------------
This table lists all of the special methods together with their
parameter and return types. A parameter named self is of the type the
method belongs to. Other untyped parameters are generic Python
objects.

What do you understand for 'A parameter named self is of the type the
method belongs to'. Anyway, do not you believe the current behavior is
a bit counter-intuitive?

On 5/10/08, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>  > I believe you are right, it seems a bug,
>  >
>
> > cdef class myTest
>  >     def __add__(myTest self, other): # note declaration: myTest self
>  >         return self.thisPtr.add(other)
>
>
> It's not a bug. With extension types, the first argument
>  of operator methods isn't necessarily self, so it's not
>  automatically typed as such.
>
>  See the section on Special Methods of Extension Types in
>  the docs for a full explanation.
>
>  --
>
> Greg
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Sun May 11 00:03:27 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 10 May 2008 19:03:27 -0300
Subject: [Cython] [Pyrex] ANN: Pyrex 0.9.7
In-Reply-To: <48259663.3070009@behnel.de>
References: <48243C21.2020207@canterbury.ac.nz>
	<205284A8-3452-4FEE-BDCD-D5DECAD61CE9@bryant.edu>
	<48259663.3070009@behnel.de>
Message-ID: <e7ba66e40805101503v41be962eheb0d32fdcff8d46d@mail.gmail.com>

On 5/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>  I don't quite see why we shouldn't just always convert
>
>     for var in range(begin, end):
>
>  to
>
>     for var in begin <= var < end:
>
>
> *iff* var is cdef-ed as a C integer type.

I'm definitely +1 on this!!  Regarding to the overflow issue, perhaps
a check can be added before entering the loop. And Cython should take
care if the user modifies the range() arguments inside de loop. What
will you do in this case? IMHO,  matching the Python semantics is the
right way.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Sun May 11 02:04:16 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 12:04:16 +1200
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <48253372.9060802@behnel.de>
References: <48243C21.2020207@canterbury.ac.nz> <48246C10.6050703@behnel.de>
	<48250902.2020804@canterbury.ac.nz> <48253372.9060802@behnel.de>
Message-ID: <48263800.8070606@canterbury.ac.nz>

Stefan Behnel wrote:
> Wouldn't special casing a type test for builtins and extension classes be
> better here than adding new functions?

I'm not sure what you mean by that. There's no run-time
distinction between a builtin class, an extension class
implemented with Pyrex, or an extension class created some
other way. They're all just types.

I could make the Pyrex version of isinstance() behave
differently, but then it wouldn't have quite  the same
semantics as it does in Python. That could be confusing,
especially since you aren't guaranteed to be always
getting the directly-called version.

Rather than fiddle with the semantics, I felt it was
better to provide different functions -- EIBTI and
all that.

They're not really new functions, anyway -- they're
just exposing functions that are in the Python API,
and they're the ones you would use if you were writing
the extension module in C.

> I mean, we already had that with getattr3(), where the right
 > thing to do would have been to fix builtin functions.

That was a matter of pragmatism. I don't currently have
any mechanism for dealing with a C function that can have
more than one signature. I would have had to build a very
special case into the guts of the compiler somewhere,
which I didn't feel was worth doing just for that one
function. So I did the next best thing and provided
something that works, even if it's not ideal. I can
always come back and do something else later.

I think we're coming up against those philosophical
differences again. Pyrex code is not Python code, and
doesn't pretend to be. I try to make things compatible
where reasonably possible, but it's not an overriding
principle.

> So if you added doctests
> to your test modules (as we do for ours), you could still validate the
> resulting source code for Pyrex, but we could both join forces and benefit
> from a growing test suite.

Okay, so you just want me to run the Cython tests as
well. That shouldn't be too hard.

Is there somewhere I can download them from?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun May 11 02:23:12 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 12:23:12 +1200
Subject: [Cython] String interning and Python 3
In-Reply-To: <4825AFFB.6040305@behnel.de>
References: <4825AFFB.6040305@behnel.de>
Message-ID: <48263C70.4000400@canterbury.ac.nz>

Stefan Behnel wrote:

> Personally, I favour 4) - although I could live with 1) - but since I'm not
> quite sure what the original intention of string interning was (saving
> memory?)

No, it's the same reason Python does it -- to speed
up namespace lookups.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun May 11 02:53:03 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 12:53:03 +1200
Subject: [Cython] cython add vs __add__
In-Reply-To: <e7ba66e40805101430t3d39e54dx490d1dade90eaca5@mail.gmail.com>
References: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>
	<e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>
	<482510A1.7090806@canterbury.ac.nz>
	<e7ba66e40805101430t3d39e54dx490d1dade90eaca5@mail.gmail.com>
Message-ID: <4826436F.90608@canterbury.ac.nz>

Lisandro Dalcin wrote:
> 'A parameter named self is of the type the
> method belongs to'.

I mean the documentation indicates when a parameter has
the type of 'self' by using the word 'self' in the table.

It *doesn't* mean that naming it 'self' in your code
will give it the type of self. Sorry about the confusion;
I'll see if I can re-word that part of the docs to make
it clearer.

> Anyway, do not you believe the current behavior is
> a bit counter-intuitive?

It's different from Python, but that's because the whole
way that the operator methods work at the C type slot
level is very different. It's something you just have to
know about and deal with. The section on "Arithmetic
Methods" explains what's going on.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun May 11 02:55:45 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 12:55:45 +1200
Subject: [Cython] cimport'ing module with cdef public declarations
In-Reply-To: <4825D1A2.5000900@semipol.de>
References: <4825D1A2.5000900@semipol.de>
Message-ID: <48264411.7070907@canterbury.ac.nz>

Johannes Wienke wrote:

> a few weeks ago I reported a problem[1] about cimport'ing a module that
> contains cdef public declarations.

 > from ship.icewing.gui cimport plug_observ_data

With extension modules that live inside packages, you need
to make sure that the compiler knows the fully qualified
name of the module.

With the current version of Pyrex, you need to do this by
naming the source files with the full dotted name, e.g.
ship.icewing.gui.pyx and ship.icewing.gui.pxd.

But Cython does things differently in this area, and I'm
not sure exactly what you need to do.

-- 
Greg

From wstein at gmail.com  Sun May 11 03:12:59 2008
From: wstein at gmail.com (William Stein)
Date: Sat, 10 May 2008 18:12:59 -0700
Subject: [Cython] Arc Riley, PyMill, and FUD
Message-ID: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>

Hi Cython-Devel,

Mike Hansen pointed out to me that this Arc Riley guy is making
a lot of noise today about forking Pyrex and Cython.   Everyone
here should be aware of this if you aren't already.   Here's Arc's
blog post:

    http://arcriley.blogspot.com/2008/05/radical-redirection.html

Here's a mailing list post where outlines why and how:

   http://www.pysoy.org/pipermail/pysoy-dev/2008-May/000173.html

Here's the PyMill project website:
    http://pysoy.org:8000/

There are in my opinion some confused and wrong statements
above about Pyrex/Cython there.  Judge for yourself.
Basically he takes all the good aspects of the many valuable
recent discussions and brainstorming threads that have happened
on cython-devel and transforms them into a bunch of FUD.

 -- William

-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From greg.ewing at canterbury.ac.nz  Sun May 11 03:18:29 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 13:18:29 +1200
Subject: [Cython] cimport'ing module with cdef public declarations
In-Reply-To: <48264411.7070907@canterbury.ac.nz>
References: <4825D1A2.5000900@semipol.de> <48264411.7070907@canterbury.ac.nz>
Message-ID: <48264965.2010008@canterbury.ac.nz>

I wrote:

> With extension modules that live inside packages, you need
> to make sure that the compiler knows the fully qualified
> name of the module.

Sorry, I should have read what you wrote more carefully.
The problem isn't what I thought it was.

I'm not sure I can help you with this, because I
believe Cython handles cimporting of functions a
bit differently from Pyrex.

But in Pyrex you wouldn't need the "public"
declaration, as it's only for making functions
available to external C code, not other Pyrex
modules.

-- 
Greg

From arcriley at gmail.com  Sun May 11 03:37:11 2008
From: arcriley at gmail.com (Arc Riley)
Date: Sat, 10 May 2008 21:37:11 -0400
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
Message-ID: <bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>

Take it as FUD if you want.

The truth of the matter is we have our timeline, and we need a
language that is going to work for us, and neither Pyrex nor Cython
fit the bill.  In our discussions tracking Cython's continuing
changes, given the amount of work it'd take us to modify Cython to our
needs and maintain a variant given radical changes being discussed, is
completely unfeasible.

You may want to consider, however, that while you have very good
reasons for making choices, and I am not at all saying you should not
be making those exact choices, those same choices don't work for
everyone.

On Sat, May 10, 2008 at 9:12 PM, William Stein <wstein at gmail.com> wrote:
> Hi Cython-Devel,
>
> Mike Hansen pointed out to me that this Arc Riley guy is making
> a lot of noise today about forking Pyrex and Cython.   Everyone
> here should be aware of this if you aren't already.   Here's Arc's
> blog post:
>
>    http://arcriley.blogspot.com/2008/05/radical-redirection.html
>
> Here's a mailing list post where outlines why and how:
>
>   http://www.pysoy.org/pipermail/pysoy-dev/2008-May/000173.html
>
> Here's the PyMill project website:
>    http://pysoy.org:8000/
>
> There are in my opinion some confused and wrong statements
> above about Pyrex/Cython there.  Judge for yourself.
> Basically he takes all the good aspects of the many valuable
> recent discussions and brainstorming threads that have happened
> on cython-devel and transforms them into a bunch of FUD.
>
>  -- William
>
> --
> William Stein
> Associate Professor of Mathematics
> University of Washington
> http://wstein.org
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ggellner at uoguelph.ca  Sun May 11 04:06:04 2008
From: ggellner at uoguelph.ca (Gabriel Gellner)
Date: Sat, 10 May 2008 22:06:04 -0400
Subject: [Cython] Comments on Documentation
In-Reply-To: <5cddadeb0805030027qe47e3e1y311c242ad2456680@mail.gmail.com>
References: <5cddadeb0805030027qe47e3e1y311c242ad2456680@mail.gmail.com>
Message-ID: <20080511020604.GA25681@basestar>

Thanks for the changes, I will apply them next week (currently getting ready
for a conference ...). If you are feeling ambitious you can try to make a
patch against: 
http://hg.cython.org/cython-docs/

If not don't worry, your email was very clear. Thanks for the help.

Gabriel

On Sat, May 03, 2008 at 03:27:14AM -0400, David Roe wrote:
> Hey guys,
> I saw the post about the documentation at
> http://www.mudskipper.ca/cython-doc/ on sage-devel, and have a few
> typos to point out.  It looks awesome though.
> 
> 1) Indentation problem
> (http://www.mudskipper.ca/cython-doc/docs/extension_types.html):
> 
> cdef Shrubbery another_shrubbery(Shrubbery sh1):
>     cdef Shrubbery sh2
>     sh2 = Shrubbery()
>     sh2.width = sh1.width
> sh2.height = sh1.height
> return sh2
> 
> 2) Another indentation error
> (http://www.mudskipper.ca/cython-doc/docs/extension_types.html):
> cdef class CheeseShop:
> 
> cdef object cheeses
> 
> def __cinit__(self):
>     self.cheeses = []
> 
> property cheese:
> 
>     def __get__(self):
>         return "We don't have: %s" % self.cheeses
> 
>     def __set__(self, value):
>         self.cheeses.append(value)
> 
>     def __del__(self):
>         del self.cheeses[:]
> 
> 3) Again (http://www.mudskipper.ca/cython-doc/docs/extension_types.html):
> cdef class Parrot:
> 
> cdef void describe(self):
>     print "This parrot is resting."
> 
> cdef class Norwegian(Parrot):
> 
> cdef void describe(self):
>     Parrot.describe(self)
>     print "Lovely plumage!"
> 
> 4) This should be a header?
> (http://www.mudskipper.ca/cython-doc/docs/sharing_declarations.html):
> What a Definition File contains A definition file can contain:
> 
> 5) A linebreak error in
> (http://www.mudskipper.ca/cython-doc/docs/sharing_declarations.html):
> "cdef float cube(float)" should go before "spammery.pyx"
> 
> 6) The Sharing extension types example is confusing because the file
> names are mixed into the code.  They should be split (and in different
> colors as in the "Using cimport to resolve naming conflicts" section).
> 
> 7) The highlighting on "The cdef extern from clause does three
> things:" is wrong
> (http://www.mudskipper.ca/cython-doc/docs/external_C_code.html)
> 
> 8) There's a table missing in the "Styles of struct, union and enum
> declaration" section
> (http://www.mudskipper.ca/cython-doc/docs/external_C_code.html)
> 
> 9) In "Declaring a function as callable without the GIL," I think
> there should be a with before the nogil in the final example.  The
> last sentence may have a typo as well.  I could be wrong though: I
> don't actually know the syntax you're explaining.
> 
> 10) Capitalization problem
> (http://www.mudskipper.ca/cython-doc/docs/pyrex_differences.html)
> Cdef bint b = x
> 
> David
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

From wstein at gmail.com  Sun May 11 04:22:12 2008
From: wstein at gmail.com (William Stein)
Date: Sat, 10 May 2008 19:22:12 -0700
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
Message-ID: <85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>

On Sat, May 10, 2008 at 6:37 PM, Arc Riley <arcriley at gmail.com> wrote:
> Take it as FUD if you want.
>
> The truth of the matter is we have our timeline, and we need a
> language that is going to work for us, and neither Pyrex nor Cython
> fit the bill.  In our discussions tracking Cython's continuing
> changes, given the amount of work it'd take us to modify Cython to our
> needs and maintain a variant given radical changes being discussed, is
> completely unfeasible.
>
> You may want to consider, however, that while you have very good
> reasons for making choices, and I am not at all saying you should not
> be making those exact choices, those same choices don't work for
> everyone.
>

1. You wrote: "Cython was founded with the primary goal of remaining
compatible with Pyrex."

I started Cython, so I can say with certainty that the above
was absolutely not the primary goal for starting Cython.
The goals for Cython are listed on the cython website and
they are: "(1) good test suite, (2) easyinstall support,
(3) good documentation, (4) make cython part of Python
(like ctypes), (5) compile most python code, (6) mitigate
or eliminate the need for users to invoke the Python/C API
directly without sacrificing performance."

2. You write "Cython guys are obviously upset, because Greg didn't
consult -anyone- for even opinions before doing so, a behavior many of
us may recognize the author of Soya demonstrating frequently, and they
are realizing that their goal of remaining Pyrex-compatible puts them
at Greg's
mercy as far as language changes."

This is total FUD.   The Cython developers (1) in no way have that
goal, and (2) are certainly not "at Greg's mercy".

And by the way, if anything Greg has in my opinion
been tremendously supportive of Cython lately, having
joined the Cython mailing list, answering and asking
many questions, etc.  Thanks Greg!

3. "build the main mill [cython/pyrex fork] package
from scratch, using the wisdom of
the Cython guys (reverse engineering in some cases), even the people
working on Cython agree it needs to be rewritten since the barrier to
entry for new developers is obscenely high".

One person who is new to Cython made a claim on cython-devel
about the barrier to entry.  That is much different  than "the
people working on Cython
agreeing the barrier to entry for new developers is obscenely high".
There certainly is a barrier to entry -- Cython is a compiler of
a highly nontrivial language after all.  But I think describing
it as obscenely high due to the particular implementation is FUD.
And it is exactly the kind of FUD that will hurt the Cython project
right now.

Arc, I have nothing against you starting whatever C-extension
generating project you want, etc.  I hope the above helps clarify
some misunderstandings you might have about Cython/Pyrex.

 -- William

From arcriley at gmail.com  Sun May 11 04:51:06 2008
From: arcriley at gmail.com (Arc Riley)
Date: Sat, 10 May 2008 22:51:06 -0400
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
Message-ID: <bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>

> 1. You wrote: "Cython was founded with the primary goal of remaining
> compatible with Pyrex."

Primary may be a misnomer, but it is listed in your faq.  If you are
changing your mind on this, it may be something to edit.  IMHO, it
would be a good move at this juncture.


> This is total FUD.   The Cython developers (1) in no way have that
> goal, and (2) are certainly not "at Greg's mercy".

FUD implies that I'm intentionally trying to spread Fear, Uncertainty,
and Doubt.

And the latter is true if you wish to remain Pyrex compatible.  It's
true whenever you provide compatibility with something, whenever Greg
decides to change the language you'll need to support his changes.


> 3. "build the main mill [cython/pyrex fork] package
> from scratch

your added [] is inconsistent with that sentence - mill cannot be a
fork if it is being written from scratch.

> One person who is new to Cython made a claim on cython-devel
> about the barrier to entry.  That is much different  than "the
> people working on Cython
> agreeing the barrier to entry for new developers is obscenely high".

You assume your mailing list is the only place I've gotten that
impression from.  We've had several discussions on IRC about the state
of the codebase.  You also missed part of that statement, "agree" is
not meaning "Cython devs all agree" but "[some] Cython devs  agree
[with the assessment we have made]"

You're digging into our mailing list archives, and that's fine, but
you have to remember when reading these that the people reading them
are having discussions both on and off the mailing list.  You can feel
free to post a refute in reply to our mailing list, since I believe
I'm the only one on this one, but it really matters not to either
Cython or us.

It's not like people reading my blog posting on some rss feed
syndicate is going to read our list archives.


> There certainly is a barrier to entry -- Cython is a compiler of
> a highly nontrivial language after all.  But I think describing
> it as obscenely high due to the particular implementation is FUD.
> And it is exactly the kind of FUD that will hurt the Cython project
> right now.

You seem to like the word FUD, however, you are misusing it.

FUD is something that sleezy marketing types conjure up intentionally
to mislead the public into not buying a product.  If you look at the
blog posting itself from a neutral POV you'll see I included nothing
which would lead someone to choose our solution (which is currently
vaporware, as we're just starting it, and we have no intention at
marketing to the larger community) over Cython or otherwise twist
perceptions.

When I post to my blog, knowing how many places it's syndicated, I'm
usually careful in the words I use.  The mailing list is an internal
development list with maybe 25 people on it, we're a lot more casual
there and venting is common.  If there is false information in the
blog posting, something specific, let me know and I'll fix it.  Any
fix made will get to everywhere it's posted.

From wstein at gmail.com  Sun May 11 05:14:26 2008
From: wstein at gmail.com (William Stein)
Date: Sat, 10 May 2008 20:14:26 -0700
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>
Message-ID: <85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>

On Sat, May 10, 2008 at 7:51 PM, Arc Riley <arcriley at gmail.com> wrote:
>> 1. You wrote: "Cython was founded with the primary goal of remaining
>> compatible with Pyrex."
>
> Primary may be a misnomer, but it is listed in your faq.  If you are
> changing your mind on this, it may be something to edit.  IMHO, it
> would be a good move at this juncture.

I just read our FAQ and it does not say that.  The closest I got was
"It usually follows in compatibility once Pyrex implements a comparable
feature, though."

>> This is total FUD.   The Cython developers (1) in no way have that
>> goal, and (2) are certainly not "at Greg's mercy".
>
> FUD implies that I'm intentionally trying to spread Fear, Uncertainty,
> and Doubt.
>
> And the latter is true if you wish to remain Pyrex compatible.  It's
> true whenever you provide compatibility with something, whenever Greg
> decides to change the language you'll need to support his changes.

We definitely have made no commitment to do that, haven't said
we will, and historically haven't done that in a rigid manner.  That
said Robert and Stefan have put a lot of effort into merging most of
the great work Greg does on Pyrex into Cython.

>> 3. "build the main mill [cython/pyrex fork] package
>> from scratch
>
> your added [] is inconsistent with that sentence - mill cannot be a
> fork if it is being written from scratch.

I wrote "fork" because that's exactly how you describe
PyMill on the Pymill trac server page: http://pysoy.org:8000/
That page disappeared so I can't paste from it.

> You seem to like the word FUD, however, you are misusing it.
>
> FUD is something that sleezy marketing types conjure up intentionally
> to mislead the public into not buying a product.  If you look at the

It's FUD about the Cython and Pyrex projects.
Anyway, I have the impression you thrive on attention
so I'm not going to bother with this further.

 -- William

-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From arcriley at gmail.com  Sun May 11 05:54:52 2008
From: arcriley at gmail.com (Arc Riley)
Date: Sat, 10 May 2008 23:54:52 -0400
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>
	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>
Message-ID: <bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>

> I just read our FAQ and it does not say that.

"The intention is to make it a drop-in replacement for existing Pyrex code."

If this is not the meaning you intended, then you may want to change
the way the FAQ reads.


> I wrote "fork" because that's exactly how you describe
> PyMill on the Pymill trac server page: http://pysoy.org:8000/
> That page disappeared so I can't paste from it.

That "page" is a development tracd that is not open to the public and
nothing on it is official in any way.  The numerous broken images on
it should have been a clue, if the port wasn't enough.  Your reposting
it here was the reason it was moved to another port.

The reason we're not using the name "PyMill" in any public posting is
that may not even be the name we end up with.  There's a good chance
it isn't.  We're nowhere near the point of publishing a definition of
it.  You'll notice my blog entry didn't use the name.

In fact your entire panic mode seems to be derived from a warped sense
of importance of a posting to our internal development list a link you
followed from it to a draft site we're working from.  Nothing you
pasted, besides a link to the blog entry, was read by the general
Python community until you posted it here.

Again - if there's something in the blog posting that is inaccurate,
specify it, and I'll correct it.  That is the only referenced place
where FUD (sic) could have been spread, since everything else was
internal.

From wstein at gmail.com  Sun May 11 06:06:07 2008
From: wstein at gmail.com (William Stein)
Date: Sat, 10 May 2008 21:06:07 -0700
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>
	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>
	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
Message-ID: <85e81ba30805102106u3ab17ddlc95b1b70dba2a0e9@mail.gmail.com>

On Sat, May 10, 2008 at 8:54 PM, Arc Riley <arcriley at gmail.com> wrote:
>> I just read our FAQ and it does not say that.
>
> "The intention is to make it a drop-in replacement for existing Pyrex code."
>
> If this is not the meaning you intended, then you may want to change
> the way the FAQ reads.
>

You're right.  I've changed that sentence to read "The intention is to
make it for the most part a drop-in replacement for existing Pyrex code,
though some changes to that existing code may have to be made."
Thanks for your other clarifications as well.

 -- William

-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From gfurnish at gfurnish.net  Sun May 11 09:30:26 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 11 May 2008 01:30:26 -0600
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>
	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>
	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
Message-ID: <8f8f8530805110030l534317d7j763f1534a9a509c3@mail.gmail.com>

Arc, at this point all I ever hear from you is how Cython is so
horrible and has to be rewritten to be even tolerable.  What exactly
is so horrible about Cython?  You have never given any specifics on
why Cython is so horrible that it is better to start from scratch and
thus violate about every good rule of programming (see Joel on
Software).  If this isn't intended to be FUD why don't you name some
specific features that require a full rewrite as opposed to making
grand movements, which are in many cases flat out wrong, that seem
designed only to attack Cython and Pyrex?  Are you actually going to
produce code at some point?  Last I heard there was some grand plan to
fork Cython and license it under GPL3 that went predictably nowhere.
Why are you taking on an even grander project?  If your goal is to
produce a simple graphics engine why are you getting into the compiler
business?

On Sat, May 10, 2008 at 9:54 PM, Arc Riley <arcriley at gmail.com> wrote:
>> I just read our FAQ and it does not say that.
>
> "The intention is to make it a drop-in replacement for existing Pyrex code."
>
> If this is not the meaning you intended, then you may want to change
> the way the FAQ reads.
>
>
>> I wrote "fork" because that's exactly how you describe
>> PyMill on the Pymill trac server page: http://pysoy.org:8000/
>> That page disappeared so I can't paste from it.
>
> That "page" is a development tracd that is not open to the public and
> nothing on it is official in any way.  The numerous broken images on
> it should have been a clue, if the port wasn't enough.  Your reposting
> it here was the reason it was moved to another port.
>
> The reason we're not using the name "PyMill" in any public posting is
> that may not even be the name we end up with.  There's a good chance
> it isn't.  We're nowhere near the point of publishing a definition of
> it.  You'll notice my blog entry didn't use the name.
>
> In fact your entire panic mode seems to be derived from a warped sense
> of importance of a posting to our internal development list a link you
> followed from it to a draft site we're working from.  Nothing you
> pasted, besides a link to the blog entry, was read by the general
> Python community until you posted it here.
>
> Again - if there's something in the blog posting that is inaccurate,
> specify it, and I'll correct it.  That is the only referenced place
> where FUD (sic) could have been spread, since everything else was
> internal.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From languitar at semipol.de  Sun May 11 10:07:55 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Sun, 11 May 2008 10:07:55 +0200
Subject: [Cython] cimport'ing module with cdef public declarations
In-Reply-To: <48264965.2010008@canterbury.ac.nz>
References: <4825D1A2.5000900@semipol.de> <48264411.7070907@canterbury.ac.nz>
	<48264965.2010008@canterbury.ac.nz>
Message-ID: <4826A95B.40509@semipol.de>

Am 05/11/2008 03:18 AM schrieb Greg Ewing:
> But in Pyrex you wouldn't need the "public"
> declaration, as it's only for making functions
> available to external C code, not other Pyrex
> modules.

But that's what I need, too. The module contains functions that must be
available to external C code and others that are needed for Cython to work.

Johannes



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080511/071331da/attachment-0001.pgp 

From dagss at student.matnat.uio.no  Sun May 11 10:10:23 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 11 May 2008 10:10:23 +0200
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <8f8f8530805110030l534317d7j763f1534a9a509c3@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
	<8f8f8530805110030l534317d7j763f1534a9a509c3@mail.gmail.com>
Message-ID: <4826A9EF.7090305@student.matnat.uio.no>

Gary Furnish wrote:
> Arc, at this point all I ever hear from you is how Cython is so
> horrible and has to be rewritten to be even tolerable.  What exactly
> is so horrible about Cython?  You have never given any specifics on

According to the blog post, it is

"recent developments in their community"

(I wonder if it's me, and he's just being sensitive and trying to 
protect me :-) )

> why Cython is so horrible that it is better to start from scratch and
> thus violate about every good rule of programming (see Joel on
> Software).  If this isn't intended to be FUD why don't you name some

I second that (strongly!) with the exact URL:

http://www.joelonsoftware.com/articles/fog0000000069.html

Still, while the premise might seem strange to us ("Cython codebase is 
moving so quickly and they are so aware of possible problems with their 
codebase that I should rather start from scratch"), there's an important 
aspect to this rebuild: While whatever they build will be moving quickly 
as well, whatever they build will also be under their control, and only 
move when it is fitting for them. (I'm not jesting here: That can be a 
valid reason! As Arc says:

"""
You may want to consider, however, that while you have very good
reasons for making choices, and I am not at all saying you should not
be making those exact choices, those same choices don't work for
everyone.
"""

).

I can't argue with that, even if Arc's conclusion seems strange to me. I 
don't know how much developer resources he has available, after all. 
(But I think it would be great if Arc would want discuss any problems 
with the Cython release process on the Cython list; as far as I can see 
he only raised the point on the Pyrex list).

And he does raise an important point that should be discussed: Pledging 
to language stability. But I'll start another thread on that.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun May 11 10:14:32 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 11 May 2008 10:14:32 +0200
Subject: [Cython] Language stability
Message-ID: <4826AAE8.6010706@student.matnat.uio.no>

Given recent developments (Pyrex release with new syntax and Arc's new 
project) I think Cython should make some public website statements on 
language policy and stability. I think these are already present in the 
"community spirit", but we should make them very explicit.

Rough suggestions aimed for starting this thread:

- "Cython want to be a dynamic work in progress and so new language 
features will happen quickly over the next year. *However*, each change 
will be related to previously unavailable features and will have a 
backwards-compatible syntax."

- If we need to break syntax backwards-incompatibly, it will be done by 
the scheme discussed earlier: A "#lang: cython-ver" comment header. 
Files without such a header will be compiled with the current ("v0.9") 
syntax.

- Some decision should be made on Pyrex compatability. I think it should 
either be flat out "the language looks like Pyrex because of its roots 
but is *not* Pyrex", or, we should aim to correctly compile Pyrex 
(including future changes) but do so only when "#lang: pyrex" is specified.

(Of course, any "lang" instruction would be command-line-overridable so 
that existing Pyrex code could be built with a "--pyrex-mode" flag)

- We need a consistent keyword policy. I'm going to be controversial and 
suggest that this will be "figure it out from context", because we aim 
to compile any Python code. I.e.:

  * Any Python keywords are also Cython keywords (which exact keywords 
depends on the Python language level we are compiling)
  * Any non-Python keywords are only keywords when used in a context 
where they can mean a keyword instruction. That is, we should start 
supporting Cython code like this:

cdef = 4
# Created cdef variable, does not alter behaviour of cdef keyword

We already support

final = 4

anyway... I think any new Cython keywords (over Python) are always used 
in contexts where they could not be a variable name so I think this is 
possible.

This is unusual, but because our aim is to go towards full Python 
support, but still support Cython code, I think such a policy is a 
possible compromise (the important thing is to have a solid, dependable 
rule we follow).

Another possibility is to only allow

cdef = 4

when we are in (also on the idea-stage, from the #lang-thread) Python 
mode (compiling a .py-file or having a "#lang: python" header).


-- 
Dag Sverre

From robertwb at math.washington.edu  Sun May 11 10:23:12 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 01:23:12 -0700
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <48263800.8070606@canterbury.ac.nz>
References: <48243C21.2020207@canterbury.ac.nz> <48246C10.6050703@behnel.de>
	<48250902.2020804@canterbury.ac.nz> <48253372.9060802@behnel.de>
	<48263800.8070606@canterbury.ac.nz>
Message-ID: <E14CD7C8-C4A7-4E8C-8232-7FB96E34DFD9@math.washington.edu>

On May 10, 2008, at 5:04 PM, Greg Ewing wrote:

> Stefan Behnel wrote:
>> Wouldn't special casing a type test for builtins and extension  
>> classes be
>> better here than adding new functions?
>
> I'm not sure what you mean by that. There's no run-time
> distinction between a builtin class, an extension class
> implemented with Pyrex, or an extension class created some
> other way. They're all just types.
>
> I could make the Pyrex version of isinstance() behave
> differently, but then it wouldn't have quite  the same
> semantics as it does in Python. That could be confusing,
> especially since you aren't guaranteed to be always
> getting the directly-called version.
>
> Rather than fiddle with the semantics, I felt it was
> better to provide different functions -- EIBTI and
> all that.
>
> They're not really new functions, anyway -- they're
> just exposing functions that are in the Python API,
> and they're the ones you would use if you were writing
> the extension module in C.
>
>> I mean, we already had that with getattr3(), where the right
>> thing to do would have been to fix builtin functions.
>
> That was a matter of pragmatism. I don't currently have
> any mechanism for dealing with a C function that can have
> more than one signature. I would have had to build a very
> special case into the guts of the compiler somewhere,
> which I didn't feel was worth doing just for that one
> function. So I did the next best thing and provided
> something that works, even if it's not ideal. I can
> always come back and do something else later.
>
> I think we're coming up against those philosophical
> differences again. Pyrex code is not Python code, and
> doesn't pretend to be. I try to make things compatible
> where reasonably possible, but it's not an overriding
> principle.
>
>> So if you added doctests
>> to your test modules (as we do for ours), you could still validate  
>> the
>> resulting source code for Pyrex, but we could both join forces and  
>> benefit
>> from a growing test suite.
>
> Okay, so you just want me to run the Cython tests as
> well. That shouldn't be too hard.

Yes. When we add new features, we try and write tests for them.

> Is there somewhere I can download them from?

You can get them all here: http://hg.cython.org/cython/file/ 
0927890724ab/tests/

- Robert

From greg.ewing at canterbury.ac.nz  Sun May 11 10:18:35 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 20:18:35 +1200
Subject: [Cython] cimport'ing module with cdef public declarations
In-Reply-To: <4826A95B.40509@semipol.de>
References: <4825D1A2.5000900@semipol.de> <48264411.7070907@canterbury.ac.nz>
	<48264965.2010008@canterbury.ac.nz> <4826A95B.40509@semipol.de>
Message-ID: <4826ABDB.9080906@canterbury.ac.nz>

Johannes Wienke wrote:

> But that's what I need, too. The module contains functions that must be
> available to external C code and others that are needed for Cython to work.

That's fine, then. I don't think it's the cause of
your problem -- it shouldn't do any harm, whether
it's needed or not.

Someone who knows more about the way function
cimporting is supposed to work in Cython will be
needed to sort this out.

-- 
Greg

From gfurnish at gfurnish.net  Sun May 11 10:26:47 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 11 May 2008 02:26:47 -0600
Subject: [Cython] Some small phase refactorings
In-Reply-To: <48233D62.7000200@student.matnat.uio.no>
References: <481AF97B.8030305@student.matnat.uio.no>
	<481BFDDA.8060007@behnel.de>
	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>
	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
	<48215041.7070805@student.matnat.uio.no>
	<5303982D-51B8-4D6F-B363-BDC8C9729892@math.washington.edu>
	<48233D62.7000200@student.matnat.uio.no>
Message-ID: <8f8f8530805110126v697012a2pd5770fa877b89dbd@mail.gmail.com>

Efficiency does matter to Sage though, and an O(1) overhead is very
far from negligible.  Cython uses real C vtables, so there are
absolutely no dict-based lookups involved with cdef class
polymorphism.  -Infinity to any proposal that makes code slower.

On Thu, May 8, 2008 at 11:50 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>
>> any replies than I guess that's an indication. I am actually a bit
>> worried about how the use of decorators will impact the ability of
>> Cython to compile itself into ultra-efficient C.
>>
> Efficiency doesn't bother me at all, and I have reasons! What the
> decorators would do (does, in prototype code) is stuff the class name
> and function into a dictionary at *class construction time* (or, without
> metaclass support, at object construction time), i.e. a O(1) overhead
> that is negligible even when running in the Python interpreter. During
> tree traversal, it's just a dict lookup on class id and a dispatch to
> the resulting function.
>
> If one were generating C++ objects then the classical visitor pattern
> would be faster because real vtables are faster than dict lookups,
> however Cython polymorphism is (if I'm not wrong, haven't looked at this
> closely) dict-based anyway so it shouldn't make a difference.
>
> Here's another way to use 2.3 that might be acceptable:
>
> class WithStatementHandler(VisitorTransform):
>    def handle_with(self, node): ...
>
>    matches = [
>        class_match(WithStatementNode, handle_with)
>    ]
>
>
> It would end up as a dict of class -> function like the other approach
> (but written like a list of objects because I'd like to leave a way open
> up for other types of matches though).
>
>> It just didn't seem to add anything except another level of
>> indirection, but that will of course eventually be needed. There is a
>> lot of re-factoring that will need to be done. I've just got other
>> obligations (general exam) to be able to put much time/though into
>> this for the next two weeks or so :-(.
>>
> What it added was the capability to plug in new transforms after type
> analysis, but before generation, on a module-wide basis, rather than
> having the transform run once for each function.
>
> I didn't make that very clear; I suppose because you're right, it's real
> value is in another layer of indirection -- but one that will be needed
> and is useful during refactoring, and can allow "in-production"
> refactoring, rather than applying it all at once.
>
> " The only problem that cannot be solved by another layer of indirection
> is too many layers of indirection " :-)
>
> --
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Sun May 11 10:28:02 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 01:28:02 -0700
Subject: [Cython] [Pyrex] ANN: Pyrex 0.9.7
In-Reply-To: <e7ba66e40805101503v41be962eheb0d32fdcff8d46d@mail.gmail.com>
References: <48243C21.2020207@canterbury.ac.nz>
	<205284A8-3452-4FEE-BDCD-D5DECAD61CE9@bryant.edu>
	<48259663.3070009@behnel.de>
	<e7ba66e40805101503v41be962eheb0d32fdcff8d46d@mail.gmail.com>
Message-ID: <4FBCBADF-F795-455F-9BA4-B4EF45FF212A@math.washington.edu>

On May 10, 2008, at 3:03 PM, Lisandro Dalcin wrote:
> On 5/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>  I don't quite see why we shouldn't just always convert
>>
>>     for var in range(begin, end):
>>
>>  to
>>
>>     for var in begin <= var < end:
>>
>>
>> *iff* var is cdef-ed as a C integer type.
>
> I'm definitely +1 on this!!

This is actually enabled in the current version of Cython: http:// 
hg.cython.org/cython/file/0927890724ab/Cython/Compiler/Options.py

> Regarding to the overflow issue, perhaps
> a check can be added before entering the loop.

This is the semantic difference--if there will be an overflow then  
Cython throws an error before entering the loop (rather than looping  
until an overflow would occur). I think this is acceptable.

> And Cython should take
> care if the user modifies the range() arguments inside de loop. What
> will you do in this case? IMHO,  matching the Python semantics is the
> right way.

One can't do that (in Python or Cython) as range() is evaluated  
exactly once, before entering the loop, so this is a non-issue.

- Robert


From gfurnish at gfurnish.net  Sun May 11 10:29:51 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sun, 11 May 2008 02:29:51 -0600
Subject: [Cython] Some small phase refactorings
In-Reply-To: <8f8f8530805110126v697012a2pd5770fa877b89dbd@mail.gmail.com>
References: <481AF97B.8030305@student.matnat.uio.no>
	<481BFDDA.8060007@behnel.de>
	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>
	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>
	<48215041.7070805@student.matnat.uio.no>
	<5303982D-51B8-4D6F-B363-BDC8C9729892@math.washington.edu>
	<48233D62.7000200@student.matnat.uio.no>
	<8f8f8530805110126v697012a2pd5770fa877b89dbd@mail.gmail.com>
Message-ID: <8f8f8530805110129w395c93a9r535f36615fbf365@mail.gmail.com>

Well, clarifying what I meant a bit more, the *biggest* speed loss
anywhere is dictionary lookups.  If your going to use dictionary
lookups at object time, you lose most of the advantage of using cython
to compile cython.

On Sun, May 11, 2008 at 2:26 AM, Gary Furnish <gfurnish at gfurnish.net> wrote:
> Efficiency does matter to Sage though, and an O(1) overhead is very
> far from negligible.  Cython uses real C vtables, so there are
> absolutely no dict-based lookups involved with cdef class
> polymorphism.  -Infinity to any proposal that makes code slower.
>
> On Thu, May 8, 2008 at 11:50 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>>
>>> any replies than I guess that's an indication. I am actually a bit
>>> worried about how the use of decorators will impact the ability of
>>> Cython to compile itself into ultra-efficient C.
>>>
>> Efficiency doesn't bother me at all, and I have reasons! What the
>> decorators would do (does, in prototype code) is stuff the class name
>> and function into a dictionary at *class construction time* (or, without
>> metaclass support, at object construction time), i.e. a O(1) overhead
>> that is negligible even when running in the Python interpreter. During
>> tree traversal, it's just a dict lookup on class id and a dispatch to
>> the resulting function.
>>
>> If one were generating C++ objects then the classical visitor pattern
>> would be faster because real vtables are faster than dict lookups,
>> however Cython polymorphism is (if I'm not wrong, haven't looked at this
>> closely) dict-based anyway so it shouldn't make a difference.
>>
>> Here's another way to use 2.3 that might be acceptable:
>>
>> class WithStatementHandler(VisitorTransform):
>>    def handle_with(self, node): ...
>>
>>    matches = [
>>        class_match(WithStatementNode, handle_with)
>>    ]
>>
>>
>> It would end up as a dict of class -> function like the other approach
>> (but written like a list of objects because I'd like to leave a way open
>> up for other types of matches though).
>>
>>> It just didn't seem to add anything except another level of
>>> indirection, but that will of course eventually be needed. There is a
>>> lot of re-factoring that will need to be done. I've just got other
>>> obligations (general exam) to be able to put much time/though into
>>> this for the next two weeks or so :-(.
>>>
>> What it added was the capability to plug in new transforms after type
>> analysis, but before generation, on a module-wide basis, rather than
>> having the transform run once for each function.
>>
>> I didn't make that very clear; I suppose because you're right, it's real
>> value is in another layer of indirection -- but one that will be needed
>> and is useful during refactoring, and can allow "in-production"
>> refactoring, rather than applying it all at once.
>>
>> " The only problem that cannot be solved by another layer of indirection
>> is too many layers of indirection " :-)
>>
>> --
>> Dag Sverre
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>

From stefan_ml at behnel.de  Sun May 11 10:34:19 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 May 2008 10:34:19 +0200
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <48263800.8070606@canterbury.ac.nz>
References: <48243C21.2020207@canterbury.ac.nz> <48246C10.6050703@behnel.de>
	<48250902.2020804@canterbury.ac.nz> <48253372.9060802@behnel.de>
	<48263800.8070606@canterbury.ac.nz>
Message-ID: <4826AF8B.9010208@behnel.de>

Hi Greg,

Greg Ewing wrote:
> Stefan Behnel wrote:
>> Wouldn't special casing a type test for builtins and extension classes be
>> better here than adding new functions?
> 
> I'm not sure what you mean by that. There's no run-time
> distinction between a builtin class, an extension class
> implemented with Pyrex, or an extension class created some
> other way. They're all just types.

I didn't mean at runtime. The compiler knows how to distinguish them.


> I could make the Pyrex version of isinstance() behave
> differently, but then it wouldn't have quite  the same
> semantics as it does in Python. That could be confusing,
> especially since you aren't guaranteed to be always
> getting the directly-called version.

I could live with different semantics even in this case:

    allowed_type = ExtType
    print isinstance(obj, allowed_type) # Python semantics

and

    print isinstance(obj, ExtType) # non-Python semantics

The Pyrex/Cython semanics would be:

- testing for an explicitly named extension type or builtin type would give
you a direct (sub-)type check (maybe even for tuples of subtypes? It would be
nice to optimise that as well)

- testing for a runtime type will fall back to Python semantics.

I think those are the perfect semantics for the context of Pyrex/Cython.

Besides, the actual difference of allowing or disallowing a type override
through ".__class__" is so tiny that almost no user will actually notice, and
those who need that distinction can well be asked to use a somewhat less
straight forward syntax. That's much better than forcing users to use a
non-standard function for the much more likely case that they do the type test
to make sure you can rely on a specific struct layout. One of the major use
cases of Pyrex/Cython is wrapping external libraries with Python classes,
after all.


> Rather than fiddle with the semantics, I felt it was
> better to provide different functions -- EIBTI and
> all that.

I would be fine with that if the new function was for the less common case.
But I would say it's for the most common case, which would better be handled
by the normal function that is named as in Python.


>> So if you added doctests
>> to your test modules (as we do for ours), you could still validate the
>> resulting source code for Pyrex, but we could both join forces and
>> benefit
>> from a growing test suite.
> 
> Okay, so you just want me to run the Cython tests as
> well. That shouldn't be too hard.

Sort of. I just want to avoid doubling the work on both sides. After all, I
prefer investing more time into the compiler than into the test suite.


> Is there somewhere I can download them from?

Yes, as Robert just pointed out. Note that most of them are actually based on
your own test suite, but most of the tests (those in the tests/run/ directory)
have an added doctest that is executed by the test runner script after
building the module.

You will also notice that the tests/errors/ directory has modules that produce
compiler errors, and the expected error messages are part of the modules
themselves.

When I wrote the test runner, my main goal was to keep the expected results in
the same file as the test code itself, so that you only have one file to edit
and look at (ok, minus .pxd and .h files). Doctests are just perfect for this.

Stefan

From robertwb at math.washington.edu  Sun May 11 10:38:30 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 01:38:30 -0700
Subject: [Cython] String interning and Python 3
In-Reply-To: <e7ba66e40805101406n468ef323i4aa49b7b39c8cb62@mail.gmail.com>
References: <4825AFFB.6040305@behnel.de>
	<e7ba66e40805101406n468ef323i4aa49b7b39c8cb62@mail.gmail.com>
Message-ID: <31D60ED7-91BE-435B-A04B-AF14E733FFAB@math.washington.edu>

On May 10, 2008, at 2:06 PM, Lisandro Dalcin wrote:
> Stefan , If  (4) is doable with no much effort, then I believe this is
> the right way.
>
> As you said, one of the points of string interning is saving memory.
> But there is also another very important benefit. Look at this for
> Python 2.6 sources in stringobject.c
>
> int
> _PyString_Eq(PyObject *o1, PyObject *o2)
> {
> 	PyStringObject *a = (PyStringObject*) o1;
> 	PyStringObject *b = (PyStringObject*) o2;
>         return Py_SIZE(a) == Py_SIZE(b)
>           && *a->ob_sval == *b->ob_sval
>           && memcmp(a->ob_sval, b->ob_sval, Py_SIZE(a)) == 0;
> }
>
>
> As you can see the line with '*a->ob_sval == *b->ob_sval' provides a
> fast path for string equality. So if o1 and o2 are the same (interned)
> strings then with a pointer comparison you avoid at all the memcmp
> call. And this is very, very important in dictionary lookups to make
> that operation faster in the case the keys are strings.

I want to second this, we want to keep interned strings if at all  
possible for the above reason. Making everything a dictionary lookup  
is one of the ways Python is so dynamic, but it means that the speed  
of dictionary lookups is extremely important (in fact I would say  
this is one of the primary bottlenecks of Python). It also offers the  
advantage that the lookup strings don't need to be re-allocated each  
time they're needed.

> On 5/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>>  I'm wondering how to continue the support for this feature given  
>> the fact that
>>  identifiers are Unicode strings in Py3. We currently only intern  
>> byte strings
>>  that look like Python identifiers, so in Py3, they simply no  
>> longer look like
>>  identifiers, as they are not Unicode strings.
>>
>>  I can see four ways how to deal with this:
>>
>>  1) drop string interning completely
>>
>>  2) disable string interning in Py3 and use normally created byte  
>> strings instead
>>
>>  3) keep separate sets of identifier-like byte strings and unicode  
>> strings in
>>  the compiler and write them into the C file. Then, depending on  
>> the Python
>>  version, either intern the byte strings or the unicode strings,  
>> and create the
>>  other set as un-interned strings.
>>
>>  4) keep the information if a string should be interned for all  
>> strings we deal
>>  with (bytes and unicode), remove the intern tab and merge it with  
>> the general
>>  string tab by adding an additional field "intern". Then  
>> __Pyx_InitStrings()
>>  would create the strings differently depending on the compile  
>> time Python
>>  version, i.e., it would intern Unicode identifiers in Py3 and  
>> byte string
>>  identifiers in Py2, and create everything else as normal strings.
>>
>>  Personally, I favour 4) - although I could live with 1) - but  
>> since I'm not
>>  quite sure what the original intention of string interning was  
>> (saving
>>  memory?), I'd like to hear other opinions first.
>>
>>  Stefan
>>  _______________________________________________
>>  Cython-dev mailing list
>>  Cython-dev at codespeak.net
>>  http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Sun May 11 10:55:42 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 11 May 2008 10:55:42 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
Message-ID: <4826B48E.3060708@student.matnat.uio.no>

> ---------a.pxd----------
> cdef class A:
>      cdef int len
>      cdef int* data
> 
>      cdef inline [final?] int __getitem__(A a, int i):
>          """
>          Note that subtypes can't override this.
>          """
>          if i < 0 or i > a.len:
>              raise IndexError
>          return data[i]
> 
> ---------b.pyx--------
> 
> from a cimport A
> cdef A(len=10) a = a([1,2,3,4,5,6,7,8,9,10]) # I'll leave the init  
> function to your imagination
> print a[9] # the code from __getitem__ gets inlined here, and since  
> len is known the a.len is resolved to 10 at compile time.
> 
> (Here a.len tries to do a lookup first on the compile time type of a,  
> and that failing the runtime type of a. The compile time types need  
> not be struct members, but if they're not then they must be specified  
> because the "runtime" lookup would fail.)

On the type arguments solution:

My first thought is that I don't think it will cover all cases -- if
native C++ template support is added, for instance, then the type
arguments must probably behave differently there. So type arguments in 
itself is a more generic thing (related to the parser and type handling 
etc.), and this specifies one concrete use of them (ie, what happens to 
"cdef class"es when given type parameters).

At first it struck me as way too magic. Then it grew on me. But then I 
dislike it again :-) So these are some non-conclusive thoughts:

1) A "disadvantage" is that it looks like one has to break down the type
specification vs. run-time parsing context seperation that we've talked
about earlier? -- how would you specify that a type parameter "T" should
take "unsigned short int*"? So to have consistency in any sane way one 
needs to make "all compile-time types also available run-time types"

2) It seems to leave the way open for some confusing (if not impossible 
to solve compiler-wise) results:

cdef A(len=10) a = ...
cdef A(len=8) b = a # What does this mean? Compile-time error?

Note, for instance, that while assigning a ndarray(2, int8) to a 
ndarray(3, int8) or ndarray(2, float32); it should (or rather, might be 
wanted behaviour) for it to be legal to assign ndarray(2, uint8, 
flat=True) to ndarray(2, uint8, flat=False); where flat is flag to 
toggle multi-dimensional indexing.

Also consider:

cdef A(len=10) a = ...
a.len = 8 # legal or not?
cdef A(len=8) b = a # Is this legal now?

Such things would have to be described in more detail, and since it is 
hard to "guess" what the semantics should be (at least for me) it might 
be an indication that this is too magic.


-- 
Dag Sverre

From stefan_ml at behnel.de  Sun May 11 10:53:39 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 May 2008 10:53:39 +0200
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>
	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
Message-ID: <4826B413.4080608@behnel.de>

Hi,

Arc Riley wrote:
>> I just read our FAQ and it does not say that.
> 
> "The intention is to make it a drop-in replacement for existing Pyrex code."

Right, "existing" Pyrex code. We are not necessarily talking about newly
developed code that uses new Pyrex features, neither are we talking about code
that uses Cython features that are not supported by Pyrex. Cython is not
Pyrex, otherwise we wouldn't have two code bases.

That aside, language compatibility between Cython and Pyrex *is* a goal. It
would be stupid to have two compilers that understand almost the same
language, and allow them to diverge in subtleties that make it impossible for
users to change between the two if they need to.

The latest discussion with Greg on the integer-loop syntax should make it
clear that Cython and Pyrex are not fundamentally diverging projects. And I
really don't think they should be.

Stefan

From stefan_ml at behnel.de  Sun May 11 10:58:57 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 May 2008 10:58:57 +0200
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
Message-ID: <4826B551.2010004@behnel.de>

Hi,

William Stein wrote:
> On Sat, May 10, 2008 at 6:37 PM, Arc Riley <arcriley at gmail.com> wrote:
> 3. "build the main mill [cython/pyrex fork] package from scratch, using the
> wisdom of the Cython guys (reverse engineering in some cases), even the
> people working on Cython agree it needs to be rewritten since the barrier
> to entry for new developers is obscenely high".
> 
> One person who is new to Cython made a claim on cython-devel about the
> barrier to entry.  That is much different  than "the people working on
> Cython agreeing the barrier to entry for new developers is obscenely high".
> There certainly is a barrier to entry -- Cython is a compiler of a highly
> nontrivial language after all.  But I think describing it as obscenely high
> due to the particular implementation is FUD. And it is exactly the kind of
> FUD that will hurt the Cython project right now.

This sounds like Arc doesn't know that code is actually harder to read than to
write.

http://www.joelonsoftware.com/articles/fog0000000069.html

Stefan


From greg.ewing at canterbury.ac.nz  Sun May 11 11:26:07 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 21:26:07 +1200
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <4826AF8B.9010208@behnel.de>
References: <48243C21.2020207@canterbury.ac.nz> <48246C10.6050703@behnel.de>
	<48250902.2020804@canterbury.ac.nz> <48253372.9060802@behnel.de>
	<48263800.8070606@canterbury.ac.nz> <4826AF8B.9010208@behnel.de>
Message-ID: <4826BBAE.4060603@canterbury.ac.nz>

Stefan Behnel wrote:

> Greg Ewing wrote:
> 
>>There's no run-time
>>distinction between a builtin class, an extension class
>>implemented with Pyrex, or an extension class created some
>>other way.
> 
> I didn't mean at runtime. The compiler knows how to distinguish them.

Well, there's no compile-time distinction either, really.
They're all the same thing. Treating a type differently
depending on whether it was implemented using a Pyrex
extension class would be bizarre and confusing.

>     allowed_type = ExtType
>     print isinstance(obj, allowed_type) # Python semantics

That's not the only possibility. There's also

   f = isinstance
   if f(ob, MyExtensionClass):
     ...

You're testing against an explicitly-named class -- but
using the Python version of isinstance rather than the
Pyrex one.

> Besides, the actual difference of allowing or disallowing a type override
> through ".__class__" is so tiny that almost no user will actually notice,

Possibly you're right. However, I do have a guideline
of my own concerning Python compatibility: if a function
has the same name as a Python function, then you can
rely on it to have the same semantics.

There's another consideration as well: What do you do
if you really *do* want exactly the same semantics as
the Python isinstance()? There would have to be another
function for that, which *is* exactly the same as
isinstance(), but has a different name.

That situation could be considered somewhat farcical.

> Note that most of them are actually based on
> your own test suite, but most of the tests (those in the tests/run/ directory)
> have an added doctest that is executed by the test runner script after
> building the module.

Hmmm. I don't really want to include a lot of tests
that are more or less duplicates of the existing ones.

If you have any new, Cython-specific tests that you
think ought to work in Pyrex as well, I'd be happy to
run those.

But I don't see how much will be gained by just including
all the Cython tests as-is. I'm not going to be changing
my own test setup, so you're not going to be able to share
any new tests that I add without converting them to your
format. And new tests for Cython-specific features probably
won't apply to Pyrex anyway.

If you find a bug and come up with a new test to guard
it against regression, I think I would actually rather
convert it to my test format and make it one of my tests.
So I would be happy for you to bring any such tests to
my attention.

> When I wrote the test runner, my main goal was to keep the expected results in
> the same file as the test code itself, so that you only have one file to edit
> and look at

There is one advantage to keeping them separate -- after
I've fixed everything so that the expected and actual results
are near enough to being in sync, I can run a script that
automatically updates the expected results from the last
actual results. That would be hard to do if the expected
results were embedded in the test files.

This is probably more important for the compilation tests,
where the "expected result" is large and can change in ways
that are ignorable. But having built the framework to cope
with that, it was easiest to just use it for everything.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun May 11 11:34:39 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 21:34:39 +1200
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <4826B413.4080608@behnel.de>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>
	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>
	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
	<4826B413.4080608@behnel.de>
Message-ID: <4826BDAF.9050106@canterbury.ac.nz>

Stefan Behnel wrote:

> The latest discussion with Greg on the integer-loop syntax should make it
> clear that Cython and Pyrex are not fundamentally diverging projects. And I
> really don't think they should be.

That's good to hear. However, if one of the priorities
of Cython is really going to be to compile pure Python code,
there are likely to be problems at some point, because
I'm going to want to make Pyrex do things that you won't
like, and vice versa.

BTW, the discussion we've been having about isinstance
vs. typecheck is a bit ironic in this light. I'm arguing
for *more* compatibility with Python in this area,
and you're arguing for *less*. :-)

-- 
Greg

From robertwb at math.washington.edu  Sun May 11 11:41:12 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 02:41:12 -0700
Subject: [Cython] Language stability
In-Reply-To: <4826AAE8.6010706@student.matnat.uio.no>
References: <4826AAE8.6010706@student.matnat.uio.no>
Message-ID: <91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>

On May 11, 2008, at 1:14 AM, Dag Sverre Seljebotn wrote:
> Given recent developments (Pyrex release with new syntax and Arc's new
> project) I think Cython should make some public website statements on
> language policy and stability. I think these are already present in  
> the
> "community spirit", but we should make them very explicit.

This is a very good idea.

> Rough suggestions aimed for starting this thread:
>
> - "Cython want to be a dynamic work in progress and so new language
> features will happen quickly over the next year. *However*, each  
> change
> will be related to previously unavailable features and will have a
> backwards-compatible syntax."

I think there is a place for backwards-incompatibility. For example,  
the __new__() being renamed to __cinit__() (as __new__() a very  
different meaning in Python). However, if there is ever backwards- 
compatibility issues, it should be for a *very* good reason. Also, we  
might want to add syntax candy, so I'm not sure about the "previously  
unavailable features" clause.

> - If we need to break syntax backwards-incompatibly, it will be  
> done by
> the scheme discussed earlier: A "#lang: cython-ver" comment header.
> Files without such a header will be compiled with the current ("v0.9")
> syntax.
>
> - Some decision should be made on Pyrex compatability. I think it  
> should
> either be flat out "the language looks like Pyrex because of its roots
> but is *not* Pyrex", or, we should aim to correctly compile Pyrex
> (including future changes) but do so only when "#lang: pyrex" is  
> specified.
>
> (Of course, any "lang" instruction would be command-line- 
> overridable so
> that existing Pyrex code could be built with a "--pyrex-mode" flag)

I don't think supporting multiple versions of the language, and  
having a specific Pyrex flag, is a good idea. If one wants a specific  
version of Cython, one can download and use that one. If one wants  
*exatly* Pyrex syntax, use Pyrex itself. Any difference should be  
clearly documented, and be for good reason, but having different  
"modes" that we continue supporting seems cumbersome.

> - We need a consistent keyword policy. I'm going to be  
> controversial and
> suggest that this will be "figure it out from context", because we aim
> to compile any Python code. I.e.:
>
>   * Any Python keywords are also Cython keywords (which exact keywords
> depends on the Python language level we are compiling)
>   * Any non-Python keywords are only keywords when used in a context
> where they can mean a keyword instruction. That is, we should start
> supporting Cython code like this:
>
> cdef = 4
> # Created cdef variable, does not alter behaviour of cdef keyword

I would much rather this be a syntax error. It also will make the  
parser a lot more complicated (and the code a lot harder to read.)

> We already support
>
> final = 4
>
> anyway...

Final is not a keyword, and I don't think it will ever be. It could  
still have a special meaning, but being a keyword is a lot stronger.

> I think any new Cython keywords (over Python) are always used
> in contexts where they could not be a variable name so I think this is
> possible.
>
> This is unusual, but because our aim is to go towards full Python
> support, but still support Cython code, I think such a policy is a
> possible compromise (the important thing is to have a solid,  
> dependable
> rule we follow).
>
> Another possibility is to only allow
>
> cdef = 4
>
> when we are in (also on the idea-stage, from the #lang-thread) Python
> mode (compiling a .py-file or having a "#lang: python" header).

Maybe... I really don't like the idea of mixing the two modes, e.g.

cdef cdef x = cdef(3)

- Robert



From greg.ewing at canterbury.ac.nz  Sun May 11 11:50:28 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 21:50:28 +1200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826B48E.3060708@student.matnat.uio.no>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
Message-ID: <4826C164.1090206@canterbury.ac.nz>

Another thing to consider is how this would
interact with subclassing. Will it be legal
to do something like

   cdef class B(A(len=10)):
     ...

and if so, how does the initialisation of B
ensure that the length constraint is satisfied?

-- 
Greg


From stefan_ml at behnel.de  Sun May 11 12:08:33 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 May 2008 12:08:33 +0200
Subject: [Cython] ANN: Pyrex 0.9.7
In-Reply-To: <4826BBAE.4060603@canterbury.ac.nz>
References: <48243C21.2020207@canterbury.ac.nz> <48246C10.6050703@behnel.de>
	<48250902.2020804@canterbury.ac.nz> <48253372.9060802@behnel.de>
	<48263800.8070606@canterbury.ac.nz> <4826AF8B.9010208@behnel.de>
	<4826BBAE.4060603@canterbury.ac.nz>
Message-ID: <4826C5A1.9050308@behnel.de>

Hi,

Greg Ewing wrote:
> Stefan Behnel wrote:
> 
>> Greg Ewing wrote:
>>
>>> There's no run-time
>>> distinction between a builtin class, an extension class
>>> implemented with Pyrex, or an extension class created some
>>> other way.
>>
>> I didn't mean at runtime. The compiler knows how to distinguish them.
> 
> Well, there's no compile-time distinction either, really.
> They're all the same thing. Treating a type differently
> depending on whether it was implemented using a Pyrex
> extension class would be bizarre and confusing.

Why? If the docs stated "explicitly testing isinstance(obj, ExtType) for a
Pyrex extension type will make sure it's really that specific type with the
expected field structure", then that's just another Pyrex/Cython specific
enhancement that deals with the grey area between Python and C, and that most
(should I say all?) users would expect anyway when calling isinstance(). This
does in no way interfere with Python semantics, as testing for (non-cdefed)
Python types will not imply a specific struct layout anyway.

Make it a feature: tell people that they can assert the C type structure by
calling isinstance(obj, MyExtType) explicitly, and that all other calls will
use the normal Python semantics.

Then, you could still use the following to do a non-explicit test:

>>     allowed_type = ExtType
>>     print isinstance(obj, allowed_type) # Python semantics

This could even become an example in the docs, right next to the paragraph
that explains the lack of "security" in the Python semantics.


> I'm not going to be changing
> my own test setup, so you're not going to be able to share
> any new tests that I add without converting them to your
> format.

Ok, then we'll continue to do that.

Stefan


From greg.ewing at canterbury.ac.nz  Sun May 11 12:05:59 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 22:05:59 +1200
Subject: [Cython] Language stability
In-Reply-To: <91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>
References: <4826AAE8.6010706@student.matnat.uio.no>
	<91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>
Message-ID: <4826C507.7040602@canterbury.ac.nz>

Robert Bradshaw wrote:
> Maybe... I really don't like the idea of mixing the two modes, e.g.
> 
> cdef cdef x = cdef(3)

This puts me in mind of PL/I, where it's perfectly
legal to write things like

  IF IF = THEN THEN THEN = ELSE ELSE ELSE = END END

But anyone even contemplating writing something like
that should be promptly shipped off to Guantanamo
Bay for a spot of waterboarding practice.

Also I'm not sure how the PL/I parser coped with
that and retained its sanity.

-- 
Greg

From robertwb at math.washington.edu  Sun May 11 12:25:21 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 03:25:21 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826C164.1090206@canterbury.ac.nz>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<4826C164.1090206@canterbury.ac.nz>
Message-ID: <F755B6FC-54B4-4DC7-AE31-BA5CE92A8A4D@math.washington.edu>

On May 11, 2008, at 2:50 AM, Greg Ewing wrote:
> Another thing to consider is how this would
> interact with subclassing. Will it be legal
> to do something like
>
>    cdef class B(A(len=10)):
>      ...
>
> and if so, how does the initialisation of B
> ensure that the length constraint is satisfied?

No, I don't think the above would be allowed. One would do

cdef class B(A):
     ...

cdef B(len=1) b


It should also be noted that some of the stuff being discussed here  
circumvents subclassing--the prototypical example is if x is a NumPy  
array with float entries then

x[i]

can be accessed directly via internal knowledge of the internal  
layout of the type of x. If x is an instance of a subclass that  
overrides __getitem__ to do something different this will break. This  
is bad, but so is manually hard-coding against the internal structure  
of x, and accessing arrays via __getitem__ is just way to slow for  
serious numerical algorithms. (Yes, you could say "just use a  
float*", but NumPy provides a lot of niceites, among them interfacing  
well with pure Python code and memory management).

- Robert


From stefan_ml at behnel.de  Sun May 11 12:34:05 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 May 2008 12:34:05 +0200
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <4826BDAF.9050106@canterbury.ac.nz>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>	<4826B413.4080608@behnel.de>
	<4826BDAF.9050106@canterbury.ac.nz>
Message-ID: <4826CB9D.4080309@behnel.de>

Hi,

Greg Ewing wrote:
> Stefan Behnel wrote:
> 
>> The latest discussion with Greg on the integer-loop syntax should make it
>> clear that Cython and Pyrex are not fundamentally diverging projects. And I
>> really don't think they should be.
> 
> That's good to hear.

And I really mean it. We even adapted our versioning scheme to Pyrex and put a
great deal of work into merging the changes in Pyrex 0.9.6.


> However, if one of the priorities
> of Cython is really going to be to compile pure Python code,
> there are likely to be problems at some point, because
> I'm going to want to make Pyrex do things that you won't
> like, and vice versa.

As long as there is a way to discuss these things and decide that the
difference is worth it, that's fine.


> BTW, the discussion we've been having about isinstance
> vs. typecheck is a bit ironic in this light. I'm arguing
> for *more* compatibility with Python in this area,
> and you're arguing for *less*. :-)

Read my answer. :) This is not about Python compatibility, as it deals with
the semantics of C types.

Stefan

From greg.ewing at canterbury.ac.nz  Sun May 11 12:41:25 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 22:41:25 +1200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <F755B6FC-54B4-4DC7-AE31-BA5CE92A8A4D@math.washington.edu>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<4826C164.1090206@canterbury.ac.nz>
	<F755B6FC-54B4-4DC7-AE31-BA5CE92A8A4D@math.washington.edu>
Message-ID: <4826CD55.9000103@canterbury.ac.nz>

Robert Bradshaw wrote:
> On May 11, 2008, at 2:50 AM, Greg Ewing wrote:
> 
>> Will it be legal to do something like
>>
>>   cdef class B(A(len=10)):
> 
> No, I don't think the above would be allowed.

That would sidestep some problems, I suppose, although
it would make it less of a general parameterised-types
mechanism.

Another thing bothers me a bit. Your simple little
__getitem__ example is all very well, but a full-blown
NumPy array with multiple dimensions and strides and
whatnot is a rather complicated beast.

Are you sure it will be possible to write a __getitem__
that deals with all that in its full generality while
still providing the efficiencies you're after? What
would such a __getitem__ implementation look like?

Also, as well as accessing single items, there's slicing
to consider. Is it going to be feasible to write a
__getitem__ and a __setitem__ that work together such
that things like

   a[i:j, k:l] = b[x, y, i:j, k:l]

do the right thing efficiently?

-- 
Greg

From robertwb at math.washington.edu  Sun May 11 12:50:12 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 03:50:12 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826B48E.3060708@student.matnat.uio.no>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
Message-ID: <9EA11E81-35A0-418D-AD47-5C9BEA51368B@math.washington.edu>

On May 11, 2008, at 1:55 AM, Dag Sverre Seljebotn wrote:
>> ---------a.pxd----------
>> cdef class A:
>>      cdef int len
>>      cdef int* data
>>
>>      cdef inline [final?] int __getitem__(A a, int i):
>>          """
>>          Note that subtypes can't override this.
>>          """
>>          if i < 0 or i > a.len:
>>              raise IndexError
>>          return data[i]
>>
>> ---------b.pyx--------
>>
>> from a cimport A
>> cdef A(len=10) a = a([1,2,3,4,5,6,7,8,9,10]) # I'll leave the init
>> function to your imagination
>> print a[9] # the code from __getitem__ gets inlined here, and since
>> len is known the a.len is resolved to 10 at compile time.
>>
>> (Here a.len tries to do a lookup first on the compile time type of a,
>> and that failing the runtime type of a. The compile time types need
>> not be struct members, but if they're not then they must be specified
>> because the "runtime" lookup would fail.)
>
> On the type arguments solution:
>
> My first thought is that I don't think it will cover all cases -- if
> native C++ template support is added, for instance, then the type
> arguments must probably behave differently there. So type arguments in
> itself is a more generic thing (related to the parser and type  
> handling
> etc.), and this specifies one concrete use of them (ie, what  
> happens to
> "cdef class"es when given type parameters).
>
> At first it struck me as way too magic. Then it grew on me. But then I
> dislike it again :-) So these are some non-conclusive thoughts:
>
> 1) A "disadvantage" is that it looks like one has to break down the  
> type
> specification vs. run-time parsing context seperation that we've  
> talked
> about earlier? -- how would you specify that a type parameter "T"  
> should
> take "unsigned short int*"? So to have consistency in any sane way one
> needs to make "all compile-time types also available run-time types"

No matter what mechanism is used, the issue of specifying types will  
take some work. I think the question of having types as parameters in  
other types is a difficult but somewhat orthogonal issue that needs  
to be dealt with, and having runtime equivalents of types will play  
into this (does ctypes already provide a mechanism?). or perhaps A 
(type=(cdef unsigned short int*)) as a way to bind a parameter to a  
type. Here "(cdef ...)" would be an allowable expression resolving to  
some (dynamically generated?) runtype variable that represents this  
type.

> 2) It seems to leave the way open for some confusing (if not  
> impossible
> to solve compiler-wise) results:
>
> cdef A(len=10) a = ...
> cdef A(len=8) b = a # What does this mean? Compile-time error?

Yep, compile time error. If a was a plain "cdef A" then it may be a  
runtime error. Of course if one is feeling risky one can cast ;-).

> Note, for instance, that while assigning a ndarray(2, int8) to a
> ndarray(3, int8) or ndarray(2, float32);

I assume you're trying to say this is not allowed...

> it should (or rather, might be
> wanted behaviour) for it to be legal to assign ndarray(2, uint8,
> flat=True) to ndarray(2, uint8, flat=False); where flat is flag to
> toggle multi-dimensional indexing.

Perhaps, or one can cast. I think it would also be useful (especially  
in the context of C++ support) to have the notion of "assignable from."

> Also consider:
>
> cdef A(len=10) a = ...
> a.len = 8 # legal or not?

Compile-time error--when it does a lookup on a.len, it will note it's  
trying to assign to a "constant."

> cdef A(len=8) b = a # Is this legal now?

Nope. The compile-time type of a is fixed.

> Such things would have to be described in more detail, and since it is
> hard to "guess" what the semantics should be (at least for me) it  
> might
> be an indication that this is too magic.

Think of len as an (compile-time fixed) attribute of a which has  
precedence over, and is used in place of a the (hypothetical runtime)  
attribute of a. This should answer the semantics questions. Again, it  
need not actually correspond to a actual attribute of A (in which  
case trying to access it would be a compile-time error when no  
compile-time value is given), though often it would.

One thing I like about this syntax is that it allows one to write a  
single piece of code that handles the generic (value only known at  
runtime) case and can be optimized if the value is known at compile  
time.

- Robert


From robertwb at math.washington.edu  Sun May 11 13:08:24 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 04:08:24 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826CD55.9000103@canterbury.ac.nz>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<4826C164.1090206@canterbury.ac.nz>
	<F755B6FC-54B4-4DC7-AE31-BA5CE92A8A4D@math.washington.edu>
	<4826CD55.9000103@canterbury.ac.nz>
Message-ID: <94D210AA-9967-4D0A-AEC5-0F84017555C2@math.washington.edu>

On May 11, 2008, at 3:41 AM, Greg Ewing wrote:
> Robert Bradshaw wrote:
>> On May 11, 2008, at 2:50 AM, Greg Ewing wrote:
>>
>>> Will it be legal to do something like
>>>
>>>   cdef class B(A(len=10)):
>>
>> No, I don't think the above would be allowed.
>
> That would sidestep some problems, I suppose, although
> it would make it less of a general parameterised-types
> mechanism.

Yes, that is true. We are not trying to re-invent C++ here (which is  
powerful but at the expense of a much steeper learning curve).

> Another thing bothers me a bit. Your simple little
> __getitem__ example is all very well, but a full-blown
> NumPy array with multiple dimensions and strides and
> whatnot is a rather complicated beast.
>
> Are you sure it will be possible to write a __getitem__
> that deals with all that in its full generality while
> still providing the efficiencies you're after? What
> would such a __getitem__ implementation look like?

I would imagine it would look a lot like the actual implementation of  
NumPy's current __getitem__ (well, after inlining some of the  
function calls, the current code does the actual work several levels  
down). Actually, probably more realistically, it would handle some of  
the simpler cases (one-dimensional for sure), and have an "else"  
clause that calls the current very-generic code (which stands the  
least to gain from an overhead-reducing perspective anyways). One  
hitch would be coming up with a nice syntax for handling a variable  
number of dimensions. If that wasn't feasible, hardcoding the method  
for 1, 2, and 3 dimensions would handle (I bet) most of cases, and it  
would call of to more generic code if the dimension was larger.

Related to this I am pretty sure we want to implement polymorphism  
for cdef (and perhaps special) methods.

> Also, as well as accessing single items, there's slicing
> to consider. Is it going to be feasible to write a
> __getitem__ and a __setitem__ that work together such
> that things like
>
>    a[i:j, k:l] = b[x, y, i:j, k:l]
>
> do the right thing efficiently?

Ah, that certainly is quite the challenge :-). Unwrapping it into a  
single loop that does the copy would be tricky (for one thing, the  
analysis would have to happen on a much higher level) but I think the  
right hand side can do things with strides to avoid any copying, and  
the assignment would do the efficient loop. This may actually already  
be quite efficient--the big performance hits come when one wants to  
manipulate elements one-by-one (rather than as blocks, for which  
NumPy has efficient code).

- Robert




From greg.ewing at canterbury.ac.nz  Sun May 11 13:05:45 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 23:05:45 +1200
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <4826CB9D.4080309@behnel.de>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>
	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>
	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>
	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>
	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>
	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>
	<4826B413.4080608@behnel.de> <4826BDAF.9050106@canterbury.ac.nz>
	<4826CB9D.4080309@behnel.de>
Message-ID: <4826D309.1050307@canterbury.ac.nz>

Stefan Behnel wrote:
> Read my answer. :) This is not about Python compatibility, as it deals with
> the semantics of C types.

But C types are Python types too. There's no way
of reading the programmer's mind to find out whether
he's thinking of his extension type as a C type or
a Python type when he passes it to isinstance().

So there is a semantic difference that could affect
the outcome in some cases, albeit rare ones.

Another consideration is that while I try to make
it so that you don't need to know about the C
API in order to use Pyrex, I also want to make it
so that it seems natural to someone who *does*
know something about the C API.

The C API manual documents PyObject_IsInstance as
having semantics that correspond to the the Python
isinstance(). Someone who knows that is likely to
expect them to correspond in Pyrex.

-- 
Greg



From greg.ewing at canterbury.ac.nz  Sun May 11 13:23:07 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 23:23:07 +1200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <9EA11E81-35A0-418D-AD47-5C9BEA51368B@math.washington.edu>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<9EA11E81-35A0-418D-AD47-5C9BEA51368B@math.washington.edu>
Message-ID: <4826D71B.4020306@canterbury.ac.nz>

On May 11, 2008, at 1:55 AM, Dag Sverre Seljebotn wrote:

> So to have consistency in any sane way one
> needs to make "all compile-time types also available run-time types"

I don't understand what you're getting at here. I thought
that all these type parameters -- whether they're "values"
like ints, or other types -- would be resolved at compile
time. In other words, in

   cdef A(len = x) a

the x would have to be a constant expression. There's no
problem with constant expressions, as they're already
used in C array declarations.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun May 11 13:28:55 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 23:28:55 +1200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <94D210AA-9967-4D0A-AEC5-0F84017555C2@math.washington.edu>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<4826C164.1090206@canterbury.ac.nz>
	<F755B6FC-54B4-4DC7-AE31-BA5CE92A8A4D@math.washington.edu>
	<4826CD55.9000103@canterbury.ac.nz>
	<94D210AA-9967-4D0A-AEC5-0F84017555C2@math.washington.edu>
Message-ID: <4826D877.3050500@canterbury.ac.nz>

Robert Bradshaw wrote:

> Yes, that is true. We are not trying to re-invent C++ here (which is  
> powerful but at the expense of a much steeper learning curve).

If you were going to reinvent some parameterised type
system, I'd suggest reinventing Eiffel instead -- it has
quite a nice one, without any of the bizarrities of C++
templates.

Another nice model to follow would be Haskell (although
I'd excuse you for skipping the typeclasses if you
weren't feeling up to it. :-)

-- 
Greg

From robertwb at math.washington.edu  Sun May 11 13:36:29 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 04:36:29 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826D71B.4020306@canterbury.ac.nz>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<9EA11E81-35A0-418D-AD47-5C9BEA51368B@math.washington.edu>
	<4826D71B.4020306@canterbury.ac.nz>
Message-ID: <DC89205F-3E6E-4080-A3B1-B58935AFCA79@math.washington.edu>

On May 11, 2008, at 4:23 AM, Greg Ewing wrote:

> On May 11, 2008, at 1:55 AM, Dag Sverre Seljebotn wrote:
>
>> So to have consistency in any sane way one
>> needs to make "all compile-time types also available run-time types"
>
> I don't understand what you're getting at here. I thought
> that all these type parameters -- whether they're "values"
> like ints, or other types -- would be resolved at compile
> time. In other words, in
>
>    cdef A(len = x) a
>
> the x would have to be a constant expression. There's no
> problem with constant expressions, as they're already
> used in C array declarations.

The issue here is handling something like

cdef A x
cdef A(type=cdef int) y = x # this needs to be able to do type  
checking at runtime

it also is inline with writing code like

if A.type == cdef int:
     ...
else:
     ....

to be able to "easily" handle types with type parameters.

- Robert


From greg.ewing at canterbury.ac.nz  Sun May 11 13:45:41 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 11 May 2008 23:45:41 +1200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <DC89205F-3E6E-4080-A3B1-B58935AFCA79@math.washington.edu>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<9EA11E81-35A0-418D-AD47-5C9BEA51368B@math.washington.edu>
	<4826D71B.4020306@canterbury.ac.nz>
	<DC89205F-3E6E-4080-A3B1-B58935AFCA79@math.washington.edu>
Message-ID: <4826DC65.7020807@canterbury.ac.nz>

Robert Bradshaw wrote:

> The issue here is handling something like
> 
> cdef A x
> cdef A(type=cdef int) y = x # this needs to be able to do type  
> checking at runtime

Hmmm. I think I would be happy if you could only do things
like that for type parameters which are Python types, not
C types.

It may help to disallow writing a bare declaration like

   cdef A x

if A has any unspecified type parameters. In the case of
C types, you would have to commit yourself to some concrete
type, and then the compiler can perform type checking.

In the case of Python types, you could write things like

   cdef A(object) x
   cdef A(Foo) y
   y = x

and a run-time type test would be done on the type parameters.

If you went on to say

   cdef A(int) z
   y = z

this would be a compile-time error, since a C int can never
be a subclass of Foo.

Or perhaps, as you say, something like ctypes could be used to
provide a runtime representation of C types.

-- 
Greg

From robertwb at math.washington.edu  Sun May 11 14:01:03 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 11 May 2008 05:01:03 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826DC65.7020807@canterbury.ac.nz>
References: <482423C4.9040005@student.matnat.uio.no>
	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>
	<4826B48E.3060708@student.matnat.uio.no>
	<9EA11E81-35A0-418D-AD47-5C9BEA51368B@math.washington.edu>
	<4826D71B.4020306@canterbury.ac.nz>
	<DC89205F-3E6E-4080-A3B1-B58935AFCA79@math.washington.edu>
	<4826DC65.7020807@canterbury.ac.nz>
Message-ID: <E96407B0-9F87-4089-BC62-1C1C9AADC50F@math.washington.edu>

On May 11, 2008, at 4:45 AM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> The issue here is handling something like
>>
>> cdef A x
>> cdef A(type=cdef int) y = x # this needs to be able to do type
>> checking at runtime
>
> Hmmm. I think I would be happy if you could only do things
> like that for type parameters which are Python types, not
> C types.
>
> It may help to disallow writing a bare declaration like
>
>    cdef A x
>
> if A has any unspecified type parameters. In the case of
> C types, you would have to commit yourself to some concrete
> type, and then the compiler can perform type checking.
>
> In the case of Python types, you could write things like
>
>    cdef A(object) x
>    cdef A(Foo) y
>    y = x
>
> and a run-time type test would be done on the type parameters.
>
> If you went on to say
>
>    cdef A(int) z
>    y = z
>
> this would be a compile-time error, since a C int can never
> be a subclass of Foo.
>
> Or perhaps, as you say, something like ctypes could be used to
> provide a runtime representation of C types.

The difficulty is that we want to be able to support things like  
NumPy arrays (whose element times are literally c doubles, ints, etc)  
and hopefully at least some support for wrapping C++ templated types  
(such as vector).

- Robert



From dagss at student.matnat.uio.no  Sun May 11 14:48:01 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 11 May 2008 14:48:01 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826CD55.9000103@canterbury.ac.nz>
References: <482423C4.9040005@student.matnat.uio.no>	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>	<4826B48E.3060708@student.matnat.uio.no>	<4826C164.1090206@canterbury.ac.nz>	<F755B6FC-54B4-4DC7-AE31-BA5CE92A8A4D@math.washington.edu>
	<4826CD55.9000103@canterbury.ac.nz>
Message-ID: <4826EB01.4050406@student.matnat.uio.no>

> Another thing bothers me a bit. Your simple little
> __getitem__ example is all very well, but a full-blown
> NumPy array with multiple dimensions and strides and
> whatnot is a rather complicated beast.

Robert's answered already but here's my thoughts on the matter.

If you haven't read it already, I've written about this in my GSoC 
application:

http://wiki.cython.org/DagSverreSeljebotn/soc/details

Highlights:
  * It does rely on some pretty "advanced" compile-time inlining (which 
I've called unrolling a couple of places). To handle multiple dimensions 
one must ideally also inline loops of compile-time values (but *only* in 
functions marked with a @cython.unroll, you don't normally want loops 
unrolled). I've written about my thoughts on this here:

http://wiki.cython.org/enhancements/inlining

  * If there's slices, simply hand them to the existing NumPy (i.e. 
(return (<object>self)[index]). (In particular, we should *not* inline 
slice copying, as that can potentially be handed to special MMX 
instructions etc. and is very CPU-specific). Multiple dimensions, 
negative indices etc. will probably be handled though.

  * It is possible to above is too utopic and one can say that it won't 
work. But, this is pretty much critical functionality for numerical 
Python use; and will be done anyway. If nothing else it will have to be 
implemented as a custom plugin to Cython, treating NumPy arrays as a 
language primitive type; but we really hope to find a way that is more 
in line with general Cython development.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun May 11 14:48:57 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 11 May 2008 14:48:57 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <4826EB01.4050406@student.matnat.uio.no>
References: <482423C4.9040005@student.matnat.uio.no>	<9CA6BD87-F619-4E3B-9C0E-3EF0B552FAD9@math.washington.edu>	<4826B48E.3060708@student.matnat.uio.no>	<4826C164.1090206@canterbury.ac.nz>	<F755B6FC-54B4-4DC7-AE31-BA5CE92A8A4D@math.washington.edu>	<4826CD55.9000103@canterbury.ac.nz>
	<4826EB01.4050406@student.matnat.uio.no>
Message-ID: <4826EB39.6060909@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
>> Another thing bothers me a bit. Your simple little
>> __getitem__ example is all very well, but a full-blown
>> NumPy array with multiple dimensions and strides and
>> whatnot is a rather complicated beast.
> 
> Robert's answered already but here's my thoughts on the matter.
> 
> If you haven't read it already, I've written about this in my GSoC 
> application:
> 
> http://wiki.cython.org/DagSverreSeljebotn/soc/details

BTW, scroll to the very bottom of that page and you'll find a full 
__getitem__ example.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun May 11 14:58:57 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 11 May 2008 14:58:57 +0200
Subject: [Cython] Language stability
In-Reply-To: <91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>
References: <4826AAE8.6010706@student.matnat.uio.no>
	<91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>
Message-ID: <4826ED91.70706@student.matnat.uio.no>

> I think there is a place for backwards-incompatibility. For example,  
> the __new__() being renamed to __cinit__() (as __new__() a very  
> different meaning in Python). However, if there is ever backwards- 
> compatibility issues, it should be for a *very* good reason. Also, we  
> might want to add syntax candy, so I'm not sure about the "previously  
> unavailable features" clause.

I'll agree to the last sentence, but not what is before (see below).

<snip>

> I don't think supporting multiple versions of the language, and  
> having a specific Pyrex flag, is a good idea. If one wants a specific  
> version of Cython, one can download and use that one. If one wants  
> *exatly* Pyrex syntax, use Pyrex itself. Any difference should be  
> clearly documented, and be for good reason, but having different  
> "modes" that we continue supporting seems cumbersome.

The sum of this is, I think, exactly why some people are sceptical to 
Pyrex and Cython as software projects. The thing is, if you are 
responsible for a big project and wondering if Cython is for you, 
there's *two* needs:

- Is it stable in the sense that I don't have to keep up with new 
developments in the languages? (Doesn't matter how trivial the change is 
as long as you cannot compile previously working code -- that's 
essentially wasted time, no matter how nice-looking the new syntax is.)

- Bugfixes! You can't simply say download an old revision and use that 
one, because when you stumble upon bugs in Cython (which there are!), 
you have to consistently work around them because you cannot upgrade for 
the bugfixes without also upgrading your syntax.

Look to Python and how they promise to keep 2.6 alive for a long time 
even if 3.0 arrives...

I think there's basically two models:
- The big project way: Fork off new development branches but keep 
backporting bugfixes to older releases.
- Keep supporting multiple language revisions in the same branch. (This 
is not too hard in many cases; one can probably often simply stick a 
backwards-compatability transform for the syntax in the pipeline if the 
right conditions are met. Though there's the danger of a growingly 
complex parser.).

I don't think the current model (keep adding bugfixes to newest release, 
which also reserves the right to change the language without being 
backwards-compatible) is a good idea.


> Maybe... I really don't like the idea of mixing the two modes, e.g.
> 
> cdef cdef x = cdef(3)

The idea is really only that Cython should be able to compile *all* 
Python code, and that includes Python code with the cdef used as 
variable names. If Cython cannot compile Python code containing "cdef" 
as a variable it could really, as you've said put it sometimes, be 
considered a bug since it's possible to write Python code that's not 
Cython-compilable. But a mode flag probably fixes that in a good way 
without allowing stuff like above.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun May 11 15:50:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 11 May 2008 15:50:00 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
Message-ID: <3293365801.1640468@smtp.netcom.no>

Robert wrote:
>One thing I like about this syntax is that it allows one to write a  
>single piece of code that handles the generic (value only known at  
>runtime) case and can be optimized if the value is known at compile  
>time.

Indeed, that's what I liked about it. What I dislike is the exact way the type parameters are put into fields of the same name. (It basically means that the caller decides what is compile-time values and not, and things can easily get "out of sync", if method implementations depends on them being runtime?)

What about this modification:

cdef A(len=10) a

then "a" is considered assignable "any instance of A which had len=10 passed to the constructor" (or instances which are willing to convert to that). The constructor can then decide how to assign the values to fields (or entirely drop them for compile-time treatment). Might require smart inlining of the constructor in perhaps impossible ways though, I'm waving my hands and will think more...

That also solves the  overloaded syntax problems -- () always do in fact refer to the constructor...

Don't take this post too seriously, have to think more...

Dag Sverre


From jim-crow at rambler.ru  Sun May 11 20:07:43 2008
From: jim-crow at rambler.ru (Anatoly A. Kazantsev)
Date: Mon, 12 May 2008 01:07:43 +0700
Subject: [Cython] defining module constants
Message-ID: <20080512010743.5a583b3c.jim-crow@rambler.ru>

Hello.

I have same problem.
Will be any answers or advices? :-)

-- 
Protect your digital freedom and privacy, eliminate DRM, learn more at
http://www.defectivebydesign.org/what_is_drm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080512/725d1622/attachment.pgp 

From dagss at student.matnat.uio.no  Sun May 11 23:00:33 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 11 May 2008 23:00:33 +0200
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <3293365801.1640468@smtp.netcom.no>
References: <3293365801.1640468@smtp.netcom.no>
Message-ID: <48275E71.2030109@student.matnat.uio.no>

Ok, been thinking some more. To sum up, what we're after is some way of 
having the following work (A is a class with a value field, printme is a 
method somehow selected for inlining and optimization for Cython):

cdef A a = get_a()
a.printme()
a.printme()

turns into

cdef A a = get_a()
print a.value
print a.value

, while this:

cdef A(value=23) a = get_a()
a.printme()
a.printme()

would be turned into

cdef A a = get_a()
if a.value != 23: raise TypeError(...) # line (a)
print 23 # line (b)
print 23

(This is the final result, not saying it is a recipe for how it happens 
magically).

So, what is happening is that we want to set up some assumptions about 
an object (without actually changing the object); and have the code 
after generated by making use of those assumptions.

I have some seperate proposals building further on Robert's ideas here.

1) An explicit method for the assumptions bit. I.e:

cdef class A:
     cdef int value

     cdef __assume__(self, int value):
         if self.value != value: raise AssumptionError(...)

The argument list to this function is basically a more explicit 
declaration of type arguments. The reason I want this, so explicitly 
(rather than using a more general __coerce__ which must probably also be 
added) is that it hints strongly to the writer of the class A to treat 
self.value as "const". Once an assumption is made by having __assume__ 
called, you cannot "back out" and change self.value. This seem to make 
it explicit that value should be treated as a const after construction; 
and it leaves the contract for the name, number and types of "type 
arguments" on the class creator side rather than the caller side.

In addition to just checking assumptions, this method can also put 
general constraints on the arguments (i.e., "if value < 0: raise 
ValueError(...)").

Important: This does *not* address how one can make optimizations later 
on, line (b), it is simply a way to insert the assumption line, line 
(a). Also, __assume__ can simply be called in the normal way.

2) With this in place, it seems ok to follow Robert's proposal and 
automatically treat fields having the same name as type arguments as 
known compile-time. The parameter list to __assume__ restricts which 
fields can be used.

I am still thinking about something more explicit like

cdef class A:
     cdef int value
     cdef int not_possible_typearg

     __typearguments__ = ["value", "compiletimeonlyarg"]

but it doesn't seem strictly necesarry as the argument list to 
__assume__ can serve the same role; and in a possibly more dynamic way, ie

cdef A(constant_alpha=True, alpha=4) a = x # ok
print a.alpha # ok to optimize...

cdef A(constant_alpha=False, alpha=4) a = x # __assume__ might raise...
# .. run-time error because of invalid combination,
# .. so code below will never run, which is lucky because
# .. alpha is now for some reason changing constantly.
print a.alpha # Will be optimized but never run

3) I still want to throw __init__ into the mix. The main reason: For 
type inference, it would be nice if

A = ndarray(shape=(4, 4), dtype=float64, buffer=arr)

would automatically (because type arguments are somehow interlinked with 
constructor arguments) be type-inferred to

cdef ndarray(shape=(4,4), dtype=float64) A
A = ndarray(shape=(4,4), dtype=float64, buffer=arr)

If so, keeping () for the type argument syntax would also make more 
sense and be less confusing.

Perhaps all that is needed is this rule at the type-inference stage:

  * A call to a known constructor (of a type which is a candidate for 
typing) obviously leads to typing it explicitly
  * At the same time, any arguments passed to the constructor is checked 
for a match with the __assume__ signature -- the arguments that 
__assume__ can take are then put into the type arguments list.

(If so, we can pretty much ignore this for now, but we have a "defense" 
for using the () syntax.)


Dag Sverre


-- 
Dag Sverre

From stefan_ml at behnel.de  Mon May 12 10:13:46 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 10:13:46 +0200
Subject: [Cython] Arc Riley, PyMill, and FUD
In-Reply-To: <4826D309.1050307@canterbury.ac.nz>
References: <85e81ba30805101812r15979a9dk8c178123ade0b347@mail.gmail.com>	<bad82a80805101837u619a97f6m22468b8060d02600@mail.gmail.com>	<85e81ba30805101922m158e9614s5f01fddc68e48fc5@mail.gmail.com>	<bad82a80805101951n18203d8cma061c8277a3a67e@mail.gmail.com>	<85e81ba30805102014n7b0f779fm2ac27a6b915ddbae@mail.gmail.com>	<bad82a80805102054k4f6b50d6nc6651a38223eb95b@mail.gmail.com>	<4826B413.4080608@behnel.de>
	<4826BDAF.9050106@canterbury.ac.nz>	<4826CB9D.4080309@behnel.de>
	<4826D309.1050307@canterbury.ac.nz>
Message-ID: <4827FC3A.90109@behnel.de>

Hi,

Greg Ewing wrote:
> But C types are Python types too. There's no way
> of reading the programmer's mind to find out whether
> he's thinking of his extension type as a C type or
> a Python type when he passes it to isinstance().

But Pyrex types are already optimised in a couple of ways. If you add a struct
field, Pyrex will not use Python calls to access it but plain C calls. If you
assign it to a variable, Pyrex will not look it up in the module dict.
However, if you test its type, you expect the user to assume that Pyrex does
not handle this its own way? I think most users will not agree with you here,
and I feel comfortable saying so from the (although unrelated) feedback we get
on the mailing list. At least Cython users expect optimised code that special
cases common use patterns internally (such as "for ... in range()"), and they
are surprised when they find that Cython/Pyrex gives them normal Python
semantics in a "sub-optimal" way.


> Another consideration is that while I try to make
> it so that you don't need to know about the C
> API in order to use Pyrex, I also want to make it
> so that it seems natural to someone who *does*
> know something about the C API.

Taking this a bit out of context, this is exactly my point. If you don't know
much about the C-API, you a) won't notice the difference and b) will expect
isinstance(obj, ExtType) to make sure that the type has the expected type struct.

However, if you know about the C-API and about the possible problems with
isinstance() and PyObject_IsInstance(), you will either write the type check
in an explicit C-API call or check the Pyrex/Cython docs for an easy way to do
it for you, and then find the section on the semantics difference between
Python and Pyrex here. In this case, you will most likely appreciate the fact
that isinstance() has semantics in Pyrex that make sure isinstance(obj,
ExtType) will give you the expected struct, and that you can easily get back
the original Python semantics by not testing for the extension type explicitly.

There is only one case where things are unexpected, which is if you rely on
isinstance() being overridable by type.__class__. In this case, your code will
fail and you will have to debug it - but only if you have written it under the
expectation that extension types behave exactly like Python types. And I think
that is a very unlikely expectation to have because users who found their path
all the way through the language docs to figure out how to write an extension
type will already have lost this expectation.

For me, that's not the path of highest Python language compatibility, but it's
the clear path of least surprises - as long as it's documented.

Stefan

From stefan_ml at behnel.de  Mon May 12 10:23:05 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 10:23:05 +0200
Subject: [Cython] Language stability
In-Reply-To: <4826ED91.70706@student.matnat.uio.no>
References: <4826AAE8.6010706@student.matnat.uio.no>	<91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>
	<4826ED91.70706@student.matnat.uio.no>
Message-ID: <4827FE69.7020804@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> - Is it stable in the sense that I don't have to keep up with new 
> developments in the languages?

It should be (and stay) that stable. The latest syntax change regarding the
for loop is not required for Cython, where the (IMHO much more obvious) cdef +
range() syntax is optimised, which also works in Pyrex. We should just take
care not to remove the older for-from syntax for compatibility reasons (which
I don't remember considering for Cython anyway). And I'm still fighting
against the two new functions that Greg introduced for type testing.

To make the rest a bit shorter: I'm stronly -1 on supporting different
language versions and even -1 on bug-fixing multiple branches. That's just two
reasons more to keep the language itself stable. And if we really have to
change the syntax, that's another two reasons more why we should consider this
very, very carefully and only let syntax changes in that are very well backed
by arguments.

BTW, I would find it somewhat ironic if the Cython project, which I found to
be a lot more agile than Pyrex lately, could agree on keeping the language
more stable than Pyrex itself. ;)

Stefan


From dagss at student.matnat.uio.no  Mon May 12 10:34:15 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 12 May 2008 10:34:15 +0200
Subject: [Cython] Language stability
In-Reply-To: <4827FE69.7020804@behnel.de>
References: <4826AAE8.6010706@student.matnat.uio.no>	<91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>	<4826ED91.70706@student.matnat.uio.no>
	<4827FE69.7020804@behnel.de>
Message-ID: <48280107.204@student.matnat.uio.no>


> To make the rest a bit shorter: I'm stronly -1 on supporting different
> language versions and even -1 on bug-fixing multiple branches. That's just two

Yes, yes, I'm not saying we do! I really hope we stay backwards-compatible.

What I am saying is that we should have a policy (and a website 
statement) saying that *if* we break backwards-compatability, *then* we 
will commit to either backporting bugfixes or support multiple language 
versions (and *if* that happens, I think the latter one is going to be 
less of a pain, but one can discuss that if that happens).

Without that, we can't really give people the security they need, 
they're at our mercy that we do not break the compatability, so to speak.

I think "from __future__ import division" is an excellent example of 
supporting multiple language levels in a nice way, it is not that 
horrible to do. But let's hope not.

-- 
Dag Sverre

From stefan_ml at behnel.de  Mon May 12 10:36:36 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 10:36:36 +0200
Subject: [Cython] cython add vs __add__
In-Reply-To: <e7ba66e40805101430t3d39e54dx490d1dade90eaca5@mail.gmail.com>
References: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>	<e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>	<482510A1.7090806@canterbury.ac.nz>
	<e7ba66e40805101430t3d39e54dx490d1dade90eaca5@mail.gmail.com>
Message-ID: <48280194.5080708@behnel.de>

Hi,

Lisandro Dalcin wrote:
> On 5/10/08, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>> Lisandro Dalcin wrote:
>>  > I believe you are right, it seems a bug,
>>  >
>>
>>> cdef class myTest
>>  >     def __add__(myTest self, other): # note declaration: myTest self
>>  >         return self.thisPtr.add(other)
>>
>>
>> It's not a bug. With extension types, the first argument
>>  of operator methods isn't necessarily self, so it's not
>>  automatically typed as such.
>
> What do you understand for 'A parameter named self is of the type the
> method belongs to'. Anyway, do not you believe the current behavior is
> a bit counter-intuitive?

I agree that it is counter-intuitive for Python users, but these things are
also non-trivial in Python. The Python API actually distinguishes between
__add__ etc. and __radd__ etc., which are merged into a single function in
Pyrex. Maybe that should be mentioned in the docs.

http://docs.python.org/ref/numeric-types.html

I think anyone who proves to be able to implement a complete numeric type in
Python should be able to implement it in Pyrex - possibly minus some fixes in
the Pyrex/Cython documentation. But you definitely will have to read the docs
to get this right in Python *and* in Pyrex.

Stefan

From dagss at student.matnat.uio.no  Mon May 12 10:41:31 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 12 May 2008 10:41:31 +0200
Subject: [Cython] cython add vs __add__
In-Reply-To: <48280194.5080708@behnel.de>
References: <4A9A59A6-7CFF-4F93-A91C-8F22804DC0F6@gmail.com>	<e7ba66e40805091559w5600414dy439571f9c49643be@mail.gmail.com>	<482510A1.7090806@canterbury.ac.nz>	<e7ba66e40805101430t3d39e54dx490d1dade90eaca5@mail.gmail.com>
	<48280194.5080708@behnel.de>
Message-ID: <482802BB.80004@student.matnat.uio.no>


> I agree that it is counter-intuitive for Python users, but these things are
> also non-trivial in Python. The Python API actually distinguishes between
> __add__ etc. and __radd__ etc., which are merged into a single function in
> Pyrex. Maybe that should be mentioned in the docs.

Wouldn't it be an idea (well, I suppose it's too late now, just 
wondering what you think) to call it __cadd__ instead then? Having 
custom functionality is ok, but I think the expectations of Python 
compliancy gets bigger when it has the same name.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon May 12 10:43:02 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 12 May 2008 10:43:02 +0200
Subject: [Cython] Visitors and speed issues
In-Reply-To: <8f8f8530805110126v697012a2pd5770fa877b89dbd@mail.gmail.com>
References: <481AF97B.8030305@student.matnat.uio.no>	<481BFDDA.8060007@behnel.de>	<1078.195.159.185.117.1209803236.squirrel@webmail.uio.no>	<452383C0-47A5-4E0D-840E-0158348C5755@math.washington.edu>	<48215041.7070805@student.matnat.uio.no>	<5303982D-51B8-4D6F-B363-BDC8C9729892@math.washington.edu>	<48233D62.7000200@student.matnat.uio.no>
	<8f8f8530805110126v697012a2pd5770fa877b89dbd@mail.gmail.com>
Message-ID: <48280316.7090503@student.matnat.uio.no>

Gary Furnish wrote:
> Efficiency does matter to Sage though, and an O(1) overhead is very
> far from negligible.  Cython uses real C vtables, so there are
> absolutely no dict-based lookups involved with cdef class
> polymorphism.  -Infinity to any proposal that makes code slower.

This is interesting; if there's a real-world, noticeable speed penalty 
to dict-based visitors then I agree they should be made vtable-based 
instead. I'd like to discuss it a bit more first though:

Just to have more data I benchmarked this on my own computer; 10^7 
visitor lookups were done in a loop. Doing everything as typed as 
possible (all arguments typed, all classes and methods cdef-ed) it took 
1.7 seconds, while using a full-blown dict-lookup (with 100 items in the 
dict) it took 8.4 seconds.

Assuming an "average" file has 10 000 nodes syntax tree nodes in it, and 
that there will be 40 visitors passing through the tree in a 
compilation, that gives ~0,07 seconds for typed and ~0,34 seconds for 
dict-based, ie a difference in 0,27 seconds per file. So on 100 files 
that is half a minute wasted.

I don't know if that is much or little, how long does Cython spend on 
compiling 100 files of 10 000 nodes today?

So far all is well. However, the problem is that Cython code is not 
using cdef classes, nor typing the method arguments! Nor will "cdef 
class" be likely to hit the source, because even if we'd like to be able 
to compile it, it is a must that it can use the Python interpreter in 
order to compile Cython with Cython (we don't want to bother with having 
a bootstrapping chain for Cython I think...).

So in order to cash out on this one must first make Python code produce 
cdef classes with typed method signatures. Of course full-blown 
inference of this is not needed, decorator support (or some comment 
format support, if we want it compilable in Python 2.3) could be used 
instead.

Thoughts?

(BTW, when I said O(1) I was referring to class construction, which 
happens once per invocation of Cython, not dict lookups (in this context 
I consider dict lookups as giving an "O(n) penalty", since there would 
be a constant number of them per node, and the number of nodes is 
proportional to input file size). If the invocation time of Cython is a 
problem then one better just have Cython compile many files in one 
process session, the Python interpreter starts up relatively slowly 
anyway...)


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon May 12 10:57:18 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 12 May 2008 10:57:18 +0200
Subject: [Cython] Visitors and compiling Cython in Cython
Message-ID: <4828066E.6040307@student.matnat.uio.no>

Gary recently voiced concern over dict-based visitor lookups perhaps 
giving a speed penalty

Now, I might be wrong, but vtables can only be used if classes or 
methods are declared as "cdef", right? And on the other hand, I think we 
definitely want Cython to be runnable in Python, so that we don't have 
to ship Cython in .c-form to people who don't already have Cython 
installed, not to mention having to "bootstrap" any new Cython feature 
using the current Cython features (agreed or not?)

Is there going to be a way to have this (i.e., vtable-based Cython) in 
the foreseeable future? Using decorators certainly comes to mind, 
however that breaks 2.3 support. Perhaps special comments?

#cython: cdef Visitor:
class Visitor:

    #cython: cdef object pre_FuncDefNode(FuncDefNode node):
    def pre_FuncDefNode(node):
       ...


Because if this is never going to happen, I think we might as well go 
for nice, dict-based visitors. If it is going to happen though, it is a 
case for vtable-based visitors.

Dag Sverre

Appendix 1:

For people who haven't followed that discussion, dict-based visitors 
refers to writing visitors in a way similar to this:

class MyVisitor(Visitor):
     def make_function_scope(node):
         ...

     matches = [
         pre(FuncDefNode, make_function_scope)
     ]


with no changes to existing node classes, while vtable-bases would be 
like this:

class MyVisitor(Visitor):
     def pre_FuncDefNode(self, node):
         # make function scope
         ....

and each node class must have something like this added (*no*, this 
can't be done magically some other place like you're used to, that would 
break the vtable-ness of it):

class FuncDefNode(Node):
     def accept(self, v):
         v.visit_FuncDefNode(self)




From stefan_ml at behnel.de  Mon May 12 11:18:44 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 11:18:44 +0200
Subject: [Cython] Language stability
In-Reply-To: <48280107.204@student.matnat.uio.no>
References: <4826AAE8.6010706@student.matnat.uio.no>	<91E6D358-EAB0-4322-BF5D-9964594C90B0@math.washington.edu>	<4826ED91.70706@student.matnat.uio.no>	<4827FE69.7020804@behnel.de>
	<48280107.204@student.matnat.uio.no>
Message-ID: <48280B74.1030106@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> I think "from __future__ import division" is an excellent example of 
> supporting multiple language levels in a nice way, it is not that 
> horrible to do.

BTW, I actually think we need to support that anyway if we want to compile
Python code. Things like

 >>> from __future__ import unicode_literals
 >>> from __future__ import print_function

provide a simple upgrade path for Py3 syntax support. Both would just be
enabled for plain Python code (.py) when the compiler runs in a Py3 environment.

Now that you mention it, maybe changes in Cython and Pyrex could be enabled by

 >>> from __future__ cimport some_new_pyrex_syntax_feature

(mind the "cimport") ? We would then still need a roadmap for a versioned
upgrade path if we add a syntax change.

Stefan

From stefan_ml at behnel.de  Mon May 12 11:49:55 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 11:49:55 +0200
Subject: [Cython] Visitors and compiling Cython in Cython
In-Reply-To: <4828066E.6040307@student.matnat.uio.no>
References: <4828066E.6040307@student.matnat.uio.no>
Message-ID: <482812C3.1040908@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Gary recently voiced concern over dict-based visitor lookups perhaps
> giving a speed penalty

Why? You could use lazy initialisation. Just look up a node type in the dict
(which is fast) and if it's not in there yet, walk it's base types (adding
each one to the dict) until there is one that already is in the dict, which
then determines the result for the lookup and for the newly added base types.
That way, you'd quickly end up with a complete flattened type hierarchy in the
dict. The total number of dict updates will be <= the total number of node
types and the time for a node type lookup is still O(1). The type registration
would follow the same algorithm as a lookup, but the new callbacks would be
added to the results of the found base types.


> I think we 
> definitely want Cython to be runnable in Python, so that we don't have 
> to ship Cython in .c-form to people who don't already have Cython 
> installed, not to mention having to "bootstrap" any new Cython feature 
> using the current Cython features (agreed or not?)

Running it in C will only ever be an optimisation, possibly triggered by an
option to setup.py that will compile Cython using Cython itself before
installing it as binary module.


> Is there going to be a way to have this (i.e., vtable-based Cython) in 
> the foreseeable future? Using decorators certainly comes to mind, 
> however that breaks 2.3 support. Perhaps special comments?
> 
> #cython: cdef Visitor:
> class Visitor:

Robert already stated his opinion on this a couple of times and I second it:
hiding code semantics in Python comments is a bad idea. We should only do this
if we really find we can't do something any other way. I don't see that here.

Stefan


From dagss at student.matnat.uio.no  Mon May 12 12:06:15 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 12 May 2008 12:06:15 +0200
Subject: [Cython] Visitors and compiling Cython in Cython
Message-ID: <48281697.2000907@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> Gary recently voiced concern over dict-based visitor lookups perhaps
>> giving a speed penalty
> 
> Why? You could use lazy initialisation. Just look up a node type in the dict
> (which is fast) and if it's not in there yet, walk it's base types (adding
> each one to the dict) until there is one that already is in the dict, which
> then determines the result for the lookup and for the newly added base types.
<snip>

Sure, sure, that's already being done and in fact you'll see that 
*exact* procedure in my patch! :-) The issue Gary has is indeed with raw 
dict lookup performance. (Consider that this will be between 10 and 50 
times to every single node in the tree of a file.)

I'll quote Gary so you have the context:

"""
Efficiency does matter to Sage though, and an O(1) overhead is very
far from negligible.  Cython uses real C vtables, so there are
absolutely no dict-based lookups involved with cdef class
polymorphism.  -Infinity to any proposal that makes code slower.

<new post>

Well, clarifying what I meant a bit more, the *biggest* speed loss
anywhere is dictionary lookups.  If your going to use dictionary
lookups at object time, you lose most of the advantage of using cython
to compile cython.
"""

Apparently, you don't agree with Gary here :-)

Since this will have a big effect on all of Cython I thought I'd 
investigate it properly before truncating Gary's -Infinity to -1 :-)

--
Dag Sverre



-- 
Dag Sverre


From stefan_ml at behnel.de  Mon May 12 12:54:41 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 12:54:41 +0200
Subject: [Cython] Visitors and compiling Cython in Cython
In-Reply-To: <48281697.2000907@student.matnat.uio.no>
References: <48281697.2000907@student.matnat.uio.no>
Message-ID: <482821F1.8060808@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> I'll quote Gary so you have the context:
> 
> """
> Efficiency does matter to Sage though, and an O(1) overhead is very
> far from negligible.  Cython uses real C vtables, so there are
> absolutely no dict-based lookups involved with cdef class
> polymorphism.  -Infinity to any proposal that makes code slower.
> 
> <new post>
> 
> Well, clarifying what I meant a bit more, the *biggest* speed loss
> anywhere is dictionary lookups.  If your going to use dictionary
> lookups at object time, you lose most of the advantage of using cython
> to compile cython.
> """

I don't think the intention to compile Cython should impact our design
decisions. If a Cython compiled Cython compiler is not any faster than a
Python based one, then that's a reason to improve Cython, not its code.

Stefan

From stefan_ml at behnel.de  Mon May 12 12:58:02 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 12:58:02 +0200
Subject: [Cython] Visitors and compiling Cython in Cython
In-Reply-To: <48281697.2000907@student.matnat.uio.no>
References: <48281697.2000907@student.matnat.uio.no>
Message-ID: <482822BA.8070800@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> You could use lazy initialisation. Just look up a node type in the dict
>> (which is fast) and if it's not in there yet, walk it's base types (adding
>> each one to the dict) until there is one that already is in the dict, which
>> then determines the result for the lookup and for the newly added base types.
> 
> Sure, sure, that's already being done and in fact you'll see that 
> *exact* procedure in my patch! :-)

Obviously. :)


> The issue Gary has is indeed with raw 
> dict lookup performance. (Consider that this will be between 10 and 50 
> times to every single node in the tree of a file.)

Unless you annotate a parse tree with all its transformers in one traversal step.

Stefan


From stefan_ml at behnel.de  Mon May 12 13:04:49 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 12 May 2008 13:04:49 +0200
Subject: [Cython] Visitors and compiling Cython in Cython
In-Reply-To: <4828066E.6040307@student.matnat.uio.no>
References: <4828066E.6040307@student.matnat.uio.no>
Message-ID: <48282451.1000202@behnel.de>

Hi again,

Dag Sverre Seljebotn wrote:
> #cython: cdef Visitor:
> class Visitor:
> 
>     #cython: cdef object pre_FuncDefNode(FuncDefNode node):
>     def pre_FuncDefNode(node):

What about just using plain old "cdef" to make it valid Cython code, and then
run a cy2py tool over it in setup.py that strips it down into valid Python
code? That's how 2to3 works. The tool could just bail out if it finds anything
that's not pythonisable, such as cimports and C pointer types.

The down side is obviously that this would prevent us from running Cython from
the source directory ...

Stefan


From dalcinl at gmail.com  Mon May 12 14:29:34 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 12 May 2008 09:29:34 -0300
Subject: [Cython] String interning and Python 3
In-Reply-To: <31D60ED7-91BE-435B-A04B-AF14E733FFAB@math.washington.edu>
References: <4825AFFB.6040305@behnel.de>
	<e7ba66e40805101406n468ef323i4aa49b7b39c8cb62@mail.gmail.com>
	<31D60ED7-91BE-435B-A04B-AF14E733FFAB@math.washington.edu>
Message-ID: <e7ba66e40805120529q74324031v40b19e4e29c30790@mail.gmail.com>

On 5/11/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>  It also offers the
>  advantage that the lookup strings don't need to be re-allocated each
>  time they're needed.

Thats very true on Cython as it has the table. Interestingly, Python
sources uses PyString_InternFromString() in many places, but this
function do actually create a new tmp string, but in the end you get
the interned string.




>  > On 5/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>  >> Hi,
>  >>
>  >>  I'm wondering how to continue the support for this feature given
>  >> the fact that
>  >>  identifiers are Unicode strings in Py3. We currently only intern
>  >> byte strings
>  >>  that look like Python identifiers, so in Py3, they simply no
>  >> longer look like
>  >>  identifiers, as they are not Unicode strings.
>  >>
>  >>  I can see four ways how to deal with this:
>  >>
>  >>  1) drop string interning completely
>  >>
>  >>  2) disable string interning in Py3 and use normally created byte
>  >> strings instead
>  >>
>  >>  3) keep separate sets of identifier-like byte strings and unicode
>  >> strings in
>  >>  the compiler and write them into the C file. Then, depending on
>  >> the Python
>  >>  version, either intern the byte strings or the unicode strings,
>  >> and create the
>  >>  other set as un-interned strings.
>  >>
>  >>  4) keep the information if a string should be interned for all
>  >> strings we deal
>  >>  with (bytes and unicode), remove the intern tab and merge it with
>  >> the general
>  >>  string tab by adding an additional field "intern". Then
>  >> __Pyx_InitStrings()
>  >>  would create the strings differently depending on the compile
>  >> time Python
>  >>  version, i.e., it would intern Unicode identifiers in Py3 and
>  >> byte string
>  >>  identifiers in Py2, and create everything else as normal strings.
>  >>
>  >>  Personally, I favour 4) - although I could live with 1) - but
>  >> since I'm not
>  >>  quite sure what the original intention of string interning was
>  >> (saving
>  >>  memory?), I'd like to hear other opinions first.
>  >>
>  >>  Stefan
>  >>  _______________________________________________
>  >>  Cython-dev mailing list
>  >>  Cython-dev at codespeak.net
>  >>  http://codespeak.net/mailman/listinfo/cython-dev
>  >>
>  >
>  >
>  > --
>  > Lisandro Dalc?n
>  > ---------------
>  > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>  > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>  > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>  > PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>  > Tel/Fax: +54-(0)342-451.1594
>  > _______________________________________________
>  > Cython-dev mailing list
>  > Cython-dev at codespeak.net
>  > http://codespeak.net/mailman/listinfo/cython-dev
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From languitar at semipol.de  Mon May 12 15:29:45 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Mon, 12 May 2008 15:29:45 +0200
Subject: [Cython] assigning to struct member
Message-ID: <48284649.9030709@semipol.de>

Hi again,

maybe I'm blind or I don't know but how do I assign a value to a struct
member?

This code:
cdef plugData *data = <plugData*>malloc(sizeof(plugData))
data.ident = "foo"

with this definition of plugData:
ctypedef struct plugData:
	plugDefinition *plug
	char *ident
	void *data

generates an error:
"Object of type 'plugData' has no attribute 'ident'"

What's wrong with this?

Thanks
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080512/9c4cbaae/attachment.pgp 

From dagss at student.matnat.uio.no  Mon May 12 16:12:25 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 12 May 2008 16:12:25 +0200
Subject: [Cython] Visitors and compiling Cython in Cython
In-Reply-To: <482821F1.8060808@behnel.de>
References: <48281697.2000907@student.matnat.uio.no>
	<482821F1.8060808@behnel.de>
Message-ID: <48285049.5060404@student.matnat.uio.no>


> I don't think the intention to compile Cython should impact our design
> decisions. If a Cython compiled Cython compiler is not any faster than a
> Python based one, then that's a reason to improve Cython, not its code.

The issue here is that using the "classical visitor pattern" is 
theoretically a much more likely candidate for optimization than 
dict-based visitors.

Consider them different algorithms which currently perform at equal 
speed, but one of them has a much higher potential for hypothetical 
future optimization. No matter how smart Cython becomes it cannot switch 
to faster algorithms...

 >> The issue Gary has is indeed with raw
 >> dict lookup performance. (Consider that this will be between 10 and 50
 >> times to every single node in the tree of a file.)
 >
 > Unless you annotate a parse tree with all its transformers in one 
traversal step.

This is highly non-trivial though -- the output of one transform is 
changed and new nodes, which are the input to the next transform. And, 
the output of one "node transform" is potentially dependant on anything 
in child nodes, parent nodes, sibling nodes etc.

To get anywhere one would at least have to put in restrictions on what a 
node visitors was allowed to read from the tree (likely inspect children 
and ancestors but not following-siblings). And still it is not simple. 
Let's do that when everything else is working :-)

 > What about just using plain old "cdef" to make it valid Cython code, 
and then
 > run a cy2py tool over it in setup.py that strips it down into valid 
Python
 > code? That's how 2to3 works. The tool could just bail out if it finds 
anything
 > that's not pythonisable, such as cimports and C pointer types.
 >
 > The down side is obviously that this would prevent us from running 
Cython from
 > the source directory ...

Good idea. (I actually hope nothing like this is attempted for a while, 
but what our possibilities are affects how visitors are written...).

Still my own summary is: This doesn't matter that much now -- the 
important thing is getting visitors at all. No matter what solution is 
picked, going from one to the other will take about a day of 
conceptually very simple work. So I'm still for dict-based; allows us to 
move faster, until something like the above is actually in place.

-- 
Dag Sverre

From dalcinl at gmail.com  Mon May 12 16:12:56 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 12 May 2008 11:12:56 -0300
Subject: [Cython] assigning to struct member
In-Reply-To: <48284649.9030709@semipol.de>
References: <48284649.9030709@semipol.de>
Message-ID: <e7ba66e40805120712j5ab6ba13i370d8377812bac8a@mail.gmail.com>

Do you also have a 'cdef class plugData' defined somewhere ?


On 5/12/08, Johannes Wienke <languitar at semipol.de> wrote:
> Hi again,
>
>  maybe I'm blind or I don't know but how do I assign a value to a struct
>  member?
>
>  This code:
>  cdef plugData *data = <plugData*>malloc(sizeof(plugData))
>  data.ident = "foo"
>
>  with this definition of plugData:
>  ctypedef struct plugData:
>         plugDefinition *plug
>         char *ident
>         void *data
>
>  generates an error:
>  "Object of type 'plugData' has no attribute 'ident'"
>
>  What's wrong with this?
>
>  Thanks
>
> Johannes
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>
>
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Mon May 12 19:05:20 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 12 May 2008 10:05:20 -0700
Subject: [Cython] defining module constants
In-Reply-To: <20080512010743.5a583b3c.jim-crow@rambler.ru>
References: <20080512010743.5a583b3c.jim-crow@rambler.ru>
Message-ID: <07C7DAC6-38D0-43DE-8145-A82366BBD2F2@math.washington.edu>

Could you clarify? I did add something that lets you do

cdef public enum foo:
     a
     b
     c = 10
     ...

and theses will be exported into the public module dict.


On May 11, 2008, at 11:07 AM, Anatoly A. Kazantsev wrote:

> Hello.
>
> I have same problem.
> Will be any answers or advices? :-)
>
> -- 
> Protect your digital freedom and privacy, eliminate DRM, learn more at
> http://www.defectivebydesign.org/what_is_drm
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080512/fbe6445f/attachment.pgp 

From robertwb at math.washington.edu  Mon May 12 19:30:58 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 12 May 2008 10:30:58 -0700
Subject: [Cython] Python type optimizations (NumPy GSoC-related)
In-Reply-To: <48275E71.2030109@student.matnat.uio.no>
References: <3293365801.1640468@smtp.netcom.no>
	<48275E71.2030109@student.matnat.uio.no>
Message-ID: <3169E86B-8A2F-4763-A158-63F142E05523@math.washington.edu>

On May 11, 2008, at 2:00 PM, Dag Sverre Seljebotn wrote:

> Ok, been thinking some more. To sum up, what we're after is some  
> way of
> having the following work (A is a class with a value field, printme  
> is a
> method somehow selected for inlining and optimization for Cython):
>
> cdef A a = get_a()
> a.printme()
> a.printme()
>
> turns into
>
> cdef A a = get_a()
> print a.value
> print a.value
>
> , while this:
>
> cdef A(value=23) a = get_a()
> a.printme()
> a.printme()
>
> would be turned into
>
> cdef A a = get_a()
> if a.value != 23: raise TypeError(...) # line (a)
> print 23 # line (b)
> print 23
>
> (This is the final result, not saying it is a recipe for how it  
> happens
> magically).
>
> So, what is happening is that we want to set up some assumptions about
> an object (without actually changing the object); and have the code
> after generated by making use of those assumptions.

Yes, this is exactly what I was saying, and I think it'll be very  
easy to implement. sorry if it wasn't totally clear.

>
> I have some seperate proposals building further on Robert's ideas  
> here.
>
> 1) An explicit method for the assumptions bit. I.e:
>
> cdef class A:
>      cdef int value
>
>      cdef __assume__(self, int value):
>          if self.value != value: raise AssumptionError(...)
>
> The argument list to this function is basically a more explicit
> declaration of type arguments. The reason I want this, so explicitly
> (rather than using a more general __coerce__ which must probably  
> also be
> added) is that it hints strongly to the writer of the class A to treat
> self.value as "const". Once an assumption is made by having __assume__
> called, you cannot "back out" and change self.value. This seem to make
> it explicit that value should be treated as a const after  
> construction;
> and it leaves the contract for the name, number and types of "type
> arguments" on the class creator side rather than the caller side.
>
> In addition to just checking assumptions, this method can also put
> general constraints on the arguments (i.e., "if value < 0: raise
> ValueError(...)").
>
> Important: This does *not* address how one can make optimizations  
> later
> on, line (b), it is simply a way to insert the assumption line, line
> (a). Also, __assume__ can simply be called in the normal way.
>
> 2) With this in place, it seems ok to follow Robert's proposal and
> automatically treat fields having the same name as type arguments as
> known compile-time. The parameter list to __assume__ restricts which
> fields can be used.
>
> I am still thinking about something more explicit like
>
> cdef class A:
>      cdef int value
>      cdef int not_possible_typearg
>
>      __typearguments__ = ["value", "compiletimeonlyarg"]
>
> but it doesn't seem strictly necesarry as the argument list to
> __assume__ can serve the same role; and in a possibly more dynamic  
> way, ie

I'm generally opposed to adding extra keywords like  
__typearguments__, that's why I've been writing A(len=11) rather than  
A(11).

> cdef A(constant_alpha=True, alpha=4) a = x # ok
> print a.alpha # ok to optimize...
>
> cdef A(constant_alpha=False, alpha=4) a = x # __assume__ might  
> raise...
> # .. run-time error because of invalid combination,
> # .. so code below will never run, which is lucky because
> # .. alpha is now for some reason changing constantly.
> print a.alpha # Will be optimized but never run
>
> 3) I still want to throw __init__ into the mix. The main reason: For
> type inference, it would be nice if
>
> A = ndarray(shape=(4, 4), dtype=float64, buffer=arr)
>
> would automatically (because type arguments are somehow interlinked  
> with
> constructor arguments) be type-inferred to
>
> cdef ndarray(shape=(4,4), dtype=float64) A
> A = ndarray(shape=(4,4), dtype=float64, buffer=arr)
>
> If so, keeping () for the type argument syntax would also make more
> sense and be less confusing.
>
> Perhaps all that is needed is this rule at the type-inference stage:
>
>   * A call to a known constructor (of a type which is a candidate for
> typing) obviously leads to typing it explicitly
>   * At the same time, any arguments passed to the constructor is