From vinjvinj at gmail.com  Fri Oct  1 20:17:35 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Fri, 1 Oct 2010 14:17:35 -0400
Subject: [Cython] cython compiler crash
Message-ID: <AANLkTimSZYTYMC0b4TFJdEKYq_NejGmSxS1asuAapXcW@mail.gmail.com>

Last night while trying out the following cython example:

    http://dpinte.wordpress.com/2010/04/22/interfacing-ta-lib-with-cython/

I got the following crash in the cython compiler (cython version 0.13):

Compiler crash traceback from this point on:
  File "Visitor.py", line 179, in
Cython.Compiler.Visitor.TreeVisitor.visitchild
(/tmp/easy_install-fEGaFj/Cython-0.13/Cython/Compiler/Visitor.c:3407)
  File "Visitor.py", line 28, in
Cython.Compiler.Visitor.BasicVisitor.visit
(/tmp/easy_install-fEGaFj/Cython-0.13/Cython/Compiler/Visitor.c:1178)
  File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/
Cython/Compiler/Buffer.py",
line 106, in visit_FuncDefNode
    self.handle_scope(node, node.local_scope)
  File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/
Cython/Compiler/Buffer.py",
line 61, in handle_scope
    if entry.type.dtype.is_ptr:
AttributeError: 'NoneType' object has no attribute 'is_ptr'
building 'talib' extension

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC
-I/usr/local/lib/python2.6/dist-packages/numpy/core/include
-I/usr/local/include/ta-lib/ -I/usr/include/python2.6 -c talib.c -o
build/temp.linux-x86_64-2.6/talib.o

From vinjvinj at gmail.com  Fri Oct  1 20:18:54 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Fri, 1 Oct 2010 14:18:54 -0400
Subject: [Cython] cython types for numpy array types
Message-ID: <AANLkTiksQrJKiJLzFrToMcVPOKvrRL7oafbtX+XjaAp1@mail.gmail.com>

I'm trying to get my head around how to let cython know the type of
the numpy array. I've seen a few examples with:

from cimport numpy as np
p.ndarray[np.float_t, mdim=1]

while numpy's documentation has float64 float32, from the following
link. Is there a mapping between numpy types and cython numpy types?

    http://docs.scipy.org/doc/numpy/user/basics.types.html

Thanks,

Vineet

From matej at laitl.cz  Fri Oct  1 20:47:58 2010
From: matej at laitl.cz (=?utf-8?q?Mat=C4=9Bj_Laitl?=)
Date: Fri, 1 Oct 2010 20:47:58 +0200
Subject: [Cython] cython types for numpy array types
In-Reply-To: <AANLkTiksQrJKiJLzFrToMcVPOKvrRL7oafbtX+XjaAp1@mail.gmail.com>
References: <AANLkTiksQrJKiJLzFrToMcVPOKvrRL7oafbtX+XjaAp1@mail.gmail.com>
Message-ID: <201010012047.59370.matej@laitl.cz>

On 1.?10.?2010 Vineet Jain wrote:
> I'm trying to get my head around how to let cython know the type of
> the numpy array. I've seen a few examples with:
> 
> from cimport numpy as np
> p.ndarray[np.float_t, mdim=1]
> 
> while numpy's documentation has float64 float32, from the following
> link. Is there a mapping between numpy types and cython numpy types?

Yes, look at numpy.pxd and ctypedefs there (distributed with cython, for me 
for example in /usr/lib/python2.6/site-packages/Cython/Includes/numpy.pxd)

	Mat?j

From dalcinl at gmail.com  Fri Oct  1 22:36:40 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 1 Oct 2010 17:36:40 -0300
Subject: [Cython] cython compiler crash
In-Reply-To: <AANLkTimSZYTYMC0b4TFJdEKYq_NejGmSxS1asuAapXcW@mail.gmail.com>
References: <AANLkTimSZYTYMC0b4TFJdEKYq_NejGmSxS1asuAapXcW@mail.gmail.com>
Message-ID: <AANLkTi=3ZBhoOAAfpxCPiUjTSHVFyjNvX+uUR5bVaVN5@mail.gmail.com>

On 1 October 2010 15:17, Vineet Jain <vinjvinj at gmail.com> wrote:
> Last night while trying out the following cython example:
>
> ? ?http://dpinte.wordpress.com/2010/04/22/interfacing-ta-lib-with-cython/
>
> I got the following crash in the cython compiler (cython version 0.13):
>
> Compiler crash traceback from this point on:
> ?File "Visitor.py", line 179, in
> Cython.Compiler.Visitor.TreeVisitor.visitchild
> (/tmp/easy_install-fEGaFj/Cython-0.13/Cython/Compiler/Visitor.c:3407)
> ?File "Visitor.py", line 28, in
> Cython.Compiler.Visitor.BasicVisitor.visit
> (/tmp/easy_install-fEGaFj/Cython-0.13/Cython/Compiler/Visitor.c:1178)
> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/
> Cython/Compiler/Buffer.py",
> line 106, in visit_FuncDefNode
> ? ?self.handle_scope(node, node.local_scope)
> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/
> Cython/Compiler/Buffer.py",
> line 61, in handle_scope
> ? ?if entry.type.dtype.is_ptr:
> AttributeError: 'NoneType' object has no attribute 'is_ptr'
> building 'talib' extension
>
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC
> -I/usr/local/lib/python2.6/dist-packages/numpy/core/include
> -I/usr/local/include/ta-lib/ -I/usr/include/python2.6 -c talib.c -o
> build/temp.linux-x86_64-2.6/talib.o
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Could you send the full code attached to my private mail?


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From vinjvinj at gmail.com  Fri Oct  1 23:00:18 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Fri, 1 Oct 2010 17:00:18 -0400
Subject: [Cython] cython compiler crash
In-Reply-To: <AANLkTi=3ZBhoOAAfpxCPiUjTSHVFyjNvX+uUR5bVaVN5@mail.gmail.com>
References: <AANLkTimSZYTYMC0b4TFJdEKYq_NejGmSxS1asuAapXcW@mail.gmail.com>
	<AANLkTi=3ZBhoOAAfpxCPiUjTSHVFyjNvX+uUR5bVaVN5@mail.gmail.com>
Message-ID: <AANLkTi=igK=asfse1R-UoGdNQswTpADKd=1v42BwwN0N@mail.gmail.com>

Problem was a typo:

cdef np.ndarray[np, ndim=1] outreal = numpy.zeros_like(inreal)

should have been:

cdef np.ndarray[np.float_t, ndim=1] outreal = numpy.zeros_like(inreal)

Which caused the compiler to crash.


On Fri, Oct 1, 2010 at 4:36 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 1 October 2010 15:17, Vineet Jain <vinjvinj at gmail.com> wrote:
>> Last night while trying out the following cython example:
>>
>> ? ?http://dpinte.wordpress.com/2010/04/22/interfacing-ta-lib-with-cython/
>>
>> I got the following crash in the cython compiler (cython version 0.13):
>>
>> Compiler crash traceback from this point on:
>> ?File "Visitor.py", line 179, in
>> Cython.Compiler.Visitor.TreeVisitor.visitchild
>> (/tmp/easy_install-fEGaFj/Cython-0.13/Cython/Compiler/Visitor.c:3407)
>> ?File "Visitor.py", line 28, in
>> Cython.Compiler.Visitor.BasicVisitor.visit
>> (/tmp/easy_install-fEGaFj/Cython-0.13/Cython/Compiler/Visitor.c:1178)
>> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/
>> Cython/Compiler/Buffer.py",
>> line 106, in visit_FuncDefNode
>> ? ?self.handle_scope(node, node.local_scope)
>> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/
>> Cython/Compiler/Buffer.py",
>> line 61, in handle_scope
>> ? ?if entry.type.dtype.is_ptr:
>> AttributeError: 'NoneType' object has no attribute 'is_ptr'
>> building 'talib' extension
>>
>> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
>> -Wstrict-prototypes -fPIC
>> -I/usr/local/lib/python2.6/dist-packages/numpy/core/include
>> -I/usr/local/include/ta-lib/ -I/usr/include/python2.6 -c talib.c -o
>> build/temp.linux-x86_64-2.6/talib.o
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> Could you send the full code attached to my private mail?
>
>
> --
> Lisandro Dalcin
> ---------------
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dalcinl at gmail.com  Fri Oct  1 23:05:40 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 1 Oct 2010 18:05:40 -0300
Subject: [Cython] cython compiler crash
In-Reply-To: <AANLkTi=igK=asfse1R-UoGdNQswTpADKd=1v42BwwN0N@mail.gmail.com>
References: <AANLkTimSZYTYMC0b4TFJdEKYq_NejGmSxS1asuAapXcW@mail.gmail.com>
	<AANLkTi=3ZBhoOAAfpxCPiUjTSHVFyjNvX+uUR5bVaVN5@mail.gmail.com>
	<AANLkTi=igK=asfse1R-UoGdNQswTpADKd=1v42BwwN0N@mail.gmail.com>
Message-ID: <AANLkTik8C8voogU0kfKNfwEOyh_wPakCCQa+O0=X_e1Y@mail.gmail.com>

On 1 October 2010 18:00, Vineet Jain <vinjvinj at gmail.com> wrote:
> Problem was a typo:
>
> cdef np.ndarray[np, ndim=1] outreal = numpy.zeros_like(inreal)
>
> should have been:
>
> cdef np.ndarray[np.float_t, ndim=1] outreal = numpy.zeros_like(inreal)
>
> Which caused the compiler to crash.
>

OK, now all is clear. Anyway, Cython should not crash. We should fix this.



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Sun Oct  3 08:51:59 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 2 Oct 2010 23:51:59 -0700
Subject: [Cython] cython compiler crash
In-Reply-To: <AANLkTik8C8voogU0kfKNfwEOyh_wPakCCQa+O0=X_e1Y@mail.gmail.com>
References: <AANLkTimSZYTYMC0b4TFJdEKYq_NejGmSxS1asuAapXcW@mail.gmail.com>
	<AANLkTi=3ZBhoOAAfpxCPiUjTSHVFyjNvX+uUR5bVaVN5@mail.gmail.com>
	<AANLkTi=igK=asfse1R-UoGdNQswTpADKd=1v42BwwN0N@mail.gmail.com>
	<AANLkTik8C8voogU0kfKNfwEOyh_wPakCCQa+O0=X_e1Y@mail.gmail.com>
Message-ID: <AANLkTikKa71yY9Tf0zCNYztYt4fcun4BQVDh5yseXqed@mail.gmail.com>

On Fri, Oct 1, 2010 at 2:05 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 1 October 2010 18:00, Vineet Jain <vinjvinj at gmail.com> wrote:
>> Problem was a typo:
>>
>> cdef np.ndarray[np, ndim=1] outreal = numpy.zeros_like(inreal)
>>
>> should have been:
>>
>> cdef np.ndarray[np.float_t, ndim=1] outreal = numpy.zeros_like(inreal)
>>
>> Which caused the compiler to crash.
>>
>
> OK, now all is clear. Anyway, Cython should not crash. We should fix this.

Fixed. http://trac.cython.org/cython_trac/ticket/579

When trying to write a test case, I noticed that buffers are not very
good at continuing once they hit an error... This should be improved.

- Robert

From vinjvinj at gmail.com  Sun Oct  3 15:14:43 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Sun, 3 Oct 2010 09:14:43 -0400
Subject: [Cython] Cython distutils does not recompile files if a pxd file
	changes.
Message-ID: <AANLkTi=uwYGnc81fRNeZDa4qrQa=uievvbgBUG=AnfGv@mail.gmail.com>

Not a big deal, but thought I would just mention it. If a pxd file
changes and you run setup.py build it will not rebuild the package.
You have to change the corresponding pyx file for the module to be
rebuilt.

Vineet

From stefan at sun.ac.za  Sun Oct  3 20:29:37 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 3 Oct 2010 20:29:37 +0200
Subject: [Cython] Unroll power expressions?
Message-ID: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>

Hi all,

I got bitten tonight by the following behaviour:

x**4

translates to

  __pyx_r = (pow(__pyx_v_x, 4.0);

which, in my case, is roughly 10x slower than x*x*x*x.

Maybe a neat feature for the new type inferencer would be to
automatically expand the power operator for integer powers below a
certain threshold?

Regards
St?fan

From matej at laitl.cz  Sun Oct  3 20:59:21 2010
From: matej at laitl.cz (=?utf-8?q?Mat=C4=9Bj_Laitl?=)
Date: Sun, 3 Oct 2010 20:59:21 +0200
Subject: [Cython] Cython distutils does not recompile files if a pxd
	file changes.
In-Reply-To: <AANLkTi=uwYGnc81fRNeZDa4qrQa=uievvbgBUG=AnfGv@mail.gmail.com>
References: <AANLkTi=uwYGnc81fRNeZDa4qrQa=uievvbgBUG=AnfGv@mail.gmail.com>
Message-ID: <201010032059.21744.matej@laitl.cz>

On 3.?10.?2010 Vineet Jain wrote:
> Not a big deal, but thought I would just mention it. If a pxd file
> changes and you run setup.py build it will not rebuild the package.
> You have to change the corresponding pyx file for the module to be
> rebuilt.

+1

The reason behind this is that cython.Distutils' build_ext does it's own "is-
source-newer-than-target" checks. It doesn't parse pyx file for dependencies, 
but it relies on you specifying all dependencies in Extension source files 
declaration, e.g:

setup(... ext_modules(Extension('module', ['module.pyx', 'module.pxd', 
'dependency.pxd'])))

But I also find it suboptimal. Cython distutils can be changed to use cython 
compiler to do dependency mtime checking, I may post a patch in a couple
of days.

	Mat?j

From stefan at sun.ac.za  Sun Oct  3 21:01:52 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 3 Oct 2010 21:01:52 +0200
Subject: [Cython] Equivalent? @locals+pxd vs defined in file
Message-ID: <AANLkTi=f9SoGBDymnb=ntTvyUzRv0CSGbKUpEq4B+2Vk@mail.gmail.com>

Hi all,

I have a snippet borrowed from Dag's Notur talk, which I've annotated
in two ways:

a) Using pure Python decorators, compiler directives and a .pxd.
b) Using "cdef" declarations inside a .pyx file.

I have the feeling these should generate equivalent results, but
approach (a) is a bit slower.  Do you have an explanation?

I attach the source files, and the annotated versions are here:

http://mentat.za.net/refer/integrate_hints.html
http://mentat.za.net/refer/integrate_cy.html

Regards
St?fan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: integrate_hints.py
Type: text/x-python
Size: 655 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101003/f82fcc07/attachment.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: integrate_hints.pxd
Type: application/octet-stream
Size: 23 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101003/f82fcc07/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: integrate_cy.pyx
Type: application/octet-stream
Size: 316 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101003/f82fcc07/attachment-0001.obj 

From matej at laitl.cz  Sun Oct  3 21:17:49 2010
From: matej at laitl.cz (=?utf-8?q?Mat=C4=9Bj_Laitl?=)
Date: Sun, 3 Oct 2010 21:17:49 +0200
Subject: [Cython] Equivalent? @locals+pxd vs defined in file
In-Reply-To: <AANLkTi=f9SoGBDymnb=ntTvyUzRv0CSGbKUpEq4B+2Vk@mail.gmail.com>
References: <AANLkTi=f9SoGBDymnb=ntTvyUzRv0CSGbKUpEq4B+2Vk@mail.gmail.com>
Message-ID: <201010032117.50292.matej@laitl.cz>

On 3.?10.?2010 St?fan van der Walt wrote:
> a) Using pure Python decorators, compiler directives and a .pxd.
> b) Using "cdef" declarations inside a .pyx file.
> 
> I have the feeling these should generate equivalent results, but
> approach (a) is a bit slower.  Do you have an explanation?
> 
> I attach the source files, and the annotated versions are here:
> 
> http://mentat.za.net/refer/integrate_hints.html
> http://mentat.za.net/refer/integrate_cy.html

Hey, it seems that cython.float maps to C float in integrate_hints. So you are 
effectively comparing double vs. single precision.

	Mat?j

From stefan at sun.ac.za  Sun Oct  3 23:16:42 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 3 Oct 2010 23:16:42 +0200
Subject: [Cython] Equivalent? @locals+pxd vs defined in file
In-Reply-To: <201010032117.50292.matej@laitl.cz>
References: <AANLkTi=f9SoGBDymnb=ntTvyUzRv0CSGbKUpEq4B+2Vk@mail.gmail.com>
	<201010032117.50292.matej@laitl.cz>
Message-ID: <AANLkTin178oHK0eFcfkFyHSETEWcDCXO02_mQdtizJxs@mail.gmail.com>

On Sun, Oct 3, 2010 at 9:17 PM, Mat?j Laitl <matej at laitl.cz> wrote:
> Hey, it seems that cython.float maps to C float in integrate_hints. So you are
> effectively comparing double vs. single precision.

Thanks, that was the problem!

Regards
St?fan

From stefan_ml at behnel.de  Mon Oct  4 08:24:00 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 04 Oct 2010 08:24:00 +0200
Subject: [Cython] Unroll power expressions?
In-Reply-To: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>
References: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>
Message-ID: <4CA97300.9090701@behnel.de>

St?fan van der Walt, 03.10.2010 20:29:
> I got bitten tonight by the following behaviour:
>
> x**4
>
> translates to
>
>    __pyx_r = (pow(__pyx_v_x, 4.0);

That's not true for C integer types (since at least 0.12 IIRC). They 
translate to multiplication code for x**N, N<=3, and to a loop for N>3.

What type does 'x' have in your code? If it's a FP type, then the above is 
the correct way to do it (although a similar optimisation for constant 
exponents may be worth it). If it's an integer type, please double check 
the C code and provide a complete example.

Stefan

From stefan at sun.ac.za  Mon Oct  4 10:29:36 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 4 Oct 2010 10:29:36 +0200
Subject: [Cython] Unroll power expressions?
In-Reply-To: <4CA97300.9090701@behnel.de>
References: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>
	<4CA97300.9090701@behnel.de>
Message-ID: <AANLkTimZ=v6FzyzX3oWjJUo7kqkBCWh-AJm5qa-kG5v1@mail.gmail.com>

Hi Stefan

On Mon, Oct 4, 2010 at 8:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> St?fan van der Walt, 03.10.2010 20:29:
>>
>> x**4
>>
> the correct way to do it (although a similar optimisation for constant
> exponents may be worth it). If it's an integer type, please double check

Yes, it's a floating point number; such an optimisation would be very useful!

Regards
St?fan

From stefan at sun.ac.za  Mon Oct  4 14:35:19 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 4 Oct 2010 14:35:19 +0200
Subject: [Cython] Unroll power expressions?
In-Reply-To: <4CA97300.9090701@behnel.de>
References: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>
	<4CA97300.9090701@behnel.de>
Message-ID: <AANLkTinceGOB=bDHL5PQ8xmxCLWF9J1HvPtbNOULuXwN@mail.gmail.com>

On Mon, Oct 4, 2010 at 8:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> That's not true for C integer types (since at least 0.12 IIRC). They
> translate to multiplication code for x**N, N<=3, and to a loop for N>3.

This works for type "int", but not cython.int.

Input:

cimport cython

def foo(cython.int x):
    return x**4

Output:

__pyx_t_1 = PyNumber_Power(((PyObject *)__pyx_v_x), __pyx_int_4, Py_None);

Does `cython.int` then not point to a C int but to Python int?

Regards
St?fan

From mostawesomedude at gmail.com  Mon Oct  4 19:11:19 2010
From: mostawesomedude at gmail.com (Corbin Simpson)
Date: Mon, 4 Oct 2010 10:11:19 -0700
Subject: [Cython] Unroll power expressions?
In-Reply-To: <AANLkTimZ=v6FzyzX3oWjJUo7kqkBCWh-AJm5qa-kG5v1@mail.gmail.com>
References: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>
	<4CA97300.9090701@behnel.de>
	<AANLkTimZ=v6FzyzX3oWjJUo7kqkBCWh-AJm5qa-kG5v1@mail.gmail.com>
Message-ID: <AANLkTin9xTgFgqNZKpcznkb+5=ck7VX-9DPia8m-ZiwL@mail.gmail.com>

Trying to remember floating-point rules. Isn't repeated multiplication less
precise than pow()? I thought that was why even integer exponents were only
unrollable if things like -ffast-math were set. I might be remembering wrong
though.

Sending from a mobile, pardon the brevity. ~ C.

On Oct 4, 2010 1:30 AM, "St?fan van der Walt" <stefan at sun.ac.za> wrote:
> Hi Stefan
>
> On Mon, Oct 4, 2010 at 8:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> St?fan van der Walt, 03.10.2010 20:29:
>>>
>>> x**4
>>>
>> the correct way to do it (although a similar optimisation for constant
>> exponents may be worth it). If it's an integer type, please double check
>
> Yes, it's a floating point number; such an optimisation would be very
useful!
>
> Regards
> St?fan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101004/a762a50d/attachment.htm 

From dalcinl at gmail.com  Mon Oct  4 19:54:08 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 4 Oct 2010 14:54:08 -0300
Subject: [Cython] c++, type inference, and reference types
Message-ID: <AANLkTimsXbuYzeieiKFBZt09Ke9MoXwn2sESnPqTF7pA@mail.gmail.com>

Please consider this code:

from libcpp.vector cimport vector
v = new vector[double]()
from math import pi
v.push_back(pi)


This generated bad code, more or less:

double &tmp; // bad code, un-init reference!
tmp = PyFloat_AsDouble(pi)
v.push_back(tmp)



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Tue Oct  5 05:43:55 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 4 Oct 2010 20:43:55 -0700
Subject: [Cython] c++, type inference, and reference types
In-Reply-To: <AANLkTimsXbuYzeieiKFBZt09Ke9MoXwn2sESnPqTF7pA@mail.gmail.com>
References: <AANLkTimsXbuYzeieiKFBZt09Ke9MoXwn2sESnPqTF7pA@mail.gmail.com>
Message-ID: <AANLkTi=bLgQr=un=V0SoVX8qKWQGdz17tPPZWNvzAOeq@mail.gmail.com>

On Mon, Oct 4, 2010 at 10:54 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Please consider this code:
>
> from libcpp.vector cimport vector
> v = new vector[double]()
> from math import pi
> v.push_back(pi)
>
>
> This generated bad code, more or less:
>
> double &tmp; // bad code, un-init reference!
> tmp = PyFloat_AsDouble(pi)
> v.push_back(tmp)

I don't think this has anything to do with type inference, but it is bad code...

- Robert

From robertwb at math.washington.edu  Tue Oct  5 06:04:34 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 4 Oct 2010 21:04:34 -0700
Subject: [Cython] c++, type inference, and reference types
In-Reply-To: <AANLkTi=bLgQr=un=V0SoVX8qKWQGdz17tPPZWNvzAOeq@mail.gmail.com>
References: <AANLkTimsXbuYzeieiKFBZt09Ke9MoXwn2sESnPqTF7pA@mail.gmail.com>
	<AANLkTi=bLgQr=un=V0SoVX8qKWQGdz17tPPZWNvzAOeq@mail.gmail.com>
Message-ID: <AANLkTinQfgJQdnB2VpeUt8bD62N_GHHzZa+PYr3pdJEn@mail.gmail.com>

On Mon, Oct 4, 2010 at 8:43 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Mon, Oct 4, 2010 at 10:54 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> Please consider this code:
>>
>> from libcpp.vector cimport vector
>> v = new vector[double]()
>> from math import pi
>> v.push_back(pi)
>>
>>
>> This generated bad code, more or less:
>>
>> double &tmp; // bad code, un-init reference!
>> tmp = PyFloat_AsDouble(pi)
>> v.push_back(tmp)
>
> I don't think this has anything to do with type inference, but it is bad code...
>

http://trac.cython.org/cython_trac/ticket/581

From robertwb at math.washington.edu  Tue Oct  5 06:20:52 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 4 Oct 2010 21:20:52 -0700
Subject: [Cython] Unroll power expressions?
In-Reply-To: <AANLkTinceGOB=bDHL5PQ8xmxCLWF9J1HvPtbNOULuXwN@mail.gmail.com>
References: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>
	<4CA97300.9090701@behnel.de>
	<AANLkTinceGOB=bDHL5PQ8xmxCLWF9J1HvPtbNOULuXwN@mail.gmail.com>
Message-ID: <AANLkTiniT60JvjSV8ai6ArTSzakWhTq4Ez+trkQbE=0K@mail.gmail.com>

2010/10/4 St?fan van der Walt <stefan at sun.ac.za>:
> On Mon, Oct 4, 2010 at 8:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> That's not true for C integer types (since at least 0.12 IIRC). They
>> translate to multiplication code for x**N, N<=3, and to a loop for N>3.
>
> This works for type "int", but not cython.int.
>
> Input:
>
> cimport cython
>
> def foo(cython.int x):
> ? ?return x**4
>
> Output:
>
> __pyx_t_1 = PyNumber_Power(((PyObject *)__pyx_v_x), __pyx_int_4, Py_None);
>
> Does `cython.int` then not point to a C int but to Python int?

Hmm... that is the case, but almost certainly not what we want, and
not consistant with the other uses of cython.int. Did anyone
intentionally implement this behavior?

From robertwb at math.washington.edu  Tue Oct  5 06:27:06 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 4 Oct 2010 21:27:06 -0700
Subject: [Cython] Unroll power expressions?
In-Reply-To: <AANLkTin9xTgFgqNZKpcznkb+5=ck7VX-9DPia8m-ZiwL@mail.gmail.com>
References: <AANLkTikor_G=oh++0E9FzSZYb5ychKWGaZdCdNz1KWea@mail.gmail.com>
	<4CA97300.9090701@behnel.de>
	<AANLkTimZ=v6FzyzX3oWjJUo7kqkBCWh-AJm5qa-kG5v1@mail.gmail.com>
	<AANLkTin9xTgFgqNZKpcznkb+5=ck7VX-9DPia8m-ZiwL@mail.gmail.com>
Message-ID: <AANLkTi=e-5+SmxUy6domiaXjwd-RONnWinowNjYFCy6m@mail.gmail.com>

On Mon, Oct 4, 2010 at 10:11 AM, Corbin Simpson
<mostawesomedude at gmail.com> wrote:
> Trying to remember floating-point rules. Isn't repeated multiplication less
> precise than pow()? I thought that was why even integer exponents were only
> unrollable if things like -ffast-math were set. I might be remembering wrong
> though.

Makes sense, and certainly agrees with my simple experiments with gcc
(-ffast-math does unroll them for me). Another option is
-funsafe-math-optimizations (which does the pow unrolling, but does
not disable as much of the error checking.

> Sending from a mobile, pardon the brevity. ~ C.
>
> On Oct 4, 2010 1:30 AM, "St?fan van der Walt" <stefan at sun.ac.za> wrote:
>> Hi Stefan
>>
>> On Mon, Oct 4, 2010 at 8:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> St?fan van der Walt, 03.10.2010 20:29:
>>>>
>>>> x**4
>>>>
>>> the correct way to do it (although a similar optimisation for constant
>>> exponents may be worth it). If it's an integer type, please double check
>>
>> Yes, it's a floating point number; such an optimisation would be very
>> useful!
>>
>> Regards
>> St?fan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From robertwb at math.washington.edu  Tue Oct  5 06:33:29 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 4 Oct 2010 21:33:29 -0700
Subject: [Cython] Cython distutils does not recompile files if a pxd
	file changes.
In-Reply-To: <201010032059.21744.matej@laitl.cz>
References: <AANLkTi=uwYGnc81fRNeZDa4qrQa=uievvbgBUG=AnfGv@mail.gmail.com>
	<201010032059.21744.matej@laitl.cz>
Message-ID: <AANLkTim2zGfwwQ-Zo3KODLCkmjdpcYjPk7ZD_2_pCg4x@mail.gmail.com>

On Sun, Oct 3, 2010 at 11:59 AM, Mat?j Laitl <matej at laitl.cz> wrote:
> On 3.?10.?2010 Vineet Jain wrote:
>> Not a big deal, but thought I would just mention it. If a pxd file
>> changes and you run setup.py build it will not rebuild the package.
>> You have to change the corresponding pyx file for the module to be
>> rebuilt.
>
> +1
>
> The reason behind this is that cython.Distutils' build_ext does it's own "is-
> source-newer-than-target" checks. It doesn't parse pyx file for dependencies,
> but it relies on you specifying all dependencies in Extension source files
> declaration, e.g:
>
> setup(... ext_modules(Extension('module', ['module.pyx', 'module.pxd',
> 'dependency.pxd'])))
>
> But I also find it suboptimal.

For sure, especially for a project like Sage. (We had to resort to
managing our dependancies manually...) This was one of the goals of
http://wiki.cython.org/enhancements/distutils_preprocessing .

> Cython distutils can be changed to use cython
> compiler to do dependency mtime checking, I may post a patch in a couple
> of days.

That would be really cool. Note that if you don't amortize this across
multiple files (by doing the dependency resolution for each one
separately) you may find that O(n^2) doesn't grow very nicely :).

- Robert

From dalcinl at gmail.com  Wed Oct  6 16:46:41 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 6 Oct 2010 11:46:41 -0300
Subject: [Cython] Asking for advice: exception handling with nogil
In-Reply-To: <AANLkTinmq+ktvJyVSA5DTEHwodV89c2UOV8QdggO7+Lw@mail.gmail.com>
References: <AANLkTimqGMZR6YEFvJPkaPRCEBHgR+ObEV_5b2ykZjr3@mail.gmail.com>
	<AANLkTimgNw4N8o7M7PsVdT+Tbrnz2N7Kp9oNyCjsXxSt@mail.gmail.com>
	<AANLkTinmq+ktvJyVSA5DTEHwodV89c2UOV8QdggO7+Lw@mail.gmail.com>
Message-ID: <AANLkTi=i74zfv+j5p2JFmnopmEEjArp3dLJgRna3e4yg@mail.gmail.com>

On 28 September 2010 13:33, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 28 September 2010 13:02, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Mon, Sep 27, 2010 at 1:55 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>> consider this code:
>>>
>>> cdef void foo() except * with gil:
>>> ? ?raise ValueError
>>>
>>> cdef int bar() except ? -1 with gil:
>>> ? ?raise ValueError
>>>
>>> with nogil:
>>> ? ?foo()
>>>
>>> with nogil:
>>> ? ?bar()
>>>
>>>
>>> Currently, Cython generates segfaulting code because in both cases it
>>> is using PyErr_Occurred() without re-acquiring the GIL. I think the
>>> easiest solution would be to use a utility code wrapping around
>>> PyErr_Occurred() and implemented with the PyGILState_XXX API's, more
>>> or less like below:
>>>
>>> int __Pyx_PyErr_Occurred_WithGIL()
>>> {
>>> int err;
>>> _save = PyGILState_Ensure();
>>> err = !!PyErr_Occurred(); /* note: PyErr_Occurred() returns PyObject* ?*/
>>> PyGILState_Release(_save);
>>> return err;
>>> }
>>>
>>> Of course, this function is going to be used ONLY when calling C
>>> functions within nogil blocks.
>>>
>>> Would such fix be fine? I would prefer to use the Py_[UN]BLOCK_THREAD
>>> macros, but in such case the fix is not trivial for me and I do not
>>> want to make a mess...
>>>
>>
>> Yes, I would be fine with that for a fix.
>>
>
> Still not sure about it... We also have to handle this:
> http://bugs.python.org/issue9972
>
> And I would like to change GILStatNode&GILExitNode:
>
> {PyThreadState *_save;
> Py_UNBLOCK_THREADS
> ....
> Py_BLOCK_THREADS}
>
> with these macros:
>
> Py_BEGIN_ALLOW_THREADS
> ...
> Py_BEGIN_ALLOW_THREADS
>
>

Fixed: http://hg.cython.org/cython-devel/rev/812f81b23dc8

@Stephane: Sorry, I forgot to give you credits in the commit message.

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Sat Oct  9 08:21:38 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 09 Oct 2010 08:21:38 +0200
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <87eic0kvxu.fsf_-_@vostro.rath.org>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>
	<87eic0kvxu.fsf_-_@vostro.rath.org>
Message-ID: <4CB009F2.9090708@behnel.de>

Nikolaus Rath, 09.10.2010 03:15:
> It seems that Cython translates the line
>
>          if ent.d_name in (b'.', b'..'):
>
> as well as
>
>          if ent.d_name == b'.' or ent.d_name == '..':
>
> into pointer comparison rather than a strcmp():
>
>      if (!(__pyx_v_ent->d_name == __pyx_k_1)) {
>        __pyx_t_3 = (__pyx_v_ent->d_name == __pyx_k_2);
>      } else {
>        __pyx_t_3 = (__pyx_v_ent->d_name == __pyx_k_1);
>      }
>      if (__pyx_t_3) {
>
> (__pyx_k_2 and __pyx_k_1 are "." and "00" respectively).

That's not explicitly intended but a side effect of a couple of 
implementation details and optimisations in Cython. The main problem here 
is that Cython's byte strings start off as char* values, and "==" on 
pointers is the same as "is", i.e. it compares the pointers.

I think it's ok to consider this specific case a bug. If a user wants 
pointer comparison, "is" is the most explicit operator w.r.t. Python 
semantics. Using strcmp() for the "==" operator on char* values makes sense 
to me. We special case char* all over the place, so this is just another 
exception.

So, for char* values, "s1 == s2" should translate into

    (s1 == s2) || (
       (s1 != NULL) && (s2 != NULL) && (strcmp(s1, s2) == 0))

Note that this allows both pointers to be NULL in the success case. I think 
it makes sense to include this.

This change certainly introduces new corner cases, though, as "ptr == ptr" 
means "ptr is ptr" for everything else. So this will surprise at least some 
users and also break existing code. But I think it's at least mostly 
obvious to the average Python developer what the right fix is.

One step further: Would it be worth making "is" the correct operator for 
pointers in the long run? I.e. discourage the use of "==" for pointers and 
eventually make it an error? Personally, I often feel an itch in the back 
of my head when I write "ptr == ptr" and I know that what I actually mean 
is "ptr is ptr". I still write it most of the time because I find "==" more 
quickly readable than the chain of words involving "is", which involves 
text reading instead of just operator spotting.

Opinions?

Stefan

From stefan_ml at behnel.de  Sat Oct  9 13:25:18 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 09 Oct 2010 13:25:18 +0200
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <4CB009F2.9090708@behnel.de>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>
	<87eic0kvxu.fsf_-_@vostro.rath.org> <4CB009F2.9090708@behnel.de>
Message-ID: <4CB0511E.5000203@behnel.de>

Stefan Behnel, 09.10.2010 08:21:
> Nikolaus Rath, 09.10.2010 03:15:
>> It seems that Cython translates the line
>>
>> if ent.d_name in (b'.', b'..'):
>>
>> as well as
>>
>> if ent.d_name == b'.' or ent.d_name == '..':
>>
>> into pointer comparison rather than a strcmp():
>>
>> if (!(__pyx_v_ent->d_name == __pyx_k_1)) {
>> __pyx_t_3 = (__pyx_v_ent->d_name == __pyx_k_2);
>> } else {
>> __pyx_t_3 = (__pyx_v_ent->d_name == __pyx_k_1);
>> }
>> if (__pyx_t_3) {
>>
>> (__pyx_k_2 and __pyx_k_1 are "." and "00" respectively).
>
> That's not explicitly intended but a side effect of a couple of
> implementation details and optimisations in Cython. The main problem
> here is that Cython's byte strings start off as char* values, and "=="
> on pointers is the same as "is", i.e. it compares the pointers.
>
> I think it's ok to consider this specific case a bug. If a user wants
> pointer comparison, "is" is the most explicit operator w.r.t. Python
> semantics. Using strcmp() for the "==" operator on char* values makes
> sense to me. We special case char* all over the place, so this is just
> another exception.
>
> So, for char* values, "s1 == s2" should translate into
>
> (s1 == s2) || (
>     (s1 != NULL) && (s2 != NULL) && (strcmp(s1, s2) == 0))
>
> Note that this allows both pointers to be NULL in the success case. I
> think it makes sense to include this.
>
> This change certainly introduces new corner cases, though, as "ptr ==
> ptr" means "ptr is ptr" for everything else. So this will surprise at
> least some users and also break existing code. But I think it's at least
> mostly obvious to the average Python developer what the right fix is.
>
> One step further: Would it be worth making "is" the correct operator for
> pointers in the long run? I.e. discourage the use of "==" for pointers
> and eventually make it an error? Personally, I often feel an itch in the
> back of my head when I write "ptr == ptr" and I know that what I
> actually mean is "ptr is ptr". I still write it most of the time because
> I find "==" more quickly readable than the chain of words involving
> "is", which involves text reading instead of just operator spotting.

http://trac.cython.org/cython_trac/ticket/582

Stefan

From stefan_ml at behnel.de  Sat Oct  9 14:04:16 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 09 Oct 2010 14:04:16 +0200
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <AANLkTikTcyecXWf-vhG1sDdxjyHrPRMhbiWaTPte5=y6@mail.gmail.com>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>
	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>
	<87eic0kvxu.fsf_-_@vostro.rath.org> <4CB009F2.9090708@behnel.de>
	<AANLkTikTcyecXWf-vhG1sDdxjyHrPRMhbiWaTPte5=y6@mail.gmail.com>
Message-ID: <4CB05A40.5010809@behnel.de>

Robert Bradshaw, 09.10.2010 13:21:
> On Fri, Oct 8, 2010 at 11:21 PM, Stefan Behnel wrote:
>> So, for char* values, "s1 == s2" should translate into
>>
>>    (s1 == s2) || (
>>       (s1 != NULL) && (s2 != NULL) && (strcmp(s1, s2) == 0))
>>
>> Note that this allows both pointers to be NULL in the success case. I think
>> it makes sense to include this.
>>
>> This change certainly introduces new corner cases, though, as "ptr == ptr"
>> means "ptr is ptr" for everything else. So this will surprise at least some
>> users and also break existing code. But I think it's at least mostly obvious
>> to the average Python developer what the right fix is.
>
> +0.5 to this. If not, it would at least be worth a warning.

Thinking about this some more, "==" isn't the only operator where this 
makes sense. Basically, all comparison operators (also <, <=, >, >=) should 
potentially behave the same on bytes objects and char* objects. However, 
for everything but "==", there isn't a simple operator replacement like 
"is" that explicitly compares the pointers if the need arises, but a 
pointer cast to <void*> should do the trick.

I personally think that the main use case for comparison operators on char* 
values really *is* the comparison of the string values and not the 
pointers. So, would it be acceptable to give all comparison operators their 
Python bytes string semantics when applied to C string types? That would 
give us the following implementations:

== : (s1 == s2) || ((s1 != NULL) && (s2 != NULL) && (strcmp(s1, s2) == 0))

!= : (s1 != s2) && ((s1 == NULL) || (s2 == NULL) || (strcmp(s1, s2) != 0))

<,<=,>,>= : (s1 != NULL) && (s2 != NULL) && (strcmp(s1, s2)  [<=>]  0)

Note how NULL is allowed for == and !=, but disallowed for < and >. 
Actually, Python 2 semantics suggest that NULL is smaller than anything, 
and Python 3 semantics suggest raising a TypeError for NULL values. We 
could potentially emulate that. The Python 2 semantics would give this:

< : (s2 != NULL) && ((s1 == NULL) || (strcmp(s1,s2) < 0))

<= : (s1 == NULL) || ((s2 != NULL) && (strcmp(s1,s2) <= 0))

 > : (s1 != NULL) && ((s2 == NULL) || (strcmp(s1,s2) > 0))

 >= : (s2 == NULL) || ((s1 != NULL) && (strcmp(s1,s2) >= 0))

And the Python 3 semantics would simply raise a TypeError if one of the two 
values is NULL and otherwise use strcmp(). We could support both depending 
on the -3 switch, at the cost of disallowing the Python 3 semantics in 
nogil sections.

What do you think?

If we go that route, I think it might be worth waiting for a 0.14 release.

Stefan

From dagss at student.matnat.uio.no  Sat Oct  9 16:22:38 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 09 Oct 2010 16:22:38 +0200
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <4CB05A40.5010809@behnel.de>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>	<87eic0kvxu.fsf_-_@vostro.rath.org>
	<4CB009F2.9090708@behnel.de>	<AANLkTikTcyecXWf-vhG1sDdxjyHrPRMhbiWaTPte5=y6@mail.gmail.com>
	<4CB05A40.5010809@behnel.de>
Message-ID: <4CB07AAE.7020202@student.matnat.uio.no>

On 10/09/2010 02:04 PM, Stefan Behnel wrote:
> Robert Bradshaw, 09.10.2010 13:21:
>    
>> On Fri, Oct 8, 2010 at 11:21 PM, Stefan Behnel wrote:
>>      
>>> So, for char* values, "s1 == s2" should translate into
>>>
>>>     (s1 == s2) || (
>>>        (s1 != NULL)&&  (s2 != NULL)&&  (strcmp(s1, s2) == 0))
>>>
>>> Note that this allows both pointers to be NULL in the success case. I think
>>> it makes sense to include this.
>>>
>>> This change certainly introduces new corner cases, though, as "ptr == ptr"
>>> means "ptr is ptr" for everything else. So this will surprise at least some
>>> users and also break existing code. But I think it's at least mostly obvious
>>> to the average Python developer what the right fix is.
>>>        
>> +0.5 to this. If not, it would at least be worth a warning.
>>      
> Thinking about this some more, "==" isn't the only operator where this
> makes sense. Basically, all comparison operators (also<,<=,>,>=) should
> potentially behave the same on bytes objects and char* objects. However,
> for everything but "==", there isn't a simple operator replacement like
> "is" that explicitly compares the pointers if the need arises, but a
> pointer cast to<void*>  should do the trick.
>    

I agree that < and == should be consistent. But < is used for pointer 
comparison as well in C...

for (i = start; i < end; ++i) ...

In such situations, one can normally use != ("is not"), but it is not 
completely unimaginable that < on pointers can have a use in some algorithm.

I just suggest that this is something to be aware of (put in release 
notes etc.), and suggest

<size_t>i < <size_t>end

as an alternative if one really needs this.

Dag Sverre

From stefan_ml at behnel.de  Sat Oct  9 17:21:38 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 09 Oct 2010 17:21:38 +0200
Subject: [Cython] String comparison
In-Reply-To: <4CB07AAE.7020202@student.matnat.uio.no>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>	<87eic0kvxu.fsf_-_@vostro.rath.org>	<4CB009F2.9090708@behnel.de>	<AANLkTikTcyecXWf-vhG1sDdxjyHrPRMhbiWaTPte5=y6@mail.gmail.com>	<4CB05A40.5010809@behnel.de>
	<4CB07AAE.7020202@student.matnat.uio.no>
Message-ID: <4CB08882.2050009@behnel.de>

Dag Sverre Seljebotn, 09.10.2010 16:22:
> it is not
> completely unimaginable that < on pointers can have a use in some algorithm.

There is a good example in libxml2. For a string interning hash table, it 
allocates string memory in larger blocks and appends strings to it as long 
as they fit into an existing block. That way, checking whether a key is 
owned (i.e. memory managed) by the hash table can be done quickly by 
comparing its start pointer to the memory bounds, rather than (expensively) 
calculating the hash key, looking it up and comparing the pointers with "==".

So there are legitimate use cases for > and < pointer comparisons. I'm just 
suggesting that for string values, it's a lot more common to compare the 
strings than the pointers. I think that Cython's semantics should reflect 
that, as long as there's a usable (i.e. simple enough) way to get the 
alternative behaviour.

I totally agree that such a change is worth noting in the release notes.

Stefan

From craigcitro at gmail.com  Sun Oct 10 09:07:37 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Sun, 10 Oct 2010 00:07:37 -0700
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <4CB009F2.9090708@behnel.de>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>
	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>
	<87eic0kvxu.fsf_-_@vostro.rath.org> <4CB009F2.9090708@behnel.de>
Message-ID: <AANLkTi=rXV+rd1FKBDpse1BqN8wnh3rh4b-Te2h-PnNH@mail.gmail.com>

> I think it's ok to consider this specific case a bug. If a user wants
> pointer comparison, "is" is the most explicit operator w.r.t. Python
> semantics. Using strcmp() for the "==" operator on char* values makes sense
> to me. We special case char* all over the place, so this is just another
> exception.
>

Personally, I'm +1 on considering this a bug, but I'm a big -1
(basically -as_many_as_my_vote_is_worth) on changing the behavior of
comparison operators on some/all pointer types to fix it. As I
understand, the chain of events causing the problem is this:

 - user writes code with == on Python (bytes) objects, expecting
Python semantics
 - we do type inference and replace bytes with char *
 - we run into trouble because == already has a different semantics
for char * objects.

I think the change that we should make is in the second step there,
not the third -- I think we're throwing away valuable type information
by replacing "Python bytes object" with "char *", so that's the place
to fix it. I think we should just introduce a new type which
represents a bytes object that we're modeling with a char *, which we
can call whatever -- maybe "InferredCharStar". It can simply be a
wrapper around a char * -- in fact, it can just *be* char * in the
generated C/C++ code -- but it maintains Python semantics in any
Python statement. We don't even need to expose it to the programmer if
we don't want, since it only really exists at compile-time, though I'd
probably be in favor of exposing it. (I'm a long-time Scheme and
Python programmer that loves types ... I wonder if I should seek help
for that. :P) Then there's also no need to introduce any more special
cases on char * behavior, or by extension any other C types. This
closely mirrors the situation with trying to use a C int for a Python
int without changing the semantics, and probably runs into some of the
other weird and thorny issues.

-cc

From craigcitro at gmail.com  Sun Oct 10 09:07:37 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Sun, 10 Oct 2010 00:07:37 -0700
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <4CB009F2.9090708@behnel.de>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>
	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>
	<87eic0kvxu.fsf_-_@vostro.rath.org> <4CB009F2.9090708@behnel.de>
Message-ID: <AANLkTi=rXV+rd1FKBDpse1BqN8wnh3rh4b-Te2h-PnNH@mail.gmail.com>

> I think it's ok to consider this specific case a bug. If a user wants
> pointer comparison, "is" is the most explicit operator w.r.t. Python
> semantics. Using strcmp() for the "==" operator on char* values makes sense
> to me. We special case char* all over the place, so this is just another
> exception.
>

Personally, I'm +1 on considering this a bug, but I'm a big -1
(basically -as_many_as_my_vote_is_worth) on changing the behavior of
comparison operators on some/all pointer types to fix it. As I
understand, the chain of events causing the problem is this:

 - user writes code with == on Python (bytes) objects, expecting
Python semantics
 - we do type inference and replace bytes with char *
 - we run into trouble because == already has a different semantics
for char * objects.

I think the change that we should make is in the second step there,
not the third -- I think we're throwing away valuable type information
by replacing "Python bytes object" with "char *", so that's the place
to fix it. I think we should just introduce a new type which
represents a bytes object that we're modeling with a char *, which we
can call whatever -- maybe "InferredCharStar". It can simply be a
wrapper around a char * -- in fact, it can just *be* char * in the
generated C/C++ code -- but it maintains Python semantics in any
Python statement. We don't even need to expose it to the programmer if
we don't want, since it only really exists at compile-time, though I'd
probably be in favor of exposing it. (I'm a long-time Scheme and
Python programmer that loves types ... I wonder if I should seek help
for that. :P) Then there's also no need to introduce any more special
cases on char * behavior, or by extension any other C types. This
closely mirrors the situation with trying to use a C int for a Python
int without changing the semantics, and probably runs into some of the
other weird and thorny issues.

-cc

From stefan_ml at behnel.de  Sun Oct 10 16:53:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 10 Oct 2010 16:53:06 +0200
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <AANLkTi=rXV+rd1FKBDpse1BqN8wnh3rh4b-Te2h-PnNH@mail.gmail.com>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>	<87eic0kvxu.fsf_-_@vostro.rath.org>
	<4CB009F2.9090708@behnel.de>
	<AANLkTi=rXV+rd1FKBDpse1BqN8wnh3rh4b-Te2h-PnNH@mail.gmail.com>
Message-ID: <4CB1D352.6080201@behnel.de>

Craig Citro, 10.10.2010 09:07:
>> I think it's ok to consider this specific case a bug. If a user wants
>> pointer comparison, "is" is the most explicit operator w.r.t. Python
>> semantics. Using strcmp() for the "==" operator on char* values makes sense
>> to me. We special case char* all over the place, so this is just another
>> exception.
>
> Personally, I'm +1 on considering this a bug, but I'm a big -1
> (basically -as_many_as_my_vote_is_worth) on changing the behavior of
> comparison operators on some/all pointer types to fix it. As I
> understand, the chain of events causing the problem is this:
>
>   - user writes code with == on Python (bytes) objects, expecting
> Python semantics

Not quite. In the specific example at hand, it's a byte string literal and 
a char* variable. There is no syntactic difference between a char* literal 
and a bytes literal in Cython, so the fact that Cython currently assumes 
char* types for both sides is totally valid and legitimate.


>   - we do type inference and replace bytes with char *

No type inference involved in the example, just plain char* values on both 
sides. Quite the contrary, the current behaviour is that byte string 
literals are in fact char* literals and may eventually get coerced into 
Python bytes objects at compile time or runtime when used in a Python 
context. So we never replace bytes by char*, only char* by bytes.


> [...]Then there's also no need to introduce any more special
> cases on char * behavior, or by extension any other C types. This
> closely mirrors the situation with trying to use a C int for a Python
> int without changing the semantics, and probably runs into some of the
> other weird and thorny issues.

Note that types only have very little intrinsic behaviour in Cython. Most 
of their visible behaviour just results from the way the syntax nodes treat 
them. So it's not like a new type would reduce the need for special cases.

That being said, I bet there are also cases where it would come as a 
surprise that a char* value has Python semantics in some contexts but not 
in others, depending on whether Cython is able to infer a bytes/char* 
duality or not. It's worth taking a deeper look, but I'd want to avoid 
complicating the semantics by introducing a special type for special cases 
that Cython can figure out, thus making it less predictable how the type 
will behave in a certain situation.

Stefan

From stefan_ml at behnel.de  Mon Oct 11 09:44:30 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 11 Oct 2010 09:44:30 +0200
Subject: [Cython] [cython-users] String comparison
In-Reply-To: <AANLkTink+Ketn=jCLwtbTgb9b3=uhe7v+y95mELOO0XF@mail.gmail.com>
References: <87hbh0tfot.fsf@inspiron.ap.columbia.edu>
	<AANLkTikri+T+h8L3Vpg4CPKc+N81ES0C_FCu7SomYrzG@mail.gmail.com>
	<87eic0kvxu.fsf_-_@vostro.rath.org>
	<4CB009F2.9090708@behnel.de> <87ocb3bh2s.fsf@vostro.rath.org>
	<AANLkTikHfXZq2C4aNv+=dqsqec1BKi=UWBocAHC1Ocd_@mail.gmail.com>
	<4CB0926E.2090101@behnel.de>
	<AANLkTink+Ketn=jCLwtbTgb9b3=uhe7v+y95mELOO0XF@mail.gmail.com>
Message-ID: <4CB2C05E.8070602@behnel.de>

Robert Bradshaw, 11.10.2010 07:27:
> On Sat, Oct 9, 2010 at 9:03 AM, Stefan Behnel wrote:
>> My impression is that the dominating use case for comparison of char* values
>> is to compare the strings rather than the pointers. The only truly common
>> use case for pointer comparison (as Dag pointed out) is for loop variables,
>> e.g. when traversing a char* string value byte by byte. Except for this
>> case, I wouldn't be surprised if most new users (especially Python
>> programmers) intuitively expected "==" to compare the string values and "is"
>> to compare the pointers.
>
> On an orthogonal note, I'm surprised you of all people are talking
> about treating char* as strings

I admit that my terminology is a bit sloppy here. In the above, "string" 
refers to what C calls strings, i.e. null terminated byte sequences.


> (and assuming the encoding is a
> null-terminated one like ASCII or UTF-8).

I'm not making any assumptions about encodings. The content may be encoded 
text, base64 encoded binary data or plain binary data. However, I am aware 
that if it contains null bytes, the Python comparison operators will not 
work "correctly", e.g.

     "" == "\0"

will return True with Python operator semantics when bytes literals are 
treated as plain char*.

That's always the case with char* values, though, and Cython's semantics 
ignore that in other places, too. I think it's fine to leave this case to 
the user. If there are (or may be) null bytes involved, you must pass an 
explicit length. Simple rule.


> For all pointers, it might be useful (and fairly easy) to be able to
> do a[:5] == b[:5]. As we do with loops, char* could have an implicit
> end, so a[:] would be a[:strlen(a)] if a is a char*.

I agree that this would be a nice feature. (BTW, these things are currently 
easiest to implement in Optimise.py, but I think they deserve a better 
place as they actually implement language features, not optimisations).

I should note that Python semantics dictate that a[:5] creates a copy of 
the array. However, it's perfectly reasonable to "optimise" the copy away 
inside of a comparison so that the above examples do the right thing 
efficiently.

On top of that, we could then change the semantics only of bytes *literals* 
to make the comparison operators compare their content. I.e.

     cdef char* s = ...
     print s == b"abcdefg"

will strcmp() s to the byte sequence b"abcdefg" (maybe even using 
strncmp()), whereas

     cdef char* s = ...
     cdef char* cmpval = b"abcdefg"
     print s == cmpval

compares the pointers as before, as would

     cdef char* s = ...
     print s == <char*>b"abcdefg"

I don't see a use case for comparing a char* to a bytes literal as 
pointers. Everyone who does that deserves breaking his or her code in a 
future Cython version.

In general, I think that bytes literals should behave more like Python 
bytes objects than char*, if only to preserve their correct length in the 
face of null bytes. They can easily (and safely) coerce to a char* at any 
time, so we don't really loose anything by switching them into Python 
objects by default and (as Craig hinted) inferring char* where it is safe.

(The above might sounds like a 180? turn of myself, but it's good to change 
your mind when the facts change)

Stefan

From stefan at sun.ac.za  Tue Oct 12 17:41:56 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 12 Oct 2010 17:41:56 +0200
Subject: [Cython] Cython slides / examples / exercises
In-Reply-To: <4C99E4B1.8010707@student.matnat.uio.no>
References: <AANLkTimWQ51qjpiaDDNeURNB2kLBueH+eT7ysJZiVvEs@mail.gmail.com>
	<4C99E4B1.8010707@student.matnat.uio.no>
Message-ID: <AANLkTikvYnPJHxPJH1TscO3eLztouGaACUZG1_a3rk2-@mail.gmail.com>

On Wed, Sep 22, 2010 at 1:12 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>> I'm presenting a Cython introduction at the Trento Summerschool and,
>> while I have some material (mostly NumPy related), I was wondering if
>> there is any "stock" material floating around?
>>
> My last talk is here:
> http://github.com/dagss/euroscipy2010/
>
> and some other talks:
>
> http://wiki.cython.org/talks

I've uploaded my slides to the above webpage and added them to the
list (I assume newer talks go at the top?).  Thanks, Dag, for the
multiplication example that I used.

Regards
St?fan

From stefan_ml at behnel.de  Thu Oct 14 11:53:52 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 Oct 2010 11:53:52 +0200
Subject: [Cython] [cython-users] sponsoring generators?
In-Reply-To: <4C74D354.7000301@behnel.de>
References: <AANLkTi=H08aXGgsocnLM68aDxCcot4shMGrNARxsA9+D@mail.gmail.com>
	<4C74D354.7000301@behnel.de>
Message-ID: <4CB6D330.5060406@behnel.de>

Stefan Behnel, 25.08.2010 10:24:
>>  * Closures are fully supported for Python functions. Cython supports
>> inner functions and lambda expressions. Generators and generator
>> expressions are __not__ supported in this release.
>
> Regarding this bit - is anyone interested in sponsoring the generator
> implementation? That would allow me to implement it in the next couple
> of months.

Coming back to this after a while: it appears that there is no interest in 
paying the development. So generators are back to status "open".

Stefan

From matej at laitl.cz  Thu Oct 14 19:56:50 2010
From: matej at laitl.cz (=?utf-8?q?Mat=C4=9Bj_Laitl?=)
Date: Thu, 14 Oct 2010 19:56:50 +0200
Subject: [Cython] Wrong C code generated (cdef class < cdef class
	inheritance, cpdef methods, pure python mode)
In-Reply-To: <AANLkTimjeGfaE820gOyW=Yds+3EbtVEhfndoz5hPs3RN@mail.gmail.com>
References: <4ca22262.8e05df0a.2a39.ffffaa6d@mx.google.com>
	<AANLkTimjeGfaE820gOyW=Yds+3EbtVEhfndoz5hPs3RN@mail.gmail.com>
Message-ID: <201010141956.52189.matej@laitl.cz>

On 28.?9.?2010 Robert Bradshaw wrote:
> On Tue, Sep 28, 2010 at 10:14 AM, Mat?j Laitl <matej at laitl.cz> wrote:
> > I have problems with cython dealing with following example case (python
> > 2.6.5, cython 0.13):
> > 
> > 2 cdef classes, Pdf and GaussPdf; GaussPdf inherits Pdf. Both are written
> > in pure python file pdfs.py and use augmentation pdfs.pxd for cythoning.
> > (attached) Both have just cpdef methods, GaussPdf overrides 4 of 5 Pdf's
> > methods. When tested as Python code, everything is okay, but when
> > compiled
> > 
> > (...)
> > 
> > Also, vtabstruct for Pdf seems strange (it contains GaussPdf methods),
> > but that may be correct inheritance magic.
> > 
> > Is this really a bug in Cython, or am I doing something completely wrong?
> 
> No, this looks like a bug on our side. Clearly the pure mode +
> augmented .pxd is insufficiently tested. I have an idea where it might
> be happening too...

Okay. Should I file a bug then? (I can attach a testcase maybe)

Robert, would you tell your secret about where you think the problem lies? :-)

Regards,
	Mat?j

From robertwb at math.washington.edu  Thu Oct 14 22:33:21 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 Oct 2010 13:33:21 -0700
Subject: [Cython] Wrong C code generated (cdef class < cdef class
 inheritance, cpdef methods, pure python mode)
In-Reply-To: <201010141956.52189.matej@laitl.cz>
References: <4ca22262.8e05df0a.2a39.ffffaa6d@mx.google.com>
	<AANLkTimjeGfaE820gOyW=Yds+3EbtVEhfndoz5hPs3RN@mail.gmail.com>
	<201010141956.52189.matej@laitl.cz>
Message-ID: <AANLkTi=JRMm1rqFW5XuGFN+6M61T5zybchg1c4OQJTTR@mail.gmail.com>

On Thu, Oct 14, 2010 at 10:56 AM, Mat?j Laitl <matej at laitl.cz> wrote:
> On 28.?9.?2010 Robert Bradshaw wrote:
>> On Tue, Sep 28, 2010 at 10:14 AM, Mat?j Laitl <matej at laitl.cz> wrote:
>> > I have problems with cython dealing with following example case (python
>> > 2.6.5, cython 0.13):
>> >
>> > 2 cdef classes, Pdf and GaussPdf; GaussPdf inherits Pdf. Both are written
>> > in pure python file pdfs.py and use augmentation pdfs.pxd for cythoning.
>> > (attached) Both have just cpdef methods, GaussPdf overrides 4 of 5 Pdf's
>> > methods. When tested as Python code, everything is okay, but when
>> > compiled
>> >
>> > (...)
>> >
>> > Also, vtabstruct for Pdf seems strange (it contains GaussPdf methods),
>> > but that may be correct inheritance magic.
>> >
>> > Is this really a bug in Cython, or am I doing something completely wrong?
>>
>> No, this looks like a bug on our side. Clearly the pure mode +
>> augmented .pxd is insufficiently tested. I have an idea where it might
>> be happening too...
>
> Okay. Should I file a bug then? (I can attach a testcase maybe)

Yes, please do.

> Robert, would you tell your secret about where you think the problem lies? :-)

I think this has to do with the order in which method signatures are
resolved and the cdef declarations applied. Essentially, it find the
superclass method (as it was not overwritten) and applies the current
class signature to it.

- Robert

From ryan at rfk.id.au  Sat Oct 16 06:42:18 2010
From: ryan at rfk.id.au (Ryan Kelly)
Date: Sat, 16 Oct 2010 15:42:18 +1100
Subject: [Cython] feature suggestion:  honour __metaclass__ declarations
Message-ID: <1287204138.2263.33.camel@durian>


Hi All,


  One feature I'm currently missing in Cython is the ability to define
the metaclass for a given class, i.e. to have code like this work as it
does in regular python:


    class MyMetaclass(type):
        pass

    class MyClass(object):
        ___metaclass__ = MyMetaclass

    assert isinstance(MyClass,MyMetaclass)


  This currently compiles and runs fine under Cython, but the assertion
fails.  The __metaclass__ declaration is treated as a normal attribute
assignment and does not affect the construction of MyClass.

  I've done some experimentation, and it seems like the full semantics
of metaclasses can be achieved with very little change to the code
generated by Cython - just five small tweaks to the output.  At least,
they seem like small tweaks to my untrained eye :-)

  Unfortunately I'm not familiar enough Cython code to simply send along
a patch, but I'm posting the results of my experiment in the hope that
someone more knowledgeable can put them to good use.

  Attached are:

    * meta.pyx:  a cython extension module using metaclasses
                 in a variety of different ways.

    * test_meta.py:  a test suite for proper metaclass behaviour

    * meta.c.orig:  the unmodified output from cython 0.13; this code
                    fails all of the tests in test_meta.py

    * meta.c:   modified output that passes all of test_meta.py

    * changes.txt:  a description of the five changes I made to the
                    generated code and why each was necessary.


  I will continue poking at the Cython sources in the hope of
implementing this myself, but I don't know if or when I'll be able to
find enough time to work it all through...


   Cheers,

       Ryan

-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan at rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details

-------------- next part --------------

cdef extern from "object.h":
    ctypedef class __builtin__.type [object PyHeapTypeObject]:
        pass

cdef class CMetaclass(type):
    """A cdef metaclass."""
    cpdef testit(cls):
        return "CMetaclass"

class PyMetaclass(type):
    """A python metaclass."""
    def testit(cls):
        return "PyMetaclass"

cdef class CClass(object):
    """A normal everyday cdef class."""
    pass

cdef class CClass_CM(object):
    """A cdef class with a cdef metaclass."""
    __metaclass__ = CMetaclass

class PyClass(object):
    """A normal everyday python class."""
    pass

class PyClass_CM(object):
    """A python class with a cdef metaclass."""
    __metaclass__ = CMetaclass

class PyClass_PM(object):
    """A python class with a python metaclass."""
    __metaclass__ = PyMetaclass


-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_meta.py
Type: text/x-python
Size: 1416 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101016/ee38da07/attachment-0001.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: meta.c.orig
Type: text/x-csrc
Size: 65787 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101016/ee38da07/attachment-0002.c 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: meta.c
Type: text/x-csrc
Size: 66901 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101016/ee38da07/attachment-0003.c 
-------------- next part --------------

====================

(1)  For each cdef class, provide a typedef that can be used to declare a
     static instance of that struct.  This name is used in (2) below.

Original output:

    struct __pyx_obj_4meta_CMetaclass {
      PyHeapTypeObject __pyx_base;
      struct __pyx_vtabstruct_4meta_CMetaclass *__pyx_vtab;
    };


Modified output:

    typedef struct __pyx_obj_4meta_CMetaclass {
      PyHeapTypeObject __pyx_base;
      struct __pyx_vtabstruct_4meta_CMetaclass *__pyx_vtab;
    } __pyx_typeobj_4meta_CMetaclass;


====================

(2)  When a cdef class has __metaclass__ defined, declare its static
     type object as an instance of the metaclass rather than of PyTypeObject.
     Also initialise an extra pointer for the metaclass vtab.
     This allows methods from the metaclass to be called on the class.

Original output:

    PyTypeObject __pyx_type_4meta_CClass_CM = {
      PyVarObject_HEAD_INIT(0, 0)
      ...
      ...
      0, /*tp_subclasses*/
      0, /*tp_weaklist*/
    };


Modified output:

    __pyx_typeobj_4meta_CMetaclass __pyx_type_4meta_CClass_CM = {
      PyVarObject_HEAD_INIT(0, 0)
      ...
      ...
      0, /*tp_subclasses*/
      0, /*tp_weaklist*/
      0, /*metaclass vtab */
    };


====================

(3)  When a cdef class has __metaclass__ defined, set the type pointer
     and initialise the vtab pointer before calling PyType_Ready.

     This would ordinarily be done by the overridden tp_new slot of the
     metaclass, but it's not called because the type object was allocated
     statically in (2) above.

Original output:

    if (PyType_Ready(&__pyx_type_4meta_CClass_CM) < 0) {...}


Modified output:

    //  Tell python what type the object is.
    Py_TYPE(&__pyx_type_4meta_CClass_CM) = __pyx_ptype_4meta_CMetaclass;
    //  Set the vtab pointer
    __pyx_type_4meta_CClass_CM.__pyx_vtab = __pyx_vtabptr_4meta_CMetaclass;
    if (PyType_Ready((PyTypeObject*)&__pyx_type_4meta_CClass_CM) < 0) {...}

====================

(4)  Have __Pyx_CreateClass check for __metaclass__ and call it if present.
     This allows python classes to specify a metaclass in the normal way.


Original output:


    #if PY_MAJOR_VERSION < 3
    result = PyClass_New(bases, dict, name);
    #else
    result = PyObject_CallFunctionObjArgs((PyObject *)&PyType_Type, name, bases, dict, NULL);
    #endif


Modified outout:

     metaclass = PyDict_GetItemString(dict, "__metaclass__");
     if(metaclass != NULL) {
         result = PyObject_CallFunctionObjArgs(metaclass, name, bases, dict, NULL);
     } else {
         #if PY_MAJOR_VERSION < 3
         result = PyClass_New(bases, dict, name);
         #else
         result = PyObject_CallFunctionObjArgs((PyObject *)&PyType_Type, name, bases, dict, NULL);
         #endif
     }

====================

(5)  When a non-cdef class has __metaclass__ defined, include it in the
     dict that is passed to __Pyx_CreateClass.

     It may be necessary to load the metaclass object if it's a python class
     rather than a cdef class.

Original output:

  if (PyDict_SetItemString(((PyObject *)__pyx_1), "__doc__", __pyx_kp_3) < 0) {...}
  __pyx_2 = __Pyx_CreateClass(...)


Modified output (cdef metaclass):

  if (PyDict_SetItemString(((PyObject *)__pyx_1), "__doc__", __pyx_kp_3) < 0) {...}
  if (PyDict_SetItemString(((PyObject *)__pyx_1), "__metaclass__", __pyx_ptype_4meta_CMetaclass) < 0) {...}
  __pyx_2 = __Pyx_CreateClass(...)


Modified output (python metaclass):

  if (PyDict_SetItemString(((PyObject *)__pyx_3), "__doc__", __pyx_kp_4) < 0) {...}
  __pyx_1 = __Pyx_GetName(__pyx_m, __pyx_kp_PyMetaclass); ...
  __Pyx_GOTREF(__pyx_1);
  if (PyDict_SetItemString(((PyObject *)__pyx_3), "__metaclass__", __pyx_1) < 0) {...}
  __Pyx_DECREF(__pyx_1); __pyx_1 = 0;
  __pyx_2 = __Pyx_CreateClass(...)



-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101016/ee38da07/attachment-0001.pgp 

From mistobaan at gmail.com  Sat Oct 16 22:45:58 2010
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Sat, 16 Oct 2010 13:45:58 -0700
Subject: [Cython] pyrex_include_dirs
Message-ID: <AANLkTinMFORWoBf4ANVJ230_-DEjbtPgjVx938FysPkM@mail.gmail.com>

Hi folks,

I spend a bit of time today trying to figure out how to specify
Cython's import path from other modules
Cython.Distutils.extension import Extension
...
Extension [
pyrex_include_dirs = [ dir; dir2;]
]

Maybe this line should be mentioned inside:
http://docs.cython.org/src/userguide/sharing_declarations.html#sharing-extension-types?

Is there any good reason other than workforce why it still with pyrex name ?

Cheers

Fabrizio
--------------------------
Luck favors the prepared mind. (Pasteur)

From takowl at gmail.com  Sat Oct 16 23:55:06 2010
From: takowl at gmail.com (Thomas Kluyver)
Date: Sat, 16 Oct 2010 22:55:06 +0100
Subject: [Cython] Python 3 & circular cimports
Message-ID: <AANLkTi=2eby1mdaBnOSiPAd2zS2H8ewZhwSkEC-kYi9E@mail.gmail.com>

Hi,

I believe I'm seeing the same error that Darren Dale was seeing last month:
http://www.mail-archive.com/cython-dev at codespeak.net/msg09703.html

As in his case, two modules are attempting to cimport from one another,
which works in Python 2.6, but hits a runtime error when compiled and
imported with Python 3.1:
RuntimeError: maximum recursion depth exceeded while calling a Python object

In my case, I'm attempting to make the Python bindings for Zero MQ work in
Python 3. The offending modules are zmq.core.context and zmq.core.socket:
http://github.com/takowl/pyzmq/tree/py3zmq-attempt3

This is using Cython-0.13 as downloaded from the website.

Did anyone find anything following the discussion last month?

Thanks,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101016/f5c34b11/attachment.htm 

From greg.ewing at canterbury.ac.nz  Sat Oct 16 23:05:09 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 17 Oct 2010 10:05:09 +1300
Subject: [Cython] [cython-users] cdef metaclass - possible?
Message-ID: <4CBA1385.8030805@canterbury.ac.nz>

Robert Bradshaw wrote:

 > On Fri, Oct 15, 2010 at 11:11 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
 >
 >> Currently, Cython creates a class with empty namespace, next
 >> populates the class with setattr
 >
 > I have actually always wondered why we did it this way, but have never
 > looked into changing it myself.


Pyrex did it that way because it was necessary to wrap
the methods in unbound method objects before putting them
in the class, and the class had to exist before UBMs could
be created.

There was some talk once about giving C-implemented functions
method-binding behaviour, which would make the UBMs uneccessary,
but I don't know whether it's been done.

-- Greg

From robertwb at math.washington.edu  Wed Oct 20 05:53:27 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 19 Oct 2010 20:53:27 -0700
Subject: [Cython] pyrex_include_dirs
In-Reply-To: <AANLkTinMFORWoBf4ANVJ230_-DEjbtPgjVx938FysPkM@mail.gmail.com>
References: <AANLkTinMFORWoBf4ANVJ230_-DEjbtPgjVx938FysPkM@mail.gmail.com>
Message-ID: <AANLkTim7HWvCEV0a-BZUf_SHCN=07GqBiO_h_rWt3uuW@mail.gmail.com>

On Sat, Oct 16, 2010 at 1:45 PM, Fabrizio Milo aka misto
<mistobaan at gmail.com> wrote:
> Hi folks,
>
> I spend a bit of time today trying to figure out how to specify
> Cython's import path from other modules
> Cython.Distutils.extension import Extension
> ...
> Extension [
> pyrex_include_dirs = [ dir; dir2;]
> ]
>
> Maybe this line should be mentioned inside:
> http://docs.cython.org/src/userguide/sharing_declarations.html#sharing-extension-types?
>
> Is there any good reason other than workforce why it still with pyrex name ?

I think it's purely historical. Pyrex and Cython are still mostly
compatible, but by now I think they've diverged enough that expecting
to be able to switch back and forth is probably not realistic for
anything but the simplest projects (and even then you'd have to
restrict what features you use).

- Robert

From stefan_ml at behnel.de  Fri Oct 22 09:44:05 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 22 Oct 2010 09:44:05 +0200
Subject: [Cython] automatic build of cython-docs in Hudson build server
Message-ID: <4CC140C5.4010808@behnel.de>

Hi,

I added a perpetual, change triggered build of cython-docs to Hudson.

https://sage.math.washington.edu:8091/hudson/job/cython-docs/

The latest generated HTML documentation is published here:

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/


Stefan

From stefan_ml at behnel.de  Thu Oct 28 11:50:27 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 28 Oct 2010 11:50:27 +0200
Subject: [Cython] Fwd: Re: Fastest way to detect a non-ASCII character in a
 list of strings.
Message-ID: <4CC94763.6000205@behnel.de>

Hi,

here's a quote from c.l.py.

BTW, I think we should put up a couple of testimonials on the front page. 
Recommendation is worth something in this world.

Have fun,

Stefan


-------- Original Message --------
Subject: Re: Fastest way to detect a non-ASCII character in a list of strings.
Date: Thu, 28 Oct 2010 08:10:27 +0100
From: Dun Peal <dunpealer (at) gmail.com>

I find myself surprised at the relatively little use that Cython is seeing.

You would expect a whole lot of organizations and people to fancy a
language that's about as high-level as Python, yet almost as fast and
down-to-the-metal as C.

Add to that the ability to seamlessly integrate with both your
existing C/++ codebase and your Python codebase, easily mix very high
level abstractions with very low-level machine access... clear winner.

I'd expect Cython to have a far larger userbase, with a long line of
sponsors queuing up to fund further development. Maybe it will get
there some day :-)

D

From anders at embl.de  Thu Oct 28 18:52:50 2010
From: anders at embl.de (Simon Anders)
Date: Thu, 28 Oct 2010 18:52:50 +0200
Subject: [Cython] Incorrect handling of properties overlaying attributes
Message-ID: <4CC9AA62.8070100@embl.de>

Hi

I might have found a subtle bug regarding Cython's handling of 
polymorphic classes: When a class contains a field, and a daughter class 
overlays the field with a property, a method of the base class will fail 
to notice this when called for an object of the daughter class.

If this sounded convoluted, an example might be clearer:

--8<--
cdef class A( object ):

    cdef str me

    def __init__( self ):
       self.me = "A"

    def whoami( self ):
       return self.me

cdef class B( A ):

    @property
    def me( self ):
       return "B"

def main():
    b = B()
    print b.me
    print b.whoami()
--8<--

The output of main() is:
B
A


However, the equivalent Python code, i.e., the same file, without the 
'cdef's before the 'class' statement and without the line 'cdef str me', 
produces an error, namely:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "test.pyx", line 16, in test.main (test.c:489)
     b = B()
   File "test.pyx", line 4, in test.A.__init__ (test.c:380)
     self.me = "A"
AttributeError: can't set attribute


I haven't tested it, but I'd guess that, had I provided a setter method 
for the property, too, A.__init__ would not have used it when called for 
a B object.


Cheers
   Simon


+---
| Dr. Simon Anders, Dipl.-Phys.
| European Molecular Biology Laboratory (EMBL), Heidelberg
| office phone +49-6221-387-8632
| preferred (permanent) e-mail: sanders at fs.tum.de

From robertwb at math.washington.edu  Fri Oct 29 09:30:01 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 29 Oct 2010 00:30:01 -0700
Subject: [Cython] Fwd: Re: Fastest way to detect a non-ASCII character
 in a list of strings.
In-Reply-To: <4CC94763.6000205@behnel.de>
References: <4CC94763.6000205@behnel.de>
Message-ID: <AANLkTi=fVYM2hdc5jjw_OeOp45xYDr--hXS09WZ3ynW8@mail.gmail.com>

That's a great idea.

On Thu, Oct 28, 2010 at 2:50 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> here's a quote from c.l.py.
>
> BTW, I think we should put up a couple of testimonials on the front page.
> Recommendation is worth something in this world.
>
> Have fun,
>
> Stefan
>
>
> -------- Original Message --------
> Subject: Re: Fastest way to detect a non-ASCII character in a list of strings.
> Date: Thu, 28 Oct 2010 08:10:27 +0100
> From: Dun Peal <dunpealer (at) gmail.com>
>
> I find myself surprised at the relatively little use that Cython is seeing.
>
> You would expect a whole lot of organizations and people to fancy a
> language that's about as high-level as Python, yet almost as fast and
> down-to-the-metal as C.
>
> Add to that the ability to seamlessly integrate with both your
> existing C/++ codebase and your Python codebase, easily mix very high
> level abstractions with very low-level machine access... clear winner.
>
> I'd expect Cython to have a far larger userbase, with a long line of
> sponsors queuing up to fund further development. Maybe it will get
> there some day :-)
>
> D
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Fri Oct 29 13:36:12 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 29 Oct 2010 13:36:12 +0200
Subject: [Cython] Fwd: Re: Fastest way to detect a non-ASCII character
 in a list of strings.
In-Reply-To: <AANLkTi=fVYM2hdc5jjw_OeOp45xYDr--hXS09WZ3ynW8@mail.gmail.com>
References: <4CC94763.6000205@behnel.de>
	<AANLkTi=fVYM2hdc5jjw_OeOp45xYDr--hXS09WZ3ynW8@mail.gmail.com>
Message-ID: <4CCAB1AC.2070009@behnel.de>

Robert Bradshaw, 29.10.2010 09:30:
> On Thu, Oct 28, 2010 at 2:50 AM, Stefan Behnel wrote:
>> here's a quote from c.l.py.
>>
>> BTW, I think we should put up a couple of testimonials on the front page.
>> Recommendation is worth something in this world.
>>
> That's a great idea.

Done. It uses a bit of JavaScript, please test if it works with your 
browsers. Anyone with an old IE?

Stefan

From robertwb at math.washington.edu  Fri Oct 29 20:06:15 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 29 Oct 2010 11:06:15 -0700
Subject: [Cython] Fwd: Re: Fastest way to detect a non-ASCII character
 in a list of strings.
In-Reply-To: <4CCAB1AC.2070009@behnel.de>
References: <4CC94763.6000205@behnel.de>
	<AANLkTi=fVYM2hdc5jjw_OeOp45xYDr--hXS09WZ3ynW8@mail.gmail.com>
	<4CCAB1AC.2070009@behnel.de>
Message-ID: <AANLkTi=dNntddQuDsoqQ2W6588TAy0uKtyV19gCXBEsL@mail.gmail.com>

On Fri, Oct 29, 2010 at 4:36 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 29.10.2010 09:30:
>> On Thu, Oct 28, 2010 at 2:50 AM, Stefan Behnel wrote:
>>> here's a quote from c.l.py.
>>>
>>> BTW, I think we should put up a couple of testimonials on the front page.
>>> Recommendation is worth something in this world.
>>>
>> That's a great idea.
>
> Done. It uses a bit of JavaScript, please test if it works with your
> browsers. Anyone with an old IE?

Nice.

From dalcinl at gmail.com  Fri Oct 29 22:30:42 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 29 Oct 2010 23:30:42 +0300
Subject: [Cython] Incorrect handling of properties overlaying attributes
In-Reply-To: <4CC9AA62.8070100@embl.de>
References: <4CC9AA62.8070100@embl.de>
Message-ID: <AANLkTimzPE_Jo9v7ewGTX23Vp3dt4=BVYh+s1ehMAQ6c@mail.gmail.com>

On 28 October 2010 19:52, Simon Anders <anders at embl.de> wrote:
> Hi
>
> I might have found a subtle bug regarding Cython's handling of
> polymorphic classes: When a class contains a field, and a daughter class
> overlays the field with a property, a method of the base class will fail
> to notice this when called for an object of the daughter class.
>
> If this sounded convoluted, an example might be clearer:
>
> --8<--
> cdef class A( object ):
>
> ? ?cdef str me
>
> ? ?def __init__( self ):
> ? ? ? self.me = "A"
>
> ? ?def whoami( self ):
> ? ? ? return self.me
>
> cdef class B( A ):
>
> ? ?@property
> ? ?def me( self ):
> ? ? ? return "B"
>
> def main():
> ? ?b = B()
> ? ?print b.me
> ? ?print b.whoami()
> --8<--
>
> The output of main() is:
> B
> A
>

Expected.

>
> However, the equivalent Python code, i.e., the same file, without the
> 'cdef's before the 'class' statement and without the line 'cdef str me',
> produces an error, namely:
>

Actually, you Python code is not equivalent. cdef classes are
extension types, and they behave different. if you remove "cdef", you
will get regular Python classes, and the code will work as you are
expecting.


> Traceback (most recent call last):
> ? File "<stdin>", line 1, in <module>
> ? File "test.pyx", line 16, in test.main (test.c:489)
> ? ? b = B()
> ? File "test.pyx", line 4, in test.A.__init__ (test.c:380)
> ? ? self.me = "A"
> AttributeError: can't set attribute
>
>
> I haven't tested it, but I'd guess that, had I provided a setter method
> for the property, too, A.__init__ would not have used it when called for
> a B object.
>
>
> Cheers
> ? Simon
>
>
> +---
> | Dr. Simon Anders, Dipl.-Phys.
> | European Molecular Biology Laboratory (EMBL), Heidelberg
> | office phone +49-6221-387-8632
> | preferred (permanent) e-mail: sanders at fs.tum.de
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Fri Oct 29 23:53:37 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 29 Oct 2010 14:53:37 -0700
Subject: [Cython] Incorrect handling of properties overlaying attributes
In-Reply-To: <AANLkTimzPE_Jo9v7ewGTX23Vp3dt4=BVYh+s1ehMAQ6c@mail.gmail.com>
References: <4CC9AA62.8070100@embl.de>
	<AANLkTimzPE_Jo9v7ewGTX23Vp3dt4=BVYh+s1ehMAQ6c@mail.gmail.com>
Message-ID: <AANLkTin4DNX0zgWom5YvxPEdrFW2MXgYUAh6U4Zt_2L+@mail.gmail.com>

On Fri, Oct 29, 2010 at 1:30 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 28 October 2010 19:52, Simon Anders <anders at embl.de> wrote:
>> Hi
>>
>> I might have found a subtle bug regarding Cython's handling of
>> polymorphic classes: When a class contains a field, and a daughter class
>> overlays the field with a property, a method of the base class will fail
>> to notice this when called for an object of the daughter class.
>>
>> If this sounded convoluted, an example might be clearer:
>>
>> --8<--
>> cdef class A( object ):
>>
>> ? ?cdef str me
>>
>> ? ?def __init__( self ):
>> ? ? ? self.me = "A"
>>
>> ? ?def whoami( self ):
>> ? ? ? return self.me
>>
>> cdef class B( A ):
>>
>> ? ?@property
>> ? ?def me( self ):
>> ? ? ? return "B"
>>
>> def main():
>> ? ?b = B()
>> ? ?print b.me
>> ? ?print b.whoami()
>> --8<--
>>
>> The output of main() is:
>> B
>> A
>>
>
> Expected.
>
>>
>> However, the equivalent Python code, i.e., the same file, without the
>> 'cdef's before the 'class' statement and without the line 'cdef str me',
>> produces an error, namely:
>>
>
> Actually, you Python code is not equivalent. cdef classes are
> extension types, and they behave different. if you remove "cdef", you
> will get regular Python classes, and the code will work as you are
> expecting.

Of course, unless there's a compelling performance (or major backwards
compatibility) reason why things can't be otherwise, I'd prefer cdef
classes to work as much like def classes as possible.

From vinjvinj at gmail.com  Sat Oct 30 00:07:28 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Fri, 29 Oct 2010 18:07:28 -0400
Subject: [Cython] numpy and pure python mode
Message-ID: <AANLkTikjWcKXfBzQ_rv5TMvornwh8sVGoLyFnvTFaFc-@mail.gmail.com>

Are there any examples of how to use numpy in pure python mode. I was
going to annotate a pure python file with definitions in a pxd file. I
can't seem to find any code with how to show declare numpy local
variables and parameters.

Thanks,

Vineet

From stefan_ml at behnel.de  Sat Oct 30 09:04:16 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 Oct 2010 09:04:16 +0200
Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1?
In-Reply-To: <4C6E763A.80602@behnel.de>
References: <4C6E763A.80602@behnel.de>
Message-ID: <4CCBC370.2090604@behnel.de>

Hi,

so, what's the status? I wouldn't like to see this dangling around for too 
long. There's a lot of interesting and nice stuff to merge here.


Stefan Behnel, 20.08.2010 14:34:
> your GSoC branch has been around for a while now. Given that 0.13 will be
> out soon (soon-isher than ever before), is there anything that you consider
> safe enough to get merged into mainline after the release?
>
> IMHO, any feature patch is a candidate, especially when it depends on
> currently illegal syntax (i.e. not impacting existing code). Everything
> that changes current behaviour is worth considering if it fixes a bug or
> Python compatibility issue but may have to wait for 0.14 if it impacts code
> that currently works with 0.13.

Haoyu, you uploaded patches to Rietvelt. However, they don't include the hg 
metadata, so I can't just apply them. Is it easy enough for you (or Craig) 
to provide complete exported patches for each ticket?


>   From a quick look, I think #488 (ellipsis) is safe, but I'd like to see
> some more syntax tests - I bet there are some in Python's own test suite.

Is the space issue resolved now?


> Ticket #487 (multiple 'with' statements) looks nice, but clearly lacks
> test. I can't even see a test that makes sure the context managers are
> executed in the correct nesting order, neither can I see anything that
> tests the chaining of cleanup actions during exception propagation, let
> alone partial propagation in cases where one of the context managers
> swallows an exception. Again, Python's test suite will provide hints on
> better tests here.
>
> Ticket #490 (nonlocal), while I'd be happy to get it in, seems too big a
> change to go into 0.13.1.
>
> It seems you also have a fix for #477. That would be another candidate for
> 0.13.1. Note that the related test case doesn't actually test that the
> argument typing has the expected effect. This could be done using a tree
> assertion based on coercion nodes.

Do we have more complete tests for these now?

I also took another quick look at the relative import code. Currently, we 
have the restriction that Cython requires (and consequently has) knowledge 
about the exact package layout for an extension module. I wonder if we 
shouldn't just use that information for relative imports.

Stefan

From stefan_ml at behnel.de  Sat Oct 30 13:10:50 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 Oct 2010 13:10:50 +0200
Subject: [Cython]  numpy and pure python mode
In-Reply-To: <AANLkTikjWcKXfBzQ_rv5TMvornwh8sVGoLyFnvTFaFc-@mail.gmail.com>
References: <AANLkTikjWcKXfBzQ_rv5TMvornwh8sVGoLyFnvTFaFc-@mail.gmail.com>
Message-ID: <4CCBFD3A.4000509@behnel.de>

[forwarding to cython-users list]

Vineet Jain, 30.10.2010 00:07:
> Are there any examples of how to use numpy in pure python mode. I was
> going to annotate a pure python file with definitions in a pxd file. I
> can't seem to find any code with how to show declare numpy local
> variables and parameters.
>
> Thanks,
>
> Vineet

From stefan_ml at behnel.de  Sat Oct 30 21:16:56 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 Oct 2010 21:16:56 +0200
Subject: [Cython] New cdef class directives: "final" and "internal"
Message-ID: <4CCC6F28.1040705@behnel.de>

Hi,

a while ago, we had agreed that a "final" directive would be nice to have 
for cdef classes. I implemented the basics for that (i.e. "final" types are 
no longer subclassable from Python), and I also added an "internal" 
directive that prevents a type from appearing in the module dict. So you 
can now write something like this for a type that you only use internally 
in your module:

    @cython.final
    @cython.internal
    cdef class _MyInternalType:
       ...

http://trac.cython.org/cython_trac/ticket/263
http://trac.cython.org/cython_trac/ticket/585

I think the "final" directive name was pretty much agreed on, but what 
about the "internal" directive? Any objections to that?

There are currently no optimisations for final types, but that may come in 
the future.

BTW, since I expect that both will not be features that users deploy all 
over the place (and I actually think that "final" should be used with some 
care), I didn't add a modifier to the parser syntax. It's currently a pure 
directive. And I would prefer leaving it that way.

Stefan

From dalcinl at gmail.com  Sat Oct 30 23:12:45 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sun, 31 Oct 2010 00:12:45 +0300
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCC6F28.1040705@behnel.de>
References: <4CCC6F28.1040705@behnel.de>
Message-ID: <AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>

On 30 October 2010 22:16, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> a while ago, we had agreed that a "final" directive would be nice to have
> for cdef classes. I implemented the basics for that (i.e. "final" types are
> no longer subclassable from Python), and I also added an "internal"
> directive that prevents a type from appearing in the module dict. So you
> can now write something like this for a type that you only use internally
> in your module:
>
> ? ?@cython.final
> ? ?@cython.internal
> ? ?cdef class _MyInternalType:
> ? ? ? ...
>
> http://trac.cython.org/cython_trac/ticket/263
> http://trac.cython.org/cython_trac/ticket/585
>

Many thanks for this, Stefan!

> I think the "final" directive name was pretty much agreed on, but what
> about the "internal" directive? Any objections to that?
>

Perhaps "private" ? I still prefer "internal".

> There are currently no optimisations for final types, but that may come in
> the future.
>
> BTW, since I expect that both will not be features that users deploy all
> over the place (and I actually think that "final" should be used with some
> care), I didn't add a modifier to the parser syntax. It's currently a pure
> directive. And I would prefer leaving it that way.
>

I agree, +1 for the current implementation.



-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Sun Oct 31 04:34:52 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 Oct 2010 20:34:52 -0700
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>
	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>
Message-ID: <AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>

On Sat, Oct 30, 2010 at 2:12 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 30 October 2010 22:16, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>> a while ago, we had agreed that a "final" directive would be nice to have
>> for cdef classes. I implemented the basics for that (i.e. "final" types are
>> no longer subclassable from Python), and I also added an "internal"
>> directive that prevents a type from appearing in the module dict. So you
>> can now write something like this for a type that you only use internally
>> in your module:
>>
>> ? ?@cython.final
>> ? ?@cython.internal
>> ? ?cdef class _MyInternalType:
>> ? ? ? ...
>>
>> http://trac.cython.org/cython_trac/ticket/263
>> http://trac.cython.org/cython_trac/ticket/585
>>
>
> Many thanks for this, Stefan!
>
>> I think the "final" directive name was pretty much agreed on, but what
>> about the "internal" directive? Any objections to that?
>>
>
> Perhaps "private" ? I still prefer "internal".

I'd lean towards "private" but that's not a strong preference. I also
don't think such a feature is strictly necessary and complicates the
language, but wouldn't stand in its way if I'm the only one.

>> There are currently no optimisations for final types, but that may come in
>> the future.
>>
>> BTW, since I expect that both will not be features that users deploy all
>> over the place (and I actually think that "final" should be used with some
>> care), I didn't add a modifier to the parser syntax. It's currently a pure
>> directive. And I would prefer leaving it that way.
>>
>
> I agree, +1 for the current implementation.

+1 from me too.

- Robert

From stefan_ml at behnel.de  Sun Oct 31 07:09:56 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 31 Oct 2010 07:09:56 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>
	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>
Message-ID: <4CCD0834.20801@behnel.de>

Robert Bradshaw, 31.10.2010 04:34:
> On Sat, Oct 30, 2010 at 2:12 PM, Lisandro Dalcin wrote:
>> On 30 October 2010 22:16, Stefan Behnel wrote:
>>> a while ago, we had agreed that a "final" directive would be nice to have
>>> for cdef classes. I implemented the basics for that (i.e. "final" types are
>>> no longer subclassable from Python), and I also added an "internal"
>>> directive that prevents a type from appearing in the module dict. So you
>>> can now write something like this for a type that you only use internally
>>> in your module:
>>>
>>>     @cython.final
>>>     @cython.internal
>>>     cdef class _MyInternalType:
>>>        ...
>>>
>>> http://trac.cython.org/cython_trac/ticket/263
>>> http://trac.cython.org/cython_trac/ticket/585
>>>
>>
>> Many thanks for this, Stefan!

I'm as happy as you are.


>>> I think the "final" directive name was pretty much agreed on, but what
>>> about the "internal" directive? Any objections to that?
>>
>> Perhaps "private" ? I still prefer "internal".
>
> I'd lean towards "private" but that's not a strong preference.

I may be mistaken, but it looks like we don't actually use "private" 
anywhere in the language so far. I thought we did, but it currently seems 
to be a purely internal visibility marker. So to me, "internal" makes as 
much sense as "private".

Robert, if you want me to change it, just say so clearly. ;)


 > I also
 > don't think such a feature is strictly necessary and complicates the
 > language, but wouldn't stand in its way if I'm the only one.

I just wrote a patch for lxml.etree that adds one or both of the decorators 
where they make sense. (I won't apply it right away as the next release is 
hopefully closer than the next release of Cython, but I'll keep it around 
for one of the next releases after Cython 0.13.1.)

It turns out that out of the 117 extension types in lxml.etree, 39 would be 
happy about at least one of the decorators: 30x final and 36x internal. So 
almost every third class is an internal one. Most of them are context 
classes that are passed through C in one way or another and that are used 
by internal callback code. Pure implementation details that no user should 
ever mess around with or even care about.

Stefan

From robertwb at math.washington.edu  Sun Oct 31 08:09:51 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 31 Oct 2010 00:09:51 -0700
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCD0834.20801@behnel.de>
References: <4CCC6F28.1040705@behnel.de>
	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>
	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>
	<4CCD0834.20801@behnel.de>
Message-ID: <AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>

On Sat, Oct 30, 2010 at 11:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 31.10.2010 04:34:
>> On Sat, Oct 30, 2010 at 2:12 PM, Lisandro Dalcin wrote:
>>> On 30 October 2010 22:16, Stefan Behnel wrote:
>>>> a while ago, we had agreed that a "final" directive would be nice to have
>>>> for cdef classes. I implemented the basics for that (i.e. "final" types are
>>>> no longer subclassable from Python), and I also added an "internal"
>>>> directive that prevents a type from appearing in the module dict. So you
>>>> can now write something like this for a type that you only use internally
>>>> in your module:
>>>>
>>>> ? ? @cython.final
>>>> ? ? @cython.internal
>>>> ? ? cdef class _MyInternalType:
>>>> ? ? ? ?...
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/263
>>>> http://trac.cython.org/cython_trac/ticket/585
>>>>
>>>
>>> Many thanks for this, Stefan!
>
> I'm as happy as you are.
>
>
>>>> I think the "final" directive name was pretty much agreed on, but what
>>>> about the "internal" directive? Any objections to that?
>>>
>>> Perhaps "private" ? I still prefer "internal".
>>
>> I'd lean towards "private" but that's not a strong preference.
>
> I may be mistaken, but it looks like we don't actually use "private"
> anywhere in the language so far. I thought we did, but it currently seems
> to be a purely internal visibility marker. So to me, "internal" makes as
> much sense as "private".
>
> Robert, if you want me to change it, just say so clearly. ;)
>
>
> ?> I also
> ?> don't think such a feature is strictly necessary and complicates the
> ?> language, but wouldn't stand in its way if I'm the only one.
>
> I just wrote a patch for lxml.etree that adds one or both of the decorators
> where they make sense. (I won't apply it right away as the next release is
> hopefully closer than the next release of Cython, but I'll keep it around
> for one of the next releases after Cython 0.13.1.)
>
> It turns out that out of the 117 extension types in lxml.etree, 39 would be
> happy about at least one of the decorators: 30x final and 36x internal. So
> almost every third class is an internal one. Most of them are context
> classes that are passed through C in one way or another and that are used
> by internal callback code. Pure implementation details that no user should
> ever mess around with or even care about.

What about using the

[cdef] class A:
    ...

del A

idiom that people use for pure Python?

- Robert

From stefan_ml at behnel.de  Sun Oct 31 08:23:05 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 31 Oct 2010 08:23:05 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>	<4CCD0834.20801@behnel.de>
	<AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>
Message-ID: <4CCD1959.9070806@behnel.de>

Robert Bradshaw, 31.10.2010 08:09:
> On Sat, Oct 30, 2010 at 11:09 PM, Stefan Behnel wrote:
>> Robert Bradshaw, 31.10.2010 04:34:
>>>> On 30 October 2010 22:16, Stefan Behnel wrote:
>>>>>      @cython.internal
>>> I also
>>> don't think such a feature is strictly necessary and complicates the
>>> language, but wouldn't stand in its way if I'm the only one.
>>
>> I just wrote a patch for lxml.etree that adds one or both of the decorators
>> where they make sense. (I won't apply it right away as the next release is
>> hopefully closer than the next release of Cython, but I'll keep it around
>> for one of the next releases after Cython 0.13.1.)
>>
>> It turns out that out of the 117 extension types in lxml.etree, 39 would be
>> happy about at least one of the decorators: 30x final and 36x internal. So
>> almost every third class is an internal one. Most of them are context
>> classes that are passed through C in one way or another and that are used
>> by internal callback code. Pure implementation details that no user should
>> ever mess around with or even care about.
>
> What about using the
>
> [cdef] class A:
>      ...
>
> del A
>
> idiom that people use for pure Python?

For example, the following doesn't work in Python:

     class Stuff(object): pass

     def use_stuff():
        Stuff()

     del Stuff

     use_stuff()

This is different for cdef classes in Cython, but that can be considered a 
bug to some extent. Arguably, that also sort-of applies to the decorator, 
but I think it makes it at least clearer what happens because it is part of 
the class declaration.

Stefan

From stefan_ml at behnel.de  Sun Oct 31 08:27:56 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 31 Oct 2010 08:27:56 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCC6F28.1040705@behnel.de>
References: <4CCC6F28.1040705@behnel.de>
Message-ID: <4CCD1A7C.3010805@behnel.de>

Stefan Behnel, 30.10.2010 21:16:
> can now write something like this for a type that you only use internally
> in your module:
>
>      @cython.final
>      @cython.internal
>      cdef class _MyInternalType:
>         ...

One problem I spotted was the auto test dict (again). It takes the classes 
from the module dict if they have a docstring, which now fails for internal 
types (it would also fail for 'del'-ed types, BTW). I always wondered why 
it used the module dict anyway - I guess it's just simplicity. Any reason 
not to rewrite that?

Stefan

From dalcinl at gmail.com  Sun Oct 31 08:22:09 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sun, 31 Oct 2010 10:22:09 +0300
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>
	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>
	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>
	<4CCD0834.20801@behnel.de>
	<AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>
Message-ID: <AANLkTinQ3sWbo6N=i+T6=96jquvHPOHRMck0fOExZL6j@mail.gmail.com>

On 31 October 2010 10:09, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Sat, Oct 30, 2010 at 11:09 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Robert Bradshaw, 31.10.2010 04:34:
>>> On Sat, Oct 30, 2010 at 2:12 PM, Lisandro Dalcin wrote:
>>>> On 30 October 2010 22:16, Stefan Behnel wrote:
>>>>> a while ago, we had agreed that a "final" directive would be nice to have
>>>>> for cdef classes. I implemented the basics for that (i.e. "final" types are
>>>>> no longer subclassable from Python), and I also added an "internal"
>>>>> directive that prevents a type from appearing in the module dict. So you
>>>>> can now write something like this for a type that you only use internally
>>>>> in your module:
>>>>>
>>>>> ? ? @cython.final
>>>>> ? ? @cython.internal
>>>>> ? ? cdef class _MyInternalType:
>>>>> ? ? ? ?...
>>>>>
>>>>> http://trac.cython.org/cython_trac/ticket/263
>>>>> http://trac.cython.org/cython_trac/ticket/585
>>>>>
>>>>
>>>> Many thanks for this, Stefan!
>>
>> I'm as happy as you are.
>>
>>
>>>>> I think the "final" directive name was pretty much agreed on, but what
>>>>> about the "internal" directive? Any objections to that?
>>>>
>>>> Perhaps "private" ? I still prefer "internal".
>>>
>>> I'd lean towards "private" but that's not a strong preference.
>>
>> I may be mistaken, but it looks like we don't actually use "private"
>> anywhere in the language so far. I thought we did, but it currently seems
>> to be a purely internal visibility marker. So to me, "internal" makes as
>> much sense as "private".
>>
>> Robert, if you want me to change it, just say so clearly. ;)
>>
>>
>> ?> I also
>> ?> don't think such a feature is strictly necessary and complicates the
>> ?> language, but wouldn't stand in its way if I'm the only one.
>>
>> I just wrote a patch for lxml.etree that adds one or both of the decorators
>> where they make sense. (I won't apply it right away as the next release is
>> hopefully closer than the next release of Cython, but I'll keep it around
>> for one of the next releases after Cython 0.13.1.)
>>
>> It turns out that out of the 117 extension types in lxml.etree, 39 would be
>> happy about at least one of the decorators: 30x final and 36x internal. So
>> almost every third class is an internal one. Most of them are context
>> classes that are passed through C in one way or another and that are used
>> by internal callback code. Pure implementation details that no user should
>> ever mess around with or even care about.
>
> What about using the
>
> [cdef] class A:
> ? ?...
>
> del A
>
> idiom that people use for pure Python?
>

I have to admit that this looks even simpler.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Sun Oct 31 14:56:49 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 31 Oct 2010 14:56:49 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <AANLkTinQ3sWbo6N=i+T6=96jquvHPOHRMck0fOExZL6j@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>	<4CCD0834.20801@behnel.de>	<AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>
	<AANLkTinQ3sWbo6N=i+T6=96jquvHPOHRMck0fOExZL6j@mail.gmail.com>
Message-ID: <4CCD75A1.8050909@behnel.de>

Lisandro Dalcin, 31.10.2010 08:22:
> On 31 October 2010 10:09, Robert Bradshaw wrote:
>> What about using the
>>
>> [cdef] class A:
>>     ...
>>
>> del A
>>
>> idiom that people use for pure Python?
>
> I have to admit that this looks even simpler.

Emphasis on "looks". I can see this easily working for Python classes, but 
what would be the semantics for cdef classes? Would it have any impact on 
their accessibility from Cython, and more specifically, from inside of the 
module that defines the type? What if a user externally overrides a Python 
class as cdef class in pure mode? And would it (or should it ever) be 
possible to redefine the type after deleting it? Should Cython prevent this 
for public/api classes, or would you just get the ImportError at runtime 
when it tries to read the type from the module dict?

Or would you just say "well, it's not really 'del' what's happening here, 
it's something that looks like Python, doesn't have an immediate impact, 
but removes the name from something that you can't see in your code"?

The advantage of a decorator is that it's easy to get right, to read and 
comprehend. I'm not really opposed to using "del" here, but I'm not sure I 
can oversee all the implications of allowing it on cdef types.

Stefan

From stefan_ml at behnel.de  Sun Oct 31 19:38:59 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 31 Oct 2010 19:38:59 +0100
Subject: [Cython] auto __test__ dict
In-Reply-To: <4CCD1A7C.3010805@behnel.de>
References: <4CCC6F28.1040705@behnel.de> <4CCD1A7C.3010805@behnel.de>
Message-ID: <4CCDB7C3.20208@behnel.de>

Stefan Behnel, 31.10.2010 08:27:
> Stefan Behnel, 30.10.2010 21:16:
>> can now write something like this for a type that you only use internally
>> in your module:
>>
>>       @cython.final
>>       @cython.internal
>>       cdef class _MyInternalType:
>>          ...
>
> One problem I spotted was the auto test dict (again). It takes the classes
> from the module dict if they have a docstring, which now fails for internal
> types (it would also fail for 'del'-ed types, BTW). I always wondered why
> it used the module dict anyway - I guess it's just simplicity. Any reason
> not to rewrite that?

I simplified it now to use the compile time docstrings directly, instead of 
looking them up at runtime. I don't think this breaks anything important, 
as the docstrings of functions and methods cannot be modified anyway. On 
the plus side, it removes a *ton* of ugly C code, and also saves some time 
during module initialisation.

The only visible thing that changes is when a decorator covers a function 
with another one that has a different docstring, as the original docstring 
will remain in the test dict. But this is a) rare (people usually go to 
great length trying to *make* the new function look like the old one) and 
b) something you can fix by disabling the __test__ dict generation. It's 
just a helpful feature, after all, not a base-level requirement for using 
Cython.

Stefan