From ondrej at certik.cz  Wed Jul  1 02:26:35 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 30 Jun 2009 18:26:35 -0600
Subject: [Cython] accessing ctype array's b_ptr
Message-ID: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>

Hi,

I need to access ctype array's b_ptr. The array is defined in
Python-2.6.2/Modules/_ctypes/ctypes.h  as:

struct tagCDataObject {
    PyObject_HEAD
    char *b_ptr;        /* pointer to memory block */
    int  b_needsfree;   /* need _we_ free the memory? */
    CDataObject *b_base;    /* pointer to base object or NULL */
    Py_ssize_t b_size;  /* size of memory block in bytes */
    Py_ssize_t b_length;    /* number of references we need */
    Py_ssize_t b_index; /* index of this object into base's
                   b_object list */
    PyObject *b_objects;    /* dictionary of references we need to
keep, or Py_None */
    union value b_value;
};


So I wrote this cython code:

cdef extern from "Python.h":
    ctypedef void PyTypeObject

cdef struct CDataObject:
    Py_ssize_t ob_refcnt
    PyTypeObject *ob_type
    char *b_ptr

and then I use it like this:

    cdef object b = ptr
    cdef CDataObject *a = <CDataObject *>b
    cdef char *x = a.b_ptr
    c_glNormalPointer(type, stride, <GLvoid *> x)


where "ptr" is a normal python variable, which contains the ctype
array, I then extract the b_ptr and use it. It works.

However, this assumes some particular form of PyObject_HEAD. Once it
changes, my code will segfault. So I guess the right way to do this is
to write my own C header file with the above struct (e.g. using the
PyObject_HEAD macro) and then just reference it using:

cdef extern from "Python.h":
    ctypedef void PyTypeObject

cdef extern from "my_header.h":
    ctypedef struct CDataObject:
        char *b_ptr

which should always work.

What do you think?

Ondrej

From dalcinl at gmail.com  Wed Jul  1 02:42:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 30 Jun 2009 21:42:09 -0300
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
Message-ID: <e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>

On Tue, Jun 30, 2009 at 9:26 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>
> However, this assumes some particular form of PyObject_HEAD. Once it
> changes, my code will segfault.
>

Indeed. That changes when you use a debug build of Python (./configure
--with-pydebug).

> So I guess the right way to do this is
> to write my own C header file with the above struct (e.g. using the
> PyObject_HEAD macro) and then just reference it using:
>
> cdef extern from "Python.h":
> ? ?ctypedef void PyTypeObject
>
> cdef extern from "my_header.h":
> ? ?ctypedef struct CDataObject:
> ? ? ? ?char *b_ptr
>
> which should always work.
>
> What do you think?
>

Provided that ctypes.h is not a public header, I would do exactly the
same as you...

PS: Can you believe that ctypes, being intended to be a tool for easy
interoperation with C code, does not provide a public C API in order
to let third-party codes to take advantage?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ondrej at certik.cz  Wed Jul  1 03:07:28 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 30 Jun 2009 19:07:28 -0600
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
	<e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
Message-ID: <85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>

On Tue, Jun 30, 2009 at 6:42 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> On Tue, Jun 30, 2009 at 9:26 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>>
>> However, this assumes some particular form of PyObject_HEAD. Once it
>> changes, my code will segfault.
>>
>
> Indeed. That changes when you use a debug build of Python (./configure
> --with-pydebug).
>
>> So I guess the right way to do this is
>> to write my own C header file with the above struct (e.g. using the
>> PyObject_HEAD macro) and then just reference it using:
>>
>> cdef extern from "Python.h":
>> ? ?ctypedef void PyTypeObject
>>
>> cdef extern from "my_header.h":
>> ? ?ctypedef struct CDataObject:
>> ? ? ? ?char *b_ptr
>>
>> which should always work.
>>
>> What do you think?
>>
>
> Provided that ctypes.h is not a public header, I would do exactly the
> same as you...
>
> PS: Can you believe that ctypes, being intended to be a tool for easy
> interoperation with C code, does not provide a public C API in order
> to let third-party codes to take advantage?

Indeed, it doesn't provide it on my Ubuntu. It sucks.

One more question. I am using ctypes because using it I can take a
list of floats and create a C array of it:

(c_float * len(ptr))(*ptr)

from historical reasons (I am adapting some opengl example from
pyglet, that used that approach --- since pyglet uses ctypes to wrap
opengl). I use cython, so I think a better way is to use numpy?

opengl supports strides, but it's just one number, e.g. it has to be
the same for all rows (as opposed to numpy). Is it safe to just copy
the numpy array, to make sure it's contiguous and then just use the
pointer to the .data member of ndarray? That's what I always do, but I
don't know if this is the safe or best approach.

Ondrej

From ondrej at certik.cz  Wed Jul  1 03:14:10 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 30 Jun 2009 19:14:10 -0600
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
	<e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
	<85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
Message-ID: <85b5c3130906301814m369a63d8x96a4cbfa59b7bfe7@mail.gmail.com>

On Tue, Jun 30, 2009 at 7:07 PM, Ondrej Certik<ondrej at certik.cz> wrote:
> On Tue, Jun 30, 2009 at 6:42 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
>> On Tue, Jun 30, 2009 at 9:26 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>>>
>>> However, this assumes some particular form of PyObject_HEAD. Once it
>>> changes, my code will segfault.
>>>
>>
>> Indeed. That changes when you use a debug build of Python (./configure
>> --with-pydebug).
>>
>>> So I guess the right way to do this is
>>> to write my own C header file with the above struct (e.g. using the
>>> PyObject_HEAD macro) and then just reference it using:
>>>
>>> cdef extern from "Python.h":
>>> ? ?ctypedef void PyTypeObject
>>>
>>> cdef extern from "my_header.h":
>>> ? ?ctypedef struct CDataObject:
>>> ? ? ? ?char *b_ptr
>>>
>>> which should always work.
>>>
>>> What do you think?
>>>
>>
>> Provided that ctypes.h is not a public header, I would do exactly the
>> same as you...
>>
>> PS: Can you believe that ctypes, being intended to be a tool for easy
>> interoperation with C code, does not provide a public C API in order
>> to let third-party codes to take advantage?
>
> Indeed, it doesn't provide it on my Ubuntu. It sucks.
>
> One more question. I am using ctypes because using it I can take a
> list of floats and create a C array of it:
>
> (c_float * len(ptr))(*ptr)
>
> from historical reasons (I am adapting some opengl example from
> pyglet, that used that approach --- since pyglet uses ctypes to wrap
> opengl). I use cython, so I think a better way is to use numpy?
>
> opengl supports strides, but it's just one number, e.g. it has to be
> the same for all rows (as opposed to numpy). Is it safe to just copy

Ah, I think I am wrong. Numpy's strides is exactly what I need, right?

http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.strides.html

so I just take *any*  numpy array, as long as it has the right dtype,
and then just pass the pointer to .data and strides to the opengl
function.

Very nice.

Ondrej

From dalcinl at gmail.com  Wed Jul  1 03:27:26 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 30 Jun 2009 22:27:26 -0300
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
	<e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
	<85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
Message-ID: <e7ba66e40906301827g371b7603mc07a392fce3bacce@mail.gmail.com>

On Tue, Jun 30, 2009 at 10:07 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>
> One more question. I am using ctypes because using it I can take a
> list of floats and create a C array of it:
>
> (c_float * len(ptr))(*ptr)
>
> from historical reasons (I am adapting some opengl example from
> pyglet, that used that approach --- since pyglet uses ctypes to wrap
> opengl).

You can even use the builtin array.array to convert a list of float to
an array of floats.. Of course. Next you will need to make some Python
C-API calls to get the pointer to the memory buffer. The second part
is what I'm currently doing in mpi4py, and it is working from Py2.3 to
Py3.1

>
>I use cython, so I think a better way is to use numpy?
>

If the numpy dependency is not a problem, perhaps this is the better way.

> opengl supports strides, but it's just one number, e.g. it has to be
> the same for all rows (as opposed to numpy). Is it safe to just copy
> the numpy array, to make sure it's contiguous and then just use the
> pointer to the .data member of ndarray? That's what I always do, but I
> don't know if this is the safe or best approach.
>

Well, not really sure what OpenGL wants... you can look at petsc4py
sources to the NumPy C-API calls I'm using (more specifically, the
flags I'm passing)... These NumPy C-API calls return a new
(contiguous) array by copying only if required.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Jul  1 03:30:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 30 Jun 2009 22:30:05 -0300
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <85b5c3130906301814m369a63d8x96a4cbfa59b7bfe7@mail.gmail.com>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
	<e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
	<85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
	<85b5c3130906301814m369a63d8x96a4cbfa59b7bfe7@mail.gmail.com>
Message-ID: <e7ba66e40906301830i10fb7367id1c3f73cae522abf@mail.gmail.com>

On Tue, Jun 30, 2009 at 10:14 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>
> Ah, I think I am wrong. Numpy's strides is exactly what I need, right?
>
> http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.strides.html
>
> so I just take *any* ?numpy array, as long as it has the right dtype,
> and then just pass the pointer to .data and strides to the opengl
> function.
>

Perhaps it works... Just in case, test it with a numpy array using
Fortran ordering, perhaps OpenGL does not like it?

BTW, do arrays with more than two dimensions make sense in OpenGL? Do
you know exactly what numpy strides mean (any nd-array have
nd-strides, even the contiguous ones) ?




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ondrej at certik.cz  Wed Jul  1 04:08:11 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 30 Jun 2009 20:08:11 -0600
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <e7ba66e40906301830i10fb7367id1c3f73cae522abf@mail.gmail.com>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
	<e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
	<85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
	<85b5c3130906301814m369a63d8x96a4cbfa59b7bfe7@mail.gmail.com>
	<e7ba66e40906301830i10fb7367id1c3f73cae522abf@mail.gmail.com>
Message-ID: <85b5c3130906301908u3e25a145x7329cd7b61870776@mail.gmail.com>

On Tue, Jun 30, 2009 at 7:30 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> On Tue, Jun 30, 2009 at 10:14 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>>
>> Ah, I think I am wrong. Numpy's strides is exactly what I need, right?
>>
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.strides.html
>>
>> so I just take *any* ?numpy array, as long as it has the right dtype,
>> and then just pass the pointer to .data and strides to the opengl
>> function.
>>
>
> Perhaps it works... Just in case, test it with a numpy array using
> Fortran ordering, perhaps OpenGL does not like it?

So far all functions in opengl use 1D arrays only.

>
> BTW, do arrays with more than two dimensions make sense in OpenGL? Do
> you know exactly what numpy strides mean (any nd-array have
> nd-strides, even the contiguous ones) ?

Yes, a stride is the number of bytes you need to move your pointer to
get to the next element in the array. opengl only accepts 1D arrays,
but it *does* accept a stride.

Can it happen, that a 1D numpy array has stride which is bigger than
sizeof(dtype)? so far I didn't manage to create one.


Ondrej

From robertwb at math.washington.edu  Wed Jul  1 06:11:18 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Jun 2009 21:11:18 -0700
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <85b5c3130906301908u3e25a145x7329cd7b61870776@mail.gmail.com>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
	<e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
	<85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
	<85b5c3130906301814m369a63d8x96a4cbfa59b7bfe7@mail.gmail.com>
	<e7ba66e40906301830i10fb7367id1c3f73cae522abf@mail.gmail.com>
	<85b5c3130906301908u3e25a145x7329cd7b61870776@mail.gmail.com>
Message-ID: <7064A370-662D-4177-90EA-635466B49C5C@math.washington.edu>

On Jun 30, 2009, at 7:08 PM, Ondrej Certik wrote:

> On Tue, Jun 30, 2009 at 7:30 PM, Lisandro Dalcin<dalcinl at gmail.com>  
> wrote:
>> On Tue, Jun 30, 2009 at 10:14 PM, Ondrej Certik<ondrej at certik.cz>  
>> wrote:
>>>
>>> Ah, I think I am wrong. Numpy's strides is exactly what I need,  
>>> right?
>>>
>>> http://docs.scipy.org/doc/numpy/reference/generated/ 
>>> numpy.ndarray.strides.html
>>>
>>> so I just take *any*  numpy array, as long as it has the right  
>>> dtype,
>>> and then just pass the pointer to .data and strides to the opengl
>>> function.
>>>
>>
>> Perhaps it works... Just in case, test it with a numpy array using
>> Fortran ordering, perhaps OpenGL does not like it?
>
> So far all functions in opengl use 1D arrays only.
>
>>
>> BTW, do arrays with more than two dimensions make sense in OpenGL? Do
>> you know exactly what numpy strides mean (any nd-array have
>> nd-strides, even the contiguous ones) ?
>
> Yes, a stride is the number of bytes you need to move your pointer to
> get to the next element in the array. opengl only accepts 1D arrays,
> but it *does* accept a stride.
>
> Can it happen, that a 1D numpy array has stride which is bigger than
> sizeof(dtype)? so far I didn't manage to create one.

Yes.

sage: a = numpy.zeros(10)[::5]
sage: a.strides
(40,)

- Robert

From ondrej at certik.cz  Wed Jul  1 06:24:15 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 30 Jun 2009 22:24:15 -0600
Subject: [Cython] accessing ctype array's b_ptr
In-Reply-To: <7064A370-662D-4177-90EA-635466B49C5C@math.washington.edu>
References: <85b5c3130906301726r5a0ddb36xc00fd0fa441cad9b@mail.gmail.com>
	<e7ba66e40906301742l20d805a4u21518d5f4224be21@mail.gmail.com>
	<85b5c3130906301807r7902d73andddff4397cc846ec@mail.gmail.com>
	<85b5c3130906301814m369a63d8x96a4cbfa59b7bfe7@mail.gmail.com>
	<e7ba66e40906301830i10fb7367id1c3f73cae522abf@mail.gmail.com>
	<85b5c3130906301908u3e25a145x7329cd7b61870776@mail.gmail.com>
	<7064A370-662D-4177-90EA-635466B49C5C@math.washington.edu>
Message-ID: <85b5c3130906302124j7f570667ybc1645cdae6bbe69@mail.gmail.com>

On Tue, Jun 30, 2009 at 10:11 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 30, 2009, at 7:08 PM, Ondrej Certik wrote:
>
>> On Tue, Jun 30, 2009 at 7:30 PM, Lisandro Dalcin<dalcinl at gmail.com>
>> wrote:
>>> On Tue, Jun 30, 2009 at 10:14 PM, Ondrej Certik<ondrej at certik.cz>
>>> wrote:
>>>>
>>>> Ah, I think I am wrong. Numpy's strides is exactly what I need,
>>>> right?
>>>>
>>>> http://docs.scipy.org/doc/numpy/reference/generated/
>>>> numpy.ndarray.strides.html
>>>>
>>>> so I just take *any* ?numpy array, as long as it has the right
>>>> dtype,
>>>> and then just pass the pointer to .data and strides to the opengl
>>>> function.
>>>>
>>>
>>> Perhaps it works... Just in case, test it with a numpy array using
>>> Fortran ordering, perhaps OpenGL does not like it?
>>
>> So far all functions in opengl use 1D arrays only.
>>
>>>
>>> BTW, do arrays with more than two dimensions make sense in OpenGL? Do
>>> you know exactly what numpy strides mean (any nd-array have
>>> nd-strides, even the contiguous ones) ?
>>
>> Yes, a stride is the number of bytes you need to move your pointer to
>> get to the next element in the array. opengl only accepts 1D arrays,
>> but it *does* accept a stride.
>>
>> Can it happen, that a 1D numpy array has stride which is bigger than
>> sizeof(dtype)? so far I didn't manage to create one.
>
> Yes.
>
> sage: a = numpy.zeros(10)[::5]
> sage: a.strides
> (40,)

Cool example, thanks for that, I'll use it for tests.

Ondrej

From seb.binet at gmail.com  Wed Jul  1 09:02:18 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Wed, 1 Jul 2009 09:02:18 +0200
Subject: [Cython] Search paths for definition files
In-Reply-To: <e7ba66e40906301443p7aeecec2o587584449fec6db7@mail.gmail.com>
References: <42c0ab790906301337u7f8c4f9rdcbf60f99a4f74b9@mail.gmail.com>
	<e7ba66e40906301443p7aeecec2o587584449fec6db7@mail.gmail.com>
Message-ID: <200907010902.19531.binet@cern.ch>

hi,

On Tuesday 30 June 2009 23:43:00 Lisandro Dalcin wrote:
> On Tue, Jun 30, 2009 at 5:37 PM, Elliott
>
> Slaughter<elliottslaughter at gmail.com> wrote:
> > Hi,
> >
> > The manual states that "the Cython compiler searches for a file called
> > modulename.pxd along the search path for include files, as specified by
> > -I command line options" [1].
>
> A good thing about open source projects is that the best
> docummentation are the actual sources :-)
> http://hg.cython.org/cython-devel/file/tip/Cython/Distutils/extension.py
>
> Try this:
>
> from Cython.Distutils.extension import Extension
> ...
> Extension('foo', ['foo.pyx'], pyrex_include_dirs=['dir1', 'dir2', 'dir3'])
>
>
> Other way (not recommended, not portable for POSIX/Windows):
> add a setup.cfg file alongside your setup.py file, and write inside it
> (use ";" to separate on Windows):

I am a bit surprised by this non-portable behaviour.
is it a (stdlib) distutils overlook or a Cython.Distutils one ?

IIRC one can standardize on the POSIX syntax for path-separators and friends, 
and have the posixpath module do the conversion for us:

>>> import posixpath
>>> assert posixpath.pathsep == ':'
>>> assert posixpath.sep == '/'

so always using the POSIX syntax in xyz.cfg files would be portable.

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From sccolbert at gmail.com  Wed Jul  1 19:52:52 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 1 Jul 2009 13:52:52 -0400
Subject: [Cython] how would I wrap something like this in a .pxd file
Message-ID: <7f014ea60907011052l7bee3598w23b014def6222b01@mail.gmail.com>

In wrapping a C library, I come across many #define statements that I
need to use elsewhere. When they simply define a constant, I use an
anonymous enum and all is well.

But how would I wrap something like this that shown up in a header
file that i'm cdef extern'ing from:

#define CV_TREE_NODE_FIELDS(node_type)                               \

    int       flags;                \

    int       header_size;          \

    struct    node_type* h_prev;      \

    struct    node_type* h_next;     \

    struct    node_type* v_prev;       \

    struct    node_type* v_next ;


or something like this:

#define CV_IS_STORAGE(storage)  \

    ((storage) != NULL &&       \

    (((CvMemStorage*)(storage))->signature & CV_MAGIC_MASK) ==
CV_STORAGE_MAGIC_VAL)




Thanks and Cheers!

Chris

From sccolbert at gmail.com  Wed Jul  1 20:24:00 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 1 Jul 2009 14:24:00 -0400
Subject: [Cython] how would I wrap something like this in a .pxd file
In-Reply-To: <7f014ea60907011052l7bee3598w23b014def6222b01@mail.gmail.com>
References: <7f014ea60907011052l7bee3598w23b014def6222b01@mail.gmail.com>
Message-ID: <7f014ea60907011124u3ae6d4bak9cc7406f969e3dc6@mail.gmail.com>

i just saw this line in the docs:

6. If the header file defines a function using a macro, declare it as
though it were an ordinary function, with appropriate argument and
result types.

So i'll start there.

Sorry for the bogus post.


On Wed, Jul 1, 2009 at 1:52 PM, Chris Colbert<sccolbert at gmail.com> wrote:
> In wrapping a C library, I come across many #define statements that I
> need to use elsewhere. When they simply define a constant, I use an
> anonymous enum and all is well.
>
> But how would I wrap something like this that shown up in a header
> file that i'm cdef extern'ing from:
>
> #define CV_TREE_NODE_FIELDS(node_type) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \
>
> ? ?int ? ? ? flags; ? ? ? ? ? ? ? ?\
>
> ? ?int ? ? ? header_size; ? ? ? ? ?\
>
> ? ?struct ? ?node_type* h_prev; ? ? ?\
>
> ? ?struct ? ?node_type* h_next; ? ? \
>
> ? ?struct ? ?node_type* v_prev; ? ? ? \
>
> ? ?struct ? ?node_type* v_next ;
>
>
> or something like this:
>
> #define CV_IS_STORAGE(storage) ?\
>
> ? ?((storage) != NULL && ? ? ? \
>
> ? ?(((CvMemStorage*)(storage))->signature & CV_MAGIC_MASK) ==
> CV_STORAGE_MAGIC_VAL)
>
>
>
>
> Thanks and Cheers!
>
> Chris
>

From dalcinl at gmail.com  Wed Jul  1 20:57:12 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Jul 2009 15:57:12 -0300
Subject: [Cython] Fwd:  Search paths for definition files
In-Reply-To: <e7ba66e40907011156m2084606ai237058f574f39477@mail.gmail.com>
References: <42c0ab790906301337u7f8c4f9rdcbf60f99a4f74b9@mail.gmail.com>
	<e7ba66e40906301443p7aeecec2o587584449fec6db7@mail.gmail.com>
	<200907010902.19531.binet@cern.ch>
	<e7ba66e40907011156m2084606ai237058f574f39477@mail.gmail.com>
Message-ID: <e7ba66e40907011157l7fbc0c53n425745de22dc2f4e@mail.gmail.com>

---------- Forwarded message ----------
From: Lisandro Dalcin <dalcinl at gmail.com>
Date: Wed, Jul 1, 2009 at 3:56 PM
Subject: Re: [Cython] Search paths for definition files
To: Sebastien Binet <seb.binet at gmail.com>


On Wed, Jul 1, 2009 at 4:02 AM, Sebastien Binet<seb.binet at gmail.com> wrote:
>
>> Other way (not recommended, not portable for POSIX/Windows):
>> add a setup.cfg file alongside your setup.py file, and write inside it
>> (use ";" to separate on Windows):
>
> I am a bit surprised by this non-portable behaviour.
> is it a (stdlib) distutils overlook or a Cython.Distutils one ?
>

It is (stdlib) distutils behaviour, and Cython.Distutils just followed
the rules... Though I would not say that is an overlook, but a
feature...

>
> IIRC one can standardize on the POSIX syntax for path-separators and friends,
> and have the posixpath module do the conversion for us:
>
>>>> import posixpath
>>>> assert posixpath.pathsep == ':'
>>>> assert posixpath.sep == '/'
>
> so always using the POSIX syntax in xyz.cfg files would be portable.
>

But what would happen if you want to pass a Windows path that has a
":" inside the name? That would be a valid Windows path...

IMHO, in the particular case of setup.cfg files, they should be used
as a last resort for letting users to pass include/library paths or
even compiler flags... distutils should not second-guess users about
pathnames...

Moreover, any Python package that export some sort of API, let say C
header files or Cython pxd files, should do what numpy does:

$ python -c 'import numpy; print numpy.get_include()'
/usr/lib/python2.5/site-packages/numpy/core/include

I'm doing that for my own projects:

$ python -c 'import mpi4py; print mpi4py.get_include()'
/u/dalcinl/lib/python/mpi4py/include

$ python -c 'import petsc4py; print petsc4py.get_include()'
/u/dalcinl/lib/python/petsc4py/include

$ python -c 'import slepc4py; print slepc4py.get_include()'
/u/dalcinl/lib/python/slepc4py/include


Then, if you want to use C headers and Cython pxd's, you have a
programatic way to pass things in your setup.py files...

--
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Jul  1 21:01:38 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Jul 2009 16:01:38 -0300
Subject: [Cython] how would I wrap something like this in a .pxd file
In-Reply-To: <7f014ea60907011124u3ae6d4bak9cc7406f969e3dc6@mail.gmail.com>
References: <7f014ea60907011052l7bee3598w23b014def6222b01@mail.gmail.com>
	<7f014ea60907011124u3ae6d4bak9cc7406f969e3dc6@mail.gmail.com>
Message-ID: <e7ba66e40907011201v5c412889hac99eb845eae4dc8@mail.gmail.com>

On Wed, Jul 1, 2009 at 3:24 PM, Chris Colbert<sccolbert at gmail.com> wrote:
> i just saw this line in the docs:
>
> 6. If the header file defines a function using a macro, declare it as
> though it were an ordinary function, with appropriate argument and
> result types.
>
> So i'll start there.
>
> Sorry for the bogus post.
>

IMHO, your post was not bogus... Making Cython use such kind of macros
(which resembles a C++ template emulation) can be not so easy...
Perhaps the easiest way would be to define a helper struct (one of
each node_type) in some C header, and then include it using cdef
extern from "mydefs.h": ...


>
> On Wed, Jul 1, 2009 at 1:52 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>> In wrapping a C library, I come across many #define statements that I
>> need to use elsewhere. When they simply define a constant, I use an
>> anonymous enum and all is well.
>>
>> But how would I wrap something like this that shown up in a header
>> file that i'm cdef extern'ing from:
>>
>> #define CV_TREE_NODE_FIELDS(node_type) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \
>>
>> ? ?int ? ? ? flags; ? ? ? ? ? ? ? ?\
>>
>> ? ?int ? ? ? header_size; ? ? ? ? ?\
>>
>> ? ?struct ? ?node_type* h_prev; ? ? ?\
>>
>> ? ?struct ? ?node_type* h_next; ? ? \
>>
>> ? ?struct ? ?node_type* v_prev; ? ? ? \
>>
>> ? ?struct ? ?node_type* v_next ;
>>
>>
>> or something like this:
>>
>> #define CV_IS_STORAGE(storage) ?\
>>
>> ? ?((storage) != NULL && ? ? ? \
>>
>> ? ?(((CvMemStorage*)(storage))->signature & CV_MAGIC_MASK) ==
>> CV_STORAGE_MAGIC_VAL)
>>
>>
>>
>>
>> Thanks and Cheers!
>>
>> Chris
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed Jul  1 21:53:59 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 1 Jul 2009 15:53:59 -0400
Subject: [Cython] how would I wrap something like this in a .pxd file
In-Reply-To: <e7ba66e40907011201v5c412889hac99eb845eae4dc8@mail.gmail.com>
References: <7f014ea60907011052l7bee3598w23b014def6222b01@mail.gmail.com>
	<7f014ea60907011124u3ae6d4bak9cc7406f969e3dc6@mail.gmail.com>
	<e7ba66e40907011201v5c412889hac99eb845eae4dc8@mail.gmail.com>
Message-ID: <7f014ea60907011253u732d6003oa0d5bc29645131d6@mail.gmail.com>

Thanks Lisandro,

I may just end up declaring each struct like this with a 'pass'

and then implementing the fields on the python side (this is how ctypes does it)

Cheers!

On Wed, Jul 1, 2009 at 3:01 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> On Wed, Jul 1, 2009 at 3:24 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>> i just saw this line in the docs:
>>
>> 6. If the header file defines a function using a macro, declare it as
>> though it were an ordinary function, with appropriate argument and
>> result types.
>>
>> So i'll start there.
>>
>> Sorry for the bogus post.
>>
>
> IMHO, your post was not bogus... Making Cython use such kind of macros
> (which resembles a C++ template emulation) can be not so easy...
> Perhaps the easiest way would be to define a helper struct (one of
> each node_type) in some C header, and then include it using cdef
> extern from "mydefs.h": ...
>
>
>>
>> On Wed, Jul 1, 2009 at 1:52 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>> In wrapping a C library, I come across many #define statements that I
>>> need to use elsewhere. When they simply define a constant, I use an
>>> anonymous enum and all is well.
>>>
>>> But how would I wrap something like this that shown up in a header
>>> file that i'm cdef extern'ing from:
>>>
>>> #define CV_TREE_NODE_FIELDS(node_type) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \
>>>
>>> ? ?int ? ? ? flags; ? ? ? ? ? ? ? ?\
>>>
>>> ? ?int ? ? ? header_size; ? ? ? ? ?\
>>>
>>> ? ?struct ? ?node_type* h_prev; ? ? ?\
>>>
>>> ? ?struct ? ?node_type* h_next; ? ? \
>>>
>>> ? ?struct ? ?node_type* v_prev; ? ? ? \
>>>
>>> ? ?struct ? ?node_type* v_next ;
>>>
>>>
>>> or something like this:
>>>
>>> #define CV_IS_STORAGE(storage) ?\
>>>
>>> ? ?((storage) != NULL && ? ? ? \
>>>
>>> ? ?(((CvMemStorage*)(storage))->signature & CV_MAGIC_MASK) ==
>>> CV_STORAGE_MAGIC_VAL)
>>>
>>>
>>>
>>>
>>> Thanks and Cheers!
>>>
>>> Chris
>>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From sccolbert at gmail.com  Wed Jul  1 23:35:40 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 1 Jul 2009 17:35:40 -0400
Subject: [Cython] how would I wrap something like this in a .pxd file
In-Reply-To: <7f014ea60907011253u732d6003oa0d5bc29645131d6@mail.gmail.com>
References: <7f014ea60907011052l7bee3598w23b014def6222b01@mail.gmail.com>
	<7f014ea60907011124u3ae6d4bak9cc7406f969e3dc6@mail.gmail.com>
	<e7ba66e40907011201v5c412889hac99eb845eae4dc8@mail.gmail.com>
	<7f014ea60907011253u732d6003oa0d5bc29645131d6@mail.gmail.com>
Message-ID: <7f014ea60907011435y437d791fy2e0c07b474f5993@mail.gmail.com>

what about functions defined in a macro where i wont know the return
type in advance?

such as:

#define  CV_NEXT_GRAPH_EDGE( edge, vertex )                              \

     (assert((edge)->vtx[0] == (vertex) || (edge)->vtx[1] == (vertex)),  \

      (edge)->next[(edge)->vtx[1] == (vertex)])

should I just implement this as a python function, or is there a way
to wrap it?

I would wrapping wouldn't be possible since the type isn't determined
till runtime...


On Wed, Jul 1, 2009 at 3:53 PM, Chris Colbert<sccolbert at gmail.com> wrote:
> Thanks Lisandro,
>
> I may just end up declaring each struct like this with a 'pass'
>
> and then implementing the fields on the python side (this is how ctypes does it)
>
> Cheers!
>
> On Wed, Jul 1, 2009 at 3:01 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
>> On Wed, Jul 1, 2009 at 3:24 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>> i just saw this line in the docs:
>>>
>>> 6. If the header file defines a function using a macro, declare it as
>>> though it were an ordinary function, with appropriate argument and
>>> result types.
>>>
>>> So i'll start there.
>>>
>>> Sorry for the bogus post.
>>>
>>
>> IMHO, your post was not bogus... Making Cython use such kind of macros
>> (which resembles a C++ template emulation) can be not so easy...
>> Perhaps the easiest way would be to define a helper struct (one of
>> each node_type) in some C header, and then include it using cdef
>> extern from "mydefs.h": ...
>>
>>
>>>
>>> On Wed, Jul 1, 2009 at 1:52 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>>> In wrapping a C library, I come across many #define statements that I
>>>> need to use elsewhere. When they simply define a constant, I use an
>>>> anonymous enum and all is well.
>>>>
>>>> But how would I wrap something like this that shown up in a header
>>>> file that i'm cdef extern'ing from:
>>>>
>>>> #define CV_TREE_NODE_FIELDS(node_type) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \
>>>>
>>>> ? ?int ? ? ? flags; ? ? ? ? ? ? ? ?\
>>>>
>>>> ? ?int ? ? ? header_size; ? ? ? ? ?\
>>>>
>>>> ? ?struct ? ?node_type* h_prev; ? ? ?\
>>>>
>>>> ? ?struct ? ?node_type* h_next; ? ? \
>>>>
>>>> ? ?struct ? ?node_type* v_prev; ? ? ? \
>>>>
>>>> ? ?struct ? ?node_type* v_next ;
>>>>
>>>>
>>>> or something like this:
>>>>
>>>> #define CV_IS_STORAGE(storage) ?\
>>>>
>>>> ? ?((storage) != NULL && ? ? ? \
>>>>
>>>> ? ?(((CvMemStorage*)(storage))->signature & CV_MAGIC_MASK) ==
>>>> CV_STORAGE_MAGIC_VAL)
>>>>
>>>>
>>>>
>>>>
>>>> Thanks and Cheers!
>>>>
>>>> Chris
>>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>

From sccolbert at gmail.com  Thu Jul  2 01:13:55 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 1 Jul 2009 19:13:55 -0400
Subject: [Cython] how to allow access to data in a C array?
Message-ID: <7f014ea60907011613g2587c330i2c7ec732efbb723a@mail.gmail.com>

I'm thinking this may be what the new array type is for, but i'm not sure.

say I have a C struct which I wrap as a class:


#mycfile.h
struct Foo
{
       int* data[32];
};

#my_dec_file.pxd
cdef exter from "mycfile.h":

     cdef struct Foo:
           int* data[32]

#my_cy_file.pxd
cimport my_dec_file

cdef class Foo:
       cdef my_dec_file.Foo* thisptr

#my_cy_file.pyx
cimport my_dec_file

cdef class Foo:
    property data:
    def _get_(self):
         return self.thisptr.data   <----i dont think this will work



what I want to be able to do in python is this:

<....get an instance of Foo  i.e. foo = Foo() ....>

b = foo.data[3:6]   # where b can be a numpy array or a list, but a
numpy array would be better




Further, is there a way to manipulate data like that directly via
python (i.e. without making a copy).

suppose I want to add the value 2 to every value in foo.data[:]

it would be nice to do (from python):

foo.data[:] + 2

Is there a way to implement that on the Cython side?

Cheers,

Chris

From dalcinl at gmail.com  Thu Jul  2 03:28:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 1 Jul 2009 22:28:15 -0300
Subject: [Cython] how would I wrap something like this in a .pxd file
In-Reply-To: <7f014ea60907011435y437d791fy2e0c07b474f5993@mail.gmail.com>
References: <7f014ea60907011052l7bee3598w23b014def6222b01@mail.gmail.com>
	<7f014ea60907011124u3ae6d4bak9cc7406f969e3dc6@mail.gmail.com>
	<e7ba66e40907011201v5c412889hac99eb845eae4dc8@mail.gmail.com>
	<7f014ea60907011253u732d6003oa0d5bc29645131d6@mail.gmail.com>
	<7f014ea60907011435y437d791fy2e0c07b474f5993@mail.gmail.com>
Message-ID: <e7ba66e40907011828k3359ce4s864d43fab537df7c@mail.gmail.com>

I still think that the safest thing would be to use a C header
defining functions with different names for each type you intend to
use.

Until Cython does not provide some support for C++-ish templatized
declarations, I cannot warranty you that thinks will not break if we
try to be smart and use these macros directly...


On Wed, Jul 1, 2009 at 6:35 PM, Chris Colbert<sccolbert at gmail.com> wrote:
> what about functions defined in a macro where i wont know the return
> type in advance?
>
> such as:
>
> #define ?CV_NEXT_GRAPH_EDGE( edge, vertex ) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?\
>
> ? ? (assert((edge)->vtx[0] == (vertex) || (edge)->vtx[1] == (vertex)), ?\
>
> ? ? ?(edge)->next[(edge)->vtx[1] == (vertex)])
>
> should I just implement this as a python function, or is there a way
> to wrap it?
>
> I would wrapping wouldn't be possible since the type isn't determined
> till runtime...
>
>
> On Wed, Jul 1, 2009 at 3:53 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>> Thanks Lisandro,
>>
>> I may just end up declaring each struct like this with a 'pass'
>>
>> and then implementing the fields on the python side (this is how ctypes does it)
>>
>> Cheers!
>>
>> On Wed, Jul 1, 2009 at 3:01 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
>>> On Wed, Jul 1, 2009 at 3:24 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>>> i just saw this line in the docs:
>>>>
>>>> 6. If the header file defines a function using a macro, declare it as
>>>> though it were an ordinary function, with appropriate argument and
>>>> result types.
>>>>
>>>> So i'll start there.
>>>>
>>>> Sorry for the bogus post.
>>>>
>>>
>>> IMHO, your post was not bogus... Making Cython use such kind of macros
>>> (which resembles a C++ template emulation) can be not so easy...
>>> Perhaps the easiest way would be to define a helper struct (one of
>>> each node_type) in some C header, and then include it using cdef
>>> extern from "mydefs.h": ...
>>>
>>>
>>>>
>>>> On Wed, Jul 1, 2009 at 1:52 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>>>> In wrapping a C library, I come across many #define statements that I
>>>>> need to use elsewhere. When they simply define a constant, I use an
>>>>> anonymous enum and all is well.
>>>>>
>>>>> But how would I wrap something like this that shown up in a header
>>>>> file that i'm cdef extern'ing from:
>>>>>
>>>>> #define CV_TREE_NODE_FIELDS(node_type) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \
>>>>>
>>>>> ? ?int ? ? ? flags; ? ? ? ? ? ? ? ?\
>>>>>
>>>>> ? ?int ? ? ? header_size; ? ? ? ? ?\
>>>>>
>>>>> ? ?struct ? ?node_type* h_prev; ? ? ?\
>>>>>
>>>>> ? ?struct ? ?node_type* h_next; ? ? \
>>>>>
>>>>> ? ?struct ? ?node_type* v_prev; ? ? ? \
>>>>>
>>>>> ? ?struct ? ?node_type* v_next ;
>>>>>
>>>>>
>>>>> or something like this:
>>>>>
>>>>> #define CV_IS_STORAGE(storage) ?\
>>>>>
>>>>> ? ?((storage) != NULL && ? ? ? \
>>>>>
>>>>> ? ?(((CvMemStorage*)(storage))->signature & CV_MAGIC_MASK) ==
>>>>> CV_STORAGE_MAGIC_VAL)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks and Cheers!
>>>>>
>>>>> Chris
>>>>>
>>>> _______________________________________________
>>>> Cython-dev mailing list
>>>> Cython-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>>
>>>
>>>
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Jul  2 06:32:43 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 02 Jul 2009 06:32:43 +0200
Subject: [Cython] how to allow access to data in a C array?
In-Reply-To: <7f014ea60907011613g2587c330i2c7ec732efbb723a@mail.gmail.com>
References: <7f014ea60907011613g2587c330i2c7ec732efbb723a@mail.gmail.com>
Message-ID: <4A4C386B.4060809@behnel.de>

Hi,

Chris Colbert wrote:
> I'm thinking this may be what the new array type is for, but i'm not sure.
> 
> say I have a C struct which I wrap as a class:
> 
> 
> #mycfile.h
> struct Foo
> {
>        int* data[32];
> };
> 
> #my_dec_file.pxd
> cdef exter from "mycfile.h":
> 
>      cdef struct Foo:
>            int* data[32]
> 
> #my_cy_file.pxd
> cimport my_dec_file
> 
> cdef class Foo:
>        cdef my_dec_file.Foo* thisptr
> 
> #my_cy_file.pyx
> cimport my_dec_file
> 
> cdef class Foo:
>     property data:
>     def _get_(self):
>          return self.thisptr.data   <----i dont think this will work

Right, it won't.


> what I want to be able to do in python is this:
> 
> <....get an instance of Foo  i.e. foo = Foo() ....>
> 
> b = foo.data[3:6]   # where b can be a numpy array or a list, but a
> numpy array would be better
> 
> Further, is there a way to manipulate data like that directly via
> python (i.e. without making a copy).
> 
> suppose I want to add the value 2 to every value in foo.data[:]
> 
> it would be nice to do (from python):
> 
> foo.data[:] + 2
> 
> Is there a way to implement that on the Cython side?

Sure, just return an object from the "data" property that wraps the array
and implements __setitem__ and __getitem__ appropriately. Not exactly
trivial, but not too hard either.

Once Cython has better array support, I guess this would simply become a
builtin feature.

Stefan

From sccolbert at gmail.com  Thu Jul  2 17:39:31 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 2 Jul 2009 11:39:31 -0400
Subject: [Cython] how to allow access to data in a C array?
In-Reply-To: <4A4C386B.4060809@behnel.de>
References: <7f014ea60907011613g2587c330i2c7ec732efbb723a@mail.gmail.com>
	<4A4C386B.4060809@behnel.de>
Message-ID: <7f014ea60907020839u3e1450f1w8d39b321fc11c42f@mail.gmail.com>

Thanks Stefan!

On Thu, Jul 2, 2009 at 12:32 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
> Hi,
>
> Chris Colbert wrote:
>> I'm thinking this may be what the new array type is for, but i'm not sure.
>>
>> say I have a C struct which I wrap as a class:
>>
>>
>> #mycfile.h
>> struct Foo
>> {
>> ? ? ? ?int* data[32];
>> };
>>
>> #my_dec_file.pxd
>> cdef exter from "mycfile.h":
>>
>> ? ? ?cdef struct Foo:
>> ? ? ? ? ? ?int* data[32]
>>
>> #my_cy_file.pxd
>> cimport my_dec_file
>>
>> cdef class Foo:
>> ? ? ? ?cdef my_dec_file.Foo* thisptr
>>
>> #my_cy_file.pyx
>> cimport my_dec_file
>>
>> cdef class Foo:
>> ? ? property data:
>> ? ? def _get_(self):
>> ? ? ? ? ?return self.thisptr.data ? <----i dont think this will work
>
> Right, it won't.
>
>
>> what I want to be able to do in python is this:
>>
>> <....get an instance of Foo ?i.e. foo = Foo() ....>
>>
>> b = foo.data[3:6] ? # where b can be a numpy array or a list, but a
>> numpy array would be better
>>
>> Further, is there a way to manipulate data like that directly via
>> python (i.e. without making a copy).
>>
>> suppose I want to add the value 2 to every value in foo.data[:]
>>
>> it would be nice to do (from python):
>>
>> foo.data[:] + 2
>>
>> Is there a way to implement that on the Cython side?
>
> Sure, just return an object from the "data" property that wraps the array
> and implements __setitem__ and __getitem__ appropriately. Not exactly
> trivial, but not too hard either.
>
> Once Cython has better array support, I guess this would simply become a
> builtin feature.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Thu Jul  2 19:30:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 02 Jul 2009 19:30:51 +0200
Subject: [Cython] To raise or not to raise
In-Reply-To: <c266a7930907012152k5a02e675l823ccecf5990b4c3@mail.gmail.com>
References: <c266a7930907012152k5a02e675l823ccecf5990b4c3@mail.gmail.com>
Message-ID: <4A4CEECB.8060507@student.matnat.uio.no>

I'm taking this to the list...

Kurt Smith wrote:
> Dag,
> 
> When an exception or an error needs to be generated, what are the
> guidelines?  It seems Errors.error() is normally called, although an
> exception isn't raised at this point, & compilation continues -- the
> errors are collected and reported at the end of everything then.  A
> few other times I see a 'raise CompileError' or 'raise InternalError'
> instead.  When should these be raised instead of just calling
> Errors.error()?  I seem to recall that InternalError is an error with
> the compiler itself, while CompileError is raised when there's
> something wrong with the source.

I wouldn't trust these rules to be followed everywhere, but:

In general, exceptions should not be raised, but compilation go on while 
"ignoring" what caused the error.

In e.g. declaration analysis, the right procedure is to call 
Errors.error and set "self.type" to PyrexTypes.error_type (or similar). 
Then code which need to use the type should check if it is error_type 
and if so, return early (i.e. "ignore" this part of the tree; 
compilation will be aborted later anyway).

Some places though an exception may still be the right thing -- if it is 
trapped further out in the node tree. I.e. to recover from an error in a 
subnode it might be proper to use exception to go back to a parent node, 
then that parent node is skipped and its sibling processed next.

Also there's the problem that currently some error reporting is done in 
code generation. This should NOT be done for new code, as a long-term 
goal is to have a stage in the pipeline where we say "everything is 
parsed and is correct; from now on, pipeline transforms can assume no 
errors have occurred".


> 
> Sometimes when simply calling Errors.error(), the rest of the code
> raises another exeption, like AttributeError because things are in an
> inconsistent state after the error() call.  It would be nice to ensure
> that everything stops at the error() call, but then cython won't
> continue past the first error then which makes testing a pain.
> 
> I'm almost done with the MemoryviewType initialization -- just putting
> in the (many) testcases right now & doing some sanity checks.  I'll
> put it up for review tomorrow.
> 
> Kurt


-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Jul  3 05:32:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 2 Jul 2009 20:32:43 -0700
Subject: [Cython] To raise or not to raise
In-Reply-To: <4A4CEECB.8060507@student.matnat.uio.no>
References: <c266a7930907012152k5a02e675l823ccecf5990b4c3@mail.gmail.com>
	<4A4CEECB.8060507@student.matnat.uio.no>
Message-ID: <EA79B6D8-FF1E-4E1C-B895-434A39EB3D66@math.washington.edu>

On Jul 2, 2009, at 10:30 AM, Dag Sverre Seljebotn wrote:

> I'm taking this to the list...
>
> Kurt Smith wrote:
>> Dag,
>>
>> When an exception or an error needs to be generated, what are the
>> guidelines?  It seems Errors.error() is normally called, although an
>> exception isn't raised at this point, & compilation continues -- the
>> errors are collected and reported at the end of everything then.  A
>> few other times I see a 'raise CompileError' or 'raise InternalError'
>> instead.  When should these be raised instead of just calling
>> Errors.error()?  I seem to recall that InternalError is an error with
>> the compiler itself, while CompileError is raised when there's
>> something wrong with the source.
>
> I wouldn't trust these rules to be followed everywhere, but:
>
> In general, exceptions should not be raised, but compilation go on  
> while
> "ignoring" what caused the error.
>
> In e.g. declaration analysis, the right procedure is to call
> Errors.error and set "self.type" to PyrexTypes.error_type (or  
> similar).
> Then code which need to use the type should check if it is error_type
> and if so, return early (i.e. "ignore" this part of the tree;
> compilation will be aborted later anyway).
>
> Some places though an exception may still be the right thing -- if  
> it is
> trapped further out in the node tree. I.e. to recover from an error  
> in a
> subnode it might be proper to use exception to go back to a parent  
> node,
> then that parent node is skipped and its sibling processed next.

I can't think of any cases, but this could be acceptable as well.  
Ideally any such errors thrown would be caught before hitting the top  
level.

In general, my attitude is to raise an exception when you get to a  
point that something has gone irrevocably wrong. If this is due to  
bad user code, typically the right thing to do is use Errors.error  
and continue (perhaps skipping hte node, setting things like type to  
dummy variables. You can raise InternalError which will get ignored  
if errors were encountered at an earlier stage, but again, if you can  
continue it'd be better try and do so.

> Also there's the problem that currently some error reporting is  
> done in
> code generation. This should NOT be done for new code, as a long-term
> goal is to have a stage in the pipeline where we say "everything is
> parsed and is correct; from now on, pipeline transforms can assume no
> errors have occurred".

+1

>> Sometimes when simply calling Errors.error(), the rest of the code
>> raises another exeption, like AttributeError because things are in an
>> inconsistent state after the error() call.  It would be nice to  
>> ensure
>> that everything stops at the error() call, but then cython won't
>> continue past the first error then which makes testing a pain.


We could catch everything and ignore it if Errors.error() had been  
previously, but even though it makes developing a bit more difficult  
(in the short run, probably easier in the long run), and it is very  
useful when writing Cython code to have it go as far as possible  
before giving up so I'm -1 on that idea. (At the other extreme,  
imagine the pain writing code if the compiler always stopped at the  
first parse error it hit.

- Robert


From dalcinl at gmail.com  Fri Jul  3 20:00:38 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 3 Jul 2009 15:00:38 -0300
Subject: [Cython] new idea for typedef integrals
Message-ID: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>

As posted some time ago, here you have some work (BTW, awaiting
review) http://trac.cython.org/cython_trac/ticket/333

But recently I was thinking on something... What about a parser
modification to accept code like the below one?:

cdef extern from *: # note the "?" annotation
    ctypedef singed MyIntType1?
    ctypedef singed MyIntType2? "something"


Perhaps the uses knows that some typedef ACTUALLY match the extern
one. And then, we should not second-guess the user.

Moreover, this is fully backwards... The new unknow-size handling code
will only be used if a typedef is explicitly marked as an unknow-size
type.

What do you think?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Sat Jul  4 03:25:25 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 04 Jul 2009 13:25:25 +1200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
Message-ID: <4A4EAF85.3070003@canterbury.ac.nz>

Lisandro Dalcin wrote:

> cdef extern from *: # note the "?" annotation
>     ctypedef singed MyIntType1?
                ^^^^^^

Is this a special data type for CPUs that are
running too hot?

-- 
Greg

From dalcinl at gmail.com  Sat Jul  4 03:42:23 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 3 Jul 2009 22:42:23 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A4EAF85.3070003@canterbury.ac.nz>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
Message-ID: <e7ba66e40907031842v4734b6d2oa1498b55ad29f7d6@mail.gmail.com>

On Fri, Jul 3, 2009 at 10:25 PM, Greg Ewing<greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> cdef extern from *: # note the "?" annotation
>> ? ? ctypedef singed MyIntType1?
> ? ? ? ? ? ? ? ?^^^^^^
>
> Is this a special data type for CPUs that are
> running too hot?
>

Thanks, Greg... Now I know a new English word (I've never seen it before)...

> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Sat Jul  4 07:55:14 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jul 2009 07:55:14 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907031842v4734b6d2oa1498b55ad29f7d6@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>
	<e7ba66e40907031842v4734b6d2oa1498b55ad29f7d6@mail.gmail.com>
Message-ID: <4A4EEEC2.40105@behnel.de>


Lisandro Dalcin wrote:
> On Fri, Jul 3, 2009 at 10:25 PM, Greg Ewing wrote:
>> Lisandro Dalcin wrote:
>>
>>> cdef extern from *: # note the "?" annotation
>>>     ctypedef singed MyIntType1?
>>                ^^^^^^
>>
>> Is this a special data type for CPUs that are
>> running too hot?
> 
> Now I know a new English word (I've never seen it before)...

Me neither. It's funny how some German words got mangled since our last
major hike to that little North See island. I actually had to look up the
dates, that was somewhere around 450 AD...

http://de.wikipedia.org/w/index.php?title=Datei:Karte_v?lkerwanderung.jpg&filetimestamp=20051213124036

Stefan

From stefan_ml at behnel.de  Sat Jul  4 12:21:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jul 2009 12:21:40 +0200
Subject: [Cython] __future__.division on constants and C integers
Message-ID: <4A4F2D34.20703@behnel.de>

Hi,

in the current implementation, __future__.division only applies when
dividing non-constant Python values, i.e.

	from __future__ import division

	a, b = 1, 2
	print a / b

will print 0.5, whereas

	from __future__ import division

	print 1 / 2

will print 0 as the calculation happens in C space (or during constant
folding, which currently gives the same result).

http://trac.cython.org/cython_trac/ticket/343

My fix would be to handle future division in the parser (instead of code
generation, as it is now), and to let the parser directly replace "/" by
"//" when future division is *not* enabled, so that the "/" operator can
always have true division semantics when it appears inside the parse tree.

I know that these things tend to be controversial, but explicitly stating
in your code that you want "future division", and then not getting it
simply is a bug. If you want integer division, there's the "//" operator.

Now, the above is clear for the Python case. But what about C integers? If
you do this:

	from __future__ import division

	cdef int result = 1 / 2
	print result

Cython would have to do the equivalent of this C code to be correct:

	int result = 1 / ((float) 2);

but given that we generally do not fold float constants, would constant
folding be allowed in this case? And what would be the result type? Imagine
this case:

	cdef int result = 1 / (1 / 2)

Running this with true division would assign 2, whereas floor division
would result in a division by zero error. Same for Python:

  Python 3.1 (r31:73572, Jun 28 2009, 21:07:35)
  [GCC 4.3.2] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> 1 / (1/2)
  2.0

  Python 2.6.2 (r262:71600, Apr 17 2009, 11:29:30)
  [GCC 4.3.2] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> 1/(1/2)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  ZeroDivisionError: integer division or modulo by zero

Another example - would this get optimised:

	cdef int i
	for i in range(1,10,4/2):
	    pass

(note that the for-in-range optimisation needs to know if the step is
positive or negative - not folding the above float value into "2.0" will
prevent that)

This similarly applies to non-constants:

	cdef int a = 1
	cdef int result = a / 2
	print result

would currently print 0 as the calculation happens in C space again. That's
ok when the result is a C integer (as that would result in doing floor
division anyway), but if the result is a Python value:

	cdef int a = 1
	py_result = a / 2
	print py_result

Calculating the result in C space would not yield the expected Python result.

My opinion on this is: if you request true division semantics by importing
__future__.division, you should get true division in all places where you
use the "/" operator.

Comments?

Stefan


From stefan_ml at behnel.de  Sat Jul  4 18:01:54 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jul 2009 18:01:54 +0200
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A4F2D34.20703@behnel.de>
References: <4A4F2D34.20703@behnel.de>
Message-ID: <4A4F7CF2.2060005@behnel.de>

Hi,

quick update:

Stefan Behnel wrote:
> My fix would be to handle future division in the parser (instead of code
> generation, as it is now), and to let the parser directly replace "/" by
> "//" when future division is *not* enabled, so that the "/" operator can
> always have true division semantics when it appears inside the parse tree.

That won't work, since "/" and "//" do different things to integers and
floats. So this can't be decided before the type analysis phase, which runs
after the constant folding phase...

I guess all we can do for now is to ignore "/" during constant folding and
then handle it when we know the operand types...

Stefan


From kwmsmith at gmail.com  Sat Jul  4 20:31:33 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sat, 4 Jul 2009 13:31:33 -0500
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A4EEEC2.40105@behnel.de>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<e7ba66e40907031842v4734b6d2oa1498b55ad29f7d6@mail.gmail.com>
	<4A4EEEC2.40105@behnel.de>
Message-ID: <c266a7930907041131q195476cfnc9c10b5e6602f78a@mail.gmail.com>

On Sat, Jul 4, 2009 at 12:55 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
> > On Fri, Jul 3, 2009 at 10:25 PM, Greg Ewing wrote:
> >> Lisandro Dalcin wrote:
> >>
> >>> cdef extern from *: # note the "?" annotation
> >>>     ctypedef singed MyIntType1?
> >>                ^^^^^^
> >>
> >> Is this a special data type for CPUs that are
> >> running too hot?
> >
> > Now I know a new English word (I've never seen it before)...
>
> Me neither. It's funny how some German words got mangled since our last
> major hike to that little North See island. I actually had to look up the
                                  ^^^

Oh, the irony ;-)


> dates, that was somewhere around 450 AD...
>
> http://de.wikipedia.org/w/index.php?title=Datei:Karte_v?lkerwanderung.jpg&filetimestamp=20051213124036
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net

> http://codespeak.net/mailman/listinfo/cython-dev

From dagss at student.matnat.uio.no  Sat Jul  4 21:32:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 04 Jul 2009 21:32:15 +0200
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A4F2D34.20703@behnel.de>
References: <4A4F2D34.20703@behnel.de>
Message-ID: <4A4FAE3F.703@student.matnat.uio.no>

Stefan Behnel wrote:
> My opinion on this is: if you request true division semantics by importing
> __future__.division, you should get true division in all places where you
> use the "/" operator.
> 
> Comments?

Thanks for raising this issue. Without diving into the details myself, 
I'm +1 -- we've pretty much decided to have Python semantics on division.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Jul  4 21:33:37 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 04 Jul 2009 21:33:37 +0200
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A4F7CF2.2060005@behnel.de>
References: <4A4F2D34.20703@behnel.de> <4A4F7CF2.2060005@behnel.de>
Message-ID: <4A4FAE91.10202@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> quick update:
> 
> Stefan Behnel wrote:
>> My fix would be to handle future division in the parser (instead of code
>> generation, as it is now), and to let the parser directly replace "/" by
>> "//" when future division is *not* enabled, so that the "/" operator can
>> always have true division semantics when it appears inside the parse tree.
> 
> That won't work, since "/" and "//" do different things to integers and
> floats. So this can't be decided before the type analysis phase, which runs
> after the constant folding phase...
> 
> I guess all we can do for now is to ignore "/" during constant folding and
> then handle it when we know the operand types...

Why is compile time folding done before expression analysis? It would 
seem sensible that as compile time folding shouldn't affect semantics (? 
at least expression type semantics?) it can just as well be done after 
expression analysis.

(I think there was a reason but I forgot.)

-- 
Dag Sverre

From greg.ewing at canterbury.ac.nz  Sun Jul  5 03:12:00 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 05 Jul 2009 13:12:00 +1200
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A4F7CF2.2060005@behnel.de>
References: <4A4F2D34.20703@behnel.de> <4A4F7CF2.2060005@behnel.de>
Message-ID: <4A4FFDE0.2000204@canterbury.ac.nz>

Stefan Behnel wrote:

> I guess all we can do for now is to ignore "/" during constant folding and
> then handle it when we know the operand types...

You could replace "/" with an internal token
meaning "type-sensitive division".

-- 
Greg

From stefan_ml at behnel.de  Sun Jul  5 10:20:57 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 05 Jul 2009 10:20:57 +0200
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A4FAE91.10202@student.matnat.uio.no>
References: <4A4F2D34.20703@behnel.de> <4A4F7CF2.2060005@behnel.de>
	<4A4FAE91.10202@student.matnat.uio.no>
Message-ID: <4A506269.9080500@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Why is compile time folding done before expression analysis?

;) Because you proposed to move it there at the time? The alternative was
to split the type analysis phase to make some type information available
before the 'real' analysis.


> It would 
> seem sensible that as compile time folding shouldn't affect semantics (? 
> at least expression type semantics?) it can just as well be done after 
> expression analysis.

I did that to fix places where the "result_code" was used during the type
analysis phase. It's also great to know that an expression is constant in a
couple of other places, such as the decision if division needs a zero check.

This somewhat covers the problem that type analysis is still doing stuff in
some nodes that would fit a lot better into later phases, especially into
code generation. We did most of this for 0.12 already (e.g. gil checking),
but there are still issues.

Stefan

From stefan_ml at behnel.de  Sun Jul  5 10:26:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 05 Jul 2009 10:26:04 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <c266a7930907041131q195476cfnc9c10b5e6602f78a@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<e7ba66e40907031842v4734b6d2oa1498b55ad29f7d6@mail.gmail.com>	<4A4EEEC2.40105@behnel.de>
	<c266a7930907041131q195476cfnc9c10b5e6602f78a@mail.gmail.com>
Message-ID: <4A50639C.4080807@behnel.de>


Kurt Smith wrote:
> On Sat, Jul 4, 2009 at 12:55 AM, Stefan Behnel wrote:
>> It's funny how some German words got mangled since our last
>> major hike to that little North See island.
>                                   ^^^
> 
> Oh, the irony ;-)

:) so true.

Stefan


From dagss at student.matnat.uio.no  Sun Jul  5 11:04:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 05 Jul 2009 11:04:27 +0200
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A506269.9080500@behnel.de>
References: <4A4F2D34.20703@behnel.de>
	<4A4F7CF2.2060005@behnel.de>	<4A4FAE91.10202@student.matnat.uio.no>
	<4A506269.9080500@behnel.de>
Message-ID: <4A506C9B.4000003@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> Why is compile time folding done before expression analysis?
> 
> ;) Because you proposed to move it there at the time? The alternative was
> to split the type analysis phase to make some type information available
> before the 'real' analysis.

Oh, right :-) Glad *someone* keeps track of things in here :-)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun Jul  5 11:23:16 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 05 Jul 2009 11:23:16 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A47BACD.6080906@behnel.de>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>	<45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>	<4A45B13E.7030302@behnel.de>	<6ADF23BE-4021-4143-891F-57DF9CFB95DE@math.washington.edu>
	<4A47BACD.6080906@behnel.de>
Message-ID: <4A507104.2080502@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Robert Bradshaw wrote:
>> On Jun 26, 2009, at 10:42 PM, Stefan Behnel wrote:
>>> Somebody snipped wrote:
>>>> Also, objects declared inside a C++ class are taken as C types  
>>>> (Python objects not allowed)
>>> Why not? They'd map to a PyObject*, which Cython code would ref- 
>>> count its
>>> access to. If you change it from C++ code, you'd just be on your  
>>> own, as
>>> usual. Same for the C++ destructor, which Cython won't generate for  
>>> you. Call it an advanced feature.
>> It is unclear what the semantics of an object member 
>> would be. Would it there be a guarantee that it is set to a non-NULL  
>> increfed value on construction? Does the dectructor have a contract  
>> to decref it? It's probably better to require a specific cast here to  
>> signal from this point on refcounting has to be done manually (either  
>> by the user, or by the C++ library author).
> 
> Good points. Let's forbid that for the time being.

I don't disagree with forbidding it at first, but I actually think that 
getting this right is very easy by using some C++.

It would be very nice to allow e.g.

cdef cpp.vector[object] x

(I.e. std::vector<PyObject*>). Of course, that means that all the
methods on x etc. would take "object". This can be implemented correctly 
(and guaranteed to be correct) by mapping object (in a cppclass context) 
to __Pyx_SmartPtr, which is implemented thusly:

class __Pyx_SmartPtr {
private:
     PyObject* m_ptr;
public:
     __Pyx_SmartPtr(PyObject* ptr) : m_ptr(ptr) {
         if (!ptr) throw new std::exception("NULL not allowed")
         Py_INCREF(ptr);
     }
     ~__Pyx_SmartPtr() {
         Py_DECREF(m_ptr);
     }
     PyObject* rawptr() const { return m_ptr; }
     PyObject* operator*() const { return m_ptr; }
     PyObject* operator->() const { return m_ptr; }
     const __Pyx_SmartPtr& operator=(const __Pyx_SmartPtr& other) const {
         Py_INCREF(other.m_ptr);
         Py_DECREF(m_ptr);
         m_ptr = other.m_ptr;
     }
};

(This kind of thing actually makes me fond of C++ -- it's just the 
thought that a __Pyx_SmartPtr has the same size and run-time storage 
format as a PyObject*, yet refcounting happens correctly).

(For non-template classes the author would need to know about 
__Pyx_SmartPtr from a header; which might not be far from object being 
disallowed -- I think this is ok.)

Extending on this concept, using a C++ template, one could add
type-checking as well, so that we can further correctly implement:

cdef cpp.vector<MyCdefClass> x

-- 
Dag Sverre


From stefan_ml at behnel.de  Sun Jul  5 14:24:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 05 Jul 2009 14:24:40 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
Message-ID: <4A509B88.50307@behnel.de>

Hi,

should we let Cython know certain stdlib modules and consider their import
as something that won't change meaning at runtime?

This is mainly driven by loop optimisations. Imagine

	from itertools import izip

always referred to the same thing. Then Cython could translate

	for a,b in izip(seq1, seq2):
	    ...

into (the equivalent of)

	while True:
	    try:
	        temp_a = seq1.next()
	        temp_b = seq2.next()
	    except StopIteration:
	        break
            a,b = temp_a, temp_b
	    ...

instead of letting izip create a new iterator and tons of tuples along the
way, plus the obvious optimisations for constant sequences, lists, dicts,
etc. (Note that doing this for Py2's builtin zip() would not be correct as
the sequences are allowed to change during iteration in that case).

http://wiki.cython.org/enhancements/forin

Or think of the math module and its constants and functions. We could use
specialised C functions instead if we know the types that we are applying
them to. That way, users wouldn't have to care about two sets of functions
in Python's math module and C's math.h.

Comments?

Stefan


From dagss at student.matnat.uio.no  Sun Jul  5 16:19:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 05 Jul 2009 16:19:54 +0200
Subject: [Cython] Robert: numpy.pxd etc.
Message-ID: <4A50B68A.10401@student.matnat.uio.no>

(Context:

http://trac.cython.org/cython_trac/ticket/339
http://trac.sagemath.org/sage_trac/ticket/4571
)

I want to use Sage to demo Cython at SciPy 09, so I'm going to take care 
of #339 within a day or so. I'm thinking about doing it against the 
cython repo, so that 0.11.2.1 could be released and it could go into 
Sage right away, what do you think about that? (There's already a 
regression bugfix in there).

A problem with the merge is that in Cython's numpy.pxd many fields are 
kept as Python (e.g. flags), for better API compatability with Python.

This is a fundamental problem (and I'm not sure I like the "solution" I 
went for), but at least for now I'll do something like this:

ctypedef extern class numpy.ndarray [object PyArrayObject]:
     ...
     cdef int c_flags "flags"

so that the Python name is the default. It's easier to change Sage here 
than to break backwards compatability.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sun Jul  5 21:04:52 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 05 Jul 2009 21:04:52 +0200
Subject: [Cython] Robert: numpy.pxd etc.
In-Reply-To: <4A50B68A.10401@student.matnat.uio.no>
References: <4A50B68A.10401@student.matnat.uio.no>
Message-ID: <4A50F954.6030701@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> (Context:
> 
> http://trac.cython.org/cython_trac/ticket/339
> http://trac.sagemath.org/sage_trac/ticket/4571
> )
> 
> I want to use Sage to demo Cython at SciPy 09, so I'm going to take care 
> of #339 within a day or so. I'm thinking about doing it against the 
> cython repo, so that 0.11.2.1 could be released and it could go into 
> Sage right away, what do you think about that? (There's already a 
> regression bugfix in there).
> 
> A problem with the merge is that in Cython's numpy.pxd many fields are 
> kept as Python (e.g. flags), for better API compatability with Python.
> 
> This is a fundamental problem (and I'm not sure I like the "solution" I 
> went for), but at least for now I'll do something like this:
> 
> ctypedef extern class numpy.ndarray [object PyArrayObject]:
>      ...
>      cdef int c_flags "flags"
> 
> so that the Python name is the default. It's easier to change Sage here 
> than to break backwards compatability.
> 

Actually, I reconsidered: I will not in general make the C fields 
available at all; since the NumPy recommandation is to use macros (like 
PyArray_CHKFLAGS and so on) anyway.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sun Jul  5 21:16:13 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 05 Jul 2009 21:16:13 +0200
Subject: [Cython] Users list revisited
In-Reply-To: <4A48D877.8010001@student.matnat.uio.no>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>	<7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>	<33DA9D7A-EB99-4248-AB6F-5BECEC637AB4@math.washington.edu>	<4A20E3CC.2040508@behnel.de>
	<4A48D877.8010001@student.matnat.uio.no>
Message-ID: <4A50FBFD.6030906@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Robert Bradshaw wrote:
>>> On May 27, 2009, at 1:00 PM, Chris Colbert wrote:
>>>
>>>> I agree, I think a Cython-users is a great idea.
>>>>
>>>> You all have a been a great help to me in learning Cython, but I  
>>>> feel horrible having to ask such newbie questions all the time, and  
>>>> fear I will one day wear out my welcome.
>>> No worries, you won't be a newbie forever, and then you can pay it  
>>> back by answering other people's questions :).
>>>
>>>> So +1 for a Cython-users list.
>>> +1 from me too, for all the reasons listed below. Stefan, do you want  
>>> to set up one at codespeak.net for consistency?
>> Asking there right now.
> 
> Got an answer yet?

Hmm, no, not really. :-/


> If it was negative, perhaps we could ask PSF? It appears to me that 
> python.org hosts a few 3rd party libraries not in core, I don't get the 
> impression that core inclusion is strictly necesarry to have a mailing 
> list hosted there.

Well, fine with me. Want to ask there?

Stefan


From dagss at student.matnat.uio.no  Mon Jul  6 18:46:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 06 Jul 2009 18:46:58 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A509B88.50307@behnel.de>
References: <4A509B88.50307@behnel.de>
Message-ID: <4A522A82.9080904@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> should we let Cython know certain stdlib modules and consider their import
> as something that won't change meaning at runtime?
> 
> This is mainly driven by loop optimisations. Imagine
> 
> 	from itertools import izip
> 
> always referred to the same thing. Then Cython could translate
> 
> 	for a,b in izip(seq1, seq2):
> 	    ...
> 
> into (the equivalent of)
> 
> 	while True:
> 	    try:
> 	        temp_a = seq1.next()
> 	        temp_b = seq2.next()
> 	    except StopIteration:
> 	        break
>             a,b = temp_a, temp_b
> 	    ...
> 
> instead of letting izip create a new iterator and tons of tuples along the
> way, plus the obvious optimisations for constant sequences, lists, dicts,
> etc. (Note that doing this for Py2's builtin zip() would not be correct as
> the sequences are allowed to change during iteration in that case).
> 
> http://wiki.cython.org/enhancements/forin
> 
> Or think of the math module and its constants and functions. We could use
> specialised C functions instead if we know the types that we are applying
> them to. That way, users wouldn't have to care about two sets of functions
> in Python's math module and C's math.h.
> 
> Comments?

This has been on my mind too for a while, and I'm undecided, or rather 
ambient: I both love and hate the idea :-)

I guess it depends on the motivation. I'm sceptical of hit-or-miss 
optimization in general -- that is, I don't think we can aim for 
"magically optimizing" very much user code through single optimization 
points like this. They are more for users who already know that their 
code will be both interpreted in Python and compiled with Cython -- of 
course, for such users an optimized izip could still be useful.

I think my opinion is that I'd like to see this split this into a couple 
of orthogonal and generic features:

a) Auto-cimport (if a directive is set, which defaults to on?). I.e. if 
you do a Python import, and a pxd file can be found, a cimport is done 
automatically.

b) Overloading combined with fall-through to Python space. I started a 
CEP on this [1], Robert discussed it with me, but it never got finished.

One way is to allow e.g. (in "math.pxd")

double sin(double x): return libc.math.sin(x)
object sin(object x)

Somehow it must be added here though that sin should be looked up at 
module initialization using getattr rather than through 
__pyx_capi/Cython call convention.

c) For izip (and, in time, zip in __builtin__ too) I'm more in favour of 
a slightly magic optimization -- it's easier to be an expert on hacking 
pxd files than the Cython compiler. Namely:

If a generator function contains exactly one "yield", and is declared 
"inline", and used in a for-loop, then the entire generator will be 
inlined into the function (putting the for-loop body at the position of 
the yield).

[1] http://wiki.cython.org/enhancements/overlaypythonmodules, but I 
wouldn't bother with it

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jul  6 18:54:38 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 06 Jul 2009 18:54:38 +0200
Subject: [Cython] Users list revisited
In-Reply-To: <4A50FBFD.6030906@behnel.de>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>	<7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>	<33DA9D7A-EB99-4248-AB6F-5BECEC637AB4@math.washington.edu>	<4A20E3CC.2040508@behnel.de>	<4A48D877.8010001@student.matnat.uio.no>
	<4A50FBFD.6030906@behnel.de>
Message-ID: <4A522C4E.7090102@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> Robert Bradshaw wrote:
>>>> On May 27, 2009, at 1:00 PM, Chris Colbert wrote:
>>>>
>>>>> I agree, I think a Cython-users is a great idea.
>>>>>
>>>>> You all have a been a great help to me in learning Cython, but I  
>>>>> feel horrible having to ask such newbie questions all the time, and  
>>>>> fear I will one day wear out my welcome.
>>>> No worries, you won't be a newbie forever, and then you can pay it  
>>>> back by answering other people's questions :).
>>>>
>>>>> So +1 for a Cython-users list.
>>>> +1 from me too, for all the reasons listed below. Stefan, do you want  
>>>> to set up one at codespeak.net for consistency?
>>> Asking there right now.
>> Got an answer yet?
> 
> Hmm, no, not really. :-/
> 
> 
>> If it was negative, perhaps we could ask PSF? It appears to me that 
>> python.org hosts a few 3rd party libraries not in core, I don't get the 
>> impression that core inclusion is strictly necesarry to have a mailing 
>> list hosted there.
> 
> Well, fine with me. Want to ask there?

Asked now, so let's wait and see.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jul  6 20:08:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 06 Jul 2009 20:08:14 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A522A82.9080904@student.matnat.uio.no>
References: <4A509B88.50307@behnel.de> <4A522A82.9080904@student.matnat.uio.no>
Message-ID: <4A523D8E.8060406@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Hi,
>>
>> should we let Cython know certain stdlib modules and consider their import
>> as something that won't change meaning at runtime?
>>
>> This is mainly driven by loop optimisations. Imagine
>>
>> 	from itertools import izip
>>
>> always referred to the same thing. Then Cython could translate
>>
>> 	for a,b in izip(seq1, seq2):
>> 	    ...
>>
>> into (the equivalent of)
>>
>> 	while True:
>> 	    try:
>> 	        temp_a = seq1.next()
>> 	        temp_b = seq2.next()
>> 	    except StopIteration:
>> 	        break
>>             a,b = temp_a, temp_b
>> 	    ...
>>
>> instead of letting izip create a new iterator and tons of tuples along the
>> way, plus the obvious optimisations for constant sequences, lists, dicts,
>> etc. (Note that doing this for Py2's builtin zip() would not be correct as
>> the sequences are allowed to change during iteration in that case).
>>
>> http://wiki.cython.org/enhancements/forin
>>
>> Or think of the math module and its constants and functions. We could use
>> specialised C functions instead if we know the types that we are applying
>> them to. That way, users wouldn't have to care about two sets of functions
>> in Python's math module and C's math.h.
>>
>> Comments?
> 
> This has been on my mind too for a while, and I'm undecided, or rather 
> ambient: I both love and hate the idea :-)
> 
> I guess it depends on the motivation. I'm sceptical of hit-or-miss 
> optimization in general -- that is, I don't think we can aim for 
> "magically optimizing" very much user code through single optimization 
> points like this. They are more for users who already know that their 
> code will be both interpreted in Python and compiled with Cython -- of 
> course, for such users an optimized izip could still be useful.
> 
> I think my opinion is that I'd like to see this split this into a couple 
> of orthogonal and generic features:
> 
> a) Auto-cimport (if a directive is set, which defaults to on?). I.e. if 
> you do a Python import, and a pxd file can be found, a cimport is done 
> automatically.
> 
> b) Overloading combined with fall-through to Python space. I started a 
> CEP on this [1], Robert discussed it with me, but it never got finished.
> 
> One way is to allow e.g. (in "math.pxd")
> 
> double sin(double x): return libc.math.sin(x)
> object sin(object x)
> 
> Somehow it must be added here though that sin should be looked up at 
> module initialization using getattr rather than through 
> __pyx_capi/Cython call convention.
> 
> c) For izip (and, in time, zip in __builtin__ too) I'm more in favour of 
> a slightly magic optimization -- it's easier to be an expert on hacking 
> pxd files than the Cython compiler. Namely:
> 
> If a generator function contains exactly one "yield", and is declared 
> "inline", and used in a for-loop, then the entire generator will be 
> inlined into the function (putting the for-loop body at the position of 
> the yield).
> 
> [1] http://wiki.cython.org/enhancements/overlaypythonmodules, but I 
> wouldn't bother with it
> 

Note that these were only my comments; I certainly don't mind you doing 
what you proposed if you have an interest (as a temporary or final 
solution).

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jul  6 20:45:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 06 Jul 2009 20:45:59 +0200
Subject: [Cython] Stealing/borrowed references semantics
Message-ID: <4A524667.7040409@student.matnat.uio.no>

In updating numpy.pxd, I encountered the issue that there are exceptions 
from the standard refcounting rules (this also applies somewhere in 
CPython, e.g. PyList_SET_ITEM/GET_ITEM).

Is there a way of declaring such functions in extern blocks, e.g. 
something like

cdef extern borrowed object PyList_GET_ITEM(object, Py_ssize_t)
cdef extern void PyList_SET_ITEM(object, Py_ssize_t, borrowed object)

If not, should I make a ticket for it? (Just wondering what workflow I 
should assign to numpy.pxd; I don't have time for implementing this.)

-- 
Dag Sverre

From stefan_ml at behnel.de  Mon Jul  6 22:16:14 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 06 Jul 2009 22:16:14 +0200
Subject: [Cython] Running the test suite in Py3.1
Message-ID: <4A525B8E.8070900@behnel.de>

Hi,

I pushed a series of code fixes to cython-unstable to support running the
compiler in Python 3.1 and to run the test suite as usual, just by calling
runtests.py. Enabling that made me realise that Cython still isn't Py3
clean, as there are still issues related to byte strings, mostly when
writing them out to the C code stream. A lovely example is that

	"some unicode string formatted with %s" % some_byte_string

didn't work, as it results in this:

	"some unicode string formatted with b'byte string content'"

being written. These problems appear in the weirdest and best hidden
places. For example, the "compile/c_directives" test fails because a
(defaulted?) buffer option pops up from the middle of nowhere that is (for
whatever reason) encoded as a byte string and gets passed on as a keyword
argument. In Py3, keyword argument names must be unicode, so you get a
beautiful compiler crash.

Another problem is string comparisons. Byte strings and unicode strings
never compare equal in Py3, even when they look exactly the same. So we get
a couple of test failures due to unknown or misinterpreted names, and a
couple of failing doctests because the output contains a 'b' prefix at some
point.

Regarding code generation: given that all generated code gets written as
Unicode strings in Py3, I decided to go the route of fake-decoding byte
strings from ISO-8859-1 to Unicode (which generates character code points
from 0-255 that are identical to the original byte sequence), and then
re-encoding the written C code as ISO-8859-1 on its way into the target
file to make them re-appear as the correct bytes. This is slightly hackish,
but everything else would largely complicate the way we write strings into
the code stream. To work around this, I gave the BytesLiteral class its own
__str__() method that returns a 'decoded' unicode string, so that %
formatting on unicode strings works now.

I'll keep carving my way through the code to find the places to fix.
However, I would already like to ask the other core developers to install
the official Python 3.1 release and to make a test run on that platform
part of your normal test cycle. It's easy to get things wrong, but now that
it's easy to run the test suite, I hope it'll also become easier to catch
unexpected problems before non-portable code gets committed.

It would be great if you could run the test suite on your side and skip
through the failures to check for tests and tested code that you are
familiar with. There are definitely tests where it would take me quite a
while to even finger point the place where things go wrong.

Stefan

From dalcinl at gmail.com  Tue Jul  7 00:52:52 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 6 Jul 2009 19:52:52 -0300
Subject: [Cython] Running the test suite in Py3.1
In-Reply-To: <4A525B8E.8070900@behnel.de>
References: <4A525B8E.8070900@behnel.de>
Message-ID: <e7ba66e40907061552v7abd3350l1acbc50ba4937b1f@mail.gmail.com>

How are we going to handle 'operator.div' (not available in Py3) at
Cython/Compiler/ExprNodes.py ? Just got a failure...

-

On Mon, Jul 6, 2009 at 5:16 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
> Hi,
>
> I pushed a series of code fixes to cython-unstable to support running the
> compiler in Python 3.1 and to run the test suite as usual, just by calling
> runtests.py. Enabling that made me realise that Cython still isn't Py3
> clean, as there are still issues related to byte strings, mostly when
> writing them out to the C code stream. A lovely example is that
>
> ? ? ? ?"some unicode string formatted with %s" % some_byte_string
>
> didn't work, as it results in this:
>
> ? ? ? ?"some unicode string formatted with b'byte string content'"
>
> being written. These problems appear in the weirdest and best hidden
> places. For example, the "compile/c_directives" test fails because a
> (defaulted?) buffer option pops up from the middle of nowhere that is (for
> whatever reason) encoded as a byte string and gets passed on as a keyword
> argument. In Py3, keyword argument names must be unicode, so you get a
> beautiful compiler crash.
>
> Another problem is string comparisons. Byte strings and unicode strings
> never compare equal in Py3, even when they look exactly the same. So we get
> a couple of test failures due to unknown or misinterpreted names, and a
> couple of failing doctests because the output contains a 'b' prefix at some
> point.
>
> Regarding code generation: given that all generated code gets written as
> Unicode strings in Py3, I decided to go the route of fake-decoding byte
> strings from ISO-8859-1 to Unicode (which generates character code points
> from 0-255 that are identical to the original byte sequence), and then
> re-encoding the written C code as ISO-8859-1 on its way into the target
> file to make them re-appear as the correct bytes. This is slightly hackish,
> but everything else would largely complicate the way we write strings into
> the code stream. To work around this, I gave the BytesLiteral class its own
> __str__() method that returns a 'decoded' unicode string, so that %
> formatting on unicode strings works now.
>
> I'll keep carving my way through the code to find the places to fix.
> However, I would already like to ask the other core developers to install
> the official Python 3.1 release and to make a test run on that platform
> part of your normal test cycle. It's easy to get things wrong, but now that
> it's easy to run the test suite, I hope it'll also become easier to catch
> unexpected problems before non-portable code gets committed.
>
> It would be great if you could run the test suite on your side and skip
> through the failures to check for tests and tested code that you are
> familiar with. There are definitely tests where it would take me quite a
> while to even finger point the place where things go wrong.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jul  7 06:57:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jul 2009 06:57:23 +0200
Subject: [Cython] Running the test suite in Py3.1
In-Reply-To: <e7ba66e40907061552v7abd3350l1acbc50ba4937b1f@mail.gmail.com>
References: <4A525B8E.8070900@behnel.de>
	<e7ba66e40907061552v7abd3350l1acbc50ba4937b1f@mail.gmail.com>
Message-ID: <4A52D5B3.1070705@behnel.de>

Hi Lisandro,

Lisandro Dalcin wrote:
> How are we going to handle 'operator.div' (not available in Py3) at
> Cython/Compiler/ExprNodes.py ? Just got a failure...

Ah, right. I didn't commit that yet. There's a thread on future division I
started three days ago. I'm currently mapping '/' and '//' to either
floordiv or truediv, depending on the input types. Trying to do that in a
portable way made me notice that I'd need to find a way to run the test
suite in Py3, which gets us back to this thread's topic. :)

Stefan

From robertwb at math.washington.edu  Tue Jul  7 12:12:48 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:12:48 -0700
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <4A524667.7040409@student.matnat.uio.no>
References: <4A524667.7040409@student.matnat.uio.no>
Message-ID: <512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>

On Jul 6, 2009, at 11:45 AM, Dag Sverre Seljebotn wrote:

> In updating numpy.pxd, I encountered the issue that there are  
> exceptions
> from the standard refcounting rules (this also applies somewhere in
> CPython, e.g. PyList_SET_ITEM/GET_ITEM).
>
> Is there a way of declaring such functions in extern blocks, e.g.
> something like
>
> cdef extern borrowed object PyList_GET_ITEM(object, Py_ssize_t)
> cdef extern void PyList_SET_ITEM(object, Py_ssize_t, borrowed object)
>
> If not, should I make a ticket for it? (Just wondering what workflow I
> should assign to numpy.pxd; I don't have time for implementing this.)

The convention we've taken in Sage is that borrowed references are  
declared as PyObject* (so to store them anywhere requires casting to  
object, thus an incref). When you say "borrowed" for an argument, I'm  
assuming you mean arguments that steal references, right? I agree it  
might be useful to have something there.

- Robert


From robertwb at math.washington.edu  Tue Jul  7 12:23:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:23:11 -0700
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A522A82.9080904@student.matnat.uio.no>
References: <4A509B88.50307@behnel.de> <4A522A82.9080904@student.matnat.uio.no>
Message-ID: <2DCB0117-B0EA-4D1E-A51B-3DB33082B8AF@math.washington.edu>

On Jul 6, 2009, at 9:46 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Hi,
>>
>> should we let Cython know certain stdlib modules and consider  
>> their import
>> as something that won't change meaning at runtime?
>>
>> This is mainly driven by loop optimisations. Imagine
>>
>> 	from itertools import izip
>>
>> always referred to the same thing. Then Cython could translate
>>
>> 	for a,b in izip(seq1, seq2):
>> 	    ...
>>
>> into (the equivalent of)
>>
>> 	while True:
>> 	    try:
>> 	        temp_a = seq1.next()
>> 	        temp_b = seq2.next()
>> 	    except StopIteration:
>> 	        break
>>             a,b = temp_a, temp_b
>> 	    ...
>>
>> instead of letting izip create a new iterator and tons of tuples  
>> along the
>> way, plus the obvious optimisations for constant sequences, lists,  
>> dicts,
>> etc. (Note that doing this for Py2's builtin zip() would not be  
>> correct as
>> the sequences are allowed to change during iteration in that case).
>>
>> http://wiki.cython.org/enhancements/forin
>>
>> Or think of the math module and its constants and functions. We  
>> could use
>> specialised C functions instead if we know the types that we are  
>> applying
>> them to. That way, users wouldn't have to care about two sets of  
>> functions
>> in Python's math module and C's math.h.
>>
>> Comments?
>
> This has been on my mind too for a while, and I'm undecided, or rather
> ambient: I both love and hate the idea :-)

Same here :)

> I guess it depends on the motivation. I'm sceptical of hit-or-miss
> optimization in general -- that is, I don't think we can aim for
> "magically optimizing" very much user code through single optimization
> points like this. They are more for users who already know that their
> code will be both interpreted in Python and compiled with Cython -- of
> course, for such users an optimized izip could still be useful.
>
> I think my opinion is that I'd like to see this split this into a  
> couple
> of orthogonal and generic features:
>
> a) Auto-cimport (if a directive is set, which defaults to on?).  
> I.e. if
> you do a Python import, and a pxd file can be found, a cimport is done
> automatically.

For Sage, we need a way to turn this off. (A lot of stuff is in  
Cython, but doesn't need to be cimported, which could slow down the  
compilation time a lot).

> b) Overloading combined with fall-through to Python space. I started a
> CEP on this [1], Robert discussed it with me, but it never got  
> finished.
>
> One way is to allow e.g. (in "math.pxd")
>
> double sin(double x): return libc.math.sin(x)
> object sin(object x)
>
> Somehow it must be added here though that sin should be looked up at
> module initialization using getattr rather than through
> __pyx_capi/Cython call convention.

Danilo's working on function overloading right now, which would make  
this lots easier. (It's still unclear how to declare the non- 
optimized version).

> c) For izip (and, in time, zip in __builtin__ too) I'm more in  
> favour of
> a slightly magic optimization -- it's easier to be an expert on  
> hacking
> pxd files than the Cython compiler. Namely:
>
> If a generator function contains exactly one "yield", and is declared
> "inline", and used in a for-loop, then the entire generator will be
> inlined into the function (putting the for-loop body at the  
> position of
> the yield).

Given the benefit and positive reaction we've had with the range  
optimization, I'm all for this as well. If the semantics could ever  
differ, we should allow it to be disabled, but the proposal above  
looks just fine.

> [1] http://wiki.cython.org/enhancements/overlaypythonmodules, but I
> wouldn't bother with it

On that, I'm not so sure about the typemixin idea, and handling  
subclasses, but we did decide that "cdef list" means exactly a list  
(and not a subclass thereof) so we may be able to perform  
optimizations of this sort.

- Robert


From robertwb at math.washington.edu  Tue Jul  7 12:26:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:26:45 -0700
Subject: [Cython] Running the test suite in Py3.1
In-Reply-To: <4A525B8E.8070900@behnel.de>
References: <4A525B8E.8070900@behnel.de>
Message-ID: <5FB0923C-B0C5-4FB3-8431-B69554B3AB5F@math.washington.edu>

On Jul 6, 2009, at 1:16 PM, Stefan Behnel wrote:

> I'll keep carving my way through the code to find the places to fix.
> However, I would already like to ask the other core developers to  
> install
> the official Python 3.1 release and to make a test run on that  
> platform
> part of your normal test cycle. It's easy to get things wrong, but  
> now that
> it's easy to run the test suite, I hope it'll also become easier to  
> catch
> unexpected problems before non-portable code gets committed.
>
> It would be great if you could run the test suite on your side and  
> skip
> through the failures to check for tests and tested code that you are
> familiar with. There are definitely tests where it would take me  
> quite a
> while to even finger point the place where things go wrong.

Thanks for doing this--I'm probably a guilty party as much or more so  
than anyone else. You've inspired me to install 3.1 on my computer,  
and I'll start trying to test with that as well.

- Robert


From robertwb at math.washington.edu  Tue Jul  7 12:29:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:29:55 -0700
Subject: [Cython] Robert: numpy.pxd etc.
In-Reply-To: <4A50B68A.10401@student.matnat.uio.no>
References: <4A50B68A.10401@student.matnat.uio.no>
Message-ID: <BC1A3888-155E-4D2C-AAF4-39358CBE888A@math.washington.edu>

On Jul 5, 2009, at 7:19 AM, Dag Sverre Seljebotn wrote:

> (Context:
>
> http://trac.cython.org/cython_trac/ticket/339
> http://trac.sagemath.org/sage_trac/ticket/4571
> )
>
> I want to use Sage to demo Cython at SciPy 09, so I'm going to take  
> care
> of #339 within a day or so. I'm thinking about doing it against the
> cython repo, so that 0.11.2.1 could be released and it could go into
> Sage right away, what do you think about that? (There's already a
> regression bugfix in there).

Cool. I got 0.11.2 working with Sage, but never got the spkg in.

> A problem with the merge is that in Cython's numpy.pxd many fields are
> kept as Python (e.g. flags), for better API compatability with Python.
>
> This is a fundamental problem (and I'm not sure I like the  
> "solution" I
> went for), but at least for now I'll do something like this:
>
> ctypedef extern class numpy.ndarray [object PyArrayObject]:
>      ...
>      cdef int c_flags "flags"
>
> so that the Python name is the default. It's easier to change Sage  
> here
> than to break backwards compatability.

Yes, feel free to fix Sage (and I do see these as fixes). Also, feel  
free to ping me with any questions as I've looked at this code recently.

- Robert


From robertwb at math.washington.edu  Tue Jul  7 12:30:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:30:24 -0700
Subject: [Cython] Robert: numpy.pxd etc.
In-Reply-To: <4A50F954.6030701@student.matnat.uio.no>
References: <4A50B68A.10401@student.matnat.uio.no>
	<4A50F954.6030701@student.matnat.uio.no>
Message-ID: <0AA0C9B7-9634-4484-A5BB-7548D03AAF8B@math.washington.edu>

On Jul 5, 2009, at 12:04 PM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> (Context:
>>
>> http://trac.cython.org/cython_trac/ticket/339
>> http://trac.sagemath.org/sage_trac/ticket/4571
>> )
>>
>> I want to use Sage to demo Cython at SciPy 09, so I'm going to  
>> take care
>> of #339 within a day or so. I'm thinking about doing it against the
>> cython repo, so that 0.11.2.1 could be released and it could go into
>> Sage right away, what do you think about that? (There's already a
>> regression bugfix in there).
>>
>> A problem with the merge is that in Cython's numpy.pxd many fields  
>> are
>> kept as Python (e.g. flags), for better API compatability with  
>> Python.
>>
>> This is a fundamental problem (and I'm not sure I like the  
>> "solution" I
>> went for), but at least for now I'll do something like this:
>>
>> ctypedef extern class numpy.ndarray [object PyArrayObject]:
>>      ...
>>      cdef int c_flags "flags"
>>
>> so that the Python name is the default. It's easier to change Sage  
>> here
>> than to break backwards compatability.
>>
>
> Actually, I reconsidered: I will not in general make the C fields
> available at all; since the NumPy recommandation is to use macros  
> (like
> PyArray_CHKFLAGS and so on) anyway.

Even better. (I really should stick to answering emails in reverse  
order...)

- Robert

From robertwb at math.washington.edu  Tue Jul  7 12:32:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:32:42 -0700
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A507104.2080502@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>	<45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>	<4A45B13E.7030302@behnel.de>	<6ADF23BE-4021-4143-891F-57DF9CFB95DE@math.washington.edu>
	<4A47BACD.6080906@behnel.de>
	<4A507104.2080502@student.matnat.uio.no>
Message-ID: <310F70B4-5E9C-4672-9DD6-A804A640D541@math.washington.edu>


On Jul 5, 2009, at 2:23 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Hi,
>>
>> Robert Bradshaw wrote:
>>> On Jun 26, 2009, at 10:42 PM, Stefan Behnel wrote:
>>>> Somebody snipped wrote:
>>>>> Also, objects declared inside a C++ class are taken as C types
>>>>> (Python objects not allowed)
>>>> Why not? They'd map to a PyObject*, which Cython code would ref-
>>>> count its
>>>> access to. If you change it from C++ code, you'd just be on your
>>>> own, as
>>>> usual. Same for the C++ destructor, which Cython won't generate for
>>>> you. Call it an advanced feature.
>>> It is unclear what the semantics of an object member
>>> would be. Would it there be a guarantee that it is set to a non-NULL
>>> increfed value on construction? Does the dectructor have a contract
>>> to decref it? It's probably better to require a specific cast  
>>> here to
>>> signal from this point on refcounting has to be done manually  
>>> (either
>>> by the user, or by the C++ library author).
>>
>> Good points. Let's forbid that for the time being.
>
> I don't disagree with forbidding it at first, but I actually think  
> that
> getting this right is very easy by using some C++.
>
> It would be very nice to allow e.g.
>
> cdef cpp.vector[object] x
>
> (I.e. std::vector<PyObject*>). Of course, that means that all the
> methods on x etc. would take "object". This can be implemented  
> correctly
> (and guaranteed to be correct) by mapping object (in a cppclass  
> context)
> to __Pyx_SmartPtr, which is implemented thusly:
>
> class __Pyx_SmartPtr {
> private:
>      PyObject* m_ptr;
> public:
>      __Pyx_SmartPtr(PyObject* ptr) : m_ptr(ptr) {
>          if (!ptr) throw new std::exception("NULL not allowed")
>          Py_INCREF(ptr);
>      }
>      ~__Pyx_SmartPtr() {
>          Py_DECREF(m_ptr);
>      }
>      PyObject* rawptr() const { return m_ptr; }
>      PyObject* operator*() const { return m_ptr; }
>      PyObject* operator->() const { return m_ptr; }
>      const __Pyx_SmartPtr& operator=(const __Pyx_SmartPtr& other)  
> const {
>          Py_INCREF(other.m_ptr);
>          Py_DECREF(m_ptr);
>          m_ptr = other.m_ptr;
>      }
> };
>
> (This kind of thing actually makes me fond of C++ -- it's just the
> thought that a __Pyx_SmartPtr has the same size and run-time storage
> format as a PyObject*, yet refcounting happens correctly).
>
> (For non-template classes the author would need to know about
> __Pyx_SmartPtr from a header; which might not be far from object being
> disallowed -- I think this is ok.)

Hmm... very cool idea. Not something that should go in first pass,  
but it makes it a real possibility.

- Robert


From robertwb at math.washington.edu  Tue Jul  7 12:36:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:36:55 -0700
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A4EAF85.3070003@canterbury.ac.nz>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
Message-ID: <9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>

On Jul 3, 2009, at 6:25 PM, Greg Ewing wrote:

> Lisandro Dalcin wrote:
>
>> cdef extern from *: # note the "?" annotation
>>     ctypedef singed MyIntType1?
>                 ^^^^^^
>
> Is this a special data type for CPUs that are
> running too hot?

:). The "unknown size" is because you don't know how many useable  
bits are left.

On a more serious note, I'd probably prefer

cdef extern from *:
     ctypedef signed int? MyIntType1

- Robert


From robertwb at math.washington.edu  Tue Jul  7 12:44:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 03:44:24 -0700
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A4FAE3F.703@student.matnat.uio.no>
References: <4A4F2D34.20703@behnel.de> <4A4FAE3F.703@student.matnat.uio.no>
Message-ID: <4983CF7A-09F4-47D4-974E-F3464E3F6C9D@math.washington.edu>

On Jul 4, 2009, at 12:32 PM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> My opinion on this is: if you request true division semantics by  
>> importing
>> __future__.division, you should get true division in all places  
>> where you
>> use the "/" operator.
>>
>> Comments?
>
> Thanks for raising this issue. Without diving into the details myself,
> I'm +1 -- we've pretty much decided to have Python semantics on  
> division.

Yep. As ugly as it seems (to me), when __future__.division is asked  
for, the user should get it. This really shouldn't be too hard to  
implement, now that the cdivision stuff is in place.

- Robert



From stefan_ml at behnel.de  Tue Jul  7 13:00:10 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jul 2009 13:00:10 +0200
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>
References: <4A524667.7040409@student.matnat.uio.no>
	<512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>
Message-ID: <4A532ABA.1000701@behnel.de>


Robert Bradshaw wrote:
> On Jul 6, 2009, at 11:45 AM, Dag Sverre Seljebotn wrote:
> 
>> In updating numpy.pxd, I encountered the issue that there are  
>> exceptions
>> from the standard refcounting rules (this also applies somewhere in
>> CPython, e.g. PyList_SET_ITEM/GET_ITEM).
>>
>> Is there a way of declaring such functions in extern blocks, e.g.
>> something like
>>
>> cdef extern borrowed object PyList_GET_ITEM(object, Py_ssize_t)
>> cdef extern void PyList_SET_ITEM(object, Py_ssize_t, borrowed object)
>>
>> If not, should I make a ticket for it? (Just wondering what workflow I
>> should assign to numpy.pxd; I don't have time for implementing this.)
> 
> The convention we've taken in Sage is that borrowed references are  
> declared as PyObject* (so to store them anywhere requires casting to  
> object, thus an incref).

+1


> When you say "borrowed" for an argument, I'm  
> assuming you mean arguments that steal references, right? I agree it  
> might be useful to have something there.

... although "borrowed" doesn't make that much sense in this context.

However, without flow analysis, it's impossible to get ref-counting right here:

    def test(x):
        a = None
        if x:
            steal_reference(a)

Should "a" get decref-ed at function exit?

Stefan


From dagss at student.matnat.uio.no  Tue Jul  7 13:16:52 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 13:16:52 +0200
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <4A532ABA.1000701@behnel.de>
References: <4A524667.7040409@student.matnat.uio.no>	<512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>
	<4A532ABA.1000701@behnel.de>
Message-ID: <4A532EA4.6060801@student.matnat.uio.no>

Stefan Behnel wrote:
> Robert Bradshaw wrote:
>> On Jul 6, 2009, at 11:45 AM, Dag Sverre Seljebotn wrote:
>>
>>> In updating numpy.pxd, I encountered the issue that there are  
>>> exceptions
>>> from the standard refcounting rules (this also applies somewhere in
>>> CPython, e.g. PyList_SET_ITEM/GET_ITEM).
>>>
>>> Is there a way of declaring such functions in extern blocks, e.g.
>>> something like
>>>
>>> cdef extern borrowed object PyList_GET_ITEM(object, Py_ssize_t)
>>> cdef extern void PyList_SET_ITEM(object, Py_ssize_t, borrowed object)
>>>
>>> If not, should I make a ticket for it? (Just wondering what workflow I
>>> should assign to numpy.pxd; I don't have time for implementing this.)
>> The convention we've taken in Sage is that borrowed references are  
>> declared as PyObject* (so to store them anywhere requires casting to  
>> object, thus an incref).
> 
> +1
> 
> 
>> When you say "borrowed" for an argument, I'm  
>> assuming you mean arguments that steal references, right? I agree it  
>> might be useful to have something there.
> 
> ... although "borrowed" doesn't make that much sense in this context.

OK, stolen then.

> 
> However, without flow analysis, it's impossible to get ref-counting right here:
> 
>     def test(x):
>         a = None
>         if x:
>             steal_reference(a)
> 
> Should "a" get decref-ed at function exit?

Well, it would be syntax candy for an extra incref. I can work around 
this now by:

cdef extern void Real_PyList_SET_ITEM "PyList_SET_ITEM"(object, 
Py_ssize_t, object)

cdef inline void PyList_SET_ITEM(object a, Py_ssize_t b, object c):
     Py_INCREF(c)
     Real_PyList_SET_ITEM(a, b, c)


For ~20 functions I am likely not going to bother; while simply declaring

cdef extern void PyList_SET_ITEM(object, Py_ssize_t, stolen object):

I could do.

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Jul  7 13:32:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jul 2009 13:32:47 +0200
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <4A532EA4.6060801@student.matnat.uio.no>
References: <4A524667.7040409@student.matnat.uio.no>	<512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>	<4A532ABA.1000701@behnel.de>
	<4A532EA4.6060801@student.matnat.uio.no>
Message-ID: <4A53325F.8060503@behnel.de>


Dag Sverre Seljebotn wrote:
> it would be syntax candy for an extra incref. I can work around 
> this now by:
> 
> cdef extern void Real_PyList_SET_ITEM "PyList_SET_ITEM"(object, 
> Py_ssize_t, object)
> 
> cdef inline void PyList_SET_ITEM(object a, Py_ssize_t b, object c):
>      Py_INCREF(c)
>      Real_PyList_SET_ITEM(a, b, c)
> 
> For ~20 functions I am likely not going to bother; while simply declaring
> 
> cdef extern void PyList_SET_ITEM(object, Py_ssize_t, stolen object):

I'm fine with that, as long as we only allow it inside of argument lists
(preferrably only "extern" argument lists).

Stefan


From stefan_ml at behnel.de  Tue Jul  7 14:00:11 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jul 2009 14:00:11 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A522A82.9080904@student.matnat.uio.no>
References: <4A509B88.50307@behnel.de> <4A522A82.9080904@student.matnat.uio.no>
Message-ID: <4A5338CB.6020309@behnel.de>

Hi Dag,

Dag Sverre Seljebotn wrote:
> I guess it depends on the motivation. I'm sceptical of hit-or-miss 
> optimization in general -- that is, I don't think we can aim for 
> "magically optimizing" very much user code through single optimization 
> points like this. They are more for users who already know that their 
> code will be both interpreted in Python and compiled with Cython -- of 
> course, for such users an optimized izip could still be useful.

Besides the obvious advantage of optimising Python code, I think even
Cython code will benefit, as it's ugly when you have to unroll izip()
yourself in your code. When you need that functionality, there should be no
reason to work around it for performance reason.


> I think my opinion is that I'd like to see this split this into a couple 
> of orthogonal and generic features:
> 
> a) Auto-cimport (if a directive is set, which defaults to on?). I.e. if 
> you do a Python import, and a pxd file can be found, a cimport is done 
> automatically.

+0.7 given that this may really have a performance impact, as Robert
stated. There should at least be a cache for the standard include path that
survives multiple compilations in one compiler run (another orthogonal
feature that would be nice for .pxd files in general).


> b) Overloading combined with fall-through to Python space. I started a 
> CEP on this [1], Robert discussed it with me, but it never got finished.
> 
> One way is to allow e.g. (in "math.pxd")
> 
> double sin(double x): return libc.math.sin(x)
> object sin(object x)

+1, nice feature.


> Somehow it must be added here though that sin should be looked up at 
> module initialization using getattr rather than through 
> __pyx_capi/Cython call convention.

That would be implied by the fact that you used a Python import. But
leaving this to the poor importer might turn out to be error prone, given
that the math module actually is a C extension...


> c) For izip (and, in time, zip in __builtin__ too) I'm more in favour of 
> a slightly magic optimization -- it's easier to be an expert on hacking 
> pxd files than the Cython compiler. Namely:
> 
> If a generator function contains exactly one "yield", and is declared 
> "inline", and used in a for-loop, then the entire generator will be 
> inlined into the function (putting the for-loop body at the position of 
> the yield).

+1, another cool feature - once we have generator support in general. :)

Stefan

From dagss at student.matnat.uio.no  Tue Jul  7 14:19:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 14:19:40 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A5338CB.6020309@behnel.de>
References: <4A509B88.50307@behnel.de> <4A522A82.9080904@student.matnat.uio.no>
	<4A5338CB.6020309@behnel.de>
Message-ID: <4A533D5C.5070009@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi Dag,
> 
> Dag Sverre Seljebotn wrote:
>> I guess it depends on the motivation. I'm sceptical of hit-or-miss 
>> optimization in general -- that is, I don't think we can aim for 
>> "magically optimizing" very much user code through single optimization 
>> points like this. They are more for users who already know that their 
>> code will be both interpreted in Python and compiled with Cython -- of 
>> course, for such users an optimized izip could still be useful.
> 
> Besides the obvious advantage of optimising Python code, I think even
> Cython code will benefit, as it's ugly when you have to unroll izip()
> yourself in your code. When you need that functionality, there should be no
> reason to work around it for performance reason.
> 
> 
>> I think my opinion is that I'd like to see this split this into a couple 
>> of orthogonal and generic features:
>>
>> a) Auto-cimport (if a directive is set, which defaults to on?). I.e. if 
>> you do a Python import, and a pxd file can be found, a cimport is done 
>> automatically.
> 
> +0.7 given that this may really have a performance impact, as Robert
> stated. There should at least be a cache for the standard include path that
> survives multiple compilations in one compiler run (another orthogonal
> feature that would be nice for .pxd files in general).

with cython.nopxd:
     import mymodule

?

It works just fine in Python :-)

Sage would likely just do --directive nopxd=True.

(The default can be discussed once the feature is actually there I suppose).

>> b) Overloading combined with fall-through to Python space. I started a 
>> CEP on this [1], Robert discussed it with me, but it never got finished.
>>
>> One way is to allow e.g. (in "math.pxd")
>>
>> double sin(double x): return libc.math.sin(x)
>> object sin(object x)
> 
> +1, nice feature.
> 
> 
>> Somehow it must be added here though that sin should be looked up at 
>> module initialization using getattr rather than through 
>> __pyx_capi/Cython call convention.
> 
> That would be implied by the fact that you used a Python import. But
> leaving this to the poor importer might turn out to be error prone, given
> that the math module actually is a C extension...

-1 on leaving it to importer; I'd like to use "import" to cimport Cython 
extensions too! Lower learning curve etc.

This could actually be done at runtime. Support code:

static PyObject* math_sin_looked_up_with_getattr = NULL;

static PyObject* math_sin_with_getattr(PyObject* obj) {
   /* call math_sin_looked_up_with_getattr using CPython API */
}

In module initialization:

if hasattr(math module, "__pyx_capi"):
     set math_sin as today
else:
     look up and set math_sin_looked_up_with_getattr
     set math_sin to math_sin_with_getattr

Then, call math_sin as today in general.

The minus is that it blows up code size somewhat for cimported Cython 
modules; new syntax would avoid that.

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Jul  7 14:52:29 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jul 2009 14:52:29 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A533D5C.5070009@student.matnat.uio.no>
References: <4A509B88.50307@behnel.de>
	<4A522A82.9080904@student.matnat.uio.no>	<4A5338CB.6020309@behnel.de>
	<4A533D5C.5070009@student.matnat.uio.no>
Message-ID: <4A53450D.1000004@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> a) Auto-cimport (if a directive is set, which defaults to on?). I.e. if 
>>> you do a Python import, and a pxd file can be found, a cimport is done 
>>> automatically.
>> +0.7 given that this may really have a performance impact, as Robert
>> stated. There should at least be a cache for the standard include path that
>> survives multiple compilations in one compiler run (another orthogonal
>> feature that would be nice for .pxd files in general).
> 
> with cython.nopxd:
>      import mymodule
> 
> ?

-1

Besides being the wrong way round (most Python modules do not have a .pxd
file), this is clearly something that Cython should handle behind the
scenes. User code shouldn't be impacted by it. If anything, a compiler
option would fit. But thinking about that now smells like a premature
optimisation to me.


>>> b) Overloading combined with fall-through to Python space. I started a 
>>> CEP on this [1], Robert discussed it with me, but it never got finished.
>>>
>>> One way is to allow e.g. (in "math.pxd")
>>>
>>> double sin(double x): return libc.math.sin(x)
>>> object sin(object x)
>> +1, nice feature.
>>
>>
>>> Somehow it must be added here though that sin should be looked up at 
>>> module initialization using getattr rather than through 
>>> __pyx_capi/Cython call convention.
>> That would be implied by the fact that you used a Python import. But
>> leaving this to the poor importer might turn out to be error prone, given
>> that the math module actually is a C extension...
> 
> -1 on leaving it to importer; I'd like to use "import" to cimport Cython 
> extensions too! Lower learning curve etc.

Fine.


> This could actually be done at runtime. Support code:
> 
> static PyObject* math_sin_looked_up_with_getattr = NULL;
> 
> static PyObject* math_sin_with_getattr(PyObject* obj) {
>    /* call math_sin_looked_up_with_getattr using CPython API */
> }
> 
> In module initialization:
> 
> if hasattr(math module, "__pyx_capi"):
>      set math_sin as today
> else:
>      look up and set math_sin_looked_up_with_getattr
>      set math_sin to math_sin_with_getattr
> 
> Then, call math_sin as today in general.
> 
> The minus is that it blows up code size somewhat for cimported Cython 
> modules; new syntax would avoid that.

I'm more for avoiding new syntax. Lower learning curve etc. ;)

I think it's a good idea to simply generate code like this and to leave the
import handling entirely to runtime code. Note that only functions that are
pure Python functions need to be looked up at the module level. Everything
else must be imported from __pyx_capi and a failure to do so is fatal.

All ambiguous imports should then get imported from __pyx_capi one by one,
and failing to do so, get imported from the module dict next. A failure to
import from both is fatal again. Note that this allows names to get
imported from either source on a case-by-case basis.

Stefan

From dagss at student.matnat.uio.no  Tue Jul  7 14:59:52 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 14:59:52 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A53450D.1000004@behnel.de>
References: <4A509B88.50307@behnel.de>	<4A522A82.9080904@student.matnat.uio.no>	<4A5338CB.6020309@behnel.de>	<4A533D5C.5070009@student.matnat.uio.no>
	<4A53450D.1000004@behnel.de>
Message-ID: <4A5346C8.5010802@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> a) Auto-cimport (if a directive is set, which defaults to on?). I.e. if 
>>>> you do a Python import, and a pxd file can be found, a cimport is done 
>>>> automatically.
>>> +0.7 given that this may really have a performance impact, as Robert
>>> stated. There should at least be a cache for the standard include path that
>>> survives multiple compilations in one compiler run (another orthogonal
>>> feature that would be nice for .pxd files in general).
>> with cython.nopxd:
>>      import mymodule
>>
>> ?
> 
> -1
> 
> Besides being the wrong way round (most Python modules do not have a .pxd
> file), this is clearly something that Cython should handle behind the
> scenes. User code shouldn't be impacted by it. If anything, a compiler
> option would fit. But thinking about that now smells like a premature
> optimisation to me.

But (expert) users must have control and care about this, it cannot 
happen behind the scenes anyway! I.e. with no automatic cimport, you can do

import itertools

...

itertools = None
...

With auto cimport (like you seemed to want originally, at least for the 
specific case of itertools) this is a syntax error.

This early-binding is the big change in semantics; everything else is 
just details.

But I think it makes sense to say that Cython does, for module-level 
imports at the top of the file, early bind them, as it is OK anyway for 
99% of the usecases (and a syntax error is raised anyway at "itertool = 
None", which could say "you have to turn off auto-cimport").

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue Jul  7 15:07:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 07 Jul 2009 15:07:30 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A5346C8.5010802@student.matnat.uio.no>
References: <4A509B88.50307@behnel.de>	<4A522A82.9080904@student.matnat.uio.no>	<4A5338CB.6020309@behnel.de>	<4A533D5C.5070009@student.matnat.uio.no>	<4A53450D.1000004@behnel.de>
	<4A5346C8.5010802@student.matnat.uio.no>
Message-ID: <4A534892.5060801@behnel.de>


Dag Sverre Seljebotn wrote:
> But (expert) users must have control and care about this, it cannot 
> happen behind the scenes anyway! I.e. with no automatic cimport, you can do
> 
> import itertools
> 
> ...
> 
> itertools = None
> ...

Isn't that easily caught by traversing the syntax tree for global
assignments to (or re-imports of) "itertools"? That's the same problem as
writing

	list = None

in my source.

Stefan


From dalcinl at gmail.com  Tue Jul  7 15:32:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Jul 2009 10:32:39 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
Message-ID: <e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>

On Tue, Jul 7, 2009 at 7:36 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
>
> On a more serious note, I'd probably prefer
>
> cdef extern from *:
> ? ? ctypedef signed int? MyIntType1
>

Many thanks for you input, Robert... I'll try to implement this...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Jul  7 15:47:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 15:47:12 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
Message-ID: <4A5351E0.7020009@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Jul 7, 2009 at 7:36 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On a more serious note, I'd probably prefer
>>
>> cdef extern from *:
>>     ctypedef signed int? MyIntType1
>>
> 
> Many thanks for you input, Robert... I'll try to implement this...

Sorry that I entered this late, but:

What is the specific semantic differences for this type beyond what #333 
provides?

That division (or other operations requiring a temporary) will be 
disallowed?

Or that #333-like behaviour will *only* be done if you use a "?"? If so, 
for what gain? -- some fewer lines of C code? Why does it hurt to 
"second-guess the user", as you put it in the OP?

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jul  7 15:49:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 15:49:48 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A5351E0.7020009@student.matnat.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
Message-ID: <4A53527C.4020805@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:
>> On Tue, Jul 7, 2009 at 7:36 AM, Robert
>> Bradshaw<robertwb at math.washington.edu> wrote:
>>> On a more serious note, I'd probably prefer
>>>
>>> cdef extern from *:
>>>     ctypedef signed int? MyIntType1
>>>
>>
>> Many thanks for you input, Robert... I'll try to implement this...
> 
> Sorry that I entered this late, but:
> 
> What is the specific semantic differences for this type beyond what #333 
> provides?
> 
> That division (or other operations requiring a temporary) will be 
> disallowed?

Sorry, it is important that I am clearer here: I meant division 
involving two different types, of which at least one is "?"-marked.

> 
> Or that #333-like behaviour will *only* be done if you use a "?"? If so, 
> for what gain? -- some fewer lines of C code? Why does it hurt to 
> "second-guess the user", as you put it in the OP?
> 


-- 
Dag Sverre

From dalcinl at gmail.com  Tue Jul  7 19:09:31 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Jul 2009 14:09:31 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A5351E0.7020009@student.matnat.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
Message-ID: <e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>

On Tue, Jul 7, 2009 at 10:47 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> On Tue, Jul 7, 2009 at 7:36 AM, Robert
>> Bradshaw<robertwb at math.washington.edu> wrote:
>>> On a more serious note, I'd probably prefer
>>>
>>> cdef extern from *:
>>> ? ? ctypedef signed int? MyIntType1
>>>
>>
>> Many thanks for you input, Robert... I'll try to implement this...
>
> Sorry that I entered this late, but:
>
> What is the specific semantic differences for this type beyond what #333
> provides?
>
> That division (or other operations requiring a temporary) will be
> disallowed?
>

In the long term, we could special case things like division and do
something smarter (upcasting?). We can start right now by disabling
them, though I did not commented on this just because I'm not sure how
to implement that... Until something smarter is implemented, users
would require to put explicit casts in order to make arithmetic opts
with such ctypedef types...

> Or that #333-like behaviour will *only* be done if you use a "?"? If so,
> for what gain? -- some fewer lines of C code? Why does it hurt to
> "second-guess the user", as you put it in the OP?
>

Because of your fist question and my previous comment. Second-guessing
will be a bad thing...

BTW, this likely should affect you work on member descriptor, right?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Jul  7 19:30:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 19:30:18 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
Message-ID: <4A53862A.10206@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Jul 7, 2009 at 10:47 AM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Or that #333-like behaviour will *only* be done if you use a "?"? If so,
>> for what gain? -- some fewer lines of C code? Why does it hurt to
>> "second-guess the user", as you put it in the OP?
>>
> 
> Because of your fist question and my previous comment. Second-guessing
> will be a bad thing...

I disagree, I think second-guessing is wonderful. (That is, if you want 
me to understand this, you must explain more -- of course, it is not a 
necessity that I understand this.)

If the only effect is to save a few lines of C and a few cycles of CPU 
time for gcc, then I don't think it is worth it to complicate the language.

I do think this could be good to have if we inserted code at module init 
time to raise exceptions if sizeof(foo) != sizeof(bar) for all external 
typedefs, so that exactness is *enforced* in those situations. (That may 
break some backwards compatability but I think it is OK myself.) If that 
approach is also taken, at the same time, and this isn't done until 
0.12, then +1.

But otherwise I'm -1 since it adds thing to the language without any 
benefit at all (that I can see).

> BTW, this likely should affect you work on member descriptor, right?

It already second-guesses now; the only effect would be to speed up gcc 
compilation with a totally insignificant amount per member; not worth 
the time to make the likely trivial commit on Cython IMO.

-- 
Dag Sverre

From dsimmons.com at gmail.com  Tue Jul  7 20:15:16 2009
From: dsimmons.com at gmail.com (David Simmons-Duffin)
Date: Tue, 7 Jul 2009 14:15:16 -0400
Subject: [Cython] "object" keyword in cython
Message-ID: <AE0A6018-7CDC-4E1E-BCA8-119D22751507@gmail.com>

I'm trying to interface with some C code that defines a type called  
"object"

typedef union {
   struct {
            objtype type; reftype nref;
   } any;
   intcel i;
   bigint b;
   vector v;
   matrix m;
   poly   pl;
   simpgrp s;
   group g;
   tekst  t;
} objcel, *object;

However, if I try to write some cython code involving an object:

cdef extern object groupmake(char lietype, int rank)
cdef object g
g = groupmake("A", 3)

It's interpreted as a PyObject in the c code:

PyObject *__pyx_t_1 = NULL;
...
__pyx_t_1 = groupmake('A', 2);

One possible solution is to just replace the word "object" in the C  
source with something else throughout.  But I'd like to make as few  
modifications of the C source as possible.  Is there an alternate  
solution?

I actually have another unrelated question: what is the correct cython  
syntax for the union statement at the top of this message?  Cython  
doesn't seem to like me putting

struct any:
	objtype type
	reftype nfref

in the middle of a union statement.  And I'm also not sure what to do  
with "objcel, *object" at the end there.  I'm a novice at this stuff,  
so apologies if these are stupid questions.

Thanks so much,
David


From dalcinl at gmail.com  Tue Jul  7 20:24:14 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Jul 2009 15:24:14 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A53862A.10206@student.matnat.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
Message-ID: <e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>

On Tue, Jul 7, 2009 at 2:30 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> On Tue, Jul 7, 2009 at 10:47 AM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> Or that #333-like behaviour will *only* be done if you use a "?"? If so,
>>> for what gain? -- some fewer lines of C code? Why does it hurt to
>>> "second-guess the user", as you put it in the OP?
>>>
>>
>> Because of your fist question and my previous comment. Second-guessing
>> will be a bad thing...
>
> I disagree, I think second-guessing is wonderful. (That is, if you want
> me to understand this, you must explain more -- of course, it is not a
> necessity that I understand this.)
>
> If the only effect is to save a few lines of C and a few cycles of CPU
> time for gcc, then I don't think it is worth it to complicate the language.
>

It is not about saving lines of CPU cycles... It's about providing a
way for users to say: Hey! This extern ctypedef is EXACTLY what I
declared Why is Cython trying to be smart and second-guess me, I'm
adult enough ...
Moreover, what I'm proposing is 100 % backwards, and the change to the
languaje is rather minimal...

Dag, are you 100 % confident that treating all extern ctypedef
integrals as unknown-size will not break something or let
overflow/truncation errors pass silently?


> I do think this could be good to have if we inserted code at module init
> time to raise exceptions if sizeof(foo) != sizeof(bar) for all external
> typedefs, so that exactness is *enforced* in those situations. (That may
> break some backwards compatability but I think it is OK myself.) If that
> approach is also taken, at the same time, and this isn't done until
> 0.12, then +1.
>

Sorry, I'm confused here... In which situation do you want to *enfoce*
sizeof(foo) == sizeof(bar) ? Can you provide some code snippet? Do you
mean that if I write this in mpi4py

cdef extern from "mpi.h":
     ctypedef signed long MPI_Aint

then mpi4py will fail to import on Win64 (MPI_Aint is a address-sized
signed integral)?


> But otherwise I'm -1 since it adds thing to the language without any
> benefit at all (that I can see).
>
>> BTW, this likely should affect you work on member descriptor, right?
>
> It already second-guesses now; the only effect would be to speed up gcc
> compilation with a totally insignificant amount per member; not worth
> the time to make the likely trivial commit on Cython IMO.
>

I agree on this particular point... member descriptors is something
that is not worth changing. Or perhaps it would make sense to change
the whole thing? We could use the current size-agnostic handling of
ctypedef's for ALL integral types, even the builtin ones... That would
certainly make the Cython codebase simpler, right?





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Jul  7 21:31:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 21:31:44 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>	<4A5351E0.7020009@student.matnat.uio.no>	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
Message-ID: <4A53A2A0.5060109@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Jul 7, 2009 at 2:30 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Lisandro Dalcin wrote:
>>> On Tue, Jul 7, 2009 at 10:47 AM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> Or that #333-like behaviour will *only* be done if you use a "?"? If so,
>>>> for what gain? -- some fewer lines of C code? Why does it hurt to
>>>> "second-guess the user", as you put it in the OP?
>>>>
>>> Because of your fist question and my previous comment. Second-guessing
>>> will be a bad thing...
>> I disagree, I think second-guessing is wonderful. (That is, if you want
>> me to understand this, you must explain more -- of course, it is not a
>> necessity that I understand this.)
>>
>> If the only effect is to save a few lines of C and a few cycles of CPU
>> time for gcc, then I don't think it is worth it to complicate the language.
>>
> 
> It is not about saving lines of CPU cycles... It's about providing a
> way for users to say: Hey! This extern ctypedef is EXACTLY what I
> declared Why is Cython trying to be smart and second-guess me, I'm
> adult enough ...

*If* Cython can successfully generate generic code that works in all 
cases, then I don't see a need for an option to not do so, even if it 
makes me look more macho :-)

> Moreover, what I'm proposing is 100 % backwards, and the change to the
> languaje is rather minimal...

What worries me is documenting and explaining the various modes of 
operation; particularily because the current external typedef mode is so 
hard to explain (people are already relying on the second-guessing that 
is there).

> 
> Dag, are you 100 % confident that treating all extern ctypedef
> integrals as unknown-size will not break something or let
> overflow/truncation errors pass silently?

No, but such issues must be fixed for the ?-syntax too...

>> I do think this could be good to have if we inserted code at module init
>> time to raise exceptions if sizeof(foo) != sizeof(bar) for all external
>> typedefs, so that exactness is *enforced* in those situations. (That may
>> break some backwards compatability but I think it is OK myself.) If that
>> approach is also taken, at the same time, and this isn't done until
>> 0.12, then +1.
>>
> 
> Sorry, I'm confused here... In which situation do you want to *enfoce*
> sizeof(foo) == sizeof(bar) ? Can you provide some code snippet? Do you
> mean that if I write this in mpi4py
> 
> cdef extern from "mpi.h":
>      ctypedef signed long MPI_Aint
> 
> then mpi4py will fail to import on Win64 (MPI_Aint is a address-sized
> signed integral)?

What I mean is that in that in modules importing this code, the module 
initialization function gets something very similar to this piece 
inserted (probably PyErr_Format and a constant message instead though):

if (sizeof(signed long) != sizeof(MPI_Aint) || (MPI_Aint)-1 != -1
   || (MPI_Aint)1 / 2 != 0) {
     PyErr_String(PyExc_RuntimeError, "MPI_Aint is not 'signed int' on 
this system (consider declaring it as 'cdef signed int? MPI_Aint' if
the size is not known in the Cython code)");
}

This will make sure that backwards compatability is *broken* (in a 
friendly way) in cases where people are currently relying on 
second-guessing for external types (like with buffers).

Most cases are not broken though.

If this is done, then the feature is easy to understand and document and 
I'm definitely +1.

But if this is not done, then the situation just becomes too unclear for 
external typedefs (it second-guesses in some cases and not in others).

As for division etc. (of course, this would be good to have anyway, just 
making a note): What is involved is CNumericType.rank, which should be 
set to None in the case of ?-declaration. Any code depending on 
comparison of this rank should not accept ?-types (give syntax error).

Grep shows that ".rank" occurs 19 times, so this shouldn't be too hard 
(i.e. add a check for None and an error message at each point where the 
rank is needed).

> I agree on this particular point... member descriptors is something
> that is not worth changing. Or perhaps it would make sense to change
> the whole thing? We could use the current size-agnostic handling of
> ctypedef's for ALL integral types, even the builtin ones... That would
> certainly make the Cython codebase simpler, right?

Only by a single if-test as it is now.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jul  7 21:33:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 21:33:58 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A53A2A0.5060109@student.matnat.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>	<4A5351E0.7020009@student.matnat.uio.no>	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>	<4A53862A.10206@student.matnat.uio.no>	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<4A53A2A0.5060109@student.matnat.uio.no>
Message-ID: <4A53A326.8090506@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Lisandro Dalcin wrote:
>> On Tue, Jul 7, 2009 at 2:30 PM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> Lisandro Dalcin wrote:
>>>> On Tue, Jul 7, 2009 at 10:47 AM, Dag Sverre
>>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>>> Or that #333-like behaviour will *only* be done if you use a "?"? If so,
>>>>> for what gain? -- some fewer lines of C code? Why does it hurt to
>>>>> "second-guess the user", as you put it in the OP?
>>>>>
>>>> Because of your fist question and my previous comment. Second-guessing
>>>> will be a bad thing...
>>> I disagree, I think second-guessing is wonderful. (That is, if you want
>>> me to understand this, you must explain more -- of course, it is not a
>>> necessity that I understand this.)
>>>
>>> If the only effect is to save a few lines of C and a few cycles of CPU
>>> time for gcc, then I don't think it is worth it to complicate the language.
>>>
>> It is not about saving lines of CPU cycles... It's about providing a
>> way for users to say: Hey! This extern ctypedef is EXACTLY what I
>> declared Why is Cython trying to be smart and second-guess me, I'm
>> adult enough ...
> 
> *If* Cython can successfully generate generic code that works in all 
> cases, then I don't see a need for an option to not do so, even if it 
> makes me look more macho :-)
> 
>> Moreover, what I'm proposing is 100 % backwards, and the change to the
>> languaje is rather minimal...
> 
> What worries me is documenting and explaining the various modes of 
> operation; particularily because the current external typedef mode is so 
> hard to explain (people are already relying on the second-guessing that 
> is there).
> 
>> Dag, are you 100 % confident that treating all extern ctypedef
>> integrals as unknown-size will not break something or let
>> overflow/truncation errors pass silently?
> 
> No, but such issues must be fixed for the ?-syntax too...
> 
>>> I do think this could be good to have if we inserted code at module init
>>> time to raise exceptions if sizeof(foo) != sizeof(bar) for all external
>>> typedefs, so that exactness is *enforced* in those situations. (That may
>>> break some backwards compatability but I think it is OK myself.) If that
>>> approach is also taken, at the same time, and this isn't done until
>>> 0.12, then +1.
>>>
>> Sorry, I'm confused here... In which situation do you want to *enfoce*
>> sizeof(foo) == sizeof(bar) ? Can you provide some code snippet? Do you
>> mean that if I write this in mpi4py
>>
>> cdef extern from "mpi.h":
>>      ctypedef signed long MPI_Aint
>>
>> then mpi4py will fail to import on Win64 (MPI_Aint is a address-sized
>> signed integral)?
> 
> What I mean is that in that in modules importing this code, the module 
> initialization function gets something very similar to this piece 
> inserted (probably PyErr_Format and a constant message instead though):
> 
> if (sizeof(signed long) != sizeof(MPI_Aint) || (MPI_Aint)-1 != -1
>    || (MPI_Aint)1 / 2 != 0) {
>      PyErr_String(PyExc_RuntimeError, "MPI_Aint is not 'signed int' on 
> this system (consider declaring it as 'cdef signed int? MPI_Aint' if
> the size is not known in the Cython code)");
> }
> 
> This will make sure that backwards compatability is *broken* (in a 
> friendly way) in cases where people are currently relying on 
> second-guessing for external types (like with buffers).
> 
> Most cases are not broken though.
> 
> If this is done, then the feature is easy to understand and document and 
> I'm definitely +1.

Just a note: I'm actually really, really happy if this happens, I'm just 
trying to refine it a bit :-) If numpy.pxd said

ctypedef int? int16_t

I would get a lot less questions about how it works...

-- 
Dag Sverre

From ondrej at certik.cz  Tue Jul  7 21:33:46 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 7 Jul 2009 13:33:46 -0600
Subject: [Cython] problem with pure python mode
In-Reply-To: <85b5c3130906181829u330f95edw6339c719cd724671@mail.gmail.com>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
	<12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
	<85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>
	<7B1321FB-58A6-4CA3-927A-54FA609A0336@math.washington.edu>
	<85b5c3130906181829u330f95edw6339c719cd724671@mail.gmail.com>
Message-ID: <85b5c3130907071233q6f797b78w12886f264580d0ce@mail.gmail.com>

On Thu, Jun 18, 2009 at 7:29 PM, Ondrej Certik<ondrej at certik.cz> wrote:
> On Tue, Jun 16, 2009 at 11:22 PM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On Jun 10, 2009, at 5:45 PM, Ondrej Certik wrote:
>>
>>> On Wed, Apr 29, 2009 at 4:00 PM, Robert
>>> Bradshaw<robertwb at math.washington.edu> wrote:
>>>> On Apr 29, 2009, at 2:49 PM, Ondrej Certik wrote:
>>>>
>>>>>>>
>>>>>>> Not sure--have you tried without the locals decorator in the
>>>>>>> Python
>>>>>>> source? I hope this isn't a regression.
>>>>>>
>>>>>> Yes, as you can see, the locals is commented out. I'll try to
>>>>>> bisect
>>>>>> it to see which commit broke it.
>>>>>
>>>>> Ok, so it never worked. I am a bit confused, because I thought the
>>>>> pure python mode worked fine for me before.
>>>>>
>>>>> In the current cython --- is there any way to keep a pure python
>>>>> code,
>>>>> and create an accompanying pxd file that would annotate the cdef
>>>>> functions and classes?
>>>>
>>>> This should work, that's how I implemented this a long time ago. If
>>>> it's not working now it's because something broke (or, I'm not
>>>> promising that it's completely bug-free, but it should be good--we
>>>> use it to bootstrap Cython itself).
>>>
>>> So I made it to work by observing how things are done in Cython
>>> itself.
>>>
>>> If you do:
>>>
>>> git clone git://github.com/certik/cython-test.git
>>> cd cython-test
>>> cython fact.pyx
>>>
>>> then it produces nicely optimized fact.c file. However, if you do:
>>>
>>> cython fact.py
>>>
>>> it produces a fact.c, that compiles, but it is not optimized --- the
>>> "i" and "r" variables are python objects, not ints, even though my
>>> fact.pxd file contains:
>>>
>>> import cython
>>>
>>> @cython.locals(i=cython.int, r=cython.int)
>>> cdef int factorial(int n)
>>>
>>>
>>> Do you see anything wrong with my files? Is this supposed to work?
>>>
>>> I think it is not implemented yet, because when I looked at the cython
>>> sources, it doesn't work there either. This is the relevant part of
>>> Scanning.pxd:
>>>
>>> ? ? @cython.locals(current_level=cython.long, new_level=cython.long)
>>> ? ? cpdef indentation_action(self, text)
>>>
>>> and here is the declaration in the generated Scanning.c:
>>>
>>> ? PyObject *__pyx_v_new_level;
>>>
>>>
>>> And consider for example the following chunk:
>>>
>>> ? /* "/home/ondrej/repos/cython-devel/Cython/Compiler/Scanning.py":398
>>> ?* ? ? ? ? new_level = len(text)
>>> ?* ? ? ? ? #print "Changing indent level from", current_level, "to",
>>> new_level ###
>>> ?* ? ? ? ? if new_level == current_level: ? ? ? ? ? ? # <<<<<<<<<<<<<<
>>> ?* ? ? ? ? ? ? return
>>> ?* ? ? ? ? elif new_level > current_level:
>>> ?*/
>>> ? __pyx_t_4 = PyObject_RichCompare(__pyx_v_new_level,
>>> __pyx_v_current_level, Py_EQ); if (unlikely(!__pyx_t_4))
>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno =
>>> __LINE__; goto __pyx_L1_error;}
>>> ? __Pyx_GOTREF(__pyx_t_4);
>>> ? __pyx_t_3 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_3
>>> < 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno
>>> = __LINE__; goto __pyx_L1_error;}
>>> ? __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
>>> ? if (__pyx_t_3) {
>>>
>>>
>>> both the new_level and current_level are declared as long, but above
>>> you can see that Cython is calling lots of Python C/API methods...
>>
>> Looks like a bug to me.
>>
>> http://trac.cython.org/cython_trac/ticket/330
>
> Do you have some idea where the problem is? You wrote "Parsing" in the
> ticket. This pure Python mode is very important for me, so I tried to
> get it fixed, but I need to invest more time to learn Cython internals
> to produce something useful, hopefully over the weekend I'll find time
> to dig into it.

Let me know if there is any progress on this one. We are slowly
approaching the state with sympy, when I will need this.

Or any hints how I could fix this myself.

Ondrej

From dalcinl at gmail.com  Tue Jul  7 22:54:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Jul 2009 17:54:28 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A53A2A0.5060109@student.matnat.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<4A53A2A0.5060109@student.matnat.uio.no>
Message-ID: <e7ba66e40907071354r3b4fe0e4y11eaab46f528635a@mail.gmail.com>

On Tue, Jul 7, 2009 at 4:31 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
>
> if (sizeof(signed long) != sizeof(MPI_Aint) || (MPI_Aint)-1 != -1
> ? || (MPI_Aint)1 / 2 != 0) {
> ? ? PyErr_String(PyExc_RuntimeError, "MPI_Aint is not 'signed int' on
> this system (consider declaring it as 'cdef signed int? MPI_Aint' if
> the size is not known in the Cython code)");
> }
>

I'm strongly -1 on this.

If we ever do this, it will be impossible for a mpi4py release to ship
the generated C sources... Cython will be required at build time to
generate the C sources, plus some 'configure'-like step to figure out
the right C type in order to hot-fix the Cython code, and all that
implemented in pure distutils (you likely know how hard that is) for
maximum compatibility...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Jul  7 23:08:28 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 7 Jul 2009 18:08:28 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A53A326.8090506@student.matnat.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<4A53A2A0.5060109@student.matnat.uio.no>
	<4A53A326.8090506@student.matnat.uio.no>
Message-ID: <e7ba66e40907071408s1ebe1e05g9f788a893b73c965@mail.gmail.com>

On Tue, Jul 7, 2009 at 4:33 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
>
> Just a note: I'm actually really, really happy if this happens, I'm just
> trying to refine it a bit :-) If numpy.pxd said
>
> ctypedef int? int16_t
>
> I would get a lot less questions about how it works...
>

Did you mean "a lot more questions" ... ?


Well, let's stop this thread right now... I withdraw my proposal.




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Jul  7 23:13:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 23:13:40 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907071354r3b4fe0e4y11eaab46f528635a@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>	<4A5351E0.7020009@student.matnat.uio.no>	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>	<4A53862A.10206@student.matnat.uio.no>	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>	<4A53A2A0.5060109@student.matnat.uio.no>
	<e7ba66e40907071354r3b4fe0e4y11eaab46f528635a@mail.gmail.com>
Message-ID: <4A53BA84.6060902@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Jul 7, 2009 at 4:31 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> if (sizeof(signed long) != sizeof(MPI_Aint) || (MPI_Aint)-1 != -1
>>   || (MPI_Aint)1 / 2 != 0) {
>>     PyErr_String(PyExc_RuntimeError, "MPI_Aint is not 'signed int' on
>> this system (consider declaring it as 'cdef signed int? MPI_Aint' if
>> the size is not known in the Cython code)");
>> }
>>
> 
> I'm strongly -1 on this.
> 
> If we ever do this, it will be impossible for a mpi4py release to ship
> the generated C sources... Cython will be required at build time to
> generate the C sources, plus some 'configure'-like step to figure out
> the right C type in order to hot-fix the Cython code, and all that
> implemented in pure distutils (you likely know how hard that is) for
> maximum compatibility...

I think you misunderstood my suggestion. No configure step needed.

Basically it would take

cdef extern from ...:
     ctypedef FOO BAR

and turn into C code, like the above, containing sizeof(FOO) and 
sizeof(BAR), to validate that it holds true.

That's all.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jul  7 23:16:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 07 Jul 2009 23:16:58 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907071408s1ebe1e05g9f788a893b73c965@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>	<4A5351E0.7020009@student.matnat.uio.no>	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>	<4A53862A.10206@student.matnat.uio.no>	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>	<4A53A2A0.5060109@student.matnat.uio.no>	<4A53A326.8090506@student.matnat.uio.no>
	<e7ba66e40907071408s1ebe1e05g9f788a893b73c965@mail.gmail.com>
Message-ID: <4A53BB4A.2000201@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, Jul 7, 2009 at 4:33 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Just a note: I'm actually really, really happy if this happens, I'm just
>> trying to refine it a bit :-) If numpy.pxd said
>>
>> ctypedef int? int16_t
>>
>> I would get a lot less questions about how it works...
>>
> 
> Did you mean "a lot more questions" ... ?

No, a lot less. I meant to say that I'm really happy if your proposal is 
implemented!, because then it will be more obvious what is going on, and 
I will get less questions about it, which is a good thing.

> Well, let's stop this thread right now... I withdraw my proposal.

Woah, deja vu...

But please don't!! I shouldn't be allowed to get in the way of this; if 
you disregard all my comments I don't mind.

Sorry if I made a bigger issue of this than it deserved.

-- 
Dag Sverre


From sccolbert at gmail.com  Wed Jul  8 01:58:48 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 7 Jul 2009 19:58:48 -0400
Subject: [Cython] ideas for exposing numpy arrays to other C libraries
Message-ID: <7f014ea60907071658sf9e379h71f8b230f22d1370@mail.gmail.com>

I'd just like so input as to whether I have the right idea.

I would like to subclass numpy's ndarray (on the python side), this new
class will call functions in a C library that expects an array type that's
defined in its own library.
This library also uses a strided array model.

So my idea is to create an extension type in Cython that essentially
reimplements this C libraries struct and will be stored in a private
attribute of my ndarray subclass. When I instantiate an object of my ndarray
subclass, I will populate the private Cython struct with the appropriate
values and the pointer to the ndarray data. Then, when I call a function of
the C library (the function is also wrapped with cython) I pass the private
attribute as an argument.

Is that about right?

Cheers,

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090707/89f67aff/attachment.htm 

From greg.ewing at canterbury.ac.nz  Wed Jul  8 02:43:07 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 08 Jul 2009 12:43:07 +1200
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>
References: <4A524667.7040409@student.matnat.uio.no>
	<512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>
Message-ID: <4A53EB9B.90603@canterbury.ac.nz>

Robert Bradshaw wrote:
> On Jul 6, 2009, at 11:45 AM, Dag Sverre Seljebotn wrote:

>>cdef extern borrowed object PyList_GET_ITEM(object, Py_ssize_t)
>>cdef extern void PyList_SET_ITEM(object, Py_ssize_t, borrowed object)

For parameters, I'd use 'stolen' instead of 'borrowed':

   cdef extern borrowed object PyList_GET_ITEM(object, Py_ssize_t)
   cdef extern void PyList_SET_ITEM(object, Py_ssize_t, stolen object)

-- 
Greg

From robertwb at math.washington.edu  Wed Jul  8 07:02:35 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 22:02:35 -0700
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A48C1D5.80204@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>	<4A45B2AB.9060609@behnel.de>	<4A45C421.4090005@student.matnat.uio.no>	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>	<4A469706.8050509@student.matnat.uio.no>
	<8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
	<4A48C1D5.80204@student.matnat.uio.no>
Message-ID: <B5D7B245-31CE-4CC6-AEF2-805E278D14F0@math.washington.edu>

On Jun 29, 2009, at 6:29 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jun 27, 2009, at 3:02 PM, Dag Sverre Seljebotn wrote:
>>
>>> It's just the idea of being able to wrap "C++ libraries if they are
>>> not
>>> too strange" I have some issues with.
>>
>> I have issues with this approach too, so no worries about selling
>> short on that front :). (Wrapping strange libraries may be non-
>> trivial, but hopefully no more so than using them.) Also, the final
>> milestone of the project is to be able to wrap all of STL, which
>> although it doesn't cover everything, does quite a bit.
>
> Sure, was just pointing out that a given path might lead you to  
> need to
> define __inc__ or __deref__ and so on in order to be able to  
> support all
> of STL (in fact __deref__ is a much better example; as the return type
> of "*it" must be declared, and "it[0]" may not be legal).
>
> I'd rather prefer C++ syntax over __inc__ or __deref__ (i.e. if we
> didn't manage to "get rid of" the C++ semantics in the first place). I
> haven't found an alternative to inline C++ code OR supporting only a
> subset, so if you're still against inline C++ code I'm inclined to  
> give
> a +1 to C++ syntax, also on the constructor.
>
> But I don't really worry, and will leave it at this.

To summarize, we're using Python syntax for now, and if there is time  
at the end of GSoC to implement references (which are a dependancy of  
the C++ style way of doing things) we'll reconsider at that point  
(this will also give us a bit of time to see how things feel).

- Robert



From robertwb at math.washington.edu  Wed Jul  8 07:13:05 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 22:13:05 -0700
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <4A5346C8.5010802@student.matnat.uio.no>
References: <4A509B88.50307@behnel.de>	<4A522A82.9080904@student.matnat.uio.no>	<4A5338CB.6020309@behnel.de>	<4A533D5C.5070009@student.matnat.uio.no>
	<4A53450D.1000004@behnel.de>
	<4A5346C8.5010802@student.matnat.uio.no>
Message-ID: <1B122DC3-E2AD-477F-B683-193EE70CF2DB@math.washington.edu>

On Jul 7, 2009, at 5:59 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Stefan Behnel wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>> a) Auto-cimport (if a directive is set, which defaults to on?).  
>>>>> I.e. if
>>>>> you do a Python import, and a pxd file can be found, a cimport  
>>>>> is done
>>>>> automatically.
>>>> +0.7 given that this may really have a performance impact, as  
>>>> Robert
>>>> stated. There should at least be a cache for the standard  
>>>> include path that
>>>> survives multiple compilations in one compiler run (another  
>>>> orthogonal
>>>> feature that would be nice for .pxd files in general).
>>> with cython.nopxd:
>>>      import mymodule
>>>
>>> ?
>>
>> -1
>>
>> Besides being the wrong way round (most Python modules do not have  
>> a .pxd
>> file), this is clearly something that Cython should handle behind the
>> scenes. User code shouldn't be impacted by it. If anything, a  
>> compiler
>> option would fit. But thinking about that now smells like a premature
>> optimisation to me.
>
> But (expert) users must have control and care about this, it cannot
> happen behind the scenes anyway! I.e. with no automatic cimport,  
> you can do
>
> import itertools
>
> ...
>
> itertools = None
> ...
>
> With auto cimport (like you seemed to want originally, at least for  
> the
> specific case of itertools) this is a syntax error.

I think it would be possible (though not trivial--cimports are  
necessarily resolved in the same stage as declaration analyses) to  
make it fall back to being a regular import (with a warning?) in this  
case.

> This early-binding is the big change in semantics; everything else is
> just details.

Exactly. For example, if I do

import math
print math.sin(foo()), math.sin(bar())

I should be doing an attribute lookup on math twice, just in case bar  
was defined as

def bar():
     import math
     math.sin = math.cos
     return 1

> But I think it makes sense to say that Cython does, for module-level
> imports at the top of the file, early bind them, as it is OK anyway  
> for
> 99% of the usecases (and a syntax error is raised anyway at  
> "itertool =
> None", which could say "you have to turn off auto-cimport").

I could live with this too.

- Robert


From robertwb at math.washington.edu  Wed Jul  8 07:32:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 22:32:11 -0700
Subject: [Cython] "object" keyword in cython
In-Reply-To: <AE0A6018-7CDC-4E1E-BCA8-119D22751507@gmail.com>
References: <AE0A6018-7CDC-4E1E-BCA8-119D22751507@gmail.com>
Message-ID: <90A851D4-4B31-425A-96C0-E13B11F5FBE8@math.washington.edu>

On Jul 7, 2009, at 11:15 AM, David Simmons-Duffin wrote:

> I'm trying to interface with some C code that defines a type called
> "object"

The type "object" is used to denote a Python object (essentially a  
refcounted PyObject*), which is why you're having trouble redefining it.

> typedef union {
>    struct {
>             objtype type; reftype nref;
>    } any;
>    intcel i;
>    bigint b;
>    vector v;
>    matrix m;
>    poly   pl;
>    simpgrp s;
>    group g;
>    tekst  t;
> } objcel, *object;
>
> However, if I try to write some cython code involving an object:
>
> cdef extern object groupmake(char lietype, int rank)
> cdef object g
> g = groupmake("A", 3)
>
> It's interpreted as a PyObject in the c code:
>
> PyObject *__pyx_t_1 = NULL;
> ...
> __pyx_t_1 = groupmake('A', 2);
>
> One possible solution is to just replace the word "object" in the C
> source with something else throughout.  But I'd like to make as few
> modifications of the C source as possible.  Is there an alternate
> solution?

Yes, you can use the magic c-namespace-renaming quotes. For your  
example.

cdef extern from *:
     ctypedef int objtype
     ctypedef int reftype
     ctypedef int intcel
     cdef struct any_struct:
         objtype type
         reftype nref
     ctypedef union objcel:
         any_struct any
         intcel i
         ...
     ctypedef objcel *my_c_object "object"

Now use my_c_object, and in the c sources it will come out as object.

>
> I actually have another unrelated question: what is the correct cython
> syntax for the union statement at the top of this message?  Cython
> doesn't seem to like me putting
>
> struct any:
> 	objtype type
> 	reftype nfref
>
> in the middle of a union statement.

Cython doesn't support anonymous structs. Give that struct a name,  
and then you can declare your variable with that type. (The name you  
choose will never actually be used.)

> And I'm also not sure what to do
> with "objcel, *object" at the end there.  I'm a novice at this stuff,
> so apologies if these are stupid questions.

No worries. The only stupid questions are those that should have been  
answered by a couple minutes of googling and reading the wiki. These  
ones don't fall into that boat.

Just out of curiosity, what are you wrapping? It looks like a math  
library of sorts.

- Robert


From robertwb at math.washington.edu  Wed Jul  8 07:48:52 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 22:48:52 -0700
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
Message-ID: <7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>

On Jul 7, 2009, at 11:24 AM, Lisandro Dalcin wrote:

> On Tue, Jul 7, 2009 at 2:30 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Lisandro Dalcin wrote:
>>> On Tue, Jul 7, 2009 at 10:47 AM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> Or that #333-like behaviour will *only* be done if you use a  
>>>> "?"? If so,
>>>> for what gain? -- some fewer lines of C code? Why does it hurt to
>>>> "second-guess the user", as you put it in the OP?
>>>>
>>>
>>> Because of your fist question and my previous comment. Second- 
>>> guessing
>>> will be a bad thing...
>>
>> I disagree, I think second-guessing is wonderful. (That is, if you  
>> want
>> me to understand this, you must explain more -- of course, it is  
>> not a
>> necessity that I understand this.)
>>
>> If the only effect is to save a few lines of C and a few cycles of  
>> CPU
>> time for gcc, then I don't think it is worth it to complicate the  
>> language.
>>
>
> It is not about saving lines of CPU cycles... It's about providing a
> way for users to say: Hey! This extern ctypedef is EXACTLY what I
> declared Why is Cython trying to be smart and second-guess me, I'm
> adult enough ...
> Moreover, what I'm proposing is 100 % backwards, and the change to the
> languaje is rather minimal...

The main point of using external typedefs is that one often doesn't  
know the exact size of the type, e.g. it's determined at c-compile  
and/or configure time. (Otherwise, one could just use int, long, or  
whatever it actually *is* and the C compiler should never complain  
after unwinding all the definitions...) I am -1 on requiring a ? when  
the type is not explicitly known.

If one needs the sized to match exactly, one can "assert sizeof 
(my_int_type) == sizeof(long)."

The only difference I see with the ? notation is that it would be  
explicitly unranked (whereas the standard typedef receives its  
ranking from its definition). I guess in that case certain operations  
(like arithmetic with other types) would be disallowed? I just don't  
see this as a common enough use case to need new syntax (though I'd  
be curious to hear if it does fit the bill for what someone's trying  
to do).

> I agree on this particular point... member descriptors is something
> that is not worth changing. Or perhaps it would make sense to change
> the whole thing? We could use the current size-agnostic handling of
> ctypedef's for ALL integral types, even the builtin ones... That would
> certainly make the Cython codebase simpler, right?

ints, longs, etc. are so common that it's probably not worth the C  
code bloat.

- Robert



From robertwb at math.washington.edu  Wed Jul  8 07:58:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 7 Jul 2009 22:58:36 -0700
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <4A53325F.8060503@behnel.de>
References: <4A524667.7040409@student.matnat.uio.no>	<512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>	<4A532ABA.1000701@behnel.de>
	<4A532EA4.6060801@student.matnat.uio.no>
	<4A53325F.8060503@behnel.de>
Message-ID: <98136C37-F6C1-40AC-A40C-434BCF019718@math.washington.edu>

On Jul 7, 2009, at 4:32 AM, Stefan Behnel wrote:

> Dag Sverre Seljebotn wrote:
>> it would be syntax candy for an extra incref. I can work around
>> this now by:
>>
>> cdef extern void Real_PyList_SET_ITEM "PyList_SET_ITEM"(object,
>> Py_ssize_t, object)
>>
>> cdef inline void PyList_SET_ITEM(object a, Py_ssize_t b, object c):
>>      Py_INCREF(c)
>>      Real_PyList_SET_ITEM(a, b, c)
>>
>> For ~20 functions I am likely not going to bother; while simply  
>> declaring
>>
>> cdef extern void PyList_SET_ITEM(object, Py_ssize_t, stolen object):
>
> I'm fine with that, as long as we only allow it inside of argument  
> lists
> (preferrably only "extern" argument lists).

Of course, there may already be code out there that counts on the  
true PyList_SET_ITEM semantics.

Another option:

cdef extern from *:
     ctypedef void stolen_object "PyObject*"

cdef inline stolen_object steal(object):
     Py_INCREF(o)
     return <stolen_object>o

cdef extern void PyList_SET_ITEM(object, Py_ssize_t, stolen_object)


One would then call it as

PyList_SET_ITEM(L, 0, steal(item))

(Of course, PyList_SET_ITEM is worse than this, as one needs to  
manually (x)decref the original spot in the list as well... This  
makes me lean towards not providing it as a friendly function taking  
ordinary objects).

- Robert


From stefan_ml at behnel.de  Wed Jul  8 09:29:59 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 08 Jul 2009 09:29:59 +0200
Subject: [Cython] Should we optimise certain stdlib modules?
In-Reply-To: <1B122DC3-E2AD-477F-B683-193EE70CF2DB@math.washington.edu>
References: <4A509B88.50307@behnel.de>	<4A522A82.9080904@student.matnat.uio.no>	<4A5338CB.6020309@behnel.de>	<4A533D5C.5070009@student.matnat.uio.no>	<4A53450D.1000004@behnel.de>	<4A5346C8.5010802@student.matnat.uio.no>
	<1B122DC3-E2AD-477F-B683-193EE70CF2DB@math.washington.edu>
Message-ID: <4A544AF7.9010900@behnel.de>


Robert Bradshaw wrote:
> For example, if I do
> 
> import math
> print math.sin(foo()), math.sin(bar())
> 
> I should be doing an attribute lookup on math twice, just in case bar  
> was defined as
> 
> def bar():
>      import math
>      math.sin = math.cos
>      return 1
>
> On Jul 7, 2009, at 5:59 AM, Dag Sverre Seljebotn wrote:
>> But I think it makes sense to say that Cython does, for module-level
>> imports at the top of the file, early bind them, as it is OK anyway  
>> for 99% of the usecases (and a syntax error is raised anyway at  
>> "itertool =
>> None", which could say "you have to turn off auto-cimport").
> 
> I could live with this too.

I think we should only optimise imports that happen at the module level. If
the module is (re-)imported anywhere else, or the imported name is
reassigned anywhere in the module, this would simply disable the
optimisation. This is easy to check for. A warning would be OK in that
case, and I'm pretty sure users can live with this.

We'd have to rethink this restriction if we ever consider changing the
functionality of an imported module, but as long as it's really only an
optimisation, I'm fine with handling only the most simple and most common
case, and just make sure that we don't break things.

Stefan

From stefan_ml at behnel.de  Wed Jul  8 10:05:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 08 Jul 2009 10:05:17 +0200
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <98136C37-F6C1-40AC-A40C-434BCF019718@math.washington.edu>
References: <4A524667.7040409@student.matnat.uio.no>	<512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>	<4A532ABA.1000701@behnel.de>	<4A532EA4.6060801@student.matnat.uio.no>	<4A53325F.8060503@behnel.de>
	<98136C37-F6C1-40AC-A40C-434BCF019718@math.washington.edu>
Message-ID: <4A54533D.2040206@behnel.de>


Robert Bradshaw wrote:
> Of course, there may already be code out there that counts on the  
> true PyList_SET_ITEM semantics.

Certainly.


> Another option:
> 
> cdef extern from *:
>      ctypedef void stolen_object "PyObject*"
> 
> cdef inline stolen_object steal(object):
>      Py_INCREF(o)
>      return <stolen_object>o
> 
> cdef extern void PyList_SET_ITEM(object, Py_ssize_t, stolen_object)
> 
> 
> One would then call it as
> 
> PyList_SET_ITEM(L, 0, steal(item))
> 
> (Of course, PyList_SET_ITEM is worse than this, as one needs to  
> manually (x)decref the original spot in the list as well... This  
> makes me lean towards not providing it as a friendly function taking  
> ordinary objects).

That's because it's a macro. It's a different situation for functions -
although I can't currently find any in the C-API that really steal
references. If users actually have to special case stolen references
anyway, maybe this whole "stolen" business is really a non-problem...

Stefan


From dsurviver at gmail.com  Wed Jul  8 11:26:32 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Wed, 8 Jul 2009 06:26:32 -0300
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <B5D7B245-31CE-4CC6-AEF2-805E278D14F0@math.washington.edu>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no> <4A45B2AB.9060609@behnel.de>
	<4A45C421.4090005@student.matnat.uio.no>
	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
	<4A469706.8050509@student.matnat.uio.no>
	<8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
	<4A48C1D5.80204@student.matnat.uio.no>
	<B5D7B245-31CE-4CC6-AEF2-805E278D14F0@math.washington.edu>
Message-ID: <45239150907080226v796f18f8ldf1094633e1c5cba@mail.gmail.com>

2009/7/8 Robert Bradshaw <robertwb at math.washington.edu>:
> On Jun 29, 2009, at 6:29 AM, Dag Sverre Seljebotn wrote:
>
>> Robert Bradshaw wrote:
>>> On Jun 27, 2009, at 3:02 PM, Dag Sverre Seljebotn wrote:
>>>
>>>> It's just the idea of being able to wrap "C++ libraries if they are
>>>> not
>>>> too strange" I have some issues with.
>>>
>>> I have issues with this approach too, so no worries about selling
>>> short on that front :). (Wrapping strange libraries may be non-
>>> trivial, but hopefully no more so than using them.) Also, the final
>>> milestone of the project is to be able to wrap all of STL, which
>>> although it doesn't cover everything, does quite a bit.
>>
>> Sure, was just pointing out that a given path might lead you to
>> need to
>> define __inc__ or __deref__ and so on in order to be able to
>> support all
>> of STL (in fact __deref__ is a much better example; as the return type
>> of "*it" must be declared, and "it[0]" may not be legal).
>>
>> I'd rather prefer C++ syntax over __inc__ or __deref__ (i.e. if we
>> didn't manage to "get rid of" the C++ semantics in the first place). I
>> haven't found an alternative to inline C++ code OR supporting only a
>> subset, so if you're still against inline C++ code I'm inclined to
>> give
>> a +1 to C++ syntax, also on the constructor.
>>
>> But I don't really worry, and will leave it at this.
>
> To summarize, we're using Python syntax for now, and if there is time
> at the end of GSoC to implement references (which are a dependancy of
> the C++ style way of doing things) we'll reconsider at that point
> (this will also give us a bit of time to see how things feel).

If we have time until the end of the program (and I hope we will), we
can use it and see how it will work (if fine or not fine).
If better, we may use it.
For now, we continue using Python syntax. If we decide it will need to
change it, we do it.

I think that even we don't have time for it in the project, it could
be taken after project time-life. I it happens, I would like to
continue this work here :).



-- 
- Danilo Freitas

From thorstenkranz at googlemail.com  Wed Jul  8 14:24:22 2009
From: thorstenkranz at googlemail.com (Thorsten Kranz)
Date: Wed, 8 Jul 2009 14:24:22 +0200
Subject: [Cython] Cython and lapack-library
Message-ID: <f9968a7e0907080524j709334d9l44219de6675aa554@mail.gmail.com>

Hi,

I'm quite new to cython, so maybe my question is trivial. I try to
wrap the lapack-routine dgesv using cython. I was glad when I found

http://codespeak.net/pipermail/cython-dev/2009-May/005702.html

covering exactly that topic. I tried to compile the lapack-part of it,
but then it seemed like the lapack.pxd file was missing. So I went
ahead creating this file, using the informations I could grep from the
slides contained in this tutorial.

But when I do so, it does indeed compile, but on execution of

testlapack.test()

an error occurs that lapack doesn't have the attribute CblasRowMajor.

Can anybody please help me? Has anybody of you done this before or
does anybody have the correct lapack.pxd file?

Greetings,
Thorsten

From dsimmons.com at gmail.com  Wed Jul  8 18:17:10 2009
From: dsimmons.com at gmail.com (David Simmons-Duffin)
Date: Wed, 8 Jul 2009 12:17:10 -0400
Subject: [Cython] "object" keyword in cython
In-Reply-To: <90A851D4-4B31-425A-96C0-E13B11F5FBE8@math.washington.edu>
References: <AE0A6018-7CDC-4E1E-BCA8-119D22751507@gmail.com>
	<90A851D4-4B31-425A-96C0-E13B11F5FBE8@math.washington.edu>
Message-ID: <685DF6AA-356A-45D7-B042-780BEEBFCB72@gmail.com>

On Jul 8, 2009, at 1:32 AM, Robert Bradshaw wrote:

>> One possible solution is to just replace the word "object" in the C
>> source with something else throughout.  But I'd like to make as few
>> modifications of the C source as possible.  Is there an alternate
>> solution?
>
> Yes, you can use the magic c-namespace-renaming quotes. For your
> example.
>
>> I actually have another unrelated question: what is the correct  
>> cython
>> syntax for the union statement at the top of this message?  Cython
>> doesn't seem to like me putting
>>
>> struct any:
>> 	objtype type
>> 	reftype nfref
>>
>> in the middle of a union statement.
>
> Cython doesn't support anonymous structs. Give that struct a name,
> and then you can declare your variable with that type. (The name you
> choose will never actually be used.)

Thanks, your suggestions worked perfectly.

> Just out of curiosity, what are you wrapping? It looks like a math
> library of sorts.

I'm wrapping a program called LiE for doing computations in lie group  
representation theory.

http://www-math.univ-poitiers.fr/~maavl/LiE/

This sort of thing is very useful in high energy theoretical physics,  
which is what I work on.  The typical choice among us theorists for  
doing computations is Mathematica (with the exception of collider  
simulations, which are run with dedicated programs like Pythia).   
However, representation theory is something Mathematica doesn't seem  
to know about, and I haven't been able to find a package for it.

LiE does basically what I want, except that everything is wrapped up  
in a custom language that's not very expressive, hard to debug, and  
generally unpleasant to use.  Python is much lovelier, especially  
since the object-oriented-ness is well-suited for writing  
mathematics.  So I'm doing my best to provide a python interface as an  
alternative to LiE's built-in lexer, parser, and interpreter.

Thanks again for all your help,
David

From robertwb at math.washington.edu  Wed Jul  8 18:18:48 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 8 Jul 2009 09:18:48 -0700
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <45239150907080226v796f18f8ldf1094633e1c5cba@mail.gmail.com>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no>
	<4A45B2AB.9060609@behnel.de>
	<4A45C421.4090005@student.matnat.uio.no>
	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
	<4A469706.8050509@student.matnat.uio.no>
	<8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
	<4A48C1D5.80204@student.matnat.uio.no>
	<B5D7B245-31CE-4CC6-AEF2-805E278D14F0@math.washington.edu>
	<45239150907080226v796f18f8ldf1094633e1c5cba@mail.gmail.com>
Message-ID: <B16A1C3C-394C-404E-9810-C8D41F9DFE6E@math.washington.edu>

On Jul 8, 2009, at 2:26 AM, Danilo Freitas wrote:

> 2009/7/8 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Jun 29, 2009, at 6:29 AM, Dag Sverre Seljebotn wrote:
>>
>>> Robert Bradshaw wrote:
>>>> On Jun 27, 2009, at 3:02 PM, Dag Sverre Seljebotn wrote:
>>>>
>>>>> It's just the idea of being able to wrap "C++ libraries if they  
>>>>> are
>>>>> not
>>>>> too strange" I have some issues with.
>>>>
>>>> I have issues with this approach too, so no worries about selling
>>>> short on that front :). (Wrapping strange libraries may be non-
>>>> trivial, but hopefully no more so than using them.) Also, the final
>>>> milestone of the project is to be able to wrap all of STL, which
>>>> although it doesn't cover everything, does quite a bit.
>>>
>>> Sure, was just pointing out that a given path might lead you to
>>> need to
>>> define __inc__ or __deref__ and so on in order to be able to
>>> support all
>>> of STL (in fact __deref__ is a much better example; as the return  
>>> type
>>> of "*it" must be declared, and "it[0]" may not be legal).
>>>
>>> I'd rather prefer C++ syntax over __inc__ or __deref__ (i.e. if we
>>> didn't manage to "get rid of" the C++ semantics in the first  
>>> place). I
>>> haven't found an alternative to inline C++ code OR supporting only a
>>> subset, so if you're still against inline C++ code I'm inclined to
>>> give
>>> a +1 to C++ syntax, also on the constructor.
>>>
>>> But I don't really worry, and will leave it at this.
>>
>> To summarize, we're using Python syntax for now, and if there is time
>> at the end of GSoC to implement references (which are a dependancy of
>> the C++ style way of doing things) we'll reconsider at that point
>> (this will also give us a bit of time to see how things feel).
>
> If we have time until the end of the program (and I hope we will), we
> can use it and see how it will work (if fine or not fine).
> If better, we may use it.
> For now, we continue using Python syntax. If we decide it will need to
> change it, we do it.
>
> I think that even we don't have time for it in the project, it could
> be taken after project time-life. I it happens, I would like to
> continue this work here :).

You are most certainly encouraged to keep working here! After you're  
initial investment to get to know the compiler, you'll even have an  
easier time of it.

- Robert



From robertwb at math.washington.edu  Wed Jul  8 18:19:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 8 Jul 2009 09:19:45 -0700
Subject: [Cython] Stealing/borrowed references semantics
In-Reply-To: <4A54533D.2040206@behnel.de>
References: <4A524667.7040409@student.matnat.uio.no>	<512D9015-71F5-4755-947A-D9281AF1FA7F@math.washington.edu>	<4A532ABA.1000701@behnel.de>	<4A532EA4.6060801@student.matnat.uio.no>	<4A53325F.8060503@behnel.de>
	<98136C37-F6C1-40AC-A40C-434BCF019718@math.washington.edu>
	<4A54533D.2040206@behnel.de>
Message-ID: <F15CE1FB-6165-403F-8B81-65082CADDAD1@math.washington.edu>

On Jul 8, 2009, at 1:05 AM, Stefan Behnel wrote:

>
> Robert Bradshaw wrote:
>> Of course, there may already be code out there that counts on the
>> true PyList_SET_ITEM semantics.
>
> Certainly.
>
>
>> Another option:
>>
>> cdef extern from *:
>>      ctypedef void stolen_object "PyObject*"
>>
>> cdef inline stolen_object steal(object):
>>      Py_INCREF(o)
>>      return <stolen_object>o
>>
>> cdef extern void PyList_SET_ITEM(object, Py_ssize_t, stolen_object)
>>
>>
>> One would then call it as
>>
>> PyList_SET_ITEM(L, 0, steal(item))
>>
>> (Of course, PyList_SET_ITEM is worse than this, as one needs to
>> manually (x)decref the original spot in the list as well... This
>> makes me lean towards not providing it as a friendly function taking
>> ordinary objects).
>
> That's because it's a macro. It's a different situation for  
> functions -
> although I can't currently find any in the C-API that really steal
> references. If users actually have to special case stolen references
> anyway, maybe this whole "stolen" business is really a non-problem...


Yep. The last thing I want is to make it easy to almost, but not  
quite, get it right. Raw PyObject* arguments make it clear that you  
have to at least think about it.

- Robert



From wstein at gmail.com  Wed Jul  8 18:22:49 2009
From: wstein at gmail.com (William Stein)
Date: Wed, 8 Jul 2009 09:22:49 -0700
Subject: [Cython] "object" keyword in cython
In-Reply-To: <685DF6AA-356A-45D7-B042-780BEEBFCB72@gmail.com>
References: <AE0A6018-7CDC-4E1E-BCA8-119D22751507@gmail.com> 
	<90A851D4-4B31-425A-96C0-E13B11F5FBE8@math.washington.edu> 
	<685DF6AA-356A-45D7-B042-780BEEBFCB72@gmail.com>
Message-ID: <85e81ba30907080922x1c7824at53acef90d1f635bd@mail.gmail.com>

On Wed, Jul 8, 2009 at 9:17 AM, David
Simmons-Duffin<dsimmons.com at gmail.com> wrote:
> On Jul 8, 2009, at 1:32 AM, Robert Bradshaw wrote:
>
>>> One possible solution is to just replace the word "object" in the C
>>> source with something else throughout. ?But I'd like to make as few
>>> modifications of the C source as possible. ?Is there an alternate
>>> solution?
>>
>> Yes, you can use the magic c-namespace-renaming quotes. For your
>> example.
>>
>>> I actually have another unrelated question: what is the correct
>>> cython
>>> syntax for the union statement at the top of this message? ?Cython
>>> doesn't seem to like me putting
>>>
>>> struct any:
>>> ? ? ?objtype type
>>> ? ? ?reftype nfref
>>>
>>> in the middle of a union statement.
>>
>> Cython doesn't support anonymous structs. Give that struct a name,
>> and then you can declare your variable with that type. (The name you
>> choose will never actually be used.)
>
> Thanks, your suggestions worked perfectly.
>
>> Just out of curiosity, what are you wrapping? It looks like a math
>> library of sorts.
>
> I'm wrapping a program called LiE for doing computations in lie group
> representation theory.
>
> http://www-math.univ-poitiers.fr/~maavl/LiE/
>
> This sort of thing is very useful in high energy theoretical physics,
> which is what I work on. ?The typical choice among us theorists for
> doing computations is Mathematica (with the exception of collider
> simulations, which are run with dedicated programs like Pythia).
> However, representation theory is something Mathematica doesn't seem
> to know about, and I haven't been able to find a package for it.
>
> LiE does basically what I want, except that everything is wrapped up
> in a custom language that's not very expressive, hard to debug, and
> generally unpleasant to use. ?Python is much lovelier, especially
> since the object-oriented-ness is well-suited for writing
> mathematics. ?So I'm doing my best to provide a python interface as an
> alternative to LiE's built-in lexer, parser, and interpreter.
>

(1) The Sage project (http://sagemath.org) may be very interested in
getting your code that wraps LiE when you finish it.  Having a Sage
<--> Lie interface has been requested feature.

(2) Sage itself has substantial functionality for combinatorics of Lie
algebras.   See, e.g., the talk by Dan Bump that is linked to from
here: http://wiki.sagemath.org/days14


 -- William

From dagss at student.matnat.uio.no  Wed Jul  8 20:31:39 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 8 Jul 2009 20:31:39 +0200
Subject: [Cython] Cython and lapack-library
In-Reply-To: <f9968a7e0907080524j709334d9l44219de6675aa554@mail.gmail.com>
References: <f9968a7e0907080524j709334d9l44219de6675aa554@mail.gmail.com>
Message-ID: <28edbcef3ac9821d6ec4c02794d0bfde.squirrel@webmail.uio.no>

Thorsten Kranz wrote:
> Hi,
>
> I'm quite new to cython, so maybe my question is trivial. I try to
> wrap the lapack-routine dgesv using cython. I was glad when I found
>
> http://codespeak.net/pipermail/cython-dev/2009-May/005702.html
>
> covering exactly that topic. I tried to compile the lapack-part of it,
> but then it seemed like the lapack.pxd file was missing. So I went
> ahead creating this file, using the informations I could grep from the
> slides contained in this tutorial.
>
> But when I do so, it does indeed compile, but on execution of
>
> testlapack.test()
>
> an error occurs that lapack doesn't have the attribute CblasRowMajor.
>
> Can anybody please help me? Has anybody of you done this before or
> does anybody have the correct lapack.pxd file?

Now this is spooky ... two questions about exactly this within a few hours
:-)

Anyway, I have now updated the tutorial resources (as I should have done
long ago); see wiki.cython.org/talks leading you to

http://sage.math.washington.edu/home/dagss/cython-notur09/lapack/lapack.pxd

Dag Sverre


From dagss at student.matnat.uio.no  Wed Jul  8 20:46:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 8 Jul 2009 20:46:50 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
Message-ID: <fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Jul 7, 2009, at 11:24 AM, Lisandro Dalcin wrote:
>
>> On Tue, Jul 7, 2009 at 2:30 PM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> I disagree, I think second-guessing is wonderful. (That is, if you
>>> want
>>> me to understand this, you must explain more -- of course, it is
>>> not a
>>> necessity that I understand this.)
>>>
>>> If the only effect is to save a few lines of C and a few cycles of
>>> CPU
>>> time for gcc, then I don't think it is worth it to complicate the
>>> language.
>>>
>>
>> It is not about saving lines of CPU cycles... It's about providing a
>> way for users to say: Hey! This extern ctypedef is EXACTLY what I
>> declared Why is Cython trying to be smart and second-guess me, I'm
>> adult enough ...
>> Moreover, what I'm proposing is 100 % backwards, and the change to the
>> languaje is rather minimal...
>
> The main point of using external typedefs is that one often doesn't
> know the exact size of the type, e.g. it's determined at c-compile
> and/or configure time. (Otherwise, one could just use int, long, or
> whatever it actually *is* and the C compiler should never complain
> after unwinding all the definitions...) I am -1 on requiring a ? when
> the type is not explicitly known.
>
> If one needs the sized to match exactly, one can "assert sizeof
> (my_int_type) == sizeof(long)."
>
> The only difference I see with the ? notation is that it would be
> explicitly unranked (whereas the standard typedef receives its
> ranking from its definition). I guess in that case certain operations
> (like arithmetic with other types) would be disallowed? I just don't
> see this as a common enough use case to need new syntax (though I'd
> be curious to hear if it does fit the bill for what someone's trying
> to do).

(Lisandro has exited the thread it seems but I still want to finish this
because I've been wanting to do the same thing. This is another question
that's been on and off the entire spring and we should just finish it.)

I think there is a problem here in Cython that must be fixed; it is just
too complicated. The problem is that if you need to use a type of which C
definition vary with platform (which is very common, e.g. numpy.pxd) one
currently has to say

"Just use something, it doesn't matter in most situations; but remember to
stay away from the situations where it does matter; which ones that are we
are not even sure of ourselves."

This is hardly satisfying, and SOME kind of solution is needed here. If
only to prevent subtle bugs from mixing types from numpy.pxd with other
types in arithmetic!

I think the following is a solution. It might not be the only one, but it
is the only one presented so far:

1) Introduce ? notation, and make rank unknown => disable some arithmetic
2) If ? is not used, require and enforce exact match.

This has the advantage that it is a) extremely easy to explain, b) guards
against any kinds of errors (and if one needs type-combining arithmetic,
just cast).

Dag Sverre


From stefan_ml at behnel.de  Wed Jul  8 21:22:19 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 08 Jul 2009 21:22:19 +0200
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A4F2D34.20703@behnel.de>
References: <4A4F2D34.20703@behnel.de>
Message-ID: <4A54F1EB.6040808@behnel.de>


Stefan Behnel wrote:
> in the current implementation, __future__.division only applies when
> dividing non-constant Python values
> [...]
> My opinion on this is: if you request true division semantics by importing
> __future__.division, you should get true division in all places where you
> use the "/" operator.

I committed the change in

http://hg.cython.org/cython-unstable/rev/a3ad25325fe4

I may be a little more complicated than necessary, but it works in all
cases I could come up with (see the end of the changeset for tests).

Stefan


From dsimmons.com at gmail.com  Wed Jul  8 21:37:50 2009
From: dsimmons.com at gmail.com (David Simmons-Duffin)
Date: Wed, 8 Jul 2009 15:37:50 -0400
Subject: [Cython] "object" keyword in cython
In-Reply-To: <85e81ba30907080922x1c7824at53acef90d1f635bd@mail.gmail.com>
References: <AE0A6018-7CDC-4E1E-BCA8-119D22751507@gmail.com>
	<90A851D4-4B31-425A-96C0-E13B11F5FBE8@math.washington.edu>
	<685DF6AA-356A-45D7-B042-780BEEBFCB72@gmail.com>
	<85e81ba30907080922x1c7824at53acef90d1f635bd@mail.gmail.com>
Message-ID: <505929e0907081237h7a186ebdn1fd22b7e158e90f@mail.gmail.com>

I didn't realize that Sage could do Lie algebra manipulations.  What are the
pros and cons for writing an interface to LiE as opposed to simply extending
Sage's existing functionality?  Is LiE faster, or has it implemented some
tricky algorithms that people don't want to write in sage?

I'd be happy to donate my code to Sage, although I don't know if it will
have the right organization or be of the right quality standards.  I haven't
used Sage, but it looks like it has some kind of system for dealing with
general mathematical objects, for instance keeping track of whether
something is a group or ring element, or in a vector space over some field.
I made some decisions about what kind of classes to include in my interface
based on trying to achieve the simplest code for my particular applications,
and not with an eye towards conforming to some general math system.  For
instance, whereas LiE uses a polynomial to stand for a representaion, I have
a Rep object that wraps a polynomial and a group, where multiplication on
Rep's is defined as tensor product.  If I were writing a sage extension, I
imagine I would have to somehow also encode that Rep is a type of ring over
the integers, etc..

Would my code be useful even without conforming to sage's math conventions?
I'd be interested in learning how to write a sage extension at some point,
but I need my code for a physics project I'm actively working on now, and
I'm not sure I have very much time to spend on programming.

Thanks,
David


On Wed, Jul 8, 2009 at 12:22 PM, William Stein <wstein at gmail.com> wrote:

> On Wed, Jul 8, 2009 at 9:17 AM, David
> Simmons-Duffin<dsimmons.com at gmail.com> wrote:
> > On Jul 8, 2009, at 1:32 AM, Robert Bradshaw wrote:
> >
> >>> One possible solution is to just replace the word "object" in the C
> >>> source with something else throughout.  But I'd like to make as few
> >>> modifications of the C source as possible.  Is there an alternate
> >>> solution?
> >>
> >> Yes, you can use the magic c-namespace-renaming quotes. For your
> >> example.
> >>
> >>> I actually have another unrelated question: what is the correct
> >>> cython
> >>> syntax for the union statement at the top of this message?  Cython
> >>> doesn't seem to like me putting
> >>>
> >>> struct any:
> >>>      objtype type
> >>>      reftype nfref
> >>>
> >>> in the middle of a union statement.
> >>
> >> Cython doesn't support anonymous structs. Give that struct a name,
> >> and then you can declare your variable with that type. (The name you
> >> choose will never actually be used.)
> >
> > Thanks, your suggestions worked perfectly.
> >
> >> Just out of curiosity, what are you wrapping? It looks like a math
> >> library of sorts.
> >
> > I'm wrapping a program called LiE for doing computations in lie group
> > representation theory.
> >
> > http://www-math.univ-poitiers.fr/~maavl/LiE/<http://www-math.univ-poitiers.fr/%7Emaavl/LiE/>
> >
> > This sort of thing is very useful in high energy theoretical physics,
> > which is what I work on.  The typical choice among us theorists for
> > doing computations is Mathematica (with the exception of collider
> > simulations, which are run with dedicated programs like Pythia).
> > However, representation theory is something Mathematica doesn't seem
> > to know about, and I haven't been able to find a package for it.
> >
> > LiE does basically what I want, except that everything is wrapped up
> > in a custom language that's not very expressive, hard to debug, and
> > generally unpleasant to use.  Python is much lovelier, especially
> > since the object-oriented-ness is well-suited for writing
> > mathematics.  So I'm doing my best to provide a python interface as an
> > alternative to LiE's built-in lexer, parser, and interpreter.
> >
>
> (1) The Sage project (http://sagemath.org) may be very interested in
> getting your code that wraps LiE when you finish it.  Having a Sage
> <--> Lie interface has been requested feature.
>
> (2) Sage itself has substantial functionality for combinatorics of Lie
> algebras.   See, e.g., the talk by Dan Bump that is linked to from
> here: http://wiki.sagemath.org/days14
>
>
>  -- William
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090708/b0685ab2/attachment-0001.htm 

From wstein at gmail.com  Wed Jul  8 21:50:26 2009
From: wstein at gmail.com (William Stein)
Date: Wed, 8 Jul 2009 12:50:26 -0700
Subject: [Cython] "object" keyword in cython
In-Reply-To: <505929e0907081237h7a186ebdn1fd22b7e158e90f@mail.gmail.com>
References: <AE0A6018-7CDC-4E1E-BCA8-119D22751507@gmail.com> 
	<90A851D4-4B31-425A-96C0-E13B11F5FBE8@math.washington.edu> 
	<685DF6AA-356A-45D7-B042-780BEEBFCB72@gmail.com>
	<85e81ba30907080922x1c7824at53acef90d1f635bd@mail.gmail.com> 
	<505929e0907081237h7a186ebdn1fd22b7e158e90f@mail.gmail.com>
Message-ID: <85e81ba30907081250s31f61f2ej808d7dd182f4addd@mail.gmail.com>

On Wed, Jul 8, 2009 at 12:37 PM, David
Simmons-Duffin<dsimmons.com at gmail.com> wrote:
> I didn't realize that Sage could do Lie algebra manipulations.? What are the
> pros and cons for writing an interface to LiE as opposed to simply extending
> Sage's existing functionality?

It depends on what you need to do.

>  Is LiE faster, or has it implemented some
> tricky algorithms that people don't want to write in sage?

I don't know.

> I'd be happy to donate my code to Sage, although I don't know if it will
> have the right organization or be of the right quality standards.? I haven't
> used Sage, but it looks like it has some kind of system for dealing with
> general mathematical objects, for instance keeping track of whether
> something is a group or ring element, or in a vector space over some field.

True.

> I made some decisions about what kind of classes to include in my interface
> based on trying to achieve the simplest code for my particular applications,
> and not with an eye towards conforming to some general math system.? For
> instance, whereas LiE uses a polynomial to stand for a representaion, I have
> a Rep object that wraps a polynomial and a group, where multiplication on
> Rep's is defined as tensor product.? If I were writing a sage extension, I
> imagine I would have to somehow also encode that Rep is a type of ring over
> the integers, etc..
>
> Would my code be useful even without conforming to sage's math conventions?
> I'd be interested in learning how to write a sage extension at some point,
> but I need my code for a physics project I'm actively working on now, and
> I'm not sure I have very much time to spend on programming.

I don't really.  It sounds to me like you should focus on your current
project first, and you might also want to investigate the capabilities
of Sage (and Sage + the sage-combinat extensions)....

William

>
> Thanks,
> David
>
>
> On Wed, Jul 8, 2009 at 12:22 PM, William Stein <wstein at gmail.com> wrote:
>>
>> On Wed, Jul 8, 2009 at 9:17 AM, David
>> Simmons-Duffin<dsimmons.com at gmail.com> wrote:
>> > On Jul 8, 2009, at 1:32 AM, Robert Bradshaw wrote:
>> >
>> >>> One possible solution is to just replace the word "object" in the C
>> >>> source with something else throughout. ?But I'd like to make as few
>> >>> modifications of the C source as possible. ?Is there an alternate
>> >>> solution?
>> >>
>> >> Yes, you can use the magic c-namespace-renaming quotes. For your
>> >> example.
>> >>
>> >>> I actually have another unrelated question: what is the correct
>> >>> cython
>> >>> syntax for the union statement at the top of this message? ?Cython
>> >>> doesn't seem to like me putting
>> >>>
>> >>> struct any:
>> >>> ? ? ?objtype type
>> >>> ? ? ?reftype nfref
>> >>>
>> >>> in the middle of a union statement.
>> >>
>> >> Cython doesn't support anonymous structs. Give that struct a name,
>> >> and then you can declare your variable with that type. (The name you
>> >> choose will never actually be used.)
>> >
>> > Thanks, your suggestions worked perfectly.
>> >
>> >> Just out of curiosity, what are you wrapping? It looks like a math
>> >> library of sorts.
>> >
>> > I'm wrapping a program called LiE for doing computations in lie group
>> > representation theory.
>> >
>> > http://www-math.univ-poitiers.fr/~maavl/LiE/
>> >
>> > This sort of thing is very useful in high energy theoretical physics,
>> > which is what I work on. ?The typical choice among us theorists for
>> > doing computations is Mathematica (with the exception of collider
>> > simulations, which are run with dedicated programs like Pythia).
>> > However, representation theory is something Mathematica doesn't seem
>> > to know about, and I haven't been able to find a package for it.
>> >
>> > LiE does basically what I want, except that everything is wrapped up
>> > in a custom language that's not very expressive, hard to debug, and
>> > generally unpleasant to use. ?Python is much lovelier, especially
>> > since the object-oriented-ness is well-suited for writing
>> > mathematics. ?So I'm doing my best to provide a python interface as an
>> > alternative to LiE's built-in lexer, parser, and interpreter.
>> >
>>
>> (1) The Sage project (http://sagemath.org) may be very interested in
>> getting your code that wraps LiE when you finish it. ?Having a Sage
>> <--> Lie interface has been requested feature.
>>
>> (2) Sage itself has substantial functionality for combinatorics of Lie
>> algebras. ? See, e.g., the talk by Dan Bump that is linked to from
>> here: http://wiki.sagemath.org/days14
>>
>>
>> ?-- William
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From dalcinl at gmail.com  Wed Jul  8 22:21:35 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 8 Jul 2009 17:21:35 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
Message-ID: <e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>

On Wed, Jul 8, 2009 at 3:46 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Robert Bradshaw wrote:
>> On Jul 7, 2009, at 11:24 AM, Lisandro Dalcin wrote:
>>
>>> On Tue, Jul 7, 2009 at 2:30 PM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> I disagree, I think second-guessing is wonderful. (That is, if you
>>>> want
>>>> me to understand this, you must explain more -- of course, it is
>>>> not a
>>>> necessity that I understand this.)
>>>>
>>>> If the only effect is to save a few lines of C and a few cycles of
>>>> CPU
>>>> time for gcc, then I don't think it is worth it to complicate the
>>>> language.
>>>>
>>>
>>> It is not about saving lines of CPU cycles... It's about providing a
>>> way for users to say: Hey! This extern ctypedef is EXACTLY what I
>>> declared Why is Cython trying to be smart and second-guess me, I'm
>>> adult enough ...
>>> Moreover, what I'm proposing is 100 % backwards, and the change to the
>>> languaje is rather minimal...
>>
>> The main point of using external typedefs is that one often doesn't
>> know the exact size of the type, e.g. it's determined at c-compile
>> and/or configure time. (Otherwise, one could just use int, long, or
>> whatever it actually *is* and the C compiler should never complain
>> after unwinding all the definitions...) I am -1 on requiring a ? when
>> the type is not explicitly known.
>>
>> If one needs the sized to match exactly, one can "assert sizeof
>> (my_int_type) == sizeof(long)."
>>
>> The only difference I see with the ? notation is that it would be
>> explicitly unranked (whereas the standard typedef receives its
>> ranking from its definition). I guess in that case certain operations
>> (like arithmetic with other types) would be disallowed? I just don't
>> see this as a common enough use case to need new syntax (though I'd
>> be curious to hear if it does fit the bill for what someone's trying
>> to do).
>
> (Lisandro has exited the thread it seems but I still want to finish this
> because I've been wanting to do the same thing. This is another question
> that's been on and off the entire spring and we should just finish it.)
>

I'm back... Have to spend some time helping a coworker...

> I think there is a problem here in Cython that must be fixed; it is just
> too complicated. The problem is that if you need to use a type of which C
> definition vary with platform (which is very common, e.g. numpy.pxd) one
> currently has to say
>
> "Just use something, it doesn't matter in most situations; but remember to
> stay away from the situations where it does matter; which ones that are we
> are not even sure of ourselves."
>

I agree 100 % with these comments. From the very beginning I moved
from SWIG/hand-C to Cython, I knew my code was broken because it was
not platform-agnostic. Then I got my hands dirty to try to fix this...
I've started handling the size_t/Py_ssize_t, treating then like
builtin integral types. Next I added code to catch integer overflow
conversions. Next Dag fixed member descriptor. Now I want to finish
this for external ctypedef, in a backward way, still letting adult
people do what they want, but still save them from their own bugs a
possible wrong assumptions, and even from current Cython wrong
assumptions...

> This is hardly satisfying, and SOME kind of solution is needed here. If
> only to prevent subtle bugs from mixing types from numpy.pxd with other
> types in arithmetic!
>

Indeed.

> I think the following is a solution. It might not be the only one, but it
> is the only one presented so far:
>
> 1) Introduce ? notation, and make rank unknown => disable some arithmetic
> 2) If ? is not used, require and enforce exact match.
>
> This has the advantage that it is a) extremely easy to explain, b) guards
> against any kinds of errors (and if one needs type-combining arithmetic,
> just cast).
>

Moreover, if we take (2) we just make Cython a not so useful tool for
wrapping external libs. It would be impossible to write portable
Cython code, end users will have to fix type definitions by hand (of
developers start wasting time with distutils based configure-like
support). Any external lib (MPI in my case) defining a integral type
for address-sized integers will be a nightmare to handle. Also I will
have problems in petsc4py. You can build core PETSc in different ways,
with 32/64 bit integers (PetscInt type), with single/double presicion
reals (PetscReal type)... So the actual sizeof of PetscInt/PetscReal
depends on how you built core PETSc...

In short, if we do (2), we will break on of the (at least in my POV)
most important use cases of Cython, i.e. wrapping C/C++ stuff.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Jul  8 23:07:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 8 Jul 2009 23:07:09 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
Message-ID: <a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>

Lisandro wrote:
> On Wed, Jul 8, 2009 at 3:46 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Robert Bradshaw wrote:
>>> On Jul 7, 2009, at 11:24 AM, Lisandro Dalcin wrote:
>>>
>>>> On Tue, Jul 7, 2009 at 2:30 PM, Dag Sverre
>>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>>> I disagree, I think second-guessing is wonderful. (That is, if you
>>>>> want
>>>>> me to understand this, you must explain more -- of course, it is
>>>>> not a
>>>>> necessity that I understand this.)
>>>>>
>>>>> If the only effect is to save a few lines of C and a few cycles of
>>>>> CPU
>>>>> time for gcc, then I don't think it is worth it to complicate the
>>>>> language.
>>>>>
>>>>
>>>> It is not about saving lines of CPU cycles... It's about providing a
>>>> way for users to say: Hey! This extern ctypedef is EXACTLY what I
>>>> declared Why is Cython trying to be smart and second-guess me, I'm
>>>> adult enough ...
>>>> Moreover, what I'm proposing is 100 % backwards, and the change to the
>>>> languaje is rather minimal...
>>>
>>> The main point of using external typedefs is that one often doesn't
>>> know the exact size of the type, e.g. it's determined at c-compile
>>> and/or configure time. (Otherwise, one could just use int, long, or
>>> whatever it actually *is* and the C compiler should never complain
>>> after unwinding all the definitions...) I am -1 on requiring a ? when
>>> the type is not explicitly known.
>>>
>>> If one needs the sized to match exactly, one can "assert sizeof
>>> (my_int_type) == sizeof(long)."
>>>
>>> The only difference I see with the ? notation is that it would be
>>> explicitly unranked (whereas the standard typedef receives its
>>> ranking from its definition). I guess in that case certain operations
>>> (like arithmetic with other types) would be disallowed? I just don't
>>> see this as a common enough use case to need new syntax (though I'd
>>> be curious to hear if it does fit the bill for what someone's trying
>>> to do).
>>
>> (Lisandro has exited the thread it seems but I still want to finish this
>> because I've been wanting to do the same thing. This is another question
>> that's been on and off the entire spring and we should just finish it.)
>>
>
> I'm back... Have to spend some time helping a coworker...
>
>> I think there is a problem here in Cython that must be fixed; it is just
>> too complicated. The problem is that if you need to use a type of which
>> C
>> definition vary with platform (which is very common, e.g. numpy.pxd) one
>> currently has to say
>>
>> "Just use something, it doesn't matter in most situations; but remember
>> to
>> stay away from the situations where it does matter; which ones that are
>> we
>> are not even sure of ourselves."
>>
>
> I agree 100 % with these comments. From the very beginning I moved
> from SWIG/hand-C to Cython, I knew my code was broken because it was
> not platform-agnostic. Then I got my hands dirty to try to fix this...
> I've started handling the size_t/Py_ssize_t, treating then like
> builtin integral types. Next I added code to catch integer overflow
> conversions. Next Dag fixed member descriptor. Now I want to finish
> this for external ctypedef, in a backward way, still letting adult
> people do what they want, but still save them from their own bugs a
> possible wrong assumptions, and even from current Cython wrong
> assumptions...
>
>> This is hardly satisfying, and SOME kind of solution is needed here. If
>> only to prevent subtle bugs from mixing types from numpy.pxd with other
>> types in arithmetic!
>>
>
> Indeed.
>
>> I think the following is a solution. It might not be the only one, but
>> it
>> is the only one presented so far:
>>
>> 1) Introduce ? notation, and make rank unknown => disable some
>> arithmetic
>> 2) If ? is not used, require and enforce exact match.
>>
>> This has the advantage that it is a) extremely easy to explain, b)
>> guards
>> against any kinds of errors (and if one needs type-combining arithmetic,
>> just cast).
>>
>
> Moreover, if we take (2) we just make Cython a not so useful tool for
> wrapping external libs. It would be impossible to write portable
> Cython code, end users will have to fix type definitions by hand (of
> developers start wasting time with distutils based configure-like
> support). Any external lib (MPI in my case) defining a integral type
> for address-sized integers will be a nightmare to handle. Also I will
> have problems in petsc4py. You can build core PETSc in different ways,
> with 32/64 bit integers (PetscInt type), with single/double presicion
> reals (PetscReal type)... So the actual sizeof of PetscInt/PetscReal
> depends on how you built core PETSc...
>
> In short, if we do (2), we will break on of the (at least in my POV)
> most important use cases of Cython, i.e. wrapping C/C++ stuff.

I think you mistunderstood me... I meant (1) and (2) in combination.
Trying again:

1) We introduce "?"
2) If you do not know the exact type, you HAVE to use ?. Not using ?
implies you know the exact type.

Dag Sverre


From dalcinl at gmail.com  Thu Jul  9 02:07:23 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 8 Jul 2009 21:07:23 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
Message-ID: <e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>

On Wed, Jul 8, 2009 at 6:07 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> I think you mistunderstood me... I meant (1) and (2) in combination.

Indeed... My fault this time...

> Trying again:
>
> 1) We introduce "?"
> 2) If you do not know the exact type, you HAVE to use ?. Not using ?
> implies you know the exact type.
>

Of course...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Jul  9 03:00:24 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 09 Jul 2009 13:00:24 +1200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
Message-ID: <4A554128.5020604@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> This is hardly satisfying, and SOME kind of solution is needed here. If
> only to prevent subtle bugs from mixing types from numpy.pxd with other
> types in arithmetic!

If you do this, you might like to consider a syntax
I suggested earlier:

   ctypedef some int foo
   ctypedef some float bar

etc.

I think this is better than something like int? or
?int because it makes it clear that the uncertainty
is about the size of the type, not its basic kind.

Some thoughts on implementation:

* Treat it as having a rank higher than any known
rank for that kind. That way arithmetic etc. between
known and unknown sizes will get promoted to an
unknown size.

* Only allow it to be introduced via a typedef, so
that generated code always has a named type for the
C compiler to go on.

-- 
Greg

From dalcinl at gmail.com  Thu Jul  9 07:07:52 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 9 Jul 2009 02:07:52 -0300
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A554128.5020604@canterbury.ac.nz>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<4A554128.5020604@canterbury.ac.nz>
Message-ID: <e7ba66e40907082207o7d100267u69e53068fa390cb8@mail.gmail.com>

On Wed, Jul 8, 2009 at 10:00 PM, Greg Ewing<greg.ewing at canterbury.ac.nz> wrote:
> Dag Sverre Seljebotn wrote:
>
>> This is hardly satisfying, and SOME kind of solution is needed here. If
>> only to prevent subtle bugs from mixing types from numpy.pxd with other
>> types in arithmetic!
>
> If you do this, you might like to consider a syntax
> I suggested earlier:
>
> ? ctypedef some int foo
> ? ctypedef some float bar
>
> etc.
>
> I think this is better than something like int? or
> ?int because it makes it clear that the uncertainty
> is about the size of the type, not its basic kind.
>

I agree...

> Some thoughts on implementation:
>
> * Treat it as having a rank higher than any known
> rank for that kind. That way arithmetic etc. between
> known and unknown sizes will get promoted to an
> unknown size.
>

I was also thinking about this way... I cannot imagine right now a
case were this breaks...

> * Only allow it to be introduced via a typedef, so
> that generated code always has a named type for the
> C compiler to go on.
>

Likely you mean EXTERNAL ctypedefs, right? In such case, that's the intention...

Greg, many thanks for your input...

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Jul  9 07:44:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 9 Jul 2009 07:44:02 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A554128.5020604@canterbury.ac.nz>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<4A554128.5020604@canterbury.ac.nz>
Message-ID: <bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>

Greg Ewing wrote:
> Dag Sverre Seljebotn wrote:
>
>> This is hardly satisfying, and SOME kind of solution is needed here. If
>> only to prevent subtle bugs from mixing types from numpy.pxd with other
>> types in arithmetic!
>
> If you do this, you might like to consider a syntax
> I suggested earlier:
>
>    ctypedef some int foo
>    ctypedef some float bar
>
> etc.

Good idea!

Here's another worse-looking but more featureful alternative:

cdef extern from *:
    ctypedef rank("mylib", 3) smallint
    ctypedef rank("mylib", 4) largeint

This way one can create groups of ranked types. Arithmetic is allowed
within a group (adding smallint and largeint will result in largeint), but
not from one group to another (i.e. adding an "largeint" and a "long"
requires a cast or gives syntax error).

Internally, short, int, long etc. would be defined rank("c", x), and one
could then also support C99 int32_t, int64_t etc. as rank("c99", x).

Of course, there's no harm in making "some" an alias for a new rank-group
with only one element.

> Some thoughts on implementation:
>
> * Treat it as having a rank higher than any known
> rank for that kind. That way arithmetic etc. between
> known and unknown sizes will get promoted to an
> unknown size.

This would be a bad idea for some of the NumPy types which are unknown yet
very short. I'd prefer to require an explicit cast from one "side" to the
other.

> * Only allow it to be introduced via a typedef, so
> that generated code always has a named type for the
> C compiler to go on.

Definitely.

Dag Sverre


From greg.ewing at canterbury.ac.nz  Thu Jul  9 08:15:55 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 09 Jul 2009 18:15:55 +1200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907082207o7d100267u69e53068fa390cb8@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<4A554128.5020604@canterbury.ac.nz>
	<e7ba66e40907082207o7d100267u69e53068fa390cb8@mail.gmail.com>
Message-ID: <4A558B1B.2000802@canterbury.ac.nz>

Lisandro Dalcin wrote:
> On Wed, Jul 8, 2009 at 10:00 PM, Greg Ewing<greg.ewing at canterbury.ac.nz> wrote:

>>* Only allow it to be introduced via a typedef
> 
> Likely you mean EXTERNAL ctypedefs, right?

Yes, it would have to be restricted to external
typedefs, I think.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Jul  9 08:19:08 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 09 Jul 2009 18:19:08 +1200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<4A554128.5020604@canterbury.ac.nz>
	<bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>
Message-ID: <4A558BDC.5000305@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> cdef extern from *:
>     ctypedef rank("mylib", 3) smallint
>     ctypedef rank("mylib", 4) largeint

Do you have any use cases where that would actually
be helpful? I.e. where you don't know the exact types
but you are sure of their ranking, and knowing the
ranking is useful?

-- 
Greg

From stefan_ml at behnel.de  Thu Jul  9 08:44:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 09 Jul 2009 08:44:46 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>	<4A4EAF85.3070003@canterbury.ac.nz>	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>	<4A5351E0.7020009@student.matnat.uio.no>	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>	<4A53862A.10206@student.matnat.uio.no>	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>	<4A554128.5020604@canterbury.ac.nz>
	<bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>
Message-ID: <4A5591DE.7000109@behnel.de>


Dag Sverre Seljebotn wrote:
> cdef extern from *:
>     ctypedef rank("mylib", 3) smallint
>     ctypedef rank("mylib", 4) largeint

... looks like implementation detail leakage to me ...

Stefan

From robertwb at math.washington.edu  Thu Jul  9 09:38:05 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Jul 2009 00:38:05 -0700
Subject: [Cython] __future__.division on constants and C integers
In-Reply-To: <4A54F1EB.6040808@behnel.de>
References: <4A4F2D34.20703@behnel.de> <4A54F1EB.6040808@behnel.de>
Message-ID: <A1881AFB-B06C-4645-B1ED-A1F20F51ED35@math.washington.edu>

On Jul 8, 2009, at 12:22 PM, Stefan Behnel wrote:

> Stefan Behnel wrote:
>> in the current implementation, __future__.division only applies when
>> dividing non-constant Python values
>> [...]
>> My opinion on this is: if you request true division semantics by  
>> importing
>> __future__.division, you should get true division in all places  
>> where you
>> use the "/" operator.
>
> I committed the change in
>
> http://hg.cython.org/cython-unstable/rev/a3ad25325fe4
>
> I may be a little more complicated than necessary, but it works in all
> cases I could come up with (see the end of the changeset for tests).

Looks good to me.

- Robert



From dagss at student.matnat.uio.no  Thu Jul  9 09:51:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 9 Jul 2009 09:51:14 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A5591DE.7000109@behnel.de>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<4A554128.5020604@canterbury.ac.nz>
	<bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>
	<4A5591DE.7000109@behnel.de>
Message-ID: <bcb9b80bc91bfce3c414ab7e5058ea16.squirrel@webmail.uio.no>

Greg Ewing wrote:
> Dag Sverre Seljebotn wrote:
>
>> cdef extern from *:
>>     ctypedef rank("mylib", 3) smallint
>>     ctypedef rank("mylib", 4) largeint
>
> Do you have any use cases where that would actually
> be helpful? I.e. where you don't know the exact types
> but you are sure of their ranking, and knowing the
> ranking is useful?

I mentioned the C99 types which is such a case; they add at least three
"rank-groups": int32_t and so on, int_least32_t and so on, and
int_fast32_t and so on.

Something very similar is going on in numpy.pxd (shipped with Cython),
since the NumPy C headers go to great pains to define these in a
cross-platform way (i.e. numpy.int32_t and so on is defined). In addition,
numpy.pxd defines float32_t, float64_t, which AFAIK is not part of C99.

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> cdef extern from *:
>>     ctypedef rank("mylib", 3) smallint
>>     ctypedef rank("mylib", 4) largeint
>
> ... looks like implementation detail leakage to me ...

Hmm; all that is needed is a syntax though. Other alternatives:

cdef extern from *:
    ctypedefgroup:
        int smallint
        int largerint

or:

cdef extern from *:
    ctypedef some int smallint
    ctypedef some int largerint
    ctyperelationship smallint < largerint

or similar ideas, I believe the point is taken, although I don't like
either of these concrete proposals.

Dag Sverre


From robertwb at math.washington.edu  Thu Jul  9 12:48:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Jul 2009 03:48:55 -0700
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <bcb9b80bc91bfce3c414ab7e5058ea16.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<4A554128.5020604@canterbury.ac.nz>
	<bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>
	<4A5591DE.7000109@behnel.de>
	<bcb9b80bc91bfce3c414ab7e5058ea16.squirrel@webmail.uio.no>
Message-ID: <495AE39A-E286-496D-BFD8-56CEBB50412E@math.washington.edu>

On Jul 9, 2009, at 12:51 AM, Dag Sverre Seljebotn wrote:

> Greg Ewing wrote:
>> Dag Sverre Seljebotn wrote:
>>
>>> cdef extern from *:
>>>     ctypedef rank("mylib", 3) smallint
>>>     ctypedef rank("mylib", 4) largeint
>>
>> Do you have any use cases where that would actually
>> be helpful? I.e. where you don't know the exact types
>> but you are sure of their ranking, and knowing the
>> ranking is useful?
>
> I mentioned the C99 types which is such a case; they add at least  
> three
> "rank-groups": int32_t and so on, int_least32_t and so on, and
> int_fast32_t and so on.
>
> Something very similar is going on in numpy.pxd (shipped with Cython),
> since the NumPy C headers go to great pains to define these in a
> cross-platform way (i.e. numpy.int32_t and so on is defined). In  
> addition,
> numpy.pxd defines float32_t, float64_t, which AFAIK is not part of  
> C99.
>
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> cdef extern from *:
>>>     ctypedef rank("mylib", 3) smallint
>>>     ctypedef rank("mylib", 4) largeint
>>
>> ... looks like implementation detail leakage to me ...
>
> Hmm; all that is needed is a syntax though. Other alternatives:
>
> cdef extern from *:
>     ctypedefgroup:
>         int smallint
>         int largerint
>
> or:
>
> cdef extern from *:
>     ctypedef some int smallint
>     ctypedef some int largerint
>     ctyperelationship smallint < largerint
>
> or similar ideas, I believe the point is taken, although I don't like
> either of these concrete proposals.

The last is the best I've seen so far, but I'm still not a fan. Also,  
rank will have to be a poset, not just an integer (which isn't too  
bad). Would we want to special case literals, so x+2 still works even  
if x's relationship to int is unknown?

- Robert



From robertwb at math.washington.edu  Thu Jul  9 13:02:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Jul 2009 04:02:03 -0700
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
Message-ID: <8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>

On Jul 8, 2009, at 5:07 PM, Lisandro Dalcin wrote:

> On Wed, Jul 8, 2009 at 6:07 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> I think you mistunderstood me... I meant (1) and (2) in combination.
>
> Indeed... My fault this time...
>
>> Trying again:
>>
>> 1) We introduce "?"
>> 2) If you do not know the exact type, you HAVE to use ?. Not using ?
>> implies you know the exact type.


I think this is a bad idea. If you have an extern typedef int, how  
often do you really know what it is? On all platforms? With all c  
configuration options and into the future? Nearly every typedef int  
would have to be unranked, which would be a pain to code with.

I'm not against the unranked/partially ordered types--it's a good  
one, but think it will be burdensome (as well as hugely backwards  
incompatible) to make it the default. (We should document the issues  
that may arise in the users manual though.)

- Robert


From dagss at student.matnat.uio.no  Thu Jul  9 13:32:32 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 9 Jul 2009 13:32:32 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
	<8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
Message-ID: <d7a91c2e4a2b688c638411fae250180e.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Jul 8, 2009, at 5:07 PM, Lisandro Dalcin wrote:
>
>> On Wed, Jul 8, 2009 at 6:07 PM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> I think you mistunderstood me... I meant (1) and (2) in combination.
>>
>> Indeed... My fault this time...
>>
>>> Trying again:
>>>
>>> 1) We introduce "?"
>>> 2) If you do not know the exact type, you HAVE to use ?. Not using ?
>>> implies you know the exact type.
>
>
> I think this is a bad idea. If you have an extern typedef int, how
> often do you really know what it is? On all platforms? With all c
> configuration options and into the future? Nearly every typedef int
> would have to be unranked, which would be a pain to code with.

I just don't see what alternative you are proposing though? Today's
behaviour is not even defined except through an ever-changing
implementation, and leads to bugs; is that what you are proposing to keep?

Just write down the exact behaviour that you want and I'll be able to
follow you here.

Dag Sverre


From greg.ewing at canterbury.ac.nz  Fri Jul 10 02:01:57 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 10 Jul 2009 12:01:57 +1200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <bcb9b80bc91bfce3c414ab7e5058ea16.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<4A4EAF85.3070003@canterbury.ac.nz>
	<9EAC899E-E390-460A-AC3B-BCFDF3ED004D@math.washington.edu>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<4A554128.5020604@canterbury.ac.nz>
	<bd67115b1ffd2ccff87886521663bb76.squirrel@webmail.uio.no>
	<4A5591DE.7000109@behnel.de>
	<bcb9b80bc91bfce3c414ab7e5058ea16.squirrel@webmail.uio.no>
Message-ID: <4A5684F5.8040700@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> I mentioned the C99 types which is such a case; they add at least three
> "rank-groups": int32_t and so on, int_least32_t and so on, and
> int_fast32_t and so on.

But does it help Cython to know the *ranking* of these
types?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Jul 10 02:17:11 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 10 Jul 2009 12:17:11 +1200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
	<8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
Message-ID: <4A568887.5060904@canterbury.ac.nz>

Robert Bradshaw wrote:

> I'm not against the unranked/partially ordered types--it's a good  
> one, but think it will be burdensome (as well as hugely backwards  
> incompatible) to make it the default.

I'm not sure what you mean by "make it the default".
The proposal *doesn't* make unrankedness the default --
you have to ask for it.

However, it sounds like you *do* want it to be the
default, i.e. no rank is assumed unless you explicitly
say the type is exact.

That would be strange, IMO. Built-in type names such as
"int" have a precise meaning to the C compiler, and
it's reasonable to assume that if you say "int" in
Cython code, it will mean the same thing.

I suppose external typedefs could be treated differently
in this respect, but that would be somewhat odd as well.

-- 
Greg

From robertwb at math.washington.edu  Fri Jul 10 08:11:35 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Jul 2009 23:11:35 -0700
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <4A568887.5060904@canterbury.ac.nz>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
	<8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
	<4A568887.5060904@canterbury.ac.nz>
Message-ID: <E22A3F03-BD2E-4AC0-A563-7758C6CE10DE@math.washington.edu>

On Jul 9, 2009, at 5:17 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> I'm not against the unranked/partially ordered types--it's a good
>> one, but think it will be burdensome (as well as hugely backwards
>> incompatible) to make it the default.
>
> I'm not sure what you mean by "make it the default".
> The proposal *doesn't* make unrankedness the default --
> you have to ask for it.
>
> However, it sounds like you *do* want it to be the
> default, i.e. no rank is assumed unless you explicitly
> say the type is exact.

Sorry, I worded things poorly... What I didn't want to be default is  
having extern ctypedef sizes checked at runtime, raising an error if  
they didn't match.

> That would be strange, IMO. Built-in type names such as
> "int" have a precise meaning to the C compiler, and
> it's reasonable to assume that if you say "int" in
> Cython code, it will mean the same thing.
>
> I suppose external typedefs could be treated differently
> in this respect, but that would be somewhat odd as well.

Yes, I'm only talking about extern types, because those may not even  
be decided until C compile time.

- Robert



From robertwb at math.washington.edu  Fri Jul 10 08:23:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 9 Jul 2009 23:23:57 -0700
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <d7a91c2e4a2b688c638411fae250180e.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
	<8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
	<d7a91c2e4a2b688c638411fae250180e.squirrel@webmail.uio.no>
Message-ID: <9D942F94-B491-4A41-9BF4-BFEA5F4AEC06@math.washington.edu>

On Jul 9, 2009, at 4:32 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jul 8, 2009, at 5:07 PM, Lisandro Dalcin wrote:
>>
>>> On Wed, Jul 8, 2009 at 6:07 PM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> I think you mistunderstood me... I meant (1) and (2) in  
>>>> combination.
>>>
>>> Indeed... My fault this time...
>>>
>>>> Trying again:
>>>>
>>>> 1) We introduce "?"
>>>> 2) If you do not know the exact type, you HAVE to use ?. Not  
>>>> using ?
>>>> implies you know the exact type.
>>
>>
>> I think this is a bad idea. If you have an extern typedef int, how
>> often do you really know what it is? On all platforms? With all c
>> configuration options and into the future? Nearly every typedef int
>> would have to be unranked, which would be a pain to code with.
>
> I just don't see what alternative you are proposing though? Today's
> behaviour is not even defined except through an ever-changing
> implementation, and leads to bugs; is that what you are proposing  
> to keep?
>
> Just write down the exact behaviour that you want and I'll be able to
> follow you here.

"ctypdef int foo" means that foo will be a signed integer type, and  
most of the code involved will be size-agnostic, but when mixing with  
other types it will have the ranking of an int. This is what it means  
now, right? Perhaps we could say that rankings being equal, we defer  
to the typedef type.

I just looked at the Sage codebase and there aren't nearly as many  
extern ctypedef ints as I thought. Nearly all of them are not a  
specific size, yet we still often do arithmetic with them (e.g. with  
literal ints). Short of a configure script, there is often no way to  
know at Cython compile time what the actual sizes are.

If I understand your proposal, you want to make an "ctypedef int foo"  
mean

ctypedef int foo # as it is now
assert sizeof(int) == sizeof(foo)

and

ctypedef some int foo # or whatever notation we have

mean we disallow all rank-based decisions (unless we provide explicit  
rank information).

- Robert


From robertwb at math.washington.edu  Fri Jul 10 09:46:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Jul 2009 00:46:51 -0700
Subject: [Cython] problem with pure python mode
In-Reply-To: <85b5c3130907071233q6f797b78w12886f264580d0ce@mail.gmail.com>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
	<12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
	<85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>
	<7B1321FB-58A6-4CA3-927A-54FA609A0336@math.washington.edu>
	<85b5c3130906181829u330f95edw6339c719cd724671@mail.gmail.com>
	<85b5c3130907071233q6f797b78w12886f264580d0ce@mail.gmail.com>
Message-ID: <D1FF3F5D-28B9-4174-A019-30A1E680138C@math.washington.edu>

On Jul 7, 2009, at 12:33 PM, Ondrej Certik wrote:

> On Thu, Jun 18, 2009 at 7:29 PM, Ondrej Certik<ondrej at certik.cz>  
> wrote:
>> On Tue, Jun 16, 2009 at 11:22 PM, Robert
>> Bradshaw<robertwb at math.washington.edu> wrote:
>>> On Jun 10, 2009, at 5:45 PM, Ondrej Certik wrote:
>>>
>>>> On Wed, Apr 29, 2009 at 4:00 PM, Robert
>>>> Bradshaw<robertwb at math.washington.edu> wrote:
>>>>> On Apr 29, 2009, at 2:49 PM, Ondrej Certik wrote:
>>>>>
>>>>>>>>
>>>>>>>> Not sure--have you tried without the locals decorator in the
>>>>>>>> Python
>>>>>>>> source? I hope this isn't a regression.
>>>>>>>
>>>>>>> Yes, as you can see, the locals is commented out. I'll try to
>>>>>>> bisect
>>>>>>> it to see which commit broke it.
>>>>>>
>>>>>> Ok, so it never worked. I am a bit confused, because I thought  
>>>>>> the
>>>>>> pure python mode worked fine for me before.
>>>>>>
>>>>>> In the current cython --- is there any way to keep a pure python
>>>>>> code,
>>>>>> and create an accompanying pxd file that would annotate the cdef
>>>>>> functions and classes?
>>>>>
>>>>> This should work, that's how I implemented this a long time  
>>>>> ago. If
>>>>> it's not working now it's because something broke (or, I'm not
>>>>> promising that it's completely bug-free, but it should be good--we
>>>>> use it to bootstrap Cython itself).
>>>>
>>>> So I made it to work by observing how things are done in Cython
>>>> itself.
>>>>
>>>> If you do:
>>>>
>>>> git clone git://github.com/certik/cython-test.git
>>>> cd cython-test
>>>> cython fact.pyx
>>>>
>>>> then it produces nicely optimized fact.c file. However, if you do:
>>>>
>>>> cython fact.py
>>>>
>>>> it produces a fact.c, that compiles, but it is not optimized ---  
>>>> the
>>>> "i" and "r" variables are python objects, not ints, even though my
>>>> fact.pxd file contains:
>>>>
>>>> import cython
>>>>
>>>> @cython.locals(i=cython.int, r=cython.int)
>>>> cdef int factorial(int n)
>>>>
>>>>
>>>> Do you see anything wrong with my files? Is this supposed to work?
>>>>
>>>> I think it is not implemented yet, because when I looked at the  
>>>> cython
>>>> sources, it doesn't work there either. This is the relevant part of
>>>> Scanning.pxd:
>>>>
>>>>     @cython.locals(current_level=cython.long,  
>>>> new_level=cython.long)
>>>>     cpdef indentation_action(self, text)
>>>>
>>>> and here is the declaration in the generated Scanning.c:
>>>>
>>>>   PyObject *__pyx_v_new_level;
>>>>
>>>>
>>>> And consider for example the following chunk:
>>>>
>>>>   /* "/home/ondrej/repos/cython-devel/Cython/Compiler/ 
>>>> Scanning.py":398
>>>>  *         new_level = len(text)
>>>>  *         #print "Changing indent level from", current_level,  
>>>> "to",
>>>> new_level ###
>>>>  *         if new_level == current_level:             #  
>>>> <<<<<<<<<<<<<<
>>>>  *             return
>>>>  *         elif new_level > current_level:
>>>>  */
>>>>   __pyx_t_4 = PyObject_RichCompare(__pyx_v_new_level,
>>>> __pyx_v_current_level, Py_EQ); if (unlikely(!__pyx_t_4))
>>>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno =
>>>> __LINE__; goto __pyx_L1_error;}
>>>>   __Pyx_GOTREF(__pyx_t_4);
>>>>   __pyx_t_3 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely 
>>>> (__pyx_t_3
>>>> < 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398;  
>>>> __pyx_clineno
>>>> = __LINE__; goto __pyx_L1_error;}
>>>>   __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
>>>>   if (__pyx_t_3) {
>>>>
>>>>
>>>> both the new_level and current_level are declared as long, but  
>>>> above
>>>> you can see that Cython is calling lots of Python C/API methods...
>>>
>>> Looks like a bug to me.
>>>
>>> http://trac.cython.org/cython_trac/ticket/330
>>
>> Do you have some idea where the problem is? You wrote "Parsing" in  
>> the
>> ticket. This pure Python mode is very important for me, so I tried to
>> get it fixed, but I need to invest more time to learn Cython  
>> internals
>> to produce something useful, hopefully over the weekend I'll find  
>> time
>> to dig into it.
>
> Let me know if there is any progress on this one. We are slowly
> approaching the state with sympy, when I will need this.

I was about to say "no haven't had a chance," but I figured I'd give  
it a quick glance right now, and fortunately it was a quick fix. I've  
pushed to cython-devel.

- Robert



From dagss at student.matnat.uio.no  Fri Jul 10 11:42:32 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 10 Jul 2009 11:42:32 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <9D942F94-B491-4A41-9BF4-BFEA5F4AEC06@math.washington.edu>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
	<8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
	<d7a91c2e4a2b688c638411fae250180e.squirrel@webmail.uio.no>
	<9D942F94-B491-4A41-9BF4-BFEA5F4AEC06@math.washington.edu>
Message-ID: <ad46cdbc42737766ad32292ce8e955a9.squirrel@webmail.uio.no>

> On Jul 9, 2009, at 4:32 AM, Dag Sverre Seljebotn wrote:
>
>> Robert Bradshaw wrote:
>>> On Jul 8, 2009, at 5:07 PM, Lisandro Dalcin wrote:
>>>
>>>> On Wed, Jul 8, 2009 at 6:07 PM, Dag Sverre
>>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>>> I think you mistunderstood me... I meant (1) and (2) in
>>>>> combination.
>>>>
>>>> Indeed... My fault this time...
>>>>
>>>>> Trying again:
>>>>>
>>>>> 1) We introduce "?"
>>>>> 2) If you do not know the exact type, you HAVE to use ?. Not
>>>>> using ?
>>>>> implies you know the exact type.
>>>
>>>
>>> I think this is a bad idea. If you have an extern typedef int, how
>>> often do you really know what it is? On all platforms? With all c
>>> configuration options and into the future? Nearly every typedef int
>>> would have to be unranked, which would be a pain to code with.
>>
>> I just don't see what alternative you are proposing though? Today's
>> behaviour is not even defined except through an ever-changing
>> implementation, and leads to bugs; is that what you are proposing
>> to keep?
>>
>> Just write down the exact behaviour that you want and I'll be able to
>> follow you here.
>
> "ctypdef int foo" means that foo will be a signed integer type, and
> most of the code involved will be size-agnostic, but when mixing with
> other types it will have the ranking of an int. This is what it means
> now, right? Perhaps we could say that rankings being equal, we defer
> to the typedef type.

It doesn't mean that yet, but when #333 is reviewed and applied, then yes.

> I just looked at the Sage codebase and there aren't nearly as many
> extern ctypedef ints as I thought. Nearly all of them are not a
> specific size, yet we still often do arithmetic with them (e.g. with
> literal ints). Short of a configure script, there is often no way to
> know at Cython compile time what the actual sizes are.
>
> If I understand your proposal, you want to make an "ctypedef int foo"
> mean
>
> ctypedef int foo # as it is now
> assert sizeof(int) == sizeof(foo)
>
> and
>
> ctypedef some int foo # or whatever notation we have
>
> mean we disallow all rank-based decisions (unless we provide explicit
> rank information).

Yes.

OK. I suppose this thread should be nearing termination;  since this is a
messy topic and no resolution seems at hand, I suppose the default is to
keep status quo (but apply #333) until somebody can invest much more time
in this.

I'll just state my reponse:

a) I agree with you that most external typedefs are in reality unknown.

b) I agree with Greg and Lisandro that when somebody write "ctypedef int
foo" it is rather unobvious that Cython doesn's assume it is exact.
(Documentation isn't always the best solution for unintuitive syntax!)

c) On one hand you say the type is usually unknown, on the other that it
is better to pretend that the rank is known and live with any disrepancies
and errors this may cause. If the type isn't known, isn't it better to get
an error and require a cast in situations where the type MUST be known?
You are saying it is better to generate incorrect code...

d) A simple "a = b + 34" (with different C external typedef types for a
and b) is not a problem because Cython doesn't need to do anything with
the rank information. Leaving the rank undefined is OK in a lot of
situations.

e) I believe one can modify Cython to rely on common sub-expression
elimination in the C compiler instead of temporaries in some situations to
eliminate most rank-based decisions and leave most to the C compiler.

But, we can never get around cases like this (with C++-style overloading):

int func(short)
float func(long)

because func can have side-effects and we can't know the type of a
temporary to put it in.

I must also note that the contrived example above leads to serious
problems for your approach as well.

Dag Sverre


From robertwb at math.washington.edu  Fri Jul 10 18:33:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 10 Jul 2009 09:33:51 -0700
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <ad46cdbc42737766ad32292ce8e955a9.squirrel@webmail.uio.no>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
	<8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
	<d7a91c2e4a2b688c638411fae250180e.squirrel@webmail.uio.no>
	<9D942F94-B491-4A41-9BF4-BFEA5F4AEC06@math.washington.edu>
	<ad46cdbc42737766ad32292ce8e955a9.squirrel@webmail.uio.no>
Message-ID: <2ABCFC2B-03C3-4CC1-8F92-A9504CECD098@math.washington.edu>

On Jul 10, 2009, at 2:42 AM, Dag Sverre Seljebotn wrote:

> OK. I suppose this thread should be nearing termination;  since  
> this is a
> messy topic and no resolution seems at hand, I suppose the default  
> is to
> keep status quo (but apply #333)

Yep. I've been meaning to look at that.

> until somebody can invest much more time in this.

I think it would be worth adding unranked types ("ctypedef some int  
foo" or "ctypedef unknown int foo"), and possibly even a way to  
declare relative rankings (though I don't know that we've come across  
the right syntax to do this yet). Once they're in and we start using  
them, then we can re-open the debate about what the default should  
be. At this point we'll have a better migration path and a better  
feel how restrictive (or not) using these "unranked" types actually is.

> I'll just state my reponse:
>
> a) I agree with you that most external typedefs are in reality  
> unknown.
>
> b) I agree with Greg and Lisandro that when somebody write  
> "ctypedef int
> foo" it is rather unobvious that Cython doesn's assume it is exact.
> (Documentation isn't always the best solution for unintuitive syntax!)
>
> c) On one hand you say the type is usually unknown, on the other  
> that it
> is better to pretend that the rank is known and live with any  
> disrepancies
> and errors this may cause. If the type isn't known, isn't it better  
> to get
> an error and require a cast in situations where the type MUST be  
> known?
> You are saying it is better to generate incorrect code...
>
> d) A simple "a = b + 34" (with different C external typedef types  
> for a
> and b) is not a problem because Cython doesn't need to do anything  
> with
> the rank information. Leaving the rank undefined is OK in a lot of
> situations.
>
> e) I believe one can modify Cython to rely on common sub-expression
> elimination in the C compiler instead of temporaries in some  
> situations to
> eliminate most rank-based decisions and leave most to the C compiler.
>
> But, we can never get around cases like this (with C++-style  
> overloading):
>
> int func(short)
> float func(long)
>
> because func can have side-effects and we can't know the type of a
> temporary to put it in.
>
> I must also note that the contrived example above leads to serious
> problems for your approach as well.

Yes, this is an inherent restriction of having the Cython compile  
stage absolutely proceed the C compilation stage.

- Robert




From dagss at student.matnat.uio.no  Fri Jul 10 20:12:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 10 Jul 2009 20:12:23 +0200
Subject: [Cython] new idea for typedef integrals
In-Reply-To: <2ABCFC2B-03C3-4CC1-8F92-A9504CECD098@math.washington.edu>
References: <e7ba66e40907031100w113fe79bidcb3843589f42027@mail.gmail.com>
	<e7ba66e40907070632n15dbced0l5ef3a546d598c8d8@mail.gmail.com>
	<4A5351E0.7020009@student.matnat.uio.no>
	<e7ba66e40907071009j19766ebei7a61ee7e0d7a68bc@mail.gmail.com>
	<4A53862A.10206@student.matnat.uio.no>
	<e7ba66e40907071124g2b2b1c50m94b5dae4ad680798@mail.gmail.com>
	<7C0793E6-442C-4EBF-BBCC-32443CB81227@math.washington.edu>
	<fdf27141c9acc6987dc2b44b2eb27ff0.squirrel@webmail.uio.no>
	<e7ba66e40907081321x16fd15ccl986f1c62916c962@mail.gmail.com>
	<a7d787807c0740e030f1deb0a9af74bc.squirrel@webmail.uio.no>
	<e7ba66e40907081707u2fbcbbebs75e70058b187c06c@mail.gmail.com>
	<8D155FDC-BC6C-4380-B5B4-7575F711C66F@math.washington.edu>
	<d7a91c2e4a2b688c638411fae250180e.squirrel@webmail.uio.no>
	<9D942F94-B491-4A41-9BF4-BFEA5F4AEC06@math.washington.edu>
	<ad46cdbc42737766ad32292ce8e955a9.squirrel@webmail.uio.no>
	<2ABCFC2B-03C3-4CC1-8F92-A9504CECD098@math.washington.edu>
Message-ID: <bc3da1504bc715287d4622d2a0a96dbd.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> I think it would be worth adding unranked types ("ctypedef some int
> foo" or "ctypedef unknown int foo"), and possibly even a way to
> declare relative rankings (though I don't know that we've come across
> the right syntax to do this yet). Once they're in and we start using
> them, then we can re-open the debate about what the default should
> be. At this point we'll have a better migration path and a better
> feel how restrictive (or not) using these "unranked" types actually is.

Sounds wise.

As for syntax, I think it should be tied to ctypedef-ing the type, like

ctypedef some int foo:
    > int
    < bar
    < baz

or

ctypedef some int foo [>int, <bar, <baz]



Dag Sverre


From joschu at caltech.edu  Fri Jul 10 22:55:11 2009
From: joschu at caltech.edu (John Schulman)
Date: Fri, 10 Jul 2009 16:55:11 -0400
Subject: [Cython] why is this code so slow
Message-ID: <185761440907101355h70f470aev188a9af465ae8391@mail.gmail.com>

After reading the "Cython for Numpy Users" page, I tried rewriting a
function in cython, which I had previously implemented in python and
then scipy.weave.
It runs just as slow as the python version, which is 100 times slower
than the C version.
Maybe someone can tell me what is slowing it down.
Thanks,
John

#cython_scripts.py
from __future__ import division
import numpy as np
cimport numpy as np

DTYPE = np.int
ctypedef np.int_t DTYPE_t

DTYPE2 = np.double
ctypedef np.double_t DTYPE2_t

cdef inline int int_max(int a, int b): return a if a >= b else b
cdef inline int int_min(int a, int b): return a if a <= b else b

def firstpass_labels(np.ndarray[DTYPE2_t,ndim=2] arr,list links,int
s_back,double thresh):
    cdef int n_s,n_ch
    n_s = arr.shape[0]; n_ch = arr.shape[1]
    cdef np.ndarray labels = np.zeros([n_s,n_ch],dtype=DTYPE)
    cdef DTYPE_t c_label
    c_label = 1
    cdef int i_s, i_ch, j_s, j_ch, j_ch_ind, j_sstart

    cdef np.ndarray links_arr = np.zeros([n_ch,n_ch+1],dtype=DTYPE)
    for (source,targs) in enumerate(links):
        links_arr[source,0] = len(targs)
        links_arr[source,1:(len(targs)+1)] = links[source]

    for i_s in range(n_s):
        for i_ch in range(n_ch):
            if arr[i_s,i_ch] > thresh:
                j_sstart = int_max(0,i_s-s_back)
                for j_s in range(j_sstart,i_s+1):
                    for j_ch_ind in range(1,links_arr[i_ch][0]+1):
                        j_ch = links_arr[i_ch,j_ch_ind]
                        if labels[j_s,j_ch] != 0:
                            labels[i_s,i_ch] = labels[j_s,j_ch]

                if labels[i_s,i_ch] == 0:
                    labels[i_s,i_ch] = c_label
                    c_label += 1

    return labels



And here's the function I use to run it:
#test_cy.py
import time
import numpy as np

import pyximport; pyximport.install()
import cython_scripts



arr = np.random.random( (400000,3))
s_back = 3
thresh = .7
links2 = [[0,1],[0,1,2],[1,2]]
t = time.time()
print cython_scripts.firstpass_labels(arr,links2,s_back,thresh)
print "cython %f"%(time.time()-t)

From kwmsmith at gmail.com  Fri Jul 10 23:21:31 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 10 Jul 2009 16:21:31 -0500
Subject: [Cython] why is this code so slow
In-Reply-To: <185761440907101355h70f470aev188a9af465ae8391@mail.gmail.com>
References: <185761440907101355h70f470aev188a9af465ae8391@mail.gmail.com>
Message-ID: <c266a7930907101421q59bff5f9hc35ecec889411ddf@mail.gmail.com>

On Fri, Jul 10, 2009 at 3:55 PM, John Schulman<joschu at caltech.edu> wrote:
> After reading the "Cython for Numpy Users" page, I tried rewriting a
> function in cython, which I had previously implemented in python and
> then scipy.weave.
> It runs just as slow as the python version, which is 100 times slower
> than the C version.
> Maybe someone can tell me what is slowing it down.
> Thanks,
> John
>
> #cython_scripts.py
> from __future__ import division
> import numpy as np
> cimport numpy as np
>
> DTYPE = np.int
> ctypedef np.int_t DTYPE_t
>
> DTYPE2 = np.double
> ctypedef np.double_t DTYPE2_t
>
> cdef inline int int_max(int a, int b): return a if a >= b else b
> cdef inline int int_min(int a, int b): return a if a <= b else b
>
> def firstpass_labels(np.ndarray[DTYPE2_t,ndim=2] arr,list links,int
> s_back,double thresh):
> ? ?cdef int n_s,n_ch
> ? ?n_s = arr.shape[0]; n_ch = arr.shape[1]
> ? ?cdef np.ndarray labels = np.zeros([n_s,n_ch],dtype=DTYPE)
> ? ?cdef DTYPE_t c_label
> ? ?c_label = 1
> ? ?cdef int i_s, i_ch, j_s, j_ch, j_ch_ind, j_sstart
>
> ? ?cdef np.ndarray links_arr = np.zeros([n_ch,n_ch+1],dtype=DTYPE)
> ? ?for (source,targs) in enumerate(links):
> ? ? ? ?links_arr[source,0] = len(targs)
> ? ? ? ?links_arr[source,1:(len(targs)+1)] = links[source]
>
> ? ?for i_s in range(n_s):
> ? ? ? ?for i_ch in range(n_ch):
> ? ? ? ? ? ?if arr[i_s,i_ch] > thresh:
> ? ? ? ? ? ? ? ?j_sstart = int_max(0,i_s-s_back)
> ? ? ? ? ? ? ? ?for j_s in range(j_sstart,i_s+1):
> ? ? ? ? ? ? ? ? ? ?for j_ch_ind in range(1,links_arr[i_ch][0]+1):
> ? ? ? ? ? ? ? ? ? ? ? ?j_ch = links_arr[i_ch,j_ch_ind]
> ? ? ? ? ? ? ? ? ? ? ? ?if labels[j_s,j_ch] != 0:
> ? ? ? ? ? ? ? ? ? ? ? ? ? ?labels[i_s,i_ch] = labels[j_s,j_ch]
>
> ? ? ? ? ? ? ? ?if labels[i_s,i_ch] == 0:
> ? ? ? ? ? ? ? ? ? ?labels[i_s,i_ch] = c_label
> ? ? ? ? ? ? ? ? ? ?c_label += 1
>
> ? ?return labels
>
>
>
> And here's the function I use to run it:
> #test_cy.py
> import time
> import numpy as np
>
> import pyximport; pyximport.install()
> import cython_scripts
>
>
>
> arr = np.random.random( (400000,3))
> s_back = 3
> thresh = .7
> links2 = [[0,1],[0,1,2],[1,2]]
> t = time.time()
> print cython_scripts.firstpass_labels(arr,links2,s_back,thresh)
> print "cython %f"%(time.time()-t)

I get a X 70-80 speedup with the following changes.  You can boost it
more by turning off boundschecking & declaring every buffer's mode to
be 'c'; see the documentation here:

http://docs.cython.org/docs/numpy_tutorial.html#tuning-indexing-further

#---------------------------------------------------------------------------
from __future__ import division
import numpy as np
cimport numpy as np

DTYPE = np.int
ctypedef np.int_t DTYPE_t

DTYPE2 = np.double
ctypedef np.double_t DTYPE2_t

cdef inline int int_max(int a, int b): return a if a >= b else b
cdef inline int int_min(int a, int b): return a if a <= b else b

def firstpass_labels(np.ndarray[DTYPE2_t,ndim=2] arr,list links,int
s_back,double thresh):
   cdef int n_s,n_ch
   n_s = arr.shape[0]; n_ch = arr.shape[1]
   # XXX: changed!
   cdef np.ndarray[DTYPE_t, ndim=2] labels = np.zeros([n_s,n_ch],dtype=DTYPE)
   cdef DTYPE_t c_label
   c_label = 1
   cdef int i_s, i_ch, j_s, j_ch, j_ch_ind, j_sstart

   #XXX: changed!
   cdef np.ndarray[DTYPE_t, ndim=2] links_arr =
np.zeros([n_ch,n_ch+1],dtype=DTYPE)
   for (source,targs) in enumerate(links):
       links_arr[source,0] = len(targs)
       links_arr[source,1:(len(targs)+1)] = links[source]

   for i_s in range(n_s):
       for i_ch in range(n_ch):
           if arr[i_s,i_ch] > thresh:
               j_sstart = int_max(0,i_s-s_back)
               for j_s in range(j_sstart,i_s+1):
                   for j_ch_ind in range(1,links_arr[i_ch,0]+1):
                       j_ch = links_arr[i_ch,j_ch_ind]
                       if labels[j_s,j_ch] != 0:
                           labels[i_s,i_ch] = labels[j_s,j_ch]

               if labels[i_s,i_ch] == 0:
                   labels[i_s,i_ch] = c_label
                   c_label += 1

   return labels
#-------------------------------------------------------------------------------------------------------------------

From joschu at caltech.edu  Fri Jul 10 23:36:27 2009
From: joschu at caltech.edu (John Schulman)
Date: Fri, 10 Jul 2009 17:36:27 -0400
Subject: [Cython] why is this code so slow
In-Reply-To: <c266a7930907101421q59bff5f9hc35ecec889411ddf@mail.gmail.com>
References: <185761440907101355h70f470aev188a9af465ae8391@mail.gmail.com>
	<c266a7930907101421q59bff5f9hc35ecec889411ddf@mail.gmail.com>
Message-ID: <185761440907101436n76277bc6i2da91fabbec998fe@mail.gmail.com>

Wow! Thanks for the quick reply!

When I run it with your changes, plus removing bounds checks, it goes
faster than when I build it with weave.inline and weave.ext_tools

pure python 7.932142
weave 0.084316
ext_tools 0.081650
cython 0.076264


On Fri, Jul 10, 2009 at 5:21 PM, Kurt Smith<kwmsmith at gmail.com> wrote:
> On Fri, Jul 10, 2009 at 3:55 PM, John Schulman<joschu at caltech.edu> wrote:
>> After reading the "Cython for Numpy Users" page, I tried rewriting a
>> function in cython, which I had previously implemented in python and
>> then scipy.weave.
>> It runs just as slow as the python version, which is 100 times slower
>> than the C version.
>> Maybe someone can tell me what is slowing it down.
>> Thanks,
>> John
>>
>> #cython_scripts.py
>> from __future__ import division
>> import numpy as np
>> cimport numpy as np
>>
>> DTYPE = np.int
>> ctypedef np.int_t DTYPE_t
>>
>> DTYPE2 = np.double
>> ctypedef np.double_t DTYPE2_t
>>
>> cdef inline int int_max(int a, int b): return a if a >= b else b
>> cdef inline int int_min(int a, int b): return a if a <= b else b
>>
>> def firstpass_labels(np.ndarray[DTYPE2_t,ndim=2] arr,list links,int
>> s_back,double thresh):
>> ? ?cdef int n_s,n_ch
>> ? ?n_s = arr.shape[0]; n_ch = arr.shape[1]
>> ? ?cdef np.ndarray labels = np.zeros([n_s,n_ch],dtype=DTYPE)
>> ? ?cdef DTYPE_t c_label
>> ? ?c_label = 1
>> ? ?cdef int i_s, i_ch, j_s, j_ch, j_ch_ind, j_sstart
>>
>> ? ?cdef np.ndarray links_arr = np.zeros([n_ch,n_ch+1],dtype=DTYPE)
>> ? ?for (source,targs) in enumerate(links):
>> ? ? ? ?links_arr[source,0] = len(targs)
>> ? ? ? ?links_arr[source,1:(len(targs)+1)] = links[source]
>>
>> ? ?for i_s in range(n_s):
>> ? ? ? ?for i_ch in range(n_ch):
>> ? ? ? ? ? ?if arr[i_s,i_ch] > thresh:
>> ? ? ? ? ? ? ? ?j_sstart = int_max(0,i_s-s_back)
>> ? ? ? ? ? ? ? ?for j_s in range(j_sstart,i_s+1):
>> ? ? ? ? ? ? ? ? ? ?for j_ch_ind in range(1,links_arr[i_ch][0]+1):
>> ? ? ? ? ? ? ? ? ? ? ? ?j_ch = links_arr[i_ch,j_ch_ind]
>> ? ? ? ? ? ? ? ? ? ? ? ?if labels[j_s,j_ch] != 0:
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ?labels[i_s,i_ch] = labels[j_s,j_ch]
>>
>> ? ? ? ? ? ? ? ?if labels[i_s,i_ch] == 0:
>> ? ? ? ? ? ? ? ? ? ?labels[i_s,i_ch] = c_label
>> ? ? ? ? ? ? ? ? ? ?c_label += 1
>>
>> ? ?return labels
>>
>>
>>
>> And here's the function I use to run it:
>> #test_cy.py
>> import time
>> import numpy as np
>>
>> import pyximport; pyximport.install()
>> import cython_scripts
>>
>>
>>
>> arr = np.random.random( (400000,3))
>> s_back = 3
>> thresh = .7
>> links2 = [[0,1],[0,1,2],[1,2]]
>> t = time.time()
>> print cython_scripts.firstpass_labels(arr,links2,s_back,thresh)
>> print "cython %f"%(time.time()-t)
>
> I get a X 70-80 speedup with the following changes. ?You can boost it
> more by turning off boundschecking & declaring every buffer's mode to
> be 'c'; see the documentation here:
>
> http://docs.cython.org/docs/numpy_tutorial.html#tuning-indexing-further
>
> #---------------------------------------------------------------------------
> from __future__ import division
> import numpy as np
> cimport numpy as np
>
> DTYPE = np.int
> ctypedef np.int_t DTYPE_t
>
> DTYPE2 = np.double
> ctypedef np.double_t DTYPE2_t
>
> cdef inline int int_max(int a, int b): return a if a >= b else b
> cdef inline int int_min(int a, int b): return a if a <= b else b
>
> def firstpass_labels(np.ndarray[DTYPE2_t,ndim=2] arr,list links,int
> s_back,double thresh):
> ? cdef int n_s,n_ch
> ? n_s = arr.shape[0]; n_ch = arr.shape[1]
> ? # XXX: changed!
> ? cdef np.ndarray[DTYPE_t, ndim=2] labels = np.zeros([n_s,n_ch],dtype=DTYPE)
> ? cdef DTYPE_t c_label
> ? c_label = 1
> ? cdef int i_s, i_ch, j_s, j_ch, j_ch_ind, j_sstart
>
> ? #XXX: changed!
> ? cdef np.ndarray[DTYPE_t, ndim=2] links_arr =
> np.zeros([n_ch,n_ch+1],dtype=DTYPE)
> ? for (source,targs) in enumerate(links):
> ? ? ? links_arr[source,0] = len(targs)
> ? ? ? links_arr[source,1:(len(targs)+1)] = links[source]
>
> ? for i_s in range(n_s):
> ? ? ? for i_ch in range(n_ch):
> ? ? ? ? ? if arr[i_s,i_ch] > thresh:
> ? ? ? ? ? ? ? j_sstart = int_max(0,i_s-s_back)
> ? ? ? ? ? ? ? for j_s in range(j_sstart,i_s+1):
> ? ? ? ? ? ? ? ? ? for j_ch_ind in range(1,links_arr[i_ch,0]+1):
> ? ? ? ? ? ? ? ? ? ? ? j_ch = links_arr[i_ch,j_ch_ind]
> ? ? ? ? ? ? ? ? ? ? ? if labels[j_s,j_ch] != 0:
> ? ? ? ? ? ? ? ? ? ? ? ? ? labels[i_s,i_ch] = labels[j_s,j_ch]
>
> ? ? ? ? ? ? ? if labels[i_s,i_ch] == 0:
> ? ? ? ? ? ? ? ? ? labels[i_s,i_ch] = c_label
> ? ? ? ? ? ? ? ? ? c_label += 1
>
> ? return labels
> #-------------------------------------------------------------------------------------------------------------------
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From kwmsmith at gmail.com  Sat Jul 11 00:03:21 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 10 Jul 2009 17:03:21 -0500
Subject: [Cython] why is this code so slow
In-Reply-To: <185761440907101436n76277bc6i2da91fabbec998fe@mail.gmail.com>
References: <185761440907101355h70f470aev188a9af465ae8391@mail.gmail.com>
	<c266a7930907101421q59bff5f9hc35ecec889411ddf@mail.gmail.com>
	<185761440907101436n76277bc6i2da91fabbec998fe@mail.gmail.com>
Message-ID: <c266a7930907101503s77593750lb17081112ecffd9f@mail.gmail.com>

On Fri, Jul 10, 2009 at 4:36 PM, John Schulman<joschu at caltech.edu> wrote:
> Wow! Thanks for the quick reply!
>
> When I run it with your changes, plus removing bounds checks, it goes
> faster than when I build it with weave.inline and weave.ext_tools
>
> pure python 7.932142
> weave 0.084316
> ext_tools 0.081650
> cython 0.076264

Glad to see it do well!  Although the credit goes to Cython as a whole
& Dag Sverre Seljebotn for putting the buffer stuff in place last
year.

BTW, the reason it was slow in your original version is that it used
Python C API PyObject_GetItem /SetItem in the inner loop, boxed the
integers into Python Objects, and created tuples for the
getitem/setitem calls.  Adding the buffer declarations told Cython to
generate fast indexing on the underlying ndarray data and not generate
any Python API calls.

One way you can tell is by looking at the generated C code (or run
with cython -a cython_script.pyx for a html annotation file) and look
inside the inner loop to see if there are any Python C API calls
there.  The first order of business (and often last) is to arrange
things to get Python API calls removed.

>
>
> On Fri, Jul 10, 2009 at 5:21 PM, Kurt Smith<kwmsmith at gmail.com> wrote:
>> On Fri, Jul 10, 2009 at 3:55 PM, John Schulman<joschu at caltech.edu> wrote:
>>> After reading the "Cython for Numpy Users" page, I tried rewriting a
>>> function in cython, which I had previously implemented in python and
>>> then scipy.weave.
>>> It runs just as slow as the python version, which is 100 times slower
>>> than the C version.
>>> Maybe someone can tell me what is slowing it down.
>>> Thanks,
>>> John
>>>
>>> #cython_scripts.py
>>> from __future__ import division
>>> import numpy as np
>>> cimport numpy as np
>>>
>>> DTYPE = np.int
>>> ctypedef np.int_t DTYPE_t
>>>
>>> DTYPE2 = np.double
>>> ctypedef np.double_t DTYPE2_t
>>>
>>> cdef inline int int_max(int a, int b): return a if a >= b else b
>>> cdef inline int int_min(int a, int b): return a if a <= b else b
>>>
>>> def firstpass_labels(np.ndarray[DTYPE2_t,ndim=2] arr,list links,int
>>> s_back,double thresh):
>>> ? ?cdef int n_s,n_ch
>>> ? ?n_s = arr.shape[0]; n_ch = arr.shape[1]
>>> ? ?cdef np.ndarray labels = np.zeros([n_s,n_ch],dtype=DTYPE)
>>> ? ?cdef DTYPE_t c_label
>>> ? ?c_label = 1
>>> ? ?cdef int i_s, i_ch, j_s, j_ch, j_ch_ind, j_sstart
>>>
>>> ? ?cdef np.ndarray links_arr = np.zeros([n_ch,n_ch+1],dtype=DTYPE)
>>> ? ?for (source,targs) in enumerate(links):
>>> ? ? ? ?links_arr[source,0] = len(targs)
>>> ? ? ? ?links_arr[source,1:(len(targs)+1)] = links[source]
>>>
>>> ? ?for i_s in range(n_s):
>>> ? ? ? ?for i_ch in range(n_ch):
>>> ? ? ? ? ? ?if arr[i_s,i_ch] > thresh:
>>> ? ? ? ? ? ? ? ?j_sstart = int_max(0,i_s-s_back)
>>> ? ? ? ? ? ? ? ?for j_s in range(j_sstart,i_s+1):
>>> ? ? ? ? ? ? ? ? ? ?for j_ch_ind in range(1,links_arr[i_ch][0]+1):
>>> ? ? ? ? ? ? ? ? ? ? ? ?j_ch = links_arr[i_ch,j_ch_ind]
>>> ? ? ? ? ? ? ? ? ? ? ? ?if labels[j_s,j_ch] != 0:
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ?labels[i_s,i_ch] = labels[j_s,j_ch]
>>>
>>> ? ? ? ? ? ? ? ?if labels[i_s,i_ch] == 0:
>>> ? ? ? ? ? ? ? ? ? ?labels[i_s,i_ch] = c_label
>>> ? ? ? ? ? ? ? ? ? ?c_label += 1
>>>
>>> ? ?return labels
>>>
>>>
>>>
>>> And here's the function I use to run it:
>>> #test_cy.py
>>> import time
>>> import numpy as np
>>>
>>> import pyximport; pyximport.install()
>>> import cython_scripts
>>>
>>>
>>>
>>> arr = np.random.random( (400000,3))
>>> s_back = 3
>>> thresh = .7
>>> links2 = [[0,1],[0,1,2],[1,2]]
>>> t = time.time()
>>> print cython_scripts.firstpass_labels(arr,links2,s_back,thresh)
>>> print "cython %f"%(time.time()-t)
>>
>> I get a X 70-80 speedup with the following changes. ?You can boost it
>> more by turning off boundschecking & declaring every buffer's mode to
>> be 'c'; see the documentation here:
>>
>> http://docs.cython.org/docs/numpy_tutorial.html#tuning-indexing-further
>>
>> #---------------------------------------------------------------------------
>> from __future__ import division
>> import numpy as np
>> cimport numpy as np
>>
>> DTYPE = np.int
>> ctypedef np.int_t DTYPE_t
>>
>> DTYPE2 = np.double
>> ctypedef np.double_t DTYPE2_t
>>
>> cdef inline int int_max(int a, int b): return a if a >= b else b
>> cdef inline int int_min(int a, int b): return a if a <= b else b
>>
>> def firstpass_labels(np.ndarray[DTYPE2_t,ndim=2] arr,list links,int
>> s_back,double thresh):
>> ? cdef int n_s,n_ch
>> ? n_s = arr.shape[0]; n_ch = arr.shape[1]
>> ? # XXX: changed!
>> ? cdef np.ndarray[DTYPE_t, ndim=2] labels = np.zeros([n_s,n_ch],dtype=DTYPE)
>> ? cdef DTYPE_t c_label
>> ? c_label = 1
>> ? cdef int i_s, i_ch, j_s, j_ch, j_ch_ind, j_sstart
>>
>> ? #XXX: changed!
>> ? cdef np.ndarray[DTYPE_t, ndim=2] links_arr =
>> np.zeros([n_ch,n_ch+1],dtype=DTYPE)
>> ? for (source,targs) in enumerate(links):
>> ? ? ? links_arr[source,0] = len(targs)
>> ? ? ? links_arr[source,1:(len(targs)+1)] = links[source]
>>
>> ? for i_s in range(n_s):
>> ? ? ? for i_ch in range(n_ch):
>> ? ? ? ? ? if arr[i_s,i_ch] > thresh:
>> ? ? ? ? ? ? ? j_sstart = int_max(0,i_s-s_back)
>> ? ? ? ? ? ? ? for j_s in range(j_sstart,i_s+1):
>> ? ? ? ? ? ? ? ? ? for j_ch_ind in range(1,links_arr[i_ch,0]+1):
>> ? ? ? ? ? ? ? ? ? ? ? j_ch = links_arr[i_ch,j_ch_ind]
>> ? ? ? ? ? ? ? ? ? ? ? if labels[j_s,j_ch] != 0:
>> ? ? ? ? ? ? ? ? ? ? ? ? ? labels[i_s,i_ch] = labels[j_s,j_ch]
>>
>> ? ? ? ? ? ? ? if labels[i_s,i_ch] == 0:
>> ? ? ? ? ? ? ? ? ? labels[i_s,i_ch] = c_label
>> ? ? ? ? ? ? ? ? ? c_label += 1
>>
>> ? return labels
>> #-------------------------------------------------------------------------------------------------------------------
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dsimmons.com at gmail.com  Sun Jul 12 05:06:46 2009
From: dsimmons.com at gmail.com (David Simmons-Duffin)
Date: Sat, 11 Jul 2009 23:06:46 -0400
Subject: [Cython] replacing LiE error() with a python exception
Message-ID: <C5B2723B-A703-4B3B-89B5-E580DE9431EA@gmail.com>

I'm still working on wrapping the computer algebra package LiE in  
cython.  For those unfamiliar, LiE is basically it's own language,  
written in C.  And I'm interested in calling some of the math-heavy  
functions, but bypassing the interpreter.

Error checking is sprinkled throughout the LiE source, for instance:

matrix *mat_mul_mat_mat(matrix *a, matrix *b) {
     if (a->ncols != b->nrows)
         error("Number columns arg1 unequal number of rows arg2 .\n");
     return Matmult(a, b);
}

When an error is found, LiE calls the error() function, which prints  
out a traceback, cleans up a little bit, and finally does a longjump  
to "envbuf," which is at the beginning of the interpreter loop:

void error(char *format, ...)
{
     symblst         list;
     extern jmp_buf  envbuf;
     va_list ap;
     va_start(ap, format);
     vfprintf(stderr, format, ap);
     if (am_monitor) vfprintf(monfile, format, ap);
     if (label->name)
     {
          boolean printin = (label->name != seq_name);
          if (printin)
             Printf("(in %s",name_tab[label->name]);
          if (no_terminal(cur_in) || strcmp(label->fname, "stdin")!=0) {
             if (!printin) Printf("(");
             Printf(" at line %d of file %s)\n",label->line, label- 
 >fname);
          }
          else
             if (printin) Printf(")\n");
     }
     if (fun_name)
         Printf("[in function %s ]\n", name_tab[fun_name]);
     for (list = top_definitions; list!=NULL; list = list->next)
         /* Recover symbol table */
     {   if (list->class == FUNCTION_COPIED) list->class = FUNCTION;
	if (list->next && list->next->class == DUMMY)
             list->next = list->next->next;      /* Remove sym */
     }
     if (repair_obj) {
         setshared(repair_obj);
         repair_obj = (object) NULL;
     }
     if (cur_in==stdin)
       clear_input();
     else
       do exit_input_file(parsing); while (cur_in!=stdin); /* pop  
input files */
     longjmp(envbuf, -1);
}

Instead of doing all this stuff, and longjumping to the LiE  
interpreter, I'd like to raise a python exception that could be caught  
by whatever cython function called "mat_mul_mat_mat."  Does anyone  
have any advice on how to do this?  I took a look at the python  
exception portion of the C/API.  It seems like I might want to make a  
C file with my own error function:

error(char* format, ...) {
     turn format into a string
     PyErr_SetString(errorstring)
     longjump somewhere ???
}

and then comment out the error function in the LiE source... but I'm  
not totally sure how to put it all together.  Also, If it's possible,  
I'd like to avoid modifying the LiE source too much -- i.e. commenting  
out an entire function.  I'm not a C master, so I wouldn't be  
surprised if I've missed an obvious way to do this.

Thanks for the advice!

David





From robertwb at math.washington.edu  Sun Jul 12 05:21:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 11 Jul 2009 20:21:26 -0700
Subject: [Cython] replacing LiE error() with a python exception
In-Reply-To: <C5B2723B-A703-4B3B-89B5-E580DE9431EA@gmail.com>
References: <C5B2723B-A703-4B3B-89B5-E580DE9431EA@gmail.com>
Message-ID: <7D43BEC2-8454-4488-B461-576D1EEA0340@math.washington.edu>

Sounds like the thing to do here is signal handling. I'm not an  
expert at it myself, but I know it can be done because we use it a  
lot in Sage.

- Robert

On Jul 11, 2009, at 8:06 PM, David Simmons-Duffin wrote:

> I'm still working on wrapping the computer algebra package LiE in
> cython.  For those unfamiliar, LiE is basically it's own language,
> written in C.  And I'm interested in calling some of the math-heavy
> functions, but bypassing the interpreter.
>
> Error checking is sprinkled throughout the LiE source, for instance:
>
> matrix *mat_mul_mat_mat(matrix *a, matrix *b) {
>      if (a->ncols != b->nrows)
>          error("Number columns arg1 unequal number of rows arg2 .\n");
>      return Matmult(a, b);
> }
>
> When an error is found, LiE calls the error() function, which prints
> out a traceback, cleans up a little bit, and finally does a longjump
> to "envbuf," which is at the beginning of the interpreter loop:
>
> void error(char *format, ...)
> {
>      symblst         list;
>      extern jmp_buf  envbuf;
>      va_list ap;
>      va_start(ap, format);
>      vfprintf(stderr, format, ap);
>      if (am_monitor) vfprintf(monfile, format, ap);
>      if (label->name)
>      {
>           boolean printin = (label->name != seq_name);
>           if (printin)
>              Printf("(in %s",name_tab[label->name]);
>           if (no_terminal(cur_in) || strcmp(label->fname, "stdin")! 
> =0) {
>              if (!printin) Printf("(");
>              Printf(" at line %d of file %s)\n",label->line, label-
>> fname);
>           }
>           else
>              if (printin) Printf(")\n");
>      }
>      if (fun_name)
>          Printf("[in function %s ]\n", name_tab[fun_name]);
>      for (list = top_definitions; list!=NULL; list = list->next)
>          /* Recover symbol table */
>      {   if (list->class == FUNCTION_COPIED) list->class = FUNCTION;
> 	if (list->next && list->next->class == DUMMY)
>              list->next = list->next->next;      /* Remove sym */
>      }
>      if (repair_obj) {
>          setshared(repair_obj);
>          repair_obj = (object) NULL;
>      }
>      if (cur_in==stdin)
>        clear_input();
>      else
>        do exit_input_file(parsing); while (cur_in!=stdin); /* pop
> input files */
>      longjmp(envbuf, -1);
> }
>
> Instead of doing all this stuff, and longjumping to the LiE
> interpreter, I'd like to raise a python exception that could be caught
> by whatever cython function called "mat_mul_mat_mat."  Does anyone
> have any advice on how to do this?  I took a look at the python
> exception portion of the C/API.  It seems like I might want to make a
> C file with my own error function:
>
> error(char* format, ...) {
>      turn format into a string
>      PyErr_SetString(errorstring)
>      longjump somewhere ???
> }
>
> and then comment out the error function in the LiE source... but I'm
> not totally sure how to put it all together.  Also, If it's possible,
> I'd like to avoid modifying the LiE source too much -- i.e. commenting
> out an entire function.  I'm not a C master, so I wouldn't be
> surprised if I've missed an obvious way to do this.
>
> Thanks for the advice!
>
> David
>
>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Mon Jul 13 17:51:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 13 Jul 2009 08:51:27 -0700
Subject: [Cython] casting strings to unsigned char*
In-Reply-To: <42ef4ee0907130647g661e9095x5cc9bbbf3239e0ca@mail.gmail.com>
References: <42ef4ee0907130643l6d00acb0o90aaf0589dc98483@mail.gmail.com>
	<42ef4ee0907130647g661e9095x5cc9bbbf3239e0ca@mail.gmail.com>
Message-ID: <C7EEBC69-1A85-4757-ADC5-1EEBA6AC5FCA@math.washington.edu>

On Jul 13, 2009, at 6:47 AM, Eric Eisner wrote:

> Hi,
>
> I was working on a wrapper for a c function that took an unsigned
> char* and its length (the string could have null bytes, so it needs a
> specific length). I was having some trouble getting cython to compile
> a simple conversion of string to unsigned char*, the way I eventually
> got it to work is:
>
> udata = <unsigned char*><char*>pydata
>
> This was a surprising requirement that took me a while to figure out.
> Is it intentional that strings cannot be directly cast to unsigned
> char?

No, I don't think that's intentional.

> If not, I assume this can be fixed easily...by someone who
> understands the code of course.

Yes, I think that could be fixed relatively easy. However, note that  
casting Python objects directly to char* is skirting all unicode/ 
charset issues.

- Robert



From soeren at apelpie.net  Mon Jul 13 23:19:36 2009
From: soeren at apelpie.net (Soeren Apel)
Date: Mon, 13 Jul 2009 23:19:36 +0200
Subject: [Cython] Troubles with "...is not a cimported module"
Message-ID: <1247519976.7035.28.camel@doodie.home>

Hello everyone,

I'm trying to use cython to build a customized python extension that
interfaces evas from the enlightenment foundation libraries. Their team
has already written a binding which my work relies upon:
http://trac.enlightenment.org/e/browser/trunk/BINDINGS/python/python-evas

I took the files in evas/ and include/, added my move_relative.pyx file
and tried to compile the bunch. However, there are problems arising.
Unfortunately I couldn't find any way to solve this, nor could I find
anyone able to help me with this. For this reason, you're my last
hope :)


Scenario 1: If my move_relative.pyx contains
------------------------------------------------------------------
import evas

cdef _evas_move_relative(evas.c_evas.Evas_Object *obj, int dx, int dy):
        cdef int x, y

        c_evas.evas_object_geometry_get(obj, &x, &y, NULL, NULL)
        c_evas.evas_object_move(obj, x + dx, y + dy)


def evas_move_relative(evas.c_evas.Object object, dx, dy):
        _evas_move_relative(object.obj, dx, dy)
------------------------------------------------------------------
the code makes use of the python-evas binding as intended but fails to
compile:

move_relative.pyx:10:23: 'move_relative' is not a cimported module

I neither understand the meaning of the error message, nor does it give
any helpful clues as to how the problem can be resolved. With that, I'm
kind of stuck.


Scenario 2: If my move_relative.pyx contains
------------------------------------------------------------------
cimport c_evas

cdef _evas_move_relative(c_evas.Evas_Object *obj, int dx, int dy):
        cdef int x, y

        c_evas.evas_object_geometry_get(obj, &x, &y, NULL, NULL)
        c_evas.evas_object_move(obj, x + dx, y + dy)


def evas_move_relative(c_evas.Object object, dx, dy):
        _evas_move_relative(object.obj, dx, dy)
------------------------------------------------------------------
the code compiles just fine but an import fails on runtime:

Traceback (most recent call last):
  File "01-smart_object_cython.py", line 7, in <module>
    from move_relative import evas_move_relative
  File "c_evas.pxd", line 682, in move_relative (move_relative.c:1036)
ImportError: No module named c_evas

Line 1036 contains a call to __Pyx_ImportType() that imports c_evas.

I can see why the import would fail as c_evas is a submodule of evas but
the fact that cython doesn't fail compilation with the "cimported
module" message confuses me.


I would be very happy if someone could shed some light into this issue
and give me a pointer on where to look.

 -Soeren Apel




From kwmsmith at gmail.com  Mon Jul 13 05:24:39 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sun, 12 Jul 2009 22:24:39 -0500
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
Message-ID: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>

In the midst of GSoC, I wanted to float an idea about a Cython
profiler, similar to the current Python profiler.  Specifically I
wanted to see if there's interest, what sorts of features would be
desired, and, broadly, some implementation ideas.

I think the arguments in favor are clear:

Often people use Cython incrementally -- take a .py file, profile it,
put the hotspots in Cython.  If fast enough -- you're done.  If not,
continue to put more slow python code into Cython, or determine if the
Cython code should be massaged to yield better performance.  In the
latter case it is difficult to zone in on the Cython hotspot, for it
is not possible for the Python profiler to see it.  Other external
tools must be used (gprof?, valgrind, mac os x-specific profiler),
which is inconvenient at best; other times they are just plain
unavailable.

I'd like to see a profiling option to the cython compiler that would
allow the user to, as seamlessly as possible, run the compiled
extension module with profiling enabled, and have the user interaction
be just like the current python profiler, specifically with meaningful
output for the Cython-level functions.  If possible, I'd like to have
the cython profiling code make use of the python cProfiler module so
profiling isn't any different between pure Python & Cython compiled
code.  I don't know how feasible hooking up with the python cProfiler
is, so it is just in the pie-in-the-sky stage right now.

Another tantalizing possibility would be line-by-line profiling,
beyond the the regular function-level profiling.

What other features would be desired?  What user interaction features
would be good to have?

Implementation-wise, it could be done as its own transform, enabled by
compile-time switch.

Someone else (Robert? in some thread?) mentioned somewhere about
getting debugger-support in Cython, allowing one to step through
Cython code just like pdb (one can use gdb currently, of course).  I
imagine the implementation issues for a Cython debugger would overlap
a good deal with the profiler (it would be its own separate tranform,
obviously).

I'll put each in a CEP, hopefully soon (and after the benefits of
comments, etc.)

Kurt

From robertwb at math.washington.edu  Tue Jul 14 08:55:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 13 Jul 2009 23:55:45 -0700
Subject: [Cython] Troubles with "...is not a cimported module"
In-Reply-To: <1247519976.7035.28.camel@doodie.home>
References: <1247519976.7035.28.camel@doodie.home>
Message-ID: <684ED397-628E-4E20-BBFE-D173BD17C4D2@math.washington.edu>

On Jul 13, 2009, at 2:19 PM, Soeren Apel wrote:

> Hello everyone,
>
> I'm trying to use cython to build a customized python extension that
> interfaces evas from the enlightenment foundation libraries. Their  
> team
> has already written a binding which my work relies upon:
> http://trac.enlightenment.org/e/browser/trunk/BINDINGS/python/ 
> python-evas
>
> I took the files in evas/ and include/, added my move_relative.pyx  
> file
> and tried to compile the bunch. However, there are problems arising.
> Unfortunately I couldn't find any way to solve this, nor could I find
> anyone able to help me with this. For this reason, you're my last
> hope :)
>
>
> Scenario 1: If my move_relative.pyx contains
> ------------------------------------------------------------------
> import evas
>
> cdef _evas_move_relative(evas.c_evas.Evas_Object *obj, int dx, int  
> dy):
>         cdef int x, y
>
>         c_evas.evas_object_geometry_get(obj, &x, &y, NULL, NULL)
>         c_evas.evas_object_move(obj, x + dx, y + dy)
>
>
> def evas_move_relative(evas.c_evas.Object object, dx, dy):
>         _evas_move_relative(object.obj, dx, dy)
> ------------------------------------------------------------------
> the code makes use of the python-evas binding as intended but fails to
> compile:
>
> move_relative.pyx:10:23: 'move_relative' is not a cimported module
>
> I neither understand the meaning of the error message, nor does it  
> give
> any helpful clues as to how the problem can be resolved. With that,  
> I'm
> kind of stuck.

I'm not sure what that error means either. I could expect "'evas is  
not a cimported module."

>
> Scenario 2: If my move_relative.pyx contains
> ------------------------------------------------------------------
> cimport c_evas
>
> cdef _evas_move_relative(c_evas.Evas_Object *obj, int dx, int dy):
>         cdef int x, y
>
>         c_evas.evas_object_geometry_get(obj, &x, &y, NULL, NULL)
>         c_evas.evas_object_move(obj, x + dx, y + dy)
>
>
> def evas_move_relative(c_evas.Object object, dx, dy):
>         _evas_move_relative(object.obj, dx, dy)
> ------------------------------------------------------------------
> the code compiles just fine but an import fails on runtime:
>
> Traceback (most recent call last):
>   File "01-smart_object_cython.py", line 7, in <module>
>     from move_relative import evas_move_relative
>   File "c_evas.pxd", line 682, in move_relative (move_relative.c:1036)
> ImportError: No module named c_evas
>
> Line 1036 contains a call to __Pyx_ImportType() that imports c_evas.
>
> I can see why the import would fail as c_evas is a submodule of  
> evas but
> the fact that cython doesn't fail compilation with the "cimported
> module" message confuses me.
>
> I would be very happy if someone could shed some light into this issue
> and give me a pointer on where to look.

This is because there is no module c_evas, it's in the evas package  
so has to be imported as evas.c_evas.

It looks like their directory structure is

/evas
     evas.c_evas.pyx  # creates the module evas.c_evas, I'm really  
tempted to depreciate the dotted-filename style of package naming
     ...
/include
     /evas
         c_evas.pxd

Somehow /include/evas is in your include path, rather than just / 
include. You need to "cimport evas.c_evas as c_evas" (or "as evas" if  
you want).

- Robert

From robertwb at math.washington.edu  Tue Jul 14 10:25:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 14 Jul 2009 01:25:42 -0700
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
Message-ID: <682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>

On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote:

> In the midst of GSoC, I wanted to float an idea about a Cython
> profiler, similar to the current Python profiler.  Specifically I
> wanted to see if there's interest, what sorts of features would be
> desired, and, broadly, some implementation ideas.
>
> I think the arguments in favor are clear:
>
> Often people use Cython incrementally -- take a .py file, profile it,
> put the hotspots in Cython.  If fast enough -- you're done.  If not,
> continue to put more slow python code into Cython, or determine if the
> Cython code should be massaged to yield better performance.  In the
> latter case it is difficult to zone in on the Cython hotspot, for it
> is not possible for the Python profiler to see it.  Other external
> tools must be used (gprof?, valgrind, mac os x-specific profiler),
> which is inconvenient at best; other times they are just plain
> unavailable.
>
> I'd like to see a profiling option to the cython compiler that would
> allow the user to, as seamlessly as possible, run the compiled
> extension module with profiling enabled, and have the user interaction
> be just like the current python profiler, specifically with meaningful
> output for the Cython-level functions.  If possible, I'd like to have
> the cython profiling code make use of the python cProfiler module so
> profiling isn't any different between pure Python & Cython compiled
> code.  I don't know how feasible hooking up with the python cProfiler
> is, so it is just in the pie-in-the-sky stage right now.
>
> Another tantalizing possibility would be line-by-line profiling,
> beyond the the regular function-level profiling.

After talking a bit about it at the last Sage days, and looking (a  
tiny bit) into how this stuff works, this really is quite feasible,  
and would be awesome to have.

> What other features would be desired?  What user interaction features
> would be good to have?
>
> Implementation-wise, it could be done as its own transform, enabled by
> compile-time switch.

Certainly we would want to enable this with a compile-time switch  
(and, if it can be done not to verbosely, maybe even via macros).   
I'm not sure transformations would be the approach to take here  
(seems not the right level)--I'd probably add some stuff to the Code  
object and maybe even latch onto the emit_marker for the line-by-line.

> Someone else (Robert? in some thread?) mentioned somewhere about
> getting debugger-support in Cython, allowing one to step through
> Cython code just like pdb (one can use gdb currently, of course).  I
> imagine the implementation issues for a Cython debugger would overlap
> a good deal with the profiler (it would be its own separate tranform,
> obviously).

Yep. I don't have any idea how pdb works, but I'd imagine if we can  
provide fake frames (the code is there for exceptions and locals())  
it wouldn't be too hard to hook up. Making gdb friendlier is an  
orthogonal but still very useful idea to pursue as well, especially  
for code that interfaces with external libraries.

> I'll put each in a CEP, hopefully soon (and after the benefits of
> comments, etc.)

That would be good. I'll try to post my thoughts/findings there as well.

- Robert



From soeren at apelpie.net  Tue Jul 14 16:55:38 2009
From: soeren at apelpie.net (Soeren Apel)
Date: Tue, 14 Jul 2009 16:55:38 +0200
Subject: [Cython] Troubles with "...is not a cimported module"
In-Reply-To: <684ED397-628E-4E20-BBFE-D173BD17C4D2@math.washington.edu>
References: <1247519976.7035.28.camel@doodie.home>
	<684ED397-628E-4E20-BBFE-D173BD17C4D2@math.washington.edu>
Message-ID: <1247583338.7035.39.camel@doodie.home>

On Mon, 2009-07-13 at 23:55 -0700, Robert Bradshaw wrote:
> On Jul 13, 2009, at 2:19 PM, Soeren Apel wrote:
> 
> > move_relative.pyx:10:23: 'move_relative' is not a cimported module
> >
> > I neither understand the meaning of the error message, nor does it  
> > give
> > any helpful clues as to how the problem can be resolved. With that,  
> > I'm
> > kind of stuck.
> 
> I'm not sure what that error means either. I could expect "'evas is  
> not a cimported module."

That's exactly what I was thinking, too. Since you and I both think this
I'm wondering if this is a bug. Should I file one?


> It looks like their directory structure is
> 
> /evas
>      evas.c_evas.pyx  # creates the module evas.c_evas, I'm really  
> tempted to depreciate the dotted-filename style of package naming

While you may of course go ahead and deprecate this, python-evas is
apparently on the way to solving this for python-evas:
http://trac.enlightenment.org/e/changeset/35233


> Somehow /include/evas is in your include path, rather than just / 
> include. You need to "cimport evas.c_evas as c_evas" (or "as evas" if  
> you want).

Thanks, fixing the include path along with the cimport is exactly what
solved my problem. Thanks a ton! :)

-Soeren



From stefan_ml at behnel.de  Tue Jul 14 18:32:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Jul 2009 18:32:56 +0200
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
	<682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
Message-ID: <4A5CB338.1000909@behnel.de>


Robert Bradshaw wrote:
> On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote:
> 
>> In the midst of GSoC, I wanted to float an idea about a Cython
>> profiler, similar to the current Python profiler.  Specifically I
>> wanted to see if there's interest, what sorts of features would be
>> desired, and, broadly, some implementation ideas.

+1, totally interested.


>> I think the arguments in favor are clear:
>>
>> Often people use Cython incrementally -- take a .py file, profile it,
>> put the hotspots in Cython.  If fast enough -- you're done.  If not,
>> continue to put more slow python code into Cython, or determine if the
>> Cython code should be massaged to yield better performance.  In the
>> latter case it is difficult to zone in on the Cython hotspot, for it
>> is not possible for the Python profiler to see it.  Other external
>> tools must be used (gprof?, valgrind, mac os x-specific profiler),
>> which is inconvenient at best; other times they are just plain
>> unavailable.

The main problem I always had here is the tool barrier. You can use
cProfile for profiling Python code, but it can't look into the Cython
functions. Profiling Cython itself will show you loads of time spent inside
the main parser function, without further detail. That's not quite the
right level of granularity for optimisations.

And you can use callgrind for C-level profiling, which works very well
thanks to kcachegrind, but is much too low-level for the Python stuff. So
all you see is the internal interpreter calls instead of the code you are
really looking at. And line-level profiling obviously means C-lines, not
Cython lines. Can be ok sometimes, but it's still more low-level than
necessary, and hard to follow in the code you write.

Having something in-between, maybe something that interfaces with cProfile
and Python's stats module would make things a lot easier. There's already a
conversion script that allows you to look at Python profiling dumps with
kcachegrind, so it might not even be too hard to make it work with both, so
that you could get profiling information at the Python, Cython and C level.

http://ddaa.net/blog/python/lsprof-calltree
http://oubiwann.blogspot.com/2006/08/python-and-kcachegrind.html
http://www.gnome.org/~johan/lsprofcalltree.py


>> I'd like to see a profiling option to the cython compiler that would
>> allow the user to, as seamlessly as possible, run the compiled
>> extension module with profiling enabled, and have the user interaction
>> be just like the current python profiler, specifically with meaningful
>> output for the Cython-level functions.  If possible, I'd like to have
>> the cython profiling code make use of the python cProfiler module so
>> profiling isn't any different between pure Python & Cython compiled
>> code.  I don't know how feasible hooking up with the python cProfiler
>> is, so it is just in the pie-in-the-sky stage right now.
>>
>> Another tantalizing possibility would be line-by-line profiling,
>> beyond the the regular function-level profiling.
> 
> After talking a bit about it at the last Sage days, and looking (a  
> tiny bit) into how this stuff works, this really is quite feasible,  
> and would be awesome to have.
> 
>> What other features would be desired?  What user interaction features
>> would be good to have?
>>
>> Implementation-wise, it could be done as its own transform, enabled by
>> compile-time switch.
> 
> Certainly we would want to enable this with a compile-time switch  
> (and, if it can be done not to verbosely, maybe even via macros).   
> I'm not sure transformations would be the approach to take here  
> (seems not the right level)--I'd probably add some stuff to the Code  
> object and maybe even latch onto the emit_marker for the line-by-line.

Absolutely. After all, profiling is not more than collecting information
like "I'm here, time is XYZ". We emit tons of error handling code that
contains line information, so this would just be one line more for each
line of Cython source, and likely similar to the code generated for the
ref-nanny.

I'd also certainly go for emit_marker() here (which simply needs to get
fixed wherever it's not accurate enough to mark each line). We need to make
sure that this only emits code inside of C function bodies, though.


>> Someone else (Robert? in some thread?) mentioned somewhere about
>> getting debugger-support in Cython, allowing one to step through
>> Cython code just like pdb (one can use gdb currently, of course).  I
>> imagine the implementation issues for a Cython debugger would overlap
>> a good deal with the profiler (it would be its own separate tranform,
>> obviously).
> 
> Yep. I don't have any idea how pdb works, but I'd imagine if we can  
> provide fake frames (the code is there for exceptions and locals())  
> it wouldn't be too hard to hook up. Making gdb friendlier is an  
> orthogonal but still very useful idea to pursue as well, especially  
> for code that interfaces with external libraries.

It would be good to look into the Python extensions to gdb first, I'd say.
Something like that might work well enough already. After all, when you
debug, you *must* see the C code, so that's different from profiling.

Stefan


From barthelemy at crans.org  Wed Jul 15 16:48:53 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Wed, 15 Jul 2009 16:48:53 +0200
Subject: [Cython] ImportError: ./pycdd.so: undefined symbol: __gmpq_init
In-Reply-To: <27D29499-6609-4A59-9AB6-880493B4A797@math.washington.edu>
References: <78f7ab620906300721s39b3b82fl659881c598614dbe@mail.gmail.com>
	<27D29499-6609-4A59-9AB6-880493B4A797@math.washington.edu>
Message-ID: <78f7ab620907150748p29fb824w414ec78485265de8@mail.gmail.com>

2009/6/30 Robert Bradshaw <robertwb at math.washington.edu>:
>> any idea of what might happen ?
>
> My guess would be that you're declaring something wrong (given the
> amount of warnings below).

your guess was right, I missed a DEFINE which was messing everything.

Thanks !

From kwmsmith at gmail.com  Thu Jul 16 05:37:54 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 15 Jul 2009 23:37:54 -0400
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
	<682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
Message-ID: <c266a7930907152037g46fc4ab4o87a17322cf483f61@mail.gmail.com>

On Tue, Jul 14, 2009 at 4:25 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote:
>
>> In the midst of GSoC, I wanted to float an idea about a Cython
>> profiler, similar to the current Python profiler. ?Specifically I
>> wanted to see if there's interest, what sorts of features would be
>> desired, and, broadly, some implementation ideas.
>>
>> I think the arguments in favor are clear:
>>
>> Often people use Cython incrementally -- take a .py file, profile it,
>> put the hotspots in Cython. ?If fast enough -- you're done. ?If not,
>> continue to put more slow python code into Cython, or determine if the
>> Cython code should be massaged to yield better performance. ?In the
>> latter case it is difficult to zone in on the Cython hotspot, for it
>> is not possible for the Python profiler to see it. ?Other external
>> tools must be used (gprof?, valgrind, mac os x-specific profiler),
>> which is inconvenient at best; other times they are just plain
>> unavailable.
>>
>> I'd like to see a profiling option to the cython compiler that would
>> allow the user to, as seamlessly as possible, run the compiled
>> extension module with profiling enabled, and have the user interaction
>> be just like the current python profiler, specifically with meaningful
>> output for the Cython-level functions. ?If possible, I'd like to have
>> the cython profiling code make use of the python cProfiler module so
>> profiling isn't any different between pure Python & Cython compiled
>> code. ?I don't know how feasible hooking up with the python cProfiler
>> is, so it is just in the pie-in-the-sky stage right now.
>>
>> Another tantalizing possibility would be line-by-line profiling,
>> beyond the the regular function-level profiling.
>
> After talking a bit about it at the last Sage days, and looking (a
> tiny bit) into how this stuff works, this really is quite feasible,
> and would be awesome to have.

Good!  I'm at a 2-week workshop currently; I'll hopefully get to
respond in more detail this weekend.  Here's what I have for now.

>
>> What other features would be desired? ?What user interaction features
>> would be good to have?
>>
>> Implementation-wise, it could be done as its own transform, enabled by
>> compile-time switch.
>
> Certainly we would want to enable this with a compile-time switch
> (and, if it can be done not to verbosely, maybe even via macros).
> I'm not sure transformations would be the approach to take here
> (seems not the right level)--I'd probably add some stuff to the Code
> object and maybe even latch onto the emit_marker for the line-by-line.
>

One reason I thought of a transform is to deal with other transforms
that might modify the python statements in such a way that makes it
difficult to reproduce the 'generated-C -> python-statement' mapping
at the code-generation phase.  Not having looked much at this part of
Cython (the Code object that is) maybe I'm way off here.

For, e.g. the 'with' transform, would it be easy to figure out all the
generated C code that corresponds to the original 'with' statement in
the Python source through the Code object (essentially at code
generation phase, right)?  Or is it mixed around enough that this
would be difficult to sort out?

My basic thinking is something like:

Have the profiling transform inject 'marker' lines at the necessary
level of detail (for function-level or line-by-line level profiling).
This would come fairly early in the pipeline before mangling
transforms take place.  The markers would contain all the necessary
info -- the source line text, location, etc.

At code generation phase, these marker lines would be turned into
profiling code (macros as you mention, or something else if too
complex).

>> Someone else (Robert? in some thread?) mentioned somewhere about
>> getting debugger-support in Cython, allowing one to step through
>> Cython code just like pdb (one can use gdb currently, of course). ?I
>> imagine the implementation issues for a Cython debugger would overlap
>> a good deal with the profiler (it would be its own separate tranform,
>> obviously).
>
> Yep. I don't have any idea how pdb works, but I'd imagine if we can
> provide fake frames (the code is there for exceptions and locals())
> it wouldn't be too hard to hook up. Making gdb friendlier is an
> orthogonal but still very useful idea to pursue as well, especially
> for code that interfaces with external libraries.
>

I misspoke somewhat -- I meant to say that one can do C-level
debugging with gdb right now if necessary, but we need a pdb-like
debugging for Cython-level source code.  Without looking under the
hood at pdb either, I think the fake frame idea is a good start.

>> I'll put each in a CEP, hopefully soon (and after the benefits of
>> comments, etc.)
>
> That would be good. I'll try to post my thoughts/findings there as well.
>

Great.  I won't be able to get to it anytime in the next 2 weeks, but
soon thereafter.

Thanks for the thoughts.

Kurt

From kwmsmith at gmail.com  Thu Jul 16 05:48:18 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 15 Jul 2009 23:48:18 -0400
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <4A5CB338.1000909@behnel.de>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
	<682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
	<4A5CB338.1000909@behnel.de>
Message-ID: <c266a7930907152048o517ac9c0wc271aa3b2704340@mail.gmail.com>

On Tue, Jul 14, 2009 at 12:32 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Robert Bradshaw wrote:
>> On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote:
>>
>>> In the midst of GSoC, I wanted to float an idea about a Cython
>>> profiler, similar to the current Python profiler. ?Specifically I
>>> wanted to see if there's interest, what sorts of features would be
>>> desired, and, broadly, some implementation ideas.
>
> +1, totally interested.
>
[snip]
>
> The main problem I always had here is the tool barrier. You can use
> cProfile for profiling Python code, but it can't look into the Cython
> functions. Profiling Cython itself will show you loads of time spent inside
> the main parser function, without further detail. That's not quite the
> right level of granularity for optimisations.

Yep.  My thinking too.

>
> And you can use callgrind for C-level profiling, which works very well
> thanks to kcachegrind, but is much too low-level for the Python stuff. So
> all you see is the internal interpreter calls instead of the code you are
> really looking at. And line-level profiling obviously means C-lines, not
> Cython lines. Can be ok sometimes, but it's still more low-level than
> necessary, and hard to follow in the code you write.
>
> Having something in-between, maybe something that interfaces with cProfile
> and Python's stats module would make things a lot easier. There's already a
> conversion script that allows you to look at Python profiling dumps with
> kcachegrind, so it might not even be too hard to make it work with both, so
> that you could get profiling information at the Python, Cython and C level.

This is good -- thanks for the links.  Once I get some time I'll take
a closer look.

>
> http://ddaa.net/blog/python/lsprof-calltree
> http://oubiwann.blogspot.com/2006/08/python-and-kcachegrind.html
> http://www.gnome.org/~johan/lsprofcalltree.py
>
>
[snip]
>
> Absolutely. After all, profiling is not more than collecting information
> like "I'm here, time is XYZ". We emit tons of error handling code that
> contains line information, so this would just be one line more for each
> line of Cython source, and likely similar to the code generated for the
> ref-nanny.
>
> I'd also certainly go for emit_marker() here (which simply needs to get
> fixed wherever it's not accurate enough to mark each line). We need to make
> sure that this only emits code inside of C function bodies, though.
>

Alright, so maybe my transform idea is going the way of the dodo.
That's fine, especially if it's too much for this task.

>
> It would be good to look into the Python extensions to gdb first, I'd say.
> Something like that might work well enough already. After all, when you
> debug, you *must* see the C code, so that's different from profiling.
>

I had no idea there were python extensions to gdb.  I've got a bit of
reading to do for the CEP.

Thanks,

Kurt

From ondrej at certik.cz  Thu Jul 16 06:48:03 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 15 Jul 2009 22:48:03 -0600
Subject: [Cython] problem with pure python mode
In-Reply-To: <D1FF3F5D-28B9-4174-A019-30A1E680138C@math.washington.edu>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
	<12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
	<85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>
	<7B1321FB-58A6-4CA3-927A-54FA609A0336@math.washington.edu>
	<85b5c3130906181829u330f95edw6339c719cd724671@mail.gmail.com>
	<85b5c3130907071233q6f797b78w12886f264580d0ce@mail.gmail.com>
	<D1FF3F5D-28B9-4174-A019-30A1E680138C@math.washington.edu>
Message-ID: <85b5c3130907152148uc60f748l24770da1c9aa0ffd@mail.gmail.com>

>> Let me know if there is any progress on this one. We are slowly
>> approaching the state with sympy, when I will need this.
>
> I was about to say "no haven't had a chance," but I figured I'd give
> it a quick glance right now, and fortunately it was a quick fix. I've
> pushed to cython-devel.

I checked this out and it works! Many thanks for that. This is very
exciting. I will start using it in sympy very soon:

http://groups.google.com/group/sympy/browse_thread/thread/325cc8b99e50895

Ondrej

From stefan_ml at behnel.de  Thu Jul 16 07:00:05 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 07:00:05 +0200
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <c266a7930907152037g46fc4ab4o87a17322cf483f61@mail.gmail.com>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>	<682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
	<c266a7930907152037g46fc4ab4o87a17322cf483f61@mail.gmail.com>
Message-ID: <4A5EB3D5.5090403@behnel.de>


Kurt Smith wrote:
> One reason I thought of a transform is to deal with other transforms
> that might modify the python statements in such a way that makes it
> difficult to reproduce the 'generated-C -> python-statement' mapping
> at the code-generation phase.  Not having looked much at this part of
> Cython (the Code object that is) maybe I'm way off here.
> 
> For, e.g. the 'with' transform, would it be easy to figure out all the
> generated C code that corresponds to the original 'with' statement in
> the Python source through the Code object (essentially at code
> generation phase, right)?  Or is it mixed around enough that this
> would be difficult to sort out?

At least in hand-written transforms, you can easily assign original code
positions to newly created nodes, usually based on what original code
triggered their creation. So you wouldn't loose any positional information.

Given that the with-transform basically transforms one line of code, the
granularity you'd get wouldn't be great, but you wouldn't loose any code
line information either.

Stefan


From stefan_ml at behnel.de  Thu Jul 16 07:03:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 07:03:17 +0200
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
Message-ID: <4A5EB495.7020704@behnel.de>

[forwarding to the list]

Date: Thu, 16 Jul 2009 00:27:27 -0400
From: Chuck Blake <cb at pdos.cail.mit.edu>

Kurt Smith wrote:
>Have the profiling transform inject 'marker' lines at the necessary
>level of detail (for function-level or line-by-line level profiling).
>This would come fairly early in the pipeline before mangling
>transforms take place.  The markers would contain all the necessary
>info -- the source line text, location, etc.
>
>At code generation phase, these marker lines would be turned into
>profiling code (macros as you mention, or something else if too
>complex).

Your proposal seems more about the compile-time instrumentation style
of profiling by exact counting.

If you are willing to accept or begin with a statistical profile as
guidance, I have a solution that works, at least under Linux:
    http://pdos.csail.mit.edu/~cblake/pct/index.html

Basically, you just download and install pct (make && sudo make install)
and compile your Cython modules with --line-directives.  Then you can go
    PCT_FMT=line% profile python myscript.py
to get a line-by-line report.

If you don't do the --line-directives, the line numbers will refer to
the generated .c files.

You can also get symbol-level profiling and a variety of other reports.

Some ambitious soul could perhaps extend the HTML annotation of
Pythonic-ness to include CPU slowness to really highlight areas
where careful attention might pay off a lot.

cb

From robertwb at math.washington.edu  Thu Jul 16 11:06:33 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Jul 2009 02:06:33 -0700
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <c266a7930907152048o517ac9c0wc271aa3b2704340@mail.gmail.com>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
	<682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
	<4A5CB338.1000909@behnel.de>
	<c266a7930907152048o517ac9c0wc271aa3b2704340@mail.gmail.com>
Message-ID: <C7BBD467-A4F8-43F0-9BF1-0AFD55D31059@math.washington.edu>

On Jul 15, 2009, at 8:48 PM, Kurt Smith wrote:

>> Absolutely. After all, profiling is not more than collecting  
>> information
>> like "I'm here, time is XYZ". We emit tons of error handling code  
>> that
>> contains line information, so this would just be one line more for  
>> each
>> line of Cython source, and likely similar to the code generated  
>> for the
>> ref-nanny.
>>
>> I'd also certainly go for emit_marker() here (which simply needs  
>> to get
>> fixed wherever it's not accurate enough to mark each line). We  
>> need to make
>> sure that this only emits code inside of C function bodies, though.
>>
>
> Alright, so maybe my transform idea is going the way of the dodo.
> That's fine, especially if it's too much for this task.

A transform would be hard because it has no concept of "once per  
line" and one would probably like to add the profiling information at  
the very top and bottom of an function (e.g. before arguments are  
parsed, though after acquiring the GIL).

>> It would be good to look into the Python extensions to gdb first,  
>> I'd say.
>> Something like that might work well enough already. After all,  
>> when you
>> debug, you *must* see the C code, so that's different from profiling.

For many debugging situations, viewing the C code is no more  
necessary than viewing the Python bytecodes for CPython. Of course,  
if you're doing C-level stuff, sometimes you need to, and for  
debugging Cython itself you have to look at the source.

- Robert

From robertwb at math.washington.edu  Thu Jul 16 11:07:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 16 Jul 2009 02:07:51 -0700
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <4A5EB495.7020704@behnel.de>
References: <4A5EB495.7020704@behnel.de>
Message-ID: <8E011FB4-477A-495F-887A-1C051A3AE2C9@math.washington.edu>

On Jul 15, 2009, at 10:03 PM, Stefan Behnel wrote:

> [forwarding to the list]
>
> Date: Thu, 16 Jul 2009 00:27:27 -0400
> From: Chuck Blake <cb at pdos.cail.mit.edu>
>
> Kurt Smith wrote:
>> Have the profiling transform inject 'marker' lines at the necessary
>> level of detail (for function-level or line-by-line level profiling).
>> This would come fairly early in the pipeline before mangling
>> transforms take place.  The markers would contain all the necessary
>> info -- the source line text, location, etc.
>>
>> At code generation phase, these marker lines would be turned into
>> profiling code (macros as you mention, or something else if too
>> complex).
>
> Your proposal seems more about the compile-time instrumentation style
> of profiling by exact counting.
>
> If you are willing to accept or begin with a statistical profile as
> guidance, I have a solution that works, at least under Linux:
>     http://pdos.csail.mit.edu/~cblake/pct/index.html
>
> Basically, you just download and install pct (make && sudo make  
> install)
> and compile your Cython modules with --line-directives.  Then you  
> can go
>     PCT_FMT=line% profile python myscript.py
> to get a line-by-line report.
>
> If you don't do the --line-directives, the line numbers will refer to
> the generated .c files.
>
> You can also get symbol-level profiling and a variety of other  
> reports.

One issue that's hard to solve this way is that if you're going back  
and forth between Cython and Python, the Python interpreter calls add  
a lot of noise to the output.

> Some ambitious soul could perhaps extend the HTML annotation of
> Pythonic-ness to include CPU slowness to really highlight areas
> where careful attention might pay off a lot.

That would be cool.

- Robert



From dagss at student.matnat.uio.no  Thu Jul 16 15:54:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 16 Jul 2009 15:54:10 +0200
Subject: [Cython] replacing LiE error() with a python exception
Message-ID: <d0b42236164ef3933c6331fe91bb06ac.squirrel@webmail.uio.no>

(I composed this earlier but there was an SMTP error. Resending in case it
is still relevant.)

(On that note I'm only semi-available the next 12 days or so.)

David Simmons-Duffin wrote:
> I'm still working on wrapping the computer algebra package LiE in
cython.  For those unfamiliar, LiE is basically it's own language,
written in C.  And I'm interested in calling some of the math-heavy
functions, but bypassing the interpreter.
> Error checking is sprinkled throughout the LiE source, for instance:
matrix *mat_mul_mat_mat(matrix *a, matrix *b) {
>      if (a->ncols != b->nrows)
>          error("Number columns arg1 unequal number of rows arg2 .\n");
>      return Matmult(a, b);
> }
> When an error is found, LiE calls the error() function, which prints out
a traceback, cleans up a little bit, and finally does a longjump to
"envbuf," which is at the beginning of the interpreter loop:
> void error(char *format, ...)
> {
>      symblst         list;
>      extern jmp_buf  envbuf;
>      va_list ap;
>      va_start(ap, format);
>      vfprintf(stderr, format, ap);
>      if (am_monitor) vfprintf(monfile, format, ap);
>      if (label->name)
>      {
>           boolean printin = (label->name != seq_name);
>           if (printin)
>              Printf("(in %s",name_tab[label->name]);
>           if (no_terminal(cur_in) || strcmp(label->fname, "stdin")!=0) {
>              if (!printin) Printf("(");
>              Printf(" at line %d of file %s)\n",label->line, label-
>  >fname);
>           }
>           else
>              if (printin) Printf(")\n");
>      }
>      if (fun_name)
>          Printf("[in function %s ]\n", name_tab[fun_name]);
>      for (list = top_definitions; list!=NULL; list = list->next)
>          /* Recover symbol table */
>      {   if (list->class == FUNCTION_COPIED) list->class = FUNCTION;
> 	if (list->next && list->next->class == DUMMY)
>              list->next = list->next->next;      /* Remove sym */
>      }
>      if (repair_obj) {
>          setshared(repair_obj);
>          repair_obj = (object) NULL;
>      }
>      if (cur_in==stdin)
>        clear_input();
>      else
>        do exit_input_file(parsing); while (cur_in!=stdin); /* pop
> input files */
>      longjmp(envbuf, -1);
> }
> Instead of doing all this stuff, and longjumping to the LiE
> interpreter, I'd like to raise a python exception that could be caught
by whatever cython function called "mat_mul_mat_mat."  Does anyone have
any advice on how to do this?  I took a look at the python

I think you should avoid using jumps since you will not call the functions
from the interpreter loop but instead call them directly; so there's no
canonical place to jump back to.

There are two ways of achieving this:

I) The usual CPython way of doing this is to have a seperate return path.
So you need to do

matrix *mat_mul_mat_mat(matrix *a, matrix *b) {
     if (a->ncols != b->nrows) {
         error("Number columns arg1 unequal number of rows arg2 .\n");
return NULL;
     }
     return Matmult(a, b);
}

The problem is when a function calls another function which may call error
-- then you must start to insert checks like this:

if ((ret = mat_mul_mat_mat(...)) == NULL) return NULL;
... use ret ...

>From the error function using a jump it looks like memory is never
dynamically allocated using malloc or similar? Anyway, if it is, then such
memory must be freed also during error-returns.

You must then declare the functions in Cython with an "except NULL"
clause. See docs.cython.org on exceptions.

II) Potentially easier: Try to compile the library using C++. Then error
can raise a C++ exception which frees you of having to check return values
(but memory allocated using malloc must still be dealt with, through using
try/catch clauses or std::auto_ptr). Using "except +" Cython-side, Cython
can trap C++ exceptions and convert them to Python exceptions.

> exception portion of the C/API.  It seems like I might want to make a C
file with my own error function:
> error(char* format, ...) {
>      turn format into a string
>      PyErr_SetString(errorstring)

Correct, PyErr_Format(...) might be easier in some contexts but I suppose
here the error string is already assembled.

Dag Sverre






From davidnovakovic at gmail.com  Mon Jul 20 07:17:11 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Mon, 20 Jul 2009 15:17:11 +1000
Subject: [Cython] creating structs from extern declaractions
Message-ID: <59d13e7d0907192217xa4ced9fied11ed56b143666@mail.gmail.com>

Hey,


i'm trying to create an instance of a struct from an external header:

cdef exern from "api.h":
    ctypedef struct foo_config:
        int member_1
        ...


cdef class Session:
    foo_config config

    def __init__(self):
        self.config = foo_config()




Also, I need to create a void pointer for a struct and pass that into the c
library so it can create an instance of a struct in the memory, I can't see
a way of doing this.. when i try and assign None to an uninitialised
variable the cython compiler complains about a type mismatch.

Any help would be greatly appreciated :)


David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090720/d845dcab/attachment.htm 

From davidnovakovic at gmail.com  Mon Jul 20 07:25:25 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Mon, 20 Jul 2009 15:25:25 +1000
Subject: [Cython] creating structs from extern declaractions
In-Reply-To: <59d13e7d0907192217xa4ced9fied11ed56b143666@mail.gmail.com>
References: <59d13e7d0907192217xa4ced9fied11ed56b143666@mail.gmail.com>
Message-ID: <59d13e7d0907192225p7704f2afx29b35bc6fd8d480b@mail.gmail.com>

I found http://ldots.org/pyrex-guide/ which alerted me to being able to use
NULL ... still trying to figure out how to create an instance of the
struct...

On Mon, Jul 20, 2009 at 3:17 PM, David P. Novakovic <
davidnovakovic at gmail.com> wrote:

> Hey,
>
>
> i'm trying to create an instance of a struct from an external header:
>
> cdef exern from "api.h":
>     ctypedef struct foo_config:
>         int member_1
>         ...
>
>
> cdef class Session:
>     foo_config config
>
>     def __init__(self):
>         self.config = foo_config()
>
>
>
>
> Also, I need to create a void pointer for a struct and pass that into the c
> library so it can create an instance of a struct in the memory, I can't see
> a way of doing this.. when i try and assign None to an uninitialised
> variable the cython compiler complains about a type mismatch.
>
> Any help would be greatly appreciated :)
>
>
> David
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090720/e20e8654/attachment.htm 

From davidnovakovic at gmail.com  Mon Jul 20 07:50:08 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Mon, 20 Jul 2009 15:50:08 +1000
Subject: [Cython] creating structs from extern declaractions
In-Reply-To: <59d13e7d0907192225p7704f2afx29b35bc6fd8d480b@mail.gmail.com>
References: <59d13e7d0907192217xa4ced9fied11ed56b143666@mail.gmail.com>
	<59d13e7d0907192225p7704f2afx29b35bc6fd8d480b@mail.gmail.com>
Message-ID: <59d13e7d0907192250h3b7e24efw885541f471160786@mail.gmail.com>

oh yes, I feel like an idiot... for anyone as idiotic as me for who finds
this post, simply declaring the variable "instantiates it" just like in C...
:)

On Mon, Jul 20, 2009 at 3:25 PM, David P. Novakovic <
davidnovakovic at gmail.com> wrote:

> I found http://ldots.org/pyrex-guide/ which alerted me to being able to
> use NULL ... still trying to figure out how to create an instance of the
> struct...
>
>
> On Mon, Jul 20, 2009 at 3:17 PM, David P. Novakovic <
> davidnovakovic at gmail.com> wrote:
>
>> Hey,
>>
>>
>> i'm trying to create an instance of a struct from an external header:
>>
>> cdef exern from "api.h":
>>     ctypedef struct foo_config:
>>         int member_1
>>         ...
>>
>>
>> cdef class Session:
>>     foo_config config
>>
>>     def __init__(self):
>>         self.config = foo_config()
>>
>>
>>
>>
>> Also, I need to create a void pointer for a struct and pass that into the
>> c library so it can create an instance of a struct in the memory, I can't
>> see a way of doing this.. when i try and assign None to an uninitialised
>> variable the cython compiler complains about a type mismatch.
>>
>> Any help would be greatly appreciated :)
>>
>>
>> David
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090720/c305b45d/attachment.htm 

From robertwb at math.washington.edu  Sat Jul 18 23:57:31 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 18 Jul 2009 14:57:31 -0700
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <4A5CB338.1000909@behnel.de>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
	<682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
	<4A5CB338.1000909@behnel.de>
Message-ID: <86C394EA-E7D3-4DF6-9DB0-5377E707AAD2@math.washington.edu>

On Jul 14, 2009, at 9:32 AM, Stefan Behnel wrote:

>
> Robert Bradshaw wrote:
>> On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote:
>>
>>> In the midst of GSoC, I wanted to float an idea about a Cython
>>> profiler, similar to the current Python profiler.  Specifically I
>>> wanted to see if there's interest, what sorts of features would be
>>> desired, and, broadly, some implementation ideas.
>
> +1, totally interested.
>
>
>>> I think the arguments in favor are clear:
>>>
>>> Often people use Cython incrementally -- take a .py file, profile  
>>> it,
>>> put the hotspots in Cython.  If fast enough -- you're done.  If not,
>>> continue to put more slow python code into Cython, or determine  
>>> if the
>>> Cython code should be massaged to yield better performance.  In the
>>> latter case it is difficult to zone in on the Cython hotspot, for it
>>> is not possible for the Python profiler to see it.  Other external
>>> tools must be used (gprof?, valgrind, mac os x-specific profiler),
>>> which is inconvenient at best; other times they are just plain
>>> unavailable.
>
> The main problem I always had here is the tool barrier. You can use
> cProfile for profiling Python code, but it can't look into the Cython
> functions. Profiling Cython itself will show you loads of time  
> spent inside
> the main parser function, without further detail. That's not quite the
> right level of granularity for optimisations.

So, I've felt like hacking Cython on my trans-continental flight, and  
got Cython working very nice with cProfile. Of course there's a lot  
of tracing and other things one could want to do from here (e.g. pdb)  
but I don't think I'll have time for that.

I'll push as soon as I have internet access again.

- Robert


From evlutte at gmail.com  Mon Jul 20 16:32:09 2009
From: evlutte at gmail.com (Ethan Van Andel)
Date: Mon, 20 Jul 2009 10:32:09 -0400
Subject: [Cython] incompatible types (complex)
Message-ID: <f67de9dc0907200732h403e3f97ld35543982be11f99@mail.gmail.com>

when I compile some code containing this snippet: (np = numpy)

COMPLEX = np.complex128
ctypedef np.complex128_t COMPLEX_T

class Riemann_Map():
    def __init__(self,fs,fprimes,a1,N,int ncorners = 4,opp = False):
        cdef COMPLEX_T a = np.complex128(a1)

I get these errors (among others):
c:1316: error: incompatible types in assignment
and
c:1316: error: previous implicit declaration of
?__pyx_PyObject_As___pyx_t_double_complex? was here

where c:1316 is the line corresponding to, cdef COMPLEX_T a ...

I am compiling this using Cython 0.11.2.beta in the sage -b environment.

What am I doing wrong?

Thanks for any help you can give,

Ethan Van Andel

From kwmsmith at gmail.com  Tue Jul 21 03:39:09 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 20 Jul 2009 21:39:09 -0400
Subject: [Cython] Cython-specific profiler (& debugger) ideas...
In-Reply-To: <86C394EA-E7D3-4DF6-9DB0-5377E707AAD2@math.washington.edu>
References: <c266a7930907122024l112c1613ycc0690277149856f@mail.gmail.com>
	<682B3834-6AEE-4761-84C9-90E4C151B90C@math.washington.edu>
	<4A5CB338.1000909@behnel.de>
	<86C394EA-E7D3-4DF6-9DB0-5377E707AAD2@math.washington.edu>
Message-ID: <c266a7930907201839s3bf283cbw9c3da6008ef4fd2c@mail.gmail.com>

On Sat, Jul 18, 2009 at 5:57 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jul 14, 2009, at 9:32 AM, Stefan Behnel wrote:
>
>>
>> Robert Bradshaw wrote:
>>> On Jul 12, 2009, at 8:24 PM, Kurt Smith wrote:
>>>
>>>> In the midst of GSoC, I wanted to float an idea about a Cython
>>>> profiler, similar to the current Python profiler. ?Specifically I
>>>> wanted to see if there's interest, what sorts of features would be
>>>> desired, and, broadly, some implementation ideas.
>>
>> +1, totally interested.
>>
>>
>>>> I think the arguments in favor are clear:
>>>>
>>>> Often people use Cython incrementally -- take a .py file, profile
>>>> it,
>>>> put the hotspots in Cython. ?If fast enough -- you're done. ?If not,
>>>> continue to put more slow python code into Cython, or determine
>>>> if the
>>>> Cython code should be massaged to yield better performance. ?In the
>>>> latter case it is difficult to zone in on the Cython hotspot, for it
>>>> is not possible for the Python profiler to see it. ?Other external
>>>> tools must be used (gprof?, valgrind, mac os x-specific profiler),
>>>> which is inconvenient at best; other times they are just plain
>>>> unavailable.
>>
>> The main problem I always had here is the tool barrier. You can use
>> cProfile for profiling Python code, but it can't look into the Cython
>> functions. Profiling Cython itself will show you loads of time
>> spent inside
>> the main parser function, without further detail. That's not quite the
>> right level of granularity for optimisations.
>
> So, I've felt like hacking Cython on my trans-continental flight, and
> got Cython working very nice with cProfile. Of course there's a lot
> of tracing and other things one could want to do from here (e.g. pdb)
> but I don't think I'll have time for that.

Excellent!  I look forward to seeing it in action.

>
> I'll push as soon as I have internet access again.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ndbecker2 at gmail.com  Tue Jul 21 17:44:53 2009
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 21 Jul 2009 11:44:53 -0400
Subject: [Cython] expy
Message-ID: <h44npl$p7m$1@ger.gmane.org>

This is an interesting approach:

http://expy.sourceforge.net/



From robertwb at math.washington.edu  Tue Jul 21 18:18:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 21 Jul 2009 09:18:45 -0700
Subject: [Cython] creating structs from extern declaractions
In-Reply-To: <59d13e7d0907192250h3b7e24efw885541f471160786@mail.gmail.com>
References: <59d13e7d0907192217xa4ced9fied11ed56b143666@mail.gmail.com>
	<59d13e7d0907192225p7704f2afx29b35bc6fd8d480b@mail.gmail.com>
	<59d13e7d0907192250h3b7e24efw885541f471160786@mail.gmail.com>
Message-ID: <C0620EF9-C3D7-4F48-B7DC-E40F3DC9FDE4@math.washington.edu>

Glad you were able to figure everything out. Also, of note, malloc  
and free are used just as in C to allocate memory (heap-allocated  
structs, pointers, etc.)

- Robert

On Jul 19, 2009, at 10:50 PM, David P. Novakovic wrote:

> oh yes, I feel like an idiot... for anyone as idiotic as me for who  
> finds this post, simply declaring the variable "instantiates it"  
> just like in C... :)
>
> On Mon, Jul 20, 2009 at 3:25 PM, David P. Novakovic  
> <davidnovakovic at gmail.com> wrote:
> I found http://ldots.org/pyrex-guide/ which alerted me to being  
> able to use NULL ... still trying to figure out how to create an  
> instance of the struct...
>
>
> On Mon, Jul 20, 2009 at 3:17 PM, David P. Novakovic  
> <davidnovakovic at gmail.com> wrote:
> Hey,
>
>
> i'm trying to create an instance of a struct from an external header:
>
> cdef exern from "api.h":
>     ctypedef struct foo_config:
>         int member_1
>         ...
>
>
> cdef class Session:
>     foo_config config
>
>     def __init__(self):
>         self.config = foo_config()
>
>
>
>
> Also, I need to create a void pointer for a struct and pass that  
> into the c library so it can create an instance of a struct in the  
> memory, I can't see a way of doing this.. when i try and assign  
> None to an uninitialised variable the cython compiler complains  
> about a type mismatch.
>
> Any help would be greatly appreciated :)
>
>
> David
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Tue Jul 21 21:34:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 21 Jul 2009 21:34:23 +0200
Subject: [Cython] expy
In-Reply-To: <h44npl$p7m$1@ger.gmane.org>
References: <h44npl$p7m$1@ger.gmane.org>
Message-ID: <4A66183F.9020101@behnel.de>


Neal Becker wrote:
> This is an interesting approach:
> 
> http://expy.sourceforge.net/

Well, I read about it, too, as it was advertised on python-dev (mistakenly,
I assume). It's similar to Cython in that it allows you to write Python
function glue code as Python funtions. However, that only handles the plain
call code. Having to put everything else into plain strings sounds pretty
awkward to me. It may be ok for thin wrappers, but I doubt that it's usable
to write real code at the wrapper layer.

Stefan

From vel.accel at gmail.com  Wed Jul 22 07:47:51 2009
From: vel.accel at gmail.com (Peter Alexander)
Date: Wed, 22 Jul 2009 01:47:51 -0400
Subject: [Cython] global cdef not recognized locally
Message-ID: <1e52e0880907212247q56734292v26b328c3911e3597@mail.gmail.com>

Hi all,

I'm getting a cython build[1] error[2] from the following code snippet[3].
A test file[4] using a similar implementation (stack memory only) built fine.
Could you help me see what I'm doing wrong here.
I'm using repo: cython-devel changeset: 2094:bcc850a1253a date: Mon
Jul 20 15:40:26 2009 -0700
Thanks ~Pete

[1] #######  build.sh script ##########

cython -a -I/home/travlr/src/numpy.gitsvn/doc/cython/ "$1"

gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -std=c99\
    -I/usr/include/python2.6 -o ${1%\.*}.so ${1%\.*}.c

######## end build.sh script ########

[2] ######## build error message ########

$ ./build utils.pyx

Error converting Pyrex file to C:
------------------------------------------------------------
    void* mmalloc(size_t size, size_t n_items):
        cdef:
            MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof(MemPoolItem))
        mp_cur.memory = malloc(size * n_items)
        mp_cur.next = mp_head
                            ^
------------------------------------------------------------
/home/travlr/py/cy/utils.pyx:14:29: Cannot convert Python object to
'MemPoolItem *'

[...]

######## end build error message ###########

[3] ######## code snippet #########

from defs cimport *

cdef:
    struct MemPoolItem:
        void*           memory
        MemPoolItem*    next

    MemPoolItem* mp_head=NULL

    void* mmalloc(size_t size, size_t n_items):
        cdef:
            MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof(MemPoolItem))
        mp_cur.memory = malloc(size * n_items)
        mp_cur.next = mp_head
        mp_head = mp_cur
        return mp_cur.memory

    mfree():
        cdef:
            MemPoolItem* mp_cur = mpi_head
            MemPoolItem* mp_tmp = NULL
        while mp_cur:
            free(mp_cur.memory)
            mp_cur.memory = NULL
            mp_tmp = mpi_cur.next
            free(mpi_cur)
            mp_cur = mpi_tmp
        mp_head = NULL

######## end code snippet #########

[4] ######## test file #########

cdef:
    struct S_t:
        int val
        S_t* next

    S_t s

    _run():
        s.val = 345
        print s.val

def run():
    _run()

########### end test file ############

From robertwb at math.washington.edu  Wed Jul 22 12:38:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 22 Jul 2009 03:38:46 -0700
Subject: [Cython] global cdef not recognized locally
In-Reply-To: <1e52e0880907212247q56734292v26b328c3911e3597@mail.gmail.com>
References: <1e52e0880907212247q56734292v26b328c3911e3597@mail.gmail.com>
Message-ID: <73FB403A-14D8-44DC-9C16-991AA904D3C8@math.washington.edu>

On Jul 21, 2009, at 10:47 PM, Peter Alexander wrote:

> Hi all,
>
> I'm getting a cython build[1] error[2] from the following code  
> snippet[3].
> A test file[4] using a similar implementation (stack memory only)  
> built fine.
> Could you help me see what I'm doing wrong here.

Cython variable semantics are just like they are in Python,  
specifically, note that you are assigning to mp_head in that function.

mp_head = mp_cur

This makes mp_head into a local variable (and, since you haven't  
specified a type, of type object). Just like Python, if you want to  
assign to a global variable, you need to put "global mp_head" in your  
function body.

To make this error more clear, it would be a good idea to emit a  
warning that mp_head is being used before it is defined.

- Robert

> I'm using repo: cython-devel changeset: 2094:bcc850a1253a date: Mon
> Jul 20 15:40:26 2009 -0700
> Thanks ~Pete
>
> [1] #######  build.sh script ##########
>
> cython -a -I/home/travlr/src/numpy.gitsvn/doc/cython/ "$1"
>
> gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing - 
> std=c99\
>     -I/usr/include/python2.6 -o ${1%\.*}.so ${1%\.*}.c
>
> ######## end build.sh script ########
>
> [2] ######## build error message ########
>
> $ ./build utils.pyx
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
>     void* mmalloc(size_t size, size_t n_items):
>         cdef:
>             MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof 
> (MemPoolItem))
>         mp_cur.memory = malloc(size * n_items)
>         mp_cur.next = mp_head
>                             ^
> ------------------------------------------------------------
> /home/travlr/py/cy/utils.pyx:14:29: Cannot convert Python object to
> 'MemPoolItem *'
>
> [...]
>
> ######## end build error message ###########
>
> [3] ######## code snippet #########
>
> from defs cimport *
>
> cdef:
>     struct MemPoolItem:
>         void*           memory
>         MemPoolItem*    next
>
>     MemPoolItem* mp_head=NULL
>
>     void* mmalloc(size_t size, size_t n_items):
>         cdef:
>             MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof 
> (MemPoolItem))
>         mp_cur.memory = malloc(size * n_items)
>         mp_cur.next = mp_head
>         mp_head = mp_cur
>         return mp_cur.memory
>
>     mfree():
>         cdef:
>             MemPoolItem* mp_cur = mpi_head
>             MemPoolItem* mp_tmp = NULL
>         while mp_cur:
>             free(mp_cur.memory)
>             mp_cur.memory = NULL
>             mp_tmp = mpi_cur.next
>             free(mpi_cur)
>             mp_cur = mpi_tmp
>         mp_head = NULL
>
> ######## end code snippet #########
>
> [4] ######## test file #########
>
> cdef:
>     struct S_t:
>         int val
>         S_t* next
>
>     S_t s
>
>     _run():
>         s.val = 345
>         print s.val
>
> def run():
>     _run()
>
> ########### end test file ############
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From vel.accel at gmail.com  Wed Jul 22 13:28:57 2009
From: vel.accel at gmail.com (Peter Alexander)
Date: Wed, 22 Jul 2009 07:28:57 -0400
Subject: [Cython] global cdef not recognized locally
In-Reply-To: <73FB403A-14D8-44DC-9C16-991AA904D3C8@math.washington.edu>
References: <1e52e0880907212247q56734292v26b328c3911e3597@mail.gmail.com>
	<73FB403A-14D8-44DC-9C16-991AA904D3C8@math.washington.edu>
Message-ID: <1e52e0880907220428v14d6fab6sf096f7177585741c@mail.gmail.com>

On Wed, Jul 22, 2009 at 6:38 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jul 21, 2009, at 10:47 PM, Peter Alexander wrote:
>
>> Hi all,
>>
>> I'm getting a cython build[1] error[2] from the following code
>> snippet[3].
>> A test file[4] using a similar implementation (stack memory only)
>> built fine.
>> Could you help me see what I'm doing wrong here.
>
> Cython variable semantics are just like they are in Python,
> specifically, note that you are assigning to mp_head in that function.
>
> mp_head = mp_cur
>
> This makes mp_head into a local variable (and, since you haven't
> specified a type, of type object). Just like Python, if you want to
> assign to a global variable, you need to put "global mp_head" in your
> function body.
>
> To make this error more clear, it would be a good idea to emit a
> warning that mp_head is being used before it is defined.
>
> - Robert
>
>> I'm using repo: cython-devel changeset: 2094:bcc850a1253a date: Mon
>> Jul 20 15:40:26 2009 -0700
>> Thanks ~Pete
>>
>> [1] ####### ?build.sh script ##########
>>
>> cython -a -I/home/travlr/src/numpy.gitsvn/doc/cython/ "$1"
>>
>> gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -
>> std=c99\
>> ? ? -I/usr/include/python2.6 -o ${1%\.*}.so ${1%\.*}.c
>>
>> ######## end build.sh script ########
>>
>> [2] ######## build error message ########
>>
>> $ ./build utils.pyx
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ? ? void* mmalloc(size_t size, size_t n_items):
>> ? ? ? ? cdef:
>> ? ? ? ? ? ? MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof
>> (MemPoolItem))
>> ? ? ? ? mp_cur.memory = malloc(size * n_items)
>> ? ? ? ? mp_cur.next = mp_head
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
>> ------------------------------------------------------------
>> /home/travlr/py/cy/utils.pyx:14:29: Cannot convert Python object to
>> 'MemPoolItem *'
>>
>> [...]
>>
>> ######## end build error message ###########
>>
>> [3] ######## code snippet #########
>>
>> from defs cimport *
>>
>> cdef:
>> ? ? struct MemPoolItem:
>> ? ? ? ? void* ? ? ? ? ? memory
>> ? ? ? ? MemPoolItem* ? ?next
>>
>> ? ? MemPoolItem* mp_head=NULL
>>
>> ? ? void* mmalloc(size_t size, size_t n_items):
>> ? ? ? ? cdef:
>> ? ? ? ? ? ? MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof
>> (MemPoolItem))
>> ? ? ? ? mp_cur.memory = malloc(size * n_items)
>> ? ? ? ? mp_cur.next = mp_head
>> ? ? ? ? mp_head = mp_cur
>> ? ? ? ? return mp_cur.memory
>>
>> ? ? mfree():
>> ? ? ? ? cdef:
>> ? ? ? ? ? ? MemPoolItem* mp_cur = mpi_head
>> ? ? ? ? ? ? MemPoolItem* mp_tmp = NULL
>> ? ? ? ? while mp_cur:
>> ? ? ? ? ? ? free(mp_cur.memory)
>> ? ? ? ? ? ? mp_cur.memory = NULL
>> ? ? ? ? ? ? mp_tmp = mpi_cur.next
>> ? ? ? ? ? ? free(mpi_cur)
>> ? ? ? ? ? ? mp_cur = mpi_tmp
>> ? ? ? ? mp_head = NULL
>>
>> ######## end code snippet #########
>>
>> [4] ######## test file #########
>>
>> cdef:
>> ? ? struct S_t:
>> ? ? ? ? int val
>> ? ? ? ? S_t* next
>>
>> ? ? S_t s
>>
>> ? ? _run():
>> ? ? ? ? s.val = 345
>> ? ? ? ? print s.val
>>
>> def run():
>> ? ? _run()
>>
>> ########### end test file ############
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Hi Robert,

I'm certainly confused by the semantics. Also, I can't find the use of
'global' documented anywhere yet.
1) I'm not sure why the "test" file built ok while referencing the
varible w/ out declaring it to be "global".
2) Placing "global mp_head" as the first line of the function body in
both functions gives a different error(s):

Thank you,

################################
$ ./build utils.pyx

Error converting Pyrex file to C:
------------------------------------------------------------
        return mp_cur.memory

    mfree():
        global mp_head
        cdef:
            MemPoolItem* mp_cur = mpi_head
                                         ^
------------------------------------------------------------
/home/travlr/py/cy/utils.pyx:22:42: undeclared name not builtin: mpi_head

Error converting Pyrex file to C:
------------------------------------------------------------
            MemPoolItem* mp_cur = mpi_head
            MemPoolItem* mp_tmp = NULL
        while mp_cur:
            free(mp_cur.memory)
            mp_cur.memory = NULL
            mp_tmp = mpi_cur.next
                           ^
------------------------------------------------------------
/home/travlr/py/cy/utils.pyx:27:28: undeclared name not builtin: mpi_cur

Error converting Pyrex file to C:
------------------------------------------------------------
            MemPoolItem* mp_tmp = NULL
        while mp_cur:
            free(mp_cur.memory)
            mp_cur.memory = NULL
            mp_tmp = mpi_cur.next
            free(mpi_cur)
                       ^
------------------------------------------------------------
/home/travlr/py/cy/utils.pyx:28:24: undeclared name not builtin: mpi_cur

Error converting Pyrex file to C:
------------------------------------------------------------
        while mp_cur:
            free(mp_cur.memory)
            mp_cur.memory = NULL
            mp_tmp = mpi_cur.next
            free(mpi_cur)
            mp_cur = mpi_tmp
                           ^
------------------------------------------------------------

From vel.accel at gmail.com  Wed Jul 22 13:36:59 2009
From: vel.accel at gmail.com (Peter Alexander)
Date: Wed, 22 Jul 2009 07:36:59 -0400
Subject: [Cython] global cdef not recognized locally
In-Reply-To: <1e52e0880907220428v14d6fab6sf096f7177585741c@mail.gmail.com>
References: <1e52e0880907212247q56734292v26b328c3911e3597@mail.gmail.com>
	<73FB403A-14D8-44DC-9C16-991AA904D3C8@math.washington.edu>
	<1e52e0880907220428v14d6fab6sf096f7177585741c@mail.gmail.com>
Message-ID: <1e52e0880907220436r620d28f9r85dcf23f46103df3@mail.gmail.com>

On Wed, Jul 22, 2009 at 7:28 AM, Peter Alexander<vel.accel at gmail.com> wrote:
> On Wed, Jul 22, 2009 at 6:38 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On Jul 21, 2009, at 10:47 PM, Peter Alexander wrote:
>>
>>> Hi all,
>>>
>>> I'm getting a cython build[1] error[2] from the following code
>>> snippet[3].
>>> A test file[4] using a similar implementation (stack memory only)
>>> built fine.
>>> Could you help me see what I'm doing wrong here.
>>
>> Cython variable semantics are just like they are in Python,
>> specifically, note that you are assigning to mp_head in that function.
>>
>> mp_head = mp_cur
>>
>> This makes mp_head into a local variable (and, since you haven't
>> specified a type, of type object). Just like Python, if you want to
>> assign to a global variable, you need to put "global mp_head" in your
>> function body.
>>
>> To make this error more clear, it would be a good idea to emit a
>> warning that mp_head is being used before it is defined.
>>
>> - Robert
>>
>>> I'm using repo: cython-devel changeset: 2094:bcc850a1253a date: Mon
>>> Jul 20 15:40:26 2009 -0700
>>> Thanks ~Pete
>>>
>>> [1] ####### ?build.sh script ##########
>>>
>>> cython -a -I/home/travlr/src/numpy.gitsvn/doc/cython/ "$1"
>>>
>>> gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -
>>> std=c99\
>>> ? ? -I/usr/include/python2.6 -o ${1%\.*}.so ${1%\.*}.c
>>>
>>> ######## end build.sh script ########
>>>
>>> [2] ######## build error message ########
>>>
>>> $ ./build utils.pyx
>>>
>>> Error converting Pyrex file to C:
>>> ------------------------------------------------------------
>>> ? ? void* mmalloc(size_t size, size_t n_items):
>>> ? ? ? ? cdef:
>>> ? ? ? ? ? ? MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof
>>> (MemPoolItem))
>>> ? ? ? ? mp_cur.memory = malloc(size * n_items)
>>> ? ? ? ? mp_cur.next = mp_head
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
>>> ------------------------------------------------------------
>>> /home/travlr/py/cy/utils.pyx:14:29: Cannot convert Python object to
>>> 'MemPoolItem *'
>>>
>>> [...]
>>>
>>> ######## end build error message ###########
>>>
>>> [3] ######## code snippet #########
>>>
>>> from defs cimport *
>>>
>>> cdef:
>>> ? ? struct MemPoolItem:
>>> ? ? ? ? void* ? ? ? ? ? memory
>>> ? ? ? ? MemPoolItem* ? ?next
>>>
>>> ? ? MemPoolItem* mp_head=NULL
>>>
>>> ? ? void* mmalloc(size_t size, size_t n_items):
>>> ? ? ? ? cdef:
>>> ? ? ? ? ? ? MemPoolItem* mp_cur = <MemPoolItem*>malloc(sizeof
>>> (MemPoolItem))
>>> ? ? ? ? mp_cur.memory = malloc(size * n_items)
>>> ? ? ? ? mp_cur.next = mp_head
>>> ? ? ? ? mp_head = mp_cur
>>> ? ? ? ? return mp_cur.memory
>>>
>>> ? ? mfree():
>>> ? ? ? ? cdef:
>>> ? ? ? ? ? ? MemPoolItem* mp_cur = mpi_head
>>> ? ? ? ? ? ? MemPoolItem* mp_tmp = NULL
>>> ? ? ? ? while mp_cur:
>>> ? ? ? ? ? ? free(mp_cur.memory)
>>> ? ? ? ? ? ? mp_cur.memory = NULL
>>> ? ? ? ? ? ? mp_tmp = mpi_cur.next
>>> ? ? ? ? ? ? free(mpi_cur)
>>> ? ? ? ? ? ? mp_cur = mpi_tmp
>>> ? ? ? ? mp_head = NULL
>>>
>>> ######## end code snippet #########
>>>
>>> [4] ######## test file #########
>>>
>>> cdef:
>>> ? ? struct S_t:
>>> ? ? ? ? int val
>>> ? ? ? ? S_t* next
>>>
>>> ? ? S_t s
>>>
>>> ? ? _run():
>>> ? ? ? ? s.val = 345
>>> ? ? ? ? print s.val
>>>
>>> def run():
>>> ? ? _run()
>>>
>>> ########### end test file ############
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> Hi Robert,
>
> I'm certainly confused by the semantics. Also, I can't find the use of
> 'global' documented anywhere yet.
> 1) I'm not sure why the "test" file built ok while referencing the
> varible w/ out declaring it to be "global".
> 2) Placing "global mp_head" as the first line of the function body in
> both functions gives a different error(s):
>
> Thank you,
>
> ################################
> $ ./build utils.pyx
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ? ? ? ?return mp_cur.memory
>
> ? ?mfree():
> ? ? ? ?global mp_head
> ? ? ? ?cdef:
> ? ? ? ? ? ?MemPoolItem* mp_cur = mpi_head
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
> ------------------------------------------------------------
> /home/travlr/py/cy/utils.pyx:22:42: undeclared name not builtin: mpi_head
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ? ? ? ? ? ?MemPoolItem* mp_cur = mpi_head
> ? ? ? ? ? ?MemPoolItem* mp_tmp = NULL
> ? ? ? ?while mp_cur:
> ? ? ? ? ? ?free(mp_cur.memory)
> ? ? ? ? ? ?mp_cur.memory = NULL
> ? ? ? ? ? ?mp_tmp = mpi_cur.next
> ? ? ? ? ? ? ? ? ? ? ? ? ? ^
> ------------------------------------------------------------
> /home/travlr/py/cy/utils.pyx:27:28: undeclared name not builtin: mpi_cur
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ? ? ? ? ? ?MemPoolItem* mp_tmp = NULL
> ? ? ? ?while mp_cur:
> ? ? ? ? ? ?free(mp_cur.memory)
> ? ? ? ? ? ?mp_cur.memory = NULL
> ? ? ? ? ? ?mp_tmp = mpi_cur.next
> ? ? ? ? ? ?free(mpi_cur)
> ? ? ? ? ? ? ? ? ? ? ? ^
> ------------------------------------------------------------
> /home/travlr/py/cy/utils.pyx:28:24: undeclared name not builtin: mpi_cur
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ? ? ? ?while mp_cur:
> ? ? ? ? ? ?free(mp_cur.memory)
> ? ? ? ? ? ?mp_cur.memory = NULL
> ? ? ? ? ? ?mp_tmp = mpi_cur.next
> ? ? ? ? ? ?free(mpi_cur)
> ? ? ? ? ? ?mp_cur = mpi_tmp
> ? ? ? ? ? ? ? ? ? ? ? ? ? ^
> ------------------------------------------------------------
>

Damn, I hate when I find a solution just after I posted the problem.
Please excuse me for I had a mis-spelling.

From robertwb at math.washington.edu  Thu Jul 23 10:40:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Jul 2009 01:40:11 -0700
Subject: [Cython] [sage-support] Re: changeing sage -b (cython)
In-Reply-To: <95ee2dab-113c-45c6-87cb-dbf4017c633b@a7g2000yqk.googlegroups.com>
References: <c83b3d1c-ebbc-4d11-8cd3-fd64aa8aaa99@26g2000yqk.googlegroups.com>
	<f67335450907171237l80a3621l6f3ce5e57ca6dc39@mail.gmail.com>
	<6ebc39c0-8255-4577-89ff-8ab8e1886525@y10g2000prf.googlegroups.com>
	<20d43422-df88-471c-b903-8c7f487ad577@24g2000yqm.googlegroups.com>
	<d09943d2-9173-4789-838c-fdb59de2a86c@i18g2000pro.googlegroups.com>
	<F0F8184B-218F-4444-95A0-89983D68FB78@math.washington.edu>
	<92dae4b1-e803-4d4f-b5cb-3989a058b7e5@z4g2000prh.googlegroups.com>
	<85e81ba30907221127w53a20ab5pc2bc26b1968ee20b@mail.gmail.com>
	<5689E418-9BBD-4359-9B14-73174723821C@math.washington.edu>
	<9418b9470907221139p1885fa35gee164559ec9704d8@mail.gmail.com>
	<95ee2dab-113c-45c6-87cb-dbf4017c633b@a7g2000yqk.googlegroups.com>
Message-ID: <79B14468-6B58-41BC-A5C3-7D4BF8686802@math.washington.edu>

I'm not sure if or when either of these will be available, but  
neither are trivial. Both questions are probably better asked on the  
cython lists.

- Robert


On Jul 22, 2009, at 12:17 PM, Ethan Van Andel wrote:

> Robert,
>
> Is there any prediction for when numpy complex types will work?
> (outside of the notebook, when compiling via sage -b)


On Jul 22, 2009, at 12:56 PM, Ethan Van Andel wrote:
>
> Yet another question:
>
> When I compile my class, I get something like this:
>
> cdef class Riemann_Map:
>     cdef int N, B, ncorners
>     cdef f
>     cdef opp
>     cdef double complex a
>     cdef np.ndarray[float,ndim = 1] tester
>                   ^
> ------------------------------------------------------------
>
> /home/evlutte/opt/sage-3.2.3/devel/sage-main/sage/plot/riemann.pyx:
> 39:19: Buffer types only allowed as function local variables
>
> Is there any way around this, that is, any way to get efficient array
> performance from my self.something arrays?


From rsclarke at gmail.com  Thu Jul 23 12:39:43 2009
From: rsclarke at gmail.com (Richard Clarke)
Date: Thu, 23 Jul 2009 11:39:43 +0100
Subject: [Cython] Cannot convert pointer to Python Object
Message-ID: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>

Hi,

Can anyone point out where I am going wrong?

// foo.h

#ifndef __FOO_H__
#define __FOO_H__

typedef struct
{
??? int fd;
} foo_t;

foo_t* init(void);

#endif


// foo.c

#include "foo.h"
#include <stdlib.h>

foo_t* init(){
??? foo_t* foo = malloc(sizeof(foo_t));
??? return foo;
}


// foo.pyx

cdef extern from "foo.h":
??? ctypedef struct foo_t:
??? ??? int fd

??? foo_t* init()

class Bar:
??? def __init__(self):
??? ??? self.__foo = self.__open()

??? def __open(self):
??? ??? cdef foo_t *foo
??? ??? foo = init()
??? ??? return foo


Results in the error:
/test/foo.pyx:14:12: Cannot convert 'foo_t *' to Python object

Cython version 0.9.6.14

Thank you,
Richard.

From robertwb at math.washington.edu  Thu Jul 23 13:43:02 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Jul 2009 04:43:02 -0700
Subject: [Cython] Cannot convert pointer to Python Object
In-Reply-To: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
References: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
Message-ID: <0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>

On Jul 23, 2009, at 3:39 AM, Richard Clarke wrote:

> Hi,
>
> Can anyone point out where I am going wrong?

Sure. There is no way convert a pointer to a Python object. (Perhaps  
we could support this via http://www.python.org/doc/2.6/c-api/ 
cobject.html , but certainly that'll take a lot of thought).

You need to make Bar into a cdef class for it to have a foo_t*  
member. Also, if a return type is foo*, you need to make it a cdef  
function. I would recommend reading http://docs.cython.org/ .

> // foo.pyx
>
> cdef extern from "foo.h":
>     ctypedef struct foo_t:
>         int fd
>
>     foo_t* init()
>
> class Bar:

 > cdef class Bar:
 >     cdef foo* __foo

>     def __init__(self):
>         self.__foo = self.__open()
>
>     def __open(self):

 >     cdef foo* open(self):

>         cdef foo_t *foo
>         foo = init()
>         return foo
>
>
> Results in the error:
> /test/foo.pyx:14:12: Cannot convert 'foo_t *' to Python object
>
> Cython version 0.9.6.14
>
> Thank you,
> Richard.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From rsclarke at gmail.com  Thu Jul 23 15:29:25 2009
From: rsclarke at gmail.com (Richard Clarke)
Date: Thu, 23 Jul 2009 14:29:25 +0100
Subject: [Cython] Cannot convert pointer to Python Object
In-Reply-To: <0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>
References: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
	<0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>
Message-ID: <6352aa660907230629y478696cdpf9b8d6f1dcadb9e3@mail.gmail.com>

Thank you Robert, the explanation on how and where to use cdef was
most helpful.  I think it has just clicked(with a flag waving
'idiot').

The PyCObject concept is a nice idea, removing cdef foo_t *foo from
outside of __init__ ?  However, I see implications in memory
deallocation.  How does cython currently handle memory deallocation?

I assume the best practice would be for users to deallocate memory in
the class __del__ method (calling the appropriate wrapped c library
destructor).  Otherwise does cython simply free(foo) or interprets the
language to build a depth first deallocation method?  I guess it
should do the simplest free(foo), if the developer was lazy not to
deallocate memory, they can suffer with memory leaks.  I think I might
have answered my own question here...

Richard

On Thu, Jul 23, 2009 at 12:43 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jul 23, 2009, at 3:39 AM, Richard Clarke wrote:
>
>> Hi,
>>
>> Can anyone point out where I am going wrong?
>
> Sure. There is no way convert a pointer to a Python object. (Perhaps
> we could support this via http://www.python.org/doc/2.6/c-api/
> cobject.html , but certainly that'll take a lot of thought).
>
> You need to make Bar into a cdef class for it to have a foo_t*
> member. Also, if a return type is foo*, you need to make it a cdef
> function. I would recommend reading http://docs.cython.org/ .
>
>> // foo.pyx
>>
>> cdef extern from "foo.h":
>> ? ? ctypedef struct foo_t:
>> ? ? ? ? int fd
>>
>> ? ? foo_t* init()
>>
>> class Bar:
>
> ?> cdef class Bar:
> ?> ? ? cdef foo* __foo
>
>> ? ? def __init__(self):
>> ? ? ? ? self.__foo = self.__open()
>>
>> ? ? def __open(self):
>
> ?> ? ? cdef foo* open(self):
>
>> ? ? ? ? cdef foo_t *foo
>> ? ? ? ? foo = init()
>> ? ? ? ? return foo
>>
>>
>> Results in the error:
>> /test/foo.pyx:14:12: Cannot convert 'foo_t *' to Python object
>>
>> Cython version 0.9.6.14
>>
>> Thank you,
>> Richard.
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From anstmich at gmail.com  Thu Jul 23 20:47:57 2009
From: anstmich at gmail.com (Andy Michaels)
Date: Thu, 23 Jul 2009 14:47:57 -0400
Subject: [Cython] Cython, pointers, and scope
Message-ID: <5f35a6800907231147y70250ff2qfcacd2ff0163d5d9@mail.gmail.com>

Hello all,

I have recently started using cython for a project that I am working on.
First, understand that I am relatively new to python, let alone cython which
I started to delve into today.  Anyway... for this project, I need to
dynamically allocate memory for a 2D array (a la malloc(...)).
Unfortunately, I have become confused with what might be a scope issue,
however I am unsure.  I will start with a quick example of my problem:


********************************************** Code
*****************************************************
# gain access to malloc, free, etc
cdef extern from "stdlib.h":
    ctypedef unsigned long size_t
    void free(void *ptr)
    void *malloc(size_t size)
    void *realloc(void *ptr, size_t size)
    size_t strlen(char *s)
    char *strcpy(char *dest, char *src)

cdef struct Point:
    int x
    int y


cdef struct CalBox:
    Point *upperLeft
    Point *upperRight
    Point *lowerRight
    Point *lowerLeft


cdef CalBox **calMat

def initMatrix(numRows = 6, numCols = 8):
    calMat = <CalBox**>malloc(numCols*sizeof(CalBox*))
    for i in range(numCols):
        calMat[i] = <CalBox*>malloc(numRows*sizeof(CalBox))

****************************************************************************************************************

As you can see, I define calMat outside of any function such that its scope
is not limited: any function in the module *should* be able to access its
contents.  When I attempt compile the example, however, I get an error
stating: "Cannot convert 'CalBox *' to Python object"

This, to me, would seem to imply that the function is not "seeing" calMat
and thus is trying to create a new python object.  Furthermore, this only
seems to happen with pointers; defining C-style integers or python objects
in the main body of the module seem to be perfectly accessable.  I have done
quite a bit of searching and tweaking, but nothing has worked.

I expect it is a something simple that i have overlooked, however if anyone
could help, I would greatly appreciate it... In the meantime I guess Ill
have to keep the matrix at a predefined size.

Thanks,
Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090723/18562174/attachment.htm 

From dalcinl at gmail.com  Thu Jul 23 21:10:33 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Jul 2009 16:10:33 -0300
Subject: [Cython] Cannot convert pointer to Python Object
In-Reply-To: <6352aa660907230629y478696cdpf9b8d6f1dcadb9e3@mail.gmail.com>
References: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
	<0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>
	<6352aa660907230629y478696cdpf9b8d6f1dcadb9e3@mail.gmail.com>
Message-ID: <e7ba66e40907231210v78e01434t730f1f4ed05f555f@mail.gmail.com>

On Thu, Jul 23, 2009 at 10:29 AM, Richard Clarke<rsclarke at gmail.com> wrote:
> Thank you Robert, the explanation on how and where to use cdef was
> most helpful. ?I think it has just clicked(with a flag waving
> 'idiot').
>
> The PyCObject concept is a nice idea, removing cdef foo_t *foo from
> outside of __init__ ? ?However, I see implications in memory
> deallocation. ?How does cython currently handle memory deallocation?
>

I'm not sure you need PyCObject for your code ...

>
> I assume the best practice would be for users to deallocate memory in
> the class __del__ method (calling the appropriate wrapped c library
> destructor).

Yes. More specifically, you should allocate stuff in __cinit__(), and
deallocate stuff in __dealloc__()...


> ?Otherwise does cython simply free(foo) or interprets the
> language to build a depth first deallocation method?

Cython does not manage memory allocated with malloc()... you have to
manage deallocation yourself...


> ?I guess it
> should do the simplest free(foo), if the developer was lazy not to
> deallocate memory, they can suffer with memory leaks. ?I think I might
> have answered my own question here...
>

In short, you have to modify your class, using something like this

cdef class Bar: # NOTE: "cdef" class

    cdef foo_t *c_foo # NOTE: "cdef" instance (not class) member!!!

    def __cinit__(self): # NOTE: using "__cinit__", not "__init__"
        self.c_foo = self.c_open()

    def __dealloc__(self): # NOTE: using "__dealloc__", not "__del__"
        if self.c_foo != NULL:
            free(self.c_foo)

    cdef foo_t* c_open(self): # NOTE: using "cdef", not "def", this
method will not show-up on Python side, it is a kind of internal,
private method.
        cdef foo_t *foo = NULL
        foo = init()
        return foo


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Jul 23 21:14:18 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Jul 2009 16:14:18 -0300
Subject: [Cython] Cython, pointers, and scope
In-Reply-To: <5f35a6800907231147y70250ff2qfcacd2ff0163d5d9@mail.gmail.com>
References: <5f35a6800907231147y70250ff2qfcacd2ff0163d5d9@mail.gmail.com>
Message-ID: <e7ba66e40907231214n5cd55818x21eccab66d1b23@mail.gmail.com>

You initMatrix() function should start with the line, below:

global calMat


These are the normal Python rules when assigning a "module-level" global...


On Thu, Jul 23, 2009 at 3:47 PM, Andy Michaels<anstmich at gmail.com> wrote:
> Hello all,
>
> I have recently started using cython for a project that I am working on.
> First, understand that I am relatively new to python, let alone cython which
> I started to delve into today.? Anyway... for this project, I need to
> dynamically allocate memory for a 2D array (a la malloc(...)).
> Unfortunately, I have become confused with what might be a scope issue,
> however I am unsure.? I will start with a quick example of my problem:
>
>
> ********************************************** Code
> *****************************************************
> # gain access to malloc, free, etc
> cdef extern from "stdlib.h":
> ??? ctypedef unsigned long size_t
> ??? void free(void *ptr)
> ??? void *malloc(size_t size)
> ??? void *realloc(void *ptr, size_t size)
> ??? size_t strlen(char *s)
> ??? char *strcpy(char *dest, char *src)
>
> cdef struct Point:
> ??? int x
> ??? int y
>
>
> cdef struct CalBox:
> ??? Point *upperLeft
> ??? Point *upperRight
> ??? Point *lowerRight
> ??? Point *lowerLeft
>
>
> cdef CalBox **calMat
>
> def initMatrix(numRows = 6, numCols = 8):
> ??? calMat = <CalBox**>malloc(numCols*sizeof(CalBox*))
> ??? for i in range(numCols):
> ??? ??? calMat[i] = <CalBox*>malloc(numRows*sizeof(CalBox))
>
> ****************************************************************************************************************
>
> As you can see, I define calMat outside of any function such that its scope
> is not limited: any function in the module should be able to access its
> contents.? When I attempt compile the example, however, I get an error
> stating: "Cannot convert 'CalBox *' to Python object"
>
> This, to me, would seem to imply that the function is not "seeing" calMat
> and thus is trying to create a new python object.? Furthermore, this only
> seems to happen with pointers; defining C-style integers or python objects
> in the main body of the module seem to be perfectly accessable.? I have done
> quite a bit of searching and tweaking, but nothing has worked.
>
> I expect it is a something simple that i have overlooked, however if anyone
> could help, I would greatly appreciate it... In the meantime I guess Ill
> have to keep the matrix at a predefined size.
>
> Thanks,
> Andy
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Jul 23 21:23:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Jul 2009 12:23:11 -0700
Subject: [Cython] Cython, pointers, and scope
In-Reply-To: <5f35a6800907231147y70250ff2qfcacd2ff0163d5d9@mail.gmail.com>
References: <5f35a6800907231147y70250ff2qfcacd2ff0163d5d9@mail.gmail.com>
Message-ID: <34A29F9D-05E5-46B5-A59F-5F67291BEED1@math.washington.edu>

On Jul 23, 2009, at 11:47 AM, Andy Michaels wrote:

> Hello all,
>
> I have recently started using cython for a project that I am  
> working on.  First, understand that I am relatively new to python,  
> let alone cython which I started to delve into today.  Anyway...  
> for this project, I need to dynamically allocate memory for a 2D  
> array (a la malloc(...)).  Unfortunately, I have become confused  
> with what might be a scope issue, however I am unsure.  I will  
> start with a quick example of my problem:

This is the second time this question came up today. See

http://wiki.cython.org/FAQ#HowdoIassigntoaglobalvariable.3F

- Robert


From robertwb at math.washington.edu  Thu Jul 23 21:28:37 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Jul 2009 12:28:37 -0700
Subject: [Cython] Cannot convert pointer to Python Object
In-Reply-To: <e7ba66e40907231210v78e01434t730f1f4ed05f555f@mail.gmail.com>
References: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
	<0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>
	<6352aa660907230629y478696cdpf9b8d6f1dcadb9e3@mail.gmail.com>
	<e7ba66e40907231210v78e01434t730f1f4ed05f555f@mail.gmail.com>
Message-ID: <CA3954F8-C78C-484A-BD46-97E9837AD696@math.washington.edu>

On Jul 23, 2009, at 12:10 PM, Lisandro Dalcin wrote:

> On Thu, Jul 23, 2009 at 10:29 AM, Richard  
> Clarke<rsclarke at gmail.com> wrote:
>> Thank you Robert, the explanation on how and where to use cdef was
>> most helpful.  I think it has just clicked(with a flag waving
>> 'idiot').
>>
>> The PyCObject concept is a nice idea, removing cdef foo_t *foo from
>> outside of __init__ ?  However, I see implications in memory
>> deallocation.  How does cython currently handle memory deallocation?
>
> I'm not sure you need PyCObject for your code ...

For sure PyCObject is not needed here. I was musing about a ptr <->  
object conversion happening automatically via PyCObject, just as we  
handle the more basic numeric types. Deallocation of pointers needs  
to be handled by the user explicitly, so perhaps it's not a good idea  
to make this transparently easy.

>
>>
>> I assume the best practice would be for users to deallocate memory in
>> the class __del__ method (calling the appropriate wrapped c library
>> destructor).
>
> Yes. More specifically, you should allocate stuff in __cinit__(), and
> deallocate stuff in __dealloc__()...

+1. To clarify, __cinit__ is called exactly once per object creation  
(no such guarantee with __init__). Likewise, __dealloc__ is always  
called exactly once upon collection. This makes memory management  
easy--essentially you're boostrapping of Python's refcounting.

>
>
>>  Otherwise does cython simply free(foo) or interprets the
>> language to build a depth first deallocation method?
>
> Cython does not manage memory allocated with malloc()... you have to
> manage deallocation yourself...
>
>
>>  I guess it
>> should do the simplest free(foo), if the developer was lazy not to
>> deallocate memory, they can suffer with memory leaks.  I think I  
>> might
>> have answered my own question here...
>>
>
> In short, you have to modify your class, using something like this
>
> cdef class Bar: # NOTE: "cdef" class
>
>     cdef foo_t *c_foo # NOTE: "cdef" instance (not class) member!!!
>
>     def __cinit__(self): # NOTE: using "__cinit__", not "__init__"
>         self.c_foo = self.c_open()
>
>     def __dealloc__(self): # NOTE: using "__dealloc__", not "__del__"
>         if self.c_foo != NULL:
>             free(self.c_foo)
>
>     cdef foo_t* c_open(self): # NOTE: using "cdef", not "def", this
> method will not show-up on Python side, it is a kind of internal,
> private method.
>         cdef foo_t *foo = NULL
>         foo = init()
>         return foo
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Thu Jul 23 21:32:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Jul 2009 12:32:07 -0700
Subject: [Cython] expy
In-Reply-To: <4A66183F.9020101@behnel.de>
References: <h44npl$p7m$1@ger.gmane.org> <4A66183F.9020101@behnel.de>
Message-ID: <F0D523E6-B77D-4930-B79C-6C6BD12798D2@math.washington.edu>

On Jul 21, 2009, at 12:34 PM, Stefan Behnel wrote:

>
> Neal Becker wrote:
>> This is an interesting approach:
>>
>> http://expy.sourceforge.net/
>
> Well, I read about it, too, as it was advertised on python-dev  
> (mistakenly,
> I assume). It's similar to Cython in that it allows you to write  
> Python
> function glue code as Python funtions. However, that only handles  
> the plain
> call code. Having to put everything else into plain strings sounds  
> pretty
> awkward to me. It may be ok for thin wrappers, but I doubt that  
> it's usable
> to write real code at the wrapper layer.

Yep, that's the impression I got too. Might be easy for writing  
simple glue, but not for writing code (especially anything that needs  
to handle refcounting). Feels a lot like Weave (more powerful in some  
ways, less useful others).

- Robert


From anstmich at gmail.com  Thu Jul 23 22:10:51 2009
From: anstmich at gmail.com (Andy Michaels)
Date: Thu, 23 Jul 2009 16:10:51 -0400
Subject: [Cython] Cython, pointers, and scope
In-Reply-To: <34A29F9D-05E5-46B5-A59F-5F67291BEED1@math.washington.edu>
References: <5f35a6800907231147y70250ff2qfcacd2ff0163d5d9@mail.gmail.com>
	<34A29F9D-05E5-46B5-A59F-5F67291BEED1@math.washington.edu>
Message-ID: <5f35a6800907231310i118c71f0ma9fdcbd878800e@mail.gmail.com>

Ahh, duh, I thought it would be something simple, thanks all... I have yet
to use global variables in python as i tend to write things in an OO design
and define variables in the __init__() function of each class.


On Thu, Jul 23, 2009 at 3:23 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On Jul 23, 2009, at 11:47 AM, Andy Michaels wrote:
>
> > Hello all,
> >
> > I have recently started using cython for a project that I am
> > working on.  First, understand that I am relatively new to python,
> > let alone cython which I started to delve into today.  Anyway...
> > for this project, I need to dynamically allocate memory for a 2D
> > array (a la malloc(...)).  Unfortunately, I have become confused
> > with what might be a scope issue, however I am unsure.  I will
> > start with a quick example of my problem:
>
> This is the second time this question came up today. See
>
> http://wiki.cython.org/FAQ#HowdoIassigntoaglobalvariable.3F
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090723/e72b3687/attachment.htm 

From robertwb at math.washington.edu  Thu Jul 23 22:27:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 23 Jul 2009 13:27:26 -0700
Subject: [Cython] incompatible types (complex)
In-Reply-To: <f67de9dc0907200732h403e3f97ld35543982be11f99@mail.gmail.com>
References: <f67de9dc0907200732h403e3f97ld35543982be11f99@mail.gmail.com>
Message-ID: <EC7B5D0C-1D1B-4165-B753-AA609C1AE0BF@math.washington.edu>

On Jul 20, 2009, at 7:32 AM, Ethan Van Andel wrote:

> when I compile some code containing this snippet: (np = numpy)
>
> COMPLEX = np.complex128
> ctypedef np.complex128_t COMPLEX_T
>
> class Riemann_Map():
>     def __init__(self,fs,fprimes,a1,N,int ncorners = 4,opp = False):
>         cdef COMPLEX_T a = np.complex128(a1)
>
> I get these errors (among others):
> c:1316: error: incompatible types in assignment
> and
> c:1316: error: previous implicit declaration of
> ?__pyx_PyObject_As___pyx_t_double_complex? was here
>
> where c:1316 is the line corresponding to, cdef COMPLEX_T a ...
>
> I am compiling this using Cython 0.11.2.beta in the sage -b  
> environment.
>
> What am I doing wrong?

I said this elsewhere, but should clarify it here too, you have to  
declare the type as "complex double" which goes fine into a complex  
ndarray. I'm not sure when or how arithmetic of np.complex128_t will  
be supported, but unlike ints, on all platforms I know of, sizeof 
(float) = 32 and sizeof(double) = 64 (leading to the 64 and 128 bit  
complex types respectively).

- Robert



From rsclarke at gmail.com  Fri Jul 24 00:11:59 2009
From: rsclarke at gmail.com (Richard Clarke)
Date: Thu, 23 Jul 2009 23:11:59 +0100
Subject: [Cython] Cannot convert pointer to Python Object
In-Reply-To: <CA3954F8-C78C-484A-BD46-97E9837AD696@math.washington.edu>
References: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
	<0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>
	<6352aa660907230629y478696cdpf9b8d6f1dcadb9e3@mail.gmail.com>
	<e7ba66e40907231210v78e01434t730f1f4ed05f555f@mail.gmail.com>
	<CA3954F8-C78C-484A-BD46-97E9837AD696@math.washington.edu>
Message-ID: <6352aa660907231511h53d7f8aax16c7b4e525495d72@mail.gmail.com>

On Thu, Jul 23, 2009 at 8:28 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jul 23, 2009, at 12:10 PM, Lisandro Dalcin wrote:
>
>> On Thu, Jul 23, 2009 at 10:29 AM, Richard
>> Clarke<rsclarke at gmail.com> wrote:
>>> Thank you Robert, the explanation on how and where to use cdef was
>>> most helpful. ?I think it has just clicked(with a flag waving
>>> 'idiot').
>>>
>>> The PyCObject concept is a nice idea, removing cdef foo_t *foo from
>>> outside of __init__ ? ?However, I see implications in memory
>>> deallocation. ?How does cython currently handle memory deallocation?
>>
>> I'm not sure you need PyCObject for your code ...
>
> For sure PyCObject is not needed here. I was musing about a ptr <->
> object conversion happening automatically via PyCObject, just as we
> handle the more basic numeric types. Deallocation of pointers needs
> to be handled by the user explicitly, so perhaps it's not a good idea
> to make this transparently easy.
>
>>
>>>
>>> I assume the best practice would be for users to deallocate memory in
>>> the class __del__ method (calling the appropriate wrapped c library
>>> destructor).
>>
>> Yes. More specifically, you should allocate stuff in __cinit__(), and
>> deallocate stuff in __dealloc__()...
>
> +1. To clarify, __cinit__ is called exactly once per object creation
> (no such guarantee with __init__). Likewise, __dealloc__ is always
> called exactly once upon collection. This makes memory management
> easy--essentially you're boostrapping of Python's refcounting.
>
>>
>>
>>> ?Otherwise does cython simply free(foo) or interprets the
>>> language to build a depth first deallocation method?
>>
>> Cython does not manage memory allocated with malloc()... you have to
>> manage deallocation yourself...
>>
>>
>>> ?I guess it
>>> should do the simplest free(foo), if the developer was lazy not to
>>> deallocate memory, they can suffer with memory leaks. ?I think I
>>> might
>>> have answered my own question here...
>>>
>>
>> In short, you have to modify your class, using something like this
>>
>> cdef class Bar: # NOTE: "cdef" class
>>
>> ? ? cdef foo_t *c_foo # NOTE: "cdef" instance (not class) member!!!
>>
>> ? ? def __cinit__(self): # NOTE: using "__cinit__", not "__init__"
>> ? ? ? ? self.c_foo = self.c_open()
>>
>> ? ? def __dealloc__(self): # NOTE: using "__dealloc__", not "__del__"
>> ? ? ? ? if self.c_foo != NULL:
>> ? ? ? ? ? ? free(self.c_foo)
>>
>> ? ? cdef foo_t* c_open(self): # NOTE: using "cdef", not "def", this
>> method will not show-up on Python side, it is a kind of internal,
>> private method.
>> ? ? ? ? cdef foo_t *foo = NULL
>> ? ? ? ? foo = init()
>> ? ? ? ? return foo
>>

I'm not sure if this is the correct way (it appears to work) but here goes...

If we consider Bar to be the encapsulating class for the foo_t struct,
in my case I need access to this foo_t struct in other classes.  Using
Lisandro's Bar class as above with an extra method, class Qux
demonstrates how to obtain the foo_t pointer.  This in turn could be
passed in to other c library function calls etc.

// foo.pyx

cdef extern from "stdlib.h":
	ctypedef unsigned long size_t
	void free(void *ptr)

cdef extern from "foo.h":
	ctypedef struct foo_t:
		int fd

	foo_t* init()

cdef class Bar:
	cdef foo_t *foo

	def __cinit__(self):
		self.foo = self.__open()
		self.foo.fd = 1

	def __dealloc__(self):
		if self.foo != NULL:
			free(self.foo)

	cdef foo_t* __open(self):
		cdef foo_t *f
		f = init()
		return f

	cdef foo_t* get_foo(self):
		return self.foo

class Qux:
	def __init__(self):
		global qux
		qux = Bar()

	def something(self):
		cdef foo_t* _foo
		_foo = Bar.get_foo(qux)
		if _foo != NULL:
			#Do something with the struct

Thanks to both of you for your help,
Richard.

>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From davidnovakovic at gmail.com  Fri Jul 24 00:17:16 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Fri, 24 Jul 2009 08:17:16 +1000
Subject: [Cython] On the topic of wrapper classes
Message-ID: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>

Does cython have anything like the as_parameter functionality of ctypes?

I mean, if i have a class Foo, which wraps a struct  foo_t * foo, would be
it possible to say when i pass this object to a function the requires a
foo_t * parameter to pass the c level attribute of the correct type instead
of the actual python class?

I'm writing a bunch of wrappers at the moment and it seems like there is a
lot of code that could be automated!

Thanks... otherwise I'm really enjoying cython :)

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090724/4a563c21/attachment.htm 

From dalcinl at gmail.com  Fri Jul 24 04:03:32 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Jul 2009 23:03:32 -0300
Subject: [Cython] Cannot convert pointer to Python Object
In-Reply-To: <6352aa660907231511h53d7f8aax16c7b4e525495d72@mail.gmail.com>
References: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
	<0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>
	<6352aa660907230629y478696cdpf9b8d6f1dcadb9e3@mail.gmail.com>
	<e7ba66e40907231210v78e01434t730f1f4ed05f555f@mail.gmail.com>
	<CA3954F8-C78C-484A-BD46-97E9837AD696@math.washington.edu>
	<6352aa660907231511h53d7f8aax16c7b4e525495d72@mail.gmail.com>
Message-ID: <e7ba66e40907231903v56eb6653v280c517a1e5cd5ca@mail.gmail.com>

On Thu, Jul 23, 2009 at 7:11 PM, Richard Clarke<rsclarke at gmail.com> wrote:
>
> If we consider Bar to be the encapsulating class for the foo_t struct,
> in my case I need access to this foo_t struct in other classes.

That's common use case... And Cython supports that almost for free..

> Using
> Lisandro's Bar class as above with an extra method, class Qux
> demonstrates how to obtain the foo_t pointer.

No need at all of such hackery..

> This in turn could be
> passed in to other c library function calls etc.
>

you can do this:

def py_function1(obj):
    cdef Bar bar = obj
    some_c_function(bar.foo) # pass foo_t* to a C function

of even this:

def py_function1(Bar bar):
    some_c_function(bar.foo)


So in Cython code, cdef members are "public" and you can access them
as long as you use a "cdef" typeded variable. However, remember that
these cdef member are not accessible from Python side, outside Cython
code.



> // foo.pyx
>
> cdef extern from "stdlib.h":
> ? ? ? ?ctypedef unsigned long size_t
> ? ? ? ?void free(void *ptr)
>
> cdef extern from "foo.h":
> ? ? ? ?ctypedef struct foo_t:
> ? ? ? ? ? ? ? ?int fd
>
> ? ? ? ?foo_t* init()
>
> cdef class Bar:
> ? ? ? ?cdef foo_t *foo
>
> ? ? ? ?def __cinit__(self):
> ? ? ? ? ? ? ? ?self.foo = self.__open()
> ? ? ? ? ? ? ? ?self.foo.fd = 1
>
> ? ? ? ?def __dealloc__(self):
> ? ? ? ? ? ? ? ?if self.foo != NULL:
> ? ? ? ? ? ? ? ? ? ? ? ?free(self.foo)
>
> ? ? ? ?cdef foo_t* __open(self):
> ? ? ? ? ? ? ? ?cdef foo_t *f
> ? ? ? ? ? ? ? ?f = init()
> ? ? ? ? ? ? ? ?return f
>
> ? ? ? ?cdef foo_t* get_foo(self):
> ? ? ? ? ? ? ? ?return self.foo
>
> class Qux:
> ? ? ? ?def __init__(self):
> ? ? ? ? ? ? ? ?global qux
> ? ? ? ? ? ? ? ?qux = Bar()
>
> ? ? ? ?def something(self):
> ? ? ? ? ? ? ? ?cdef foo_t* _foo
> ? ? ? ? ? ? ? ?_foo = Bar.get_foo(qux)
> ? ? ? ? ? ? ? ?if _foo != NULL:
> ? ? ? ? ? ? ? ? ? ? ? ?#Do something with the struct
>
> Thanks to both of you for your help,
> Richard.
>
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Jul 24 04:20:35 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 23 Jul 2009 23:20:35 -0300
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
Message-ID: <e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>

So you are proposing to go against the Zen of Python's (Explicit is
better than implicit) just to save a few key strokes? However, I have
to grant you that this use case is so common (at least for all us
using Cython for wrapping) that this idea could deserve some
consideration? Two more lines to take into account:

Special cases aren't special enough to break the rules.
Although practicality beats purity.


On Thu, Jul 23, 2009 at 7:17 PM, David P.
Novakovic<davidnovakovic at gmail.com> wrote:
> Does cython have anything like the as_parameter functionality of ctypes?
>
> I mean, if i have a class Foo, which wraps a struct? foo_t * foo, would be
> it possible to say when i pass this object to a function the requires a
> foo_t * parameter to pass the c level attribute of the correct type instead
> of the actual python class?
>
> I'm writing a bunch of wrappers at the moment and it seems like there is a
> lot of code that could be automated!
>
> Thanks... otherwise I'm really enjoying cython :)
>
> David
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From davidnovakovic at gmail.com  Fri Jul 24 04:44:54 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Fri, 24 Jul 2009 12:44:54 +1000
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
Message-ID: <59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>

It's already a feature in the python stdlib through ctypes, so it obviously
doesn't break the guidelines that much.

Also, it's not any less explicit.. it'd just be nice to create the mapping
once on the class then have the calling convention respect that everywhere
else. In many ways this is more pythonic than repeatedly typing the same
'few characters' every time  i make a function call into a C library.

As you said, this isn't really a special case, so many of us are doing it...

On Fri, Jul 24, 2009 at 12:20 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> So you are proposing to go against the Zen of Python's (Explicit is
> better than implicit) just to save a few key strokes? However, I have
> to grant you that this use case is so common (at least for all us
> using Cython for wrapping) that this idea could deserve some
> consideration? Two more lines to take into account:
>
> Special cases aren't special enough to break the rules.
> Although practicality beats purity.
>
>
> On Thu, Jul 23, 2009 at 7:17 PM, David P.
> Novakovic<davidnovakovic at gmail.com> wrote:
> > Does cython have anything like the as_parameter functionality of ctypes?
> >
> > I mean, if i have a class Foo, which wraps a struct  foo_t * foo, would
> be
> > it possible to say when i pass this object to a function the requires a
> > foo_t * parameter to pass the c level attribute of the correct type
> instead
> > of the actual python class?
> >
> > I'm writing a bunch of wrappers at the moment and it seems like there is
> a
> > lot of code that could be automated!
> >
> > Thanks... otherwise I'm really enjoying cython :)
> >
> > David
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090724/86e232f3/attachment.htm 

From dalcinl at gmail.com  Fri Jul 24 05:23:49 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Jul 2009 00:23:49 -0300
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
Message-ID: <e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>

On Thu, Jul 23, 2009 at 11:44 PM, David P.
Novakovic<davidnovakovic at gmail.com> wrote:
>
> It's already a feature in the python stdlib through ctypes, so it obviously
> doesn't break the guidelines that much.
>

Sorry for my ignorance about ctypes... But could you point me to some
code/link or provide some more comments in order I can understand what
you commented above?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From davidnovakovic at gmail.com  Fri Jul 24 05:29:54 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Fri, 24 Jul 2009 13:29:54 +1000
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
Message-ID: <59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>

http://docs.python.org/library/ctypes.html#calling-functions-with-your-own-custom-data-types

I notice it only accepts basic types... and obviously the implementation
would be a lot simpler than what I am talking about for cython... but that's
the general gist of it.

I guess for cython it would be nice to be able to have a mapping of type ->
attribute (or type_name -> attribute_name) so that when the class is passed
as a parameter the attribute of the right type is passed instead.

David

On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> On Thu, Jul 23, 2009 at 11:44 PM, David P.
> Novakovic<davidnovakovic at gmail.com> wrote:
> >
> > It's already a feature in the python stdlib through ctypes, so it
> obviously
> > doesn't break the guidelines that much.
> >
>
> Sorry for my ignorance about ctypes... But could you point me to some
> code/link or provide some more comments in order I can understand what
> you commented above?
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090724/70314c5c/attachment.htm 

From jason-sage at creativetrax.com  Fri Jul 24 08:23:37 2009
From: jason-sage at creativetrax.com (Jason Grout)
Date: Thu, 23 Jul 2009 23:23:37 -0700
Subject: [Cython] any() and all()
Message-ID: <h4bk19$50q$1@ger.gmane.org>

Is it possible to support the any() and all() functions?  I'm using the 
most recent version of Cython in Sage, so forgive me if it is already 
supported in Cython, but just not in Sage's version yet.

I realize that it probably would be easy to write such functions, but 
since they are standard python functions, it would be nice if they were 
recognized by Cython.  It would be super nice if they were *fast* too :).

See http://docs.python.org/library/functions.html#all and 
http://docs.python.org/library/functions.html#any

Thanks!

Jason

--
Jason Grout


From robertwb at math.washington.edu  Fri Jul 24 13:49:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Jul 2009 04:49:50 -0700
Subject: [Cython] Cannot convert pointer to Python Object
In-Reply-To: <e7ba66e40907231903v56eb6653v280c517a1e5cd5ca@mail.gmail.com>
References: <6352aa660907230339k6c064e7dl4b53ab17c8b73936@mail.gmail.com>
	<0134A56A-B2C0-4568-AE9C-87B196EDAED0@math.washington.edu>
	<6352aa660907230629y478696cdpf9b8d6f1dcadb9e3@mail.gmail.com>
	<e7ba66e40907231210v78e01434t730f1f4ed05f555f@mail.gmail.com>
	<CA3954F8-C78C-484A-BD46-97E9837AD696@math.washington.edu>
	<6352aa660907231511h53d7f8aax16c7b4e525495d72@mail.gmail.com>
	<e7ba66e40907231903v56eb6653v280c517a1e5cd5ca@mail.gmail.com>
Message-ID: <AE7E7EB7-3D7B-4E36-A9D1-2D660E6520EC@math.washington.edu>

Note that you can declare at foo_t to be a member of Bar directly, so  
you don't even need to allocate it (this works if you're not getting  
it as a pointer from elsehwhere). For example

cdef extern from "foo.h":
        ctypedef struct foo_t:
                int fd
         cdef some_c_func(foo_t*)

cdef class Bar:
     cdef foo_t foo

     cdef __init__(self, n):
         self.foo.fd = n

def call_something_on_foo(Bar b):
     some_c_func(&b.foo)


- Robert

From robertwb at math.washington.edu  Fri Jul 24 13:53:34 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Jul 2009 04:53:34 -0700
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
	<59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
Message-ID: <D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>

This feels a lot like C++ operator overloading...

I'm not a fan of _as_parameter_, but maybe an "as foo_t" kind of  
attribute that it would use every time it needed to convert it to a  
foo_t (not just as a parameter, but in assignment, etc.) Certainly  
something like this could be useful, but I'm not sure if it's needed.

- Robert

On Jul 23, 2009, at 8:29 PM, David P. Novakovic wrote:

> http://docs.python.org/library/ctypes.html#calling-functions-with- 
> your-own-custom-data-types
>
> I notice it only accepts basic types... and obviously the  
> implementation would be a lot simpler than what I am talking about  
> for cython... but that's the general gist of it.
>
> I guess for cython it would be nice to be able to have a mapping of  
> type -> attribute (or type_name -> attribute_name) so that when the  
> class is passed as a parameter the attribute of the right type is  
> passed instead.
>
> David
>
> On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin  
> <dalcinl at gmail.com> wrote:
> On Thu, Jul 23, 2009 at 11:44 PM, David P.
> Novakovic<davidnovakovic at gmail.com> wrote:
> >
> > It's already a feature in the python stdlib through ctypes, so it  
> obviously
> > doesn't break the guidelines that much.
> >
>
> Sorry for my ignorance about ctypes... But could you point me to some
> code/link or provide some more comments in order I can understand what
> you commented above?
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri Jul 24 13:53:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 24 Jul 2009 04:53:45 -0700
Subject: [Cython] any() and all()
In-Reply-To: <h4bk19$50q$1@ger.gmane.org>
References: <h4bk19$50q$1@ger.gmane.org>
Message-ID: <96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>

On Jul 23, 2009, at 11:23 PM, Jason Grout wrote:

> Is it possible to support the any() and all() functions?  I'm using  
> the
> most recent version of Cython in Sage, so forgive me if it is already
> supported in Cython, but just not in Sage's version yet.

It's working for me...

{{{
%cython
print any([0,0,1,0]), all([0,0,1,0])
///
(True, False)
}}}

> I realize that it probably would be easy to write such functions, but
> since they are standard python functions, it would be nice if they  
> were
> recognized by Cython.  It would be super nice if they were *fast*  
> too :).

This was actually discussed recently, and yes, we'd like to support  
this kind of thing. It shouldn't be too hard to do either. However,  
it's hard to see how one would get faster than the builtin  
implementations of all and any, short of getting rid of the Python call.

For (some) builtlins, it might be interesting to extract and call the  
C function pointer directly. This could give a good speedup for  
builtins...

- Robert


From dalcinl at gmail.com  Fri Jul 24 17:21:21 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 24 Jul 2009 12:21:21 -0300
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
	<59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
	<D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>
Message-ID: <e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>

On Fri, Jul 24, 2009 at 8:53 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> This feels a lot like C++ operator overloading...
>
> I'm not a fan of _as_parameter_, but maybe an "as foo_t" kind of
> attribute that it would use every time it needed to convert it to a
> foo_t (not just as a parameter, but in assignment, etc.) Certainly
> something like this could be useful, but I'm not sure if it's needed.
>

Well, such feature would make mpi4py/petsc4py/slepc4py/tao4py codebase
simpler... All my cdef classes are proxies to C handles... I'm
constantly calling "self.handle" when calling library routines...


> On Jul 23, 2009, at 8:29 PM, David P. Novakovic wrote:
>
>> http://docs.python.org/library/ctypes.html#calling-functions-with-
>> your-own-custom-data-types
>>
>> I notice it only accepts basic types... and obviously the
>> implementation would be a lot simpler than what I am talking about
>> for cython... but that's the general gist of it.
>>
>> I guess for cython it would be nice to be able to have a mapping of
>> type -> attribute (or type_name -> attribute_name) so that when the
>> class is passed as a parameter the attribute of the right type is
>> passed instead.
>>
>> David
>>
>> On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin
>> <dalcinl at gmail.com> wrote:
>> On Thu, Jul 23, 2009 at 11:44 PM, David P.
>> Novakovic<davidnovakovic at gmail.com> wrote:
>> >
>> > It's already a feature in the python stdlib through ctypes, so it
>> obviously
>> > doesn't break the guidelines that much.
>> >
>>
>> Sorry for my ignorance about ctypes... But could you point me to some
>> code/link or provide some more comments in order I can understand what
>> you commented above?
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From davidnovakovic at gmail.com  Fri Jul 24 23:48:59 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Sat, 25 Jul 2009 07:48:59 +1000
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
	<59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
	<D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>
	<e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>
Message-ID: <59d13e7d0907241448j7408f75fo32d9b67e011dc483@mail.gmail.com>

It could be made optional..

I'm not sure about the parser rules for .pyx... but something like the
following would allow a single wrapper class to satisfy several wrapped
types. The above named param would work too, as long as people understand
where they could create attribute name conflicts.. (perhaps not as backwards
compatible)

cdef class Wrapper:
    cdef foo_t * foo

    _cython_type_map = {  #optional
    'foo_t *': foo
    }

It's not really a requirement of mine to be able to map multiple types, but
if you were going to add it you might as well give people the option.

Alternatively it could just work for a single type too.

Anyway, just my thoughts :)

David

On Sat, Jul 25, 2009 at 1:21 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> On Fri, Jul 24, 2009 at 8:53 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
> > This feels a lot like C++ operator overloading...
> >
> > I'm not a fan of _as_parameter_, but maybe an "as foo_t" kind of
> > attribute that it would use every time it needed to convert it to a
> > foo_t (not just as a parameter, but in assignment, etc.) Certainly
> > something like this could be useful, but I'm not sure if it's needed.
> >
>
> Well, such feature would make mpi4py/petsc4py/slepc4py/tao4py codebase
> simpler... All my cdef classes are proxies to C handles... I'm
> constantly calling "self.handle" when calling library routines...
>
>
> > On Jul 23, 2009, at 8:29 PM, David P. Novakovic wrote:
> >
> >> http://docs.python.org/library/ctypes.html#calling-functions-with-
> >> your-own-custom-data-types
> >>
> >> I notice it only accepts basic types... and obviously the
> >> implementation would be a lot simpler than what I am talking about
> >> for cython... but that's the general gist of it.
> >>
> >> I guess for cython it would be nice to be able to have a mapping of
> >> type -> attribute (or type_name -> attribute_name) so that when the
> >> class is passed as a parameter the attribute of the right type is
> >> passed instead.
> >>
> >> David
> >>
> >> On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin
> >> <dalcinl at gmail.com> wrote:
> >> On Thu, Jul 23, 2009 at 11:44 PM, David P.
> >> Novakovic<davidnovakovic at gmail.com> wrote:
> >> >
> >> > It's already a feature in the python stdlib through ctypes, so it
> >> obviously
> >> > doesn't break the guidelines that much.
> >> >
> >>
> >> Sorry for my ignorance about ctypes... But could you point me to some
> >> code/link or provide some more comments in order I can understand what
> >> you commented above?
> >>
> >>
> >>
> >> --
> >> Lisandro Dalc?n
> >> ---------------
> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> Tel/Fax: +54-(0)342-451.1594
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >>
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090725/f0cd065d/attachment.htm 

From jason-sage at creativetrax.com  Sat Jul 25 04:40:44 2009
From: jason-sage at creativetrax.com (Jason Grout)
Date: Fri, 24 Jul 2009 19:40:44 -0700
Subject: [Cython] any() and all()
In-Reply-To: <96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>
References: <h4bk19$50q$1@ger.gmane.org>
	<96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>
Message-ID: <h4drbc$hh2$1@ger.gmane.org>

Robert Bradshaw wrote:
> On Jul 23, 2009, at 11:23 PM, Jason Grout wrote:
> 
>> Is it possible to support the any() and all() functions?  I'm using  
>> the
>> most recent version of Cython in Sage, so forgive me if it is already
>> supported in Cython, but just not in Sage's version yet.
> 
> It's working for me...
> 
> {{{
> %cython
> print any([0,0,1,0]), all([0,0,1,0])
> ///
> (True, False)
> }}}
> 


Not if you use comprehension-like syntax.  Can Cython support that in 
the same way it supports list/set/dict comprehensions?

Note that in lots of cases, having any/all only support lists kind of 
defeats the purpose of the short-cutting.

{{{id=0|
%cython
print all(i>=0 for i in range(10))
print any(i<0 for i in range(10))
///

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File 
"/home/sage/sagenb/sage_notebook/worksheets/jason3/174/code/4.py", line 
6, in <module>
 
_support_.cython_import_all("/home/sage/sagenb/sage_notebook/worksheets/jason3/174/code/sage5.spyx", 
globals())
   File 
"/home/sage/sage_install/sage/local/lib/python2.6/site-packages/sage/server/support.py", 
line 411, in cython_import_all
     create_local_c_file=create_local_c_file)
   File 
"/home/sage/sage_install/sage/local/lib/python2.6/site-packages/sage/server/support.py", 
line 388, in cython_import
     create_local_c_file=create_local_c_file)
   File 
"/home/sage/sage_install/sage/local/lib/python2.6/site-packages/sage/misc/cython.py", 
line 367, in cython
     raise RuntimeError, "Error converting %s to C:\n%s\n%s"%(filename, 
log, err)
RuntimeError: Error converting 
/home/sage/sagenb/sage_notebook/worksheets/jason3/174/code/sage5.spyx to C:


Error converting Pyrex file to C:
------------------------------------------------------------
...

include "interrupt.pxi"  # ctrl-c interrupt block support
include "stdsage.pxi"  # ctrl-c interrupt block support

include "cdefs.pxi"
print all(i>=0 for i in range(10))
               ^
------------------------------------------------------------

/home/worksheet/.sage/temp/sagenb/18730/spyx/_home_sage_sagenb_sage_notebook_worksheets_jason3_174_code_sage5_spyx/_home_sage_sagenb_sage_notebook_worksheets_jason3_174_code_sage5_spyx_0.pyx:6:15: 
Expected ')'
}}}

{{{id=1|
print all(i>=0 for i in range(10))
print any(i<0 for i in range(10))
///

True
False
}}}







>> I realize that it probably would be easy to write such functions, but
>> since they are standard python functions, it would be nice if they  
>> were
>> recognized by Cython.  It would be super nice if they were *fast*  
>> too :).
> 
> This was actually discussed recently, and yes, we'd like to support  
> this kind of thing. It shouldn't be too hard to do either. However,  
> it's hard to see how one would get faster than the builtin  
> implementations of all and any, short of getting rid of the Python call.

Okay.  It would be enough for me if it just didn't error out on seeing 
any/all with comprehension like syntax.  Then I wouldn't have to 
obfuscate my code when converting from python.


Thanks!

Jason


From dalcinl at gmail.com  Sat Jul 25 05:15:09 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 25 Jul 2009 00:15:09 -0300
Subject: [Cython] any() and all()
In-Reply-To: <h4drbc$hh2$1@ger.gmane.org>
References: <h4bk19$50q$1@ger.gmane.org>
	<96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>
	<h4drbc$hh2$1@ger.gmane.org>
Message-ID: <e7ba66e40907242015p62da079drc1b5b204c4c3864c@mail.gmail.com>

On Fri, Jul 24, 2009 at 11:40 PM, Jason
Grout<jason-sage at creativetrax.com> wrote:
>
> Not if you use comprehension-like syntax. ?Can Cython support that in
> the same way it supports list/set/dict comprehensions?
>
> Note that in lots of cases, having any/all only support lists kind of
> defeats the purpose of the short-cutting.
>
> {{{id=0|
> %cython
> print all(i>=0 for i in range(10))
> print any(i<0 for i in range(10))
> ///
>

Wait a minute... this does not have nothing to do with any/all. For
this to work, Cython have to support GENERATORS...

Robert, Is your work on closures related to this?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jason-sage at creativetrax.com  Sat Jul 25 06:41:57 2009
From: jason-sage at creativetrax.com (Jason Grout)
Date: Fri, 24 Jul 2009 21:41:57 -0700
Subject: [Cython] any() and all()
In-Reply-To: <e7ba66e40907242015p62da079drc1b5b204c4c3864c@mail.gmail.com>
References: <h4bk19$50q$1@ger.gmane.org>	<96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>	<h4drbc$hh2$1@ger.gmane.org>
	<e7ba66e40907242015p62da079drc1b5b204c4c3864c@mail.gmail.com>
Message-ID: <h4e2em$s15$1@ger.gmane.org>

Lisandro Dalcin wrote:
> On Fri, Jul 24, 2009 at 11:40 PM, Jason
> Grout<jason-sage at creativetrax.com> wrote:
>> Not if you use comprehension-like syntax.  Can Cython support that in
>> the same way it supports list/set/dict comprehensions?
>>
>> Note that in lots of cases, having any/all only support lists kind of
>> defeats the purpose of the short-cutting.
>>
>> {{{id=0|
>> %cython
>> print all(i>=0 for i in range(10))
>> print any(i<0 for i in range(10))
>> ///
>>
> 
> Wait a minute... this does not have nothing to do with any/all. For
> this to work, Cython have to support GENERATORS...


Okay.  I was hoping that supporting

all(i>=0 for i in range(10))

would be similar to how Cython currently supports

[i>=0 for i in range(10)]


I realize that generators have been a long-requested feature in Cython. 
  To be honest, I was surprised that list/set/dict comprehensions 
worked.    I was hoping it was a small step from there to supporting any 
and all statements like the above.

However, I also really don't know what I'm talking about, so if the 
above doesn't make sense, sorry.

Thanks,

Jason


From robertwb at math.washington.edu  Sat Jul 25 12:42:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Jul 2009 03:42:50 -0700
Subject: [Cython] any() and all()
In-Reply-To: <h4e2em$s15$1@ger.gmane.org>
References: <h4bk19$50q$1@ger.gmane.org>	<96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>	<h4drbc$hh2$1@ger.gmane.org>
	<e7ba66e40907242015p62da079drc1b5b204c4c3864c@mail.gmail.com>
	<h4e2em$s15$1@ger.gmane.org>
Message-ID: <33978C9A-A0EC-4900-A155-EB60677B0638@math.washington.edu>

On Jul 24, 2009, at 9:41 PM, Jason Grout wrote:

> Lisandro Dalcin wrote:
>> On Fri, Jul 24, 2009 at 11:40 PM, Jason
>> Grout<jason-sage at creativetrax.com> wrote:
>>> Not if you use comprehension-like syntax.  Can Cython support  
>>> that in
>>> the same way it supports list/set/dict comprehensions?
>>>
>>> Note that in lots of cases, having any/all only support lists  
>>> kind of
>>> defeats the purpose of the short-cutting.
>>>
>>> {{{id=0|
>>> %cython
>>> print all(i>=0 for i in range(10))
>>> print any(i<0 for i in range(10))
>>> ///
>>>
>>
>> Wait a minute... this does not have nothing to do with any/all. For
>> this to work, Cython have to support GENERATORS...
>

Exactly.

> Okay.  I was hoping that supporting
>
> all(i>=0 for i in range(10))
>
> would be similar to how Cython currently supports
>
> [i>=0 for i in range(10)]
>
>
> I realize that generators have been a long-requested feature in  
> Cython.

They're closer now than they ever have been, but the last month or  
two what little time I've had to work on Cython has mostly been  
related to the GSoC projects.

>   To be honest, I was surprised that list/set/dict comprehensions
> worked.    I was hoping it was a small step from there to  
> supporting any
> and all statements like the above.
>
> However, I also really don't know what I'm talking about, so if the
> above doesn't make sense, sorry.


This would be possible to support, but it's a rather special case.  
One would also, perhaps want to support things like sum. Even if we  
had generators, it's unclear (but not impossible) how they'd work  
with non-object return types--maybe a next_c method? In the short  
run, it probably wouldn't be that bad of an addition. Aside from any,  
all, and sum, what other builtins would we like to support in this  
manner?

- Robert

From jason-sage at creativetrax.com  Sat Jul 25 19:10:57 2009
From: jason-sage at creativetrax.com (Jason Grout)
Date: Sat, 25 Jul 2009 10:10:57 -0700
Subject: [Cython] any() and all()
In-Reply-To: <33978C9A-A0EC-4900-A155-EB60677B0638@math.washington.edu>
References: <h4bk19$50q$1@ger.gmane.org>	<96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>	<h4drbc$hh2$1@ger.gmane.org>	<e7ba66e40907242015p62da079drc1b5b204c4c3864c@mail.gmail.com>	<h4e2em$s15$1@ger.gmane.org>
	<33978C9A-A0EC-4900-A155-EB60677B0638@math.washington.edu>
Message-ID: <h4feb1$tr0$1@ger.gmane.org>

Robert Bradshaw wrote:

> 
> This would be possible to support, but it's a rather special case.  
> One would also, perhaps want to support things like sum. Even if we  
> had generators, it's unclear (but not impossible) how they'd work  
> with non-object return types--maybe a next_c method? In the short  
> run, it probably wouldn't be that bad of an addition. Aside from any,  
> all, and sum, what other builtins would we like to support in this  
> manner?


I also use enumerate, min, and max a lot.  If it's not too hard, it'd be 
great to support these using the above comprehension-like syntax as well.

Thanks,

Jason



From sccolbert at gmail.com  Sat Jul 25 19:41:08 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Sat, 25 Jul 2009 13:41:08 -0400
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
	<59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
	<D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>
	<e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>
Message-ID: <7f014ea60907251041o27220469kda85bb4296588092@mail.gmail.com>

+1

On Fri, Jul 24, 2009 at 11:21 AM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> On Fri, Jul 24, 2009 at 8:53 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> This feels a lot like C++ operator overloading...
>>
>> I'm not a fan of _as_parameter_, but maybe an "as foo_t" kind of
>> attribute that it would use every time it needed to convert it to a
>> foo_t (not just as a parameter, but in assignment, etc.) Certainly
>> something like this could be useful, but I'm not sure if it's needed.
>>
>
> Well, such feature would make mpi4py/petsc4py/slepc4py/tao4py codebase
> simpler... All my cdef classes are proxies to C handles... I'm
> constantly calling "self.handle" when calling library routines...
>
>
>> On Jul 23, 2009, at 8:29 PM, David P. Novakovic wrote:
>>
>>> http://docs.python.org/library/ctypes.html#calling-functions-with-
>>> your-own-custom-data-types
>>>
>>> I notice it only accepts basic types... and obviously the
>>> implementation would be a lot simpler than what I am talking about
>>> for cython... but that's the general gist of it.
>>>
>>> I guess for cython it would be nice to be able to have a mapping of
>>> type -> attribute (or type_name -> attribute_name) so that when the
>>> class is passed as a parameter the attribute of the right type is
>>> passed instead.
>>>
>>> David
>>>
>>> On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin
>>> <dalcinl at gmail.com> wrote:
>>> On Thu, Jul 23, 2009 at 11:44 PM, David P.
>>> Novakovic<davidnovakovic at gmail.com> wrote:
>>> >
>>> > It's already a feature in the python stdlib through ctypes, so it
>>> obviously
>>> > doesn't break the guidelines that much.
>>> >
>>>
>>> Sorry for my ignorance about ctypes... But could you point me to some
>>> code/link or provide some more comments in order I can understand what
>>> you commented above?
>>>
>>>
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Sat Jul 25 21:21:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Jul 2009 12:21:27 -0700
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <7f014ea60907251041o27220469kda85bb4296588092@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
	<59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
	<D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>
	<e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>
	<7f014ea60907251041o27220469kda85bb4296588092@mail.gmail.com>
Message-ID: <3F575B00-CA06-42D1-9B6E-911375C59131@math.washington.edu>

Well, I propose someone write up a CEP. I'm not opposed to the idea  
but in particular I have yet to see a syntax that I'd be happy with.

- Robert


On Jul 25, 2009, at 10:41 AM, Chris Colbert wrote:

> +1
>
> On Fri, Jul 24, 2009 at 11:21 AM, Lisandro  
> Dalcin<dalcinl at gmail.com> wrote:
>> On Fri, Jul 24, 2009 at 8:53 AM, Robert
>> Bradshaw<robertwb at math.washington.edu> wrote:
>>> This feels a lot like C++ operator overloading...
>>>
>>> I'm not a fan of _as_parameter_, but maybe an "as foo_t" kind of
>>> attribute that it would use every time it needed to convert it to a
>>> foo_t (not just as a parameter, but in assignment, etc.) Certainly
>>> something like this could be useful, but I'm not sure if it's  
>>> needed.
>>>
>>
>> Well, such feature would make mpi4py/petsc4py/slepc4py/tao4py  
>> codebase
>> simpler... All my cdef classes are proxies to C handles... I'm
>> constantly calling "self.handle" when calling library routines...
>>
>>
>>> On Jul 23, 2009, at 8:29 PM, David P. Novakovic wrote:
>>>
>>>> http://docs.python.org/library/ctypes.html#calling-functions-with-
>>>> your-own-custom-data-types
>>>>
>>>> I notice it only accepts basic types... and obviously the
>>>> implementation would be a lot simpler than what I am talking about
>>>> for cython... but that's the general gist of it.
>>>>
>>>> I guess for cython it would be nice to be able to have a mapping of
>>>> type -> attribute (or type_name -> attribute_name) so that when the
>>>> class is passed as a parameter the attribute of the right type is
>>>> passed instead.
>>>>
>>>> David
>>>>
>>>> On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin
>>>> <dalcinl at gmail.com> wrote:
>>>> On Thu, Jul 23, 2009 at 11:44 PM, David P.
>>>> Novakovic<davidnovakovic at gmail.com> wrote:
>>>>>
>>>>> It's already a feature in the python stdlib through ctypes, so it
>>>> obviously
>>>>> doesn't break the guidelines that much.
>>>>>
>>>>
>>>> Sorry for my ignorance about ctypes... But could you point me to  
>>>> some
>>>> code/link or provide some more comments in order I can  
>>>> understand what
>>>> you commented above?
>>>>
>>>>
>>>>
>>>> --
>>>> Lisandro Dalc?n
>>>> ---------------
>>>> Centro Internacional de M?todos Computacionales en Ingenier?a  
>>>> (CIMEC)
>>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica  
>>>> (INTEC)
>>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas  
>>>> (CONICET)
>>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>>> Tel/Fax: +54-(0)342-451.1594
>>>> _______________________________________________
>>>> Cython-dev mailing list
>>>> Cython-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>>
>>>> _______________________________________________
>>>> Cython-dev mailing list
>>>> Cython-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Sat Jul 25 21:26:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 25 Jul 2009 12:26:24 -0700
Subject: [Cython] any() and all()
In-Reply-To: <h4feb1$tr0$1@ger.gmane.org>
References: <h4bk19$50q$1@ger.gmane.org>	<96889859-43B5-40E0-8DFE-40E511F0F0D0@math.washington.edu>	<h4drbc$hh2$1@ger.gmane.org>	<e7ba66e40907242015p62da079drc1b5b204c4c3864c@mail.gmail.com>	<h4e2em$s15$1@ger.gmane.org>
	<33978C9A-A0EC-4900-A155-EB60677B0638@math.washington.edu>
	<h4feb1$tr0$1@ger.gmane.org>
Message-ID: <284434A4-7329-4BAE-9140-451229876EA7@math.washington.edu>

On Jul 25, 2009, at 10:10 AM, Jason Grout wrote:

> Robert Bradshaw wrote:
>
>>
>> This would be possible to support, but it's a rather special case.
>> One would also, perhaps want to support things like sum. Even if we
>> had generators, it's unclear (but not impossible) how they'd work
>> with non-object return types--maybe a next_c method? In the short
>> run, it probably wouldn't be that bad of an addition. Aside from any,
>> all, and sum, what other builtins would we like to support in this
>> manner?
>
>
> I also use enumerate, min, and max a lot.  If it's not too hard,  
> it'd be
> great to support these using the above comprehension-like syntax as  
> well.

Yeah, enumerate has certainly been on the list, and min/max would be  
useful too.

- Robert



From davidnovakovic at gmail.com  Mon Jul 27 02:02:49 2009
From: davidnovakovic at gmail.com (David P. Novakovic)
Date: Mon, 27 Jul 2009 10:02:49 +1000
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <3F575B00-CA06-42D1-9B6E-911375C59131@math.washington.edu>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
	<59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
	<D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>
	<e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>
	<7f014ea60907251041o27220469kda85bb4296588092@mail.gmail.com>
	<3F575B00-CA06-42D1-9B6E-911375C59131@math.washington.edu>
Message-ID: <59d13e7d0907261702g701593b2yc8b3ae6188dab571@mail.gmail.com>

OK, when I get some time I'll throw some thoughts together. I'm assuming the
enhancement page on the wiki is the best place to drop it..

David

On Sun, Jul 26, 2009 at 5:21 AM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> Well, I propose someone write up a CEP. I'm not opposed to the idea
> but in particular I have yet to see a syntax that I'd be happy with.
>
> - Robert
>
>
> On Jul 25, 2009, at 10:41 AM, Chris Colbert wrote:
>
> > +1
> >
> > On Fri, Jul 24, 2009 at 11:21 AM, Lisandro
> > Dalcin<dalcinl at gmail.com> wrote:
> >> On Fri, Jul 24, 2009 at 8:53 AM, Robert
> >> Bradshaw<robertwb at math.washington.edu> wrote:
> >>> This feels a lot like C++ operator overloading...
> >>>
> >>> I'm not a fan of _as_parameter_, but maybe an "as foo_t" kind of
> >>> attribute that it would use every time it needed to convert it to a
> >>> foo_t (not just as a parameter, but in assignment, etc.) Certainly
> >>> something like this could be useful, but I'm not sure if it's
> >>> needed.
> >>>
> >>
> >> Well, such feature would make mpi4py/petsc4py/slepc4py/tao4py
> >> codebase
> >> simpler... All my cdef classes are proxies to C handles... I'm
> >> constantly calling "self.handle" when calling library routines...
> >>
> >>
> >>> On Jul 23, 2009, at 8:29 PM, David P. Novakovic wrote:
> >>>
> >>>> http://docs.python.org/library/ctypes.html#calling-functions-with-
> >>>> your-own-custom-data-types
> >>>>
> >>>> I notice it only accepts basic types... and obviously the
> >>>> implementation would be a lot simpler than what I am talking about
> >>>> for cython... but that's the general gist of it.
> >>>>
> >>>> I guess for cython it would be nice to be able to have a mapping of
> >>>> type -> attribute (or type_name -> attribute_name) so that when the
> >>>> class is passed as a parameter the attribute of the right type is
> >>>> passed instead.
> >>>>
> >>>> David
> >>>>
> >>>> On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin
> >>>> <dalcinl at gmail.com> wrote:
> >>>> On Thu, Jul 23, 2009 at 11:44 PM, David P.
> >>>> Novakovic<davidnovakovic at gmail.com> wrote:
> >>>>>
> >>>>> It's already a feature in the python stdlib through ctypes, so it
> >>>> obviously
> >>>>> doesn't break the guidelines that much.
> >>>>>
> >>>>
> >>>> Sorry for my ignorance about ctypes... But could you point me to
> >>>> some
> >>>> code/link or provide some more comments in order I can
> >>>> understand what
> >>>> you commented above?
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Lisandro Dalc?n
> >>>> ---------------
> >>>> Centro Internacional de M?todos Computacionales en Ingenier?a
> >>>> (CIMEC)
> >>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica
> >>>> (INTEC)
> >>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas
> >>>> (CONICET)
> >>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >>>> Tel/Fax: +54-(0)342-451.1594
> >>>> _______________________________________________
> >>>> Cython-dev mailing list
> >>>> Cython-dev at codespeak.net
> >>>> http://codespeak.net/mailman/listinfo/cython-dev
> >>>>
> >>>> _______________________________________________
> >>>> Cython-dev mailing list
> >>>> Cython-dev at codespeak.net
> >>>> http://codespeak.net/mailman/listinfo/cython-dev
> >>>
> >>> _______________________________________________
> >>> Cython-dev mailing list
> >>> Cython-dev at codespeak.net
> >>> http://codespeak.net/mailman/listinfo/cython-dev
> >>>
> >>
> >>
> >>
> >> --
> >> Lisandro Dalc?n
> >> ---------------
> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> Tel/Fax: +54-(0)342-451.1594
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >>
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090727/06631af1/attachment-0001.htm 

From robertwb at math.washington.edu  Mon Jul 27 11:00:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 27 Jul 2009 02:00:59 -0700
Subject: [Cython] On the topic of wrapper classes
In-Reply-To: <59d13e7d0907261702g701593b2yc8b3ae6188dab571@mail.gmail.com>
References: <59d13e7d0907231517r76ee6344kf2394a93a68f734d@mail.gmail.com>
	<e7ba66e40907231920m4444a227w2bd7d734a0ddbe97@mail.gmail.com>
	<59d13e7d0907231944j26d62f13g8e2f8ecd6cc36bc7@mail.gmail.com>
	<e7ba66e40907232023k4e9a2f27ue6fe60b7209e0623@mail.gmail.com>
	<59d13e7d0907232029m767c17f2w845c1f58537ae2e3@mail.gmail.com>
	<D06A9EF2-E9A0-4443-AAA3-B5326C87A4BD@math.washington.edu>
	<e7ba66e40907240821x5b0f4630o619a82970f939e45@mail.gmail.com>
	<7f014ea60907251041o27220469kda85bb4296588092@mail.gmail.com>
	<3F575B00-CA06-42D1-9B6E-911375C59131@math.washington.edu>
	<59d13e7d0907261702g701593b2yc8b3ae6188dab571@mail.gmail.com>
Message-ID: <AFBA3861-690D-4334-834F-5ED0DE84902A@math.washington.edu>

On Jul 26, 2009, at 5:02 PM, David P. Novakovic wrote:

> OK, when I get some time I'll throw some thoughts together.

Great.

> I'm assuming the enhancement page on the wiki is the best place to  
> drop it..

Yep, that would be the place.

>
>
> David
>
> On Sun, Jul 26, 2009 at 5:21 AM, Robert Bradshaw  
> <robertwb at math.washington.edu> wrote:
> Well, I propose someone write up a CEP. I'm not opposed to the idea
> but in particular I have yet to see a syntax that I'd be happy with.
>
> - Robert
>
>
> On Jul 25, 2009, at 10:41 AM, Chris Colbert wrote:
>
> > +1
> >
> > On Fri, Jul 24, 2009 at 11:21 AM, Lisandro
> > Dalcin<dalcinl at gmail.com> wrote:
> >> On Fri, Jul 24, 2009 at 8:53 AM, Robert
> >> Bradshaw<robertwb at math.washington.edu> wrote:
> >>> This feels a lot like C++ operator overloading...
> >>>
> >>> I'm not a fan of _as_parameter_, but maybe an "as foo_t" kind of
> >>> attribute that it would use every time it needed to convert it  
> to a
> >>> foo_t (not just as a parameter, but in assignment, etc.) Certainly
> >>> something like this could be useful, but I'm not sure if it's
> >>> needed.
> >>>
> >>
> >> Well, such feature would make mpi4py/petsc4py/slepc4py/tao4py
> >> codebase
> >> simpler... All my cdef classes are proxies to C handles... I'm
> >> constantly calling "self.handle" when calling library routines...
> >>
> >>
> >>> On Jul 23, 2009, at 8:29 PM, David P. Novakovic wrote:
> >>>
> >>>> http://docs.python.org/library/ctypes.html#calling-functions- 
> with-
> >>>> your-own-custom-data-types
> >>>>
> >>>> I notice it only accepts basic types... and obviously the
> >>>> implementation would be a lot simpler than what I am talking  
> about
> >>>> for cython... but that's the general gist of it.
> >>>>
> >>>> I guess for cython it would be nice to be able to have a  
> mapping of
> >>>> type -> attribute (or type_name -> attribute_name) so that  
> when the
> >>>> class is passed as a parameter the attribute of the right type is
> >>>> passed instead.
> >>>>
> >>>> David
> >>>>
> >>>> On Fri, Jul 24, 2009 at 1:23 PM, Lisandro Dalcin
> >>>> <dalcinl at gmail.com> wrote:
> >>>> On Thu, Jul 23, 2009 at 11:44 PM, David P.
> >>>> Novakovic<davidnovakovic at gmail.com> wrote:
> >>>>>
> >>>>> It's already a feature in the python stdlib through ctypes,  
> so it
> >>>> obviously
> >>>>> doesn't break the guidelines that much.
> >>>>>
> >>>>
> >>>> Sorry for my ignorance about ctypes... But could you point me to
> >>>> some
> >>>> code/link or provide some more comments in order I can
> >>>> understand what
> >>>> you commented above?
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Lisandro Dalc?n
> >>>> ---------------
> >>>> Centro Internacional de M?todos Computacionales en Ingenier?a
> >>>> (CIMEC)
> >>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica
> >>>> (INTEC)
> >>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas
> >>>> (CONICET)
> >>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >>>> Tel/Fax: +54-(0)342-451.1594
> >>>> _______________________________________________
> >>>> Cython-dev mailing list
> >>>> Cython-dev at codespeak.net
> >>>> http://codespeak.net/mailman/listinfo/cython-dev
> >>>>
> >>>> _______________________________________________
> >>>> Cython-dev mailing list
> >>>> Cython-dev at codespeak.net
> >>>> http://codespeak.net/mailman/listinfo/cython-dev
> >>>
> >>> _______________________________________________
> >>> Cython-dev mailing list
> >>> Cython-dev at codespeak.net
> >>> http://codespeak.net/mailman/listinfo/cython-dev
> >>>
> >>
> >>
> >>
> >> --
> >> Lisandro Dalc?n
> >> ---------------
> >> Centro Internacional de M?todos Computacionales en Ingenier?a  
> (CIMEC)
> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica  
> (INTEC)
> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas  
> (CONICET)
> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> Tel/Fax: +54-(0)342-451.1594
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >>
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From gpk at kochanski.org  Tue Jul 28 00:41:18 2009
From: gpk at kochanski.org (Greg Kochanski)
Date: Mon, 27 Jul 2009 23:41:18 +0100
Subject: [Cython] Design to minimize user errors: __deallocate__
Message-ID: <4A6E2D0E.80909@kochanski.org>

I'd like to propose that if an extension class has a
__cinit__() member, it must have a __dealloc__()
member.  The compiler should raise an error
or at least a warning.

I realize this will annoy a few people in the odd case or
two where one isn't necessary, but the problem is
that it is all too easy to write code like this:


cdef class foo:
   cdef double *x

   def __init__(self, whatever):
         stuff

  def __cinit__(self, whatver):
        self.x = <double *>calloc(...)

  def __deallocate__(self):
       free(self.x)


Such a class works perfectly, until you notice the memory
leak.    (Because, __deallocate__() isn't a special member
even though the name is very suggestive, so free() never
gets called.)

I've done this twice now.   The first one required several hours
of debugging and a bicycle ride in the rain to reboot my
thrashing computer at work.   The second one took less time,
but it's still a very easy mistake to make and an annoying one
to find.

People might also think of __cdel__/__cinit__ by analogy with
python's __del__/__init__.


From charleshixsn at earthlink.net  Tue Jul 28 00:55:21 2009
From: charleshixsn at earthlink.net (Charles Hixson)
Date: Mon, 27 Jul 2009 15:55:21 -0700
Subject: [Cython] suggested added example
Message-ID: <4A6E3059.1090509@earthlink.net>

FWIW, I think the Cython would become more accessible if there were 
examples of interfacing Cython to standard C library calls.  In 
particular for i/o.  (As in how to write a cint to a file.)

I'm fairly certain from observing the discussion that many consider this 
trivially obvious, but I was not able to figure it out in the time I had 
to evaluate Cython.  (And Python files wouldn't do, because I needed to 
do i/o to C compatible files written in binary mode.  [Yeah, on Linux 
it's not supposed to make an difference.  But the binary representation 
of a long or a short IS different from the Python representation.])



From greg.ewing at canterbury.ac.nz  Tue Jul 28 02:09:55 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 28 Jul 2009 12:09:55 +1200
Subject: [Cython] suggested added example
In-Reply-To: <4A6E3059.1090509@earthlink.net>
References: <4A6E3059.1090509@earthlink.net>
Message-ID: <4A6E41D3.7010900@canterbury.ac.nz>

Charles Hixson wrote:
> Python files wouldn't do, because I needed to 
> do i/o to C compatible files written in binary mode.  [Yeah, on Linux 
> it's not supposed to make an difference.  But the binary representation 
> of a long or a short IS different from the Python representation.])

I'm not sure what the problem is. The struct module uses
the platform's native byte order by default for ints and
longs, or lets you specify big-endian or little-endian.

If you want to do it with Cython, then you're really
doing it in C, and teaching people how to do I/O in C
is not Cython's responsibility.

-- 
Greg

From dalcinl at gmail.com  Tue Jul 28 02:21:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 27 Jul 2009 21:21:02 -0300
Subject: [Cython] suggested added example
In-Reply-To: <4A6E3059.1090509@earthlink.net>
References: <4A6E3059.1090509@earthlink.net>
Message-ID: <e7ba66e40907271721t3782f5a2ob1b1360f91f6d7a0@mail.gmail.com>

Sorry, I'm a bit confused... Did you managed to implement the raw I/O
in C? What kind problems are you experiencing? Or you are just asking
for more comprehensive documentation for stuff you already managed to
get it working?

On Mon, Jul 27, 2009 at 7:55 PM, Charles
Hixson<charleshixsn at earthlink.net> wrote:
> FWIW, I think the Cython would become more accessible if there were
> examples of interfacing Cython to standard C library calls. ?In
> particular for i/o. ?(As in how to write a cint to a file.)
>
> I'm fairly certain from observing the discussion that many consider this
> trivially obvious, but I was not able to figure it out in the time I had
> to evaluate Cython. ?(And Python files wouldn't do, because I needed to
> do i/o to C compatible files written in binary mode. ?[Yeah, on Linux
> it's not supposed to make an difference. ?But the binary representation
> of a long or a short IS different from the Python representation.])
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue Jul 28 11:12:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 28 Jul 2009 02:12:07 -0700
Subject: [Cython] Design to minimize user errors: __deallocate__
In-Reply-To: <4A6E2D0E.80909@kochanski.org>
References: <4A6E2D0E.80909@kochanski.org>
Message-ID: <C289EA5E-43CE-4622-BCC2-E7282060730C@math.washington.edu>

On Jul 27, 2009, at 3:41 PM, Greg Kochanski wrote:

> I'd like to propose that if an extension class has a
> __cinit__() member, it must have a __dealloc__()
> member.  The compiler should raise an error
> or at least a warning.

Certainly not an error--maybe a warning.

> I realize this will annoy a few people in the odd case or
> two where one isn't necessary, but the problem is
> that it is all too easy to write code like this:
>
>
> cdef class foo:
>    cdef double *x
>
>    def __init__(self, whatever):
>          stuff
>
>   def __cinit__(self, whatver):
>         self.x = <double *>calloc(...)
>
>   def __deallocate__(self):
>        free(self.x)
>
>
> Such a class works perfectly, until you notice the memory
> leak.    (Because, __deallocate__() isn't a special member
> even though the name is very suggestive, so free() never
> gets called.)
>
> I've done this twice now.   The first one required several hours
> of debugging and a bicycle ride in the rain to reboot my
> thrashing computer at work.   The second one took less time,
> but it's still a very easy mistake to make and an annoying one
> to find.
>
> People might also think of __cdel__/__cinit__ by analogy with
> python's __del__/__init__.

I don't think the analogy is quite as direct, because __del__ is  
optional, rare, and slightly discouraged.

- Robert



From dagss at student.matnat.uio.no  Wed Jul 29 18:46:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 29 Jul 2009 18:46:53 +0200
Subject: [Cython] [sage-support] Re: changeing sage -b (cython)
In-Reply-To: <79B14468-6B58-41BC-A5C3-7D4BF8686802@math.washington.edu>
References: <c83b3d1c-ebbc-4d11-8cd3-fd64aa8aaa99@26g2000yqk.googlegroups.com>	<f67335450907171237l80a3621l6f3ce5e57ca6dc39@mail.gmail.com>	<6ebc39c0-8255-4577-89ff-8ab8e1886525@y10g2000prf.googlegroups.com>	<20d43422-df88-471c-b903-8c7f487ad577@24g2000yqm.googlegroups.com>	<d09943d2-9173-4789-838c-fdb59de2a86c@i18g2000pro.googlegroups.com>	<F0F8184B-218F-4444-95A0-89983D68FB78@math.washington.edu>	<92dae4b1-e803-4d4f-b5cb-3989a058b7e5@z4g2000prh.googlegroups.com>	<85e81ba30907221127w53a20ab5pc2bc26b1968ee20b@mail.gmail.com>	<5689E418-9BBD-4359-9B14-73174723821C@math.washington.edu>	<9418b9470907221139p1885fa35gee164559ec9704d8@mail.gmail.com>	<95ee2dab-113c-45c6-87cb-dbf4017c633b@a7g2000yqk.googlegroups.com>
	<79B14468-6B58-41BC-A5C3-7D4BF8686802@math.washington.edu>
Message-ID: <4A707CFD.3000207@student.matnat.uio.no>

Robert Bradshaw wrote:
> I'm not sure if or when either of these will be available, but  
> neither are trivial. Both questions are probably better asked on the  
> cython lists.
> 
> - Robert
> 
> 
> On Jul 22, 2009, at 12:17 PM, Ethan Van Andel wrote:
> 
>> Robert,
>>
>> Is there any prediction for when numpy complex types will work?
>> (outside of the notebook, when compiling via sage -b)

I assume this will happen when Cython is upgraded in Sage, i.e. #6438 / 
4.1.1. If it doesn't, please send cython-dev (or me if you can't be 
bothered) a mail.

>> Yet another question:
>>
>> When I compile my class, I get something like this:
>>
>> cdef class Riemann_Map:
>>     cdef int N, B, ncorners
>>     cdef f
>>     cdef opp
>>     cdef double complex a
>>     cdef np.ndarray[float,ndim = 1] tester
>>                   ^
>> ------------------------------------------------------------
>>
>> /home/evlutte/opt/sage-3.2.3/devel/sage-main/sage/plot/riemann.pyx:
>> 39:19: Buffer types only allowed as function local variables
>>
>> Is there any way around this, that is, any way to get efficient array
>> performance from my self.something arrays?

No; it's a known deficiency, no ETA. (Of course, you can work around it 
for now by storing a plain np.ndarray and copy it to a local variable 
manually. Even when this is supported, it will likely be a tiny bit 
slower than copying to a local variable anyway!...depending on problem 
size).


-- 
Dag Sverre

From robertwb at math.washington.edu  Wed Jul 29 19:22:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 29 Jul 2009 10:22:57 -0700
Subject: [Cython] [sage-support] Re: changeing sage -b (cython)
In-Reply-To: <4A707CFD.3000207@student.matnat.uio.no>
References: <c83b3d1c-ebbc-4d11-8cd3-fd64aa8aaa99@26g2000yqk.googlegroups.com>	<f67335450907171237l80a3621l6f3ce5e57ca6dc39@mail.gmail.com>	<6ebc39c0-8255-4577-89ff-8ab8e1886525@y10g2000prf.googlegroups.com>	<20d43422-df88-471c-b903-8c7f487ad577@24g2000yqm.googlegroups.com>	<d09943d2-9173-4789-838c-fdb59de2a86c@i18g2000pro.googlegroups.com>	<F0F8184B-218F-4444-95A0-89983D68FB78@math.washington.edu>	<92dae4b1-e803-4d4f-b5cb-3989a058b7e5@z4g2000prh.googlegroups.com>	<85e81ba30907221127w53a20ab5pc2bc26b1968ee20b@mail.gmail.com>	<5689E418-9BBD-4359-9B14-73174723821C@math.washington.edu>	<9418b9470907221139p1885fa35gee164559ec9704d8@mail.gmail.com>	<95ee2dab-113c-45c6-87cb-dbf4017c633b@a7g2000yqk.googlegroups.com>
	<79B14468-6B58-41BC-A5C3-7D4BF8686802@math.washington.edu>
	<4A707CFD.3000207@student.matnat.uio.no>
Message-ID: <EA41467D-4043-4994-956E-5D9294BDDE83@math.washington.edu>

On Jul 29, 2009, at 9:46 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> I'm not sure if or when either of these will be available, but
>> neither are trivial. Both questions are probably better asked on the
>> cython lists.
>>
>> - Robert
>>
>>
>> On Jul 22, 2009, at 12:17 PM, Ethan Van Andel wrote:
>>
>>> Robert,
>>>
>>> Is there any prediction for when numpy complex types will work?
>>> (outside of the notebook, when compiling via sage -b)
>
> I assume this will happen when Cython is upgraded in Sage, i.e.  
> #6438 /
> 4.1.1. If it doesn't, please send cython-dev (or me if you can't be
> bothered) a mail.

No, that doesn't work with plain Cython either. Perhaps we need to  
require casting with extern typedefs?

-------- numpy_complex_arith.pyx ---------
cimport numpy as np

cdef np.complex128_t x = 0
print x+x

-----------------

gcc: numpy_complex_arith.c
numpy_complex_arith.c: In function 'initnumpy_complex_arith':
numpy_complex_arith.c:2536: error: incompatible types in assignment
numpy_complex_arith.c:2543: error: incompatible type for argument 1  
of '__pyx_t_double_complex_add'
numpy_complex_arith.c:2543: error: incompatible type for argument 2  
of '__pyx_t_double_complex_add'
numpy_complex_arith.c:2543: error: incompatible type for argument 1  
of '__pyx_t_double_complex_add'
numpy_complex_arith.c:2543: error: incompatible type for argument 2  
of '__pyx_t_double_complex_add'
numpy_complex_arith.c: In function 'initnumpy_complex_arith':
numpy_complex_arith.c:2536: error: incompatible types in assignment
numpy_complex_arith.c:2543: error: incompatible type for argument 1  
of '__pyx_t_double_complex_add'
numpy_complex_arith.c:2543: error: incompatible type for argument 2  
of '__pyx_t_double_complex_add'
numpy_complex_arith.c:2543: error: incompatible type for argument 1  
of '__pyx_t_double_complex_add'
numpy_complex_arith.c:2543: error: incompatible type for argument 2  
of '__pyx_t_double_complex_add'


From dagss at student.matnat.uio.no  Wed Jul 29 20:59:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 29 Jul 2009 20:59:26 +0200
Subject: [Cython] Call for hosting of users' mailing list
Message-ID: <4A709C0E.3030702@student.matnat.uio.no>

We've been trying to get somewhere to host a users' list for Cython, 
however we've got no response from the mail admins we've tried so far 
(codespeak, python.org).

Other ideas?

Anybody with a reliable, well-supported server, with email lists set up, 
who wants to host the Cython mailing lists?

Cython's server is currently hosted by the Sage project (sagemath.org). 
However the Sage project uses Google groups and doesn't have mailman or 
similar set up (or at any rate, don't administer any other email lists, 
so the administrative overhead would be huge).

Google Groups appears to be out of the question for Cython due to having 
to sign an EULA before getting posting access.


Dag Sverre

From dagss at student.matnat.uio.no  Wed Jul 29 22:13:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 29 Jul 2009 22:13:26 +0200
Subject: [Cython] [sage-support] Re: changeing sage -b (cython)
In-Reply-To: <EA41467D-4043-4994-956E-5D9294BDDE83@math.washington.edu>
References: <c83b3d1c-ebbc-4d11-8cd3-fd64aa8aaa99@26g2000yqk.googlegroups.com>	<f67335450907171237l80a3621l6f3ce5e57ca6dc39@mail.gmail.com>	<6ebc39c0-8255-4577-89ff-8ab8e1886525@y10g2000prf.googlegroups.com>	<20d43422-df88-471c-b903-8c7f487ad577@24g2000yqm.googlegroups.com>	<d09943d2-9173-4789-838c-fdb59de2a86c@i18g2000pro.googlegroups.com>	<F0F8184B-218F-4444-95A0-89983D68FB78@math.washington.edu>	<92dae4b1-e803-4d4f-b5cb-3989a058b7e5@z4g2000prh.googlegroups.com>	<85e81ba30907221127w53a20ab5pc2bc26b1968ee20b@mail.gmail.com>	<5689E418-9BBD-4359-9B14-73174723821C@math.washington.edu>	<9418b9470907221139p1885fa35gee164559ec9704d8@mail.gmail.com>	<95ee2dab-113c-45c6-87cb-dbf4017c633b@a7g2000yqk.googlegroups.com>
	<79B14468-6B58-41BC-A5C3-7D4BF8686802@math.washington.edu>
	<4A707CFD.3000207@student.matnat.uio.no>
	<EA41467D-4043-4994-956E-5D9294BDDE83@math.washington.edu>
Message-ID: <4A70AD66.5040104@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jul 29, 2009, at 9:46 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> I'm not sure if or when either of these will be available, but
>>> neither are trivial. Both questions are probably better asked on the
>>> cython lists.
>>>
>>> - Robert
>>>
>>>
>>> On Jul 22, 2009, at 12:17 PM, Ethan Van Andel wrote:
>>>
>>>> Robert,
>>>>
>>>> Is there any prediction for when numpy complex types will work?
>>>> (outside of the notebook, when compiling via sage -b)
>>
>> I assume this will happen when Cython is upgraded in Sage, i.e. #6438 /
>> 4.1.1. If it doesn't, please send cython-dev (or me if you can't be
>> bothered) a mail.
> 
> No, that doesn't work with plain Cython either. Perhaps we need to 
> require casting with extern typedefs?

Ahh, of course; "Cython complex" (either C99 or custom structs) are not 
compatible with NumPy structs.

I've selected this strategy:

http://trac.cython.org/cython_trac/ticket/347

Fix is up in cython repo.

-- 
Dag Sverre

From sccolbert at gmail.com  Thu Jul 30 03:41:25 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 29 Jul 2009 21:41:25 -0400
Subject: [Cython] Call for hosting of users' mailing list
In-Reply-To: <4A709C0E.3030702@student.matnat.uio.no>
References: <4A709C0E.3030702@student.matnat.uio.no>
Message-ID: <7f014ea60907291841u54390ec3n8fde3f85f6cde8d9@mail.gmail.com>

Well, I have a dreamhost server just sitting around. It only hosts my
blog (which is mostly non-existant), but I have ample bandwidth and
space.

It is however, on a shared box so the performance isnt stellar.

The Cython list seems to be lightweight however, so if you want to
give it a try (and want to walk me through the setup) your more than
welcome to use my box.

Cheers,

Chris

On Wed, Jul 29, 2009 at 2:59 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> We've been trying to get somewhere to host a users' list for Cython,
> however we've got no response from the mail admins we've tried so far
> (codespeak, python.org).
>
> Other ideas?
>
> Anybody with a reliable, well-supported server, with email lists set up,
> who wants to host the Cython mailing lists?
>
> Cython's server is currently hosted by the Sage project (sagemath.org).
> However the Sage project uses Google groups and doesn't have mailman or
> similar set up (or at any rate, don't administer any other email lists,
> so the administrative overhead would be huge).
>
> Google Groups appears to be out of the question for Cython due to having
> to sign an EULA before getting posting access.
>
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Thu Jul 30 07:05:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 30 Jul 2009 07:05:18 +0200
Subject: [Cython] Call for hosting of users' mailing list
In-Reply-To: <7f014ea60907291841u54390ec3n8fde3f85f6cde8d9@mail.gmail.com>
References: <4A709C0E.3030702@student.matnat.uio.no>
	<7f014ea60907291841u54390ec3n8fde3f85f6cde8d9@mail.gmail.com>
Message-ID: <4A712A0E.5050104@student.matnat.uio.no>

Chris Colbert wrote:
> Well, I have a dreamhost server just sitting around. It only hosts my
> blog (which is mostly non-existant), but I have ample bandwidth and
> space.
> 
> It is however, on a shared box so the performance isnt stellar.
> 
> The Cython list seems to be lightweight however, so if you want to
> give it a try (and want to walk me through the setup) your more than
> welcome to use my box.

Thanks. Let's keep it in mind; however I think we should try to find 
somebody who is already used to hosting email lists, to avoid double 
efforts.

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Jul 30 12:12:52 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 30 Jul 2009 03:12:52 -0700
Subject: [Cython] Call for hosting of users' mailing list
In-Reply-To: <4A709C0E.3030702@student.matnat.uio.no>
References: <4A709C0E.3030702@student.matnat.uio.no>
Message-ID: <6BE9AF19-5FD9-473E-AA4A-82115250F52C@math.washington.edu>

On Jul 29, 2009, at 11:59 AM, Dag Sverre Seljebotn wrote:

> We've been trying to get somewhere to host a users' list for Cython,
> however we've got no response from the mail admins we've tried so far
> (codespeak, python.org).
>
> Other ideas?
>
> Anybody with a reliable, well-supported server, with email lists  
> set up,
> who wants to host the Cython mailing lists?
>
> Cython's server is currently hosted by the Sage project  
> (sagemath.org).
> However the Sage project uses Google groups and doesn't have  
> mailman or
> similar set up (or at any rate, don't administer any other email  
> lists,
> so the administrative overhead would be huge).
>
> Google Groups appears to be out of the question for Cython due to  
> having
> to sign an EULA before getting posting access.

I'm pretty sure one can post to the list without signing an EULA--at  
least once the moderators approve you. I'm pretty sure one needs an  
account to administer though. I don't find it too draconian,
but for something like this finding the lowest common denominator is  
important.

- Robert


From cyrille.rosset at apc.univ-paris-diderot.fr  Fri Jul 31 18:53:22 2009
From: cyrille.rosset at apc.univ-paris-diderot.fr (Cyrille Rosset)
Date: Fri, 31 Jul 2009 18:53:22 +0200
Subject: [Cython] Include paths for pyximport (continued)
Message-ID: <1249059202.5648.19.camel@leonidas>

Hi all,

I am using cython and numpy and looked for a way to get cython modules
using numpy compiled with pyximport.
I found on the list the following message
(http://codespeak.net/pipermail/cython-dev/2009-May/005289.html)
but didn't find any modification in the cython repository in this
direction.

Following Lisandro Dalc?n's suggestion, I made a patch that adds a
ext_opts option to pyximport.install() which cascades to
pyximport.get_distutils_extension() and is used as a dictionnary with
options for distutils.Extension().
Only the dictionnary type is tested (I assume Extension will tell if an
option is not good).
I also added a pyximport.unsinstall() function (so it's possible to
change options of the importer).

Example:
In [1]: import pyximport2
In [2]: import numpy
In [3]: pyximport2.install(ext_opts={'include_dirs':
[numpy.get_include()]})
In [4]: import test
In [5]: test.total([1,2,3,4,5])
Out[5]: 15.0

I would be happy if something like this could be included in cython !
Please tell me what you think.

Cheers,
Cyrille Rosset.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pyximport.patch
Type: text/x-patch
Size: 5795 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090731/0cdab917/attachment.bin 

From seb.binet at gmail.com  Fri Jul 31 19:48:27 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Fri, 31 Jul 2009 19:48:27 +0200
Subject: [Cython] Include paths for pyximport (continued)
In-Reply-To: <1249059202.5648.19.camel@leonidas>
References: <1249059202.5648.19.camel@leonidas>
Message-ID: <200907311948.27372.binet@cern.ch>

hi,

[snip]
> Example:
> In [1]: import pyximport2
> In [2]: import numpy
> In [3]: pyximport2.install(ext_opts={'include_dirs':
> [numpy.get_include()]})
> In [4]: import test
> In [5]: test.total([1,2,3,4,5])
> Out[5]: 15.0
>
> I would be happy if something like this could be included in cython !
FWIW, I am definitely +1 on this one.

pyximport.load_module('foo', ext_opts={'libraries' = ['cfoo1', 'cbar2']})
is my new quick'n'dirty testing friend.

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From craigcitro at gmail.com  Fri Jul 31 20:14:04 2009
From: craigcitro at gmail.com (Craig Citro)
Date: Fri, 31 Jul 2009 11:14:04 -0700
Subject: [Cython] Call for hosting of users' mailing list
In-Reply-To: <6BE9AF19-5FD9-473E-AA4A-82115250F52C@math.washington.edu>
References: <4A709C0E.3030702@student.matnat.uio.no>
	<6BE9AF19-5FD9-473E-AA4A-82115250F52C@math.washington.edu>
Message-ID: <a8c4ff4c0907311114m6f4afc7av7bb8468c42be962b@mail.gmail.com>

>> Google Groups appears to be out of the question for Cython due to
>> having
>> to sign an EULA before getting posting access.
>
> I'm pretty sure one can post to the list without signing an EULA--at
> least once the moderators approve you. I'm pretty sure one needs an
> account to administer though. I don't find it too draconian,
> but for something like this finding the lowest common denominator is
> important.
>

I'm just curious -- what was it about the Google Groups EULA that
people disliked? I'm not trying to stir up trouble, but I don't know
what the reasons were when setting up this list. ;)

-cc

From dagss at student.matnat.uio.no  Fri Jul 31 22:11:43 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 31 Jul 2009 22:11:43 +0200
Subject: [Cython] Include paths for pyximport (continued)
In-Reply-To: <1249059202.5648.19.camel@leonidas>
References: <1249059202.5648.19.camel@leonidas>
Message-ID: <e87ee632d21b196c037ecf1dfbbcbd3b.squirrel@webmail.uio.no>

Cyrille Rosset wrote:
> Hi all,
>
> I am using cython and numpy and looked for a way to get cython modules
> using numpy compiled with pyximport.
> I found on the list the following message
> (http://codespeak.net/pipermail/cython-dev/2009-May/005289.html)
> but didn't find any modification in the cython repository in this
> direction.

Well, there's

http://trac.cython.org/cython_trac/ticket/312

with some patches, which unfortunately noone's found time to review and
commit.

We're happy to accept other/further patches as well (but please make a
comment on whether it is compatible with the above, in that case).

(I don't have much time now so I'll have to stop here, sorry.)

Dag Sverre


From dagss at student.matnat.uio.no  Fri Jul 31 22:08:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 31 Jul 2009 22:08:00 +0200
Subject: [Cython] Call for hosting of users' mailing list
In-Reply-To: <a8c4ff4c0907311114m6f4afc7av7bb8468c42be962b@mail.gmail.com>
References: <4A709C0E.3030702@student.matnat.uio.no>
	<6BE9AF19-5FD9-473E-AA4A-82115250F52C@math.washington.edu>
	<a8c4ff4c0907311114m6f4afc7av7bb8468c42be962b@mail.gmail.com>
Message-ID: <70e1a4a33cde19266564cbabda06f3b5.squirrel@webmail.uio.no>

Craig Citro wrote:
>>> Google Groups appears to be out of the question for Cython due to
>>> having
>>> to sign an EULA before getting posting access.
>>
>> I'm pretty sure one can post to the list without signing an EULA--at
>> least once the moderators approve you. I'm pretty sure one needs an
>> account to administer though. I don't find it too draconian,
>> but for something like this finding the lowest common denominator is
>> important.
>>
>
> I'm just curious -- what was it about the Google Groups EULA that
> people disliked? I'm not trying to stir up trouble, but I don't know
> what the reasons were when setting up this list. ;)

Well, see e.g.

http://article.gmane.org/gmane.comp.python.cython.devel/5567

Looking on that again, Stefan may have changed his mind. Stefan?

Anyway, Stefan not being able/willing to join the group is pretty much a
deal-breaker.

I'm not so much in favor of having an actual dicussion for or against
getting a Google account -- if somebody has issues with it, I'm not really
in a position to tell them otherwise :-)

Dag Sverre


From dagss at student.matnat.uio.no  Fri Jul 31 22:08:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 31 Jul 2009 22:08:01 +0200
Subject: [Cython] Call for hosting of users' mailing list
In-Reply-To: <a8c4ff4c0907311114m6f4afc7av7bb8468c42be962b@mail.gmail.com>
References: <4A709C0E.3030702@student.matnat.uio.no>
	<6BE9AF19-5FD9-473E-AA4A-82115250F52C@math.washington.edu>
	<a8c4ff4c0907311114m6f4afc7av7bb8468c42be962b@mail.gmail.com>
Message-ID: <742652ef2d3b38f43c781f6badf73b75.squirrel@webmail.uio.no>

Craig Citro wrote:
>>> Google Groups appears to be out of the question for Cython due to
>>> having
>>> to sign an EULA before getting posting access.
>>
>> I'm pretty sure one can post to the list without signing an EULA--at
>> least once the moderators approve you. I'm pretty sure one needs an
>> account to administer though. I don't find it too draconian,
>> but for something like this finding the lowest common denominator is
>> important.
>>
>
> I'm just curious -- what was it about the Google Groups EULA that
> people disliked? I'm not trying to stir up trouble, but I don't know
> what the reasons were when setting up this list. ;)

Well, see e.g.

http://article.gmane.org/gmane.comp.python.cython.devel/5567

Looking on that again, Stefan may have changed his mind. Stefan?

Anyway, Stefan not being able/willing to join the group is pretty much a
deal-breaker.

I'm not so much in favor of having an actual dicussion for or against
getting a Google account -- if somebody has issues with it, I'm not really
in a position to tell them otherwise :-)

Dag Sverre