From stefan at sun.ac.za  Mon Jun  1 01:46:02 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 1 Jun 2009 01:46:02 +0200
Subject: [Cython] Profiling Cython under OSX
Message-ID: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>

Hi all,

I needed to profile some Cython code on my mac today.  I am only
familiar with Valgrind, but this tool is not available on OSX (closest
thing is  http://www.sealiesoftware.com/valgrind/).  Luckily, I
discovered that the Activity Monitor is very handy in this regard:

1. Start your long running process
2. Go to the activity monitor, select the process, and click "Sample Process"
3. In the window that pops up, select "Display: Percent of Parent"

And there you have a beautiful tree showing how much time each call took:

http://mentat.za.net/refer/osx_profiling.png

Enjoy!
St?fan

From amit.pureenergy at gmail.com  Mon Jun  1 15:38:54 2009
From: amit.pureenergy at gmail.com (Amit Sethi)
Date: Mon, 1 Jun 2009 19:08:54 +0530
Subject: [Cython] Creating python bindings for C library using Cython
Message-ID: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>

hi ,
I guess this needs to go on cython-users but i did not see any such mailing
list so I am posting it here ,
I am trying to create python bindings for a library written in C ,
But I am wondering as to how I am suppose to handle pointers.

For example

for the C function,
int * analyse_row (int length, unsigned char *data),


Now in this function call how am i supposed to handle *data , can i pass a
string in place of *data ., Also please guide be to some projects that use
Cython for binding with C
-- 
A-M-I-T S|S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090601/56a66e9e/attachment.htm 

From dalcinl at gmail.com  Mon Jun  1 16:19:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 1 Jun 2009 11:19:11 -0300
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
Message-ID: <e7ba66e40906010719p6e570d1an96752797eaf108ab@mail.gmail.com>

On Mon, Jun 1, 2009 at 10:38 AM, Amit Sethi <amit.pureenergy at gmail.com> wrote:
>
> hi ,
> I guess this needs to go on cython-users but i did not see any such mailing
> list so I am posting it here ,
> I am trying to create python bindings for a library written in C ,
> But I am wondering as to how I am suppose to handle pointers.
>
> For example
>
> for the C function,
> int * analyse_row (int length, unsigned char *data),
>
>
> Now in this function call how am i supposed to handle *data , can i pass a
> string in place of *data .

Mmm... better you are sure what the 'data' represent. Anyway, you can
also add explicit cast and call like "analize_row(somelen, <unsigned
char*>somepointer)". I would need some more backgraund from you to
provide a better hint (What is analyse_row() for?)

> Also please guide be to some projects that use
> Cython for binding with C
>

>From my side, you could look at mpi4py, petsc4py, slepc4py (all of
them at Google Code).




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Jun  1 16:33:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 1 Jun 2009 11:33:59 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>
	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
Message-ID: <e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>

On Sat, May 30, 2009 at 12:04 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On May 28, 2009, at 3:50 PM, Lisandro Dalcin wrote:
>
>>
>> Anyway, suppose you have installed both MSVC and MinGW, an as you are
>> a serious Cython developer, you want to run the testsuite with both.
>> The easiest way to select GCC would be to pass '--mingw' to
>> runtests.py, right?
>
>
> I hadn't thought of that. Are you saying that you would have a single
> Python install that works both with MSVC and MinGW?
>

Of course!

In my particular case, I have installed stock Python 2.6 (using the
MSI installed from python.org), and I routinely test mpi4py building
it with MSVC Express 2008 and MinGW32, with a couple of different MPI
implementations (DeinoMPI, MPICH2, and very recently Microsoft MPI).
Up to now, all work just fine. Of course, I've been very careful and
my code does not call anything from the C stdlib (which is know to
have issues when building with MinGW, because of possible mismatches
of the C runtime).

OTOH, most of the Cython testsuite pass happily with the stock Python
2.6 and MinGW32. The only offending part are test like 'cdef extern
void foo()', as that end-up with linker errors.




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From seb.binet at gmail.com  Mon Jun  1 18:11:50 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Mon, 1 Jun 2009 18:11:50 +0200
Subject: [Cython] cimport from __all__
Message-ID: <200906011811.50884.binet@cern.ch>

hi there,

I am in the process of applying what I learnt at last SciPy'08 tutorial (yes, 
I am that slow) on an almost-real-life project of mine.
Up to now I am trying to translate an old C++/PyCxx project (which does not 
compile anymore) into Cython: so far so good.

One thing, though: the separation of '.pyx' and '.pxd' files to export 
declarations is a bit too C-ish.

Couldn't the same functionality be achieved by re-using the __all__ module 
attribute ?

## mymodule.pyx ##
__all__ = ( 'Foo', 'bar')

cdef class Foo:
   # implementation elided

cdef int bar():
  return 42
## EOF ##

## test.pyx ##
cimport mymodule
print "bar:",mymodule.bar()
## EOF ##

I suppose I am missing some implementation detail, but from my 20k feet 
overview, it seems that should work :)

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From dagss at student.matnat.uio.no  Mon Jun  1 18:23:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Jun 2009 18:23:56 +0200
Subject: [Cython] cimport from __all__
In-Reply-To: <200906011811.50884.binet@cern.ch>
References: <200906011811.50884.binet@cern.ch>
Message-ID: <4A24009C.30509@student.matnat.uio.no>

Sebastien Binet wrote:
> hi there,
>
> I am in the process of applying what I learnt at last SciPy'08 tutorial (yes, 
> I am that slow) on an almost-real-life project of mine.
> Up to now I am trying to translate an old C++/PyCxx project (which does not 
> compile anymore) into Cython: so far so good.
>
> One thing, though: the separation of '.pyx' and '.pxd' files to export 
> declarations is a bit too C-ish.
>
> Couldn't the same functionality be achieved by re-using the __all__ module 
> attribute ?
>
> ## mymodule.pyx ##
> __all__ = ( 'Foo', 'bar')
>
> cdef class Foo:
>    # implementation elided
>
> cdef int bar():
>   return 42
> ## EOF ##
>
> ## test.pyx ##
> cimport mymodule
> print "bar:",mymodule.bar()
> ## EOF ##
>
> I suppose I am missing some implementation detail, but from my 20k feet 
> overview, it seems that should work :)
>   
I remember that the basic idea of this feature (which would be 
implemented as "generate pxd files automatically from pyx files") has 
been discussed earlier (Sage dev 1 days?) and it's definitely on our 
would-like-to-do-if-we-get-time list :-) The idea about using the 
__all__ attribute to declare what symbols to export was new; I like it!

Dag Sverre

From sccolbert at gmail.com  Mon Jun  1 19:37:30 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 1 Jun 2009 10:37:30 -0700
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
Message-ID: <7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>

python strings are automagically converted to char* by cython.

You can also look at a wrapper i'm building for the OpenCV library.
cython-opencv at google code.

Chris

On Mon, Jun 1, 2009 at 6:38 AM, Amit Sethi <amit.pureenergy at gmail.com>wrote:

>
> hi ,
> I guess this needs to go on cython-users but i did not see any such mailing
> list so I am posting it here ,
> I am trying to create python bindings for a library written in C ,
> But I am wondering as to how I am suppose to handle pointers.
>
> For example
>
> for the C function,
> int * analyse_row (int length, unsigned char *data),
>
>
> Now in this function call how am i supposed to handle *data , can i pass a
> string in place of *data ., Also please guide be to some projects that use
> Cython for binding with C
> --
> A-M-I-T S|S
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090601/c2b57f99/attachment.htm 

From robert.kern at gmail.com  Mon Jun  1 20:18:28 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 01 Jun 2009 13:18:28 -0500
Subject: [Cython] Profiling Cython under OSX
In-Reply-To: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>
References: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>
Message-ID: <h0161k$8nj$1@ger.gmane.org>

On 2009-05-31 18:46, St?fan van der Walt wrote:
> Hi all,
>
> I needed to profile some Cython code on my mac today.  I am only
> familiar with Valgrind, but this tool is not available on OSX (closest
> thing is  http://www.sealiesoftware.com/valgrind/).

Actually, the OS X branch got merged into the trunk a few days ago.

> Luckily, I
> discovered that the Activity Monitor is very handy in this regard:
>
> 1. Start your long running process
> 2. Go to the activity monitor, select the process, and click "Sample Process"
> 3. In the window that pops up, select "Display: Percent of Parent"

Instruments.app is the canonical tool for this on OS X.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robertwb at math.washington.edu  Mon Jun  1 21:06:06 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Jun 2009 12:06:06 -0700
Subject: [Cython] Profiling Cython under OSX
In-Reply-To: <h0161k$8nj$1@ger.gmane.org>
References: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>
	<h0161k$8nj$1@ger.gmane.org>
Message-ID: <D0CCEF0A-B471-4C73-8194-3C0F81DB4781@math.washington.edu>

On Jun 1, 2009, at 11:18 AM, Robert Kern wrote:

> On 2009-05-31 18:46, St?fan van der Walt wrote:
>> Hi all,
>>
>> I needed to profile some Cython code on my mac today.  I am only
>> familiar with Valgrind, but this tool is not available on OSX  
>> (closest
>> thing is  http://www.sealiesoftware.com/valgrind/).
>
> Actually, the OS X branch got merged into the trunk a few days ago.
>
>> Luckily, I
>> discovered that the Activity Monitor is very handy in this regard:
>>
>> 1. Start your long running process
>> 2. Go to the activity monitor, select the process, and click  
>> "Sample Process"
>> 3. In the window that pops up, select "Display: Percent of Parent"
>
> Instruments.app is the canonical tool for this on OS X.

Still, very cool! Could you put this up on the wiki? (Also, any  
valgrind+Cython knowledge you might have.)

- Robert


From dagss at student.matnat.uio.no  Mon Jun  1 21:09:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Jun 2009 21:09:08 +0200
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
	<7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
Message-ID: <4A242754.5050207@student.matnat.uio.no>

Chris Colbert wrote:
> python strings are automagically converted to char* by cython.
I'll beat Stefan to it and note some very important points here. This 
applies if you actually have a string (e.g. could be stored in "unicode" 
in Py2), not if you have raw byte data ("bytes" in Py3).

1) A C char* is not a string as such, just a sequence of bytes. The C 
library should have a definition of what kind of encoding it wants its 
strings in, and one should call the library like this:

encoded = mystr.encode(encoding)
cdef char* encoded_buf = encoded
c_func(encoded_buf)

2) Note that char* does not say anything about releasing memory etc. 
I.e., the following will likely crash:

c_func(mystr.encode(encoding))

because the temporary object returned from the encode method is 
deallocated. *Always* keep a reference to the original Python object for 
the duration someone could use the char*.

(Perhaps somebody could make this a FAQ entry in the wiki if it is not 
already?)

Dag Sverre

From sccolbert at gmail.com  Mon Jun  1 21:14:02 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 1 Jun 2009 15:14:02 -0400
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <4A242754.5050207@student.matnat.uio.no>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
	<7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
	<4A242754.5050207@student.matnat.uio.no>
Message-ID: <7f014ea60906011214q17a73ba7u3a1217b468757adb@mail.gmail.com>

I never mind being corrected/updated/put-in-my-place with such a great
answer.

So there you have it :)

Cheers!

On Mon, Jun 1, 2009 at 3:09 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Chris Colbert wrote:
> > python strings are automagically converted to char* by cython.
> I'll beat Stefan to it and note some very important points here. This
> applies if you actually have a string (e.g. could be stored in "unicode"
> in Py2), not if you have raw byte data ("bytes" in Py3).
>
> 1) A C char* is not a string as such, just a sequence of bytes. The C
> library should have a definition of what kind of encoding it wants its
> strings in, and one should call the library like this:
>
> encoded = mystr.encode(encoding)
> cdef char* encoded_buf = encoded
> c_func(encoded_buf)
>
> 2) Note that char* does not say anything about releasing memory etc.
> I.e., the following will likely crash:
>
> c_func(mystr.encode(encoding))
>
> because the temporary object returned from the encode method is
> deallocated. *Always* keep a reference to the original Python object for
> the duration someone could use the char*.
>
> (Perhaps somebody could make this a FAQ entry in the wiki if it is not
> already?)
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090601/cccc2f84/attachment.htm 

From robertwb at math.washington.edu  Mon Jun  1 21:17:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Jun 2009 12:17:04 -0700
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>
	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
	<e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
Message-ID: <3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>

On Jun 1, 2009, at 7:33 AM, Lisandro Dalcin wrote:

> On Sat, May 30, 2009 at 12:04 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On May 28, 2009, at 3:50 PM, Lisandro Dalcin wrote:
>>
>>>
>>> Anyway, suppose you have installed both MSVC and MinGW, an as you  
>>> are
>>> a serious Cython developer, you want to run the testsuite with both.
>>> The easiest way to select GCC would be to pass '--mingw' to
>>> runtests.py, right?
>>
>>
>> I hadn't thought of that. Are you saying that you would have a single
>> Python install that works both with MSVC and MinGW?
>>
>
> Of course!
>
> In my particular case, I have installed stock Python 2.6 (using the
> MSI installed from python.org), and I routinely test mpi4py building
> it with MSVC Express 2008 and MinGW32, with a couple of different MPI
> implementations (DeinoMPI, MPICH2, and very recently Microsoft MPI).
> Up to now, all work just fine. Of course, I've been very careful and
> my code does not call anything from the C stdlib (which is know to
> have issues when building with MinGW, because of possible mismatches
> of the C runtime).

Ah, I didn't know avoiding the C stdlib was all that one needed to do  
here. You can tell I'm not a Windows developer :)

> OTOH, most of the Cython testsuite pass happily with the stock Python
> 2.6 and MinGW32. The only offending part are test like 'cdef extern
> void foo()', as that end-up with linker errors.


Do you have any ideas on how best to fix this? (I suppose we could  
make our own library with these fake stubs, but that seems kludgy and  
overkill)

- Robert


From dagss at student.matnat.uio.no  Mon Jun  1 21:48:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Jun 2009 21:48:58 +0200
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>
References: <4A1DB976.4060309@gmail.com>
	<4A1DC278.9080704@gmail.com>	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>	<e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
	<3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>
Message-ID: <4A2430AA.6070707@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 1, 2009, at 7:33 AM, Lisandro Dalcin wrote:
>
>   
>> On Sat, May 30, 2009 at 12:04 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>     
>>> On May 28, 2009, at 3:50 PM, Lisandro Dalcin wrote:
>>>
>>>       
>>>> Anyway, suppose you have installed both MSVC and MinGW, an as you  
>>>> are
>>>> a serious Cython developer, you want to run the testsuite with both.
>>>> The easiest way to select GCC would be to pass '--mingw' to
>>>> runtests.py, right?
>>>>         
>>> I hadn't thought of that. Are you saying that you would have a single
>>> Python install that works both with MSVC and MinGW?
>>>
>>>       
>> Of course!
>>
>> In my particular case, I have installed stock Python 2.6 (using the
>> MSI installed from python.org), and I routinely test mpi4py building
>> it with MSVC Express 2008 and MinGW32, with a couple of different MPI
>> implementations (DeinoMPI, MPICH2, and very recently Microsoft MPI).
>> Up to now, all work just fine. Of course, I've been very careful and
>> my code does not call anything from the C stdlib (which is know to
>> have issues when building with MinGW, because of possible mismatches
>> of the C runtime).
>>     
>
> Ah, I didn't know avoiding the C stdlib was all that one needed to do  
> here. You can tell I'm not a Windows developer :)
>   
IIRC, an additional point: I think using the stdlib should work as long 
as you link in the proper MinGW stdlib, and make sure not to exchange 
uses of the stdlib with modules compiled with a different compiler.

It's basically like having two different stdlibs -- file handles from 
one of them won't make sense to the other, buffering output in one of 
them won't be seen by the other, and so on.

Am I right? (This came form a numpy-discuss thread somewhere, they've 
discussed this a couple of times.)

Dag Sverre

From stefan_ml at behnel.de  Mon Jun  1 22:04:35 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Jun 2009 22:04:35 +0200
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <4A242754.5050207@student.matnat.uio.no>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>	<7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
	<4A242754.5050207@student.matnat.uio.no>
Message-ID: <4A243453.50004@behnel.de>


Dag Sverre Seljebotn wrote:
> Chris Colbert wrote:
>> python strings are automagically converted to char* by cython.
> I'll beat Stefan to it and note some very important points here.
> [...]
> (Perhaps somebody could make this a FAQ entry in the wiki if it is not 
> already?)

http://wiki.cython.org/FAQ#HowdoIpassaPythonstringparameterontoaClibrary.3F

Now guess who wrote that entry :)

Stefan


From dalcinl at gmail.com  Mon Jun  1 22:08:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 1 Jun 2009 17:08:01 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A2430AA.6070707@student.matnat.uio.no>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>
	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
	<e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
	<3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>
	<4A2430AA.6070707@student.matnat.uio.no>
Message-ID: <e7ba66e40906011308y330bc185pe041dae839673b3a@mail.gmail.com>

On Mon, Jun 1, 2009 at 4:48 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Robert Bradshaw wrote:
>>
>> Ah, I didn't know avoiding the C stdlib was all that one needed to do
>> here. You can tell I'm not a Windows developer :)
>>

You can tell the same about me :-)

> IIRC, an additional point: I think using the stdlib should work as long
> as you link in the proper MinGW stdlib, and make sure not to exchange
> uses of the stdlib with modules compiled with a different compiler.
>
> It's basically like having two different stdlibs -- file handles from
> one of them won't make sense to the other, buffering output in one of
> them won't be seen by the other, and so on.
>

> Am I right? (This came form a numpy-discuss thread somewhere, they've
> discussed this a couple of times.)
>

Yes, you are right. But I do not know how to ask MinGW use a
particular C runtime. Though I bet there is a way (and likely numpy
folks know it).



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jusa.sj at gmail.com  Tue Jun  2 12:43:29 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Tue, 2 Jun 2009 13:43:29 +0300
Subject: [Cython] Syntax for declaring Cython class variables
Message-ID: <20db9a7d0906020343u3dcb3157nc166049261dfb24b@mail.gmail.com>

Hi,
I got a simple problem regarding so called class variables in Cython. I'm
trying to declare a simple counter as a class variable which is shared
between all instances of that class. The counter's value should be
incremented as instances of the class are created. Cython allows me to
declare a Python dictionary as a class variable, but the following gives me
an error:


# foo.pyx

cdef class Foo:
    count = 0

    def __cinit__(self):
        Foo.count += 1


>>> a = Foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
    Foo.count += 1
TypeError: can't set attributes of built-in/extension type 'foo.Foo'


Is there a way to declare an integer class variable in Cython?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/1d2b260e/attachment.htm 

From darkgl0w at yahoo.com  Tue Jun  2 14:27:54 2009
From: darkgl0w at yahoo.com (Cristi Constantin)
Date: Tue, 2 Jun 2009 05:27:54 -0700 (PDT)
Subject: [Cython] Syntax for declaring Cython class variables
Message-ID: <235734.63423.qm@web52112.mail.re2.yahoo.com>


Good day.
You should try self.count += 1 instead of Foo.count += 1.

--- On Tue, 6/2/09, Juha Salo <jusa.sj at gmail.com> wrote:

From: Juha Salo <jusa.sj at gmail.com>
Subject: [Cython] Syntax for declaring Cython class variables
To: cython-dev at codespeak.net
Date: Tuesday, June 2, 2009, 3:43 AM

Hi,?
I got a simple problem regarding so called class variables in Cython. I'm trying to declare a simple counter as a class variable which is shared between all instances of that class. The counter's value should be incremented as instances of the class are created. Cython allows me to declare a Python dictionary as a class variable, but the following gives me an error:


# foo.pyx
cdef class Foo:?? ?count = 0
?? ?def __cinit__(self):?? ? ? ?Foo.count += 1


>>> a = Foo()Traceback (most recent call last):??File "<stdin>", line 1, in <module>??File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
?? ?Foo.count += 1TypeError: can't set attributes of built-in/extension type 'foo.Foo'

Is there a way to declare an integer class variable in Cython?


-----Inline Attachment Follows-----

_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/7ea06252/attachment.htm 

From jusa.sj at gmail.com  Tue Jun  2 17:17:05 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Tue, 2 Jun 2009 18:17:05 +0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <235734.63423.qm@web52112.mail.re2.yahoo.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
Message-ID: <20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>

Unfortunately that didn't seem to work. I got the following error after the
change:
>>> a = Foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
    self.count += 1
AttributeError: 'foo.Foo' object attribute 'count' is read-only


To clarify, I want the value of the count variable to remain the same
between all instances. So if I create 3 Foo objects, each object should show
3 as the value in the count variable.


2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>

>
> Good day.
> You should try self.count += 1 instead of Foo.count += 1.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/50306bda/attachment.htm 

From sccolbert at gmail.com  Tue Jun  2 18:36:50 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 2 Jun 2009 12:36:50 -0400
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
Message-ID: <7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>

you have to declare the attribute as public.

http://docs.cython.org/docs/extension_types.html

Cheers,

Chris

On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:

> Unfortunately that didn't seem to work. I got the following error after the
> change:
> >>> a = Foo()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>     self.count += 1
> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>
>
> To clarify, I want the value of the count variable to remain the same
> between all instances. So if I create 3 Foo objects, each object should show
> 3 as the value in the count variable.
>
>
> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>
>>
>> Good day.
>> You should try self.count += 1 instead of Foo.count += 1.
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/a1b9afab/attachment.htm 

From dalcinl at gmail.com  Tue Jun  2 18:44:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 13:44:17 -0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
Message-ID: <e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>

No, that will not work. AFAIK, Class attributes are not currently
supported for cdef classes.

Juha, you will have to code like this:


# foo.pyx

cdef int Foo_count = 0

cdef class Foo:

    def __cinit__(self):
        global Foo_count
        Foo_count += 1




On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com> wrote:
> you have to declare the attribute as public.
>
> http://docs.cython.org/docs/extension_types.html
>
> Cheers,
>
> Chris
>
> On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
>>
>> Unfortunately that didn't seem to work. I got the following error after
>> the change:
>> >>> a = Foo()
>> Traceback (most recent call last):
>> ??File "<stdin>", line 1, in <module>
>> ??File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>> ?? ?self.count += 1
>> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>>
>> To clarify, I want the value of the count variable to remain the same
>> between all instances. So if I create 3 Foo objects, each object should show
>> 3 as the value in the count variable.
>>
>> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>>>
>>> Good day.
>>> You should try self.count += 1 instead of Foo.count += 1.
>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Jun  2 18:51:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 13:51:06 -0300
Subject: [Cython] PATCH: get rid of __Pyx_StrEq hackery at
	__Pyx_Method_ClassMethod
Message-ID: <e7ba66e40906020951w2f98cb72x55741d3586099ab8@mail.gmail.com>

Review this. Tell me if you like it.

AFAIK, this should always work (even iff an embedder plays with
loading/unloading the Python shared library ??)

diff -r 88fa346e169d Cython/Compiler/Symtab.py
--- a/Cython/Compiler/Symtab.py	Tue May 26 22:54:46 2009 +0200
+++ b/Cython/Compiler/Symtab.py	Tue Jun 02 13:45:11 2009 -0300
@@ -1583,8 +1583,14 @@
 impl = """
 static PyObject* __Pyx_Method_ClassMethod(PyObject *method) {
     /* It appears that PyMethodDescr_Type is not anywhere exposed in
the Python/C API */
-    /* if (!PyObject_TypeCheck(method, &PyMethodDescr_Type)) { */
-    if (__Pyx_StrEq(Py_TYPE(method)->tp_name, "method_descriptor")) {
/* cdef classes */
+    static PyTypeObject *methoddescr_type = NULL;
+    if (methoddescr_type == NULL) {
+       PyObject *meth = __Pyx_GetAttrString((PyObject*)&PyList_Type, "append");
+       if (!meth) return NULL;
+       methoddescr_type = Py_TYPE(meth);
+       Py_DECREF(meth);
+    }
+    if (PyObject_TypeCheck(method, methoddescr_type)) { /* cdef classes */
         PyMethodDescrObject *descr = (PyMethodDescrObject *)method;
         return PyDescr_NewClassMethod(descr->d_type, descr->d_method);
     }

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Tue Jun  2 18:59:22 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 2 Jun 2009 12:59:22 -0400
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
	<e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
Message-ID: <7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>

This is correct, forget what I said. I should have tried it out before I
said anything. My apologies.



On Tue, Jun 2, 2009 at 12:44 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> No, that will not work. AFAIK, Class attributes are not currently
> supported for cdef classes.
>
> Juha, you will have to code like this:
>
>
> # foo.pyx
>
> cdef int Foo_count = 0
>
> cdef class Foo:
>
>    def __cinit__(self):
>        global Foo_count
>        Foo_count += 1
>
>
>
>
> On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com> wrote:
> > you have to declare the attribute as public.
> >
> > http://docs.cython.org/docs/extension_types.html
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
> >>
> >> Unfortunately that didn't seem to work. I got the following error after
> >> the change:
> >> >>> a = Foo()
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >>   File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
> >>     self.count += 1
> >> AttributeError: 'foo.Foo' object attribute 'count' is read-only
> >>
> >> To clarify, I want the value of the count variable to remain the same
> >> between all instances. So if I create 3 Foo objects, each object should
> show
> >> 3 as the value in the count variable.
> >>
> >> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
> >>>
> >>> Good day.
> >>> You should try self.count += 1 instead of Foo.count += 1.
> >>
> >>
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >>
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/2b9ffc48/attachment.htm 

From jusa.sj at gmail.com  Tue Jun  2 19:48:48 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Tue, 2 Jun 2009 20:48:48 +0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
	<e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
	<7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>
Message-ID: <20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>

OK, thanks for the info. It's interesting, though, that I can code something
like this in Cython and it works:

cdef class Foo:
    foos = {}

    def __cinit__(self, name):
        Foo.foos[name] = self


>>> a = Foo('Larry')
>>> b = Foo('Bob')
>>> a.foos
{'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
0x01C5F488>}
>>> b.foos
{'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
0x01C5F488>}


Perhaps I could use a hack workaround with the aforementioned count variable
by storing it inside a class level list object.


2009/6/2 Chris Colbert <sccolbert at gmail.com>

> This is correct, forget what I said. I should have tried it out before I
> said anything. My apologies.
>
>
>
> On Tue, Jun 2, 2009 at 12:44 PM, Lisandro Dalcin <dalcinl at gmail.com>wrote:
>
>> No, that will not work. AFAIK, Class attributes are not currently
>> supported for cdef classes.
>>
>> Juha, you will have to code like this:
>>
>>
>> # foo.pyx
>>
>> cdef int Foo_count = 0
>>
>> cdef class Foo:
>>
>>    def __cinit__(self):
>>        global Foo_count
>>        Foo_count += 1
>>
>>
>>
>>
>> On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com>
>> wrote:
>> > you have to declare the attribute as public.
>> >
>> > http://docs.cython.org/docs/extension_types.html
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
>> >>
>> >> Unfortunately that didn't seem to work. I got the following error after
>> >> the change:
>> >> >>> a = Foo()
>> >> Traceback (most recent call last):
>> >>   File "<stdin>", line 1, in <module>
>> >>   File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>> >>     self.count += 1
>> >> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>> >>
>> >> To clarify, I want the value of the count variable to remain the same
>> >> between all instances. So if I create 3 Foo objects, each object should
>> show
>> >> 3 as the value in the count variable.
>> >>
>> >> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>> >>>
>> >>> Good day.
>> >>> You should try self.count += 1 instead of Foo.count += 1.
>> >>
>> >>
>> >> _______________________________________________
>> >> Cython-dev mailing list
>> >> Cython-dev at codespeak.net
>> >> http://codespeak.net/mailman/listinfo/cython-dev
>> >>
>> >
>> >
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/47b1a3c3/attachment.htm 

From dalcinl at gmail.com  Tue Jun  2 20:03:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 15:03:25 -0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
	<e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
	<7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>
	<20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>
Message-ID: <e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>

On Tue, Jun 2, 2009 at 2:48 PM, Juha Salo <jusa.sj at gmail.com> wrote:
> OK, thanks for the info. It's interesting, though, that I can code something
> like this in Cython and it works:
>
> cdef class Foo:
> ?? ?foos = {}
> ?? ?def __cinit__(self, name):
> ?? ??? ?Foo.foos[name] = self
>
>>>> a = Foo('Larry')
>>>> b = Foo('Bob')
>>>> a.foos
> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> 0x01C5F488>}
>>>> b.foos
> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> 0x01C5F488>}
>

But that is a different beast. You are doing a 'getattr', and next
updating the attribute. What you cannot do on a cdef class is
'setattr'. Python rules for extension types...

BTW, such code creates reference cycles. If you are going to create a
lot of these objects, GC collection will have to traverse a large
dict, slowing down your runs each time the collection triggers.

> Perhaps I could use a hack workaround with the aforementioned count variable
> by storing it inside a class level list object.
>

Well, if the extra dict lookup is not an issue for you, go for it....



> 2009/6/2 Chris Colbert <sccolbert at gmail.com>
>>
>> This is correct, forget what I said. I should have tried it out before I
>> said anything. My apologies.
>>
>>
>>
>> On Tue, Jun 2, 2009 at 12:44 PM, Lisandro Dalcin <dalcinl at gmail.com>
>> wrote:
>>>
>>> No, that will not work. AFAIK, Class attributes are not currently
>>> supported for cdef classes.
>>>
>>> Juha, you will have to code like this:
>>>
>>>
>>> # foo.pyx
>>>
>>> cdef int Foo_count = 0
>>>
>>> cdef class Foo:
>>>
>>> ? ?def __cinit__(self):
>>> ? ? ? ?global Foo_count
>>> ? ? ? ?Foo_count += 1
>>>
>>>
>>>
>>>
>>> On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com>
>>> wrote:
>>> > you have to declare the attribute as public.
>>> >
>>> > http://docs.cython.org/docs/extension_types.html
>>> >
>>> > Cheers,
>>> >
>>> > Chris
>>> >
>>> > On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
>>> >>
>>> >> Unfortunately that didn't seem to work. I got the following error
>>> >> after
>>> >> the change:
>>> >> >>> a = Foo()
>>> >> Traceback (most recent call last):
>>> >> ??File "<stdin>", line 1, in <module>
>>> >> ??File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>>> >> ?? ?self.count += 1
>>> >> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>>> >>
>>> >> To clarify, I want the value of the count variable to remain the same
>>> >> between all instances. So if I create 3 Foo objects, each object
>>> >> should show
>>> >> 3 as the value in the count variable.
>>> >>
>>> >> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>>> >>>
>>> >>> Good day.
>>> >>> You should try self.count += 1 instead of Foo.count += 1.
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Cython-dev mailing list
>>> >> Cython-dev at codespeak.net
>>> >> http://codespeak.net/mailman/listinfo/cython-dev
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Cython-dev mailing list
>>> > Cython-dev at codespeak.net
>>> > http://codespeak.net/mailman/listinfo/cython-dev
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue Jun  2 20:28:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Jun 2009 11:28:19 -0700
Subject: [Cython] PATCH: get rid of __Pyx_StrEq hackery at
	__Pyx_Method_ClassMethod
In-Reply-To: <e7ba66e40906020951w2f98cb72x55741d3586099ab8@mail.gmail.com>
References: <e7ba66e40906020951w2f98cb72x55741d3586099ab8@mail.gmail.com>
Message-ID: <58D679FC-CBD2-46A9-AAA3-55CE06DF9607@math.washington.edu>

Yes, that should work fine. Please make a ticket and push.

On Jun 2, 2009, at 9:51 AM, Lisandro Dalcin wrote:

> Review this. Tell me if you like it.
>
> AFAIK, this should always work (even iff an embedder plays with
> loading/unloading the Python shared library ??)
>
> diff -r 88fa346e169d Cython/Compiler/Symtab.py
> --- a/Cython/Compiler/Symtab.py	Tue May 26 22:54:46 2009 +0200
> +++ b/Cython/Compiler/Symtab.py	Tue Jun 02 13:45:11 2009 -0300
> @@ -1583,8 +1583,14 @@
>  impl = """
>  static PyObject* __Pyx_Method_ClassMethod(PyObject *method) {
>      /* It appears that PyMethodDescr_Type is not anywhere exposed in
> the Python/C API */
> -    /* if (!PyObject_TypeCheck(method, &PyMethodDescr_Type)) { */
> -    if (__Pyx_StrEq(Py_TYPE(method)->tp_name, "method_descriptor")) {
> /* cdef classes */
> +    static PyTypeObject *methoddescr_type = NULL;
> +    if (methoddescr_type == NULL) {
> +       PyObject *meth = __Pyx_GetAttrString((PyObject*) 
> &PyList_Type, "append");
> +       if (!meth) return NULL;
> +       methoddescr_type = Py_TYPE(meth);
> +       Py_DECREF(meth);
> +    }
> +    if (PyObject_TypeCheck(method, methoddescr_type)) { /* cdef  
> classes */
>          PyMethodDescrObject *descr = (PyMethodDescrObject *)method;
>          return PyDescr_NewClassMethod(descr->d_type, descr- 
> >d_method);
>      }
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From seb.binet at gmail.com  Tue Jun  2 20:33:51 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Tue, 2 Jun 2009 20:33:51 +0200
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>
	<e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>
Message-ID: <200906022033.52142.binet@cern.ch>

On Tuesday 02 June 2009 20:03:25 Lisandro Dalcin wrote:
> On Tue, Jun 2, 2009 at 2:48 PM, Juha Salo <jusa.sj at gmail.com> wrote:
> > OK, thanks for the info. It's interesting, though, that I can code
> > something like this in Cython and it works:
> >
> > cdef class Foo:
> >     foos = {}
> >     def __cinit__(self, name):
> >         Foo.foos[name] = self
> >
> >>>> a = Foo('Larry')
> >>>> b = Foo('Bob')
> >>>> a.foos
> >
> > {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> > 0x01C5F488>}
> >
> >>>> b.foos
> >
> > {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> > 0x01C5F488>}
>
> But that is a different beast. You are doing a 'getattr', and next
> updating the attribute. What you cannot do on a cdef class is
> 'setattr'. Python rules for extension types...
>
> BTW, such code creates reference cycles. 
which are usually easily broken if one were to use a 
weakref.WeakValueDictionar :)

> If you are going to create a
> lot of these objects, GC collection will have to traverse a large
> dict, slowing down your runs each time the collection triggers.

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From stefan_ml at behnel.de  Tue Jun  2 20:49:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Jun 2009 20:49:17 +0200
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <200906022033.52142.binet@cern.ch>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>	<20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>	<e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>
	<200906022033.52142.binet@cern.ch>
Message-ID: <4A25742D.2060109@behnel.de>


Sebastien Binet wrote:
> On Tuesday 02 June 2009 20:03:25 Lisandro Dalcin wrote:
>> On Tue, Jun 2, 2009 at 2:48 PM, Juha Salo wrote:
>>> OK, thanks for the info. It's interesting, though, that I can code
>>> something like this in Cython and it works:
>>>
>>> cdef class Foo:
>>>     foos = {}
>>>     def __cinit__(self, name):
>>>         Foo.foos[name] = self
>>>
>>>>>> a = Foo('Larry')
>>>>>> b = Foo('Bob')
>>>>>> a.foos
>>> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
>>> 0x01C5F488>}
>>>
>>>>>> b.foos
>>> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
>>> 0x01C5F488>}
>> But that is a different beast. You are doing a 'getattr', and next
>> updating the attribute. What you cannot do on a cdef class is
>> 'setattr'. Python rules for extension types...
>>
>> BTW, such code creates reference cycles. 
> which are usually easily broken if one were to use a 
> weakref.WeakValueDictionar :)

... in which case the overhead is so high that you can just as well use a
plain Python class instead of an extension class.

[benchmark results left out as an exercise for the reader]

Stefan

From dalcinl at gmail.com  Wed Jun  3 02:22:26 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 21:22:26 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
Message-ID: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>

No time right now to look at this... All doctests fail on Py3.1rc1,
with errors like the traceback below. Are we needing a (trivial?) fix
or is something broken on Py3.1?

======================================================================
ERROR: compiling (cpp) and running withstat
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 345, in run
    doctest.DocTestSuite(self.module).run(result)
  File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 2226, in
DocTestSuite
    tests = test_finder.find(module, globs=globs, extraglobs=extraglobs)
  File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 820, in find
    source_lines = linecache.getlines(file, module.__dict__)
  File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 41, in getlines
    return updatecache(filename, module_globals)
  File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 130,
in updatecache
    lines = fp.readlines()
  File "/usr/local/python/3.1/lib/python3.1/codecs.py", line 300, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x94 in position
100: unexpected code byte


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Jun  3 09:00:13 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 3 Jun 2009 09:00:13 +0200 (CEST)
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
Message-ID: <cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> No time right now to look at this... All doctests fail on Py3.1rc1,
> with errors like the traceback below. Are we needing a (trivial?) fix
> or is something broken on Py3.1?
>
> ======================================================================
> ERROR: compiling (cpp) and running withstat
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "runtests.py", line 345, in run
>     doctest.DocTestSuite(self.module).run(result)
>   File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 2226, in
> DocTestSuite
>     tests = test_finder.find(module, globs=globs, extraglobs=extraglobs)
>   File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 820, in find
>     source_lines = linecache.getlines(file, module.__dict__)
>   File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 41, in
> getlines
>     return updatecache(filename, module_globals)
>   File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 130,
> in updatecache
>     lines = fp.readlines()
>   File "/usr/local/python/3.1/lib/python3.1/codecs.py", line 300, in
> decode
>     (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x94 in position
> 100: unexpected code byte

If all doctests fail like this, I doubt that there's much we can do about
this. I wonder why the UTF-8 codec is involved at all here, we are using
Unicode strings for "__doc__" everywhere in Py3 compatible tests. And why
does it look like linecache is reading from a file (even a filename!)
here? I can't look at the sources right now, but this feels awfully wrong
to me. I hope it's not trying to read the binary modules as text files...

I think we should file a bug report against Py3.1 and even bring this to
the attention of python-dev before it's too late.

Stefan


From collette at physics.ucla.edu  Wed Jun  3 09:07:18 2009
From: collette at physics.ucla.edu (Andrew Collette)
Date: Wed, 3 Jun 2009 00:07:18 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
Message-ID: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>

Hi,

I noticed there's an errant C99-style comment in the generated C code
for the method __Pyx_PyObject_Append:

return Py_None; // this is just to have an accurate signature

Unfortunately this breaks compilation with the -ansi option, which is
currently how my project is compiled on OS X.  This is with the
current public version of Cython (0.11.2).  As a temporary workaround
I'm using the += operator instead of append().

Andrew Collette

From robertwb at math.washington.edu  Wed Jun  3 09:27:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Jun 2009 00:27:46 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
In-Reply-To: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
References: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
Message-ID: <72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>

On Jun 3, 2009, at 12:07 AM, Andrew Collette wrote:

> Hi,
>
> I noticed there's an errant C99-style comment in the generated C code
> for the method __Pyx_PyObject_Append:
>
> return Py_None; // this is just to have an accurate signature
>
> Unfortunately this breaks compilation with the -ansi option, which is
> currently how my project is compiled on OS X.  This is with the
> current public version of Cython (0.11.2).  As a temporary workaround
> I'm using the += operator instead of append().

Strange--it looks like this was fixed months ago:

http://hg.cython.org/cython-devel/diff/8b6d42d16c99/Cython/Compiler/ 
ExprNodes.py

Maybe you need to touch the .pyx to re-generate the sources. It is  
gratifying to hear that is the only issue--should we be running our  
tests with this option? (Or is it gcc only?)

- Robert


From robertwb at math.washington.edu  Wed Jun  3 11:53:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Jun 2009 02:53:55 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>
	<C2F2563A-B960-422C-8E31-E5CD100209CA@math.washington.edu>
	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>
	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>
	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>
	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
Message-ID: <0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>

OK, I finally ran some simple benchmarks. To summarize, it looks like  
the non-naive algorithm is 4 times slower on my computer for doubles  
(see the code I used below).

-------- complex.pyx ----------

def time_div(double complex z, int N):
     cdef double complex a = 1
     cdef int i
     for i from 0 <= i < N:
         a = a/z
     return a

------ C code (toggled manually) --------

     static INLINE %(type)s %(type_name)s_div_fast(%(type)s a, %(type) 
s b) {
         %(type)s z;
         %(real_type)s denom = b.real*b.real + b.imag*b.imag;
         z.real = (a.real * b.real + a.imag * b.imag) / denom;
         z.imag = (a.imag * b.real - a.real * b.imag) / denom;
         return z;
     }

     static INLINE %(type)s %(type_name)s_div_slow(%(type)s a, %(type) 
s b) {
         %(type)s z;
         const %(real_type)s abs_breal = fabs(b.real);
         const %(real_type)s abs_bimag = fabs(b.imag);
         if (abs_breal >= abs_bimag) {
             const %(real_type)s ratio = b.imag / b.real;
             const %(real_type)s denom = b.real + b.imag * ratio;
             z.real = (a.real + a.imag * ratio) / denom;
             z.imag = (a.imag - a.real * ratio) / denom;
         }
         else {
             const %(real_type)s ratio = b.real / b.imag;
             const %(real_type)s denom = b.real * ratio + b.imag;
             z.real = (a.real * ratio + a.imag) / denom;
             z.imag = (a.imag * ratio - a.real) / denom;
         }
         return z;
     }

     #define %(type_name)s_div %(type_name)s_div_slow

------- session --------

In [1]: from complex import time_div; import math; a = math.e**1j
In [2]: time abs(time_div(a, 10**7))
CPU times: user 0.10 s, sys: 0.00 s, total: 0.10 s
Wall time: 0.10 s
Out[3]: 1.0000000002480893

vs.

In [1]: from complex import time_div; import math; a = math.e**1j
In [2]: time abs(time_div(a, 10**7))
CPU times: user 0.40 s, sys: 0.00 s, total: 0.40 s
Wall time: 0.40 s
Out[3]: 1.0000000000360101

------------------------------------

The difference in accumulated error after ten million divisions is:

sage: (1 - 1.0000000002480893) / (1 - 1.0000000000360101)
6.8894367097208654

 From before, relative errors for the naive vs. other algorithm,  
100000 runs, uniformly chosen in unit square (though nearly all  
distributions look basically the same):

naive
     better 26187
     avg 1.4940916064705992895601085724e-16
     worst 5.7659574333851360909896621025e-16
other
     better 63116
     avg 9.5414951065097745299683547276e-17
     worst 3.9584519821557590591785765975e-16

- Robert


From jonovik at gmail.com  Wed Jun  3 13:09:08 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Wed, 3 Jun 2009 13:09:08 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
Message-ID: <cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>

Dear Cythoneers,

I am simulating ordinary differential equation models with Pysundials
[1], a wrapper around the Sundials [2] suite of ODE solvers (written
in C). The ODE is specified by a Python function like:

def odefun(t, y, ydot, user_data):
? ?ydot[0] = ...
? ?ydot[1] = ...
? ?...
? ?return 0

corresponding to the Sundials convention [3]:

typedef int (*CVRhsFn)( realtype t, N_Vector y, N_Vector ydot, void *user_data);

After an inspiring workshop with Dag Sverre Seljebotn [4], I see that
Cython could routinely speed up my simulation work by a factor of
hundreds. My benchmark used Numpy arrays for y and ydot, and then
cythonizing the ODE function was straightforward. However, CVODE (the
Sundials solver that I use) requires that y and ydot be of type
N_Vector [5]. Pysundials wraps that type, but at the cost of
inefficient Python indexing.

Somewhere inside pysundials.cvode.NVector is an array of double
screaming to get out [6]. However, Dag Sverre told me that passing it
to Cython may require converting between ctypes and Cython pointers.
Moreover, the solver makes lots of calls to odefun(), and these would
still be Python-based. Therefore, it is probably best to bypass
Pysundials altogether and call CVODE directly.

Typical use of CVODE [7] involves manual memory allocation and lots of
routine steps. Pysundials faithfully replicates this somewhat
unpythonic interface, which could make it fairly straightforward to
convert my code (which includes some higher-level wrappers) to call
the CVODE C library directly.

I typically use ODE functions autogenerated from a model repository
[8]. Although so far I've been generating Python code [9], with Cython
I'd use C directly [10].

So, to summarize:
I would be very grateful for any help in getting started with calling
the Sundials libraries from Cython. I'm running Python 2.5.2 with
Numpy 1.1.0 under Linux:
-bash-3.2$ dmesg | head -1
Linux version 2.6.18-92.1.13.el5 (mockbuild at builder10.centos.org) (gcc
version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Sep 24 19:32:05
EDT 2008

Subtasks include:
* Installing Sundials from [2] (Done; test programs run successfully.
Also, Pysundials uses the same Sundials installation.)
* (Optional: Experienced Cythoneers might perhaps find it informative
to see how Pysundials wraps things; in that case, install from [1]
(and watch out for the CPATH gotcha mentioned on that page))
* Telling Cython where to find cvode.h, nvector.h, sundials.h (see [11])
* Importing anything from Sundials (see [12]). Currently "python
setup.py build_ext --inplace" complains about not finding nvector.h
[13]
* Allocating an NVector with Sundials (see [3, 7]).
* Handle the NVector from Python: exchange data with a Numpy array,
and pass the NVector to a C or Cython function.
* Compiling a C function (for the ODE right-hand side) to work with
CVODE. I would modify [10] to fit the calling convention [3].

Given this, I think the other steps [7] will follow fairly easily.
Despite the lengthiness of this post, I suspect that interfacing with
Sundials would be quite straightforward if I knew which buttons to
push. The benefits would carry over to all my work with cellular ODE
models, and would hopefully have some reuse value for others. Thanks
in advance for any pointers (pardon the pun).

Best regards,
Jon Olav Vik

--
Jon Olav Vik
jonovik at gmail.com
http://www.cigene.no/people/57-members/178-vik-jon-olav.html


[1] http://pysundials.sourceforge.net/
[2] https://computation.llnl.gov/casc/sundials/
[3] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/node5.html#SECTION00561000000000000000
[4] http://wiki.cython.org/talks/notur2009
[5] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/node7.html
[6]
In [25]: from pysundials import cvode
In [26]: y = cvode.NVector([3.14]) # state vector
In [38]: y.data.contents.content.contents.data[0]
Out[38]: 3.1400000000000001
In [39]: y.data.contents.content.contents.data[1]
Out[39]: 1.1297001364744253e-312
[7] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/node5.html#SECTION00540000000000000000
[8] http://www.cellml.org/models
[9] https://tracker.physiomeproject.org/show_bug.cgi?id=1514#c28
[10] http://www.cellml.org/models/fitzhugh_1961_version05/generateRawSource?lang=C
[11] First attempt at setup.py:
"""Run with e.g. python setup.py build_ext --inplace"""
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
? ?cmdclass = {'build_ext': build_ext},
? ?ext_modules = [
? ? ? ?Extension("hello",
? ? ? ? ? ?["hello.pyx"],
? ? ? ? ? ?include_dirs=['/site/VERSIONS/sundials-2.3.0/include/cvode',
? ? ? ? ? ? ? ?'/site/VERSIONS/sundials-2.3.0/include/nvector',
? ? ? ? ? ? ? ?'/site/VERSIONS/sundials-2.3.0/include/sundials'],
? ? ? ? ? ?library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
? ? ? ? ? ?libraries=['cvode']),
? ?]
)
[12] First attempt at hello.pyx:
cdef extern from "cvode.h":
? ?int CVodeSetFdata(void *cvode_mem, void *f_data)
[13] Failed attempt at building:
-bash-3.2$ python setup.py build_ext --inplace
running build_ext
building 'hello' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototy
? ? ? ? ? ? ? pes -fPIC -I/site/VERSIONS/sundials-2.3.0/include/cvode
-I/site/VERSIONS/sundial
? ? ? ? ? ? ? ? ? ? ? s-2.3.0/include/nvector
-I/site/VERSIONS/sundials-2.3.0/include/sundials -I/site

/VERSIONS/compython-2.5/Linux/include/python2.5 -c hello.c -o
build/temp.linux-x
? ? ? ? ? ? ? ? 86_64-2.5/hello.o
In file included from hello.c:135:
/site/VERSIONS/sundials-2.3.0/include/cvode/cvode.h:37:39: error:
sundials/sundi
? ? ? ? ? ? als_nvector.h: No such file or directory
In file included from hello.c:135:
/site/VERSIONS/sundials-2.3.0/include/cvode/cvode.h:177: error:
expected ')' bef
? ? ? ? ? ? ? ore 't'
...



-- 
Jon Olav Vik
jonovik at gmail.com
http://www.cigene.no/people/57-members/178-vik-jon-olav.html

From dagss at student.matnat.uio.no  Wed Jun  3 13:42:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Jun 2009 13:42:54 +0200
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>	<C2F2563A-B960-422C-8E31-E5CD100209CA@math.washington.edu>	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
Message-ID: <4A2661BE.8030605@student.matnat.uio.no>

Robert Bradshaw wrote:
>  From before, relative errors for the naive vs. other algorithm,  
> 100000 runs, uniformly chosen in unit square (though nearly all  
> distributions look basically the same):
>
> naive
>      better 26187
>      avg 1.4940916064705992895601085724e-16
>      worst 5.7659574333851360909896621025e-16
> other
>      better 63116
>      avg 9.5414951065097745299683547276e-17
>      worst 3.9584519821557590591785765975e-16
>
> - Robert
>   

This is not my speciality, but since the problem here is with things 
like fp cancellation on subtraction etc., wouldn't it be better to 
increase the odds of wildly different values?

Something like exp(uniform square)?

Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun  3 14:11:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Jun 2009 14:11:29 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
Message-ID: <4A266871.9070209@student.matnat.uio.no>

Jon Olav Vik wrote:
> Dear Cythoneers,
>   
Hi again! I'll be brief, suspecting that it's better with a quick answer 
now than putting it off until I don't remember getting back. Please get 
back with any obstacles that this doesn't solve.
> Subtasks include:
> * Installing Sundials from [2] (Done; test programs run successfully.
> Also, Pysundials uses the same Sundials installation.)
> * (Optional: Experienced Cythoneers might perhaps find it informative
> to see how Pysundials wraps things; in that case, install from [1]
> (and watch out for the CPATH gotcha mentioned on that page))
> * Telling Cython where to find cvode.h, nvector.h, sundials.h (see [11])
>   
I suspect the problem here is including "cvode", "nvector" etc. directly 
in include_path. Instead you should likely do

cdef extern from "cvode/....h":
    ...

and in setup.py simly have include_paths=[".../include"] rather than 
[".../include/cvode", ".../include/nvector", ...]. (This is how 
libraries are usually set up anyway.)

> * Importing anything from Sundials (see [12]). Currently "python
> setup.py build_ext --inplace" complains about not finding nvector.h
> [13]
> * Allocating an NVector with Sundials (see [3, 7]).
>   
Note that the call seems to allocate the ydot NVector for you (whether 
you still need to allocate other NVector's I didn't bother to check now; 
but it is likely straightforward).

The challenge here though is about function pointers.

Basically you need to define
cdef extern from ...:
   ctypedef double realtype
   ctypedef struct N_Vector:
      ...more or less copy definition from nvector header files...
   ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void 
*user_data)
   int CVodeInit(void* mem, CVRhsFn f, realtype t0, N_Vector y0)

Note that here CVRhsFn is a /type/, it is a pointer to a function, and 
CVodeInit is a function which takes such a pointer (a "callback"). So 
you'd do

cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
   ...

CVodeInit(..., your_rhs, t0, y0)

and now Sundials will know that your_rhs should be called.

Furthermore, user_data is for your use, for passing additional 
parameters to your_rhs. It can e.g. be a Python object (or Cython cdef 
class object) storing values your_rhs needs, as long as you cast it:

cdef class MyRhsProperties:
    ...

cdef MyRhsProperties props = ...
some_solve_function_taking_user_data(..., <void*>props, ...)

cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
     cdef MyRhsProperties props = <MyRhsProperties>user_data
     ...

Further refining this we get a typical pattern in these situations:

cdef class SundialsRhs:
    cdef int step(realtype t, N_vector y, N_vector ydot):
        ...

cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
     (<MyRhsProperties>user_data).step(t, y, ydot)

def foosolve(..., SundialsRhs rhs):
    some_solve_function_taking_user_data(..., <void*>rhs, ...)

which (for a small virtual dispatch penalty) allows you to subclass 
SundialsRhs to provide the callback function.

> * Handle the NVector from Python: exchange data with a Numpy array,
> and pass the NVector to a C or Cython function.
>   
Can you find the definition of NVector and paste it in an email if you 
don't succeed here? Having to fetch Sundials just to provide hints is a 
bit inconvenient...

Dag Sverre

From dalcinl at gmail.com  Wed Jun  3 18:08:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 3 Jun 2009 13:08:11 -0300
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
Message-ID: <e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>

On Wed, Jun 3, 2009 at 8:09 AM, Jon Olav Vik <jonovik at gmail.com> wrote:
>
> Somewhere inside pysundials.cvode.NVector is an array of double
> screaming to get out [6]. However, Dag Sverre told me that passing it
> to Cython may require converting between ctypes and Cython pointers.

if you have a 'ctypes' object , you can use ctypes.addressof() to get
a integer (ctypes uses PyLong_FromVoidPtr()) containing the underlying
pointer. Next, in Cython code you can use core Python C-API call
PyLong_AsVoidPtr() to get the C-side pointer.

Of course, if your odefun() is tiny, this will not alleviate the
overhead of the Python layer.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Jun  3 18:53:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 3 Jun 2009 18:53:24 +0200 (CEST)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
 differential equation models with CVODE
In-Reply-To: <e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>
Message-ID: <7626197aeb36199a01b1beaa0f266655.squirrel@webmail.uio.no>

Lisandro wrote:
> On Wed, Jun 3, 2009 at 8:09 AM, Jon Olav Vik <jonovik at gmail.com> wrote:
>>
>> Somewhere inside pysundials.cvode.NVector is an array of double
>> screaming to get out [6]. However, Dag Sverre told me that passing it
>> to Cython may require converting between ctypes and Cython pointers.
>
> if you have a 'ctypes' object , you can use ctypes.addressof() to get
> a integer (ctypes uses PyLong_FromVoidPtr()) containing the underlying
> pointer. Next, in Cython code you can use core Python C-API call
> PyLong_AsVoidPtr() to get the C-side pointer.
>
> Of course, if your odefun() is tiny, this will not alleviate the
> overhead of the Python layer.

Thanks, I wasn't aware of the addressof call. That's definitely convenient.

One thing we might want to do though is ship a ctypes.pxd for a bit faster
access directly to ctypes pointers.

Finally, while the topic is up, I personally have a wishlist item for
Cython which is automatic coercion between pointers and ctypes objects:

cdef int* i
print i # prints <ctypes.POINTER(ctypes.c_int)...> or somesuch

90% of my motivation for this behaviour is that then all cdef functions
would also be cpdef-able, and removing the need to explain that some
objects cannot be coerced to objects and back :-) Basically just more
language consistency.

Dag Sverre


From collette at physics.ucla.edu  Wed Jun  3 20:31:55 2009
From: collette at physics.ucla.edu (Andrew Collette)
Date: Wed, 3 Jun 2009 11:31:55 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
In-Reply-To: <72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>
References: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
	<72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>
Message-ID: <f125f5e0906031131nebf9677tca009911f4861bf5@mail.gmail.com>

Hi,

> Strange--it looks like this was fixed months ago:
>
> http://hg.cython.org/cython-devel/diff/8b6d42d16c99/Cython/Compiler/
> ExprNodes.py

It looks like for some reason Cython wasn't being upgraded correctly;
after sudo python setup.py install, cython --version still reported
0.10.3.  I had to manually delete the Cython directory in
site-packages and reinstall to get it to report 0.11.2.  But it works
now!

> Maybe you need to touch the .pyx to re-generate the sources. It is
> gratifying to hear that is the only issue--should we be running our
> tests with this option? (Or is it gcc only?)

I think all major compilers have a strict-ansi mode, although I'm sure
the switch is different for VC++.  I currently only use it with gcc on
OS X for compatibility with another package that looks for
__STRICT_ANSI__.  However, if Cython is intended to generate pure C89
code it might be worth adding this as a test.  At the moment it seems
this is already the case; my large-ish (9k line) Cython project now
compiles fine with -ansi.

Thanks!
Andrew

From robertwb at math.washington.edu  Wed Jun  3 21:11:22 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Jun 2009 12:11:22 -0700
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <7626197aeb36199a01b1beaa0f266655.squirrel@webmail.uio.no>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>
	<7626197aeb36199a01b1beaa0f266655.squirrel@webmail.uio.no>
Message-ID: <C9BF6336-E87B-454B-B2B8-367A60E3057E@math.washington.edu>

On Jun 3, 2009, at 9:53 AM, Dag Sverre Seljebotn wrote:

> Lisandro wrote:
>> On Wed, Jun 3, 2009 at 8:09 AM, Jon Olav Vik <jonovik at gmail.com>  
>> wrote:
>>>
>>> Somewhere inside pysundials.cvode.NVector is an array of double
>>> screaming to get out [6]. However, Dag Sverre told me that  
>>> passing it
>>> to Cython may require converting between ctypes and Cython pointers.
>>
>> if you have a 'ctypes' object , you can use ctypes.addressof() to get
>> a integer (ctypes uses PyLong_FromVoidPtr()) containing the  
>> underlying
>> pointer. Next, in Cython code you can use core Python C-API call
>> PyLong_AsVoidPtr() to get the C-side pointer.
>>
>> Of course, if your odefun() is tiny, this will not alleviate the
>> overhead of the Python layer.
>
> Thanks, I wasn't aware of the addressof call. That's definitely  
> convenient.
>
> One thing we might want to do though is ship a ctypes.pxd for a bit  
> faster
> access directly to ctypes pointers.
>
> Finally, while the topic is up, I personally have a wishlist item for
> Cython which is automatic coercion between pointers and ctypes  
> objects:
>
> cdef int* i
> print i # prints <ctypes.POINTER(ctypes.c_int)...> or somesuch
>
> 90% of my motivation for this behaviour is that then all cdef  
> functions
> would also be cpdef-able, and removing the need to explain that some
> objects cannot be coerced to objects and back :-) Basically just more
> language consistency.

Sounds mostly like a good idea to me (though it would be exposing the  
messiness of raw C memory management to Python). I think there's a  
lot we could do with ctypes integration, at least for Python 2.5+.

- Robert



From jonovik at gmail.com  Thu Jun  4 00:30:55 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Wed, 3 Jun 2009 22:30:55 +0000 (UTC)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
Message-ID: <loom.20090603T221740-806@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:
> > * Telling Cython where to find cvode.h, nvector.h, sundials.h (see [11])
> >   
> I suspect the problem here is including "cvode", "nvector" etc. directly 
> in include_path. Instead you should likely do
> 
> cdef extern from "cvode/....h":
>     ...
> 
> and in setup.py simly have include_paths=[".../include"] rather than 
> [".../include/cvode", ".../include/nvector", ...]. (This is how 
> libraries are usually set up anyway.)

Thanks for the information; I'm not familiar with C libraries. Having looked in 
the /lib directory, I found that the library is called sundials_cvode, not just 
cvode. I modified setup.py to use "libraries=['sundials_cvode'])" and could 
then recompile without error. (I had to manually remove the build/ directory 
and hello.c, though.)

== Successful compilation ==
bash-3.2$ python setup.py build_ext --inplace
running build_ext
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/sundials-2.3.0/include -I/site/VERSIONS/
compython-2.5/Linux/include/python2.5 -c hello.c -o build/temp.linux-x86_64-2.5/
hello.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/hello.o -L/site/VERSIONS/
sundials-2.3.0/lib -L/site/VERSIONS/compython-2.5/Linux/lib -lsundials_cvode -
lpython2.5 -o hello.so

== setup.py ==
"""Run with e.g. python setup.py build_ext --inplace"""
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("hello",
            ["hello.pyx"],
            include_dirs=['/site/VERSIONS/sundials-2.3.0/include'],
            library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
            libraries=['sundials_cvode']),
    ]
)

== hello.pyx ==
cdef extern from "cvode/cvode.h":
   int CVodeSetFdata(void *cvode_mem, void *f_data)

> > * Importing anything from Sundials (see [12]). Currently "python
> > setup.py build_ext --inplace" complains about not finding nvector.h
> > [13]

==> Solved 8-)

I chose CVodeSetFdata for the above example because it didn't depend on any 
user-defined types. Now onto the next challenge...

> > * Allocating an NVector with Sundials (see [3, 7]).
> >   
> Note that the call seems to allocate the ydot NVector for you (whether 
> you still need to allocate other NVector's I didn't bother to check now; 
> but it is likely straightforward).

Yes, ydot is allocated by CVODE (though not initialized to zero, as I learned 
the hard way). I do need to allocate y, however.

As a start, I would like Cython to compile the following line of code:

y0 = N_VNew_Serial(N)

However, I'm unsure how much of the C declarations I need to duplicate (with 
ctypedef?). N_VNew_Serial returns an N_Vector:
include/nvector/nvector_serial.h:N_Vector N_VNew_Serial(long int vec_length);

The generic definition of an N_Vector is here:

== BEGIN include/sundials/sundials_nvector.h (excerpt) ==
#include <sundials/sundials_types.h>

/*
 * -----------------------------------------------------------------
 * Generic definition of N_Vector
 * -----------------------------------------------------------------
 */

/* Forward reference for pointer to N_Vector_Ops object */
typedef struct _generic_N_Vector_Ops *N_Vector_Ops;

/* Forward reference for pointer to N_Vector object */
typedef struct _generic_N_Vector *N_Vector;

/* Define array of N_Vectors */
typedef N_Vector *N_Vector_S;

/* Structure containing function pointers to vector operations  */  
struct _generic_N_Vector_Ops {
  N_Vector    (*nvclone)(N_Vector);
  N_Vector    (*nvcloneempty)(N_Vector);
  void        (*nvdestroy)(N_Vector);
  void        (*nvspace)(N_Vector, long int *, long int *);
  realtype*   (*nvgetarraypointer)(N_Vector);
  void        (*nvsetarraypointer)(realtype *, N_Vector);
  void        (*nvlinearsum)(realtype, N_Vector, realtype, N_Vector, N_Vector); 
  void        (*nvconst)(realtype, N_Vector);
  void        (*nvprod)(N_Vector, N_Vector, N_Vector);
  void        (*nvdiv)(N_Vector, N_Vector, N_Vector);
  void        (*nvscale)(realtype, N_Vector, N_Vector);
  void        (*nvabs)(N_Vector, N_Vector);
  void        (*nvinv)(N_Vector, N_Vector);
  void        (*nvaddconst)(N_Vector, realtype, N_Vector);
  realtype    (*nvdotprod)(N_Vector, N_Vector);
  realtype    (*nvmaxnorm)(N_Vector);
  realtype    (*nvwrmsnorm)(N_Vector, N_Vector);
  realtype    (*nvwrmsnormmask)(N_Vector, N_Vector, N_Vector);
  realtype    (*nvmin)(N_Vector);
  realtype    (*nvwl2norm)(N_Vector, N_Vector);
  realtype    (*nvl1norm)(N_Vector);
  void        (*nvcompare)(realtype, N_Vector, N_Vector);
  booleantype (*nvinvtest)(N_Vector, N_Vector);
  booleantype (*nvconstrmask)(N_Vector, N_Vector, N_Vector);
  realtype    (*nvminquotient)(N_Vector, N_Vector);
};

/*
 * -----------------------------------------------------------------
 * A vector is a structure with an implementation-dependent
 * 'content' field, and a pointer to a structure of vector
 * operations corresponding to that implementation.
 * -----------------------------------------------------------------
 */

struct _generic_N_Vector {
  void *content;
  struct _generic_N_Vector_Ops *ops;
};
== END include/sundials/sundials_nvector.h (excerpt) ==

(Pardon the long quote; unfortunately, I couldn't find a web repository for the 
Sundials source code. An earlier version is in this Ubuntu package, but note 
that the file hierarchy is different than the include/sundials/, include/
cvode/, etc.:
http://www.google.com/codesearch/p?hl=en#jVvm31OatEM/sundials/cvode/include/
cvode.h&q=nvector%20sundials%20cvode

(For all I know, the file hierarchy may be different in the source distribution 
and once it is installed as a library -- I'm really an absolute beginner in the 
C world.)

The abovementioned realtype (and booleantype) is defined in
include/sundials/sundials_types.h
from which I roughly quote some conditional defines:
#if defined(SUNDIALS_SINGLE_PRECISION)
typedef float realtype;
#elif defined(SUNDIALS_DOUBLE_PRECISION)
typedef double realtype;
#elif defined(SUNDIALS_EXTENDED_PRECISION)
typedef long double realtype;

As for the structs, I've tried and failed along these lines:
cdef extern from "sundials/sundials_nvector.h":
    ctypedef struct _generic_N_Vector
    ctypedef struct _generic_N_Vector *N_Vector
                                     ^
------------------------------------------------------------
/xanadu/home/jonvi/svn/trunk/cysundials/hello.pyx:8:38: Syntax error in struct 
or union definition

Do I really have to replicate the entire struct definition?


Thanks a lot for your help!

Regards,
Jon Olav



From dagss at student.matnat.uio.no  Thu Jun  4 09:25:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 09:25:04 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <loom.20090603T221740-806@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
Message-ID: <4A2776D0.7090405@student.matnat.uio.no>

Jon Olav Vik wrote:
> Yes, ydot is allocated by CVODE (though not initialized to zero, as I learned 
> the hard way). I do need to allocate y, however.
> 
> As a start, I would like Cython to compile the following line of code:
> 
> y0 = N_VNew_Serial(N)
> 
> However, I'm unsure how much of the C declarations I need to duplicate (with 
> ctypedef?). N_VNew_Serial returns an N_Vector:
> include/nvector/nvector_serial.h:N_Vector N_VNew_Serial(long int vec_length);
> 
> The generic definition of an N_Vector is here:
> 
> == BEGIN include/sundials/sundials_nvector.h (excerpt) ==
> #include <sundials/sundials_types.h>
> 
> /*
>  * -----------------------------------------------------------------
>  * Generic definition of N_Vector
>  * -----------------------------------------------------------------
>  */
> 
> /* Forward reference for pointer to N_Vector_Ops object */
> typedef struct _generic_N_Vector_Ops *N_Vector_Ops;
> 
> /* Forward reference for pointer to N_Vector object */
> typedef struct _generic_N_Vector *N_Vector;
> 
> /* Define array of N_Vectors */
> typedef N_Vector *N_Vector_S;
> 
> /* Structure containing function pointers to vector operations  */  
> struct _generic_N_Vector_Ops {
>   N_Vector    (*nvclone)(N_Vector);
>   N_Vector    (*nvcloneempty)(N_Vector);
>   void        (*nvdestroy)(N_Vector);
...
> 
> struct _generic_N_Vector {
>   void *content;
>   struct _generic_N_Vector_Ops *ops;
> };

> The abovementioned realtype (and booleantype) is defined in
> include/sundials/sundials_types.h
> from which I roughly quote some conditional defines:
> #if defined(SUNDIALS_SINGLE_PRECISION)
> typedef float realtype;
> #elif defined(SUNDIALS_DOUBLE_PRECISION)
> typedef double realtype;
> #elif defined(SUNDIALS_EXTENDED_PRECISION)
> typedef long double realtype;
> 
> As for the structs, I've tried and failed along these lines:
> cdef extern from "sundials/sundials_nvector.h":
>     ctypedef struct _generic_N_Vector
>     ctypedef struct _generic_N_Vector *N_Vector
>                                      ^
> ------------------------------------------------------------
> /xanadu/home/jonvi/svn/trunk/cysundials/hello.pyx:8:38: Syntax error in struct 
> or union definition
> 
> Do I really have to replicate the entire struct definition?

(Trivia: This is a lot like NumPy's source code.)

No, you just have to write "enough" for Cython to know what C code to 
output. Knowing what that is probably takes some C experience though; 
I'll try to help.

First, the #if-s cannot be expressed in Cython, but Cython is going in 
the direction of the type of external typedefs not to matter. I think 
you can safely do

cdef extern from ...:
     ctypedef double realtype

It should work also when Sundials is compiled with float or long double.

Then N_Vector etc. can probably be expressed like this:

cdef extern from ....:
     cdef struct _generic_N_Vector:
         void* content
         # not necesarry to declare ops unless you will be using
         # the contents of ops from Cython

     ctypedef _generic_N_Vector* N_vector

     N_Vector N_VNew_Serial(long int vec_length)

Then you will be able to do

cdef NVector v = N_VNew_Serial(10)
cdef realtype* v_data = <realtype*>v.content
v_data[0] = 34

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun  4 09:27:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 09:27:25 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <4A2776D0.7090405@student.matnat.uio.no>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
Message-ID: <4A27775D.9010801@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> First, the #if-s cannot be expressed in Cython, but Cython is going in 
> the direction of the type of external typedefs not to matter. I think 

I meant to say that the "type size" should matter as little as possible 
for external typedefs. int vs. unsigned vs. float still matters.

Also things are not 100% ready in this area in Cython. But if you know 
you are on a system where realtype is e.g. double, then just use double 
for now and we can discuss the details of this point again later...


-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Jun  4 09:35:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Jun 2009 00:35:45 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <4A2661BE.8030605@student.matnat.uio.no>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>	<C2F2563A-B960-422C-8E31-E5CD100209CA@math.washington.edu>	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
	<4A2661BE.8030605@student.matnat.uio.no>
Message-ID: <089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>

On Jun 3, 2009, at 4:42 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>  From before, relative errors for the naive vs. other algorithm,
>> 100000 runs, uniformly chosen in unit square (though nearly all
>> distributions look basically the same):
>>
>> naive
>>      better 26187
>>      avg 1.4940916064705992895601085724e-16
>>      worst 5.7659574333851360909896621025e-16
>> other
>>      better 63116
>>      avg 9.5414951065097745299683547276e-17
>>      worst 3.9584519821557590591785765975e-16
>>
>> - Robert
>>
>
> This is not my speciality, but since the problem here is with things
> like fp cancellation on subtraction etc., wouldn't it be better to
> increase the odds of wildly different values?
>
> Something like exp(uniform square)?

I did exp(X) + exp(X)*I where X was a uniform distribution on  
[-10,10] and various other distributions, all yielded very similar  
results (which was actually surprising to me).

Any thoughts on the 4x speed difference?

- Robert


From dagss at student.matnat.uio.no  Thu Jun  4 11:00:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 11:00:59 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <4A2776D0.7090405@student.matnat.uio.no>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
Message-ID: <4A278D4B.2040804@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Jon Olav Vik wrote:
>   
>> Yes, ydot is allocated by CVODE (though not initialized to zero, as I learned 
>> the hard way). I do need to allocate y, however.
>>
>> As a start, I would like Cython to compile the following line of code:
>>
>> y0 = N_VNew_Serial(N)
>>
>> However, I'm unsure how much of the C declarations I need to duplicate (with 
>> ctypedef?). N_VNew_Serial returns an N_Vector:
>> include/nvector/nvector_serial.h:N_Vector N_VNew_Serial(long int vec_length);
>>
>> The generic definition of an N_Vector is here:
>>
>> == BEGIN include/sundials/sundials_nvector.h (excerpt) ==
>> #include <sundials/sundials_types.h>
>>
>> /*
>>  * -----------------------------------------------------------------
>>  * Generic definition of N_Vector
>>  * -----------------------------------------------------------------
>>  */
>>
>> /* Forward reference for pointer to N_Vector_Ops object */
>> typedef struct _generic_N_Vector_Ops *N_Vector_Ops;
>>
>> /* Forward reference for pointer to N_Vector object */
>> typedef struct _generic_N_Vector *N_Vector;
>>
>> /* Define array of N_Vectors */
>> typedef N_Vector *N_Vector_S;
>>
>> /* Structure containing function pointers to vector operations  */  
>> struct _generic_N_Vector_Ops {
>>   N_Vector    (*nvclone)(N_Vector);
>>   N_Vector    (*nvcloneempty)(N_Vector);
>>   void        (*nvdestroy)(N_Vector);
>>     
> ...
>   
>> struct _generic_N_Vector {
>>   void *content;
>>   struct _generic_N_Vector_Ops *ops;
>> };
>>     
>
>   
>> The abovementioned realtype (and booleantype) is defined in
>> include/sundials/sundials_types.h
>> from which I roughly quote some conditional defines:
>> #if defined(SUNDIALS_SINGLE_PRECISION)
>> typedef float realtype;
>> #elif defined(SUNDIALS_DOUBLE_PRECISION)
>> typedef double realtype;
>> #elif defined(SUNDIALS_EXTENDED_PRECISION)
>> typedef long double realtype;
>>
>> As for the structs, I've tried and failed along these lines:
>> cdef extern from "sundials/sundials_nvector.h":
>>     ctypedef struct _generic_N_Vector
>>     ctypedef struct _generic_N_Vector *N_Vector
>>                                      ^
>> ------------------------------------------------------------
>> /xanadu/home/jonvi/svn/trunk/cysundials/hello.pyx:8:38: Syntax error in struct 
>> or union definition
>>
>> Do I really have to replicate the entire struct definition?
>>     
>
> (Trivia: This is a lot like NumPy's source code.)
>
> No, you just have to write "enough" for Cython to know what C code to 
> output. Knowing what that is probably takes some C experience though; 
> I'll try to help.
>
> First, the #if-s cannot be expressed in Cython, but Cython is going in 
> the direction of the type of external typedefs not to matter. I think 
> you can safely do
>
> cdef extern from ...:
>      ctypedef double realtype
>
> It should work also when Sundials is compiled with float or long double.
>
> Then N_Vector etc. can probably be expressed like this:
>
> cdef extern from ....:
>      cdef struct _generic_N_Vector:
>          void* content
>          # not necesarry to declare ops unless you will be using
>          # the contents of ops from Cython
>
>      ctypedef _generic_N_Vector* N_vector
>
>      N_Vector N_VNew_Serial(long int vec_length)
>
> Then you will be able to do
>
> cdef NVector v = N_VNew_Serial(10)
> cdef realtype* v_data = <realtype*>v.content
> v_data[0] = 34
>   
Make sure to also free/delete the vector though! (There must be a 
corresponding call there somewhere)

Dag Sverre

From barthelemy at crans.org  Thu Jun  4 11:38:20 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Thu, 4 Jun 2009 11:38:20 +0200
Subject: [Cython] running doctests in pyx files
Message-ID: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>

Hello,

I'm converting some python modules to cython ones. The original
function have doctests which I would obviously like to keep. However,
if I just let them in the pyx file (in def-ed and cpdef-ed functions
docstrings), doctest does not find them.

However, each function doctstring (myfunction.__doc__) seems fine.

Does-it mean that doctest parses the source file instead of using the
docstring from the python runtime? Or am I mistaken?

Any explanation and workaround is welcome !

a simple example is attached. I run it with python  2.6.2 (ubuntu
intrepid). Cython version 0.10.3.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hg.pyx
Type: application/octet-stream
Size: 1533 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090604/5e8f585a/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rundoctest.py
Type: text/x-python
Size: 104 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090604/5e8f585a/attachment.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: text/x-python
Size: 213 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090604/5e8f585a/attachment-0001.py 

From dagss at student.matnat.uio.no  Thu Jun  4 12:21:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 12:21:14 +0200
Subject: [Cython] running doctests in pyx files
In-Reply-To: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
Message-ID: <4A27A01A.70601@student.matnat.uio.no>

S?bastien Barth?lemy wrote:
> Hello,
>
> I'm converting some python modules to cython ones. The original
> function have doctests which I would obviously like to keep. However,
> if I just let them in the pyx file (in def-ed and cpdef-ed functions
> docstrings), doctest does not find them.
>
> However, each function doctstring (myfunction.__doc__) seems fine.
>
> Does-it mean that doctest parses the source file instead of using the
> docstring from the python runtime? Or am I mistaken?
>   
This is a problem with doctest, which uses inspect.is_function to check 
whether something is a function, which fails for Cython functions (which 
instead answer to inspect.is_builtin). It would be nice if someone would 
take this to the doctest list with a patch at some point...

There are various hackaround we could do in Cython, including wrapping 
every function in a dummy Python-interpreted function (speed loss 
though?) or have a directive to generate a __test__ dictionary. Best 
thing is probably to fix doctest but it takes ages before a patch there 
gets to our users even if it is accepted (Python 3.2?).

To get a solution right away, this is what I do to invoke testing on a 
module "cmbutils":

import cmbutils
import doctest
import inspect
cmbutils.__test__ = {}
for name in dir(cmbutils):
    value = getattr(cmbutils, name)
    if inspect.isbuiltin(value) and isinstance(value.__doc__, str):
        cmbutils.__test__[name] = value.__doc__
doctest.testmod(cmbutils)

You could also just set __test__ manually in the module, write a 
decorator which adds to __test__ for functions it decorates, etc.

Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun  4 12:22:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 12:22:57 +0200
Subject: [Cython] running doctests in pyx files
In-Reply-To: <4A27A01A.70601@student.matnat.uio.no>
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
	<4A27A01A.70601@student.matnat.uio.no>
Message-ID: <4A27A081.8020807@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> S?bastien Barth?lemy wrote:
>> Hello,
>>
>> I'm converting some python modules to cython ones. The original
>> function have doctests which I would obviously like to keep. However,
>> if I just let them in the pyx file (in def-ed and cpdef-ed functions
>> docstrings), doctest does not find them.
>>
>> However, each function doctstring (myfunction.__doc__) seems fine.
>>
>> Does-it mean that doctest parses the source file instead of using the
>> docstring from the python runtime? Or am I mistaken?
>>   
> This is a problem with doctest, which uses inspect.is_function to 
> check whether something is a function, which fails for Cython 
> functions (which instead answer to inspect.is_builtin). It would be 
> nice if someone would take this to the doctest list with a patch at 
> some point...
>
> There are various hackaround we could do in Cython, including wrapping 
> every function in a dummy Python-interpreted function (speed loss 
> though?) or have a directive to generate a __test__ dictionary. Best 
> thing is probably to fix doctest but it takes ages before a patch 
> there gets to our users even if it is accepted (Python 3.2?).
>
> To get a solution right away, this is what I do to invoke testing on a 
> module "cmbutils":
>
> import cmbutils
> import doctest
> import inspect
> cmbutils.__test__ = {}
> for name in dir(cmbutils):
>    value = getattr(cmbutils, name)
>    if inspect.isbuiltin(value) and isinstance(value.__doc__, str):
>        cmbutils.__test__[name] = value.__doc__
> doctest.testmod(cmbutils)
>
> You could also just set __test__ manually in the module, write a 
> decorator which adds to __test__ for functions it decorates, etc.
BTW if you get that last solution to work, it would be nice if you could 
take the time to add it to the FAQ :-)

Dag Sverre

From dalcinl at gmail.com  Thu Jun  4 16:18:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 11:18:05 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>

On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> I think we should file a bug report against Py3.1 and even bring this to
> the attention of python-dev before it's too late.
>

Stefan, I saw your post in python-dev and corresponding issue in the
tracker, but I got confused. Is the solution know and the issue going
to be fixed?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Jun  4 16:27:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 11:27:43 -0300
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>
	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>
	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>
	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>
	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
	<4A2661BE.8030605@student.matnat.uio.no>
	<089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>
Message-ID: <e7ba66e40906040727q520c8fcemdd0c28129d73f501@mail.gmail.com>

On Thu, Jun 4, 2009 at 4:35 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Jun 3, 2009, at 4:42 AM, Dag Sverre Seljebotn wrote:
>
> Any thoughts on the 4x speed difference?
>

Any chance such difference could be related to the fabs() function
call? Could you test an explicit comparison?

const %(real_type)s abs_breal = b.real < 0 ? -b.real : b.real;
const %(real_type)s abs_bimag = b.imag < 0 ? -b.imag : b.imag;


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From barthelemy at crans.org  Thu Jun  4 17:37:19 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Thu, 4 Jun 2009 17:37:19 +0200
Subject: [Cython] running doctests in pyx files
In-Reply-To: <4A27A081.8020807@student.matnat.uio.no>
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
	<4A27A01A.70601@student.matnat.uio.no>
	<4A27A081.8020807@student.matnat.uio.no>
Message-ID: <78f7ab620906040837k67e1fb82ncb32e43de83ea93@mail.gmail.com>

Hello Dag,

thank you for your help,

>> This is a problem with doctest, which uses inspect.is_function to check
>> whether something is a function, which fails for Cython functions (which
>> instead answer to inspect.is_builtin). It would be nice if someone would
>> take this to the doctest list with a patch at some point...

I had a quick look at it but there is too much I do not understand in
both cython and doctest there to work on this now.

>> To get a solution right away, this is what I do to invoke testing on a
>> module "cmbutils":
>> ...

this works fine, as long as you're not doing "from somewhere import
something" in which case, "something" is tested too (and fails because
of bad context).

I added a _from_module() test (found in doctest.py) which seem to fix that.

> BTW if you get that last solution to work, it would be nice if you could
> take the time to add it to the FAQ :-)

Done.
http://wiki.cython.org/FAQ#Doctestskipstestsfromcythonfunctions.2Cwhy.3FIsthereaworkaround.3F

regards, and thank you again.

From stefan_ml at behnel.de  Thu Jun  4 18:29:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 4 Jun 2009 18:29:07 +0200 (CEST)
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
Message-ID: <7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel wrote:
>>
>> I think we should file a bug report against Py3.1 and even bring this to
>> the attention of python-dev before it's too late.
>>
>
> Stefan, I saw your post in python-dev and corresponding issue in the
> tracker, but I got confused. Is the solution know and the issue going
> to be fixed?

I didn't find the time to look into this any deeper. I'm not sure it will
get fixed, because a) the bug itself is a lot older than rc1, and b) the
comment sounded more like "the fix might be similar to what we did here".

I think we should still open a new bug. Given that you experienced the
problem, could you do that?

Stefan


From dalcinl at gmail.com  Thu Jun  4 19:27:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 14:27:22 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A2800E6.4030806@gmail.com>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<4A2800E6.4030806@gmail.com>
Message-ID: <e7ba66e40906041027j1281d86y154e7a0497a65a6e@mail.gmail.com>

On Thu, Jun 4, 2009 at 2:14 PM, David Martin <frostedcheerios at gmail.com> wrote:
>
> I've attached a patch for my addition of the "--mingw" option to
> runtests.py.

Looks good.

> There was a small issue regarding trailing backslashes in the
> INCLUDE_DIRS list, and aside from that, I couldn't get the mingw compiler to
> work correctly unless I emptied the INCLUDE_DIRS list (I run from a VS
> command prompt for our build system, so it will always have VS directories
> in my "include" env variable). It definitely shouldn't assign an empty list
> for INCLUDE_DIRS if --mingw is given, but it is the only way I could get it
> to work quickly. =)
>

I understand...

> If you know what needs to be done to fix those errors I can definitely help
> out. I've also attached the log containing the small amount of tests that
> are failing for me when using the MinGW compiler, which I believe are the
> same tests that are failing for you.
>

I'm tempted to either remove that getenv INCLUDE, or change the name.
That getenv was needed long ago for running numpy-specific tests, It
seems we could get rid of that hackery.

What other people on cython-dev think?



> -David
>
> --- runtests.py 2009-05-20 10:37:46.000000000 -0500
> +++ runtests222.py ? ? ?2009-06-04 11:48:07.000000000 -0500
> @@ -3,6 +3,7 @@
> ?import os, sys, re, shutil, unittest, doctest
>
> ?WITH_CYTHON = True
> +MINGW = False
>
> ?from distutils.dist import Distribution
> ?from distutils.core import Extension
> @@ -32,7 +33,9 @@
> ?# ? ?(2,4) : lambda x: x in ['run.set']
> ?}
>
> -INCLUDE_DIRS = [ d for d in os.getenv('INCLUDE', '').split(os.pathsep) if d
> ]
> +# Remove any trailing backslashes in the include paths because they cause
> +# gcc to incorrectly handle spaces in the paths that follow.
> +INCLUDE_DIRS = [ d.rstrip('\\') for d in os.getenv('INCLUDE',
> '').split(os.pathsep) if d ]
> ?CFLAGS = os.getenv('CFLAGS', '').split()
>
> ?class build_ext(_build_ext):
> @@ -277,6 +280,8 @@
> ? ? ? ? os.chdir(workdir)
> ? ? ? ? try:
> ? ? ? ? ? ? build_extension = build_ext(distutils_distro)
> + ? ? ? ? ? ?if MINGW:
> + ? ? ? ? ? ? ? ?build_extension.compiler = 'mingw32'
> ? ? ? ? ? ? build_extension.include_dirs = INCLUDE_DIRS[:]
> ? ? ? ? ? ? if incdir:
> ? ? ? ? ? ? ? ? build_extension.include_dirs.append(incdir)
> @@ -505,7 +510,11 @@
> ? ? ? ? ? ? ? ? ? ? ? help="do not run the file based tests")
> ? ? parser.add_option("--no-pyregr", dest="pyregr",
> ? ? ? ? ? ? ? ? ? ? ? action="store_false", default=True,
> - ? ? ? ? ? ? ? ? ? ? ?help="do not run the regression tests of CPython in
> tests/pyregr/")
> + ? ? ? ? ? ? ? ? ? ? ?help="do not run the regression tests of CPython in
> tests/pyregr/")
> + ? ?if sys.platform.startswith('win'):
> + ? ? ? ?parser.add_option("--mingw", dest="mingw",
> + ? ? ? ? ? ? ? ? ? ? ? ? action="store_true", default=False,
> + ? ? ? ? ? ? ? ? ? ? ? ? help="compile C/C++ code using the MinGW
> compilers")
> ? ? parser.add_option("--cython-only", dest="cython_only",
> ? ? ? ? ? ? ? ? ? ? ? action="store_true", default=False,
> ? ? ? ? ? ? ? ? ? ? ? help="only compile pyx to c, do not run C compiler or
> run the tests")
> @@ -547,6 +556,11 @@
> ? ? ? ? import coverage
> ? ? ? ? coverage.erase()
> ? ? ? ? coverage.start()
> +
> + ? ?if sys.platform.startswith('win'):
> + ? ? ? ?MINGW = options.mingw
> + ? ? ? ?if MINGW:
> + ? ? ? ? ? ?INCLUDE_DIRS = []
>
> ? ? WITH_CYTHON = options.with_cython
>
>
> Running tests against Cython 0.11.2
> Python 2.5.4 (r254:67916, Mar 25 2009, 13:58:21) [MSC v.1310 32 bit (Intel)]
>
> belchenko1.cpp: In function `void initbelchenko1()':
> belchenko1.cpp:430: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x22c): undefined reference to `p1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x230): undefined reference to `f1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x236): undefined reference to `p2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x23a): undefined reference to `f2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x240): undefined reference to `p3'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x244): undefined reference to `f3 at 0'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x24a): undefined reference to `p4'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x24e): undefined reference to `@f4 at 0'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x22c): undefined reference to `p1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x230): undefined reference to `f1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x236): undefined reference to `p2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x23a): undefined reference to `f2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x240): undefined reference to `p3'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x244): undefined reference to `f3 at 0'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x24a): undefined reference to `p4'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x24e): undefined reference to `@f4 at 0'
> collect2: ld returned 1 exit status
> cnamespec.c: In function `c_spam':
> cnamespec.c:343: warning: '__pyx_v_p' might be used uninitialized in this
> function
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\cnamespec.o:cnamespec.c:(.text+0xf):
> undefined r
> eference to `c_a'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\cnamespec.o:cnamespec.c:(.text+0x20):
> undefined
> reference to `c_b'
> collect2: ld returned 1 exit status
> cnamespec.cpp: In function `double c_spam(int, float)':
> cnamespec.cpp:343: warning: '__pyx_v_p' might be used uninitialized in this
> function
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cnamespec.o:cnamespec.cpp:(.text+0xf):
> undefin
> ed reference to `c_a'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cnamespec.o:cnamespec.cpp:(.text+0x20):
> undefi
> ned reference to `c_b'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercearraytoptr.o:coercearraytoptr.c:(.text+0xd
> ): undefined reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercearraytoptr.o:coercearraytoptr.c:(.text+0x1
> 8): undefined reference to `spam'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\coercearraytoptr.o:coercearraytoptr.cpp:(.text
> +0xd): undefined reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\coercearraytoptr.o:coercearraytoptr.cpp:(.text
> +0x18): undefined reference to `spam'
> collect2: ld returned 1 exit status
> complexbasetype.cpp: In function `void initcomplexbasetype()':
> complexbasetype.cpp:433: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp_exceptions_T265.cpp:
> (.text+0x1e6): undefined reference to `generic_error()'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp_exceptions_T265.cpp:
> (.text+0x25a): undefined reference to `specified_error()'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp_exceptions_T265.cpp:
> (.text+0x2e0): undefined reference to `dynamic_error()'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\declarations.o:declarations.c:(.text+0x5):
> undef
> ined reference to `ifnp'
> collect2: ld returned 1 exit status
> declarations.cpp: In function `void initdeclarations()':
> declarations.cpp:472: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\declarations.o:declarations.cpp:(.text+0x5):
> u
> ndefined reference to `ifnp'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.o:excvalcheck.c:(.text+0x7):
> undefin
> ed reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.o:excvalcheck.c:(.text+0x33):
> undefi
> ned reference to `grail'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.o:excvalcheck.c:(.text+0x64):
> undefi
> ned reference to `tomato'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.o:excvalcheck.cpp:(.text+0x7):
> und
> efined reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.o:excvalcheck.cpp:(.text+0x33):
> un
> defined reference to `grail'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.o:excvalcheck.cpp:(.text+0x64):
> un
> defined reference to `tomato'
> collect2: ld returned 1 exit status
> extern.cpp: In function `void initextern()':
> extern.cpp:440: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x19):
> undefined referenc
> e to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x21):
> undefined referenc
> e to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x26):
> undefined referenc
> e to `e1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x2b):
> undefined referenc
> e to `e2'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x1a):
> undefined refe
> rence to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x22):
> undefined refe
> rence to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x27):
> undefined refe
> rence to `e1()'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x2c):
> undefined refe
> rence to `e2()'
> collect2: ld returned 1 exit status
>
> ======================================================================
> ERROR: compiling (c) callingconvention
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) callingconvention
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) cnamespec
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) cnamespec
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) coercearraytoptr
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) coercearraytoptr
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) cpp_exceptions_T265
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) declarations
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) declarations
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) excvalcheck
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) excvalcheck
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) nogil
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) nogil
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ----------------------------------------------------------------------
> Ran 1477 tests in 1347.264s
>
> FAILED (errors=13)
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Jun  4 20:22:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Jun 2009 11:22:55 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <e7ba66e40906040727q520c8fcemdd0c28129d73f501@mail.gmail.com>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>
	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>
	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>
	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>
	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
	<4A2661BE.8030605@student.matnat.uio.no>
	<089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>
	<e7ba66e40906040727q520c8fcemdd0c28129d73f501@mail.gmail.com>
Message-ID: <B6C1A95B-6C67-459A-B504-06B5839B55B8@math.washington.edu>

On Jun 4, 2009, at 7:27 AM, Lisandro Dalcin wrote:

> On Thu, Jun 4, 2009 at 4:35 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Jun 3, 2009, at 4:42 AM, Dag Sverre Seljebotn wrote:
>>
>> Any thoughts on the 4x speed difference?
>>
>
> Any chance such difference could be related to the fabs() function
> call? Could you test an explicit comparison?
>
> const %(real_type)s abs_breal = b.real < 0 ? -b.real : b.real;
> const %(real_type)s abs_bimag = b.imag < 0 ? -b.imag : b.imag;

This is actually what I did first, and got the 4x slowdown, so I  
switched to fabs hoping it would be faster (despite only working for  
doubles for now), but it wasn't (on my machine at least).

- Robert


From dalcinl at gmail.com  Thu Jun  4 21:48:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 16:48:59 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A28174C.7010609@gmail.com>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<4A2800E6.4030806@gmail.com>
	<e7ba66e40906041027j1281d86y154e7a0497a65a6e@mail.gmail.com>
	<4A28174C.7010609@gmail.com>
Message-ID: <e7ba66e40906041248w20c80aecm60e37f1edb9c6dbf@mail.gmail.com>

On Thu, Jun 4, 2009 at 3:49 PM, David Martin <frostedcheerios at gmail.com> wrote:
> Lisandro Dalcin wrote:
>
> I was mostly talking about the linker errors with "cdef extern" in the
> tests, but fixing the INCLUDE_DIRS issue would also be nice. If you know
> what needs to be done to fix the tests regarding cdef extern and any of the
> other failures in that log I sent you I can get to work on those.
>

Well, I'm not sure how to fix these tests without destroying what them
are trying to test. As quick fist fix would be to disable them for the
Win case.

BTW, there are things that should be improved in Cython. For example,

1) If we write "cdef extern void foo()", then Cython should emit a
prototype with DL_IMPORT
2) If we write "cdef public void bar(): pass", then Cython should emit
a prototype with DL_EXPORT

Currently, Cython used DL_EXPORT for both of them.

Of course, a key question about using DL_IMPORT... What would happen
if the 'void foo()' function is not actually in a DLL, but on a static
library, or some extra C file built together with the Cython-generated
C code to build-up the final extension module?





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonovik at gmail.com  Thu Jun  4 22:58:54 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Thu, 4 Jun 2009 20:58:54 +0000 (UTC)
Subject: [Cython]
	=?utf-8?q?Sundials_with_Cython=3F_Integrating/solving_or?=
	=?utf-8?q?dinary=09differential_equation_models_with_CVODE?=
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
Message-ID: <loom.20090604T194945-627@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:
> > Do I really have to replicate the entire struct definition?
> 
> No, you just have to write "enough" for Cython to know what C code to 
> output. Knowing what that is probably takes some C experience though; 
> I'll try to help.
> 
> First, the #if-s cannot be expressed in Cython, but Cython is going in 
> the direction of the type of external typedefs not to matter. I think 
> you can safely do
> 
> cdef extern from ...:
>      ctypedef double realtype
> 
> It should work also when Sundials is compiled with float or long double.
> 
> Then N_Vector etc. can probably be expressed like this:
> 
> cdef extern from ....:
>      cdef struct _generic_N_Vector:
>          void* content
>          # not necesarry to declare ops unless you will be using
>          # the contents of ops from Cython
> 
>      ctypedef _generic_N_Vector* N_vector
> 
>      N_Vector N_VNew_Serial(long int vec_length)
> 
> Then you will be able to do
> 
> cdef NVector v = N_VNew_Serial(10)
> cdef realtype* v_data = <realtype*>v.content
> v_data[0] = 34

Thank you so much! This seems to work well [1] after some adjustments, and I've 
even been able to print and modify the contents of an N_Vector.

* I found I had to add libraries=['sundials_cvode', 'sundials_nvecserial'] to 
get at the N_Vector functions. (Otherwise, the module compiled without error 
but failed on import with "ImportError: ./hello.so: undefined symbol: 
N_VNew_Serial".)
* I needed to cast v.content to <N_VectorContent_Serial>, not to <realtype*>. 
N_VectorContent_Serial is a struct with a .data field which apparently is what 
I'm after. [2]
* When trying to ctypedef N_VectorContent_Serial, I didn't realize it resided 
in another header file. Until I fetched it with 'cdef extern from "nvector/
nvector_serial.h"', I got these warnings (which I won't ignore again 8-):
hello.c:418: warning: implicit declaration of function 'N_VNew_Serial'
hello.c:418: warning: assignment makes pointer from integer without a cast
hello.c:436: warning: implicit declaration of function 'N_VPrint_Serial'
hello.c:476: warning: implicit declaration of function 'N_VDestroy_Serial'


Thanks a lot for your help so far!

Best regards,
Jon Olav



[1]
== compilation, running, output ==
bash-3.2$ rm -rf build hello.so hello.c && python setup.py build_ext --inplace 
&& python -c "import hello"
running build_ext
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/sundials-2.3.0/include -I/site/VERSIONS/
compython-2.5/Linux/include/python2.5 -c hello.c -o build/temp.linux-x86_64-2.5/
hello.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/hello.o -L/site/VERSIONS/
sundials-2.3.0/lib -L/site/VERSIONS/compython-2.5/Linux/lib -lsundials_cvode -
lsundials_nvecserial -lpython2.5 -o hello.so
1.1297001e-312
1.1297001e-312
2.3547898e-310

          0
          1
          2

== hello.pyx ==
"""Trying to modify an NVector"""

cdef extern from "sundials/sundials_types.h":
    ctypedef double realtype

cdef extern from "sundials/sundials_nvector.h":
    cdef struct _generic_N_Vector:
        void* content
    ctypedef _generic_N_Vector* N_Vector
    N_Vector N_VNew_Serial(long int vec_length)
    void N_VDestroy_Serial(N_Vector v)
    void N_VPrint_Serial(N_Vector v)

cdef extern from "nvector/nvector_serial.h":
    cdef struct _N_VectorContent_Serial:
        long int length
        realtype* data
    ctypedef _N_VectorContent_Serial* N_VectorContent_Serial


cdef N_Vector v = N_VNew_Serial(3)
cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
N_VPrint_Serial(v) # prints uninitialized floats
for i in range(3):
    v_data[i] = i
N_VPrint_Serial(v) # prints 0, 1, 2
N_VDestroy_Serial(v)

== setup.py ==
"""Run with e.g. python setup.py build_ext --inplace
Or, to remove intermediate files from previous builds:
rm -rf build hello.so hello.c && python setup.py build_ext --inplace && python -
c "import hello"
"""

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("hello",
            ["hello.pyx"],
            include_dirs=['/site/VERSIONS/sundials-2.3.0/include'],
            library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
            libraries=['sundials_cvode', 'sundials_nvecserial']),
    ]
)


[2]
This is what showed the problem in treating v.content as an array of double, 
"cdef realtype* v_data = <realtype*>v.content". Before I modified v_data[0], 
N_VPrint_Serial() would correctly print the (uninitialized) floats in the 
vector. After modifying v_data[0], it printed uninitialized floats ad infinitum.

Looking at the source code of N_VPrint_Serial() [3], I saw that it used the 
macros NV_LENGTH_S and NV_DATA_S, which accessed fields of the struct type 
N_VectorContent_Serial.

== excerpt from nvector_serial.h ==
#define NV_CONTENT_S(v)  ( (N_VectorContent_Serial)(v->content) )
#define NV_LENGTH_S(v)   ( NV_CONTENT_S(v)->length )
void N_VPrint_Serial(N_Vector x)
{
  long int i, N;
  realtype *xd;
  N  = NV_LENGTH_S(x);
  xd = NV_DATA_S(x);
  for (i=0; i < N; i++) {
    printf("%11.8g\n", *xd++);
  }
  printf("\n");
}


[3]
http://www.google.com/codesearch/p?hl=en#jVvm31OatEM/sundials/nvec_ser/
nvector_serial.c&q=nvector%2520sundials%2520cvode



From dalcinl at gmail.com  Thu Jun  4 23:49:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 18:49:25 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>

On Thu, Jun 4, 2009 at 1:29 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin wrote:
>> On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel wrote:
>>>
>>> I think we should file a bug report against Py3.1 and even bring this to
>>> the attention of python-dev before it's too late.
>>>
>>
>> Stefan, I saw your post in python-dev and corresponding issue in the
>> tracker, but I got confused. Is the solution know and the issue going
>> to be fixed?
>
> I didn't find the time to look into this any deeper. I'm not sure it will
> get fixed, because a) the bug itself is a lot older than rc1, and b) the
> comment sounded more like "the fix might be similar to what we did here".
>
> I think we should still open a new bug. Given that you experienced the
> problem, could you do that?
>

Done, http://bugs.python.org/issue6195, with patch, I've added Stefan
to the noisy list.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonovik at gmail.com  Fri Jun  5 00:34:12 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Thu, 4 Jun 2009 22:34:12 +0000 (UTC)
Subject: [Cython]
	=?utf-8?q?Sundials_with_Cython=3F_Integrating/solving_or?=
	=?utf-8?q?dinary=09differential_equation_models_with_CVODE?=
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
	<loom.20090604T194945-627@post.gmane.org>
Message-ID: <loom.20090604T222116-214@post.gmane.org>

Jon Olav Vik <jonovik at ...> writes:
> Thank you so much! This seems to work well [1] after some adjustments, and 
> I've even been able to print and modify the contents of an N_Vector.

Now I've had a go at another subtask:

> * Handle the NVector from Python: exchange data with a Numpy array,
> and pass the NVector to a C or Cython function.

The functions arr2nv() and nv2arr() work [1], at least within Cython. It can 
probably be done faster, but I don't expect frequent copying to/from N_Vectors.

I did try "cdef N_Vector arr2nv(np.ndarray[np.int_t] x):", based on
http://wiki.cython.org/tutorials/numpy#Efficientindexing
but I couldn't "cimport numpy as np" as required to get at np.int_t. Any ideas 
why I'd get "error: numpy/arrayobject.h: No such file or directory"? [2]

Regards,
Jon Olav


[1]
== compilation, running, output ==
bash-3.2$ rm -rf build hello.so hello.c && python setup.py build_ext --inplace 
&& python -c "import hello"
running build_ext
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/sundials-2.3.0/include -I/site/VERSIONS/
compython-2.5/Linux/include/python2.5 -c hello.c -o build/temp.linux-x86_64-2.5/
hello.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/hello.o -L/site/VERSIONS/
sundials-2.3.0/lib -L/site/VERSIONS/compython-2.5/Linux/lib -lsundials_cvode -
lsundials_nvecserial -lpython2.5 -o hello.so
1.1297001e-312
4.4472177e-316
1.5810101e-322

          0
          1
          2

[ 0.  1.  2.]
          0
          1
          2

== hello.pyx ==
"""Trying to modify an NVector"""

cdef extern from "sundials/sundials_types.h":
    ctypedef double realtype

cdef extern from "sundials/sundials_nvector.h":
    cdef struct _generic_N_Vector:
        void* content
    ctypedef _generic_N_Vector* N_Vector
    N_Vector N_VNew_Serial(long int vec_length)
    void N_VDestroy_Serial(N_Vector v)
    void N_VPrint_Serial(N_Vector v)

cdef extern from "nvector/nvector_serial.h":
    cdef struct _N_VectorContent_Serial:
        long int length
        realtype* data
    ctypedef _N_VectorContent_Serial* N_VectorContent_Serial

import numpy as np
cdef nv2arr(N_Vector v):
    cdef long int n = (<N_VectorContent_Serial>v.content).length
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    x = np.empty(n)
    for i in range(n):
        x[i] = v_data[i]
    return x

# cimport numpy as np
# cdef N_Vector arr2nv(np.ndarray[np.int_t] x):
cdef N_Vector arr2nv(x):
    cdef long int n = len(x)
    cdef N_Vector v = N_VNew_Serial(n)
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    for i in range(n):
        v_data[i] = x[i]
    return v

cdef N_Vector v = N_VNew_Serial(3)
cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
N_VPrint_Serial(v) # prints uninitialized floats
for i in range(3):
    v_data[i] = i
N_VPrint_Serial(v)
x = nv2arr(v)
print x
N_VDestroy_Serial(v)
v = arr2nv(x)
N_VPrint_Serial(v)
N_VDestroy_Serial(v)

[2]
"cimport numpy" fails inside my hello.pyx too; here follows a minimal example.

bash-3.2$ cat > cimp.pyx
cimport numpy
bash-3.2$ python -c "import pyximport; pyximport.install(build_dir='.'); import 
cimp"
./temp.linux-x86_64-2.5/pyrex/cimp.c:136:31: error: numpy/arrayobject.h: No 
such file or directory
./temp.linux-x86_64-2.5/pyrex/cimp.c:595: error: expected '=', ',', ';', 'asm' 
or '__attribute__' before '__pyx_t_5numpy_int8_t'
...
./temp.linux-x86_64-2.5/pyrex/cimp.c:639: error: expected '=', ',', ';', 'asm' 
or '__attribute__' before '__pyx_t_5numpy_complex_t'
./temp.linux-x86_64-2.5/pyrex/cimp.c:650: error: expected ')' before '*' token
./temp.linux-x86_64-2.5/pyrex/cimp.c: In function 
'__pyx_pf_5numpy_7ndarray___getbuffer__':
./temp.linux-x86_64-2.5/pyrex/cimp.c:737: error: 'PyArray_Descr' undeclared 
(first use in this function)
./temp.linux-x86_64-2.5/pyrex/cimp.c:737: error: (Each undeclared identifier is 
reported only once
./temp.linux-x86_64-2.5/pyrex/cimp.c:737: error: for each function it appears 
in.)
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usit/titan/u1/jonvi/usr//lib/python2.5/site-packages/pyximport/
pyximport.py", line 288, in load_module
    self.pyxbuild_dir)
  File "/usit/titan/u1/jonvi/usr//lib/python2.5/site-packages/pyximport/
pyximport.py", line 154, in load_module
    raise ImportError("Building module failed: %s" % e)
ImportError: Building module failed: CompileError(DistutilsExecError("command 
'gcc' failed with exit status 1",),)



From dalcinl at gmail.com  Fri Jun  5 02:31:03 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 21:31:03 -0300
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <loom.20090604T222116-214@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
	<loom.20090604T194945-627@post.gmane.org>
	<loom.20090604T222116-214@post.gmane.org>
Message-ID: <e7ba66e40906041731v2d15672dy23fb82d492616b38@mail.gmail.com>

On Thu, Jun 4, 2009 at 7:34 PM, Jon Olav Vik <jonovik at gmail.com> wrote:
> Jon Olav Vik <jonovik at ...> writes:
>
> I did try "cdef N_Vector arr2nv(np.ndarray[np.int_t] x):", based on
> http://wiki.cython.org/tutorials/numpy#Efficientindexing
> but I couldn't "cimport numpy as np" as required to get at np.int_t. Any ideas
> why I'd get "error: numpy/arrayobject.h: No such file or directory"? [2]
>

Because the C compiler requires the include path to numpy headers. The
function numpy.get_include() returns it, however I do not remember if
pyximport has the fixes to let you pass such include path. Could you
try to do in your shell

$ export CFLAGS=-I`python -c 'import numpy; print numpy.get_include()'`

and next try to run your code?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonovik at gmail.com  Fri Jun  5 08:39:09 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Fri, 5 Jun 2009 06:39:09 +0000 (UTC)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
	<loom.20090604T194945-627@post.gmane.org>
	<loom.20090604T222116-214@post.gmane.org>
	<e7ba66e40906041731v2d15672dy23fb82d492616b38@mail.gmail.com>
Message-ID: <loom.20090605T062635-632@post.gmane.org>

Lisandro Dalcin <dalcinl at ...> writes:

> On Thu, Jun 4, 2009 at 7:34 PM, Jon Olav Vik <jonovik at ...> wrote:
> > Jon Olav Vik <jonovik at ...> writes:
> >
> > I did try "cdef N_Vector arr2nv(np.ndarray[np.int_t] x):", based on
> > http://wiki.cython.org/tutorials/numpy#Efficientindexing
> > but I couldn't "cimport numpy as np" as required to get at np.int_t. Any 
ideas
> > why I'd get "error: numpy/arrayobject.h: No such file or directory"? [2]
> >
> 
> Because the C compiler requires the include path to numpy headers. The
> function numpy.get_include() returns it, however I do not remember if
> pyximport has the fixes to let you pass such include path. Could you
> try to do in your shell
> 
> $ export CFLAGS=-I`python -c 'import numpy; print numpy.get_include()'`
> 
> and next try to run your code?

The cimport statement now passes, though I'm not sure what to make of the "type-
punned" warnings [1]. "Defined but not used" doesn't sound too scary, though.

I remember now that we used numpy.get_include() in Dag Sverre's workshop. 
Adding "import numpy as np" and "include_dirs=['/site/VERSIONS/sundials-2.3.0/
include', np.get_include ()]," worked for my setup.py too (the "hello.pyx" 
example mentioned previously).

Thank you very much for your help. Later I'll have a go at actually running 
CVODE.

Best regards,
Jon Olav Vik


[1]
bash-3.2$ cat cimp.pyx
cimport numpy
print "cimported numpy"
bash-3.2$ python -c "import pyximport; pyximport.install(build_dir='.'); import 
cimp"
./temp.linux-x86_64-2.5/pyrex/cimp.c: In function '__Pyx_PyObject_IsTrue':
./temp.linux-x86_64-2.5/pyrex/cimp.c:3131: warning: dereferencing type-punned 
pointer will break strict-aliasing rules
./temp.linux-x86_64-2.5/pyrex/cimp.c:3132: warning: dereferencing type-punned 
pointer will break strict-aliasing rules
./temp.linux-x86_64-2.5/pyrex/cimp.c: At top level:
/site/VERSIONS/compython-2.5/Linux/lib/python2.5/site-packages/numpy/core/
include/numpy/__multiarray_api.h:959: warning: '_import_array' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:687: warning: '__pyx_k_3' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:688: warning: '__pyx_k_4' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:708: warning: '__pyx_k_24' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:714: warning: '__pyx_k_26' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:715: warning: '__pyx_k_27' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:729: warning: 
'__pyx_pf_5numpy_7ndarray___getbuffer__' defined but not used
./temp.linux-x86_64-2.5/pyrex/cimp.c:1487: warning: 
'__pyx_pf_5numpy_7ndarray___releasebuffer__' defined but not used
cimported numpy



From jonovik at gmail.com  Fri Jun  5 15:41:41 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Fri, 5 Jun 2009 13:41:41 +0000 (UTC)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
Message-ID: <loom.20090605T132731-705@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:
> The challenge here though is about function pointers.
> 
> Basically you need to define
> cdef extern from ...:
>    ctypedef double realtype
>    ctypedef struct N_Vector:
>       ...more or less copy definition from nvector header files...
>    ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void 
> *user_data)
>    int CVodeInit(void* mem, CVRhsFn f, realtype t0, N_Vector y0)
> 
> Note that here CVRhsFn is a /type/, it is a pointer to a function, and 
> CVodeInit is a function which takes such a pointer (a "callback"). So 
> you'd do
> 
> cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
>    ...
> 
> CVodeInit(..., your_rhs, t0, y0)
> 
> and now Sundials will know that your_rhs should be called.

I'm pleased to report that Sundials is finally doing my bidding from within 
Cython [1]! As I become more familiar with the cdef extern syntax, it's fairly 
easy to import new functions from Sundials/CVODE. I'll come back to efficient 
indexing [2] later.

If I may make a wish, I would like Cython to find more declarations 
automatically. For instance, having

cdef extern * from "cvode/cvode.h"

or, if that is impractical,

cdef extern from "cvode/cvode.h":
    CVodeCreate, CVRhsFn, CVodeMalloc, CVodeFree, CVode, CV_ADAMS, CV_BDF

instead of:

cdef extern from "cvode/cvode.h":
    void* CVodeCreate(int lmm, int iter)
    ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void *f_data)
    int CVodeMalloc(void *cvode_mem, CVRhsFn f, realtype t0, N_Vector y0,
        int itol, realtype reltol, void *abstol)
    void CVodeFree(void **cvode_mem)
    int CVode(void *cvode_mem, realtype tout, N_Vector yout, realtype *tret, 
        int itask)
    # constants
    ## /* lmm */
    DEF CV_ADAMS = 1
    DEF CV_BDF   = 2
    ...

I haven't benchmarked anything yet, but the relative ease of adapting the C 
tutorial for CVODE [3] is uplifting. As a last question, what is the cleanest 
way to transfer arrays of double between realtype* and Numpy vectors (one-
dimensional arrays)?

Thank you very much to all who helped me!

Best regards,
Jon Olav


[1]
== setup.py ==
"""Run with e.g. python setup.py build_ext --inplace
Or, to remove intermediate files from previous builds:
rm -rf build hello.so hello.c && python setup.py build_ext --inplace && python -
c "import hello"
"""
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy as np
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("hello",
            ["hello.pyx"],
            include_dirs=['/site/VERSIONS/sundials-2.3.0/include', 
np.get_include ()],
            library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
            libraries=['sundials_cvode', 'sundials_nvecserial']),
    ]
)

== hello.pyx ==
"""
Cythonizing simple usage of CVODE

Build and test with
rm -rf build hello.so hello.c && python setup.py build_ext --inplace && python -
c "import hello"
"""

cdef extern from "sundials/sundials_types.h":
    ctypedef double realtype

cdef extern from "sundials/sundials_nvector.h":
    cdef struct _generic_N_Vector:
        void* content
    ctypedef _generic_N_Vector* N_Vector
    N_Vector N_VNew_Serial(long int vec_length)
    void N_VDestroy_Serial(N_Vector v)
    void N_VPrint_Serial(N_Vector v)

cdef extern from "nvector/nvector_serial.h":
    cdef struct _N_VectorContent_Serial:
        long int length
        realtype* data
    ctypedef _N_VectorContent_Serial* N_VectorContent_Serial

cdef extern from "cvode/cvode.h":
    void* CVodeCreate(int lmm, int iter)
    ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void *f_data)
    int CVodeMalloc(void *cvode_mem, CVRhsFn f, realtype t0, N_Vector y0,
        int itol, realtype reltol, void *abstol)
    void CVodeFree(void **cvode_mem)
    int CVode(void *cvode_mem, realtype tout, N_Vector yout, realtype *tret, 
        int itask)
    # constants
    ## /* lmm */
    DEF CV_ADAMS = 1
    DEF CV_BDF   = 2
    ## /* iter */
    DEF CV_FUNCTIONAL = 1
    DEF CV_NEWTON     = 2
    ## /* itol */
    DEF CV_SS = 1
    DEF CV_SV = 2
    DEF CV_WF = 3
    ## /* itask */
    DEF CV_NORMAL         = 1
    DEF CV_ONE_STEP       = 2
    DEF CV_NORMAL_TSTOP   = 3
    DEF CV_ONE_STEP_TSTOP = 4
    ## /* CVODE return flags */
    DEF CV_SUCCESS = 0
    DEF CV_TSTOP_RETURN = 1
    DEF CV_ROOT_RETURN = 2

# how to find this stuff: 
# grep "CVDense(" /xanadu/site/common/VERSIONS/sundials-2.3.0/include/*/*
# /xanadu/site/common/VERSIONS/sundials-2.3.0/include/
# ... cvode/cvode_dense.h:int CVDense(void *cvode_mem, long int N);
cdef extern from "cvode/cvode_dense.h":
    int CVDense(void *cvode_mem, long int N)


import numpy as np
cdef nv2arr(N_Vector v):
    """Create new numpy array from N_Vector"""
    cdef long int n = (<N_VectorContent_Serial>v.content).length
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    x = np.empty(n)
    for i in range(n):
        x[i] = v_data[i]
    return x

cimport numpy as np
# cdef N_Vector arr2nv(np.ndarray[np.int_t] x):
cdef N_Vector arr2nv(x):
    """Create new N_Vector from numpy array"""
    cdef long int n = len(x)
    cdef N_Vector v = N_VNew_Serial(n)
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    for i in range(n):
        v_data[i] = x[i]
    return v

# following https://computation.llnl.gov/casc/sundials/documentation/
# ... cv_guide/node5.html#SECTION00540000000000000000

# set problem dimensions
N = 2

# set vector of initial values
cdef N_Vector y0 = arr2nv(np.arange(N)) # N_VMake_Serial(N, ydata)

# create CVODE object
cdef void* cvode_mem = CVodeCreate(CV_BDF, CV_NEWTON)

## ODE right-hand side
cdef int vanderpol (realtype t, N_Vector y_nv, N_Vector ydot_nv, void *f_data):
    """van der Pol equation adapted from example_ode.py"""
    cdef realtype* y = (<N_VectorContent_Serial>y_nv.content).data
    cdef realtype* ydot = (<N_VectorContent_Serial>ydot_nv.content).data
    ydot[0] = y[1]
    ydot[1] = (1 - y[0] * y[0]) * y[1] - y[0] # eps[0] * ...
    return 0

### testing RHS
print "Current state:"
N_VPrint_Serial(y0) # 0, 1
cdef N_Vector ydot = arr2nv(np.zeros(N)) # N_VMake_Serial(N, ydata)
vanderpol(0.0, y0, ydot, <void*>None)
print "Rates of change:"
N_VPrint_Serial(ydot) # 1, 1

# initialize CVODE solver
# specify integration tolerances
# (note: Sundials 2.3 uses CVodeMalloc instead of CVodeInit + CVodeSStolerances)
cdef realtype abstol = 1e-4
flag = CVodeMalloc(cvode_mem, vanderpol, 0.0, y0, CV_SS, 1e-4, &abstol)
# if flag == CV_SUCCESS: print "CVodeMalloc returned CV_SUCCESS"

# set optional inputs

# attach linear solver module
flag = CVDense(cvode_mem, N)

# set linear solver optional inputs

# specify rootfinding problem

# advance solution in time
tmax = 150
y = np.zeros((tmax, N))
t = np.zeros((tmax,))
cdef realtype tout = 2.0
cdef realtype tret
cdef int i, j
cdef realtype* v_data = (<N_VectorContent_Serial>y0.content).data
for i in range(tmax):
    flag = CVode(cvode_mem, tout, y0, &tret, CV_ONE_STEP)
    if flag != CV_SUCCESS:
        break
    t[i] = tret
    for j in range(N):
        y[i,j] = v_data[j]

# get optional outputs

# deallocate memory for solution vector
N_VDestroy_Serial(ydot)
N_VDestroy_Serial(y0)

# free solver memory
CVodeFree(&cvode_mem)

# display results
import matplotlib as mpl
mpl.use("svg")
import pylab
pylab.plot(t, y)
pylab.savefig("cvode.svgz")

[2] http://docs.cython.org/docs/numpy_tutorial.html#efficient-indexing

[3] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/
node5.html#SECTION00540000000000000000


From barthelemy at crans.org  Fri Jun  5 17:46:32 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Fri, 5 Jun 2009 17:46:32 +0200
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
Message-ID: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>

Hello all,

I have a project involving cython files with the following file hierarchy.

/setup.py
/src/homogeneousmatrix.pyx
/src/homogeneousmatrix.pxd
/src/__init__.py
/src/...

However, I have some trouble writing a working  setup.py. With the
version below, generating src/homogeneousmatrix.c fails, apparently
because it does not use the corresponding pxd file, where some
necessary ctypedef are made.


I can compile everything with this setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    name='arboris',
    packages=['arboris',
 	          'arboris.robots'],
    package_dir = {'arboris':'src'},
    cmdclass = {'build_ext': build_ext},
    ext_package='arboris',
    ext_modules = [Extension("homogeneousmatrix",
                             ["src/homogeneousmatrix.pyx"])]
)


However, with the following one,  generating src/homogeneousmatrix.c
fails, apparently because cython does not find/use the corresponding
pxd file

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    name='arboris',
    packages=['arboris',
                  'arboris.robots'],
    package_dir = {'arboris':'src'},
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("arboris.homogeneousmatrix",
                             ["src/homogeneousmatrix.pyx"])]
)

I was wondering if this was an expected behaviour or a bug.

Have a nice week-end all.

From efiring at hawaii.edu  Fri Jun  5 21:38:46 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 05 Jun 2009 09:38:46 -1000
Subject: [Cython] support for numpy bool type
Message-ID: <4A297446.6030309@hawaii.edu>

In writing a cython extension to work with numpy masked arrays, I needed 
to work with the mask, which is dtype('bool').  Therefore I was 
expecting to be able to use np.bool_t, in analogy to np.float_t. 
Instead I had to use np.int8_t, and cast the input to np.int8 in a 
python wrapper.  Can bool_t be added to numpy.pxd?

Thanks.

Eric

From mattcbro at earthlink.net  Fri Jun  5 21:42:39 2009
From: mattcbro at earthlink.net (Matthew Bromberg)
Date: Fri, 05 Jun 2009 15:42:39 -0400
Subject: [Cython] Cython Memory Management
Message-ID: <4A29752F.7070901@earthlink.net>

I am not entirely clear how memory management works in cython for 
extension classes and their attributes/members.

I have a class in a .pxd file defined like,

cdef class rmat :
     """  Real matrix class binds to both vsipl and numpy ndarray.
     The ndarray object is returned via getarr.  The array is bound
     back to vsipl via admit """
     cdef vsip_mview_f *vm
     # this should be allowed but currently isn't
     # cdef np.ndarray[np.float32_t, ndim=2] arr
     # cdef vsip_block_f *block
     cdef object arr
     cpdef rmat parent
  ....


So apparently I can create an rmat object in python.  I have 
__cinit__()  and  __dealloc__() methods defined.

Question 1,  are references to an rmat object reference counted and 
garbage collected?
So far it seems like they are.

Question 2)  Are extension class members set to  other  extension 
classes or python objects,  such as  arr and parent  reference counted?
ie suppose I create a real matrix such as,
 >> m = rmat(2,3)
 >> m.arr = array1
 >> m.parent =  m0

The assignments to the class members are actually done inside a .pyx 
file so they are done via cython.  Do such assignments increment the 
refcounts and are things properly garbage collected when the reference 
counts go to zero?

Again it seems like this is true,  but it's not entirely obvious.  I 
might expect  member arr to be properly refcounted since it's explicitly 
declared as a python object,  but the parent member is declared as an 
extension type object.  In theory it could just treat it as a C pointer. 
What if I had declared it via a cdef instead of cpdef,  does that make 
any difference?

  I have some anecdotal evidence that it is properly garbage collected,  
but I don't know the rules or restrictions on this since the 
documentation only talks briefly about these issues in a couple of places.

Thanks in advance for your help,



From robertwb at math.washington.edu  Fri Jun  5 22:21:37 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Jun 2009 13:21:37 -0700
Subject: [Cython] Cython Memory Management
In-Reply-To: <4A29752F.7070901@earthlink.net>
References: <4A29752F.7070901@earthlink.net>
Message-ID: <644608A4-0148-4DC7-B861-1909C82A9648@math.washington.edu>

On Jun 5, 2009, at 12:42 PM, Matthew Bromberg wrote:

> I am not entirely clear how memory management works in cython for
> extension classes and their attributes/members.
>
> I have a class in a .pxd file defined like,
>
> cdef class rmat :
>      """  Real matrix class binds to both vsipl and numpy ndarray.
>      The ndarray object is returned via getarr.  The array is bound
>      back to vsipl via admit """
>      cdef vsip_mview_f *vm
>      # this should be allowed but currently isn't
>      # cdef np.ndarray[np.float32_t, ndim=2] arr
>      # cdef vsip_block_f *block
>      cdef object arr
>      cpdef rmat parent
>   ....

There's no such thing as a cpdef attribute, just cdef, cpdef only  
makes sense for functions.

> Question 1,  are references to an rmat object reference counted and
> garbage collected?
> So far it seems like they are.
>
> Question 2)  Are extension class members set to  other  extension
> classes or python objects,  such as  arr and parent  reference  
> counted?

Yes, anything that is an object (including subclasses thereof) is  
reference counted. You should never have to worry about reference  
counts unless you're casting objects to or from pointer types.

- Robert


From robertwb at math.washington.edu  Fri Jun  5 22:28:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Jun 2009 13:28:55 -0700
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
Message-ID: <9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>

On Jun 5, 2009, at 8:46 AM, S?bastien Barth?lemy wrote:

> Hello all,
>
> I have a project involving cython files with the following file  
> hierarchy.
>
> /setup.py
> /src/homogeneousmatrix.pyx
> /src/homogeneousmatrix.pxd
> /src/__init__.py
> /src/...
>
> However, I have some trouble writing a working  setup.py. With the
> version below, generating src/homogeneousmatrix.c fails, apparently
> because it does not use the corresponding pxd file, where some
> necessary ctypedef are made.
>
>
> I can compile everything with this setup.py:
>
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
>     name='arboris',
>     packages=['arboris',
>  	          'arboris.robots'],
>     package_dir = {'arboris':'src'},
>     cmdclass = {'build_ext': build_ext},
>     ext_package='arboris',
>     ext_modules = [Extension("homogeneousmatrix",
>                              ["src/homogeneousmatrix.pyx"])]
> )
>
>
> However, with the following one,  generating src/homogeneousmatrix.c
> fails, apparently because cython does not find/use the corresponding
> pxd file
>
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
>     name='arboris',
>     packages=['arboris',
>                   'arboris.robots'],
>     package_dir = {'arboris':'src'},
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("arboris.homogeneousmatrix",
>                              ["src/homogeneousmatrix.pyx"])]
> )
>
> I was wondering if this was an expected behaviour or a bug.

If you don't want to specify ext_package, you should probably lay out  
your sources as you would for normal .py files, e.g. put everything  
in the arboris package in an "arboris" directory rather than a "src"  
directory. Perhaps Cython should be inspecting package_dir?

- Robert


From jonovik at gmail.com  Sat Jun  6 00:57:20 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Sat, 6 Jun 2009 00:57:20 +0200
Subject: [Cython] Running doctests with Cython?
Message-ID: <cd166c030906051557m5387d0efwf191427db22e96a@mail.gmail.com>

I'm cythonizing an existing module with lots of doctests, which would be useful
in catching any errors or changes in behaviour that might result. However, I
have trouble running doctests in a cython-compiled module. The compiled module
offers the usual __doc__ strings (simple example follows below), yet python -m
fails with "No code object available". Is there a trick?

Thanks in advance for any help.

Best regards,
Jon Olav Vik


-bash-3.2$ cat cydoctest.pyx
def pyfun(x):
    """
    >>> pyfun(3)
    6
    """
    return x * 2

-bash-3.2$ rm -rf *.pyc build *.so *.c && python setup.py build_ext --inplace
running build_ext
cythoning cydoctest.pyx to cydoctest.c
building 'cydoctest' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/compython-2.5/Linux/include/python2.5 -c
cydoctest.c -o build/temp.linux-x86_64-2.5/cydoctest.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/cydoctest.o -L/site/VERSIONS/
compython-2.5/Linux/lib -lpython2.5 -o cydoctest.so

-bash-3.2$ python -c "import cydoctest; print cydoctest.pyfun.__doc__; print
cydoctest"
    >>> pyfun(3)
    6
<module 'cydoctest' from 'cydoctest.so'>

-bash-3.2$ python cydoctest.pyx -v
Trying:
    pyfun(3)
Expecting:
    6
ok
1 items had no tests:
    __main__
1 items passed all tests:
   1 tests in __main__.pyfun
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

-bash-3.2$ python -m cydoctest
Traceback (most recent call last):
  File "/site/VERSIONS/compython-2.5/Linux/lib/python2.5/runpy.py", line 90, in
run_module
    raise ImportError("No code object available for " + mod_name)
ImportError: No code object available for cydoctest

From glenn at tarbox.org  Sat Jun  6 06:00:45 2009
From: glenn at tarbox.org (Glenn Tarbox, PhD)
Date: Fri, 5 Jun 2009 21:00:45 -0700
Subject: [Cython] Trivial numpy test in sage notebook fails...
Message-ID: <f049b35c0906052100l41f5a77do73e755859b24f0dc@mail.gmail.com>

Here's the code which is the first cell of a sage notebook (sage 4.0)

%cython
import numpy as np
cimport numpy as np
DTYPE = np.int
ctypedef np.int_t DTYPE_t

Here's the error:

Traceback (most recent call last):    DTYPE = np.int
  File "/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py",
line 408, in cython_import_all
    create_local_c_file=create_local_c_file)
  File "/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py",
line 385, in cython_import
    create_local_c_file=create_local_c_file)
  File "/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/misc/cython.py",
line 367, in cython
    raise RuntimeError, "Error converting %s to C:\n%s\n%s"%(filename, log, err)
RuntimeError: Error converting
/home/tarbox/.sage/sage_notebook/worksheets/admin/89/code/sage23.spyx
to C:


Error converting Pyrex file to C:
------------------------------------------------------------
...

include "cdefs.pxi"
import numpy as np
cimport numpy as np
DTYPE = np.int
ctypedef np.int_t DTYPE_t        ^
------------------------------------------------------------

/home/tarbox/.sage/temp/hq2/12471/spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx_0.pyx:9:9:
'int_t' is not a type identifier


I tried with and without the "import numpy" directive thinking it may be
redundant...

This example comes from the "cython for numpy users" tutorial on the wiki
where it mentions that all numpy types have a corresponding _t type

There is indeed a numpy.int type... so I was wondering if the new work with
python buffers changed how things work.

-glenn

-- 
Glenn H. Tarbox, PhD ||  206-274-6919
http://www.tarbox.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090605/ccce6c30/attachment.htm 

From barthelemy at crans.org  Sat Jun  6 07:49:35 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Sat, 6 Jun 2009 07:49:35 +0200
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
Message-ID: <78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>

2009/6/5 Robert Bradshaw <robertwb at math.washington.edu>:
> On Jun 5, 2009, at 8:46 AM, S?bastien Barth?lemy wrote:
>> I have a project involving cython files with the following file
>> hierarchy.
>>
>> /setup.py
>> /src/homogeneousmatrix.pyx
>> /src/homogeneousmatrix.pxd
>> /src/__init__.py
>> /src/...

[...]

> If you don't want to specify ext_package, you should probably lay out
> your sources as you would for normal .py files, e.g. put everything
> in the arboris package in an "arboris" directory rather than a "src"
> directory.

I think my sources are currently laid out as for normal python source.
(while I agree renaming src into arboris would be nice.)


By the way, if I use

dir = {'arboris':'src'},
   ext_modules = [Extension("arboris.homogeneousmatrix",
                            ["src/homogeneousmatrix.pyx"])]

(that is without ext_package), cython -- as called by distutils --
does not use/find the pxd. But if I run cython directly from the
command line (cython src/homogeneousmatrix.pyx) it does. This
discrepancy sounds like a bug to me.


> Perhaps Cython should be inspecting package_dir?

I also think so, but I'm certainly not an expert. I also don't now why
distutils uses two different properties for packages (packages and
ext_package).

-- 
Sebastien

From dagss at student.matnat.uio.no  Sat Jun  6 09:05:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 06 Jun 2009 09:05:00 +0200
Subject: [Cython] Running doctests with Cython?
Message-ID: <3327123929.3069139@smtp.netcom.no>

This was just answered two days ago in another thread.

Dag Sverre Seljebotn
-----Original Message-----
From: Jon Olav Vik <jonovik at gmail.com>
Date: Saturday, Jun 6, 2009 12:58 am
Subject: [Cython] Running doctests with Cython?
To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net

I'm cythonizing an existing module with lots of doctests, which would be useful in catching any errors or changes in behaviour that might result. However, I have trouble running doctests in a cython-compiled module. The compiled module offers the usual __doc__ strings (simple example follows below), yet python -m fails with "No code object available". Is there a trick?
>
>Thanks in advance for any help.
>
>Best regards,
>Jon Olav Vik
>
>
>-bash-3.2$ cat cydoctest.pyx
>def pyfun(x):
>    """
>    >>> pyfun(3)
>    6
>    """
>    return x * 2
>
>-bash-3.2$ rm -rf *.pyc build *.so *.c && python setup.py build_ext --inplace running build_ext
>cythoning cydoctest.pyx to cydoctest.c
>building 'cydoctest' extension
>creating build
>creating build/temp.linux-x86_64-2.5
>gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict- prototypes -fPIC -I/site/VERSIONS/compython-2.5/Linux/include/python2.5 -c cydoctest.c -o build/temp.linux-x86_64-2.5/cydoctest.o
>gcc -pthread -shared build/temp.linux-x86_64-2.5/cydoctest.o -L/site/VERSIONS/ compython-2.5/Linux/lib -lpython2.5 -o cydoctest.so
>
>-bash-3.2$ python -c "import cydoctest; print cydoctest.pyfun.__doc__; print cydoctest"
>    >>> pyfun(3)
>    6
><module 'cydoctest' from 'cydoctest.so'>
>
>-bash-3.2$ python cydoctest.pyx -v
>Trying:
>    pyfun(3)
>Expecting:
>    6
>ok
>1 items had no tests:
>    __main__
>1 items passed all tests:
>   1 tests in __main__.pyfun
>1 tests in 2 items.
>1 passed and 0 failed.
>Test passed.
>
>-bash-3.2$ python -m cydoctest
>Traceback (most recent call last):
>  File "/site/VERSIONS/compython-2.5/Linux/lib/python2.5/runpy.py", line 90, in run_module
>    raise ImportError("No code object available for " + mod_name)
>ImportError: No code object available for cydoctest
>_______________________________________________
>Cython-dev mailing list
>Cython-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/cython-dev
>


From efiring at hawaii.edu  Sat Jun  6 09:08:46 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 05 Jun 2009 21:08:46 -1000
Subject: [Cython] support for numpy bool type
In-Reply-To: <4A297446.6030309@hawaii.edu>
References: <4A297446.6030309@hawaii.edu>
Message-ID: <4A2A15FE.8020201@hawaii.edu>

Eric Firing wrote:
> In writing a cython extension to work with numpy masked arrays, I needed 
> to work with the mask, which is dtype('bool').  Therefore I was 
> expecting to be able to use np.bool_t, in analogy to np.float_t. 
> Instead I had to use np.int8_t, and cast the input to np.int8 in a 
> python wrapper.  Can bool_t be added to numpy.pxd?
> 
> Thanks.
> 
> Eric


Attached is a diff that I think takes care of adding bool support.

Eric



-------------- next part --------------
A non-text attachment was scrubbed...
Name: cython_numpy_bool.diff
Type: text/x-patch
Size: 1722 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090605/5428c85b/attachment.bin 

From dagss at student.matnat.uio.no  Sat Jun  6 09:09:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 06 Jun 2009 09:09:00 +0200
Subject: [Cython] Trivial numpy test in sage notebook fails...
Message-ID: <3327124144.3090608@smtp.netcom.no>

afaik this is a problem with the Sage notebook. Try to Cythonize from the command line, and if that works, take it to the Sage list.

Dag Sverre Seljebotn
-----Original Message-----
From: "Glenn Tarbox, PhD" <glenn at tarbox.org>
Date: Saturday, Jun 6, 2009 6:01 am
Subject: [Cython] Trivial numpy test in sage notebook fails...
To: Cython-dev at codespeak.netReply-To: cython-dev at codespeak.net

Here's the code which is the first cell of a sage notebook (sage 4.0)
>
>%cython
>import numpy as np
>cimport numpy as np
>DTYPE = np.int
>ctypedef np.int_t DTYPE_t
>
>Here's the error:
>
>Traceback (most recent call last):    DTYPE = np.int
>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py', line 408, in cython_import_all
>    create_local_c_file=create_local_c_file)
>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py', line 385, in cython_import
>    create_local_c_file=create_local_c_file)
>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/misc/cython.py', line 367, in cython
>    raise RuntimeError, 'Error converting %s to C:\n%s\n%s'%(filename, log, err)
>RuntimeError: Error converting /home/tarbox/.sage/sage_notebook/worksheets/admin/89/code/sage23.spyx to C:
>
>
>Error converting Pyrex file to C:
>------------------------------------------------------------
>...
>
>include 'cdefs.pxi'
>import numpy as np
>cimport numpy as np
>DTYPE = np.int
> ctypedef np.int_t DTYPE_t        ^
>------------------------------------------------------------
>
>/home/tarbox/.sage/temp/hq2/12471/spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx_0.pyx:9:9: 'int_t' is not a type identifier 
>I tried with and without the 'import numpy' directive thinking it may be redundant...
>
>This example comes from the 'cython for numpy users' tutorial on the wiki where it mentions that all numpy types have a corresponding _t type
>
>There is indeed a numpy.int type... so I was wondering if the new work with python buffers changed how things work.
>
>-glenn
>
>-- 
>Glenn H. Tarbox, PhD || ?206-274-6919
>http://www.tarbox.org
>


From robertwb at math.washington.edu  Sat Jun  6 09:13:09 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 6 Jun 2009 00:13:09 -0700
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
	<78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
Message-ID: <880357DD-058E-417E-A247-E48735B98118@math.washington.edu>

On Jun 5, 2009, at 10:49 PM, S?bastien Barth?lemy wrote:

> 2009/6/5 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Jun 5, 2009, at 8:46 AM, S?bastien Barth?lemy wrote:
>>> I have a project involving cython files with the following file
>>> hierarchy.
>>>
>>> /setup.py
>>> /src/homogeneousmatrix.pyx
>>> /src/homogeneousmatrix.pxd
>>> /src/__init__.py
>>> /src/...
>
> [...]
>
>> If you don't want to specify ext_package, you should probably lay out
>> your sources as you would for normal .py files, e.g. put everything
>> in the arboris package in an "arboris" directory rather than a "src"
>> directory.
>
> I think my sources are currently laid out as for normal python source.
> (while I agree renaming src into arboris would be nice.)

Normally A/B/C.py corresponds to the module A.B.C. Is there any  
reason you can't rename src to arboris?

> By the way, if I use
>
> dir = {'arboris':'src'},
>    ext_modules = [Extension("arboris.homogeneousmatrix",
>                             ["src/homogeneousmatrix.pyx"])]
>
> (that is without ext_package), cython -- as called by distutils --
> does not use/find the pxd. But if I run cython directly from the
> command line (cython src/homogeneousmatrix.pyx) it does. This
> discrepancy sounds like a bug to me.

That is because it's actually creating a module  
"src.homogeneousmatrix" and it finds "homogeneousmatrix.pxd" just  
fine. Running Cython directly doesn't read setup.py at all.

- Robert

From barthelemy at crans.org  Sat Jun  6 09:34:47 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Sat, 6 Jun 2009 09:34:47 +0200
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <880357DD-058E-417E-A247-E48735B98118@math.washington.edu>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
	<78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
	<880357DD-058E-417E-A247-E48735B98118@math.washington.edu>
Message-ID: <78f7ab620906060034y6c9e399eo149fbb3d9a36cb1f@mail.gmail.com>

2009/6/6 Robert Bradshaw <robertwb at math.washington.edu>:
> On Jun 5, 2009, at 10:49 PM, S?bastien Barth?lemy wrote:
>> I think my sources are currently laid out as for normal python source.
>> (while I agree renaming src into arboris would be nice.)
>
> Normally A/B/C.py corresponds to the module A.B.C.

ok. I was thinking of "normal distutils way" which understands package_dir

> Is there any
> reason you can't rename src to arboris?

no, I did it indeed.

> is because it's actually creating a module
> "src.homogeneousmatrix" and it finds "homogeneousmatrix.pxd" just
> fine. Running Cython directly doesn't read setup.py at all.

thank you for the explanations. I'm almost fully enlighten. The only
dark spot left is the point of the distutil ext_package option.

-- 
Sebastien

From dagss at student.matnat.uio.no  Sat Jun  6 11:20:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 06 Jun 2009 11:20:10 +0200
Subject: [Cython] support for numpy bool type
In-Reply-To: <4A2A15FE.8020201@hawaii.edu>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
Message-ID: <4A2A34CA.8000702@student.matnat.uio.no>

Eric Firing wrote:
> Eric Firing wrote:
>> In writing a cython extension to work with numpy masked arrays, I 
>> needed to work with the mask, which is dtype('bool').  Therefore I 
>> was expecting to be able to use np.bool_t, in analogy to np.float_t. 
>> Instead I had to use np.int8_t, and cast the input to np.int8 in a 
>> python wrapper.  Can bool_t be added to numpy.pxd?
>>
>> Thanks.
>>
>> Eric
The reason it is not there is that AFAIK Cython has no support for an 
8-bit boolean type. I.e. you want

print arr[3] # should print "True", not 1.

and furthermore you want

arr[3] = obj # should mean arr[3] = bool(obj)

Which brings up the questions
1) I suppose the best thing would be to add support for a "bool" which 
would be a C99 _Bool if compiling on C99, and "unsigned char" with 
appropriate restrictions otherwise. Thoughts?
2) Is it better to keep ndarray[bool_t] disabled until this happens, or 
should it be enabled (as an "unsigned char" array) and then change its 
semantics later?

>
> Attached is a diff that I think takes care of adding bool support.

Thanks, I might apply it depending on the answers I get to the above 
questions.

Dag Sverre

From dagss at student.matnat.uio.no  Sat Jun  6 19:11:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 06 Jun 2009 19:11:28 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <loom.20090605T132731-705@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090605T132731-705@post.gmane.org>
Message-ID: <4A2AA340.2030004@student.matnat.uio.no>

Jon Olav Vik wrote:
> I haven't benchmarked anything yet, but the relative ease of adapting the C 
> tutorial for CVODE [3] is uplifting. As a last question, what is the cleanest 
> way to transfer arrays of double between realtype* and Numpy vectors (one-
> dimensional arrays)?
>   
Just a note: There is a nice and fairly canonical way of doing this, but 
it takes some typing and I'm close to a deadline for an exam project. 
I'll write it down on Tuesday or so if noone beats me to it.

Dag Sverre

From robertwb at math.washington.edu  Sat Jun  6 20:18:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 6 Jun 2009 11:18:51 -0700
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <78f7ab620906060034y6c9e399eo149fbb3d9a36cb1f@mail.gmail.com>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
	<78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
	<880357DD-058E-417E-A247-E48735B98118@math.washington.edu>
	<78f7ab620906060034y6c9e399eo149fbb3d9a36cb1f@mail.gmail.com>
Message-ID: <13DE9389-6DDE-43E4-B9CB-2BA7D7AE6494@math.washington.edu>

On Jun 6, 2009, at 12:34 AM, S?bastien Barth?lemy wrote:

> 2009/6/6 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Jun 5, 2009, at 10:49 PM, S?bastien Barth?lemy wrote:
>>> I think my sources are currently laid out as for normal python  
>>> source.
>>> (while I agree renaming src into arboris would be nice.)
>>
>> Normally A/B/C.py corresponds to the module A.B.C.
>
> ok. I was thinking of "normal distutils way" which understands  
> package_dir
>
>> Is there any
>> reason you can't rename src to arboris?
>
> no, I did it indeed.
>
>> is because it's actually creating a module
>> "src.homogeneousmatrix" and it finds "homogeneousmatrix.pxd" just
>> fine. Running Cython directly doesn't read setup.py at all.
>
> thank you for the explanations.

You're welcome.

> I'm almost fully enlighten. The only
> dark spot left is the point of the distutil ext_package option.

That's a mystery to me too...

- Robert



From robertwb at math.washington.edu  Sat Jun  6 22:02:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 6 Jun 2009 13:02:11 -0700
Subject: [Cython] support for numpy bool type
In-Reply-To: <4A2A34CA.8000702@student.matnat.uio.no>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
Message-ID: <E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>

On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:

> Eric Firing wrote:
>> Eric Firing wrote:
>>> In writing a cython extension to work with numpy masked arrays, I
>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>> was expecting to be able to use np.bool_t, in analogy to np.float_t.
>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>
>>> Thanks.
>>>
>>> Eric
> The reason it is not there is that AFAIK Cython has no support for an
> 8-bit boolean type. I.e. you want
>
> print arr[3] # should print "True", not 1.
>
> and furthermore you want
>
> arr[3] = obj # should mean arr[3] = bool(obj)
>
> Which brings up the questions
> 1) I suppose the best thing would be to add support for a "bool" which
> would be a C99 _Bool if compiling on C99, and "unsigned char" with
> appropriate restrictions otherwise. Thoughts?

How would this relate to the current bint type? bint is an int  
because logical operations in C are ints, and also several operations  
in the Python/C API (and elsewhere) return true/false values as ints.

> 2) Is it better to keep ndarray[bool_t] disabled until this  
> happens, or
> should it be enabled (as an "unsigned char" array) and then change its
> semantics later?

I would be OK with allowing bool_t to be a synonym for int8_t, and  
then widening the semantics later (I don't see what would break,  
unless one was storing actual integer (rather than boolean) values).

- Robert


From efiring at hawaii.edu  Sat Jun  6 22:14:34 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Sat, 06 Jun 2009 10:14:34 -1000
Subject: [Cython] support for numpy bool type
In-Reply-To: <E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
	<E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
Message-ID: <4A2ACE2A.8040906@hawaii.edu>

Robert Bradshaw wrote:
> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
> 
>> Eric Firing wrote:
>>> Eric Firing wrote:
>>>> In writing a cython extension to work with numpy masked arrays, I
>>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>>> was expecting to be able to use np.bool_t, in analogy to np.float_t.
>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>>
>>>> Thanks.
>>>>
>>>> Eric
>> The reason it is not there is that AFAIK Cython has no support for an
>> 8-bit boolean type. I.e. you want
>>
>> print arr[3] # should print "True", not 1.
>>
>> and furthermore you want
>>
>> arr[3] = obj # should mean arr[3] = bool(obj)
>>
>> Which brings up the questions
>> 1) I suppose the best thing would be to add support for a "bool" which
>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>> appropriate restrictions otherwise. Thoughts?
> 
> How would this relate to the current bint type? bint is an int  
> because logical operations in C are ints, and also several operations  
> in the Python/C API (and elsewhere) return true/false values as ints.
> 
>> 2) Is it better to keep ndarray[bool_t] disabled until this  
>> happens, or
>> should it be enabled (as an "unsigned char" array) and then change its
>> semantics later?
> 
> I would be OK with allowing bool_t to be a synonym for int8_t, and  
> then widening the semantics later (I don't see what would break,  
> unless one was storing actual integer (rather than boolean) values).

I think it should be uint8_t, not int8_t, because numpy booleans are 
stored as unsigned char.

Eric

> 
> - Robert
> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Sat Jun  6 22:22:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 6 Jun 2009 22:22:35 +0200 (CEST)
Subject: [Cython] support for numpy bool type
In-Reply-To: <E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
	<E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
Message-ID: <a236bdcbdd2d91e6838e52565565d5ac.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
>
>> Eric Firing wrote:
>>> Eric Firing wrote:
>>>> In writing a cython extension to work with numpy masked arrays, I
>>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>>> was expecting to be able to use np.bool_t, in analogy to np.float_t.
>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>>
>>>> Thanks.
>>>>
>>>> Eric
>> The reason it is not there is that AFAIK Cython has no support for an
>> 8-bit boolean type. I.e. you want
>>
>> print arr[3] # should print "True", not 1.
>>
>> and furthermore you want
>>
>> arr[3] = obj # should mean arr[3] = bool(obj)
>>
>> Which brings up the questions
>> 1) I suppose the best thing would be to add support for a "bool" which
>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>> appropriate restrictions otherwise. Thoughts?
>
> How would this relate to the current bint type? bint is an int
> because logical operations in C are ints, and also several operations
> in the Python/C API (and elsewhere) return true/false values as ints.

Well, bint is an int that has different semantics for conversion to/from
Python, and that seems to be needed here as well (e.g. a "bchar", which I
think is essentially C99 _Bool).

What happens if you do "cdef bint value = 4"? Is bool(4) called resulting
in 1 being placed in value? I think that's the wanted behaviour here...

Using bint directly seems to be right out though as one must access the
data using an 8-bit integer type. (Well, I could hard-code buffers to cast
back and forth between "bint" and "char", after the pointer dereference,
but that seems very unclean compared to introducing a new type).

Perhaps we should have "char bint", "short bint", "unsigned long long
bint" and so on :-) Really, the need for a seperate type for bools is a
strict Cython feature, because what it affects is Python object
conversion, so it doesn't hurt IMO that these are not in C. And size and
booleanness are orthogonal features. (But, this is a puristic approach
only, and in practice "bchar" would suffice.)

Dag Sverre


From jonovik at gmail.com  Sun Jun  7 04:06:38 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Sun, 7 Jun 2009 02:06:38 +0000 (UTC)
Subject: [Cython] running doctests in pyx files
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
	<4A27A01A.70601@student.matnat.uio.no>
	<4A27A081.8020807@student.matnat.uio.no>
	<78f7ab620906040837k67e1fb82ncb32e43de83ea93@mail.gmail.com>
Message-ID: <loom.20090607T020207-45@post.gmane.org>

S?bastien Barth?lemy <barthelemy at ...> writes:

> >> This is a problem with doctest, which uses inspect.is_function to check
> >> whether something is a function, which fails for Cython functions (which
> >> instead answer to inspect.is_builtin). It would be nice if someone would
> >> take this to the doctest list with a patch at some point...
[...]
> http://wiki.cython.org/
FAQ#Doctestskipstestsfromcythonfunctions.2Cwhy.3FIsthereaworkaround.3F

Thanks for this, it was very useful. I've tweaked it into a reusable module 
allowing this:

python -c "import cydoctest, mymod; cydoctest.testmod(mymod)"

http://wiki.cython.org/FAQ#HowcanIrundoctestsinCythoncode.28pyxfiles.29.3F



From barthelemy at crans.org  Sun Jun  7 06:54:28 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Sun, 7 Jun 2009 06:54:28 +0200
Subject: [Cython] error(pos, "'%s.pxd' not found" % module_name)
Message-ID: <78f7ab620906062154g5e964644o5164e834bc935b3f@mail.gmail.com>

Hello all,

When I run "cython arboris/core/test.pyx" cython fails with the
attached msg.txt.

Apparently, the problem comes the following line (#8 in test.pyx)

from misc_c cimport asym, sin, cos

If I replace "cimport" by "import" it works fine.

However, there might be a bug in cython because msg.txt is sooo
unhelpful in the bad case.

Regards
-------------- next part --------------
Traceback (most recent call last):
  File "/usr/bin/cython", line 8, in <module>
    main(command_line = 1)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 733, in main
    result = compile(sources, options)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 710, in compile
    return compile_multiple(source, options)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 680, in compile_multiple
    result = run_pipeline(source, options)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 542, in run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 197, in run_pipeline
    data = phase(data)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 638, in __call__
    return super(AnalyseDeclarationsTransform, self).__call__(root)        
  File "Visitor.py", line 264, in Cython.Compiler.Visitor.CythonTransform.__call__ (/home/seb/Telechargements/Cython-0.11.2/Cython/Compiler/Visitor.c:4528)
  File "Visitor.py", line 250, in Cython.Compiler.Visitor.VisitorTransform.__call__ (/home/seb/Telechargements/Cython-0.11.2/Cython/Compiler/Visitor.c:4289)
  File "Visitor.py", line 45, in Cython.Compiler.Visitor.BasicVisitor.visit (/home/seb/Telechargements/Cython-0.11.2/Cython/Compiler/Visitor.c:1328)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 646, in visit_ModuleNode
    node.analyse_declarations(self.env_stack[-1])
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/ModuleNode.py", line 60, in analyse_declarations
    self.body.analyse_declarations(env)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py", line 358, in analyse_declarations
    stat.analyse_declarations(env)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py", line 4739, in analyse_declarations
    submodule_scope = env.context.find_module(name, relative_to = module_scope, pos = self.pos)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 260, in find_module
    error(pos, "'%s.pxd' not found" % module_name)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Errors.py", line 132, in error
    report_error(err)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Errors.py", line 121, in report_error
    line = "%s\n" % err
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 170: ordinal not in range(128)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.pyx
Type: application/octet-stream
Size: 2171 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: misc_c.pxd
Type: application/octet-stream
Size: 148 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0005.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.pxd
Type: application/octet-stream
Size: 114 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0006.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: misc_c.pyx
Type: application/octet-stream
Size: 1074 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0007.obj 

From dagss at student.matnat.uio.no  Mon Jun  8 11:39:33 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 08 Jun 2009 11:39:33 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A159A96.9070907@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
Message-ID: <4A2CDC55.8000600@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> A decision is needed here within a few weeks; it would be great if any 
> core developers who don't care could respond to vote 0.

*bump*

Unless I hear something, I'm going to assume that people trust my 
opinion here and start implementing the solution below (it's kind of a 
blocker for Kurt's GSoC). Speak now or forever hold your peace...

Dag Sverre



> 
> The decision was made to use [] for templates. To reiterate the main 
> arguments:
> 
>   - It leaves the way open for exporting pre-instantiated templates to 
> Python in a /very/ natural fashion.
> 
>   - Templates are a bit like a collection of types anyway.
> 
> 
> However this creates some syntax issues:
> 
> A) Currently
> 
>    cdef object[int] arr = ...
> 
> can be used for PEP 3118 access on a generic object. While not a 
> technical problem (object will never become a template), it looks a bit 
> odd when introducing with a template feature.
> 
> B) How would a templated buffer type look like? MyType[int][float]?
> 
> Proposal:
> 
> The object[int] syntax is removed (are anyone using it?). This is 
> basically done by making any cdef class with the 
> __cythonbufferdefaults__ magic attribute become a "buffer template". 
> Such cdef classes cannot be templates. Any other class cannot be used as 
> a buffer-supporting object.
> 
> This removes functionality (much numeric code could care less about 
> whether the underlying object is an ndarray). As a substitute, the main 
> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted, 
> which is to use
> 
> cdef int[:] arr = ...
> 
> for generic access, without any object access (this means that for now 
> only indexing and getting the shape is allowed at all.)
> 
> I've been wondering a lot about whether this is the right way, and it 
> still seems so, even if it means that there will be (and likely continue 
> to be) two different ways of accessing a buffer; one through an explicit 
> "buffer type" and one through an "automatic template".
> 


-- 
Dag Sverre

From kwmsmith at gmail.com  Mon Jun  8 17:12:42 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 8 Jun 2009 10:12:42 -0500
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2CDC55.8000600@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
Message-ID: <c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>

On Mon, Jun 8, 2009 at 4:39 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> A decision is needed here within a few weeks; it would be great if any
>> core developers who don't care could respond to vote 0.
>
> *bump*
>
> Unless I hear something, I'm going to assume that people trust my
> opinion here and start implementing the solution below (it's kind of a
> blocker for Kurt's GSoC). Speak now or forever hold your peace...

The new syntax is fine with me, but you already knew that, and I'd be
presuming a lot to say I'm a core developer ;-)  I've some clarifying
questions below since my understanding of templates is fairly basic.

>>
>> The decision was made to use [] for templates. To reiterate the main
>> arguments:
>>
>> ? - It leaves the way open for exporting pre-instantiated templates to
>> Python in a /very/ natural fashion.
>>
>> ? - Templates are a bit like a collection of types anyway.
>>
>>
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>> ? ?cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?). This is
>> basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be used as
>> a buffer-supporting object.

'removed': so no one can use

cdef object[int] foo = ...

anymore?  Or it has different semantics (must be a 'buffer template'
cdef class)?  I presume all the current syntax for buffer indexing
will still be around for backwards compatibility.

>>
>> This removes functionality (much numeric code could care less about
>> whether the underlying object is an ndarray). As a substitute, the main
>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>> which is to use

What functionality is removed exactly?  The fast indexing with cdef
ints, and all buffer access is now through the object's __getitem__
and __setitem__ only?

>>
>> cdef int[:] arr = ...
>>
>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
>>
>> I've been wondering a lot about whether this is the right way, and it
>> still seems so, even if it means that there will be (and likely continue
>> to be) two different ways of accessing a buffer; one through an explicit
>> "buffer type" and one through an "automatic template".

Just to be clear: the "buffer type" is the cdef int[:], and the
'automatic template' is the cdef MyObj[int].  Both will be indexable,
with the first using default 'fast' indexing on the buffer itself, and
the second indexing through the MyObj.__getitem__.  Right?

Kurt

From ellisonbg.net at gmail.com  Mon Jun  8 20:05:33 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 8 Jun 2009 11:05:33 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A159A96.9070907@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
Message-ID: <6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>

Is there a reason the C++ <> syntax for templates was rejected?

Brian

On Thu, May 21, 2009 at 11:16 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> A decision is needed here within a few weeks; it would be great if any
> core developers who don't care could respond to vote 0.
>
> The decision was made to use [] for templates. To reiterate the main
> arguments:
>
> ?- It leaves the way open for exporting pre-instantiated templates to
> Python in a /very/ natural fashion.
>
> ?- Templates are a bit like a collection of types anyway.
>
>
> However this creates some syntax issues:
>
> A) Currently
>
> ? cdef object[int] arr = ...
>
> can be used for PEP 3118 access on a generic object. While not a
> technical problem (object will never become a template), it looks a bit
> odd when introducing with a template feature.
>
> B) How would a templated buffer type look like? MyType[int][float]?
>
> Proposal:
>
> The object[int] syntax is removed (are anyone using it?). This is
> basically done by making any cdef class with the
> __cythonbufferdefaults__ magic attribute become a "buffer template".
> Such cdef classes cannot be templates. Any other class cannot be used as
> a buffer-supporting object.
>
> This removes functionality (much numeric code could care less about
> whether the underlying object is an ndarray). As a substitute, the main
> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
> which is to use
>
> cdef int[:] arr = ...
>
> for generic access, without any object access (this means that for now
> only indexing and getting the shape is allowed at all.)
>
> I've been wondering a lot about whether this is the right way, and it
> still seems so, even if it means that there will be (and likely continue
> to be) two different ways of accessing a buffer; one through an explicit
> "buffer type" and one through an "automatic template".
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Mon Jun  8 20:35:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 8 Jun 2009 20:35:06 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>
References: <4A159A96.9070907@student.matnat.uio.no>
	<6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>
Message-ID: <82d1b76d4a698d9d26575117da01f1b8.squirrel@webmail.uio.no>

> Is there a reason the C++ <> syntax for templates was rejected?
>
> Brian

I can't seem to find the thread now, so this is from memory. There's the
points already mentioned (quoted below); I consider it extremely valuable
myself that templates written in Cython are potentially exportable to
Python using the same syntax.

Also it can be parsed more cleanly. E.g.

a<b>(c)

Is that two comparisons or calling the constructor of a template? No way
to know until you check the type of a and c, meaning the syntax is
ambiguous until you check that. (In Python this kind of thing is a
showstopper for a new syntax; Guido argues that a simple parser means a
language that's also simple to parse by humans).

(Even C++ struggles with this kind of thing; requiring you to write
"a<b<c> >", i.e. the extra, ugly space to avoid writing a right-shift
operator.)

In Cython we're not so picky about things like this and there are ways of
resolving ambiguous syntax later in the stage; but it does mean that if
Python gets generic types (which isn't entirely impossible), <> is out of
the question, while we could hope for Python using [] (Guido already used
it in mock demonstration syntax for the purpose).

> On Thu, May 21, 2009 at 11:16 AM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> A decision is needed here within a few weeks; it would be great if any
>> core developers who don't care could respond to vote 0.
>>
>> The decision was made to use [] for templates. To reiterate the main
>> arguments:
>>
>> ?- It leaves the way open for exporting pre-instantiated templates to
>> Python in a /very/ natural fashion.
>>
>> ?- Templates are a bit like a collection of types anyway.
>>
>>
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>> ? cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?). This is
>> basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be used as
>> a buffer-supporting object.
>>
>> This removes functionality (much numeric code could care less about
>> whether the underlying object is an ndarray). As a substitute, the main
>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>> which is to use
>>
>> cdef int[:] arr = ...
>>
>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
>>
>> I've been wondering a lot about whether this is the right way, and it
>> still seems so, even if it means that there will be (and likely continue
>> to be) two different ways of accessing a buffer; one through an explicit
>> "buffer type" and one through an "automatic template".
>>
>> --
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



From ellisonbg.net at gmail.com  Mon Jun  8 20:39:26 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 8 Jun 2009 11:39:26 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <82d1b76d4a698d9d26575117da01f1b8.squirrel@webmail.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>
	<82d1b76d4a698d9d26575117da01f1b8.squirrel@webmail.uio.no>
Message-ID: <6ce0ac130906081139w7dffc38cxbc00403d3376aaf4@mail.gmail.com>

> I can't seem to find the thread now, so this is from memory. There's the
> points already mentioned (quoted below); I consider it extremely valuable
> myself that templates written in Cython are potentially exportable to
> Python using the same syntax.
>
> Also it can be parsed more cleanly. E.g.
>
> a<b>(c)
>
> Is that two comparisons or calling the constructor of a template? No way
> to know until you check the type of a and c, meaning the syntax is
> ambiguous until you check that. (In Python this kind of thing is a
> showstopper for a new syntax; Guido argues that a simple parser means a
> language that's also simple to parse by humans).

Yes, this makes sense.

> (Even C++ struggles with this kind of thing; requiring you to write
> "a<b<c> >", i.e. the extra, ugly space to avoid writing a right-shift
> operator.)

I agree that the C++ syntax is not necessarily the prettiest as well.

> In Cython we're not so picky about things like this and there are ways of
> resolving ambiguous syntax later in the stage; but it does mean that if
> Python gets generic types (which isn't entirely impossible), <> is out of
> the question, while we could hope for Python using [] (Guido already used
> it in mock demonstration syntax for the purpose).

Ahh, then this makes total sense.

Cheers,

Brian

>> On Thu, May 21, 2009 at 11:16 AM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> A decision is needed here within a few weeks; it would be great if any
>>> core developers who don't care could respond to vote 0.
>>>
>>> The decision was made to use [] for templates. To reiterate the main
>>> arguments:
>>>
>>> ?- It leaves the way open for exporting pre-instantiated templates to
>>> Python in a /very/ natural fashion.
>>>
>>> ?- Templates are a bit like a collection of types anyway.
>>>
>>>
>>> However this creates some syntax issues:
>>>
>>> A) Currently
>>>
>>> ? cdef object[int] arr = ...
>>>
>>> can be used for PEP 3118 access on a generic object. While not a
>>> technical problem (object will never become a template), it looks a bit
>>> odd when introducing with a template feature.
>>>
>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>
>>> Proposal:
>>>
>>> The object[int] syntax is removed (are anyone using it?). This is
>>> basically done by making any cdef class with the
>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>> Such cdef classes cannot be templates. Any other class cannot be used as
>>> a buffer-supporting object.
>>>
>>> This removes functionality (much numeric code could care less about
>>> whether the underlying object is an ndarray). As a substitute, the main
>>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>>> which is to use
>>>
>>> cdef int[:] arr = ...
>>>
>>> for generic access, without any object access (this means that for now
>>> only indexing and getting the shape is allowed at all.)
>>>
>>> I've been wondering a lot about whether this is the right way, and it
>>> still seems so, even if it means that there will be (and likely continue
>>> to be) two different ways of accessing a buffer; one through an explicit
>>> "buffer type" and one through an "automatic template".
>>>
>>> --
>>> Dag Sverre
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dalcinl at gmail.com  Mon Jun  8 20:41:36 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 8 Jun 2009 15:41:36 -0300
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2CDC55.8000600@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
Message-ID: <e7ba66e40906081141g1f03c334ua0e84fe9c7475ad1@mail.gmail.com>

On Mon, Jun 8, 2009 at 6:39 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> A decision is needed here within a few weeks; it would be great if any
>> core developers who don't care could respond to vote 0.
>
> *bump*
>
> Unless I hear something, I'm going to assume that people trust my
> opinion here and start implementing the solution below (it's kind of a
> blocker for Kurt's GSoC). Speak now or forever hold your peace...
>
> Dag Sverre
>

Perhaps you do not remember, but I've expressed my +1 for templates using  []

I'll just point that I like "MyType[int,float]" more than
"MyType[int][float]" ...

>
>
>>
>> The decision was made to use [] for templates. To reiterate the main
>> arguments:
>>
>> ? - It leaves the way open for exporting pre-instantiated templates to
>> Python in a /very/ natural fashion.
>>
>> ? - Templates are a bit like a collection of types anyway.
>>
>>
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>> ? ?cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?). This is
>> basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be used as
>> a buffer-supporting object.
>>
>> This removes functionality (much numeric code could care less about
>> whether the underlying object is an ndarray). As a substitute, the main
>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>> which is to use
>>
>> cdef int[:] arr = ...
>>
>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
>>
>> I've been wondering a lot about whether this is the right way, and it
>> still seems so, even if it means that there will be (and likely continue
>> to be) two different ways of accessing a buffer; one through an explicit
>> "buffer type" and one through an "automatic template".
>>
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Jun  8 20:53:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 8 Jun 2009 20:53:51 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
	<c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>
Message-ID: <16477e0839ee8f2b42848ef748378222.squirrel@webmail.uio.no>

Kurt Smith wrote:
> On Mon, Jun 8, 2009 at 4:39 AM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Dag Sverre Seljebotn wrote:
> The new syntax is fine with me, but you already knew that, and I'd be
> presuming a lot to say I'm a core developer ;-)  I've some clarifying
> questions below since my understanding of templates is fairly basic.

To be clear, I definitely want input from *everybody* (users too!) I
didn't put it that well.

>>> However this creates some syntax issues:
>>>
>>> A) Currently
>>>
>>> ? ?cdef object[int] arr = ...
>>>
>>> can be used for PEP 3118 access on a generic object. While not a
>>> technical problem (object will never become a template), it looks a bit
>>> odd when introducing with a template feature.
>>>
>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>
>>> Proposal:
>>>
>>> The object[int] syntax is removed (are anyone using it?). This is
>>> basically done by making any cdef class with the
>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>> Such cdef classes cannot be templates. Any other class cannot be used
>>> as
>>> a buffer-supporting object.
>
> 'removed': so no one can use
>
> cdef object[int] foo = ...
>
> anymore?  Or it has different semantics (must be a 'buffer template'
> cdef class)?  I presume all the current syntax for buffer indexing
> will still be around for backwards compatibility.

Removed, so noone can use

cdef object[int] foo

any more. To get that exact behaviour, one has to do

cdef object foo
cdef int[:] foo_buf = foo

I think this is fair enough -- I'd like to see a real usecase for this
syntax if the new one is introduced?

ndarray[int] stays though.

>>> This removes functionality (much numeric code could care less about
>>> whether the underlying object is an ndarray). As a substitute, the main
>>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>>> which is to use
>
> What functionality is removed exactly?  The fast indexing with cdef
> ints, and all buffer access is now through the object's __getitem__
> and __setitem__ only?

Just removing object[type] by itself, without compensating, removes the
possibility of writing functions which work across multiple buffers
exporters. E.g. a single function which can handle both Image objects and
ndarray objects through the buffer API.


>
>>>
>>> cdef int[:] arr = ...
>>>
>>> for generic access, without any object access (this means that for now
>>> only indexing and getting the shape is allowed at all.)
>>>
>>> I've been wondering a lot about whether this is the right way, and it
>>> still seems so, even if it means that there will be (and likely
>>> continue
>>> to be) two different ways of accessing a buffer; one through an
>>> explicit
>>> "buffer type" and one through an "automatic template".
>
> Just to be clear: the "buffer type" is the cdef int[:], and the
> 'automatic template' is the cdef MyObj[int].  Both will be indexable,
> with the first using default 'fast' indexing on the buffer itself, and
> the second indexing through the MyObj.__getitem__.  Right?

No. I was clumsy here. If MyObj defines __cythonbufferdefaults__, Cython
treats it like ndarray is treated today. Otherwise, and when templates are
implemented, the "int" is used in whatever way the MyObj template uses it.

But what I mean is that in both cases, from the user's perspective one can
look at MyObj as a template of sorts.

However for writing more generic buffer code (where no knowledge of the
type of the object is sometimes not needed at all) a different kind of
concept is needed. object[int] used to provide this but I don't think many
used it.

Dag Sverre


From dagss at student.matnat.uio.no  Mon Jun  8 20:59:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 8 Jun 2009 20:59:12 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <e7ba66e40906081141g1f03c334ua0e84fe9c7475ad1@mail.gmail.com>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
	<e7ba66e40906081141g1f03c334ua0e84fe9c7475ad1@mail.gmail.com>
Message-ID: <d0c3953c4cf4539a5346a8f1d384882c.squirrel@webmail.uio.no>

Lisandro wrote:
> Perhaps you do not remember, but I've expressed my +1 for templates using
> []
>
> I'll just point that I like "MyType[int,float]" more than
> "MyType[int][float]" ...
>

I think I write too long emails :-)

Anyway, this wasn't about the template syntax (which is pretty much
settled I believe; Brian Granger just wanted to know why the decision went
that way), but about changing the PEP 3118 syntax, specifically, removing
the possibility of doing

cdef object[int] a

(is anybody even using it for real?) and replace it with

cdef int[:] a

with slightly different semantics. ndarray[int] stays as today.

Dag Sverre


From kwmsmith at gmail.com  Mon Jun  8 22:20:54 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 8 Jun 2009 15:20:54 -0500
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <16477e0839ee8f2b42848ef748378222.squirrel@webmail.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
	<c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>
	<16477e0839ee8f2b42848ef748378222.squirrel@webmail.uio.no>
Message-ID: <c266a7930906081320s4b5c84d4t5751a25d9fde27bd@mail.gmail.com>

On Mon, Jun 8, 2009 at 1:53 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Kurt Smith wrote:
>> On Mon, Jun 8, 2009 at 4:39 AM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> Dag Sverre Seljebotn wrote:
>> The new syntax is fine with me, but you already knew that, and I'd be
>> presuming a lot to say I'm a core developer ;-) ?I've some clarifying
>> questions below since my understanding of templates is fairly basic.
>
> To be clear, I definitely want input from *everybody* (users too!) I
> didn't put it that well.
>
>>>> However this creates some syntax issues:
>>>>
>>>> A) Currently
>>>>
>>>> ? ?cdef object[int] arr = ...
>>>>
>>>> can be used for PEP 3118 access on a generic object. While not a
>>>> technical problem (object will never become a template), it looks a bit
>>>> odd when introducing with a template feature.
>>>>
>>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>>
>>>> Proposal:
>>>>
>>>> The object[int] syntax is removed (are anyone using it?). This is
>>>> basically done by making any cdef class with the
>>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>>> Such cdef classes cannot be templates. Any other class cannot be used
>>>> as
>>>> a buffer-supporting object.
>>
>> 'removed': so no one can use
>>
>> cdef object[int] foo = ...
>>
>> anymore? ?Or it has different semantics (must be a 'buffer template'
>> cdef class)? ?I presume all the current syntax for buffer indexing
>> will still be around for backwards compatibility.
>
> Removed, so noone can use
>
> cdef object[int] foo
>
> any more. To get that exact behaviour, one has to do
>
> cdef object foo
> cdef int[:] foo_buf = foo

I see now ;-)

>
> I think this is fair enough -- I'd like to see a real usecase for this
> syntax if the new one is introduced?
>
> ndarray[int] stays though.

A possible problem I envision with keeping 'ndarray[int]' around
exactly as is (although I'm exposing my ignorance of the proposed
template usage): if using bracket-templates becomes fairly common in
Cython, won't people then start to see 'ndarray[int]' as another
template expression and *not* a (magic) buffer statement?  Or is it
obvious since 'ndarray' is special?  If someone created their own cdef
class with __cythonbufferdefaults__, wouldn't it mimick the template
bracket expression/statement and be quite confusing?

>>
>> What functionality is removed exactly? ?The fast indexing with cdef
>> ints, and all buffer access is now through the object's __getitem__
>> and __setitem__ only?
>
> Just removing object[type] by itself, without compensating, removes the
> possibility of writing functions which work across multiple buffers
> exporters. E.g. a single function which can handle both Image objects and
> ndarray objects through the buffer API.
>
>
[snip]
>>
>> Just to be clear: the "buffer type" is the cdef int[:], and the
>> 'automatic template' is the cdef MyObj[int]. ?Both will be indexable,
>> with the first using default 'fast' indexing on the buffer itself, and
>> the second indexing through the MyObj.__getitem__. ?Right?
>
> No. I was clumsy here. If MyObj defines __cythonbufferdefaults__, Cython
> treats it like ndarray is treated today. Otherwise, and when templates are
> implemented, the "int" is used in whatever way the MyObj template uses it.
>
> But what I mean is that in both cases, from the user's perspective one can
> look at MyObj as a template of sorts.

I see.  Perhaps my above comment isn't such a concern, then.

Kurt

From robertwb at math.washington.edu  Tue Jun  9 09:06:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 00:06:50 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A159A96.9070907@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
Message-ID: <717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>

On May 21, 2009, at 11:16 AM, Dag Sverre Seljebotn wrote:

> A decision is needed here within a few weeks; it would be great if any
> core developers who don't care could respond to vote 0.

Sorry, I flagged this as something I wanted to reply to after reading  
your wiki pages on the topic, and then it scrolled off the top of my  
cython folder...

> However this creates some syntax issues:
>
> A) Currently
>
>    cdef object[int] arr = ...
>
> can be used for PEP 3118 access on a generic object. While not a
> technical problem (object will never become a template), it looks a  
> bit
> odd when introducing with a template feature.
>
> B) How would a templated buffer type look like? MyType[int][float]?
>
> Proposal:
>
> The object[int] syntax is removed (are anyone using it?).

It's a good idea to have a non-conflicting syntax like int[:], but I  
don't think we should be treating ndarray[int] and object[int] any  
differently, or removing backwards compatibility.

> This is basically done by making any cdef class with the
> __cythonbufferdefaults__ magic attribute become a "buffer template".
> Such cdef classes cannot be templates. Any other class cannot be  
> used as
> a buffer-supporting object.

Are you suggesting we require __cythonbufferdefaults__ for all buffer  
classes? __cythonbufferdefaults__ doesn't make sense for, e.g.,  
ndarrays.

> This removes functionality (much numeric code could care less about
> whether the underlying object is an ndarray). As a substitute, the  
> main
> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
> which is to use
>
> cdef int[:] arr = ...
>
> for generic access, without any object access (this means that for now
> only indexing and getting the shape is allowed at all.)

I like the int[:] syntax, but I'm -1 on the raw buffer no-object- 
access idea. Having to write

cdef object foo = ...
cdef int[:] foo_buf = foo

seems clunky to me--I really like that in the current system one  
doesn't have to have to separate variables to access a buffer as a  
fast array or a Python object--this fits in well with the whole  
Cython philosophy. (I do think we should print warnings for people  
who write A[i][j] (which is slower from Python too, and should be  
obvious if cython -a is run).)

I would rather see int[:] be a type qualifier (like long,  
unsigned, ...) so one can write

cdef int[:] object foo

- Robert

From dagss at student.matnat.uio.no  Tue Jun  9 09:30:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 09:30:55 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <loom.20090605T132731-705@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090605T132731-705@post.gmane.org>
Message-ID: <4A2E0FAF.7050109@student.matnat.uio.no>

Jon Olav Vik wrote:
> If I may make a wish, I would like Cython to find more declarations 
> automatically. For instance, having
> 
> cdef extern * from "cvode/cvode.h"
> 
> or, if that is impractical,
> 
> cdef extern from "cvode/cvode.h":
>     CVodeCreate, CVRhsFn, CVodeMalloc, CVodeFree, CVode, CV_ADAMS, CV_BDF

Yes, we all would. If you find a decent CS student at your institution 
who would like to do this as his or her master thesis then drop us a 
note :-)

I think there's a script someone posted on the mailing list to try to 
autogenerate pxds, though I don't remember enough to find it -- anyone?

> I haven't benchmarked anything yet, but the relative ease of adapting the C 
> tutorial for CVODE [3] is uplifting. As a last question, what is the cleanest 
> way to transfer arrays of double between realtype* and Numpy vectors (one-
> dimensional arrays)?

OK here it is. Note that this sets up the ndarray to point to the *same* 
memory. Modifications to the resulting NumPy array will happen directly 
in the NVector!

Case A: If you are 100% sure that you do not call N_VDestroy_Serial 
before all references to the ndarray created are gone, you can simply do 
this:

cdef extern from "numpy/arrayobject.h":
     # These are simply missing from numpy.pxd due to laziness;
     # so declare manually for now
     void import_array()
     object PyArray_SimpleNewFromData(int nd,
                                      np.npy_intp* dims,
                                      int typenum,
                                      void* data)



import_array() # Needed because we call the raw NumPy C API below!

def some_func():
     cdef np.npy_intp* shape = [somehow_extract_length_of(some_nvector)]
     # Create ndarray. Assume realtype is double.
     assert sizeof(realtype) == sizeof(double)
     yourarray = PyArray_SimpleNewFromData(
       1, shape, np.NPY_DOUBLE,
       <void*>self.nvector.whatever_to_get_buffer_pointer)
     ...

But this is dangerous as the resulting yourarray will address 
nonallocated memory if you free the underlying NVector.

Case B: Proper memory handling:

First, create a C header file (workaround.h), because Cython cannot 
handle expressions of the form "foo(x) = y" so we need to "reformulate" 
the NumPy C API a bit:

#include <numpy/arrayobject.h>
#define PyArray_Set_BASE(arr, obj) PyArray_BASE(arr) = obj

Then, this Cython code should do the job. Note that we need to create a 
class because Python needs an object to associate with the memory, so 
that the memory gets freed at the right time.

cimport numpy as np

cdef extern from "workaround.h":
     void PyArray_Set_BASE(np.ndarray arr, object obj)

cdef extern from "numpy/arrayobject.h":
     # These are simply missing from numpy.pxd due to laziness;
     # so declare manually for now
     void import_array()
     object PyArray_SimpleNewFromData(int nd,
                                      np.npy_intp* dims,
                                      int typenum,
                                      void* data)



import_array() # Needed because we call the raw NumPy C API below!

cdef class CythonNVector:
     cdef N_Vector nvector
     def __init__(self, length):
         self.nvector = N_VNew_Serial(length)

     def __dealloc__(self):
         # Called when the object is being destroyed (normally there's
         # no need for this in Python, but here it is, as we need to
         # free C memory explicitly).
         #
         # Note: The "Python parts" of the object might already
         # be gone in this method -- only touch C attributes.
         N_VDestroy_Serial(self.nvector)

     cpdef toarray(self):
         cdef np.npy_intp* shape = \
[somehow_extract_length_of(self.nvector)]
         # Create ndarray. Assume realtype is double.
         assert sizeof(realtype) == sizeof(double)
         cdef np.ndarray result = PyArray_SimpleNewFromData(
   1, shape, np.NPY_DOUBLE,
   <void*>self.nvector.whatever_to_get_buffer_pointer)
         # Make the ndarray keep a reference to this object
         PyArray_Set_BASE(result, self)
         return result


Just modify this general scheme for your usecase (i.e. you can set 
nvector manually rather than allocate it the constructor, etc.)



-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun  9 09:46:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 09:46:48 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
References: <4A159A96.9070907@student.matnat.uio.no>
	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
Message-ID: <4A2E1368.4040701@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 21, 2009, at 11:16 AM, Dag Sverre Seljebotn wrote:
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>>    cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a  
>> bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?).
> 
> It's a good idea to have a non-conflicting syntax like int[:], but I  
> don't think we should be treating ndarray[int] and object[int] any  
> differently, or removing backwards compatibility.

The idea here is that given templates the object hierarchy will look 
like this:

object
|
+-- SomeTemplate[int]
|
L-- ndarray[int]

My idea is then that it will look nicer if object doesn't default to 
being a "template" as well.

We could introduce a new decorator to make this point clear (at least, 
the following code will make clear how I'd prefer to think of it):

@cython.buffer.auto_buffer_template(
   defaults={"ndim":2, "mode":"strided"})
cdef class MyImage(object):
     ...

with the understanding that Cython transforms this into
cdef class MyImage[type, ndim=2, mode="strided"](object):
     ...

with an efficient __getitem__/__setitem__ implementation.

Making this behaviour the "default" template behaviour for a type, 
unless a template is explicitly requested, just seems more confusing.

I think virtually noone are using the object[...] at the moment, and 
better remove backwards compatability sooner than later.

But, it's certainly a point where pragmatism might go before purism.

>> This is basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be  
>> used as
>> a buffer-supporting object.
> 
> Are you suggesting we require __cythonbufferdefaults__ for all buffer  
> classes? __cythonbufferdefaults__ doesn't make sense for, e.g.,  
> ndarrays.

? They are used in numpy.pxd already (and if removed, mode would default 
to "full", causing an extra if-test per dimension per access...).

But I now like the above decorator -- but this would be the backwards 
compatability with most classes. Almost all serious uses must have set

cdef __cythonbufferdefaults__ = {"mode": "strided"}

anyway.

>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
> 
> I like the int[:] syntax, but I'm -1 on the raw buffer no-object- 
> access idea. Having to write
> 
> cdef object foo = ...
> cdef int[:] foo_buf = foo
> 
> seems clunky to me--I really like that in the current system one  
> doesn't have to have to separate variables to access a buffer as a  
> fast array or a Python object--this fits in well with the whole  
> Cython philosophy. (I do think we should print warnings for people  
> who write A[i][j] (which is slower from Python too, and should be  
> obvious if cython -a is run).)
> 
> I would rather see int[:] be a type qualifier (like long,  
> unsigned, ...) so one can write
> 
> cdef int[:] object foo

Hmm. I'm not against that -- would you then be ok with

cdef int[:] foo

not giving any object access at all? (The thing is I also want to use it 
to access arrays stored in C pointers etc.; + safe passing around of 
buffers in nogil mode!)

I guess I see it as getting hold of another interface -- I see it this 
way: Objects supporting PEP 3118 kind of "inherits from" int[:] through 
multiple inheritance.

(If doing something like this I'd rather like to adopt Guido's mock 
syntax for combining interfaces -- e.g. if a class implements both 
"Iterable" and "File" interfaces, one could do

cdef Iterable & File foo

Of course, Cython currently doesn't support multiple inheritance (even 
through Java interfaces), and we might not want to over-prepare for 
that. Meaning "cdef int[:] object foo" could work.)


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun  9 09:55:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 09:55:08 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2E1368.4040701@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
	<4A2E1368.4040701@student.matnat.uio.no>
Message-ID: <4A2E155C.1030701@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> seems clunky to me--I really like that in the current system one  
>> doesn't have to have to separate variables to access a buffer as a  
>> fast array or a Python object--this fits in well with the whole  
>> Cython philosophy. (I do think we should print warnings for people  
>> who write A[i][j] (which is slower from Python too, and should be  
>> obvious if cython -a is run).)
>>
>> I would rather see int[:] be a type qualifier (like long,  
>> unsigned, ...) so one can write
>>
>> cdef int[:] object foo
> 
> Hmm. I'm not against that -- would you then be ok with
> 
> cdef int[:] foo
> 
> not giving any object access at all? (The thing is I also want to use it 
> to access arrays stored in C pointers etc.; + safe passing around of 
> buffers in nogil mode!)

I really should reiterate a problem spot here: Efficient slicing.

cdef object[int] a = np.arange(10)
cdef object[int] b

b = a[5:] # made efficient
print b[0] # prints 5
print (<object>b)[0] # huh, prints 0?

Either that, or

print b.foo() # raises exception "NoneType has no attribute"?

When loosing the underlying object reference from the syntax, this 
problems disappears, as one won't expect "b.foo()" to work in the first 
place. Then

print (<object>b)[0] # prints 5

because b is coerced to a memoryview.


-- 
Dag Sverre

From robertwb at math.washington.edu  Tue Jun  9 10:31:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:31:53 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2E1368.4040701@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
	<4A2E1368.4040701@student.matnat.uio.no>
Message-ID: <E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>

On Jun 9, 2009, at 12:46 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 21, 2009, at 11:16 AM, Dag Sverre Seljebotn wrote:
>>> However this creates some syntax issues:
>>>
>>> A) Currently
>>>
>>>    cdef object[int] arr = ...
>>>
>>> can be used for PEP 3118 access on a generic object. While not a
>>> technical problem (object will never become a template), it looks a
>>> bit
>>> odd when introducing with a template feature.
>>>
>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>
>>> Proposal:
>>>
>>> The object[int] syntax is removed (are anyone using it?).
>>
>> It's a good idea to have a non-conflicting syntax like int[:], but I
>> don't think we should be treating ndarray[int] and object[int] any
>> differently, or removing backwards compatibility.
>
> The idea here is that given templates the object hierarchy will look
> like this:
>
> object
> |
> +-- SomeTemplate[int]
> |
> L-- ndarray[int]
>
> My idea is then that it will look nicer if object doesn't default to
> being a "template" as well.
>
> We could introduce a new decorator to make this point clear (at least,
> the following code will make clear how I'd prefer to think of it):
>
> @cython.buffer.auto_buffer_template(
>    defaults={"ndim":2, "mode":"strided"})
> cdef class MyImage(object):
>      ...
>
> with the understanding that Cython transforms this into
> cdef class MyImage[type, ndim=2, mode="strided"](object):
>      ...
>
> with an efficient __getitem__/__setitem__ implementation.
>
> Making this behaviour the "default" template behaviour for a type,
> unless a template is explicitly requested, just seems more confusing.

cdef class MyImage[...](object):

would always be a template, not a buffer.

> I think virtually noone are using the object[...] at the moment, and
> better remove backwards compatability sooner than later.

I see your point, but I think explicitly disallowing object is too  
inconsistent. It's also nice to not have to even cimport numpy to use  
efficient buffer access and be able to handle anything that exports a  
buffer interface.

> But, it's certainly a point where pragmatism might go before purism.
>
>>> This is basically done by making any cdef class with the
>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>> Such cdef classes cannot be templates. Any other class cannot be
>>> used as
>>> a buffer-supporting object.
>>
>> Are you suggesting we require __cythonbufferdefaults__ for all buffer
>> classes? __cythonbufferdefaults__ doesn't make sense for, e.g.,
>> ndarrays.
>
> ? They are used in numpy.pxd already (and if removed, mode would  
> default
> to "full", causing an extra if-test per dimension per access...).

Oh, you're right. I was thinking ndim couldn't be set, but there are  
other things that can be.

> But I now like the above decorator -- but this would be the backwards
> compatability with most classes. Almost all serious uses must have set
>
> cdef __cythonbufferdefaults__ = {"mode": "strided"}
>
> anyway.

I'm OK with either the decorator of the __cythonbufferdefaults__.

>
>>> for generic access, without any object access (this means that  
>>> for now
>>> only indexing and getting the shape is allowed at all.)
>>
>> I like the int[:] syntax, but I'm -1 on the raw buffer no-object-
>> access idea. Having to write
>>
>> cdef object foo = ...
>> cdef int[:] foo_buf = foo
>>
>> seems clunky to me--I really like that in the current system one
>> doesn't have to have to separate variables to access a buffer as a
>> fast array or a Python object--this fits in well with the whole
>> Cython philosophy. (I do think we should print warnings for people
>> who write A[i][j] (which is slower from Python too, and should be
>> obvious if cython -a is run).)
>>
>> I would rather see int[:] be a type qualifier (like long,
>> unsigned, ...) so one can write
>>
>> cdef int[:] object foo
>
> Hmm. I'm not against that -- would you then be ok with
>
> cdef int[:] foo
>
> not giving any object access at all? (The thing is I also want to  
> use it
> to access arrays stored in C pointers etc.; + safe passing around of
> buffers in nogil mode!)

I could accept that, I trust that the details could be worked out  
nicely. Would be like a char* from an object, where the user is  
required to keep the object around long enough?

> I guess I see it as getting hold of another interface -- I see it this
> way: Objects supporting PEP 3118 kind of "inherits from" int[:]  
> through
> multiple inheritance.

I see "providing X api/interface" and "inheriting from" as two  
orthogonal concepts (though the two have historically been conflated  
due to statically linked languages such as C++ and Java).

> (If doing something like this I'd rather like to adopt Guido's mock
> syntax for combining interfaces -- e.g. if a class implements both
> "Iterable" and "File" interfaces, one could do
>
> cdef Iterable & File foo
>
> Of course, Cython currently doesn't support multiple inheritance (even
> through Java interfaces), and we might not want to over-prepare for
> that. Meaning "cdef int[:] object foo" could work.)

Agreed. Also, we don't want the syntax to suggest that any two types  
could be paired--there's something very different going on here.

- Robert



From robertwb at math.washington.edu  Tue Jun  9 10:38:58 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:38:58 -0700
Subject: [Cython] support for numpy bool type
In-Reply-To: <a236bdcbdd2d91e6838e52565565d5ac.squirrel@webmail.uio.no>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
	<E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
	<a236bdcbdd2d91e6838e52565565d5ac.squirrel@webmail.uio.no>
Message-ID: <88BEB4A6-11CA-4201-BDC9-A1CE4DA6C6CF@math.washington.edu>

On Jun 6, 2009, at 1:22 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
>>
>>> Eric Firing wrote:
>>>> Eric Firing wrote:
>>>>> In writing a cython extension to work with numpy masked arrays, I
>>>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>>>> was expecting to be able to use np.bool_t, in analogy to  
>>>>> np.float_t.
>>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Eric
>>> The reason it is not there is that AFAIK Cython has no support  
>>> for an
>>> 8-bit boolean type. I.e. you want
>>>
>>> print arr[3] # should print "True", not 1.
>>>
>>> and furthermore you want
>>>
>>> arr[3] = obj # should mean arr[3] = bool(obj)
>>>
>>> Which brings up the questions
>>> 1) I suppose the best thing would be to add support for a "bool"  
>>> which
>>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>>> appropriate restrictions otherwise. Thoughts?
>>
>> How would this relate to the current bint type? bint is an int
>> because logical operations in C are ints, and also several operations
>> in the Python/C API (and elsewhere) return true/false values as ints.
>
> Well, bint is an int that has different semantics for conversion to/ 
> from
> Python,

That is exactly what it is.

> and that seems to be needed here as well (e.g. a "bchar", which I
> think is essentially C99 _Bool).

a C99 _Bool can only have two values, anything non-zero is converted  
to exactly 1.

> What happens if you do "cdef bint value = 4"? Is bool(4) called  
> resulting
> in 1 being placed in value? I think that's the wanted behaviour  
> here...

No, it retains the value 4. (This is one of the reasons I didn't call  
it a bool.)

> Using bint directly seems to be right out though as one must access  
> the
> data using an 8-bit integer type. (Well, I could hard-code buffers  
> to cast
> back and forth between "bint" and "char", after the pointer  
> dereference,
> but that seems very unclean compared to introducing a new type).
>
> Perhaps we should have "char bint", "short bint", "unsigned long long
> bint" and so on :-) Really, the need for a seperate type for bools  
> is a
> strict Cython feature, because what it affects is Python object
> conversion, so it doesn't hurt IMO that these are not in C. And  
> size and
> booleanness are orthogonal features. (But, this is a puristic approach
> only, and in practice "bchar" would suffice.)

Actually, that might work well. When I first introduced it, I saw no  
need to offer various sizes, but if one has arrays of them then it  
becomes important. And just when we thought we were simplifying the  
integer type system... :)

- Robert



From robertwb at math.washington.edu  Tue Jun  9 10:39:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:39:59 -0700
Subject: [Cython] Trivial numpy test in sage notebook fails...
In-Reply-To: <3327124144.3090608@smtp.netcom.no>
References: <3327124144.3090608@smtp.netcom.no>
Message-ID: <79017BF3-B11C-4378-A1C1-33572625AB28@math.washington.edu>

To conclude this thread, Sage ships its own numpy.pxd, which predates  
the buffer interface. We need to go in and remove this (there's a  
ticket in Sage to do this).

On Jun 6, 2009, at 12:09 AM, Dag Sverre Seljebotn wrote:

> afaik this is a problem with the Sage notebook. Try to Cythonize  
> from the command line, and if that works, take it to the Sage list.
>
> Dag Sverre Seljebotn
> -----Original Message-----
> From: "Glenn Tarbox, PhD" <glenn at tarbox.org>
> Date: Saturday, Jun 6, 2009 6:01 am
> Subject: [Cython] Trivial numpy test in sage notebook fails...
> To: Cython-dev at codespeak.netReply-To: cython-dev at codespeak.net
>
> Here's the code which is the first cell of a sage notebook (sage 4.0)
>>
>> %cython
>> import numpy as np
>> cimport numpy as np
>> DTYPE = np.int
>> ctypedef np.int_t DTYPE_t
>>
>> Here's the error:
>>
>> Traceback (most recent call last):    DTYPE = np.int
>>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/ 
>> sage/server/support.py', line 408, in cython_import_all
>>    create_local_c_file=create_local_c_file)
>>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/ 
>> sage/server/support.py', line 385, in cython_import
>>    create_local_c_file=create_local_c_file)
>>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/ 
>> sage/misc/cython.py', line 367, in cython
>>    raise RuntimeError, 'Error converting %s to C:\n%s\n%s'% 
>> (filename, log, err)
>> RuntimeError: Error converting /home/tarbox/.sage/sage_notebook/ 
>> worksheets/admin/89/code/sage23.spyx to C:
>>
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>>
>> include 'cdefs.pxi'
>> import numpy as np
>> cimport numpy as np
>> DTYPE = np.int
>> ctypedef np.int_t DTYPE_t        ^
>> ------------------------------------------------------------
>>
>> /home/tarbox/.sage/temp/hq2/12471/spyx/ 
>> _home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx 
>> / 
>> _home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx 
>> _0.pyx:9:9: 'int_t' is not a type identifier
>> I tried with and without the 'import numpy' directive thinking it  
>> may be redundant...
>>
>> This example comes from the 'cython for numpy users' tutorial on  
>> the wiki where it mentions that all numpy types have a  
>> corresponding _t type
>>
>> There is indeed a numpy.int type... so I was wondering if the new  
>> work with python buffers changed how things work.
>>
>> -glenn
>>
>> -- 
>> Glenn H. Tarbox, PhD ||  206-274-6919
>> http://www.tarbox.org
>>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Tue Jun  9 10:41:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:41:32 -0700
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
Message-ID: <3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>

On Jun 4, 2009, at 2:49 PM, Lisandro Dalcin wrote:

> On Thu, Jun 4, 2009 at 1:29 PM, Stefan Behnel <stefan_ml at behnel.de>  
> wrote:
>> Lisandro Dalcin wrote:
>>> On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel wrote:
>>>>
>>>> I think we should file a bug report against Py3.1 and even bring  
>>>> this to
>>>> the attention of python-dev before it's too late.
>>>>
>>>
>>> Stefan, I saw your post in python-dev and corresponding issue in the
>>> tracker, but I got confused. Is the solution know and the issue  
>>> going
>>> to be fixed?
>>
>> I didn't find the time to look into this any deeper. I'm not sure  
>> it will
>> get fixed, because a) the bug itself is a lot older than rc1, and  
>> b) the
>> comment sounded more like "the fix might be similar to what we did  
>> here".
>>
>> I think we should still open a new bug. Given that you experienced  
>> the
>> problem, could you do that?
>>
>
> Done, http://bugs.python.org/issue6195, with patch, I've added Stefan
> to the noisy list.

Does the new patch there resolve the issue?

- Robert


From robertwb at math.washington.edu  Tue Jun  9 10:43:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:43:26 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
In-Reply-To: <f125f5e0906031131nebf9677tca009911f4861bf5@mail.gmail.com>
References: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
	<72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>
	<f125f5e0906031131nebf9677tca009911f4861bf5@mail.gmail.com>
Message-ID: <43627760-11C2-4014-87CE-07101AE45538@math.washington.edu>

On Jun 3, 2009, at 11:31 AM, Andrew Collette wrote:

> Hi,
>
>> Strange--it looks like this was fixed months ago:
>>
>> http://hg.cython.org/cython-devel/diff/8b6d42d16c99/Cython/Compiler/
>> ExprNodes.py
>
> It looks like for some reason Cython wasn't being upgraded correctly;
> after sudo python setup.py install, cython --version still reported
> 0.10.3.  I had to manually delete the Cython directory in
> site-packages and reinstall to get it to report 0.11.2.  But it works
> now!

I've seen this a couple of times before--not sure what the issue is.

>> Maybe you need to touch the .pyx to re-generate the sources. It is
>> gratifying to hear that is the only issue--should we be running our
>> tests with this option? (Or is it gcc only?)
>
> I think all major compilers have a strict-ansi mode, although I'm sure
> the switch is different for VC++.  I currently only use it with gcc on
> OS X for compatibility with another package that looks for
> __STRICT_ANSI__.  However, if Cython is intended to generate pure C89
> code it might be worth adding this as a test.  At the moment it seems
> this is already the case; my large-ish (9k line) Cython project now
> compiles fine with -ansi.

Cool. Yeah, I think we should add the flag to runtests.py if we can  
detect the compiler.

- Robert


From dagss at student.matnat.uio.no  Tue Jun  9 10:46:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 10:46:00 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>
References: <4A159A96.9070907@student.matnat.uio.no>	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>	<4A2E1368.4040701@student.matnat.uio.no>
	<E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>
Message-ID: <4A2E2148.4020204@student.matnat.uio.no>

Robert Bradshaw wrote:
>> Hmm. I'm not against that -- would you then be ok with
>>
>> cdef int[:] foo
>>
>> not giving any object access at all? (The thing is I also want to  
>> use it
>> to access arrays stored in C pointers etc.; + safe passing around of
>> buffers in nogil mode!)
> 
> I could accept that, I trust that the details could be worked out  
> nicely. Would be like a char* from an object, where the user is  
> required to keep the object around long enough?

With C pointers, yes. With buffers acquired from Python objects it would 
hold a reference, but I was thinking only acquisition/release would 
require GIL. I.e.

cdef int[:] a = obj # requires GIL
cdef int[:] b = a # does not require GIL -- extra "acquisition count"
# (remember that acquisition does have a speed penalty too, that
# really must go if one wants to use efficient slices inside
# numeric loops)

In some very contrived cases a release could be needed within a nogil 
block though...there are ways to resolve that (release queues or 
reacquiring the GIL temporarily). But users are unlikely to produce code 
requiring it.

I think I didn't manage to communicate my intentions regarding the other 
question with object[...] vs. templates, but now my daughter is awake so 
it'll have to wait :-)


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun  9 14:29:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 14:29:23 +0200
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
Message-ID: <4A2E55A3.5040509@student.matnat.uio.no>

I realize now that the recent discussion shouldn't be a discussion of 
syntax as such, but rather the overall strategy for numerics in Cython.

I think I finally know how to present the issue that I've been circling 
around during the spring reasonably clearly -- however I don't want to 
start either a) a thread where I have a longwinded start which nobody 
(understandably) bother to pick up, leaving us in the current state of 
indecision, or b) one of those 200-post threads spanning three weeks 
(which is too long and a lot of effort, summed up).

Can we try an IRC session? Do people have time? (Anybody who's 
interested, but at lest Robert and Kurt)? (If so I'll try to pose the 
question more properly on a single wiki screenful or so beforehand.)

-- 
Dag Sverre

From dalcinl at gmail.com  Tue Jun  9 15:52:38 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 9 Jun 2009 10:52:38 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
	<3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>
Message-ID: <e7ba66e40906090652v1d737514g61e4174269885c14@mail.gmail.com>

On Tue, Jun 9, 2009 at 5:41 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
>
> Does the new patch there resolve the issue?
>

Yes, it seems to fix the issue... Just tested from a fresh SVN update
with the latest patch applied.

BTW, I'm getting this failure in Py3k... we should handle this, though
not sure about what to do.

extdelslice.cpp:464: error: invalid conversion from ?int
(*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?
extsetslice.cpp:466: error: invalid conversion from ?int
(*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?


I'm also getting other failures, some are changes in exception
messages, other are related to division ....

This one is very nince :-)

<...> in complex_numbers_T305
Failed example:
    test_arithmetic(2j, 4j)
Expected:
    (-2j, 6j, -2j, (-8+0j), (0.5+0j))
Got:
    ((-0-2j), 6j, -2j, (-8+0j), (0.5+0j))


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jun  9 16:10:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 9 Jun 2009 16:10:30 +0200 (CEST)
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906090652v1d737514g61e4174269885c14@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
	<3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>
	<e7ba66e40906090652v1d737514g61e4174269885c14@mail.gmail.com>
Message-ID: <1a002aa5800dae922b2dde276fd291e3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> Yes, it seems to fix the issue... Just tested from a fresh SVN update
> with the latest patch applied.

Cool, well done.


> BTW, I'm getting this failure in Py3k... we should handle this, though
> not sure about what to do.
>
> extdelslice.cpp:464: error: invalid conversion from ?int
> (*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?
> extsetslice.cpp:466: error: invalid conversion from ?int
> (*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?

I think a warning that __*slice__() isn't portable would be nice. Apart
from that, I wouldn't care too much. Maybe remove the test alltogether.


> I'm also getting other failures, some are changes in exception
> messages, other are related to division ....

Exception message changes are ugly. My take is: if Cython generates a
similar exception or compiler error, try to follow Py3 regarding the
message (if it makes sense, but there's usually a reason why they did it).
In our doctests, try to check as much of an exception as possible to avoid
false positives. But I agree that it's really tricky to check these things
in a portable way.


> This one is very nince :-)
>
> <...> in complex_numbers_T305
> Failed example:
>     test_arithmetic(2j, 4j)
> Expected:
>     (-2j, 6j, -2j, (-8+0j), (0.5+0j))
> Got:
>     ((-0-2j), 6j, -2j, (-8+0j), (0.5+0j))

Beautiful. :)

We do a lot of doctest string manipulation already, so that's just another
case IMHO.

Stefan


From kwmsmith at gmail.com  Tue Jun  9 18:04:03 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 9 Jun 2009 11:04:03 -0500
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <4A2E55A3.5040509@student.matnat.uio.no>
References: <4A2E55A3.5040509@student.matnat.uio.no>
Message-ID: <c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>

On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> I realize now that the recent discussion shouldn't be a discussion of
> syntax as such, but rather the overall strategy for numerics in Cython.
>
> I think I finally know how to present the issue that I've been circling
> around during the spring reasonably clearly -- however I don't want to
> start either a) a thread where I have a longwinded start which nobody
> (understandably) bother to pick up, leaving us in the current state of
> indecision, or b) one of those 200-post threads spanning three weeks
> (which is too long and a lot of effort, summed up).
>
> Can we try an IRC session? Do people have time? (Anybody who's
> interested, but at lest Robert and Kurt)? (If so I'll try to pose the
> question more properly on a single wiki screenful or so beforehand.)

I can make time -- I'll just sit around on #cython.  Send an email if
I'm not responding.

Kurt

From kwmsmith at gmail.com  Tue Jun  9 18:05:09 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 9 Jun 2009 11:05:09 -0500
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>
References: <4A2E55A3.5040509@student.matnat.uio.no>
	<c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>
Message-ID: <c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>

On Tue, Jun 9, 2009 at 11:04 AM, Kurt Smith<kwmsmith at gmail.com> wrote:
> On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> I realize now that the recent discussion shouldn't be a discussion of
>> syntax as such, but rather the overall strategy for numerics in Cython.
>>
>> I think I finally know how to present the issue that I've been circling
>> around during the spring reasonably clearly -- however I don't want to
>> start either a) a thread where I have a longwinded start which nobody
>> (understandably) bother to pick up, leaving us in the current state of
>> indecision, or b) one of those 200-post threads spanning three weeks
>> (which is too long and a lot of effort, summed up).
>>
>> Can we try an IRC session? Do people have time? (Anybody who's
>> interested, but at lest Robert and Kurt)? (If so I'll try to pose the
>> question more properly on a single wiki screenful or so beforehand.)
>
> I can make time -- I'll just sit around on #cython. ?Send an email if
> I'm not responding.

Unless, of course, you were thinking of later in the week, or
something ;-)  Just say the word.

>
> Kurt
>

From robertwb at math.washington.edu  Tue Jun  9 18:12:01 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 09:12:01 -0700
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>
References: <4A2E55A3.5040509@student.matnat.uio.no>
	<c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>
	<c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>
Message-ID: <421423AC-F52E-41AF-984B-CA91732BFDF4@math.washington.edu>

On Jun 9, 2009, at 9:05 AM, Kurt Smith wrote:

> On Tue, Jun 9, 2009 at 11:04 AM, Kurt Smith<kwmsmith at gmail.com> wrote:
>> On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> I realize now that the recent discussion shouldn't be a  
>>> discussion of
>>> syntax as such, but rather the overall strategy for numerics in  
>>> Cython.
>>>
>>> I think I finally know how to present the issue that I've been  
>>> circling
>>> around during the spring reasonably clearly -- however I don't  
>>> want to
>>> start either a) a thread where I have a longwinded start which  
>>> nobody
>>> (understandably) bother to pick up, leaving us in the current  
>>> state of
>>> indecision, or b) one of those 200-post threads spanning three weeks
>>> (which is too long and a lot of effort, summed up).
>>>
>>> Can we try an IRC session? Do people have time? (Anybody who's
>>> interested, but at lest Robert and Kurt)? (If so I'll try to pose  
>>> the
>>> question more properly on a single wiki screenful or so beforehand.)
>>
>> I can make time -- I'll just sit around on #cython.  Send an email if
>> I'm not responding.
>
> Unless, of course, you were thinking of later in the week, or
> something ;-)  Just say the word.

I'll be around (much) later tonight.

- Robert


From ondrej at certik.cz  Tue Jun  9 20:03:13 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 9 Jun 2009 12:03:13 -0600
Subject: [Cython] force cython to do local imports
Message-ID: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>

Hi,

I have the following problem -- I have a package hermes2d, with the structure:

hermes2d/__init__.py
hermes2d/_hermes2d.pyx + .pxd
hermes2d/_forms.pyx

and sometimes (depending on what the users tells cmake) I want to
build the .so libraries out of the tree and put their paths to
PYTHONPATH. Things just work as long as remove the file
hermes2d/__init__.py during the cython compilation. The reason is that
if I do:

cd hermes2d
cython _forms.pyx

it produces _forms.c with lines like:

  __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
__Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
__pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
(unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
{__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
__LINE__; goto __pyx_L1_error;}

Notice the import "hermes2d._hermes2d", which fails if _hermes2d.so is
out of the tree (e..g somewhere else on the filesystem, even if it's
in the search path!). If, however I remove the __init__.py, then
cython generates:

  __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
"LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
(unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
__pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
__pyx_L1_error;}

and everything just works, no matter where the file _hermes2d.so actually is.

Now my question is -- if I want those .so libraries not to be in the
tree, am I doing things the right way? E.g. I just have a line:

from _forms import *

in the hermes2d/forms.py

and then people just do "from hermes2d import forms" and things work,
no matter where the .so files are. But I need some way to force cython
to just pretend there is no __init__.py in there. Why is cython doing
this check at all? Does it have any advantages?

Thanks,
Ondrej

From seb.binet at gmail.com  Tue Jun  9 20:33:27 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Tue, 9 Jun 2009 20:33:27 +0200
Subject: [Cython] command to only parse cython statements ?
Message-ID: <200906092033.27437.binet@cern.ch>

hi there,

Just for fun and to see how cython works, I am trying to write a 
CythonInterpreter inheriting from the usual code.InteractiveConsole.

So far so good, I manage to compile cython-oneliners (leveraging 
pyximport.load_module) but then I'd need some help for multi-lines statements:
## ex:
cdef class Foo:
  pass
##

for this to work in a reasonnable timely fashion, I'd need to see if parsing 
the snippet of code is valid cython (without compiling).
(I guess I'd need to know if the parsing failed b/c of unsupported python 
constructs or if it is b/c the cython command is incomplete)

any hint ?

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From dagss at student.matnat.uio.no  Tue Jun  9 21:41:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 21:41:14 +0200
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <421423AC-F52E-41AF-984B-CA91732BFDF4@math.washington.edu>
References: <4A2E55A3.5040509@student.matnat.uio.no>	<c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>	<c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>
	<421423AC-F52E-41AF-984B-CA91732BFDF4@math.washington.edu>
Message-ID: <4A2EBADA.3010600@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 9, 2009, at 9:05 AM, Kurt Smith wrote:
> 
>> On Tue, Jun 9, 2009 at 11:04 AM, Kurt Smith<kwmsmith at gmail.com> wrote:
>>> On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> I realize now that the recent discussion shouldn't be a  
>>>> discussion of
>>>> syntax as such, but rather the overall strategy for numerics in  
>>>> Cython.
>>>>
>>>> I think I finally know how to present the issue that I've been  
>>>> circling
>>>> around during the spring reasonably clearly -- however I don't  
>>>> want to
>>>> start either a) a thread where I have a longwinded start which  
>>>> nobody
>>>> (understandably) bother to pick up, leaving us in the current  
>>>> state of
>>>> indecision, or b) one of those 200-post threads spanning three weeks
>>>> (which is too long and a lot of effort, summed up).
>>>>
>>>> Can we try an IRC session? Do people have time? (Anybody who's
>>>> interested, but at lest Robert and Kurt)? (If so I'll try to pose  
>>>> the
>>>> question more properly on a single wiki screenful or so beforehand.)
>>> I can make time -- I'll just sit around on #cython.  Send an email if
>>> I'm not responding.
>> Unless, of course, you were thinking of later in the week, or
>> something ;-)  Just say the word.
> 
> I'll be around (much) later tonight.

I'm primarily available 2 more hours, but I guess that likely I'll catch 
you in my morning (10 hours from now).

(If it's the only option I'm at a point where I will get up in the 
middle of the night if it helps getting this stuff resolved though -- if 
so please respond with an estimate. Just sometime during the week is 
fine though.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 10 07:36:11 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 07:36:11 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <200906092033.27437.binet@cern.ch>
References: <200906092033.27437.binet@cern.ch>
Message-ID: <9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>

Sebastien Binet wrote:
> hi there,
>
> Just for fun and to see how cython works, I am trying to write a
> CythonInterpreter inheriting from the usual code.InteractiveConsole.
>
> So far so good, I manage to compile cython-oneliners (leveraging
> pyximport.load_module) but then I'd need some help for multi-lines
> statements:
> ## ex:
> cdef class Foo:
>   pass
> ##
>
> for this to work in a reasonnable timely fashion, I'd need to see if
> parsing
> the snippet of code is valid cython (without compiling).
> (I guess I'd need to know if the parsing failed b/c of unsupported python
> constructs or if it is b/c the cython command is incomplete)
>
> any hint ?

I don't think this is possible currently as Cython can generate errors at
any point in its pipeline. In time it might be possible to stop the
pipeline at an earlier stage, but I wouldn't expect great speedups here
still as parsing takes a significant part of the time (don't remember the
numbers though). Sorry.

Dag Svere


From robertwb at math.washington.edu  Wed Jun 10 08:03:12 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 23:03:12 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2E2148.4020204@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>	<4A2E1368.4040701@student.matnat.uio.no>
	<E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>
	<4A2E2148.4020204@student.matnat.uio.no>
Message-ID: <9378A591-507F-4D07-9838-B28152C4F6A0@math.washington.edu>

On Jun 9, 2009, at 1:46 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>> Hmm. I'm not against that -- would you then be ok with
>>>
>>> cdef int[:] foo
>>>
>>> not giving any object access at all? (The thing is I also want to
>>> use it
>>> to access arrays stored in C pointers etc.; + safe passing around of
>>> buffers in nogil mode!)
>>
>> I could accept that, I trust that the details could be worked out
>> nicely. Would be like a char* from an object, where the user is
>> required to keep the object around long enough?
>
> With C pointers, yes. With buffers acquired from Python objects it  
> would
> hold a reference, but I was thinking only acquisition/release would
> require GIL. I.e.
>
> cdef int[:] a = obj # requires GIL
> cdef int[:] b = a # does not require GIL -- extra "acquisition count"
> # (remember that acquisition does have a speed penalty too, that
> # really must go if one wants to use efficient slices inside
> # numeric loops)
>
> In some very contrived cases a release could be needed within a nogil
> block though...there are ways to resolve that (release queues or
> reacquiring the GIL temporarily). But users are unlikely to produce  
> code
> requiring it.

This sounds like a very good use case, and perhaps one could even  
assign (with a cast?) raw c pointers to it too (or maybe I'm being  
too ambitious there).

- Robert


From stefan_ml at behnel.de  Wed Jun 10 08:07:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 10 Jun 2009 08:07:23 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
References: <200906092033.27437.binet@cern.ch>
	<9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
Message-ID: <4A2F4D9B.9070502@behnel.de>

Hi,

Dag Sverre Seljebotn wote:
> Sebastien Binet wrote:
>> Just for fun and to see how cython works, I am trying to write a
>> CythonInterpreter inheriting from the usual code.InteractiveConsole.

Way cool. I always wanted something like that for quickly checking out code
snippets without going all the way through writing a module and
pyximporting it.


>> So far so good, I manage to compile cython-oneliners (leveraging
>> pyximport.load_module) but then I'd need some help for multi-lines
>> statements:
>> ## ex:
>> cdef class Foo:
>>   pass
>> ##
>>
>> for this to work in a reasonnable timely fashion, I'd need to see if
>> parsing the snippet of code is valid cython (without compiling).

The current parser isn't made for parsing things interactively, so I have
no idea if this is trivial or hard to do.

You might get away with checking for parser errors that indicate a missing
indentation after the last character that you passed. Every parse error
inside or before the last line would indicate a 'real' parser error.


>> (I guess I'd need to know if the parsing failed b/c of unsupported python
>> constructs or if it is b/c the cython command is incomplete)
> 
> I don't think this is possible currently as Cython can generate errors at
> any point in its pipeline.

This isn't the point here, though. All he needs is to know if a command is
complete or if he should keep reading stuff from the console.


> In time it might be possible to stop the
> pipeline at an earlier stage, but I wouldn't expect great speedups here
> still as parsing takes a significant part of the time (don't remember the
> numbers though).

Parse time really isn't a problem here. I think the main problem will be
that you need to load and reload tons of modules (again, not a timing
issue). Maybe giving the module a public C-API interface and accessing it
through ctypes would work better? But that's not something that needs
solving as a first step. I'm fine with loading a couple of thousand modules
in an interactive session for now. The main use case is testing stuff, so
restarting the interpreter when it gets too heavy isn't a huge problem IMHO.

Stefan

From robertwb at math.washington.edu  Wed Jun 10 08:13:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 23:13:24 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
Message-ID: <BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>

On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:

> Hi,
>
> I have the following problem -- I have a package hermes2d, with the  
> structure:
>
> hermes2d/__init__.py
> hermes2d/_hermes2d.pyx + .pxd
> hermes2d/_forms.pyx
>
> and sometimes (depending on what the users tells cmake) I want to
> build the .so libraries out of the tree and put their paths to
> PYTHONPATH. Things just work as long as remove the file
> hermes2d/__init__.py during the cython compilation. The reason is that
> if I do:
>
> cd hermes2d
> cython _forms.pyx
>
> it produces _forms.c with lines like:
>
>   __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}
>
> Notice the import "hermes2d._hermes2d", which fails if _hermes2d.so is
> out of the tree (e..g somewhere else on the filesystem, even if it's
> in the search path!). If, however I remove the __init__.py, then
> cython generates:
>
>   __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
> __pyx_L1_error;}
>
> and everything just works, no matter where the file _hermes2d.so  
> actually is.
>
> Now my question is -- if I want those .so libraries not to be in the
> tree, am I doing things the right way? E.g. I just have a line:
>
> from _forms import *
>
> in the hermes2d/forms.py
>
> and then people just do "from hermes2d import forms" and things work,
> no matter where the .so files are. But I need some way to force cython
> to just pretend there is no __init__.py in there. Why is cython doing
> this check at all? Does it have any advantages?

One advantage I see is that you may have two modules named _hermes2d  
in different packages that it might have to distinguish. (OK, rare  
with that name, but it could happen with more common names). Also,  
this way the module path is fully qualified as one would expect.

- Robert



From stefan at sun.ac.za  Wed Jun 10 15:58:28 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 10 Jun 2009 15:58:28 +0200
Subject: [Cython] How to build Cython extensions for Windows under Linux or
	OSX
Message-ID: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>

Hi all,

Yesterday, I had to build a Cython extension for a colleague using
Windows, so here are the steps required:

1. Install Wine (on OSX I used http://www.macports.org/)
2. Install Python 2.5 from the http://www.python.org .exe
3. Install mingw from http://www.mingw.org/.  Install the base system and g++.
4. Start the registry editor (wine regedit) and setup the system path:

 - Search for Environment and find the PATH key
 - Append to it c:\python25;c:\mingw\bin;c:\python25\scripts
 - Add a new key called "HOME" and set it to c:\windows\profiles\yourusername

5. Create a new file in ~/.wine/drive_c/windows/profiles/yourusername/
called pydistutils.cfg that contains

[build]
compiler=mingw32

This tells distutils not to look for MSVC7, but to use mingw as the
default compiler.

6. Install setuptools (not necessary, but convenient if you want to
install any other packages).

Get the .tar.gz from http://pypi.python.org/pypi/setuptools.
Unpack.
wine python setup.py install

7. Install Cython (either from source as in 6 or using `easy_install cython`)
8. Compile you extension (from your package source directory):

python setup.py bdist_wininst

This will only work if you have a setup.py file, similar to the one in

http://dip.sun.ac.za/~stefan/code/cython_demo.git

This step should then generate a .exe in the dist/ directory.

If your package requires NumPy to work, you'll have to install that
prior to step 8, by

1. Extracting the numpy source tree
2. python setup.py build -c mingw32
3. python setup.py install --skip-build

I hope someone else finds this recipe useful!

Regards
St?fan

From dalcinl at gmail.com  Wed Jun 10 16:30:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 10 Jun 2009 11:30:15 -0300
Subject: [Cython] How to build Cython extensions for Windows under Linux
	or OSX
In-Reply-To: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
References: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
Message-ID: <e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>

Just a couple of comments/questions regarding Windows, an OS I really
do not know well :-)...

1) Why did you need Wine?

2) Why did you use regedit to setup environment variables?
http://support.microsoft.com/kb/310519

3) Regarding setuptools... Why the exe installers at PyPI are not
enough for you?

4) Regarding numpy...  Why the exe installers are SourceForge are not
enough for you?


2009/6/10 St?fan van der Walt <stefan at sun.ac.za>:
> Hi all,
>
> Yesterday, I had to build a Cython extension for a colleague using
> Windows, so here are the steps required:
>
> 1. Install Wine (on OSX I used http://www.macports.org/)
> 2. Install Python 2.5 from the http://www.python.org .exe
> 3. Install mingw from http://www.mingw.org/. ?Install the base system and g++.
> 4. Start the registry editor (wine regedit) and setup the system path:
>
> ?- Search for Environment and find the PATH key
> ?- Append to it c:\python25;c:\mingw\bin;c:\python25\scripts
> ?- Add a new key called "HOME" and set it to c:\windows\profiles\yourusername
>
> 5. Create a new file in ~/.wine/drive_c/windows/profiles/yourusername/
> called pydistutils.cfg that contains
>
> [build]
> compiler=mingw32
>
> This tells distutils not to look for MSVC7, but to use mingw as the
> default compiler.
>
> 6. Install setuptools (not necessary, but convenient if you want to
> install any other packages).
>
> Get the .tar.gz from http://pypi.python.org/pypi/setuptools.
> Unpack.
> wine python setup.py install
>
> 7. Install Cython (either from source as in 6 or using `easy_install cython`)
> 8. Compile you extension (from your package source directory):
>
> python setup.py bdist_wininst
>
> This will only work if you have a setup.py file, similar to the one in
>
> http://dip.sun.ac.za/~stefan/code/cython_demo.git
>
> This step should then generate a .exe in the dist/ directory.
>
> If your package requires NumPy to work, you'll have to install that
> prior to step 8, by
>
> 1. Extracting the numpy source tree
> 2. python setup.py build -c mingw32
> 3. python setup.py install --skip-build
>
> I hope someone else finds this recipe useful!
>
> Regards
> St?fan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed Jun 10 17:08:04 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 10 Jun 2009 11:08:04 -0400
Subject: [Cython] How to build Cython extensions for Windows under Linux
	or OSX
In-Reply-To: <e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
References: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
	<e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
Message-ID: <7f014ea60906100808k3dca2698j889f04dbd2f4402c@mail.gmail.com>

I think because he was cross compiling these extensions for windows
while working in MacOSX or Linux

On Wed, Jun 10, 2009 at 10:30 AM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> Just a couple of comments/questions regarding Windows, an OS I really
> do not know well :-)...
>
> 1) Why did you need Wine?
>
> 2) Why did you use regedit to setup environment variables?
> http://support.microsoft.com/kb/310519
>
> 3) Regarding setuptools... Why the exe installers at PyPI are not
> enough for you?
>
> 4) Regarding numpy... ?Why the exe installers are SourceForge are not
> enough for you?
>
>
> 2009/6/10 St?fan van der Walt <stefan at sun.ac.za>:
>> Hi all,
>>
>> Yesterday, I had to build a Cython extension for a colleague using
>> Windows, so here are the steps required:
>>
>> 1. Install Wine (on OSX I used http://www.macports.org/)
>> 2. Install Python 2.5 from the http://www.python.org .exe
>> 3. Install mingw from http://www.mingw.org/. ?Install the base system and g++.
>> 4. Start the registry editor (wine regedit) and setup the system path:
>>
>> ?- Search for Environment and find the PATH key
>> ?- Append to it c:\python25;c:\mingw\bin;c:\python25\scripts
>> ?- Add a new key called "HOME" and set it to c:\windows\profiles\yourusername
>>
>> 5. Create a new file in ~/.wine/drive_c/windows/profiles/yourusername/
>> called pydistutils.cfg that contains
>>
>> [build]
>> compiler=mingw32
>>
>> This tells distutils not to look for MSVC7, but to use mingw as the
>> default compiler.
>>
>> 6. Install setuptools (not necessary, but convenient if you want to
>> install any other packages).
>>
>> Get the .tar.gz from http://pypi.python.org/pypi/setuptools.
>> Unpack.
>> wine python setup.py install
>>
>> 7. Install Cython (either from source as in 6 or using `easy_install cython`)
>> 8. Compile you extension (from your package source directory):
>>
>> python setup.py bdist_wininst
>>
>> This will only work if you have a setup.py file, similar to the one in
>>
>> http://dip.sun.ac.za/~stefan/code/cython_demo.git
>>
>> This step should then generate a .exe in the dist/ directory.
>>
>> If your package requires NumPy to work, you'll have to install that
>> prior to step 8, by
>>
>> 1. Extracting the numpy source tree
>> 2. python setup.py build -c mingw32
>> 3. python setup.py install --skip-build
>>
>> I hope someone else finds this recipe useful!
>>
>> Regards
>> St?fan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ondrej at certik.cz  Wed Jun 10 17:35:42 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 09:35:42 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
Message-ID: <85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>

On Wed, Jun 10, 2009 at 12:13 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:
>
>> Hi,
>>
>> I have the following problem -- I have a package hermes2d, with the
>> structure:
>>
>> hermes2d/__init__.py
>> hermes2d/_hermes2d.pyx + .pxd
>> hermes2d/_forms.pyx
>>
>> and sometimes (depending on what the users tells cmake) I want to
>> build the .so libraries out of the tree and put their paths to
>> PYTHONPATH. Things just work as long as remove the file
>> hermes2d/__init__.py during the cython compilation. The reason is that
>> if I do:
>>
>> cd hermes2d
>> cython _forms.pyx
>>
>> it produces _forms.c with lines like:
>>
>> ? __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
>> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
>> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
>> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
>> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
>> __LINE__; goto __pyx_L1_error;}
>>
>> Notice the import "hermes2d._hermes2d", which fails if _hermes2d.so is
>> out of the tree (e..g somewhere else on the filesystem, even if it's
>> in the search path!). If, however I remove the __init__.py, then
>> cython generates:
>>
>> ? __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
>> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
>> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
>> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
>> __pyx_L1_error;}
>>
>> and everything just works, no matter where the file _hermes2d.so
>> actually is.
>>
>> Now my question is -- if I want those .so libraries not to be in the
>> tree, am I doing things the right way? E.g. I just have a line:
>>
>> from _forms import *
>>
>> in the hermes2d/forms.py
>>
>> and then people just do "from hermes2d import forms" and things work,
>> no matter where the .so files are. But I need some way to force cython
>> to just pretend there is no __init__.py in there. Why is cython doing
>> this check at all? Does it have any advantages?
>
> One advantage I see is that you may have two modules named _hermes2d
> in different packages that it might have to distinguish. (OK, rare
> with that name, but it could happen with more common names). Also,
> this way the module path is fully qualified as one would expect.

Ok. So how do you propose I should fix the problem, if I need those
.so libraries at different place than the rest of the python source
files?

Ondrej

From seb.binet at gmail.com  Wed Jun 10 17:54:57 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Wed, 10 Jun 2009 17:54:57 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <4A2F4D9B.9070502@behnel.de>
References: <200906092033.27437.binet@cern.ch>
	<9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
	<4A2F4D9B.9070502@behnel.de>
Message-ID: <200906101754.57824.binet@cern.ch>

On Wednesday 10 June 2009 08:07:23 Stefan Behnel wrote:
> Hi,
>
> Dag Sverre Seljebotn wote:
> > Sebastien Binet wrote:
> >> Just for fun and to see how cython works, I am trying to write a
> >> CythonInterpreter inheriting from the usual code.InteractiveConsole.
>
> Way cool. I always wanted something like that for quickly checking out code
> snippets without going all the way through writing a module and
> pyximporting it.
>
> >> So far so good, I manage to compile cython-oneliners (leveraging
> >> pyximport.load_module) but then I'd need some help for multi-lines
> >> statements:
> >> ## ex:
> >> cdef class Foo:
> >>   pass
> >> ##
> >>
> >> for this to work in a reasonnable timely fashion, I'd need to see if
> >> parsing the snippet of code is valid cython (without compiling).
>
> The current parser isn't made for parsing things interactively, so I have
> no idea if this is trivial or hard to do.
>
> You might get away with checking for parser errors that indicate a missing
> indentation after the last character that you passed. Every parse error
> inside or before the last line would indicate a 'real' parser error.

thanks for the hint.
So I am now able to successfully enter at the cython prompt:
cython> cdef class Foo:
...              cdef int i

yeah!

the problem I get now is that I am out of luck if I want to actually do:
cdef class Foo:
  cdef int i
  cdef int j

as the class Foo is compiled just after I entered "cdef int i"...
I suppose I would have to know the indentation context of the last statement 
(if indent_lvl>0, gimme a new line)

hum... maybe I am not tackling this from the correct angle (perhaps that a 
simple buffer of strings filled while the last entry is not the_empty_string 
would do as I quick hack...)

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From robertwb at math.washington.edu  Wed Jun 10 18:01:33 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 10 Jun 2009 09:01:33 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
Message-ID: <574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>

On Jun 10, 2009, at 8:35 AM, Ondrej Certik wrote:

> On Wed, Jun 10, 2009 at 12:13 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:
>>
>>> Hi,
>>>
>>> I have the following problem -- I have a package hermes2d, with the
>>> structure:
>>>
>>> hermes2d/__init__.py
>>> hermes2d/_hermes2d.pyx + .pxd
>>> hermes2d/_forms.pyx
>>>
>>> and sometimes (depending on what the users tells cmake) I want to
>>> build the .so libraries out of the tree and put their paths to
>>> PYTHONPATH. Things just work as long as remove the file
>>> hermes2d/__init__.py during the cython compilation. The reason is  
>>> that
>>> if I do:
>>>
>>> cd hermes2d
>>> cython _forms.pyx
>>>
>>> it produces _forms.c with lines like:
>>>
>>>   __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
>>> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
>>> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
>>> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
>>> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
>>> __LINE__; goto __pyx_L1_error;}
>>>
>>> Notice the import "hermes2d._hermes2d", which fails if  
>>> _hermes2d.so is
>>> out of the tree (e..g somewhere else on the filesystem, even if it's
>>> in the search path!). If, however I remove the __init__.py, then
>>> cython generates:
>>>
>>>   __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
>>> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
>>> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
>>> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
>>> __pyx_L1_error;}
>>>
>>> and everything just works, no matter where the file _hermes2d.so
>>> actually is.
>>>
>>> Now my question is -- if I want those .so libraries not to be in the
>>> tree, am I doing things the right way? E.g. I just have a line:
>>>
>>> from _forms import *
>>>
>>> in the hermes2d/forms.py
>>>
>>> and then people just do "from hermes2d import forms" and things  
>>> work,
>>> no matter where the .so files are. But I need some way to force  
>>> cython
>>> to just pretend there is no __init__.py in there. Why is cython  
>>> doing
>>> this check at all? Does it have any advantages?
>>
>> One advantage I see is that you may have two modules named _hermes2d
>> in different packages that it might have to distinguish. (OK, rare
>> with that name, but it could happen with more common names). Also,
>> this way the module path is fully qualified as one would expect.
>
> Ok. So how do you propose I should fix the problem, if I need those
> .so libraries at different place than the rest of the python source
> files?

I don't know. A hack is to have your setup.py actually move the  
__init__.py temporarily--I'm sure there's a better longterm fix  
though. Just out of curiosity, why do you need to segregate the .so  
files?

- Robert



From robertwb at math.washington.edu  Wed Jun 10 18:02:56 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 10 Jun 2009 09:02:56 -0700
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <200906101754.57824.binet@cern.ch>
References: <200906092033.27437.binet@cern.ch>
	<9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
	<4A2F4D9B.9070502@behnel.de> <200906101754.57824.binet@cern.ch>
Message-ID: <D31AC5CD-8DA4-4460-B019-3FC1E646DBAA@math.washington.edu>

On Jun 10, 2009, at 8:54 AM, Sebastien Binet wrote:

> On Wednesday 10 June 2009 08:07:23 Stefan Behnel wrote:
>> Hi,
>>
>> Dag Sverre Seljebotn wote:
>>> Sebastien Binet wrote:
>>>> Just for fun and to see how cython works, I am trying to write a
>>>> CythonInterpreter inheriting from the usual  
>>>> code.InteractiveConsole.
>>
>> Way cool. I always wanted something like that for quickly checking  
>> out code
>> snippets without going all the way through writing a module and
>> pyximporting it.
>>
>>>> So far so good, I manage to compile cython-oneliners (leveraging
>>>> pyximport.load_module) but then I'd need some help for multi-lines
>>>> statements:
>>>> ## ex:
>>>> cdef class Foo:
>>>>   pass
>>>> ##
>>>>
>>>> for this to work in a reasonnable timely fashion, I'd need to  
>>>> see if
>>>> parsing the snippet of code is valid cython (without compiling).
>>
>> The current parser isn't made for parsing things interactively, so  
>> I have
>> no idea if this is trivial or hard to do.
>>
>> You might get away with checking for parser errors that indicate a  
>> missing
>> indentation after the last character that you passed. Every parse  
>> error
>> inside or before the last line would indicate a 'real' parser error.
>
> thanks for the hint.
> So I am now able to successfully enter at the cython prompt:
> cython> cdef class Foo:
> ...              cdef int i
>
> yeah!
>
> the problem I get now is that I am out of luck if I want to  
> actually do:
> cdef class Foo:
>   cdef int i
>   cdef int j
>
> as the class Foo is compiled just after I entered "cdef int i"...
> I suppose I would have to know the indentation context of the last  
> statement
> (if indent_lvl>0, gimme a new line)
>
> hum... maybe I am not tackling this from the correct angle (perhaps  
> that a
> simple buffer of strings filled while the last entry is not  
> the_empty_string
> would do as I quick hack...)

The way ipython does it is to require a new empty line to "leave" an  
indentation block--maybe you could do the same. (Or maybe you could  
even hook this in with ipython).

- Robert


From dagss at student.matnat.uio.no  Wed Jun 10 18:32:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 18:32:24 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
Message-ID: <4A2FE018.5060902@student.matnat.uio.no>

Thanks to chats with Kurt and Robert, I think I've managed to present my 
thoughts from a different and more constructive angle.

http://wiki.cython.org/enhancements/array

There's some time constraints in the picture for various reasons (e.g. 
Fortran GSoC development direction) so it would be great if this could 
be resolved one way or the other before too long. Part of that will 
likely happen on IRC.

Feedback from anybody (especially users) very welcome.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 10 18:35:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 18:35:12 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE018.5060902@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
Message-ID: <4A2FE0C0.9020004@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Thanks to chats with Kurt and Robert, I think I've managed to present my 
> thoughts from a different and more constructive angle.
> 
> http://wiki.cython.org/enhancements/array
> 
> There's some time constraints in the picture for various reasons (e.g. 
> Fortran GSoC development direction) so it would be great if this could 
> be resolved one way or the other before too long. Part of that will 
> likely happen on IRC.
> 
> Feedback from anybody (especially users) very welcome.
> 

Oh..: Please, let's discuss the high-level first (*if* something 
remotely looking like this is wanted at all) before discussing any of 
the lower-level stuff (syntax, implementation, feature-set etc.)

-- 
Dag Sverre

From cournape at gmail.com  Wed Jun 10 18:43:37 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 11 Jun 2009 01:43:37 +0900
Subject: [Cython] How to build Cython extensions for Windows under Linux
	or OSX
In-Reply-To: <e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
References: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
	<e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
Message-ID: <5b8d13220906100943y7e3f599dhaf2cb5155a69cbac@mail.gmail.com>

On Wed, Jun 10, 2009 at 11:30 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> Just a couple of comments/questions regarding Windows, an OS I really
> do not know well :-)...
>
> 1) Why did you need Wine?

Because then you don't have to use windows, of course ! :) The last
numpy binaries are built in wine, as a matter of fact.

David

From seb.binet at gmail.com  Wed Jun 10 19:20:01 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Wed, 10 Jun 2009 19:20:01 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <D31AC5CD-8DA4-4460-B019-3FC1E646DBAA@math.washington.edu>
References: <200906092033.27437.binet@cern.ch>
	<200906101754.57824.binet@cern.ch>
	<D31AC5CD-8DA4-4460-B019-3FC1E646DBAA@math.washington.edu>
Message-ID: <200906101920.01749.binet@cern.ch>

hi,

On Wednesday 10 June 2009 18:02:56 Robert Bradshaw wrote:
> > hum... maybe I am not tackling this from the correct angle (perhaps
> > that a
> > simple buffer of strings filled while the last entry is not
> > the_empty_string
> > would do as I quick hack...)
>
> The way ipython does it is to require a new empty line to "leave" an
> indentation block--maybe you could do the same. (Or maybe you could
> even hook this in with ipython).

attached is my hack. feel free to steal it...
there are probably many improvements to be performed.

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

-------------- next part --------------
A non-text attachment was scrubbed...
Name: icython.py
Type: text/x-python
Size: 5707 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090610/2839a547/attachment.py 

From ondrej at certik.cz  Wed Jun 10 19:41:58 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 11:41:58 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
Message-ID: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>

On Wed, Jun 10, 2009 at 10:01 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 10, 2009, at 8:35 AM, Ondrej Certik wrote:
>
>> On Wed, Jun 10, 2009 at 12:13 AM, Robert
>> Bradshaw<robertwb at math.washington.edu> wrote:
>>> On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:
>>>
>>>> Hi,
>>>>
>>>> I have the following problem -- I have a package hermes2d, with the
>>>> structure:
>>>>
>>>> hermes2d/__init__.py
>>>> hermes2d/_hermes2d.pyx + .pxd
>>>> hermes2d/_forms.pyx
>>>>
>>>> and sometimes (depending on what the users tells cmake) I want to
>>>> build the .so libraries out of the tree and put their paths to
>>>> PYTHONPATH. Things just work as long as remove the file
>>>> hermes2d/__init__.py during the cython compilation. The reason is
>>>> that
>>>> if I do:
>>>>
>>>> cd hermes2d
>>>> cython _forms.pyx
>>>>
>>>> it produces _forms.c with lines like:
>>>>
>>>> ? __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
>>>> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
>>>> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
>>>> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
>>>> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
>>>> __LINE__; goto __pyx_L1_error;}
>>>>
>>>> Notice the import "hermes2d._hermes2d", which fails if
>>>> _hermes2d.so is
>>>> out of the tree (e..g somewhere else on the filesystem, even if it's
>>>> in the search path!). If, however I remove the __init__.py, then
>>>> cython generates:
>>>>
>>>> ? __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
>>>> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
>>>> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
>>>> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
>>>> __pyx_L1_error;}
>>>>
>>>> and everything just works, no matter where the file _hermes2d.so
>>>> actually is.
>>>>
>>>> Now my question is -- if I want those .so libraries not to be in the
>>>> tree, am I doing things the right way? E.g. I just have a line:
>>>>
>>>> from _forms import *
>>>>
>>>> in the hermes2d/forms.py
>>>>
>>>> and then people just do "from hermes2d import forms" and things
>>>> work,
>>>> no matter where the .so files are. But I need some way to force
>>>> cython
>>>> to just pretend there is no __init__.py in there. Why is cython
>>>> doing
>>>> this check at all? Does it have any advantages?
>>>
>>> One advantage I see is that you may have two modules named _hermes2d
>>> in different packages that it might have to distinguish. (OK, rare
>>> with that name, but it could happen with more common names). Also,
>>> this way the module path is fully qualified as one would expect.
>>
>> Ok. So how do you propose I should fix the problem, if I need those
>> .so libraries at different place than the rest of the python source
>> files?
>
> I don't know. A hack is to have your setup.py actually move the
> __init__.py temporarily--I'm sure there's a better longterm fix
> though. Just out of curiosity, why do you need to segregate the .so
> files?

Because we use cmake to build the project (C++ and cython) and cmake
allows to build the project out of the tree, e.g. it leaves the .cpp
and .py files in the source dir and creates a build dir, where it
stores all the .o and .so files. If the project is just C++, it is
then possible to use the build directory as is, e.g. you don't have to
install it and it still works (of course if you want, you can also
install it).

With Cython, it doesn't work, because the .so files don't find each other.

One solution (besides moving __init__.py files away at each build) is
to just create an option to cython, that will disable this feature (or
pretend the __init__.py files are not there).

I am not convinced it's worth the pain though yet. So you think one
should never need to segregate the .so file in a Python + C++ project?

Ondrej

From ellisonbg.net at gmail.com  Wed Jun 10 19:58:14 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 10 Jun 2009 10:58:14 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
Message-ID: <6ce0ac130906101058k5ebe87c5q39c47a5cbd7eb022@mail.gmail.com>

> With Cython, it doesn't work, because the .so files don't find each other.
>
> One solution (besides moving __init__.py files away at each build) is
> to just create an option to cython, that will disable this feature (or
> pretend the __init__.py files are not there).
>
> I am not convinced it's worth the pain though yet. So you think one
> should never need to segregate the .so file in a Python + C++ project?

I think this should function like the --inplace option to distutils.
Once built, the .so's/.dylibs should be moved into the source tree if
the build is inplace.  Either that, or the .py files should all be
moved over to the out of place build location.  I think trying to run
things in a fragmented manner like you are describing is confusing and
is going to cause unneeded pain.

Cheers,

Brian

> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ellisonbg.net at gmail.com  Wed Jun 10 20:17:18 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 10 Jun 2009 11:17:18 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE0C0.9020004@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A2FE0C0.9020004@student.matnat.uio.no>
Message-ID: <6ce0ac130906101117o6e642454mccb923c26517170d@mail.gmail.com>

Dag,

I quickly glanced through the proposal and have two big picture questions:

* What will this make possible that is currently not possible?

* What will this make easier that is currently really difficult?

Cheers,

Brian

On Wed, Jun 10, 2009 at 9:35 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> Thanks to chats with Kurt and Robert, I think I've managed to present my
>> thoughts from a different and more constructive angle.
>>
>> http://wiki.cython.org/enhancements/array
>>
>> There's some time constraints in the picture for various reasons (e.g.
>> Fortran GSoC development direction) so it would be great if this could
>> be resolved one way or the other before too long. Part of that will
>> likely happen on IRC.
>>
>> Feedback from anybody (especially users) very welcome.
>>
>
> Oh..: Please, let's discuss the high-level first (*if* something
> remotely looking like this is wanted at all) before discussing any of
> the lower-level stuff (syntax, implementation, feature-set etc.)
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From bpederse at gmail.com  Wed Jun 10 20:37:59 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 10 Jun 2009 11:37:59 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <guhfgd$pd5$1@ger.gmane.org>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
Message-ID: <e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>

On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
> Dag Sverre Seljebotn wrote:
>> Robert wrote:
>>> How to deal with Python's array.array directly - as with numpy.pxd
>>> is there a array.pxd ?
>>
>> Which Python version?
>
> mainly 2.6; and 2.3
>
>> Under Python 3 this should probably happen automatically, try:
>>
>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>
>> Not sure about Python 2.6+
>
> 2.6.2 didn't:
> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
> TypeError: 'array.array' does not have the buffer interface
>
> in Python "buffer(myarray)" also behaves strange
>
>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>> and filling in the Py_buffer struct).
>>
>> If somebody ends up doing this, please submit it for inclusion in
>> Python.
>
> I've put a array.pxd here:
> http://trac.cython.org/cython_trac/ticket/314

hi, what's the status on this? it seemed very useful.
if it needs tests, i could try to write some given some info on where
to start, what to
cover or maybe some cython-numpy tests to from which to crib?
thanks,
-brent


>
> --
> Cython direct interface to Python's array.array type (builtin module).
>
> ? ? * 1D contiguous data view
> ? ? * 2D views - contigious or [x,y] transposed/strided
> ? ? * tools for fast array creation, maximum C-speed and handiness
> ? ? * suitable as allround light weight auto-array within Cython
> code too
>
> ? ? See also: array_example.pyx and doc strings
> --
>
> Tests needed. just used it in 2.6 so far.
>
> Robert
>
>
> PS: For 2.3 a need arose for conditional compilation within the pxd.
> http://docs.cython.org/docs/language_basics.html#conditional-compilation
> doesn't seem to allow comparison against the python version so far? )
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Wed Jun 10 20:57:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 20:57:41 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
Message-ID: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>

Brian Granger wrote:
> Dag,
>
> I quickly glanced through the proposal and have two big picture questions:
>
> * What will this make possible that is currently not possible?

1) Efficient slices

2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
Koepke's project proposal)

3) Automatic contiguous copies in/out if a function is coded to work on
contiguous memory

Why can't this be done currently?

 * Cython contains no in-compiler support for NumPy, and so cannot know
how to create new underlying ndarray objects.

 * All optimizations need to hard-code semantics at compile-time. With
single-element indexing it seemed fair to assume the usual semantics of
zero-based indexing etc., but with slices tings get worse (which kind of
object is returned) and with arithmetic downright impossible (what does *
do again?)

That's not to say there's not other options:
1) We could hard-code support for NumPy only, and only allow ndarray and
not subclasses thereof.

2) We could invent some new protocol/syntax for defining compile-time
semantics for all relevant operations.

> * What will this make easier that is currently really difficult?

Well, nothing is really difficult. But passing, say, a PIL image into C
code and back again for processing (with, say, a contiguous C array) isn't
completely straightforward, but would be with this proposal.

Dag Sverre





From dagss at student.matnat.uio.no  Wed Jun 10 21:08:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 21:08:30 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
References: <guemg0$nc7$1@ger.gmane.org>
	<4A0AE4DF.20209@student.matnat.uio.no>	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
Message-ID: <4A3004AE.2030403@student.matnat.uio.no>

Brent Pedersen wrote:
> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert wrote:
>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>> is there a array.pxd ?
>>> Which Python version?
>> mainly 2.6; and 2.3
>>
>>> Under Python 3 this should probably happen automatically, try:
>>>
>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>
>>> Not sure about Python 2.6+
>> 2.6.2 didn't:
>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>     cdef object[float, ndim=1, mode="c"] b = pyarray
>> TypeError: 'array.array' does not have the buffer interface
>>
>> in Python "buffer(myarray)" also behaves strange
>>
>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>> and filling in the Py_buffer struct).
>>>
>>> If somebody ends up doing this, please submit it for inclusion in
>>> Python.
>> I've put a array.pxd here:
>> http://trac.cython.org/cython_trac/ticket/314
> 
> hi, what's the status on this? it seemed very useful.
> if it needs tests, i could try to write some given some info on where
> to start, what to
> cover or maybe some cython-numpy tests to from which to crib?
> thanks,
> -brent

Great!

The main issue with the patch as I see it is that it tries to hack on 
multi-dimensionality. That is very easily done simply by writing a 
subclass of array instead, and so doesn't belong in the pxd like this.

If you could just remove the multi-dimensional stuff from it I'd be 
happy to accept it. Accompanying tests are strongly preferred though.

There's a section on writing tests here:
http://wiki.cython.org/HackerGuide

I don't expect the numpy tests to be too useful (though they are in 
tests/run), just make sure the basics work with a couple of different 
datatypes. I'll be happy to suggest improvements if you make a first 
iteration anyway.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 10 21:19:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 21:19:24 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A3004AE.2030403@student.matnat.uio.no>
References: <guemg0$nc7$1@ger.gmane.org>	<4A0AE4DF.20209@student.matnat.uio.no>	<guhfgd$pd5$1@ger.gmane.org>	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
Message-ID: <4A30073C.5090106@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Brent Pedersen wrote:
>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert wrote:
>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>> is there a array.pxd ?
>>>> Which Python version?
>>> mainly 2.6; and 2.3
>>>
>>>> Under Python 3 this should probably happen automatically, try:
>>>>
>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>
>>>> Not sure about Python 2.6+
>>> 2.6.2 didn't:
>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>     cdef object[float, ndim=1, mode="c"] b = pyarray
>>> TypeError: 'array.array' does not have the buffer interface
>>>
>>> in Python "buffer(myarray)" also behaves strange
>>>
>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>> and filling in the Py_buffer struct).
>>>>
>>>> If somebody ends up doing this, please submit it for inclusion in
>>>> Python.
>>> I've put a array.pxd here:
>>> http://trac.cython.org/cython_trac/ticket/314
>> hi, what's the status on this? it seemed very useful.
>> if it needs tests, i could try to write some given some info on where
>> to start, what to
>> cover or maybe some cython-numpy tests to from which to crib?
>> thanks,
>> -brent
> 
> Great!
> 
> The main issue with the patch as I see it is that it tries to hack on 
> multi-dimensionality. That is very easily done simply by writing a 
> subclass of array instead, and so doesn't belong in the pxd like this.
> 
> If you could just remove the multi-dimensional stuff from it I'd be 
> happy to accept it. Accompanying tests are strongly preferred though.
> 
> There's a section on writing tests here:
> http://wiki.cython.org/HackerGuide
> 
> I don't expect the numpy tests to be too useful (though they are in 
> tests/run), just make sure the basics work with a couple of different 
> datatypes. I'll be happy to suggest improvements if you make a first 
> iteration anyway.
> 

Also it would be great if any patches could be created using Mercurial.

-- 
Dag Sverre

From stefan_ml at behnel.de  Wed Jun 10 08:14:51 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 10 Jun 2009 08:14:51 +0200
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <4A2E55A3.5040509@student.matnat.uio.no>
References: <4A2E55A3.5040509@student.matnat.uio.no>
Message-ID: <4A2F4F5B.4080000@behnel.de>


Dag Sverre Seljebotn wrote:
> I realize now that the recent discussion shouldn't be a discussion of 
> syntax as such, but rather the overall strategy for numerics in Cython.
> 
> I think I finally know how to present the issue that I've been circling 
> around during the spring reasonably clearly -- however I don't want to 
> start either a) a thread where I have a longwinded start which nobody 
> (understandably) bother to pick up, leaving us in the current state of 
> indecision, or b) one of those 200-post threads spanning three weeks 
> (which is too long and a lot of effort, summed up).
> 
> Can we try an IRC session? Do people have time? (Anybody who's 
> interested, but at lest Robert and Kurt)? (If so I'll try to pose the 
> question more properly on a single wiki screenful or so beforehand.)

I would be interested, although numerics isn't my primary focus. However,
I'll hardly have time to schedule an IRC meeting, so I'll leave it up to
you and see if I can make it somehow.

Anyway, Dag, thanks for driving these things.

Stefan


From dalcinl at gmail.com  Wed Jun 10 23:37:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 10 Jun 2009 18:37:06 -0300
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE0C0.9020004@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A2FE0C0.9020004@student.matnat.uio.no>
Message-ID: <e7ba66e40906101437r2cea33ebxc0b3a26d3f3c122f@mail.gmail.com>

On Wed, Jun 10, 2009 at 1:35 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>
> Oh..: Please, let's discuss the high-level first (*if* something
> remotely looking like this is wanted at all) before discussing any of
> the lower-level stuff (syntax, implementation, feature-set etc.)
>

Definitely +1 for me... It looks really good...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From bpederse at gmail.com  Thu Jun 11 00:23:17 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 10 Jun 2009 15:23:17 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A3004AE.2030403@student.matnat.uio.no>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
Message-ID: <e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>

On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Brent Pedersen wrote:
>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert wrote:
>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>> is there a array.pxd ?
>>>> Which Python version?
>>> mainly 2.6; and 2.3
>>>
>>>> Under Python 3 this should probably happen automatically, try:
>>>>
>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>
>>>> Not sure about Python 2.6+
>>> 2.6.2 didn't:
>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
>>> TypeError: 'array.array' does not have the buffer interface
>>>
>>> in Python "buffer(myarray)" also behaves strange
>>>
>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>> and filling in the Py_buffer struct).
>>>>
>>>> If somebody ends up doing this, please submit it for inclusion in
>>>> Python.
>>> I've put a array.pxd here:
>>> http://trac.cython.org/cython_trac/ticket/314
>>
>> hi, what's the status on this? it seemed very useful.
>> if it needs tests, i could try to write some given some info on where
>> to start, what to
>> cover or maybe some cython-numpy tests to from which to crib?
>> thanks,
>> -brent
>
> Great!
>
> The main issue with the patch as I see it is that it tries to hack on
> multi-dimensionality. That is very easily done simply by writing a
> subclass of array instead, and so doesn't belong in the pxd like this.
>
> If you could just remove the multi-dimensional stuff from it I'd be
> happy to accept it. Accompanying tests are strongly preferred though.
>
> There's a section on writing tests here:
> http://wiki.cython.org/HackerGuide
>
> I don't expect the numpy tests to be too useful (though they are in
> tests/run), just make sure the basics work with a couple of different
> datatypes. I'll be happy to suggest improvements if you make a first
> iteration anyway.
>

thanks for the pointers,
i have a start on this, but what should i do with the arrayarray.h?
there doesnt seem to be any cases where a header is included with cython itself.

-brent


> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From bpederse at gmail.com  Thu Jun 11 02:00:16 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 10 Jun 2009 17:00:16 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
	<e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
Message-ID: <e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>

On Wed, Jun 10, 2009 at 3:23 PM, Brent Pedersen<bpederse at gmail.com> wrote:
> On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Brent Pedersen wrote:
>>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>> Robert wrote:
>>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>>> is there a array.pxd ?
>>>>> Which Python version?
>>>> mainly 2.6; and 2.3
>>>>
>>>>> Under Python 3 this should probably happen automatically, try:
>>>>>
>>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>>
>>>>> Not sure about Python 2.6+
>>>> 2.6.2 didn't:
>>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
>>>> TypeError: 'array.array' does not have the buffer interface
>>>>
>>>> in Python "buffer(myarray)" also behaves strange
>>>>
>>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>>> and filling in the Py_buffer struct).
>>>>>
>>>>> If somebody ends up doing this, please submit it for inclusion in
>>>>> Python.
>>>> I've put a array.pxd here:
>>>> http://trac.cython.org/cython_trac/ticket/314
>>>
>>> hi, what's the status on this? it seemed very useful.
>>> if it needs tests, i could try to write some given some info on where
>>> to start, what to
>>> cover or maybe some cython-numpy tests to from which to crib?
>>> thanks,
>>> -brent
>>
>> Great!
>>
>> The main issue with the patch as I see it is that it tries to hack on
>> multi-dimensionality. That is very easily done simply by writing a
>> subclass of array instead, and so doesn't belong in the pxd like this.
>>
>> If you could just remove the multi-dimensional stuff from it I'd be
>> happy to accept it. Accompanying tests are strongly preferred though.
>>
>> There's a section on writing tests here:
>> http://wiki.cython.org/HackerGuide
>>
>> I don't expect the numpy tests to be too useful (though they are in
>> tests/run), just make sure the basics work with a couple of different
>> datatypes. I'll be happy to suggest improvements if you make a first
>> iteration anyway.
>>
>
> thanks for the pointers,
> i have a start on this, but what should i do with the arrayarray.h?
> there doesnt seem to be any cases where a header is included with cython itself.
>
> -brent
>

here's a first iteration with tests for some feedback.
i'm still not sure what to do with the .h file. currently
the tests have to be run as:
INCLUDE=/path/to/cython-devel/Cython/Includes/ python runtests.py pyarray
-------------- next part --------------
A non-text attachment was scrubbed...
Name: array.patch
Type: text/x-diff
Size: 14963 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090610/31de2748/attachment-0001.bin 

From ondrej at certik.cz  Thu Jun 11 02:45:20 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 18:45:20 -0600
Subject: [Cython] problem with pure python mode
In-Reply-To: <12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
	<12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
Message-ID: <85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>

On Wed, Apr 29, 2009 at 4:00 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Apr 29, 2009, at 2:49 PM, Ondrej Certik wrote:
>
>>>>
>>>> Not sure--have you tried without the locals decorator in the Python
>>>> source? I hope this isn't a regression.
>>>
>>> Yes, as you can see, the locals is commented out. I'll try to bisect
>>> it to see which commit broke it.
>>
>> Ok, so it never worked. I am a bit confused, because I thought the
>> pure python mode worked fine for me before.
>>
>> In the current cython --- is there any way to keep a pure python code,
>> and create an accompanying pxd file that would annotate the cdef
>> functions and classes?
>
> This should work, that's how I implemented this a long time ago. If
> it's not working now it's because something broke (or, I'm not
> promising that it's completely bug-free, but it should be good--we
> use it to bootstrap Cython itself).

So I made it to work by observing how things are done in Cython itself.

If you do:

git clone git://github.com/certik/cython-test.git
cd cython-test
cython fact.pyx

then it produces nicely optimized fact.c file. However, if you do:

cython fact.py

it produces a fact.c, that compiles, but it is not optimized --- the
"i" and "r" variables are python objects, not ints, even though my
fact.pxd file contains:

import cython

@cython.locals(i=cython.int, r=cython.int)
cdef int factorial(int n)


Do you see anything wrong with my files? Is this supposed to work?

I think it is not implemented yet, because when I looked at the cython
sources, it doesn't work there either. This is the relevant part of
Scanning.pxd:

    @cython.locals(current_level=cython.long, new_level=cython.long)
    cpdef indentation_action(self, text)

and here is the declaration in the generated Scanning.c:

  PyObject *__pyx_v_new_level;


And consider for example the following chunk:

  /* "/home/ondrej/repos/cython-devel/Cython/Compiler/Scanning.py":398
 *         new_level = len(text)
 *         #print "Changing indent level from", current_level, "to",
new_level ###
 *         if new_level == current_level:             # <<<<<<<<<<<<<<
 *             return
 *         elif new_level > current_level:
 */
  __pyx_t_4 = PyObject_RichCompare(__pyx_v_new_level,
__pyx_v_current_level, Py_EQ); if (unlikely(!__pyx_t_4))
{__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno =
__LINE__; goto __pyx_L1_error;}
  __Pyx_GOTREF(__pyx_t_4);
  __pyx_t_3 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_3
< 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno
= __LINE__; goto __pyx_L1_error;}
  __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
  if (__pyx_t_3) {


both the new_level and current_level are declared as long, but above
you can see that Cython is calling lots of Python C/API methods...



Ondrej

From ondrej at certik.cz  Thu Jun 11 03:06:33 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 19:06:33 -0600
Subject: [Cython] -unstable mergeback?
In-Reply-To: <204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
References: <4A14291F.2090806@student.matnat.uio.no>
	<C9D49322-800E-4409-A32D-299E10E64525@math.washington.edu>
	<4A20DED9.5010506@behnel.de>
	<204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
Message-ID: <85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>

On Sat, May 30, 2009 at 9:01 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On May 30, 2009, at 12:23 AM, Stefan Behnel wrote:
>
>> Hi,
>>
>> sorry, was away for a while.
>>
>> Robert Bradshaw schrieb:
>>> On May 20, 2009, at 9:00 AM, Dag Sverre Seljebotn wrote:
>>>
>>>> It seems that there's now a danger of -unstable becoming the main
>>>> development branch; I also think the dangerous things are completed
>>>> myself. If so, I think it should replace the current -devel. This
>>>> will
>>>> make a few tickets blockers for the next release though.
>>>
>>> If -unstable has stabilized, I think this makes sense. I envisioned -
>>> unstable as being for truly unstable things (e.g. closures, temp
>>> allocation rewrites) rather than the main development branch.
>>
>> Same from my side. There are a couple of regressions (failing
>> tests) in the
>> current -unstable that I couldn't easily fix right away. They need
>> some
>> more work before merging it back into -devel. BTW, I'd actually go the
>> opposite way once that's done: merge -devel up into -unstable, drop
>> -devel
>> and rename -unstable.
>
> I'd like to see if there's any major fallout trying to compile sage
> with -unstable before we make the switch though, but assuming things
> aren't catastrophic, let's do that.
>
>> Regarding the -closures branch, IMHO it now works well enough to
>> merge it
>> into cython-unstable. I'd still be happy if others could give it a try
>> first, maybe just run it on some Python code that uses nested
>> functions.
>
> I found an odd bug while demoing closures, so they're not ready yet,
> but I think they're close--certainly ready for other people to start
> trying them out. This is a feature that I would like people to really
> pound on before release.

The closure branch didn't compile our project (Compiler crash in GilCheck):

$ make
[ 74%] Built target hermes2d-real
[ 75%] Cython source

Error converting Pyrex file to C:
------------------------------------------------------------
...
    ctypedef scalar (*BiFormFnVol)(RealFunction *fu, RealFunction *fv,
            RefMap *ru, RefMap *rv)

    ctypedef scalar (*LiFormFnVol)(RealFunction *fv, RefMap *rv)

    cdef BiFormFnVol BF_SYM = <BiFormFnVol> 1
                             ^
------------------------------------------------------------

/home/ondrej/repos/hermes2d/python/hermes2d/_hermes2d.pxd:110:30:
Compiler crash in GilCheck

ModuleNode.body = StatListNode(_hermes2d.pxd:1:0)
StatListNode.stats[5] = CDefExternNode(_hermes2d.pxd:31:5,
    include_file = 'hermes2d.h')
CDefExternNode.body = StatListNode(_hermes2d.pxd:34:4)
StatListNode.stats[5] = SingleAssignmentNode(_hermes2d.pxd:110:9)
SingleAssignmentNode.rhs = TypecastNode(_hermes2d.pxd:110:30)
File 'ExprNodes.py', line 4006, in gil_check: TypecastNode(_hermes2d.pxd:110:30)

Compiler crash traceback from this point on:
  File "/home/ondrej/usr/lib/python/Cython/Compiler/ExprNodes.py",
line 4007, in gil_check
    if self.type.is_pyobject and self.is_temp:
AttributeError: 'NoneType' object has no attribute 'is_pyobject'
make[2]: *** [python/hermes2d/_forms.cpp] Error 1
make[1]: *** [python/hermes2d/CMakeFiles/_forms.dir/all] Error 2
make: *** [all] Error 2



But otherwise this works:


cdef int a(int x):
    def b(int n):
        cdef int r
        r = x + n
        return r
    return b(3)

Just curious question -- if I apply this patch:

$ git diff
diff --git a/fact.pyx b/fact.pyx
index 0cadb8d..518d3ba 100644
--- a/fact.pyx
+++ b/fact.pyx
@@ -7,7 +7,7 @@ cdef int factorial(int n):
     return r

 cdef int a(int x):
-    def b(int n):
+    cdef b(int n):
         cdef int r
         r = x + n
         return r


cython will complain:

Error converting Pyrex file to C:
------------------------------------------------------------
...
    for i in range(1, n + 1):
        r *= i
    return r

cdef int a(int x):
    cdef b(int n):
                ^
------------------------------------------------------------

/home/ondrej/repos/cython-test/fact.pyx:10:17: C function definition
not allowed here



I guess there is no way to make this work in C, right? Because I could
then pass the pointer to the function "b" to some external C program
and that function still needs to access my outer scope.


Ondrej

From robertwb at math.washington.edu  Thu Jun 11 09:28:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Jun 2009 00:28:07 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
Message-ID: <004A836B-BC00-4AA8-9DAA-DA99A737B526@math.washington.edu>


On Jun 10, 2009, at 10:41 AM, Ondrej Certik wrote:

[...]

>>> Ok. So how do you propose I should fix the problem, if I need those
>>> .so libraries at different place than the rest of the python source
>>> files?
>>
>> I don't know. A hack is to have your setup.py actually move the
>> __init__.py temporarily--I'm sure there's a better longterm fix
>> though. Just out of curiosity, why do you need to segregate the .so
>> files?
>
> Because we use cmake to build the project (C++ and cython) and cmake
> allows to build the project out of the tree, e.g. it leaves the .cpp
> and .py files in the source dir and creates a build dir, where it
> stores all the .o and .so files. If the project is just C++, it is
> then possible to use the build directory as is, e.g. you don't have to
> install it and it still works (of course if you want, you can also
> install it).
>
> With Cython, it doesn't work, because the .so files don't find each  
> other.
>
> One solution (besides moving __init__.py files away at each build) is
> to just create an option to cython, that will disable this feature (or
> pretend the __init__.py files are not there).
>
> I am not convinced it's worth the pain though yet. So you think one
> should never need to segregate the .so file in a Python + C++ project?

I don't see a reason why one should need to, but I don't claim  
there's never a valid reason either. More to the point, I don't see a  
quick or good workaround or sufficient motivation for a more complex/ 
disruptive one (but if you do, go ahead and look at it more).

- Robert



From robertwb at math.washington.edu  Thu Jun 11 09:34:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Jun 2009 00:34:32 -0700
Subject: [Cython] -unstable mergeback?
In-Reply-To: <85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
References: <4A14291F.2090806@student.matnat.uio.no>
	<C9D49322-800E-4409-A32D-299E10E64525@math.washington.edu>
	<4A20DED9.5010506@behnel.de>
	<204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
	<85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
Message-ID: <96690D04-7F5B-45DA-A7ED-535D6B9CEF5E@math.washington.edu>

On Jun 10, 2009, at 6:06 PM, Ondrej Certik wrote:

> On Sat, May 30, 2009 at 9:01 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On May 30, 2009, at 12:23 AM, Stefan Behnel wrote:
>>
>>> Hi,
>>>
>>> sorry, was away for a while.
>>>
>>> Robert Bradshaw schrieb:
>>>> On May 20, 2009, at 9:00 AM, Dag Sverre Seljebotn wrote:
>>>>
>>>>> It seems that there's now a danger of -unstable becoming the main
>>>>> development branch; I also think the dangerous things are  
>>>>> completed
>>>>> myself. If so, I think it should replace the current -devel. This
>>>>> will
>>>>> make a few tickets blockers for the next release though.
>>>>
>>>> If -unstable has stabilized, I think this makes sense. I  
>>>> envisioned -
>>>> unstable as being for truly unstable things (e.g. closures, temp
>>>> allocation rewrites) rather than the main development branch.
>>>
>>> Same from my side. There are a couple of regressions (failing
>>> tests) in the
>>> current -unstable that I couldn't easily fix right away. They need
>>> some
>>> more work before merging it back into -devel. BTW, I'd actually  
>>> go the
>>> opposite way once that's done: merge -devel up into -unstable, drop
>>> -devel
>>> and rename -unstable.
>>
>> I'd like to see if there's any major fallout trying to compile sage
>> with -unstable before we make the switch though, but assuming things
>> aren't catastrophic, let's do that.
>>
>>> Regarding the -closures branch, IMHO it now works well enough to
>>> merge it
>>> into cython-unstable. I'd still be happy if others could give it  
>>> a try
>>> first, maybe just run it on some Python code that uses nested
>>> functions.
>>
>> I found an odd bug while demoing closures, so they're not ready yet,
>> but I think they're close--certainly ready for other people to start
>> trying them out. This is a feature that I would like people to really
>> pound on before release.
>
> The closure branch didn't compile our project (Compiler crash in  
> GilCheck):

Thanks for the bug report.

>     for i in range(1, n + 1):
>         r *= i
>     return r
>
> cdef int a(int x):
>     cdef b(int n):
>                 ^
> ------------------------------------------------------------
>
> /home/ondrej/repos/cython-test/fact.pyx:10:17: C function definition
> not allowed here
>
> I guess there is no way to make this work in C, right? Because I could
> then pass the pointer to the function "b" to some external C program
> and that function still needs to access my outer scope.

A closure is more than just an function, it's a function with context  
(i.e. the x from the outer scope can be accessed). It's unclear what  
type "b" would have as a cdef function--wouldn't be impossible to do  
some mangling to make this appear to work in many cases (e.g. it  
could be a struct with the context as one field and function pointer  
in another), but it wouldn't be a (PyObject*)(int) for sure. C just  
doesn't have the concept of "bound" methods.

- Robert



From stefan_ml at behnel.de  Thu Jun 11 09:59:51 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 09:59:51 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
Message-ID: <4A30B977.9090306@behnel.de>


Ondrej Certik wrote:
> we use cmake to build the project (C++ and cython) and cmake
> allows to build the project out of the tree, e.g. it leaves the .cpp
> and .py files in the source dir and creates a build dir, where it
> stores all the .o and .so files.

Uhm, so the real problem is that cmake fails to take the source directory
structure into account when copying build artefacts over to the build
directory?

How is that something to 'fix' in Cython?


> If the project is just C++, it is
> then possible to use the build directory as is, e.g. you don't have to
> install it and it still works (of course if you want, you can also
> install it).

For testing, I build my extensions using distutils' "build_ext -i", which
creates the .so files inside the source tree (and leaves everything else
out of sight).

Stefan


From dagss at student.matnat.uio.no  Thu Jun 11 11:01:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 11:01:50 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>
References: <guemg0$nc7$1@ger.gmane.org>
	<4A0AE4DF.20209@student.matnat.uio.no>	<guhfgd$pd5$1@ger.gmane.org>	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>	<4A3004AE.2030403@student.matnat.uio.no>	<e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
	<e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>
Message-ID: <4A30C7FE.7080107@student.matnat.uio.no>

Brent Pedersen wrote:
> On Wed, Jun 10, 2009 at 3:23 PM, Brent Pedersen<bpederse at gmail.com> wrote:
>> On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> Brent Pedersen wrote:
>>>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>>>> Dag Sverre Seljebotn wrote:
>>>>>> Robert wrote:
>>>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>>>> is there a array.pxd ?
>>>>>> Which Python version?
>>>>> mainly 2.6; and 2.3
>>>>>
>>>>>> Under Python 3 this should probably happen automatically, try:
>>>>>>
>>>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>>>
>>>>>> Not sure about Python 2.6+
>>>>> 2.6.2 didn't:
>>>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>>>     cdef object[float, ndim=1, mode="c"] b = pyarray
>>>>> TypeError: 'array.array' does not have the buffer interface
>>>>>
>>>>> in Python "buffer(myarray)" also behaves strange
>>>>>
>>>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>>>> and filling in the Py_buffer struct).
>>>>>>
>>>>>> If somebody ends up doing this, please submit it for inclusion in
>>>>>> Python.
>>>>> I've put a array.pxd here:
>>>>> http://trac.cython.org/cython_trac/ticket/314
>>>> hi, what's the status on this? it seemed very useful.
>>>> if it needs tests, i could try to write some given some info on where
>>>> to start, what to
>>>> cover or maybe some cython-numpy tests to from which to crib?
>>>> thanks,
>>>> -brent
>>> Great!
>>>
>>> The main issue with the patch as I see it is that it tries to hack on
>>> multi-dimensionality. That is very easily done simply by writing a
>>> subclass of array instead, and so doesn't belong in the pxd like this.
>>>
>>> If you could just remove the multi-dimensional stuff from it I'd be
>>> happy to accept it. Accompanying tests are strongly preferred though.
>>>
>>> There's a section on writing tests here:
>>> http://wiki.cython.org/HackerGuide
>>>
>>> I don't expect the numpy tests to be too useful (though they are in
>>> tests/run), just make sure the basics work with a couple of different
>>> datatypes. I'll be happy to suggest improvements if you make a first
>>> iteration anyway.
>>>
>> thanks for the pointers,
>> i have a start on this, but what should i do with the arrayarray.h?
>> there doesnt seem to be any cases where a header is included with cython itself.
>>
>> -brent
>>
> 
> here's a first iteration with tests for some feedback.
> i'm still not sure what to do with the .h file. currently

Hmm. If it wasn't for the #ifdefs it could have been ported to Cython 
code and inserted in the pxd I think.

Anyway, there's not much to do -- but you can have a look at runtests.py 
to make it add Cython/Includes to the C include path automatically?

I'm a bit uncertain about this though. Is the faster creation etc. 
really necesarry? It seems like adding a lot of complexity (basically 
one needs to keep up with all Python versions and verify that the header 
still works for each new Python release). So I'm not quite sure about 
it...thoughts? But if it is deemed important I think it can go in.

Tests seems fine -- I only glanced superficially, I just see that they 
are there and trust the submitter of the patch to know the subject 
matter better than me.

Finally, to properly support the PEP 3118, one should make sure that 
memory is not deallocated between a call to getbuffer and release. I.e. 
when the array is resized, one should not free the memory, but 
releasebuffer should do that.

I realize that this is likely impossible to do though (the array can be 
resized through Python calls, right?) so we'll just have to drop that 
and make a big warning not to resize while a buffer is acquired.

-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 16:45:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 16:45:50 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE018.5060902@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
Message-ID: <4A31189E.6000104@behnel.de>


Dag Sverre Seljebotn wrote:
> Thanks to chats with Kurt and Robert, I think I've managed to present my 
> thoughts from a different and more constructive angle.
> 
> http://wiki.cython.org/enhancements/array
> 
> There's some time constraints in the picture for various reasons (e.g. 
> Fortran GSoC development direction) so it would be great if this could 
> be resolved one way or the other before too long.

I'm mostly happy with the proposal. IMHO, this would also fit many use
cases of ticket 153

http://trac.cython.org/cython_trac/ticket/153

- I wonder why the "custom reference counting" is necessary, though. I
assume that the main advantage of doing it in Cython instead of CPython is
that we could allow memory allocation and ref-counting in nogil blocks, but
that would require us to add some other kind of synchronisation for arrays
that pass function barriers. I don't think that's worth it.

I doubt that many people would be surprised by getting a real Python object
when they use this feature, so why not just implement Cython arrays as
PyVarObjects?

- Another point is the "inout" modifier in the one example. Would that only
apply to C++ or would the semantics be the same in C? And: is it a general
modifier that would also apply to scalar variables (and how), or only to
arrays? What about pointers?

If it only applies to arrays, maybe putting it inside the array declaration
(e.g. as a kw arg) makes more sense?

- Should this
"""
Arithmetic (a + b turned into a loop on the arrays a and b)
"""

read

"""
Arithmetic (a + b returns a concatenated copy of a and b)
"""
?

Stefan

From bpederse at gmail.com  Thu Jun 11 16:48:36 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Thu, 11 Jun 2009 07:48:36 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A30C7FE.7080107@student.matnat.uio.no>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
	<e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
	<e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>
	<4A30C7FE.7080107@student.matnat.uio.no>
Message-ID: <e183a99d0906110748r71d012e8q4d4ba306536d964a@mail.gmail.com>

On Thu, Jun 11, 2009 at 2:01 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Brent Pedersen wrote:
>> On Wed, Jun 10, 2009 at 3:23 PM, Brent Pedersen<bpederse at gmail.com> wrote:
>>> On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> Brent Pedersen wrote:
>>>>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>>>>> Dag Sverre Seljebotn wrote:
>>>>>>> Robert wrote:
>>>>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>>>>> is there a array.pxd ?
>>>>>>> Which Python version?
>>>>>> mainly 2.6; and 2.3
>>>>>>
>>>>>>> Under Python 3 this should probably happen automatically, try:
>>>>>>>
>>>>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>>>>
>>>>>>> Not sure about Python 2.6+
>>>>>> 2.6.2 didn't:
>>>>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>>>> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
>>>>>> TypeError: 'array.array' does not have the buffer interface
>>>>>>
>>>>>> in Python "buffer(myarray)" also behaves strange
>>>>>>
>>>>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>>>>> and filling in the Py_buffer struct).
>>>>>>>
>>>>>>> If somebody ends up doing this, please submit it for inclusion in
>>>>>>> Python.
>>>>>> I've put a array.pxd here:
>>>>>> http://trac.cython.org/cython_trac/ticket/314
>>>>> hi, what's the status on this? it seemed very useful.
>>>>> if it needs tests, i could try to write some given some info on where
>>>>> to start, what to
>>>>> cover or maybe some cython-numpy tests to from which to crib?
>>>>> thanks,
>>>>> -brent
>>>> Great!
>>>>
>>>> The main issue with the patch as I see it is that it tries to hack on
>>>> multi-dimensionality. That is very easily done simply by writing a
>>>> subclass of array instead, and so doesn't belong in the pxd like this.
>>>>
>>>> If you could just remove the multi-dimensional stuff from it I'd be
>>>> happy to accept it. Accompanying tests are strongly preferred though.
>>>>
>>>> There's a section on writing tests here:
>>>> http://wiki.cython.org/HackerGuide
>>>>
>>>> I don't expect the numpy tests to be too useful (though they are in
>>>> tests/run), just make sure the basics work with a couple of different
>>>> datatypes. I'll be happy to suggest improvements if you make a first
>>>> iteration anyway.
>>>>
>>> thanks for the pointers,
>>> i have a start on this, but what should i do with the arrayarray.h?
>>> there doesnt seem to be any cases where a header is included with cython itself.
>>>
>>> -brent
>>>
>>
>> here's a first iteration with tests for some feedback.
>> i'm still not sure what to do with the .h file. currently
>
> Hmm. If it wasn't for the #ifdefs it could have been ported to Cython
> code and inserted in the pxd I think.
>
> Anyway, there's not much to do -- but you can have a look at runtests.py
> to make it add Cython/Includes to the C include path automatically?

i updated the patch to do that, but then what about when running outside
of tests? perhaps this is better as its own python module? robert?
i'm attaching updated patch here which adjusts the runtests.py hopefully
someone can add to the tests or suggest more. they're surely not complete
as i have only a superficial understanding of this.

-brent

>
> I'm a bit uncertain about this though. Is the faster creation etc.
> really necesarry? It seems like adding a lot of complexity (basically
> one needs to keep up with all Python versions and verify that the header
> still works for each new Python release). So I'm not quite sure about
> it...thoughts? But if it is deemed important I think it can go in.
>
> Tests seems fine -- I only glanced superficially, I just see that they
> are there and trust the submitter of the patch to know the subject
> matter better than me.
>
> Finally, to properly support the PEP 3118, one should make sure that
> memory is not deallocated between a call to getbuffer and release. I.e.
> when the array is resized, one should not free the memory, but
> releasebuffer should do that.
>
> I realize that this is likely impossible to do though (the array can be
> resized through Python calls, right?) so we'll just have to drop that
> and make a big warning not to resize while a buffer is acquired.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: array.export.patch
Type: text/x-patch
Size: 16652 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090611/58272a94/attachment-0001.bin 

From ondrej at certik.cz  Thu Jun 11 17:23:06 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 09:23:06 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A30B977.9090306@behnel.de>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
Message-ID: <85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>

On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Ondrej Certik wrote:
>> we use cmake to build the project (C++ and cython) and cmake
>> allows to build the project out of the tree, e.g. it leaves the .cpp
>> and .py files in the source dir and creates a build dir, where it
>> stores all the .o and .so files.
>
> Uhm, so the real problem is that cmake fails to take the source directory
> structure into account when copying build artefacts over to the build
> directory?

I think cmake is doing it right, it only builds binary things in the
binary directory and leaves all the sources (be it python or C++) in
the source directory.

>
> How is that something to 'fix' in Cython?

Consider this cython statement:

from hermes2d._hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
        int_grad_u_grad_v, int_v, malloc

This is essentially converted to a statement

py_module = PyImport_Import("hermes2d._hermes2d");      // Yes, I know
it's actually PyString in there and this is handled automatically by
__Pyx_ImportModule

Which is correct. Consider however this statement:

from _hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
        int_grad_u_grad_v, int_v, malloc

which is converted to:

py_module = PyImport_Import("hermes2d._hermes2d");

which I think is not right, because Cython changed my import path (and
what is worse, it depends on the positions of the __init__.py files in
my directory). All I am asking is if there is a really good reason,
why Cython mangles the import paths, why it doesn't leave it to the
user. Here is the documentation for PyImport_Import:

http://docs.python.org/c-api/import.html#PyImport_ImportModule

it says that "Changed in version 2.6: always use absolute imports".
What does it mean? Does it mean that one should always pass absolute
imports in it? If that is the case, then that would be a reason why
Cython does it. But I created a simple hack to Cython, which leaves
the paths as they are read from the .pyx file and it works for me in
python2.6 and does exactly what I want. I still need to polish the
patch, as it fails in some other cases, but essentially it's just a
change in search_include_directories() and couple other places.


>
>
>> If the project is just C++, it is
>> then possible to use the build directory as is, e.g. you don't have to
>> install it and it still works (of course if you want, you can also
>> install it).
>
> For testing, I build my extensions using distutils' "build_ext -i", which
> creates the .so files inside the source tree (and leaves everything else
> out of sight).

Right, I use the same approach with cmake (e.g. building in the tree)
and that works fine.

Some of my colleagues however like to work out of the tree (which also
has its advantages), so I am investigating if that is possible with
Cython (it works just fine with C++, where it links all the libraries
in the correct places, and when it installs, it changes the library
paths in the binaries).

Ondrej

From dagss at student.matnat.uio.no  Thu Jun 11 17:33:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 17:33:15 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31189E.6000104@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31189E.6000104@behnel.de>
Message-ID: <4A3123BB.5080603@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>   
>> Thanks to chats with Kurt and Robert, I think I've managed to present my 
>> thoughts from a different and more constructive angle.
>>
>> http://wiki.cython.org/enhancements/array
>>
>> There's some time constraints in the picture for various reasons (e.g. 
>> Fortran GSoC development direction) so it would be great if this could 
>> be resolved one way or the other before too long.
>>     
>
> I'm mostly happy with the proposal. IMHO, this would also fit many use
> cases of ticket 153
>   
Nice, and thanks for the feedback. I was mostly worried that this would 
be seen as too special-purpose for inclusion in mainline Cython.

I'll start with what I consider most important here:

> - Should this
> """
> Arithmetic (a + b turned into a loop on the arrays a and b)
> """
>
> read
>
> """
> Arithmetic (a + b returns a concatenated copy of a and b)
> """
> ?
>   
Well, that would upset all us numerical users a lot :-)

Doing arithmetic componentwise is one of the big ideas here. If + means 
concatenation you gain perhaps a convenient array type but it becomes 
much, *much* less useful for numerical purposes. I can see why one would 
argue against it though (not found in Python, array.array does 
concatenation etc.) so this is where I can only state what I'd 
personally find really useful. I feel pragmatism should win this one but 
you've got purism on your side.

PEP 3118 doesn't allow resizing buffers, which makes concatenation less 
natural to me. Though we could always return a new copy...

Anyway, say we want to make the usecases of both groups possible. Then 
the choice seems clear; if we use + for addition, one can make use of 
the full set +,-,/,*,** on arrays, while concatenation, being only one 
operation, could be moved into "cat(arr1, arr2)" or something. Using + 
for concatenation makes -,/,*,** useless...

I'm convinced that + would be used *a lot* more for componentwise 
addition than for concatenation.


> http://trac.cython.org/cython_trac/ticket/153
>
> - I wonder why the "custom reference counting" is necessary, though. I
> assume that the main advantage of doing it in Cython instead of CPython is
> that we could allow memory allocation and ref-counting in nogil blocks, but
> that would require us to add some other kind of synchronisation for arrays
> that pass function barriers. I don't think that's worth it.
>
> I doubt that many people would be surprised by getting a real Python object
> when they use this feature, so why not just implement Cython arrays as
> PyVarObjects?
>   
Yes, I've been thinking about this as well, and you are right that the 
proposal isn't quite there in that area.

I'll need to think more before I write more about this. Suffice to say 
now that there's many layers:

1) Memory container object. Definitely Python object.
2) Typed buffer acquisition, likely containing a single refcounted 
Py_buffer.
3) The actual variables, which will be structs with unpacked information 
from the Py_buffer.

Although I would like multithreading to be easier for numerical 
programming, here's an orthogonal solution: A new type of "nogil" 
section which releases the GIL but temporarily reacquires it on Python 
operations. On most code this would be slower, but it allows writing 
mostly nogil code but more easily take a slice or reassign an array here 
and there.

I need to reread that post on what issues Greg had with "with gil" though.

> - Another point is the "inout" modifier in the one example. Would that only
> apply to C++ or would the semantics be the same in C? And: is it a general
> modifier that would also apply to scalar variables (and how), or only to
> arrays? What about pointers?
>
> If it only applies to arrays, maybe putting it inside the array declaration
> (e.g. as a kw arg) makes more sense?
>   
Consider this:

cdef int[:,:] matrix
cdef func(inout matrix arg):
    cdef matrix var
    ...

So I'd like to avoid putting it in the brackets.

It comes from Fortran; there's no parallell in C or C++. I think it only 
applies to arrays, and it is necesarry to avoid unecessarry copying. It 
means that the contents of the array is copied into a new, contiguous 
array when entering the function, and then back again afterwards. This 
is just an instruction about what should happen to the *contents* of the 
array, so it's not in breach of Python semantics, and specifically would 
not make sense for scalars, and I think not for pointers either.


Dag Sverre


From dagss at student.matnat.uio.no  Thu Jun 11 17:37:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 17:37:30 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3123BB.5080603@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>
	<4A3123BB.5080603@student.matnat.uio.no>
Message-ID: <4A3124BA.70303@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:

(Re inout modifier:)
> Consider this:
>
> cdef int[:,:] matrix
> cdef func(inout matrix arg):
>     cdef matrix var
>     ...
>
> So I'd like to avoid putting it in the brackets.
>   
I made a quite critical typo there -- it should be

ctypedef int[:,:] matrix

That should make it more understandable...

Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 17:44:59 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 17:44:59 +0200
Subject: [Cython] -unstable mergeback?
In-Reply-To: <85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
References: <4A14291F.2090806@student.matnat.uio.no>	<C9D49322-800E-4409-A32D-299E10E64525@math.washington.edu>	<4A20DED9.5010506@behnel.de>	<204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
	<85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
Message-ID: <4A31267B.7040105@behnel.de>

Hi,

Ondrej Certik wrote:
> The closure branch didn't compile our project (Compiler crash in GilCheck):
> 
> $ make
> [ 74%] Built target hermes2d-real
> [ 75%] Cython source
> 
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>     ctypedef scalar (*BiFormFnVol)(RealFunction *fu, RealFunction *fv,
>             RefMap *ru, RefMap *rv)
> 
>     ctypedef scalar (*LiFormFnVol)(RealFunction *fv, RefMap *rv)
> 
>     cdef BiFormFnVol BF_SYM = <BiFormFnVol> 1
>                              ^
> ------------------------------------------------------------
> 
> /home/ondrej/repos/hermes2d/python/hermes2d/_hermes2d.pxd:110:30:
> Compiler crash in GilCheck
> 
> ModuleNode.body = StatListNode(_hermes2d.pxd:1:0)
> StatListNode.stats[5] = CDefExternNode(_hermes2d.pxd:31:5,
>     include_file = 'hermes2d.h')
> CDefExternNode.body = StatListNode(_hermes2d.pxd:34:4)
> StatListNode.stats[5] = SingleAssignmentNode(_hermes2d.pxd:110:9)
> SingleAssignmentNode.rhs = TypecastNode(_hermes2d.pxd:110:30)
> File 'ExprNodes.py', line 4006, in gil_check: TypecastNode(_hermes2d.pxd:110:30)
> 
> Compiler crash traceback from this point on:
>   File "/home/ondrej/usr/lib/python/Cython/Compiler/ExprNodes.py",
> line 4007, in gil_check
>     if self.type.is_pyobject and self.is_temp:
> AttributeError: 'NoneType' object has no attribute 'is_pyobject'

Thanks for the report (and for testing!). This isn't related to closures,
though. I bet this also fails in cython-unstable, due to the gil-check
refactoring.

http://trac.cython.org/cython_trac/ticket/329

Stefan

From stefan_ml at behnel.de  Thu Jun 11 18:05:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 18:05:02 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3123BB.5080603@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>
	<4A3123BB.5080603@student.matnat.uio.no>
Message-ID: <4A312B2E.5020602@behnel.de>


Dag Sverre Seljebotn wrote:
> I'll start with what I consider most important here:
> 
>> - Should this
>> """
>> Arithmetic (a + b turned into a loop on the arrays a and b)
>> """
>>
>> read
>>
>> """
>> Arithmetic (a + b returns a concatenated copy of a and b)
>> """
>> ?
>>   
> Well, that would upset all us numerical users a lot :-)
> 
> Doing arithmetic componentwise is one of the big ideas here. If + means 
> concatenation you gain perhaps a convenient array type but it becomes 
> much, *much* less useful for numerical purposes. I can see why one would 
> argue against it though (not found in Python, array.array does 
> concatenation etc.) so this is where I can only state what I'd 
> personally find really useful. I feel pragmatism should win this one but 
> you've got purism on your side.
> 
> PEP 3118 doesn't allow resizing buffers, which makes concatenation less 
> natural to me. Though we could always return a new copy...
> 
> Anyway, say we want to make the usecases of both groups possible. Then 
> the choice seems clear; if we use + for addition, one can make use of 
> the full set +,-,/,*,** on arrays, while concatenation, being only one 
> operation, could be moved into "cat(arr1, arr2)" or something. Using + 
> for concatenation makes -,/,*,** useless...

Tuples support '+' and '*'. My immediate intuition is that an array type
should behave exactly like a (mutable, but fixed-size) tuple, except where
C dictates otherwise.

> I'm convinced that + would be used *a lot* more for componentwise
> addition than for concatenation.

I agree - at least in numerics. But if you want a mathematical vector type,
that's a different thing than a memory managed C array IMHO.

BTW, didn't we have this discussion twice already? That shows that at least
my intuition keeps clearly dictating otherwise, each time we get here.

Stefan


From ellisonbg.net at gmail.com  Thu Jun 11 18:23:14 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 11 Jun 2009 09:23:14 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
Message-ID: <6ce0ac130906110923y8d00f04i1ebbca1454bd3218@mail.gmail.com>

Dag,

Thanks for the clarifications.  I think this would be a really great
addition.  I often find myself needing fast-than-numpy arrays and this
seems like it would open the door to make that easier.  Also, being
able to freely move between the different types of arrays (numpy, pil,
pointers, etc.) would be great.

One more question?  At times I have used SLT vector template in Cython
and it works really well of you want a fast dynamically sized array
type.  Do you think it would be possible to have cython arrays be
interchangible with these as well?  I don't see why it wouldn't work
through the c++ "ponter" member that vector has.  That would be a
killer feature.

Cheers,

Brian



>> * What will this make possible that is currently not possible?
>
> 1) Efficient slices
>
> 2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
> Koepke's project proposal)
>
> 3) Automatic contiguous copies in/out if a function is coded to work on
> contiguous memory
>
> Why can't this be done currently?
>
> ?* Cython contains no in-compiler support for NumPy, and so cannot know
> how to create new underlying ndarray objects.
>
> ?* All optimizations need to hard-code semantics at compile-time. With
> single-element indexing it seemed fair to assume the usual semantics of
> zero-based indexing etc., but with slices tings get worse (which kind of
> object is returned) and with arithmetic downright impossible (what does *
> do again?)
>
> That's not to say there's not other options:
> 1) We could hard-code support for NumPy only, and only allow ndarray and
> not subclasses thereof.
>
> 2) We could invent some new protocol/syntax for defining compile-time
> semantics for all relevant operations.
>
>> * What will this make easier that is currently really difficult?
>
> Well, nothing is really difficult. But passing, say, a PIL image into C
> code and back again for processing (with, say, a contiguous C array) isn't
> completely straightforward, but would be with this proposal.
>
> Dag Sverre
>
>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Thu Jun 11 18:24:22 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 18:24:22 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
Message-ID: <4A312FB6.7000300@behnel.de>


Ondrej Certik wrote:
> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>> Ondrej Certik wrote:
>>> we use cmake to build the project (C++ and cython) and cmake
>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>> and .py files in the source dir and creates a build dir, where it
>>> stores all the .o and .so files.
>> Uhm, so the real problem is that cmake fails to take the source directory
>> structure into account when copying build artefacts over to the build
>> directory?
> 
> I think cmake is doing it right, it only builds binary things in the
> binary directory and leaves all the sources (be it python or C++) in
> the source directory.

Ok, I get it, so all that's missing here to make this work is put either
the __init__.py files next to the binaries or the binaries next to the
__init__.py files.


> Consider this cython statement:
> 
> from hermes2d._hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>         int_grad_u_grad_v, int_v, malloc
> 
> This is essentially converted to a statement
> 
> py_module = PyImport_Import("hermes2d._hermes2d");      // Yes, I know
> it's actually PyString in there and this is handled automatically by
> __Pyx_ImportModule
> 
> Which is correct. Consider however this statement:
> 
> from _hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>         int_grad_u_grad_v, int_v, malloc
> 
> which is converted to:
> 
> py_module = PyImport_Import("hermes2d._hermes2d");
> 
> which I think is not right, because Cython changed my import path (and
> what is worse, it depends on the positions of the __init__.py files in
> my directory). All I am asking is if there is a really good reason,
> why Cython mangles the import paths

Yes (although it doesn't "mangle" them). Cython needs to know the fully
qualified module names to correctly handle class cimports. Otherwise, the
type would be known with one name in the module that implements it and with
a different name in the module that cimports it. Greg would know this better.

BTW, this has been discussed before (on the Pyrex list, I guess).

Stefan

From dagss at student.matnat.uio.no  Thu Jun 11 18:32:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 18:32:28 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A312B2E.5020602@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>
	<4A312B2E.5020602@behnel.de>
Message-ID: <4A31319C.9090309@student.matnat.uio.no>

Stefan Behnel wrote:
> Tuples support '+' and '*'. My immediate intuition is that an array type
> should behave exactly like a (mutable, but fixed-size) tuple, except where
> C dictates otherwise.
>
>   
>> I'm convinced that + would be used *a lot* more for componentwise
>> addition than for concatenation.
>>     
>
> I agree - at least in numerics. But if you want a mathematical vector type,
> that's a different thing than a memory managed C array IMHO.
>
> BTW, didn't we have this discussion twice already? That shows that at least
> my intuition keeps clearly dictating otherwise, each time we get here.
>   
Hmm, don't remember previous discussions. Anyway there's not that much 
to *discuss* as such; there's obvious interests at conflict here and the 
case seems clear for both.

I'll try to put down the interesting points:

1) Concatenation is a slow operation with large overhead where one can 
after all just use Python objects or cdef methods with negligble 
overhead. Addition is a real usecase where Cython can provide something 
Python/NumPy can not provide (by reducing memory bus traffic for large 
expressions). I.e. concatenation can be done without this feature, while 
for efficient expression evaluation one would have to switch to another 
language -- there's NO way of getting it in Cython otherwise.

2) I'd have to reconsider how much work I'm willing to put into this 
given concatenation semantics. I am scratching my own itch here. If + is 
chosen for concatenation I might end up having to use Fortran anyway in 
some cases (where Python/NumPy isn't fast enough), and so my motivation 
would be mainly on getting Kurt's GSoC through rather than empower 
Cython in general in this area.

3) Sometimes no feature at all is much better than a wrong or 
unintuitive feature, so 2) should NOT decide this, and I'll deal just 
fine with a "no" nevertheless :-)

4) Looking at mailing list activity, the numerical crowd is a very 
significant user of Cython

5) But again, the numerical stuff has happened as a byproduct, I realize 
that neither you nor Robert need it, and that Cython's real goal lies 
elsewhere


Trying one last time for constructive solutions though:
- There's always compiler directives
- Or int[:,:,componentwise=True]

OTOH, clearly

rsq = add(mul(x, x), mul(y, y))

is rather out of the question :-)

Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 11 18:41:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 18:41:01 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <6ce0ac130906110923y8d00f04i1ebbca1454bd3218@mail.gmail.com>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
	<6ce0ac130906110923y8d00f04i1ebbca1454bd3218@mail.gmail.com>
Message-ID: <4A31339D.5030901@student.matnat.uio.no>

Brian Granger wrote:
> One more question?  At times I have used SLT vector template in Cython
> and it works really well of you want a fast dynamically sized array
> type.  Do you think it would be possible to have cython arrays be
> interchangible with these as well?  I don't see why it wouldn't work
> through the c++ "ponter" member that vector has.  That would be a
> killer feature.
>   
Perhaps afterwards (without support for resize it would be supported 
straight away), resizing seems like a natural extension. I think it 
doesn't affect design decisions now and so I'd rather deal with it later 
-- let's get this proposal even accepted first :-)

Dag Sverre

From ondrej at certik.cz  Thu Jun 11 18:49:47 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 10:49:47 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A312FB6.7000300@behnel.de>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
Message-ID: <85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>

On Thu, Jun 11, 2009 at 10:24 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Ondrej Certik wrote:
>> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>>> Ondrej Certik wrote:
>>>> we use cmake to build the project (C++ and cython) and cmake
>>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>>> and .py files in the source dir and creates a build dir, where it
>>>> stores all the .o and .so files.
>>> Uhm, so the real problem is that cmake fails to take the source directory
>>> structure into account when copying build artefacts over to the build
>>> directory?
>>
>> I think cmake is doing it right, it only builds binary things in the
>> binary directory and leaves all the sources (be it python or C++) in
>> the source directory.
>
> Ok, I get it, so all that's missing here to make this work is put either
> the __init__.py files next to the binaries or the binaries next to the
> __init__.py files.

Yes, but putting binaries next to __init__.py files essentially means
building in the tree (then I can just build in the tree from the
beginning, which I do, but other people prefer not to pollute the
build tree), and putting __init__.py files next to binaries doesn't
work for me, because I then have the same module twice in my import
path (once in the source directory, once in the binary directory) and
it clashes -- at least I didn't make it work.

>
>
>> Consider this cython statement:
>>
>> from hermes2d._hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>> ? ? ? ? int_grad_u_grad_v, int_v, malloc
>>
>> This is essentially converted to a statement
>>
>> py_module = PyImport_Import("hermes2d._hermes2d"); ? ? ?// Yes, I know
>> it's actually PyString in there and this is handled automatically by
>> __Pyx_ImportModule
>>
>> Which is correct. Consider however this statement:
>>
>> from _hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>> ? ? ? ? int_grad_u_grad_v, int_v, malloc
>>
>> which is converted to:
>>
>> py_module = PyImport_Import("hermes2d._hermes2d");
>>
>> which I think is not right, because Cython changed my import path (and
>> what is worse, it depends on the positions of the __init__.py files in
>> my directory). All I am asking is if there is a really good reason,
>> why Cython mangles the import paths
>
> Yes (although it doesn't "mangle" them). Cython needs to know the fully
> qualified module names to correctly handle class cimports. Otherwise, the
> type would be known with one name in the module that implements it and with
> a different name in the module that cimports it. Greg would know this better.
>
> BTW, this has been discussed before (on the Pyrex list, I guess).

I found this:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/source_files.html

where it says:

"
If you don't do this //fully qualified module names//, you may find
that pickling doesn't work, among other problems. It also ensures that
the Pyrex compiler has the right idea about the layout of the module
namespace, which can be important when accessing extension types
defined in other modules.
"

so I guess it's not worthy to patch cython to do what I want. But I
still have a feeling that this is only an implementation detail in
cython, since Python itself can work this way, so I don't see why
Cython couldn't as well.


Ondrej

From dagss at student.matnat.uio.no  Thu Jun 11 18:53:32 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 18:53:32 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>	<4A30B977.9090306@behnel.de>	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
Message-ID: <4A31368C.7040507@student.matnat.uio.no>

Ondrej Certik wrote:
> On Thu, Jun 11, 2009 at 10:24 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>   
>> Ondrej Certik wrote:
>>     
>>> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>>>       
>>>> Ondrej Certik wrote:
>>>>         
>>>>> we use cmake to build the project (C++ and cython) and cmake
>>>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>>>> and .py files in the source dir and creates a build dir, where it
>>>>> stores all the .o and .so files.
>>>>>           
>>>> Uhm, so the real problem is that cmake fails to take the source directory
>>>> structure into account when copying build artefacts over to the build
>>>> directory?
>>>>         
>>> I think cmake is doing it right, it only builds binary things in the
>>> binary directory and leaves all the sources (be it python or C++) in
>>> the source directory.
>>>       
>> Ok, I get it, so all that's missing here to make this work is put either
>> the __init__.py files next to the binaries or the binaries next to the
>> __init__.py files.
>>     
>
> Yes, but putting binaries next to __init__.py files essentially means
> building in the tree (then I can just build in the tree from the
> beginning, which I do, but other people prefer not to pollute the
> build tree), and putting __init__.py files next to binaries doesn't
> work for me, because I then have the same module twice in my import
> path (once in the source directory, once in the binary directory) and
> it clashes -- at least I didn't make it work.
>   
This is the real problem I think. The concept of a build directory is 
that the result ends up there -- thus, copy over any pure .py files to 
the build directory, and only have that as the import path.

Really, that's how all out-of-tree builds work? Non-compiled files are 
copied.

I'm against any changes needed only because of a wierd import path setting.

Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 19:06:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 19:06:52 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31319C.9090309@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>
	<4A31319C.9090309@student.matnat.uio.no>
Message-ID: <4A3139AC.2050007@behnel.de>


Dag Sverre Seljebotn wrote:
> 5) But again, the numerical stuff has happened as a byproduct, I realize 
> that neither you nor Robert need it, and that Cython's real goal lies 
> elsewhere

It's less of a question of "who needs it", since there would be different
types of use cases for an array and vector type. This is about language
design, so the question is: what would users expect?

When I see a C array, I think of a memory block with a sequence of
identically typed items. I also think of a Python tuple because that
behaves very similar. I totally do not think of a math vector, because that
is a very (very!) special use case.

The examples you gave used "int" as a base type, simply because you were
thinking of numeric use cases. What about an array of pointers or structs?
What should adding two of those do?


> - There's always compiler directives

;) I actually thought about putting a mention of this with a big fat NO
into my last mail but then just stopped waiting for someone else to bring
this up.

A compiler directive would mean that

	a + b

would do entirely different things depending *not* on the types of a and b,
but on the state of the compiler. That's evil.


> - Or int[:,:,componentwise=True]

The main difference here (from a Python POV) is the implementation of
__add__ and friends, right? So the way I see it is that this should not be
an option. It's the entire *type* that should be different. If you want a
vector, use an explicit vector type. If you want a memory managed array,
use a standard array type. Don't mix the two, except after (explicit) coercion.

I haven't thought about how to cast this into a syntax yet, so this is just
my idea on how this should behave.

Stefan

From dagss at student.matnat.uio.no  Thu Jun 11 19:25:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 19:25:58 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3139AC.2050007@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de>
Message-ID: <4A313E26.7080006@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>   
>> 5) But again, the numerical stuff has happened as a byproduct, I realize 
>> that neither you nor Robert need it, and that Cython's real goal lies 
>> elsewhere
>>     
>
> It's less of a question of "who needs it", since there would be different
> types of use cases for an array and vector type. This is about language
> design, so the question is: what would users expect?
>   
Thanks again for supplying your thoughts in this thread.

The problem here is that the support must be built into the language, 
user-defined types won't do the trick (unless you have meta-programming 
capabilities on the level of LISP). So it's also a question of whether 
one can use Cython at all, or must use Fortran or C++ (the sad thing is 
that a language as horrible as Fortran will perhaps never be decently 
replaced, because all other languages consider math vectors a too 
special usecase.)

Anyway, thanks for clarifying the terms. So the question turns into: Is 
it possible to add a *vector* type to Cython? Or is that pulling it too 
far? (Choice of syntax aside.)

> When I see a C array, I think of a memory block with a sequence of
> identically typed items. I also think of a Python tuple because that
> behaves very similar. I totally do not think of a math vector, because that
> is a very (very!) special use case.
>   
Hmm. One can go with two different conclusions though
 a) You're ok with componentwise, but only given a different syntax
 b) Math vectors is too special a usecase to warrant native Cython support

Resolving this is my primary interest here, and we can deal with syntax 
later.

> The examples you gave used "int" as a base type, simply because you were
> thinking of numeric use cases. What about an array of pointers or structs?
> What should adding two of those do?
>   
Pointers: C semantics. I.e. you can add a pointer and an int array 
together to get new pointers, and so on. People are already doing 
exactly this with NumPy arrays (casting the pointers to integers and 
back though).

Structs: Ideally (for numericans) the arithmetic done field-wise. If you 
have a pointer in a struct it won't be allowed (unless, perhaps, one of 
the operands is another struct type with identical fields except that 
any pointer fields have been replaced with int fields. Yes, such things 
could be useful!)

But really I'm happy even if structs are dropped from this.

>> - There's always compiler directives
>>     
>
> ;) I actually thought about putting a mention of this with a big fat NO
> into my last mail but then just stopped waiting for someone else to bring
> this up.
>   
Well, but in reality, the substantial amount of numerical users would 
not consider putting it to anything but componentwise. Just consider it 
a nicer alternative to an outright fork NumCython :-) (Rest easy, I will 
not fork.)

Dag Sverre

From ondrej at certik.cz  Thu Jun 11 19:31:18 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 11:31:18 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A31368C.7040507@student.matnat.uio.no>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
Message-ID: <85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>

On Thu, Jun 11, 2009 at 10:53 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
>> On Thu, Jun 11, 2009 at 10:24 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>>
>>> Ondrej Certik wrote:
>>>
>>>> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>>>>
>>>>> Ondrej Certik wrote:
>>>>>
>>>>>> we use cmake to build the project (C++ and cython) and cmake
>>>>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>>>>> and .py files in the source dir and creates a build dir, where it
>>>>>> stores all the .o and .so files.
>>>>>>
>>>>> Uhm, so the real problem is that cmake fails to take the source directory
>>>>> structure into account when copying build artefacts over to the build
>>>>> directory?
>>>>>
>>>> I think cmake is doing it right, it only builds binary things in the
>>>> binary directory and leaves all the sources (be it python or C++) in
>>>> the source directory.
>>>>
>>> Ok, I get it, so all that's missing here to make this work is put either
>>> the __init__.py files next to the binaries or the binaries next to the
>>> __init__.py files.
>>>
>>
>> Yes, but putting binaries next to __init__.py files essentially means
>> building in the tree (then I can just build in the tree from the
>> beginning, which I do, but other people prefer not to pollute the
>> build tree), and putting __init__.py files next to binaries doesn't
>> work for me, because I then have the same module twice in my import
>> path (once in the source directory, once in the binary directory) and
>> it clashes -- at least I didn't make it work.
>>
> This is the real problem I think. The concept of a build directory is
> that the result ends up there -- thus, copy over any pure .py files to
> the build directory, and only have that as the import path.
>
> Really, that's how all out-of-tree builds work? Non-compiled files are
> copied.

When I copy all the .py files, then it of course works. So maybe you
are right, that I should just copy (or rather link them) all the .py
files into the build dir and be done with this.

>
> I'm against any changes needed only because of a wierd import path setting.

I don't think it's weird.

As far as Cython is concerned, the only problem I can see is that it
changes all import paths to the fully qualified import paths. Python
allows both ways, Cython only the fully qualified way. As Stefan
explained, this is needed, because it will then not work properly at
the moment. But I am not convinced this cannot be fixed -- though it
may happen that the fix is too tedious and not worthy.

Ondrej

From kwmsmith at gmail.com  Thu Jun 11 19:42:44 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 11 Jun 2009 12:42:44 -0500
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE018.5060902@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
Message-ID: <c266a7930906111042p182bbebi9f424d868038600c@mail.gmail.com>

On Wed, Jun 10, 2009 at 11:32 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Thanks to chats with Kurt and Robert, I think I've managed to present my
> thoughts from a different and more constructive angle.
>
> http://wiki.cython.org/enhancements/array
>
> There's some time constraints in the picture for various reasons (e.g.
> Fortran GSoC development direction) so it would be great if this could
> be resolved one way or the other before too long. Part of that will
> likely happen on IRC.
>
> Feedback from anybody (especially users) very welcome.

Nice proposal.  As mentioned in other comments, this would certainly
give Cython a strong numerical aspect and would make those using
Cython with numpy or external languages (ahem, Fortran ;-) quite
happy, I think.  The syntax seems to me nicely pythonic -- I have some
small comments on the stuff inside the brackets, but it'll wait until
(if and when) the proposal is accepted.

As an aside -- the array syntax & semantics within Fortran are, IMHO,
the one thing Fortran 90 got right and the main reason people *still*
write programs in it today.  The lack of a high-level array in
C-derivative languages is quite a hurdle for those who want
component-wise operations.  Pulling into Cython something that has
proven itself for a while now wouldn't be a bad thing.  But that's
from a numeric perspective, admittedly.

Kurt

From dalcinl at gmail.com  Thu Jun 11 20:01:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 11 Jun 2009 15:01:25 -0300
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3139AC.2050007@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31189E.6000104@behnel.de> <4A3123BB.5080603@student.matnat.uio.no>
	<4A312B2E.5020602@behnel.de> <4A31319C.9090309@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de>
Message-ID: <e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>

On Thu, Jun 11, 2009 at 2:06 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> When I see a C array, I think of a memory block with a sequence of
> identically typed items. I also think of a Python tuple because that
> behaves very similar. I totally do not think of a math vector, because that
> is a very (very!) special use case.
>

Well, for me it is not a matter about how do think of view an array,
but what do you want to do with an array. Could you tell me (apart
from char/wchar_t because of byte/unicode strings) how many times in
your life as a programmer did you need to concatenate an array of let
say integers or double precision floats ? That (I mean, concatenation)
is for me a very (very!) special use case for arrays...

Almost all APL's (array programming languages) I'm aware of do not
have concatenation semantics for a+b. Python is not an APL, but let me
ask... Why Cython should not be?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Jun 11 20:04:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:04:44 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A313E26.7080006@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>
	<4A313E26.7080006@student.matnat.uio.no>
Message-ID: <4A31473C.8020307@behnel.de>


Dag Sverre Seljebotn wrote:
> The problem here is that the support must be built into the language, 
> user-defined types won't do the trick (unless you have meta-programming 
> capabilities on the level of LISP).

I'm not arguing that the type shouldn't be a builtin. It's just a
difference if you require new syntax for a type or if you special case a
type in the compiler.


> Anyway, thanks for clarifying the terms. So the question turns into: Is 
> it possible to add a *vector* type to Cython? Or is that pulling it too 
> far? (Choice of syntax aside.)

I'm for having both. The vector type could become a subtype of an array
that overrides the numeric operations (although I wouldn't care too much if
that's too hard to implement).

We discussed the templating syntax a while ago. Even if that isn't there
yet, that doesn't mean we can't use its syntax.

	from cython cimport Vector
	cdef Vector[int] some_vector

would be the same as the equivalent "int[:]" array, except for arithmetic.


>> The examples you gave used "int" as a base type, simply because you were
>> thinking of numeric use cases. What about an array of pointers or structs?
>> What should adding two of those do?
>>   
> Pointers: C semantics. I.e. you can add a pointer and an int array 
> together to get new pointers, and so on. People are already doing 
> exactly this with NumPy arrays (casting the pointers to integers and 
> back though).

I actually meant adding two arrays of pointers.


> Structs: Ideally (for numericans) the arithmetic done field-wise.

I would expect that when I have a /vector/ of structs (and yes, that would
be a totally cool feature). An /array/ of structs should concatenate instead.

Stefan


From stefan_ml at behnel.de  Thu Jun 11 20:16:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:16:47 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>
	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>
	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>
	<e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>
Message-ID: <4A314A0F.5080505@behnel.de>


Lisandro Dalcin wrote:
> On Thu, Jun 11, 2009 at 2:06 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>> When I see a C array, I think of a memory block with a sequence of
>> identically typed items. I also think of a Python tuple because that
>> behaves very similar. I totally do not think of a math vector, because that
>> is a very (very!) special use case.
> 
> Well, for me it is not a matter about how do think of view an array,
> but what do you want to do with an array. Could you tell me (apart
> from char/wchar_t because of byte/unicode strings) how many times in
> your life as a programmer did you need to concatenate an array of let
> say integers or double precision floats ? That (I mean, concatenation)
> is for me a very (very!) special use case for arrays...

Have you never concatenated tuples? I consider this the usual chicken and
egg problem. If the functionality was available, people would use it.

Currently, I avoid C arrays in Cython wherever I can, simply because it's
really hard to deal with them (manual memory management, resizing, ...). If
Cython grew native array and vector types, I bet people would just jump on
them.

I'd even vote for allowing resizing of Cython's array type (or a subtype),
maybe through a resize() method that would call realloc() internally. That
would remove the need to keep C types in Python lists (or to do manual
memory management to achieve the same).

Stefan


From dalcinl at gmail.com  Thu Jun 11 20:31:31 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 11 Jun 2009 15:31:31 -0300
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
Message-ID: <e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>

On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>
> As far as Cython is concerned, the only problem I can see is that it
> changes all import paths to the fully qualified import paths. Python
> allows both ways, Cython only the fully qualified way. As Stefan
> explained, this is needed, because it will then not work properly at
> the moment. But I am not convinced this cannot be fixed -- though it
> may happen that the fix is too tedious and not worthy.
>

IMHO, the only "fix" Cython needs is support for relative imports...
Then you could write:

from ._hermes2d cimport what_you_need

This way, Ondrej would be able to easily write Cython code using
relative imports the right way...




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Jun 11 20:35:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 20:35:30 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31473C.8020307@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de>
Message-ID: <4A314E72.6000500@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Anyway, thanks for clarifying the terms. So the question turns into: Is 
>> it possible to add a *vector* type to Cython? Or is that pulling it too 
>> far? (Choice of syntax aside.)
> 
> I'm for having both. The vector type could become a subtype of an array
> that overrides the numeric operations (although I wouldn't care too much if
> that's too hard to implement).
> 
> We discussed the templating syntax a while ago. Even if that isn't there
> yet, that doesn't mean we can't use its syntax.
> 
> 	from cython cimport Vector
> 	cdef Vector[int] some_vector
> 
> would be the same as the equivalent "int[:]" array, except for arithmetic.

That answers most of my worries. I'll leave this to sink in for a few 
days; but I'll start brainstorming for shorter syntax alternatives which 
might make this discussion shorter :-)

Here's some starting points you all won't like:

int[:].cwise
int.cwise[:]

Also consider that "array" is already a Python builtin doing much of 
what Stefan seems to be after, at least Python side. array[int] should 
perhaps at least be an alias for what Stefan wants int[:] to be, at 
least in 1D, although Stefan's arguments from "expectation of random 
Python user" is still valid.

(BTW, this is how a 2D arrays look in Fortran, which is still a very 
dominant tool in numerics:

real, dimension(:,:) :: arr

So you can see where my inspiration with the [:] came from.)
-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 20:40:48 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:40:48 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>	<4A30B977.9090306@behnel.de>	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>	<4A312FB6.7000300@behnel.de>	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>	<4A31368C.7040507@student.matnat.uio.no>	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
Message-ID: <4A314FB0.9090507@behnel.de>


Lisandro Dalcin wrote:
> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>> As far as Cython is concerned, the only problem I can see is that it
>> changes all import paths to the fully qualified import paths. Python
>> allows both ways, Cython only the fully qualified way. As Stefan
>> explained, this is needed, because it will then not work properly at
>> the moment. But I am not convinced this cannot be fixed -- though it
>> may happen that the fix is too tedious and not worthy.
>>
> 
> IMHO, the only "fix" Cython needs is support for relative imports...
> Then you could write:
> 
> from ._hermes2d cimport what_you_need
> 
> This way, Ondrej would be able to easily write Cython code using
> relative imports the right way...

I thought about this, too, and yes, that would be a nice feature. But it
doesn't help in this case. What Ondrej wanted was to be able to import the
module both from its package and as a plain import, i.e. when put outside
of any package. The above would not handle the latter case either.

Stefan

From dagss at student.matnat.uio.no  Thu Jun 11 20:53:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 20:53:42 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A314A0F.5080505@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>
	<4A314A0F.5080505@behnel.de>
Message-ID: <4A3152B6.6070206@student.matnat.uio.no>

Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>> On Thu, Jun 11, 2009 at 2:06 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>>> When I see a C array, I think of a memory block with a sequence of
>>> identically typed items. I also think of a Python tuple because that
>>> behaves very similar. I totally do not think of a math vector, because that
>>> is a very (very!) special use case.
>> Well, for me it is not a matter about how do think of view an array,
>> but what do you want to do with an array. Could you tell me (apart
>> from char/wchar_t because of byte/unicode strings) how many times in
>> your life as a programmer did you need to concatenate an array of let
>> say integers or double precision floats ? That (I mean, concatenation)
>> is for me a very (very!) special use case for arrays...
> 
> Have you never concatenated tuples? I consider this the usual chicken and
> egg problem. If the functionality was available, people would use it.

I can actually attest to this. In my primary language these days, R 
(very much a componentwise language), I find myself doing

cbind(arr1, arr2)

for concatenating column-wise,

rbind(arr1, arr2)

for concatenating row-wise, and

c(1, 2, 3, arr1)

for unpacking concatenation all the time. Only in situations where speed 
is not an issue though.

(Of course, I use componentwise + even more often -- but having 
available functions doing this even in the componentwise case would be 
nice).

BTW, all of Stefan's examples been one-dimensional, which doesn't quite 
fit with the CEP which has a strong many-dimensional emphasis (or at 
least that was the point). Would + only work for 1D, or would it always 
add along the first dimension (i.e. like rbind above)? How to add along 
other dimensions?

-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 20:52:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:52:00 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A314E72.6000500@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
Message-ID: <4A315250.7080606@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Anyway, thanks for clarifying the terms. So the question turns into: Is 
>>> it possible to add a *vector* type to Cython? Or is that pulling it too 
>>> far? (Choice of syntax aside.)
>> I'm for having both. The vector type could become a subtype of an array
>> that overrides the numeric operations (although I wouldn't care too much if
>> that's too hard to implement).
>>
>> We discussed the templating syntax a while ago. Even if that isn't there
>> yet, that doesn't mean we can't use its syntax.
>>
>> 	from cython cimport Vector
>> 	cdef Vector[int] some_vector
>>
>> would be the same as the equivalent "int[:]" array, except for arithmetic.
> 
> That answers most of my worries. I'll leave this to sink in for a few 
> days; but I'll start brainstorming for shorter syntax alternatives which 
> might make this discussion shorter :-)
> 
> Here's some starting points you all won't like:
> 
> int[:].cwise
> int.cwise[:]

Yes, you're right. :)

Honestly, what's so bad about making the vector type explicit? Even

	Vector[int[:]]
or
	Vector(int[:])

are better than a hard to read "cwise".


> Also consider that "array" is already a Python builtin doing much of 
> what Stefan seems to be after, at least Python side.

Well, yes, except for storing pointers and structs.


> (BTW, this is how a 2D arrays look in Fortran, which is still a very 
> dominant tool in numerics:
> 
> real, dimension(:,:) :: arr
> 
> So you can see where my inspiration with the [:] came from.)

I like the "int[:]" syntax. I just think that a vector should look
different in the code.

Stefan

From ondrej at certik.cz  Thu Jun 11 21:02:15 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 13:02:15 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A314FB0.9090507@behnel.de>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
	<4A314FB0.9090507@behnel.de>
Message-ID: <85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>

On Thu, Jun 11, 2009 at 12:40 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>>> As far as Cython is concerned, the only problem I can see is that it
>>> changes all import paths to the fully qualified import paths. Python
>>> allows both ways, Cython only the fully qualified way. As Stefan
>>> explained, this is needed, because it will then not work properly at
>>> the moment. But I am not convinced this cannot be fixed -- though it
>>> may happen that the fix is too tedious and not worthy.
>>>
>>
>> IMHO, the only "fix" Cython needs is support for relative imports...
>> Then you could write:
>>
>> from ._hermes2d cimport what_you_need
>>
>> This way, Ondrej would be able to easily write Cython code using
>> relative imports the right way...
>
> I thought about this, too, and yes, that would be a nice feature. But it
> doesn't help in this case. What Ondrej wanted was to be able to import the
> module both from its package and as a plain import, i.e. when put outside
> of any package. The above would not handle the latter case either.

I think it would. I have _forms.so and _hermes2d.so. The _hermes2d.so
is imported from _forms.so and if the import was relative, e.g. if
_forms.so was doing "import ._hermes2d", then it should work both in
the tree and out of the tree.

Ondrej

Ondrej

From dagss at student.matnat.uio.no  Thu Jun 11 21:07:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 21:07:15 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315250.7080606@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de>
Message-ID: <4A3155E3.908@student.matnat.uio.no>

I never keep a promise to shut up...

Stefan Behnel wrote:
> Honestly, what's so bad about making the vector type explicit? Even
> 
> 	Vector[int[:]]
> or
> 	Vector(int[:])
> 
> are better than a hard to read "cwise".

Being very concrete, the word Vector is not something I normally use in 
this situation and I would really prefer to keep it that way. If a is a 
"vector", I expect linear algebra behaviour -- "a.T * a" is a scalar and 
so on.

E.g. Sage has a Vector type with the mathematical meaning of the word 
vector. Cython code in Sage dealing with efficient numerical linear 
algebra could probably look quite horrible as a result if the word 
vector was chosen.

In every other programming language I know about with componentwise 
operations, the word "array" is used. "vector" is something you Coder 
People try to force down on us Numericals :-)

(I know that the word is used like this in CS circles, but I prefer only 
using the word in contexts where the meaning is crystal clear (as in 
"vectorized operations") myself.)

-- 
Dag Sverre

From ondrej at certik.cz  Thu Jun 11 21:07:50 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 13:07:50 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
	<4A314FB0.9090507@behnel.de>
	<85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>
Message-ID: <85b5c3130906111207s44d5a715qfe91081566a0b5c2@mail.gmail.com>

On Thu, Jun 11, 2009 at 1:02 PM, Ondrej Certik<ondrej at certik.cz> wrote:
> On Thu, Jun 11, 2009 at 12:40 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>>
>> Lisandro Dalcin wrote:
>>> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>>>> As far as Cython is concerned, the only problem I can see is that it
>>>> changes all import paths to the fully qualified import paths. Python
>>>> allows both ways, Cython only the fully qualified way. As Stefan
>>>> explained, this is needed, because it will then not work properly at
>>>> the moment. But I am not convinced this cannot be fixed -- though it
>>>> may happen that the fix is too tedious and not worthy.
>>>>
>>>
>>> IMHO, the only "fix" Cython needs is support for relative imports...
>>> Then you could write:
>>>
>>> from ._hermes2d cimport what_you_need
>>>
>>> This way, Ondrej would be able to easily write Cython code using
>>> relative imports the right way...
>>
>> I thought about this, too, and yes, that would be a nice feature. But it
>> doesn't help in this case. What Ondrej wanted was to be able to import the
>> module both from its package and as a plain import, i.e. when put outside
>> of any package. The above would not handle the latter case either.
>
> I think it would. I have _forms.so and _hermes2d.so. The _hermes2d.so
> is imported from _forms.so and if the import was relative, e.g. if
> _forms.so was doing "import ._hermes2d", then it should work both in
> the tree and out of the tree.

Looking here, it should be easy to support relative imports:

http://docs.python.org/c-api/import.html

so unless there is some difficulty inside Cython, that I don't see yet
(e.g. the kind of thing Stefan was talking about), it should be
possible to support relative imports quite easily, I already
understand the part that takes the import path from .pyx file and then
makes it fully qualified --- so I would simply change that.

Ondrej

From stefan_ml at behnel.de  Thu Jun 11 21:18:26 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 21:18:26 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3155E3.908@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>
	<4A3155E3.908@student.matnat.uio.no>
Message-ID: <4A315882.4030305@behnel.de>


Dag Sverre Seljebotn wrote:
> I never keep a promise to shut up...
> 
> Stefan Behnel wrote:
>> Honestly, what's so bad about making the vector type explicit? Even
>>
>> 	Vector[int[:]]
>> or
>> 	Vector(int[:])
>>
>> are better than a hard to read "cwise".
> 
> Being very concrete, the word Vector is not something I normally use in 
> this situation and I would really prefer to keep it that way. If a is a 
> "vector", I expect linear algebra behaviour -- "a.T * a" is a scalar and 
> so on.
> 
> In every other programming language I know about with componentwise 
> operations, the word "array" is used. "vector" is something you Coder 
> People try to force down on us Numericals :-)
> 
> (I know that the word is used like this in CS circles, but I prefer only 
> using the word in contexts where the meaning is crystal clear (as in 
> "vectorized operations") myself.)

Ah, so all you really want is a SIMD data type, right? And I guess that's
how things work in Fortran?

Stefan

From stefan_ml at behnel.de  Thu Jun 11 21:27:57 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 21:27:57 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315882.4030305@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de>
Message-ID: <4A315ABD.3020806@behnel.de>


Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> In every other programming language I know about with componentwise 
>> operations, the word "array" is used. "vector" is something you Coder 
>> People try to force down on us Numericals :-)
>>
>> (I know that the word is used like this in CS circles, but I prefer only 
>> using the word in contexts where the meaning is crystal clear (as in 
>> "vectorized operations") myself.)
> 
> Ah, so all you really want is a SIMD data type, right?

... which would then also support parallel operations as in

	some_int_array *= 2

to multiply all entries by two, and

	some_float_array = math.sqrt(some_int_array)

to take the square-root of all entries in parallel?

Stefan


From dalcinl at gmail.com  Thu Jun 11 21:29:49 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 11 Jun 2009 16:29:49 -0300
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315882.4030305@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A312B2E.5020602@behnel.de> <4A31319C.9090309@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de> <4A313E26.7080006@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de> <4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de>
Message-ID: <e7ba66e40906111229o14a2342bxb7ab92b5c740693d@mail.gmail.com>

On Thu, Jun 11, 2009 at 4:18 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
>
> Ah, so all you really want is a SIMD data type, right?
>

Yes, and not only Dag, but almost all people doing numerics :-)

>
> And I guess that's
> how things work in Fortran?
>

Yes, and not only Fortran, but almost all APL langs out there:
Matlab/Octave/Scilab, R, IDL, D (though it uses c[]=a[]+b[])



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Jun 11 22:01:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 22:01:35 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315ABD.3020806@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>
	<4A315ABD.3020806@behnel.de>
Message-ID: <4A31629F.8000608@student.matnat.uio.no>

Stefan Behnel wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> In every other programming language I know about with componentwise 
>>> operations, the word "array" is used. "vector" is something you Coder 
>>> People try to force down on us Numericals :-)
>>>
>>> (I know that the word is used like this in CS circles, but I prefer only 
>>> using the word in contexts where the meaning is crystal clear (as in 
>>> "vectorized operations") myself.)
>> Ah, so all you really want is a SIMD data type, right?

Right :-) Now linear algebra support in the compiler *would* be taking 
things quite overboard (and probably only slow things down, unless N is 
  really small, like 4x4 3D game matrices...), I'm actually against that.

> ... which would then also support parallel operations as in
> 
> 	some_int_array *= 2
> 
> to multiply all entries by two, and

That's actually an example in the CEP :-)

> 	some_float_array = math.sqrt(some_int_array)
> 
> to take the square-root of all entries in parallel?

Well, the issue here is function namespaces -- do we "misuse" the Python 
library like this? But the idea -- certainly. At least it should be easy 
to call C stdlib's sin in parallell on a double array.

And it should be possible to create custom functions to call in 
parallel, either by

cdef extern from "stdlib.h":
     double sin(double)
cdef myfunc(double x): ...
myfunc(sin(arr)) # SIMD, by convention, as we know they
                  # take single element

or, if people think that is too magic, by a decorator:

cdef extern from "stdlib.h":
     @cython.array.elementwise
     double sin(double)
@cython.array.elementwise
cdef myfunc(double x): ...

I suggest we deal with this particular issue much later though -- for 
now it's nice to have a roadmap and figure out the syntax for 
concatenateable vs. SIMD, but any actual work on componentwise 
operations will happen after everything else works.

-- 
Dag Sverre

From roland at utk.edu  Thu Jun 11 23:01:43 2009
From: roland at utk.edu (Roland Schulz)
Date: Thu, 11 Jun 2009 17:01:43 -0400
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31629F.8000608@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de> 
	<4A313E26.7080006@student.matnat.uio.no> <4A31473C.8020307@behnel.de> 
	<4A314E72.6000500@student.matnat.uio.no> <4A315250.7080606@behnel.de> 
	<4A3155E3.908@student.matnat.uio.no> <4A315882.4030305@behnel.de> 
	<4A315ABD.3020806@behnel.de> <4A31629F.8000608@student.matnat.uio.no>
Message-ID: <c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>

On Thu, Jun 11, 2009 at 4:01 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Stefan Behnel wrote:
> > Stefan Behnel wrote:
> >> Dag Sverre Seljebotn wrote:
> >>> In every other programming language I know about with componentwise
> >>> operations, the word "array" is used. "vector" is something you Coder
> >>> People try to force down on us Numericals :-)
> >>>
> >>> (I know that the word is used like this in CS circles, but I prefer
> only
> >>> using the word in contexts where the meaning is crystal clear (as in
> >>> "vectorized operations") myself.)
> >> Ah, so all you really want is a SIMD data type, right?
>
> Right :-) Now linear algebra support in the compiler *would* be taking
> things quite overboard (and probably only slow things down, unless N is
>  really small, like 4x4 3D game matrices...), I'm actually against that.
>


Well only if you do it directly in a hard-coded datatype.

Why not having a non numeric base class as Stefan proposed. And then
allowing the operator functions be handled by Eigen/Blitz. This would enable
very fast numerics, including SSE and fast linear algebra. And without
having it as tightly coupled in the main language. Similar in that expect to
weave, but with all the advantages of cython.

Component wise operations without optimization (thus collapsing d=a*b*c*d
into one loop instead of 3 and not using temporary arrays) does not give you
any speed-up over Numpy for vectorized code with large arrays.

For vectorized Numpy code the bottleneck is not the call from Python to C,
but the inefficient use of cache because of the temporary arrays.

And I don't think you need a full metalanguage to map the array class to
Eigen/Blitz. Though it would be still be couple somewhat to the main
language otherwise I agree you would need a metalanguage or full template
support. But it seems to me that there should be an optimum between full
template support and doing the numerics directly in a language specific
numeric array type. Sorry for being vage in this central point. I just don't
know the cython details here.

For the naming: why not calling it ndarray, nparray or narray? Even numpy
has two array types to allow different operator syntax. Hardcoding one into
the default array won't allow that in the future.

Roland



-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090611/cfcc7a41/attachment-0001.htm 

From dagss at student.matnat.uio.no  Thu Jun 11 23:22:43 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 23:22:43 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>
	<4A313E26.7080006@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
Message-ID: <4A3175A3.8010304@student.matnat.uio.no>

Thanks for your input!, you definitely know more about such computations 
than me.

Roland Schulz wrote:
> 
> 
> On Thu, Jun 11, 2009 at 4:01 PM, Dag Sverre Seljebotn 
> <dagss at student.matnat.uio.no <mailto:dagss at student.matnat.uio.no>> wrote:

>     Right :-) Now linear algebra support in the compiler *would* be taking
>     things quite overboard (and probably only slow things down, unless N is
>      really small, like 4x4 3D game matrices...), I'm actually against that.
> 
> 
> 
> Well only if you do it directly in a hard-coded datatype.
> 
> Why not having a non numeric base class as Stefan proposed. And then 
> allowing the operator functions be handled by Eigen/Blitz. This would 
> enable very fast numerics, including SSE and fast linear algebra. And 
> without having it as tightly coupled in the main language. Similar in 
> that expect to weave, but with all the advantages of cython.

I'll answer this below.

> 
> Component wise operations without optimization (thus collapsing 
> d=a*b*c*d into one loop instead of 3 and not using temporary arrays) 
> does not give you any speed-up over Numpy for vectorized code with large 
> arrays.
> 
> For vectorized Numpy code the bottleneck is not the call from Python to 
> C, but the inefficient use of cache because of the temporary arrays.

I don't know enough about this, but these two paragraphs seem slightly 
contradictory to me.

Anyway: Any builtin features as such in Cython can't really depend on 
Eigen/Blitz. (I know this is perhaps not what you are suggesting, but 
bear with me.)

As the CEP 517 says, I see this as a two-step process:

1) Naive C loops produced by Cython, which will be "good enough" for 
many cases, and which gives a reference spec and implementation which 
runs out of the box without library requirements.

2) Various plugins (with a seperate release process from Cython so that 
work in this area doesn't bog down Cython development), using e.g. 
Eigen/Blitz as backends, or run it on a GPU, in parallell using OpenMP 
(well, for heavy componentwise functions), etc. etc.

Whether a library could be produced in a mixture of C++ and Cython, 
perhaps adding more Cython features along the way, which implemented 
this directly using Eigen/Blitz instead... perhaps. It seems more 
difficult to get the smooth PEP 3118 integration, but it could perhaps 
be done.

> 
> And I don't think you need a full metalanguage to map the array class to 
> Eigen/Blitz. Though it would be still be couple somewhat to the main 
> language otherwise I agree you would need a metalanguage or full 
> template support. But it seems to me that there should be an optimum 
> between full template support and doing the numerics directly in a 
> language specific numeric array type. Sorry for being vage in this 
> central point. I just don't know the cython details here.

Well, the point here is that when you use Eigen/Blitz, what you do is 
use the metalanguage of C++. Which is much more powerful than anything 
we're likely to get in Cython. So in some sense you use C++ to do what 
you cannot do in Cython.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 11 23:32:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 23:32:41 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3175A3.8010304@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>
	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>
	<4A315ABD.3020806@behnel.de>	<4A31629F.8000608@student.matnat.uio.no>	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
Message-ID: <4A3177F9.5010007@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Thanks for your input!, you definitely know more about such computations 
> than me.
> 
> Roland Schulz wrote:
>> Component wise operations without optimization (thus collapsing 
>> d=a*b*c*d into one loop instead of 3 and not using temporary arrays) 
>> does not give you any speed-up over Numpy for vectorized code with large 
>> arrays.
>>
>> For vectorized Numpy code the bottleneck is not the call from Python to 
>> C, but the inefficient use of cache because of the temporary arrays.
> 
> I don't know enough about this, but these two paragraphs seem slightly 
> contradictory to me.

Or did you mean that

optimization == collapsing d=a*b*c*d into one loop instead of 3 and not 
using temporary arrays

?

It is definitely the plan of CEP 517 that

cdef int[:] a = ..., b = ..., c = ..., d = ...
d = a * b * c * d

turn into something very similar to

cdef size_t tmp1
cdef int[:] tmpresult = new array of right length
for tmp1 in range(a.shape[0]):
     tmpresult[tmp1] = a[tmp1] + b[tmp1] + c[tmp1] + d[tmp1]
d = tmpresult

although broadcasting should be supported and makes it more complicated 
(repeat arrays of length 1, if d has length 1 it must be reallocated -- 
and so on). With multidimensional this becomes more difficult, there's 
lots of ugly details concerning broadcasting and non-contiguous arrays 
(where the "innermost" dimension must be found at runtime...)

IMPORTANT NOTE: All of this is way ahead, all that is the question now 
is a coarse roadmap, and whether this is wanted at all or not.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Jun 12 08:44:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Jun 2009 23:44:53 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
Message-ID: <52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>

If anything, your proposal has at least been interesting enough to  
generate a lot of discussion. Would a fair summary be that you're  
proposing more full-featured wrapping of PEP 3118's bufferinfo struct  
(where all the semantics/operation logic are embedded in the Cython  
compiler and code emitted inline, rather than in an external class  
like an ndarray)? And essentially you want these to act as Fortran  
arrays, a native type in the language.

I am not opposed to this feature, despite it being off the path of  
being a good, optimizing Python compiler with easy glue to external  
code, as long as it doesn't intrude on the language too much. Mostly,  
this is because there is so much interest in using Cython in a  
numerical processing context. I do, however, have some reservations.

First, it sounds a bit like you're proposing to essentially re- 
implement NumPy. Would it just be slicing and arithmetic, or where  
would you draw the line before it doesn't really belong in a  
compiler, but rather a library. More below:

On Jun 10, 2009, at 11:57 AM, Dag Sverre Seljebotn wrote:

> Brian Granger wrote:
>> Dag,
>>
>> I quickly glanced through the proposal and have two big picture  
>> questions:
>>
>> * What will this make possible that is currently not possible?

This was originally my first question too, but you beat me to it.

> 1) Efficient slices

Is the inefficiency just in the object creation and Python indexing  
semantics? It's still O(1), right? Same with the other operations. (I  
guess there's also a question of result type.)

> 2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
> Koepke's project proposal)

Could you provide a bit more of a concrete example here? There is  
avoiding allocation of temporary arrays, is there more?

> 3) Automatic contiguous copies in/out if a function is coded to  
> work on
> contiguous memory
>
> Why can't this be done currently?
>
>  * Cython contains no in-compiler support for NumPy, and so cannot  
> know
> how to create new underlying ndarray objects.
>
>  * All optimizations need to hard-code semantics at compile-time. With
> single-element indexing it seemed fair to assume the usual  
> semantics of
> zero-based indexing etc., but with slices tings get worse (which  
> kind of
> object is returned) and with arithmetic downright impossible (what  
> does *
> do again?)
>
> That's not to say there's not other options:
> 1) We could hard-code support for NumPy only, and only allow  
> ndarray and
> not subclasses thereof.
>
> 2) We could invent some new protocol/syntax for defining compile-time
> semantics for all relevant operations.

Here I am torn--I don't like defining compile-time semantics because  
it goes against the whole OO style of inheritance (and feels even  
more remote than the very dynamic, late-binding Python runtime). I  
don't like option (1) either though.

Another an idea, have you thought of using NumPy as the backend? I.e.  
an int[:,:] is any bufferinfo--supporting object, but if one needs to  
be created you create it via an ndarray? This could (potentially)  
facilitate a lot more code reuse (especially for operations that are  
more complicated than a single loop over the data). (Might be messier  
than implementing int[:,:] directly though.) Suppose one develops a  
vectorized version of ndarrays, could that be a drop-in replacement?

The plug-in idea is very interesting, both from the perspective that  
it would allow one to play with different ways to operate on arrays,  
and also it could separate some of the array-processing logic out of  
the core compiler itself.

- Robert


P.S. In terms of componentwise operations vs. concatenate, I think a  
raw list type would be very useful too. int[:] has the advantage that  
it is easy to encode dimension and other information relevant to  
arrays. Memory-managed lists of ints, doubles, structs, etc. would be  
something I would very much like to see added, but via another syntax  
(e.g. [int] or int[]). Using the keyword "vector" or "array" is IMHO  
a bad idea, as it looks like an ordinary type, and bars a user from  
actually using that keyword as well. Syntax via punctuation makes it  
clear that something non-Pythonic is going on here.


P.P.S. Were you actually suggesting that when I have

cdef struct X:
     int a
     double b

I could do

cdef X x = ..., y = ..., z
z = x+y

?

From dagss at student.matnat.uio.no  Fri Jun 12 09:30:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 09:30:50 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
	<52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>
Message-ID: <4A32042A.20709@student.matnat.uio.no>

A quick answer to some of the points.

Robert Bradshaw wrote:
> First, it sounds a bit like you're proposing to essentially re- 
> implement NumPy. Would it just be slicing and arithmetic, or where  
> would you draw the line before it doesn't really belong in a  
> compiler, but rather a library. More below:

At this point, I should stress that what I'm after is a roadmap (how do 
I deal with requests coming up, and things like Hoyt's proposed project 
when that came up). I won't have time for doing more than the basic 
array stuff (to better facilitate ourself of Kurt's work) this time 
around -- which is still very useful for exchanging array data with 
C/Fortran code.

It would be "reimplementing" NumPy as far as the core array API goes, 
but the implementation would be totally different as we have a 
compilation stage. If anything, it is reimplementing Fortran, however 
Fortran desperately need more alternatives :-)

As for the library, the CEP scetches full interopability with the 
library, you would likely still do

arr = np.dot(arr,arr)

for matrix mul.

What to include? I think slicing, arithmetic (with "broadcasting", i.e 
dimensions of length 1 are repeated to make arrays conform), transpose 
(arr.T). No "sum()" member function or similar.

In addition perhaps high-level utilities for looping, such as 
"ndenumerate" etc:

cdef int[:,:] arr = ...
for (i, j), value in cython.array.ndenumerate(arr):
     ...

These would be quite nicely contained in cython.array though (and again 
have no direct NumPy link that's not horribly slow due to Python-level 
iterators).


> 
> On Jun 10, 2009, at 11:57 AM, Dag Sverre Seljebotn wrote:
> 
>> Brian Granger wrote:
>>> Dag,
>>>
>>> I quickly glanced through the proposal and have two big picture  
>>> questions:
>>>
>>> * What will this make possible that is currently not possible?
> 
> This was originally my first question too, but you beat me to it.
> 
>> 1) Efficient slices
> 
> Is the inefficiency just in the object creation and Python indexing  
> semantics? It's still O(1), right? Same with the other operations. (I  
> guess there's also a question of result type.)

Well, and then the resulting buffer is acquired and checked for data 
type. But essentially, yes.

This is purely a matter of notational convenience -- will it hurt to 
have that O(1) in your almost-inner-loop taking a slice of 50 elements, 
or do you pass "start", "end" around in your function?

>> 2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
>> Koepke's project proposal)
> 
> Could you provide a bit more of a concrete example here? There is  
> avoiding allocation of temporary arrays, is there more?

Most systems today are for simple functions like addition constrained by 
memory bus speed, not CPU speed. Consider

B = A + A + A + A + A + A

With NumPy, the data of A has to travel over the bus 6 times, with loop 
unrolling only 1, as it would turn into

B = new memory
for idx in ...
    B[idx] = A[idx] + A[idx] + ...

This can mean a dramatic speed increase as data doesn't have to enter 
the cache more than once.

(NumPy can't do this, but there is numexpr, which allow this:

B = numexpr.eval("A+A+...")

using bytecode interpreter and working in cache-sized blocks.)

>> 3) Automatic contiguous copies in/out if a function is coded to  
>> work on
>> contiguous memory
>>
>> Why can't this be done currently?
>>
>>  * Cython contains no in-compiler support for NumPy, and so cannot  
>> know
>> how to create new underlying ndarray objects.
>>
>>  * All optimizations need to hard-code semantics at compile-time. With
>> single-element indexing it seemed fair to assume the usual  
>> semantics of
>> zero-based indexing etc., but with slices tings get worse (which  
>> kind of
>> object is returned) and with arithmetic downright impossible (what  
>> does *
>> do again?)
>>
>> That's not to say there's not other options:
>> 1) We could hard-code support for NumPy only, and only allow  
>> ndarray and
>> not subclasses thereof.
>>
>> 2) We could invent some new protocol/syntax for defining compile-time
>> semantics for all relevant operations.
> 
> Here I am torn--I don't like defining compile-time semantics because  
> it goes against the whole OO style of inheritance (and feels even  
> more remote than the very dynamic, late-binding Python runtime). I  
> don't like option (1) either though.
> 
> Another an idea, have you thought of using NumPy as the backend? I.e.  
> an int[:,:] is any bufferinfo--supporting object, but if one needs to  
> be created you create it via an ndarray? This could (potentially)  
> facilitate a lot more code reuse (especially for operations that are  
> more complicated than a single loop over the data). (Might be messier  
> than implementing int[:,:] directly though.) Suppose one develops a  
> vectorized version of ndarrays, could that be a drop-in replacement?

? ndarrays are vectorized? But subclasses may not be.

I've thought about it. If arithmetic is implemented on arrays, the only 
thing that's seems to be reused this way is malloc/free of a memory area 
though. Better loose the dependency then.

> The plug-in idea is very interesting, both from the perspective that  
> it would allow one to play with different ways to operate on arrays,  
> and also it could separate some of the array-processing logic out of  
> the core compiler itself.

I can definitely see NumPy as being the first such plugin, i.e. it would 
force each array to exactly ndarray on arithmetic, to at least provide 
some arithmetic support right away.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Jun 12 10:02:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 12 Jun 2009 01:02:28 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A32042A.20709@student.matnat.uio.no>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
	<52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>
	<4A32042A.20709@student.matnat.uio.no>
Message-ID: <5FFC2397-4769-49E3-9E00-04D22DDC4E1F@math.washington.edu>

On Jun 12, 2009, at 12:30 AM, Dag Sverre Seljebotn wrote:

> A quick answer to some of the points.
>
> Robert Bradshaw wrote:
>> First, it sounds a bit like you're proposing to essentially re-
>> implement NumPy. Would it just be slicing and arithmetic, or where
>> would you draw the line before it doesn't really belong in a
>> compiler, but rather a library. More below:
>
> At this point, I should stress that what I'm after is a roadmap  
> (how do
> I deal with requests coming up, and things like Hoyt's proposed  
> project
> when that came up). I won't have time for doing more than the basic
> array stuff (to better facilitate ourself of Kurt's work) this time
> around -- which is still very useful for exchanging array data with
> C/Fortran code.
>
> It would be "reimplementing" NumPy as far as the core array API goes,
> but the implementation would be totally different as we have a
> compilation stage. If anything, it is reimplementing Fortran, however
> Fortran desperately need more alternatives :-)
>
> As for the library, the CEP scetches full interopability with the
> library, you would likely still do
>
> arr = np.dot(arr,arr)
>
> for matrix mul.
>
> What to include? I think slicing, arithmetic (with "broadcasting", i.e
> dimensions of length 1 are repeated to make arrays conform), transpose
> (arr.T). No "sum()" member function or similar.

That sound reasonable.

> In addition perhaps high-level utilities for looping, such as
> "ndenumerate" etc:
>
> cdef int[:,:] arr = ...
> for (i, j), value in cython.array.ndenumerate(arr):
>      ...
>
> These would be quite nicely contained in cython.array though (and  
> again
> have no direct NumPy link that's not horribly slow due to Python-level
> iterators).

Something like this could certainly benefit from compile-time  
optimizations when the dimensions are fixed.

>
>
>>
>> On Jun 10, 2009, at 11:57 AM, Dag Sverre Seljebotn wrote:
>>
>>> Brian Granger wrote:
>>>> Dag,
>>>>
>>>> I quickly glanced through the proposal and have two big picture
>>>> questions:
>>>>
>>>> * What will this make possible that is currently not possible?
>>
>> This was originally my first question too, but you beat me to it.
>>
>>> 1) Efficient slices
>>
>> Is the inefficiency just in the object creation and Python indexing
>> semantics? It's still O(1), right? Same with the other operations. (I
>> guess there's also a question of result type.)
>
> Well, and then the resulting buffer is acquired and checked for data
> type. But essentially, yes.
>
> This is purely a matter of notational convenience -- will it hurt to
> have that O(1) in your almost-inner-loop taking a slice of 50  
> elements,
> or do you pass "start", "end" around in your function?

Your inner loop would typically be over largish arrays, so it  
probably wouldn't matter much, but I see your point.

>>> 2) Leave the road open for memory bus friendlier arithmetic (re:  
>>> Hoyt
>>> Koepke's project proposal)
>>
>> Could you provide a bit more of a concrete example here? There is
>> avoiding allocation of temporary arrays, is there more?
>
> Most systems today are for simple functions like addition  
> constrained by
> memory bus speed, not CPU speed.

Yep.

> Consider
>
> B = A + A + A + A + A + A
>
> With NumPy, the data of A has to travel over the bus 6 times, with  
> loop
> unrolling only 1, as it would turn into
>
> B = new memory
> for idx in ...
>     B[idx] = A[idx] + A[idx] + ...
>
> This can mean a dramatic speed increase as data doesn't have to enter
> the cache more than once.

Well, it's unlikely that you would have 6 copies of the same data to  
add, but it would save on store/load in the target array. The ability  
to combine multiple operations into a single pass is a powerful one.

> (NumPy can't do this, but there is numexpr, which allow this:
>
> B = numexpr.eval("A+A+...")
>
> using bytecode interpreter and working in cache-sized blocks.)

Didn't know about that, that's kind of neat. Have you looked at how  
it works for ideas?

>>> 3) Automatic contiguous copies in/out if a function is coded to
>>> work on
>>> contiguous memory
>>>
>>> Why can't this be done currently?
>>>
>>>  * Cython contains no in-compiler support for NumPy, and so cannot
>>> know
>>> how to create new underlying ndarray objects.
>>>
>>>  * All optimizations need to hard-code semantics at compile-time.  
>>> With
>>> single-element indexing it seemed fair to assume the usual
>>> semantics of
>>> zero-based indexing etc., but with slices tings get worse (which
>>> kind of
>>> object is returned) and with arithmetic downright impossible (what
>>> does *
>>> do again?)
>>>
>>> That's not to say there's not other options:
>>> 1) We could hard-code support for NumPy only, and only allow
>>> ndarray and
>>> not subclasses thereof.
>>>
>>> 2) We could invent some new protocol/syntax for defining compile- 
>>> time
>>> semantics for all relevant operations.
>>
>> Here I am torn--I don't like defining compile-time semantics because
>> it goes against the whole OO style of inheritance (and feels even
>> more remote than the very dynamic, late-binding Python runtime). I
>> don't like option (1) either though.
>>
>> Another an idea, have you thought of using NumPy as the backend? I.e.
>> an int[:,:] is any bufferinfo--supporting object, but if one needs to
>> be created you create it via an ndarray? This could (potentially)
>> facilitate a lot more code reuse (especially for operations that are
>> more complicated than a single loop over the data). (Might be messier
>> than implementing int[:,:] directly though.) Suppose one develops a
>> vectorized version of ndarrays, could that be a drop-in replacement?
>
> ? ndarrays are vectorized? But subclasses may not be.

I'm just thinking about taking advantage of improvements/ 
optimizations from ndarrays. But a lot of the stuff you're talking  
about is clearly and optimally O(n) so this is less of an issue  
(especially if you make it pluggable to support future parallelization).

> I've thought about it. If arithmetic is implemented on arrays, the  
> only
> thing that's seems to be reused this way is malloc/free of a memory  
> area
> though. Better loose the dependency then.

OK, makes sense. Wanted to know your thoughts at least.

- Robert


From dagss at student.matnat.uio.no  Fri Jun 12 11:37:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 11:37:28 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <5FFC2397-4769-49E3-9E00-04D22DDC4E1F@math.washington.edu>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>	<52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>	<4A32042A.20709@student.matnat.uio.no>
	<5FFC2397-4769-49E3-9E00-04D22DDC4E1F@math.washington.edu>
Message-ID: <4A3221D8.6080000@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 12, 2009, at 12:30 AM, Dag Sverre Seljebotn wrote:
>
>   
>
>> In addition perhaps high-level utilities for looping, such as
>> "ndenumerate" etc:
>>
>> cdef int[:,:] arr = ...
>> for (i, j), value in cython.array.ndenumerate(arr):
>>      ...
>>
>> These would be quite nicely contained in cython.array though (and  
>> again
>> have no direct NumPy link that's not horribly slow due to Python-level
>> iterators).
>>     
>
> Something like this could certainly benefit from compile-time  
> optimizations when the dimensions are fixed.
>   
Actually, high-level iteration can be done quite fast when the ndims are 
not fixed too, but it's a long story (main idea is to have slow code 
extract the dimension with the lowest strides, and then have an inner 
for-loop set up over pointers/increments set up by the slower code). I'm 
thinking of "int[...]" syntax for what, which will allow all operations 
except item indexing.

>> Consider
>>
>> B = A + A + A + A + A + A
>>
>> With NumPy, the data of A has to travel over the bus 6 times, with  
>> loop
>> unrolling only 1, as it would turn into
>>
>> B = new memory
>> for idx in ...
>>     B[idx] = A[idx] + A[idx] + ...
>>
>> This can mean a dramatic speed increase as data doesn't have to enter
>> the cache more than once.
>>     
>
> Well, it's unlikely that you would have 6 copies of the same data to  
> add, but it would save on store/load in the target array. The ability  
> to combine multiple operations into a single pass is a powerful one.
>   
I just see that people do use numexpr to get around it.

>> (NumPy can't do this, but there is numexpr, which allow this:
>>
>> B = numexpr.eval("A+A+...")
>>
>> using bytecode interpreter and working in cache-sized blocks.)
>>     
>
> Didn't know about that, that's kind of neat. Have you looked at how  
> it works for ideas?
>   
1) Parses expression tree
2) Repeatedly executes expression tree on ideally-sized sub-chunks of 
the arrays

I.e. it balances array block size (and cache limit) vs. the time it 
takes to execute an expression tree, in order to stay within cache but 
divide the running time of the latter with a big constant factor. With 
Cython there's no need for this as the expression tree is in the format 
the CPU speaks.

Now, it might do SSE etc. as well, that's where I see the plugins, and 
code properly refactored from numexpr could fit in there. Long-term we 
might see the recent NumPy CorePy effort kick in -- CorePy can compile 
code into memory (for use with NumPy) or to assembly text file (for 
linking into Cython code through a plugin).

It would be neat to fold numexpr into Sage BTW :-), i.e.

x, y = var("x, y")
integrate(x**2 + y**2 + x, x)(somenumpyarray)

and run that through numexpr for more efficient evaluation. If I'm to do 
anything myself, I'm as likely to contribute that /before/ working on 
arithmetic in Cython *shrug*.

But even if dynamic expressions in Sage becomes a major mode of working 
with numerics, I still think there's always going to be room for Cython 
to provide honest, no-frills compiled code. And it is anyway likely 
needed to "embrace and extend" Fortran workflow that people are used to.

I wanted to ask you about fasteval in Sage -- I think it does something 
similar (with other types), but I can't seem to find any docs on it? 
Which source file should I look in?

Dag Sverre

From roland at utk.edu  Fri Jun 12 16:11:24 2009
From: roland at utk.edu (Roland Schulz)
Date: Fri, 12 Jun 2009 10:11:24 -0400
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3175A3.8010304@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de> 
	<4A314E72.6000500@student.matnat.uio.no> <4A315250.7080606@behnel.de> 
	<4A3155E3.908@student.matnat.uio.no> <4A315882.4030305@behnel.de> 
	<4A315ABD.3020806@behnel.de> <4A31629F.8000608@student.matnat.uio.no> 
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com> 
	<4A3175A3.8010304@student.matnat.uio.no>
Message-ID: <c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>

On Thu, Jun 11, 2009 at 5:22 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Thanks for your input!, you definitely know more about such computations
> than me.
>
> Roland Schulz wrote:
> >
> >
> > On Thu, Jun 11, 2009 at 4:01 PM, Dag Sverre Seljebotn
> > <dagss at student.matnat.uio.no <mailto:dagss at student.matnat.uio.no>>
> wrote:
>
> >     Right :-) Now linear algebra support in the compiler *would* be
> taking
> >     things quite overboard (and probably only slow things down, unless N
> is
> >      really small, like 4x4 3D game matrices...), I'm actually against
> that.
> >
> >
> >
> > Well only if you do it directly in a hard-coded datatype.
> >
> > Why not having a non numeric base class as Stefan proposed. And then
> > allowing the operator functions be handled by Eigen/Blitz. This would
> > enable very fast numerics, including SSE and fast linear algebra. And
> > without having it as tightly coupled in the main language. Similar in
> > that expect to weave, but with all the advantages of cython.
>
> I'll answer this below.
>
> >
> > Component wise operations without optimization (thus collapsing
> > d=a*b*c*d into one loop instead of 3 and not using temporary arrays)
> > does not give you any speed-up over Numpy for vectorized code with large
> > arrays.
> >
> > For vectorized Numpy code the bottleneck is not the call from Python to
> > C, but the inefficient use of cache because of the temporary arrays.
>
> I don't know enough about this, but these two paragraphs seem slightly
> contradictory to me.


I think it is important to see which features of this proposals are possible
but will only be included in the unforeseeable (=may be never) future and
what could/will be done soon.
Of course this depends on how much time there is so it is difficult to say.

By point is, if collapsing d=a*b*c*d into one loop is possible, but is not
clear when it will be added, this proposal won't speed up calculations over
vectorized numpy for the foreseeable furture.

Or putting it another way: Introducing only element-wise access won't help
performance by itself, so performance wise it is not a meaningful
intermediate step.

Also this optimization is not straight forward. If you look at
http://eigen.tuxfamily.org/index.php?title=Benchmark-August2008

you see that many libraries trying to do even simple Y+=alpha X do a bad job
performance wise.
And having to do all the required optimization in the cython compiler would
basicly mean rewriting the Frotran Compiler. And of course the language is
suboptimal but the compiler can be sometimes rather smart. And putting all
this in the Cython compiler sounds like awful a lot of work.


>
> Anyway: Any builtin features as such in Cython can't really depend on
> Eigen/Blitz. (I know this is perhaps not what you are suggesting, but
> bear with me.)
>
> As the CEP 517 says, I see this as a two-step process:
>
> 1) Naive C loops produced by Cython, which will be "good enough" for
> many cases, and which gives a reference spec and implementation which
> runs out of the box without library requirements.
>
> 2) Various plugins (with a seperate release process from Cython so that
> work in this area doesn't bog down Cython development), using e.g.
> Eigen/Blitz as backends, or run it on a GPU, in parallell using OpenMP
> (well, for heavy componentwise functions), etc. etc.


Why does it bog down Cython delvelopment if it is a library with ships with
cython? Also you can always require certain version of the external
components (and make it optional for complication for people not interested
in numerics).



>
> >
> > And I don't think you need a full metalanguage to map the array class to
> > Eigen/Blitz. Though it would be still be couple somewhat to the main
> > language otherwise I agree you would need a metalanguage or full
> > template support. But it seems to me that there should be an optimum
> > between full template support and doing the numerics directly in a
> > language specific numeric array type. Sorry for being vage in this
> > central point. I just don't know the cython details here.
>
> Well, the point here is that when you use Eigen/Blitz, what you do is
> use the metalanguage of C++. Which is much more powerful than anything
> we're likely to get in Cython. So in some sense you use C++ to do what
> you cannot do in Cython.


Exactly you have rather easy usage of Eigen/Blitz and only that code has to
be generated by Cython. Then the C++ compiler takes care of the expression
templates.

Roland


-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090612/ae40cdf2/attachment.htm 

From dagss at student.matnat.uio.no  Fri Jun 12 17:59:21 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 17:59:21 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
	<c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
Message-ID: <4A327B59.7010406@student.matnat.uio.no>

I think I agree more with you than you give me credit for...

Roland Schulz wrote:
> I think it is important to see which features of this proposals are 
> possible but will only be included in the unforeseeable (=may be never) 
> future and what could/will be done soon.
> Of course this depends on how much time there is so it is difficult to say.

OK I'll by crystal clear about this: In autumn,

    cdef int[:] a = ..., b = ...
    a = a + b

will be a syntax error. The main motivation for adding this type now is 
to make it easier to pass array data between Python objects (PEP 3118) 
and external libraries.

However, adding a new, native type is not something you do without 
thinking things through, and discussing possibilities for arithmetic 
operators was important to get the whole picture of "what kind of type" 
this is and what design and implementation considerations to take.

Also there was the issue of the whole direction of Cython in the area of 
numeric programming which has now largely been resolved. This also means 
that

cdef ndarray[int] a = ..., b = ...
a = a + b

will never be optimized -- because it is vital that such code allows 
arbitrary subclasses of ndarray to overload the operators.

> By point is, if collapsing d=a*b*c*d into one loop is possible, but is 
> not clear when it will be added, this proposal won't speed up 
> calculations over vectorized numpy for the foreseeable furture.
> 
> Or putting it another way: Introducing only element-wise access won't 
> help performance by itself, so performance wise it is not a meaningful 
> intermediate step.

I agree, what it is needed for is more convenient "raw" handling of PEP 
3118 without invoking NumPy to do it (and NumPy doesn't handle all the 
memory layouts that PEP 3118 handle).

BUT: The code needed to e.g. pass Python Imaging Library images to 
Fortran code is 50% of the way towards what it takes to implement naive 
componentwise operators.


> Also this optimization is not straight forward. If you look at
> http://eigen.tuxfamily.org/index.php?title=Benchmark-August2008
> 
> you see that many libraries trying to do even simple Y+=alpha X do a bad 
> job performance wise.
> And having to do all the required optimization in the cython compiler 
> would basicly mean rewriting the Frotran Compiler. And of course the 
> language is suboptimal but the compiler can be sometimes rather smart. 
> And putting all this in the Cython compiler sounds like awful a lot of work.

I completely agree!! -- I don't see Cython doing the optimizations 
itself, but make use of external libraries (like eigen) through plugins. 
The question is how this is facilitated!

I.e. you could probably define type after the C++ integration is more 
mature that allows you to do

     cdef eigen.vector a = ..., b = ... # no pun intended
     a = a + b # faster than anything else

but can you transparently convert those to and from PIL images, or pass 
them to a SciPy/NumPy function?

That's the issues that are really at stake here.

>     2) Various plugins (with a seperate release process from Cython so that
>     work in this area doesn't bog down Cython development), using e.g.
>     Eigen/Blitz as backends, or run it on a GPU, in parallell using OpenMP
>     (well, for heavy componentwise functions), etc. etc.
> 
> 
> Why does it bog down Cython delvelopment if it is a library with ships 
> with cython? Also you can always require certain version of the external 
> components (and make it optional for complication for people not 
> interested in numerics).

What I meant to say is that we must avoid situations like this:

Dag Sverre, year 2010: "Please wait with releasing Cython 1.3 until I'm 
done supporting the next version of Eigen"

So plugins have "their own release process" in the sense that Cython can 
be released at any time shipping a previous, stable version of the plugin.

But this is premature discussion ... :-)


>     Well, the point here is that when you use Eigen/Blitz, what you do is
>     use the metalanguage of C++. Which is much more powerful than anything
>     we're likely to get in Cython. So in some sense you use C++ to do what
>     you cannot do in Cython.
> 
> 
> Exactly you have rather easy usage of Eigen/Blitz and only that code has 
> to be generated by Cython. Then the C++ compiler takes care of the 
> expression templates.

Generating the code to use Eigen/Blitz and generating naive loops are 
probably about the same amount of work, and probably something like 70% 
overlap. I see no reason for not doing both, if only to avoid having to 
say "you need C++ to use this feature of the language".

Really, neither are rocket science, but all the "buffer interopability 
infrastructure" needs to be there first.

-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Jun 12 17:31:15 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Jun 2009 17:31:15 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31629F.8000608@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>	<4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
Message-ID: <4A3274C3.1050204@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>>> Ah, so all you really want is a SIMD data type, right?
> 
> Right :-) Now linear algebra support in the compiler *would* be taking 
> things quite overboard (and probably only slow things down, unless N is 
>   really small, like 4x4 3D game matrices...), I'm actually against that.
> 
>> ... which would then also support parallel operations as in
>>
>> 	some_int_array *= 2
>>
>> to multiply all entries by two, and
> 
> That's actually an example in the CEP :-)

Yes, and I skipped over it because that was a tiny section titled
"numerical calculations". It was far from obvious to me that that was the
main point of the proposal.


>> 	some_float_array = math.sqrt(some_int_array)
>>
>> to take the square-root of all entries in parallel?
> 
> Well, the issue here is function namespaces -- do we "misuse" the Python 
> library like this?

I was more thinking of math.h here, i.e. C functions.


> But the idea -- certainly. At least it should be easy 
> to call C stdlib's sin in parallell on a double array.
> 
> And it should be possible to create custom functions to call in 
> parallel, either by
> 
> cdef extern from "stdlib.h":
>      double sin(double)
> cdef myfunc(double x): ...
> myfunc(sin(arr)) # SIMD, by convention, as we know they
>                   # take single element

Are you sure you always know where the parallelism ends?


> or, if people think that is too magic, by a decorator:
> 
> cdef extern from "stdlib.h":
>      @cython.array.elementwise
>      double sin(double)
> @cython.array.elementwise
> cdef myfunc(double x): ...

Not if we can avoid it in any way.


> I suggest we deal with this particular issue much later though -- for 
> now it's nice to have a roadmap and figure out the syntax for 
> concatenateable vs. SIMD, but any actual work on componentwise 
> operations will happen after everything else works.

I don't have the impression that this proposal is accepted yet.

Stefan


From dagss at student.matnat.uio.no  Fri Jun 12 19:55:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 19:55:50 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3274C3.1050204@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>	<4A315ABD.3020806@behnel.de>	<4A31629F.8000608@student.matnat.uio.no>
	<4A3274C3.1050204@behnel.de>
Message-ID: <4A3296A6.4030404@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>>> Ah, so all you really want is a SIMD data type, right?
>> Right :-) Now linear algebra support in the compiler *would* be taking 
>> things quite overboard (and probably only slow things down, unless N is 
>>   really small, like 4x4 3D game matrices...), I'm actually against that.
>>
>>> ... which would then also support parallel operations as in
>>>
>>> 	some_int_array *= 2
>>>
>>> to multiply all entries by two, and
>> That's actually an example in the CEP :-)
> 
> Yes, and I skipped over it because that was a tiny section titled
> "numerical calculations". It was far from obvious to me that that was the
> main point of the proposal.

It was intended as a demonstration of what went before. Anyway, 
communication is hard. Sorry.

(When the dust has settled I'll revise the CEP.)

(I wouldn't say the main point of the proposal as such, but it would be 
the main reason why it would make sense to just as well make this a 
builtin feature and not just something in a library, due to 
implementation constraints.)

>> And it should be possible to create custom functions to call in 
>> parallel, either by
>>
>> cdef extern from "stdlib.h":
>>      double sin(double)
>> cdef myfunc(double x): ...
>> myfunc(sin(arr)) # SIMD, by convention, as we know they
>>                   # take single element
> 
> Are you sure you always know where the parallelism ends?

If a function takes "object", you have to assume it means to take the 
whole array and not its elements. I think that is it. (There are pretty 
standard rules for how to deal with multiple arguments etc. too).

>> or, if people think that is too magic, by a decorator:
>>
>> cdef extern from "stdlib.h":
>>      @cython.array.elementwise
>>      double sin(double)
>> @cython.array.elementwise
>> cdef myfunc(double x): ...
> 
> Not if we can avoid it in any way.

Well that's how you'd do it in Python (given the right elementwise 
decorator).

>> I suggest we deal with this particular issue much later though -- for 
>> now it's nice to have a roadmap and figure out the syntax for 
>> concatenateable vs. SIMD, but any actual work on componentwise 
>> operations will happen after everything else works.
> 
> I don't have the impression that this proposal is accepted yet.

I had that impression until now (except for the syntax bit, where you 
wanted to use int[:] for a list-style type. Or are you suggesting to use 
+ for multi-dimensional concatenation as well?)

I guess that was before we cleared up the SIMD misunderstanding though. 
So I'll revise my impression -- obviously, if you say so, there's still 
problems.

Rather than us discussing this I think it might be more constructive 
though if you comment on Robert's first email about this yesterday? He 
tends to put things in a different way from me.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Jun 13 07:33:38 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Jun 2009 07:33:38 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
	<c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
Message-ID: <4A333A32.4040904@behnel.de>


Roland Schulz wrote:
> I think it is important to see which features of this proposals are possible
> but will only be included in the unforeseeable (=may be never) future and
> what could/will be done soon.
> Of course this depends on how much time there is so it is difficult to say.
> 
> By point is, if collapsing d=a*b*c*d into one loop is possible, but is not
> clear when it will be added, this proposal won't speed up calculations over
> vectorized numpy for the foreseeable furture.
> [...]
> Why does it bog down Cython delvelopment if it is a library with ships with
> cython? Also you can always require certain version of the external
> components (and make it optional for complication for people not interested
> in numerics).

I think the main question then is: what would be the minimum required
change to the compiler that would enable providing these things as external
modules?

Stefan

From dagss at student.matnat.uio.no  Sat Jun 13 09:29:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 13 Jun 2009 09:29:44 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A333A32.4040904@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de> <4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
	<c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
	<4A333A32.4040904@behnel.de>
Message-ID: <afd1f743de6fa54a009317d1c95bc03a.squirrel@webmail.uio.no>

Stefan Behnel wrote:
> Roland Schulz wrote:
>> I think it is important to see which features of this proposals are
>> possible
>> but will only be included in the unforeseeable (=may be never) future
>> and
>> what could/will be done soon.
>> Of course this depends on how much time there is so it is difficult to
>> say.
>>
>> By point is, if collapsing d=a*b*c*d into one loop is possible, but is
>> not
>> clear when it will be added, this proposal won't speed up calculations
>> over
>> vectorized numpy for the foreseeable furture.
>> [...]
>> Why does it bog down Cython delvelopment if it is a library with ships
>> with
>> cython? Also you can always require certain version of the external
>> components (and make it optional for complication for people not
>> interested
>> in numerics).
>
> I think the main question then is: what would be the minimum required
> change to the compiler that would enable providing these things as
> external
> modules?

Provided that everything in the proposal is implemented in core Cython
(i.e. naive SIMD loops are implemented as a simple fallback mechanism
etc.), and it is done in a way using seperate node types (e.g
SIMDArithmeticNode), I was thinking to just have a pluggable pipeline, and
allow plugins to provide transforms and positions in the pipeline to plug
them in. (Oh, and plugin transforms could insert their own nodes too.)

This would require plugin developers to follow Cython node tree changes
(which one wouldn't promise to keep stable).

I'm not sure whether such a thing would work or whether there will be
requests of the form "can you just change this thing in this node before
it gets to our plugin transform". But I think that's a luxury problem --
if people do develop such plugins, it means more eyes on the Cython
codebase too.

Dag Sverre


From dagss at student.matnat.uio.no  Sat Jun 13 12:40:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 13 Jun 2009 12:40:26 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3274C3.1050204@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>	<4A315ABD.3020806@behnel.de>	<4A31629F.8000608@student.matnat.uio.no>
	<4A3274C3.1050204@behnel.de>
Message-ID: <4A33821A.9020107@student.matnat.uio.no>

Stefan Behnel wrote:
> I don't have the impression that this proposal is accepted yet.

I looked it over and here's way too much in the proposal that didn't 
communicate to people who are not already avid NumPy users. Sorry about 
that.

Within a few days I'll rewrite the CEP (perhaps split it in two), sum up 
the discussion, and we can start over properly then.

-- 
Dag Sverre

From darkgl0w at yahoo.com  Mon Jun 15 12:18:06 2009
From: darkgl0w at yahoo.com (Cristi Constantin)
Date: Mon, 15 Jun 2009 03:18:06 -0700 (PDT)
Subject: [Cython] support for numpy bool type / unicode type
Message-ID: <707613.24072.qm@web52108.mail.re2.yahoo.com>

Good day.
I was following this discussion too.
How about unicode_t ?
I am trying to make the numpy tutorial you have on Cython Wiki, but instead of np.int_t i need numpy.unicode_ type.
Do you have any idea how i could do that?

Check this python code:
import numpy as np
a=np.array([0,1,2],np.unicode_)
print a # array([u'0', u'1', u'2'], dtype='<U1')
a[0]=u'\u2588'
a[1]=u'\u00b6'
a[2]=u'\u2248'
a.astype('I')
# UnicodeEncodeError: 'decimal' codec can't encode character u'\u2588' in position 0: invalid decimal Unicode string
a.astype(np.uint32)
# Same unicode error.

What's the best "cast" i can do to have my Unicode Numpy Array?
Thank you in advance.

--- On Tue, 6/9/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:

From: Robert Bradshaw <robertwb at math.washington.edu>
Subject: Re: [Cython] support for numpy bool type
To: cython-dev at codespeak.net
Date: Tuesday, June 9, 2009, 1:38 AM

On Jun 6, 2009, at 1:22 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
>>
>>> Eric Firing wrote:
>>>> Eric Firing wrote:
>>>>> In writing a cython extension to work with numpy masked arrays, I
>>>>> needed to work with the mask, which is dtype('bool').? Therefore I
>>>>> was expecting to be able to use np.bool_t, in analogy to? 
>>>>> np.float_t.
>>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>>> python wrapper.? Can bool_t be added to numpy.pxd?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Eric
>>> The reason it is not there is that AFAIK Cython has no support? 
>>> for an
>>> 8-bit boolean type. I.e. you want
>>>
>>> print arr[3] # should print "True", not 1.
>>>
>>> and furthermore you want
>>>
>>> arr[3] = obj # should mean arr[3] = bool(obj)
>>>
>>> Which brings up the questions
>>> 1) I suppose the best thing would be to add support for a "bool"? 
>>> which
>>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>>> appropriate restrictions otherwise. Thoughts?
>>
>> How would this relate to the current bint type? bint is an int
>> because logical operations in C are ints, and also several operations
>> in the Python/C API (and elsewhere) return true/false values as ints.
>
> Well, bint is an int that has different semantics for conversion to/ 
> from
> Python,

That is exactly what it is.

> and that seems to be needed here as well (e.g. a "bchar", which I
> think is essentially C99 _Bool).

a C99 _Bool can only have two values, anything non-zero is converted? 
to exactly 1.

> What happens if you do "cdef bint value = 4"? Is bool(4) called? 
> resulting
> in 1 being placed in value? I think that's the wanted behaviour? 
> here...

No, it retains the value 4. (This is one of the reasons I didn't call? 
it a bool.)

> Using bint directly seems to be right out though as one must access? 
> the
> data using an 8-bit integer type. (Well, I could hard-code buffers? 
> to cast
> back and forth between "bint" and "char", after the pointer? 
> dereference,
> but that seems very unclean compared to introducing a new type).
>
> Perhaps we should have "char bint", "short bint", "unsigned long long
> bint" and so on :-) Really, the need for a seperate type for bools? 
> is a
> strict Cython feature, because what it affects is Python object
> conversion, so it doesn't hurt IMO that these are not in C. And? 
> size and
> booleanness are orthogonal features. (But, this is a puristic approach
> only, and in practice "bchar" would suffice.)

Actually, that might work well. When I first introduced it, I saw no? 
need to offer various sizes, but if one has arrays of them then it? 
becomes important. And just when we thought we were simplifying the? 
integer type system... :)

- Robert




      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090615/d0929800/attachment.htm 

From dagss at student.matnat.uio.no  Mon Jun 15 16:28:45 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 15 Jun 2009 16:28:45 +0200
Subject: [Cython] support for numpy bool type / unicode type
In-Reply-To: <707613.24072.qm@web52108.mail.re2.yahoo.com>
References: <707613.24072.qm@web52108.mail.re2.yahoo.com>
Message-ID: <4A365A9D.8040101@student.matnat.uio.no>

Cristi Constantin wrote:
> Good day.
> I was following this discussion too.
> How about unicode_t ?
> I am trying to make the numpy tutorial you have on Cython Wiki, but 
> instead of np.int_t i need numpy.unicode_ type.
> Do you have any idea how i could do that?
> 
> Check this python code:
> import numpy as np
> a=np.array([0,1,2],np.unicode_)
> print a # array([u'0', u'1', u'2'], dtype='<U1')
> a[0]=u'\u2588'
> a[1]=u'\u00b6'
> a[2]=u'\u2248'
> a.astype('I')
> # UnicodeEncodeError: 'decimal' codec can't encode character u'\u2588' 
> in position 0: invalid decimal Unicode string
> a.astype(np.uint32)
> # Same unicode error.
> 
> What's the best "cast" i can do to have my Unicode Numpy Array?
> Thank you in advance.

This is very badly supported in Cython, mostly because I have no idea 
what usecases people have for such arrays. Would you care to describe 
your usecase?

But if you want to modify Unicode 1-length strings as an integer type, 
you can do (in Cython)

cdef np.ndarray[some_int_type, cast=True] arr = a

Here, some_int_type needs to be an integer of the same size as 
"a.itemsize" would give you above -- I don't know how NumPy stored 
Unicode strings so I actually can't tell you the right one.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jun 15 18:12:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 15 Jun 2009 18:12:28 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
Message-ID: <4A3672EC.90002@student.matnat.uio.no>

Thanks to everybody who contributed to the discussion on a Cython array 
type last week! Here's a summary to attempt focusing the discussion.

There are now two CEPs:
- CEP 517, array type: http://wiki.cython.org/enhancements/array
- CEP 518, SIMD operations: http://wiki.cython.org/enhancements/simd

I mostly just added a "what does this facilitate"-section is added near 
the beginning of each, and the multidimensional aspect of the arrays has 
been emphasised. No need to reread it.

The story is this:

  a) I can only commit to implementing CEP 517 this time around

  b) The arrays themselves/CEP 517 can without loosing *too* much be 
done as a library (though I would prefer them as builtins myself). 
However SIMD/CEP 518 depends on building the arrays into the language, 
and switching approaches later would mean redoing some stuff. Therefore 
SIMD (and the numerical roadmap in general) keeps popping into the 
discussion.

  c) It would be nice to finally settle a roadmap for numerics in Cython 
(how far will the language be bent for the benefit of numerics), for 
instance to economize on discussions when questions pop up on the 
mailing list (and make plans). These two CEPs would create one such roadmap.

  d) CEP 518 isn't a lot of work (totally doable in a GSoC next 
year...though of course not at the expense of e.g. a full Python 
compatability GSoC or similar).

Key impressions from last week:

  a) Numerical users seem positive about all of this.
  b) Robert Bradshaw said:

I am not opposed to this feature, despite it being off the path of
being a good, optimizing Python compiler with easy glue to external
code, as long as it doesn't intrude on the language too much. Mostly,
this is because there is so much interest in using Cython in a
numerical processing context. I do, however, have some reservations.

[I believe the reservations were all taken care of.]

  c) I'm not sure about Stefan Behnel's current opinions about this.

-- 
Dag Sverre

From daniel.yarlett at gmail.com  Tue Jun 16 03:50:16 2009
From: daniel.yarlett at gmail.com (Daniel Yarlett)
Date: Tue, 16 Jun 2009 02:50:16 +0100
Subject: [Cython] Cython and Numpy
Message-ID: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>

Hi All,

I am porting some code which uses Cython and Numpy from Ubuntu to OS  
X, and have run into some problems getting the Cython compilation  
process working. I am using distutils with the following in my  
setup.py file:

> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
>    cmdclass = {'build_ext': build_ext},
>    ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"])]
> )

neuralRoutines.pyx imports and cimports numpy, amongst other things. I  
don't think the problem lies in there because it runs fine as it is  
under Ubuntu.

When I run "python setup.py build_ext --inplace" I get the following  
error messages:

> Yarletts-Computer:scorelineModels yarlett$ python setup.py build_ext  
> --inplace
> running build_ext
> building 'neuralRoutines' extension
> creating build
> creating build/temp.macosx-10.3-i386-2.6
> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - 
> fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/ 
> Frameworks/Python.framework/Versions/2.6/include/python2.6 -c  
> neuralRoutines.c -o build/temp.macosx-10.3-i386-2.6/neuralRoutines.o
> neuralRoutines.c:136:31: error: numpy/arrayobject.h: No such file or  
> directory
> neuralRoutines.c:136:31: error: numpy/arrayobject.h: No such file or  
> directory
> neuralRoutines.c:659: error: syntax error before  
> ?__pyx_t_5numpy_int8_t?neuralRoutines.c:659: error: syntax error  
> before ?__pyx_t_5numpy_int8_t?
> neuralRoutines.c:659: warning: data definition has no type or  
> storage class
> neuralRoutines.c:661: error: syntax error before  
> ?__pyx_t_5numpy_int16_t?
> neuralRoutines.c:661: warning: data definition has no type or  
> storage class
> neuralRoutines.c:663: error: syntax error before  
> ?__pyx_t_5numpy_int32_t?
> neuralRoutines.c:663: warning: data definition has no type or  
> storage class

It looks like arrayobject.h is not being found, so do I need to modify  
my setup.py file somehow? I'm afraid I'm not sure how to do this, so  
any help would be greatly appreciated.

When I search for arrayobject.h it can be found in the following  
places, amongst others, on my system:

> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.4.0.dev6728-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h

Many thanks,

Dan.

From darkgl0w at yahoo.com  Tue Jun 16 14:06:38 2009
From: darkgl0w at yahoo.com (Cristi Constantin)
Date: Tue, 16 Jun 2009 05:06:38 -0700 (PDT)
Subject: [Cython] Cython and Numpy
Message-ID: <357936.47971.qm@web52101.mail.re2.yahoo.com>

Good day.

You need to copy all .c files from your numpy installation (on windows it's "C:\Python25\Lib\site-packages\numpy\core\include\numpy\") into MingW\Include\numpy.

gcc can't find your arrayobject.h in python directory.

Have a nice day.

--- On Mon, 6/15/09, Daniel Yarlett <daniel.yarlett at gmail.com> wrote:

It looks like arrayobject.h is not being found, so do I need to modify? 
my setup.py file somehow? I'm afraid I'm not sure how to do this, so? 
any help would be greatly appreciated.

When I search for arrayobject.h it can be found in the following? 
places, amongst others, on my system:

> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.4.0.dev6728-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h




      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090616/86965c96/attachment.htm 

From brett.calcott at gmail.com  Tue Jun 16 14:15:52 2009
From: brett.calcott at gmail.com (Brett Calcott)
Date: Tue, 16 Jun 2009 22:15:52 +1000
Subject: [Cython] Cython and Numpy
In-Reply-To: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
References: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
Message-ID: <c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>

Hi Daniel,


2009/6/16 Daniel Yarlett <daniel.yarlett at gmail.com>:
> ... SNIP ...
> It looks like arrayobject.h is not being found, so do I need to modify
> my setup.py file somehow? I'm afraid I'm not sure how to do this, so
> any help would be greatly appreciated.


This line in my .bashrc solves the same problem on my Mac

export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include

Or, I think you could add the same path to your definition of the
Extension() in setup.py. I think you can do this:

ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"],
        extra_compile_args=['-Ithe/path/you/want/here'],
)]

... but I haven't checked it.

>
> When I search for arrayobject.h it can be found in the following
> places, amongst others, on my system:
>
>> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
>> packages/numpy/core/include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
>> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/
>> include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
>> packages/numpy/core/include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
>> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/
>> include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
>> packages/numpy-1.4.0.dev6728-py2.5-macosx-10.3-i386.egg/numpy/core/
>> include/numpy/arrayobject.h

It looks like you've got a few versions installed. You'd best figure out
which one you are actually using, and delete the rest...

Brett

From dagss at student.matnat.uio.no  Tue Jun 16 15:07:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Jun 2009 15:07:48 +0200
Subject: [Cython] Cython and Numpy
In-Reply-To: <c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>
References: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
	<c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>
Message-ID: <4A379924.8040005@student.matnat.uio.no>

Brett Calcott wrote:
> Hi Daniel,
> 
> 
> 2009/6/16 Daniel Yarlett <daniel.yarlett at gmail.com>:
>> ... SNIP ...
>> It looks like arrayobject.h is not being found, so do I need to modify
>> my setup.py file somehow? I'm afraid I'm not sure how to do this, so
>> any help would be greatly appreciated.
> 
> 
> This line in my .bashrc solves the same problem on my Mac
> 
> export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include
> 
> Or, I think you could add the same path to your definition of the
> Extension() in setup.py. I think you can do this:
> 
> ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"],
>         extra_compile_args=['-Ithe/path/you/want/here'],
> )]
> 
> ... but I haven't checked it.

The usual way of doing this is

import numpy
...
setup(
    ...
    ext_modules = [Extension(..., include_path=[numpy.get_include()])]
)

No need to alter include paths etc. then.

(It would be great if somebody could take the time to add this question 
to the FAQ and/or the Cython/NumPy tutorial, it comes up a lot.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun 16 15:08:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Jun 2009 15:08:42 +0200
Subject: [Cython] Cython and Numpy
In-Reply-To: <4A379924.8040005@student.matnat.uio.no>
References: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
	<c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>
	<4A379924.8040005@student.matnat.uio.no>
Message-ID: <4A37995A.9040109@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Brett Calcott wrote:
>> Hi Daniel,
>>
>>
>> 2009/6/16 Daniel Yarlett <daniel.yarlett at gmail.com>:
>>> ... SNIP ...
>>> It looks like arrayobject.h is not being found, so do I need to modify
>>> my setup.py file somehow? I'm afraid I'm not sure how to do this, so
>>> any help would be greatly appreciated.
>>
>>
>> This line in my .bashrc solves the same problem on my Mac
>>
>> export 
>> C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include
>>
>> Or, I think you could add the same path to your definition of the
>> Extension() in setup.py. I think you can do this:
>>
>> ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"],
>>         extra_compile_args=['-Ithe/path/you/want/here'],
>> )]
>>
>> ... but I haven't checked it.
> 
> The usual way of doing this is
> 
> import numpy
> ...
> setup(
>    ...
>    ext_modules = [Extension(..., include_path=[numpy.get_include()])]
> )

Sorry, that should be include_dirs, not include_path.

-- 
Dag Sverre

From ellisonbg.net at gmail.com  Tue Jun 16 19:20:57 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 16 Jun 2009 10:20:57 -0700
Subject: [Cython] Trivial numpy test in sage notebook fails...
In-Reply-To: <79017BF3-B11C-4378-A1C1-33572625AB28@math.washington.edu>
References: <3327124144.3090608@smtp.netcom.no>
	<79017BF3-B11C-4378-A1C1-33572625AB28@math.washington.edu>
Message-ID: <6ce0ac130906161020l80c746ev770b4520620262c0@mail.gmail.com>

I too need to use the numpy+cython stuff from the notebook.

> To conclude this thread, Sage ships its own numpy.pxd, which predates
> the buffer interface. We need to go in and remove this (there's a
> ticket in Sage to do this).

In the meantime, if I simply remove Sage's old numpy.pxd file will
this start working, or is it more complicated than that?

Cheers,

Brian

From robertwb at math.washington.edu  Wed Jun 17 06:45:41 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Jun 2009 21:45:41 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3672EC.90002@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
Message-ID: <12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>

On Jun 15, 2009, at 9:12 AM, Dag Sverre Seljebotn wrote:

> Thanks to everybody who contributed to the discussion on a Cython  
> array
> type last week! Here's a summary to attempt focusing the discussion.
>
> There are now two CEPs:
> - CEP 517, array type: http://wiki.cython.org/enhancements/array
> - CEP 518, SIMD operations: http://wiki.cython.org/enhancements/simd
>
> I mostly just added a "what does this facilitate"-section is added  
> near
> the beginning of each, and the multidimensional aspect of the  
> arrays has
> been emphasised. No need to reread it.

Looks good. I assume this supersedes http://wiki.cython.org/ 
enhancements/buffersyntax ; are there any other wiki pages that are  
made obsolete by these proposals?

> The story is this:
>
>   a) I can only commit to implementing CEP 517 this time around
>
>   b) The arrays themselves/CEP 517 can without loosing *too* much be
> done as a library (though I would prefer them as builtins myself).
> However SIMD/CEP 518 depends on building the arrays into the language,
> and switching approaches later would mean redoing some stuff.  
> Therefore
> SIMD (and the numerical roadmap in general) keeps popping into the
> discussion.
>
>   c) It would be nice to finally settle a roadmap for numerics in  
> Cython
> (how far will the language be bent for the benefit of numerics), for
> instance to economize on discussions when questions pop up on the
> mailing list (and make plans). These two CEPs would create one such  
> roadmap.
>
>   d) CEP 518 isn't a lot of work (totally doable in a GSoC next
> year...though of course not at the expense of e.g. a full Python
> compatability GSoC or similar).
>
> Key impressions from last week:
>
>   a) Numerical users seem positive about all of this.
>   b) Robert Bradshaw said:
>
> I am not opposed to this feature, despite it being off the path of
> being a good, optimizing Python compiler with easy glue to external
> code, as long as it doesn't intrude on the language too much. Mostly,
> this is because th