From stefan at sun.ac.za  Mon Jun  1 01:46:02 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Mon, 1 Jun 2009 01:46:02 +0200
Subject: [Cython] Profiling Cython under OSX
Message-ID: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>

Hi all,

I needed to profile some Cython code on my mac today.  I am only
familiar with Valgrind, but this tool is not available on OSX (closest
thing is  http://www.sealiesoftware.com/valgrind/).  Luckily, I
discovered that the Activity Monitor is very handy in this regard:

1. Start your long running process
2. Go to the activity monitor, select the process, and click "Sample Process"
3. In the window that pops up, select "Display: Percent of Parent"

And there you have a beautiful tree showing how much time each call took:

http://mentat.za.net/refer/osx_profiling.png

Enjoy!
St?fan

From amit.pureenergy at gmail.com  Mon Jun  1 15:38:54 2009
From: amit.pureenergy at gmail.com (Amit Sethi)
Date: Mon, 1 Jun 2009 19:08:54 +0530
Subject: [Cython] Creating python bindings for C library using Cython
Message-ID: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>

hi ,
I guess this needs to go on cython-users but i did not see any such mailing
list so I am posting it here ,
I am trying to create python bindings for a library written in C ,
But I am wondering as to how I am suppose to handle pointers.

For example

for the C function,
int * analyse_row (int length, unsigned char *data),


Now in this function call how am i supposed to handle *data , can i pass a
string in place of *data ., Also please guide be to some projects that use
Cython for binding with C
-- 
A-M-I-T S|S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090601/56a66e9e/attachment.htm 

From dalcinl at gmail.com  Mon Jun  1 16:19:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 1 Jun 2009 11:19:11 -0300
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
Message-ID: <e7ba66e40906010719p6e570d1an96752797eaf108ab@mail.gmail.com>

On Mon, Jun 1, 2009 at 10:38 AM, Amit Sethi <amit.pureenergy at gmail.com> wrote:
>
> hi ,
> I guess this needs to go on cython-users but i did not see any such mailing
> list so I am posting it here ,
> I am trying to create python bindings for a library written in C ,
> But I am wondering as to how I am suppose to handle pointers.
>
> For example
>
> for the C function,
> int * analyse_row (int length, unsigned char *data),
>
>
> Now in this function call how am i supposed to handle *data , can i pass a
> string in place of *data .

Mmm... better you are sure what the 'data' represent. Anyway, you can
also add explicit cast and call like "analize_row(somelen, <unsigned
char*>somepointer)". I would need some more backgraund from you to
provide a better hint (What is analyse_row() for?)

> Also please guide be to some projects that use
> Cython for binding with C
>

>From my side, you could look at mpi4py, petsc4py, slepc4py (all of
them at Google Code).




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Jun  1 16:33:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 1 Jun 2009 11:33:59 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>
	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
Message-ID: <e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>

On Sat, May 30, 2009 at 12:04 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On May 28, 2009, at 3:50 PM, Lisandro Dalcin wrote:
>
>>
>> Anyway, suppose you have installed both MSVC and MinGW, an as you are
>> a serious Cython developer, you want to run the testsuite with both.
>> The easiest way to select GCC would be to pass '--mingw' to
>> runtests.py, right?
>
>
> I hadn't thought of that. Are you saying that you would have a single
> Python install that works both with MSVC and MinGW?
>

Of course!

In my particular case, I have installed stock Python 2.6 (using the
MSI installed from python.org), and I routinely test mpi4py building
it with MSVC Express 2008 and MinGW32, with a couple of different MPI
implementations (DeinoMPI, MPICH2, and very recently Microsoft MPI).
Up to now, all work just fine. Of course, I've been very careful and
my code does not call anything from the C stdlib (which is know to
have issues when building with MinGW, because of possible mismatches
of the C runtime).

OTOH, most of the Cython testsuite pass happily with the stock Python
2.6 and MinGW32. The only offending part are test like 'cdef extern
void foo()', as that end-up with linker errors.




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From seb.binet at gmail.com  Mon Jun  1 18:11:50 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Mon, 1 Jun 2009 18:11:50 +0200
Subject: [Cython] cimport from __all__
Message-ID: <200906011811.50884.binet@cern.ch>

hi there,

I am in the process of applying what I learnt at last SciPy'08 tutorial (yes, 
I am that slow) on an almost-real-life project of mine.
Up to now I am trying to translate an old C++/PyCxx project (which does not 
compile anymore) into Cython: so far so good.

One thing, though: the separation of '.pyx' and '.pxd' files to export 
declarations is a bit too C-ish.

Couldn't the same functionality be achieved by re-using the __all__ module 
attribute ?

## mymodule.pyx ##
__all__ = ( 'Foo', 'bar')

cdef class Foo:
   # implementation elided

cdef int bar():
  return 42
## EOF ##

## test.pyx ##
cimport mymodule
print "bar:",mymodule.bar()
## EOF ##

I suppose I am missing some implementation detail, but from my 20k feet 
overview, it seems that should work :)

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From dagss at student.matnat.uio.no  Mon Jun  1 18:23:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Jun 2009 18:23:56 +0200
Subject: [Cython] cimport from __all__
In-Reply-To: <200906011811.50884.binet@cern.ch>
References: <200906011811.50884.binet@cern.ch>
Message-ID: <4A24009C.30509@student.matnat.uio.no>

Sebastien Binet wrote:
> hi there,
>
> I am in the process of applying what I learnt at last SciPy'08 tutorial (yes, 
> I am that slow) on an almost-real-life project of mine.
> Up to now I am trying to translate an old C++/PyCxx project (which does not 
> compile anymore) into Cython: so far so good.
>
> One thing, though: the separation of '.pyx' and '.pxd' files to export 
> declarations is a bit too C-ish.
>
> Couldn't the same functionality be achieved by re-using the __all__ module 
> attribute ?
>
> ## mymodule.pyx ##
> __all__ = ( 'Foo', 'bar')
>
> cdef class Foo:
>    # implementation elided
>
> cdef int bar():
>   return 42
> ## EOF ##
>
> ## test.pyx ##
> cimport mymodule
> print "bar:",mymodule.bar()
> ## EOF ##
>
> I suppose I am missing some implementation detail, but from my 20k feet 
> overview, it seems that should work :)
>   
I remember that the basic idea of this feature (which would be 
implemented as "generate pxd files automatically from pyx files") has 
been discussed earlier (Sage dev 1 days?) and it's definitely on our 
would-like-to-do-if-we-get-time list :-) The idea about using the 
__all__ attribute to declare what symbols to export was new; I like it!

Dag Sverre

From sccolbert at gmail.com  Mon Jun  1 19:37:30 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 1 Jun 2009 10:37:30 -0700
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
Message-ID: <7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>

python strings are automagically converted to char* by cython.

You can also look at a wrapper i'm building for the OpenCV library.
cython-opencv at google code.

Chris

On Mon, Jun 1, 2009 at 6:38 AM, Amit Sethi <amit.pureenergy at gmail.com>wrote:

>
> hi ,
> I guess this needs to go on cython-users but i did not see any such mailing
> list so I am posting it here ,
> I am trying to create python bindings for a library written in C ,
> But I am wondering as to how I am suppose to handle pointers.
>
> For example
>
> for the C function,
> int * analyse_row (int length, unsigned char *data),
>
>
> Now in this function call how am i supposed to handle *data , can i pass a
> string in place of *data ., Also please guide be to some projects that use
> Cython for binding with C
> --
> A-M-I-T S|S
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090601/c2b57f99/attachment.htm 

From robert.kern at gmail.com  Mon Jun  1 20:18:28 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 01 Jun 2009 13:18:28 -0500
Subject: [Cython] Profiling Cython under OSX
In-Reply-To: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>
References: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>
Message-ID: <h0161k$8nj$1@ger.gmane.org>

On 2009-05-31 18:46, St?fan van der Walt wrote:
> Hi all,
>
> I needed to profile some Cython code on my mac today.  I am only
> familiar with Valgrind, but this tool is not available on OSX (closest
> thing is  http://www.sealiesoftware.com/valgrind/).

Actually, the OS X branch got merged into the trunk a few days ago.

> Luckily, I
> discovered that the Activity Monitor is very handy in this regard:
>
> 1. Start your long running process
> 2. Go to the activity monitor, select the process, and click "Sample Process"
> 3. In the window that pops up, select "Display: Percent of Parent"

Instruments.app is the canonical tool for this on OS X.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robertwb at math.washington.edu  Mon Jun  1 21:06:06 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Jun 2009 12:06:06 -0700
Subject: [Cython] Profiling Cython under OSX
In-Reply-To: <h0161k$8nj$1@ger.gmane.org>
References: <9457e7c80905311646v3588d3e1lb29f508947eabbcc@mail.gmail.com>
	<h0161k$8nj$1@ger.gmane.org>
Message-ID: <D0CCEF0A-B471-4C73-8194-3C0F81DB4781@math.washington.edu>

On Jun 1, 2009, at 11:18 AM, Robert Kern wrote:

> On 2009-05-31 18:46, St?fan van der Walt wrote:
>> Hi all,
>>
>> I needed to profile some Cython code on my mac today.  I am only
>> familiar with Valgrind, but this tool is not available on OSX  
>> (closest
>> thing is  http://www.sealiesoftware.com/valgrind/).
>
> Actually, the OS X branch got merged into the trunk a few days ago.
>
>> Luckily, I
>> discovered that the Activity Monitor is very handy in this regard:
>>
>> 1. Start your long running process
>> 2. Go to the activity monitor, select the process, and click  
>> "Sample Process"
>> 3. In the window that pops up, select "Display: Percent of Parent"
>
> Instruments.app is the canonical tool for this on OS X.

Still, very cool! Could you put this up on the wiki? (Also, any  
valgrind+Cython knowledge you might have.)

- Robert


From dagss at student.matnat.uio.no  Mon Jun  1 21:09:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Jun 2009 21:09:08 +0200
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
	<7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
Message-ID: <4A242754.5050207@student.matnat.uio.no>

Chris Colbert wrote:
> python strings are automagically converted to char* by cython.
I'll beat Stefan to it and note some very important points here. This 
applies if you actually have a string (e.g. could be stored in "unicode" 
in Py2), not if you have raw byte data ("bytes" in Py3).

1) A C char* is not a string as such, just a sequence of bytes. The C 
library should have a definition of what kind of encoding it wants its 
strings in, and one should call the library like this:

encoded = mystr.encode(encoding)
cdef char* encoded_buf = encoded
c_func(encoded_buf)

2) Note that char* does not say anything about releasing memory etc. 
I.e., the following will likely crash:

c_func(mystr.encode(encoding))

because the temporary object returned from the encode method is 
deallocated. *Always* keep a reference to the original Python object for 
the duration someone could use the char*.

(Perhaps somebody could make this a FAQ entry in the wiki if it is not 
already?)

Dag Sverre

From sccolbert at gmail.com  Mon Jun  1 21:14:02 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 1 Jun 2009 15:14:02 -0400
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <4A242754.5050207@student.matnat.uio.no>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>
	<7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
	<4A242754.5050207@student.matnat.uio.no>
Message-ID: <7f014ea60906011214q17a73ba7u3a1217b468757adb@mail.gmail.com>

I never mind being corrected/updated/put-in-my-place with such a great
answer.

So there you have it :)

Cheers!

On Mon, Jun 1, 2009 at 3:09 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Chris Colbert wrote:
> > python strings are automagically converted to char* by cython.
> I'll beat Stefan to it and note some very important points here. This
> applies if you actually have a string (e.g. could be stored in "unicode"
> in Py2), not if you have raw byte data ("bytes" in Py3).
>
> 1) A C char* is not a string as such, just a sequence of bytes. The C
> library should have a definition of what kind of encoding it wants its
> strings in, and one should call the library like this:
>
> encoded = mystr.encode(encoding)
> cdef char* encoded_buf = encoded
> c_func(encoded_buf)
>
> 2) Note that char* does not say anything about releasing memory etc.
> I.e., the following will likely crash:
>
> c_func(mystr.encode(encoding))
>
> because the temporary object returned from the encode method is
> deallocated. *Always* keep a reference to the original Python object for
> the duration someone could use the char*.
>
> (Perhaps somebody could make this a FAQ entry in the wiki if it is not
> already?)
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090601/cccc2f84/attachment.htm 

From robertwb at math.washington.edu  Mon Jun  1 21:17:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Jun 2009 12:17:04 -0700
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>
	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
	<e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
Message-ID: <3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>

On Jun 1, 2009, at 7:33 AM, Lisandro Dalcin wrote:

> On Sat, May 30, 2009 at 12:04 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On May 28, 2009, at 3:50 PM, Lisandro Dalcin wrote:
>>
>>>
>>> Anyway, suppose you have installed both MSVC and MinGW, an as you  
>>> are
>>> a serious Cython developer, you want to run the testsuite with both.
>>> The easiest way to select GCC would be to pass '--mingw' to
>>> runtests.py, right?
>>
>>
>> I hadn't thought of that. Are you saying that you would have a single
>> Python install that works both with MSVC and MinGW?
>>
>
> Of course!
>
> In my particular case, I have installed stock Python 2.6 (using the
> MSI installed from python.org), and I routinely test mpi4py building
> it with MSVC Express 2008 and MinGW32, with a couple of different MPI
> implementations (DeinoMPI, MPICH2, and very recently Microsoft MPI).
> Up to now, all work just fine. Of course, I've been very careful and
> my code does not call anything from the C stdlib (which is know to
> have issues when building with MinGW, because of possible mismatches
> of the C runtime).

Ah, I didn't know avoiding the C stdlib was all that one needed to do  
here. You can tell I'm not a Windows developer :)

> OTOH, most of the Cython testsuite pass happily with the stock Python
> 2.6 and MinGW32. The only offending part are test like 'cdef extern
> void foo()', as that end-up with linker errors.


Do you have any ideas on how best to fix this? (I suppose we could  
make our own library with these fake stubs, but that seems kludgy and  
overkill)

- Robert


From dagss at student.matnat.uio.no  Mon Jun  1 21:48:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Jun 2009 21:48:58 +0200
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>
References: <4A1DB976.4060309@gmail.com>
	<4A1DC278.9080704@gmail.com>	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>	<e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
	<3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>
Message-ID: <4A2430AA.6070707@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 1, 2009, at 7:33 AM, Lisandro Dalcin wrote:
>
>   
>> On Sat, May 30, 2009 at 12:04 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>     
>>> On May 28, 2009, at 3:50 PM, Lisandro Dalcin wrote:
>>>
>>>       
>>>> Anyway, suppose you have installed both MSVC and MinGW, an as you  
>>>> are
>>>> a serious Cython developer, you want to run the testsuite with both.
>>>> The easiest way to select GCC would be to pass '--mingw' to
>>>> runtests.py, right?
>>>>         
>>> I hadn't thought of that. Are you saying that you would have a single
>>> Python install that works both with MSVC and MinGW?
>>>
>>>       
>> Of course!
>>
>> In my particular case, I have installed stock Python 2.6 (using the
>> MSI installed from python.org), and I routinely test mpi4py building
>> it with MSVC Express 2008 and MinGW32, with a couple of different MPI
>> implementations (DeinoMPI, MPICH2, and very recently Microsoft MPI).
>> Up to now, all work just fine. Of course, I've been very careful and
>> my code does not call anything from the C stdlib (which is know to
>> have issues when building with MinGW, because of possible mismatches
>> of the C runtime).
>>     
>
> Ah, I didn't know avoiding the C stdlib was all that one needed to do  
> here. You can tell I'm not a Windows developer :)
>   
IIRC, an additional point: I think using the stdlib should work as long 
as you link in the proper MinGW stdlib, and make sure not to exchange 
uses of the stdlib with modules compiled with a different compiler.

It's basically like having two different stdlibs -- file handles from 
one of them won't make sense to the other, buffering output in one of 
them won't be seen by the other, and so on.

Am I right? (This came form a numpy-discuss thread somewhere, they've 
discussed this a couple of times.)

Dag Sverre

From stefan_ml at behnel.de  Mon Jun  1 22:04:35 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Jun 2009 22:04:35 +0200
Subject: [Cython] Creating python bindings for C library using Cython
In-Reply-To: <4A242754.5050207@student.matnat.uio.no>
References: <da81a0a80906010638g22b29470jee90255a7f42d013@mail.gmail.com>	<7f014ea60906011037h14c61539ldfa0e87a0c32ff5b@mail.gmail.com>
	<4A242754.5050207@student.matnat.uio.no>
Message-ID: <4A243453.50004@behnel.de>


Dag Sverre Seljebotn wrote:
> Chris Colbert wrote:
>> python strings are automagically converted to char* by cython.
> I'll beat Stefan to it and note some very important points here.
> [...]
> (Perhaps somebody could make this a FAQ entry in the wiki if it is not 
> already?)

http://wiki.cython.org/FAQ#HowdoIpassaPythonstringparameterontoaClibrary.3F

Now guess who wrote that entry :)

Stefan


From dalcinl at gmail.com  Mon Jun  1 22:08:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 1 Jun 2009 17:08:01 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A2430AA.6070707@student.matnat.uio.no>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<e7ba66e40905271637l1af91b66m7806916fa8163bc1@mail.gmail.com>
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
	<e7ba66e40905281550t50b384ddy9e78e6421249e055@mail.gmail.com>
	<99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>
	<e7ba66e40906010733y7ba54b21g3779ad1416e3346e@mail.gmail.com>
	<3A4B4142-B9F7-4000-A951-60686D1FF804@math.washington.edu>
	<4A2430AA.6070707@student.matnat.uio.no>
Message-ID: <e7ba66e40906011308y330bc185pe041dae839673b3a@mail.gmail.com>

On Mon, Jun 1, 2009 at 4:48 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Robert Bradshaw wrote:
>>
>> Ah, I didn't know avoiding the C stdlib was all that one needed to do
>> here. You can tell I'm not a Windows developer :)
>>

You can tell the same about me :-)

> IIRC, an additional point: I think using the stdlib should work as long
> as you link in the proper MinGW stdlib, and make sure not to exchange
> uses of the stdlib with modules compiled with a different compiler.
>
> It's basically like having two different stdlibs -- file handles from
> one of them won't make sense to the other, buffering output in one of
> them won't be seen by the other, and so on.
>

> Am I right? (This came form a numpy-discuss thread somewhere, they've
> discussed this a couple of times.)
>

Yes, you are right. But I do not know how to ask MinGW use a
particular C runtime. Though I bet there is a way (and likely numpy
folks know it).



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jusa.sj at gmail.com  Tue Jun  2 12:43:29 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Tue, 2 Jun 2009 13:43:29 +0300
Subject: [Cython] Syntax for declaring Cython class variables
Message-ID: <20db9a7d0906020343u3dcb3157nc166049261dfb24b@mail.gmail.com>

Hi,
I got a simple problem regarding so called class variables in Cython. I'm
trying to declare a simple counter as a class variable which is shared
between all instances of that class. The counter's value should be
incremented as instances of the class are created. Cython allows me to
declare a Python dictionary as a class variable, but the following gives me
an error:


# foo.pyx

cdef class Foo:
    count = 0

    def __cinit__(self):
        Foo.count += 1


>>> a = Foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
    Foo.count += 1
TypeError: can't set attributes of built-in/extension type 'foo.Foo'


Is there a way to declare an integer class variable in Cython?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/1d2b260e/attachment.htm 

From darkgl0w at yahoo.com  Tue Jun  2 14:27:54 2009
From: darkgl0w at yahoo.com (Cristi Constantin)
Date: Tue, 2 Jun 2009 05:27:54 -0700 (PDT)
Subject: [Cython] Syntax for declaring Cython class variables
Message-ID: <235734.63423.qm@web52112.mail.re2.yahoo.com>


Good day.
You should try self.count += 1 instead of Foo.count += 1.

--- On Tue, 6/2/09, Juha Salo <jusa.sj at gmail.com> wrote:

From: Juha Salo <jusa.sj at gmail.com>
Subject: [Cython] Syntax for declaring Cython class variables
To: cython-dev at codespeak.net
Date: Tuesday, June 2, 2009, 3:43 AM

Hi,?
I got a simple problem regarding so called class variables in Cython. I'm trying to declare a simple counter as a class variable which is shared between all instances of that class. The counter's value should be incremented as instances of the class are created. Cython allows me to declare a Python dictionary as a class variable, but the following gives me an error:


# foo.pyx
cdef class Foo:?? ?count = 0
?? ?def __cinit__(self):?? ? ? ?Foo.count += 1


>>> a = Foo()Traceback (most recent call last):??File "<stdin>", line 1, in <module>??File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
?? ?Foo.count += 1TypeError: can't set attributes of built-in/extension type 'foo.Foo'

Is there a way to declare an integer class variable in Cython?


-----Inline Attachment Follows-----

_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/7ea06252/attachment.htm 

From jusa.sj at gmail.com  Tue Jun  2 17:17:05 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Tue, 2 Jun 2009 18:17:05 +0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <235734.63423.qm@web52112.mail.re2.yahoo.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
Message-ID: <20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>

Unfortunately that didn't seem to work. I got the following error after the
change:
>>> a = Foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
    self.count += 1
AttributeError: 'foo.Foo' object attribute 'count' is read-only


To clarify, I want the value of the count variable to remain the same
between all instances. So if I create 3 Foo objects, each object should show
3 as the value in the count variable.


2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>

>
> Good day.
> You should try self.count += 1 instead of Foo.count += 1.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/50306bda/attachment.htm 

From sccolbert at gmail.com  Tue Jun  2 18:36:50 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 2 Jun 2009 12:36:50 -0400
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
Message-ID: <7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>

you have to declare the attribute as public.

http://docs.cython.org/docs/extension_types.html

Cheers,

Chris

On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:

> Unfortunately that didn't seem to work. I got the following error after the
> change:
> >>> a = Foo()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>     self.count += 1
> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>
>
> To clarify, I want the value of the count variable to remain the same
> between all instances. So if I create 3 Foo objects, each object should show
> 3 as the value in the count variable.
>
>
> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>
>>
>> Good day.
>> You should try self.count += 1 instead of Foo.count += 1.
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/a1b9afab/attachment.htm 

From dalcinl at gmail.com  Tue Jun  2 18:44:17 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 13:44:17 -0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
Message-ID: <e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>

No, that will not work. AFAIK, Class attributes are not currently
supported for cdef classes.

Juha, you will have to code like this:


# foo.pyx

cdef int Foo_count = 0

cdef class Foo:

    def __cinit__(self):
        global Foo_count
        Foo_count += 1




On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com> wrote:
> you have to declare the attribute as public.
>
> http://docs.cython.org/docs/extension_types.html
>
> Cheers,
>
> Chris
>
> On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
>>
>> Unfortunately that didn't seem to work. I got the following error after
>> the change:
>> >>> a = Foo()
>> Traceback (most recent call last):
>> ??File "<stdin>", line 1, in <module>
>> ??File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>> ?? ?self.count += 1
>> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>>
>> To clarify, I want the value of the count variable to remain the same
>> between all instances. So if I create 3 Foo objects, each object should show
>> 3 as the value in the count variable.
>>
>> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>>>
>>> Good day.
>>> You should try self.count += 1 instead of Foo.count += 1.
>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Jun  2 18:51:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 13:51:06 -0300
Subject: [Cython] PATCH: get rid of __Pyx_StrEq hackery at
	__Pyx_Method_ClassMethod
Message-ID: <e7ba66e40906020951w2f98cb72x55741d3586099ab8@mail.gmail.com>

Review this. Tell me if you like it.

AFAIK, this should always work (even iff an embedder plays with
loading/unloading the Python shared library ??)

diff -r 88fa346e169d Cython/Compiler/Symtab.py
--- a/Cython/Compiler/Symtab.py	Tue May 26 22:54:46 2009 +0200
+++ b/Cython/Compiler/Symtab.py	Tue Jun 02 13:45:11 2009 -0300
@@ -1583,8 +1583,14 @@
 impl = """
 static PyObject* __Pyx_Method_ClassMethod(PyObject *method) {
     /* It appears that PyMethodDescr_Type is not anywhere exposed in
the Python/C API */
-    /* if (!PyObject_TypeCheck(method, &PyMethodDescr_Type)) { */
-    if (__Pyx_StrEq(Py_TYPE(method)->tp_name, "method_descriptor")) {
/* cdef classes */
+    static PyTypeObject *methoddescr_type = NULL;
+    if (methoddescr_type == NULL) {
+       PyObject *meth = __Pyx_GetAttrString((PyObject*)&PyList_Type, "append");
+       if (!meth) return NULL;
+       methoddescr_type = Py_TYPE(meth);
+       Py_DECREF(meth);
+    }
+    if (PyObject_TypeCheck(method, methoddescr_type)) { /* cdef classes */
         PyMethodDescrObject *descr = (PyMethodDescrObject *)method;
         return PyDescr_NewClassMethod(descr->d_type, descr->d_method);
     }

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Tue Jun  2 18:59:22 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 2 Jun 2009 12:59:22 -0400
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
	<e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
Message-ID: <7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>

This is correct, forget what I said. I should have tried it out before I
said anything. My apologies.



On Tue, Jun 2, 2009 at 12:44 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> No, that will not work. AFAIK, Class attributes are not currently
> supported for cdef classes.
>
> Juha, you will have to code like this:
>
>
> # foo.pyx
>
> cdef int Foo_count = 0
>
> cdef class Foo:
>
>    def __cinit__(self):
>        global Foo_count
>        Foo_count += 1
>
>
>
>
> On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com> wrote:
> > you have to declare the attribute as public.
> >
> > http://docs.cython.org/docs/extension_types.html
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
> >>
> >> Unfortunately that didn't seem to work. I got the following error after
> >> the change:
> >> >>> a = Foo()
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >>   File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
> >>     self.count += 1
> >> AttributeError: 'foo.Foo' object attribute 'count' is read-only
> >>
> >> To clarify, I want the value of the count variable to remain the same
> >> between all instances. So if I create 3 Foo objects, each object should
> show
> >> 3 as the value in the count variable.
> >>
> >> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
> >>>
> >>> Good day.
> >>> You should try self.count += 1 instead of Foo.count += 1.
> >>
> >>
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >>
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/2b9ffc48/attachment.htm 

From jusa.sj at gmail.com  Tue Jun  2 19:48:48 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Tue, 2 Jun 2009 20:48:48 +0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
	<e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
	<7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>
Message-ID: <20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>

OK, thanks for the info. It's interesting, though, that I can code something
like this in Cython and it works:

cdef class Foo:
    foos = {}

    def __cinit__(self, name):
        Foo.foos[name] = self


>>> a = Foo('Larry')
>>> b = Foo('Bob')
>>> a.foos
{'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
0x01C5F488>}
>>> b.foos
{'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
0x01C5F488>}


Perhaps I could use a hack workaround with the aforementioned count variable
by storing it inside a class level list object.


2009/6/2 Chris Colbert <sccolbert at gmail.com>

> This is correct, forget what I said. I should have tried it out before I
> said anything. My apologies.
>
>
>
> On Tue, Jun 2, 2009 at 12:44 PM, Lisandro Dalcin <dalcinl at gmail.com>wrote:
>
>> No, that will not work. AFAIK, Class attributes are not currently
>> supported for cdef classes.
>>
>> Juha, you will have to code like this:
>>
>>
>> # foo.pyx
>>
>> cdef int Foo_count = 0
>>
>> cdef class Foo:
>>
>>    def __cinit__(self):
>>        global Foo_count
>>        Foo_count += 1
>>
>>
>>
>>
>> On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com>
>> wrote:
>> > you have to declare the attribute as public.
>> >
>> > http://docs.cython.org/docs/extension_types.html
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
>> >>
>> >> Unfortunately that didn't seem to work. I got the following error after
>> >> the change:
>> >> >>> a = Foo()
>> >> Traceback (most recent call last):
>> >>   File "<stdin>", line 1, in <module>
>> >>   File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>> >>     self.count += 1
>> >> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>> >>
>> >> To clarify, I want the value of the count variable to remain the same
>> >> between all instances. So if I create 3 Foo objects, each object should
>> show
>> >> 3 as the value in the count variable.
>> >>
>> >> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>> >>>
>> >>> Good day.
>> >>> You should try self.count += 1 instead of Foo.count += 1.
>> >>
>> >>
>> >> _______________________________________________
>> >> Cython-dev mailing list
>> >> Cython-dev at codespeak.net
>> >> http://codespeak.net/mailman/listinfo/cython-dev
>> >>
>> >
>> >
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090602/47b1a3c3/attachment.htm 

From dalcinl at gmail.com  Tue Jun  2 20:03:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 15:03:25 -0300
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906020817t1aa950bfxc7374d4b593bc821@mail.gmail.com>
	<7f014ea60906020936y38f98ccfl999aaa7601fa71a5@mail.gmail.com>
	<e7ba66e40906020944g45a4bda3g3f475811899ea8dd@mail.gmail.com>
	<7f014ea60906020959j6f46ee38qfef3275d35bb8be9@mail.gmail.com>
	<20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>
Message-ID: <e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>

On Tue, Jun 2, 2009 at 2:48 PM, Juha Salo <jusa.sj at gmail.com> wrote:
> OK, thanks for the info. It's interesting, though, that I can code something
> like this in Cython and it works:
>
> cdef class Foo:
> ?? ?foos = {}
> ?? ?def __cinit__(self, name):
> ?? ??? ?Foo.foos[name] = self
>
>>>> a = Foo('Larry')
>>>> b = Foo('Bob')
>>>> a.foos
> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> 0x01C5F488>}
>>>> b.foos
> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> 0x01C5F488>}
>

But that is a different beast. You are doing a 'getattr', and next
updating the attribute. What you cannot do on a cdef class is
'setattr'. Python rules for extension types...

BTW, such code creates reference cycles. If you are going to create a
lot of these objects, GC collection will have to traverse a large
dict, slowing down your runs each time the collection triggers.

> Perhaps I could use a hack workaround with the aforementioned count variable
> by storing it inside a class level list object.
>

Well, if the extra dict lookup is not an issue for you, go for it....



> 2009/6/2 Chris Colbert <sccolbert at gmail.com>
>>
>> This is correct, forget what I said. I should have tried it out before I
>> said anything. My apologies.
>>
>>
>>
>> On Tue, Jun 2, 2009 at 12:44 PM, Lisandro Dalcin <dalcinl at gmail.com>
>> wrote:
>>>
>>> No, that will not work. AFAIK, Class attributes are not currently
>>> supported for cdef classes.
>>>
>>> Juha, you will have to code like this:
>>>
>>>
>>> # foo.pyx
>>>
>>> cdef int Foo_count = 0
>>>
>>> cdef class Foo:
>>>
>>> ? ?def __cinit__(self):
>>> ? ? ? ?global Foo_count
>>> ? ? ? ?Foo_count += 1
>>>
>>>
>>>
>>>
>>> On Tue, Jun 2, 2009 at 1:36 PM, Chris Colbert <sccolbert at gmail.com>
>>> wrote:
>>> > you have to declare the attribute as public.
>>> >
>>> > http://docs.cython.org/docs/extension_types.html
>>> >
>>> > Cheers,
>>> >
>>> > Chris
>>> >
>>> > On Tue, Jun 2, 2009 at 11:17 AM, Juha Salo <jusa.sj at gmail.com> wrote:
>>> >>
>>> >> Unfortunately that didn't seem to work. I got the following error
>>> >> after
>>> >> the change:
>>> >> >>> a = Foo()
>>> >> Traceback (most recent call last):
>>> >> ??File "<stdin>", line 1, in <module>
>>> >> ??File "foo.pyx", line 7, in foo.Foo.__cinit__ (foo.c:388)
>>> >> ?? ?self.count += 1
>>> >> AttributeError: 'foo.Foo' object attribute 'count' is read-only
>>> >>
>>> >> To clarify, I want the value of the count variable to remain the same
>>> >> between all instances. So if I create 3 Foo objects, each object
>>> >> should show
>>> >> 3 as the value in the count variable.
>>> >>
>>> >> 2009/6/2 Cristi Constantin <darkgl0w at yahoo.com>
>>> >>>
>>> >>> Good day.
>>> >>> You should try self.count += 1 instead of Foo.count += 1.
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Cython-dev mailing list
>>> >> Cython-dev at codespeak.net
>>> >> http://codespeak.net/mailman/listinfo/cython-dev
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Cython-dev mailing list
>>> > Cython-dev at codespeak.net
>>> > http://codespeak.net/mailman/listinfo/cython-dev
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue Jun  2 20:28:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Jun 2009 11:28:19 -0700
Subject: [Cython] PATCH: get rid of __Pyx_StrEq hackery at
	__Pyx_Method_ClassMethod
In-Reply-To: <e7ba66e40906020951w2f98cb72x55741d3586099ab8@mail.gmail.com>
References: <e7ba66e40906020951w2f98cb72x55741d3586099ab8@mail.gmail.com>
Message-ID: <58D679FC-CBD2-46A9-AAA3-55CE06DF9607@math.washington.edu>

Yes, that should work fine. Please make a ticket and push.

On Jun 2, 2009, at 9:51 AM, Lisandro Dalcin wrote:

> Review this. Tell me if you like it.
>
> AFAIK, this should always work (even iff an embedder plays with
> loading/unloading the Python shared library ??)
>
> diff -r 88fa346e169d Cython/Compiler/Symtab.py
> --- a/Cython/Compiler/Symtab.py	Tue May 26 22:54:46 2009 +0200
> +++ b/Cython/Compiler/Symtab.py	Tue Jun 02 13:45:11 2009 -0300
> @@ -1583,8 +1583,14 @@
>  impl = """
>  static PyObject* __Pyx_Method_ClassMethod(PyObject *method) {
>      /* It appears that PyMethodDescr_Type is not anywhere exposed in
> the Python/C API */
> -    /* if (!PyObject_TypeCheck(method, &PyMethodDescr_Type)) { */
> -    if (__Pyx_StrEq(Py_TYPE(method)->tp_name, "method_descriptor")) {
> /* cdef classes */
> +    static PyTypeObject *methoddescr_type = NULL;
> +    if (methoddescr_type == NULL) {
> +       PyObject *meth = __Pyx_GetAttrString((PyObject*) 
> &PyList_Type, "append");
> +       if (!meth) return NULL;
> +       methoddescr_type = Py_TYPE(meth);
> +       Py_DECREF(meth);
> +    }
> +    if (PyObject_TypeCheck(method, methoddescr_type)) { /* cdef  
> classes */
>          PyMethodDescrObject *descr = (PyMethodDescrObject *)method;
>          return PyDescr_NewClassMethod(descr->d_type, descr- 
> >d_method);
>      }
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From seb.binet at gmail.com  Tue Jun  2 20:33:51 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Tue, 2 Jun 2009 20:33:51 +0200
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>
	<20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>
	<e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>
Message-ID: <200906022033.52142.binet@cern.ch>

On Tuesday 02 June 2009 20:03:25 Lisandro Dalcin wrote:
> On Tue, Jun 2, 2009 at 2:48 PM, Juha Salo <jusa.sj at gmail.com> wrote:
> > OK, thanks for the info. It's interesting, though, that I can code
> > something like this in Cython and it works:
> >
> > cdef class Foo:
> >     foos = {}
> >     def __cinit__(self, name):
> >         Foo.foos[name] = self
> >
> >>>> a = Foo('Larry')
> >>>> b = Foo('Bob')
> >>>> a.foos
> >
> > {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> > 0x01C5F488>}
> >
> >>>> b.foos
> >
> > {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
> > 0x01C5F488>}
>
> But that is a different beast. You are doing a 'getattr', and next
> updating the attribute. What you cannot do on a cdef class is
> 'setattr'. Python rules for extension types...
>
> BTW, such code creates reference cycles. 
which are usually easily broken if one were to use a 
weakref.WeakValueDictionar :)

> If you are going to create a
> lot of these objects, GC collection will have to traverse a large
> dict, slowing down your runs each time the collection triggers.

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From stefan_ml at behnel.de  Tue Jun  2 20:49:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Jun 2009 20:49:17 +0200
Subject: [Cython] Syntax for declaring Cython class variables
In-Reply-To: <200906022033.52142.binet@cern.ch>
References: <235734.63423.qm@web52112.mail.re2.yahoo.com>	<20db9a7d0906021048kd6e0587nceedfed042eac8b6@mail.gmail.com>	<e7ba66e40906021103x6ed8b723qaf39994914a8ad45@mail.gmail.com>
	<200906022033.52142.binet@cern.ch>
Message-ID: <4A25742D.2060109@behnel.de>


Sebastien Binet wrote:
> On Tuesday 02 June 2009 20:03:25 Lisandro Dalcin wrote:
>> On Tue, Jun 2, 2009 at 2:48 PM, Juha Salo wrote:
>>> OK, thanks for the info. It's interesting, though, that I can code
>>> something like this in Cython and it works:
>>>
>>> cdef class Foo:
>>>     foos = {}
>>>     def __cinit__(self, name):
>>>         Foo.foos[name] = self
>>>
>>>>>> a = Foo('Larry')
>>>>>> b = Foo('Bob')
>>>>>> a.foos
>>> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
>>> 0x01C5F488>}
>>>
>>>>>> b.foos
>>> {'Bob': <foo.Foo object at 0x01C5F490>, 'Larry': <foo.Foo object at
>>> 0x01C5F488>}
>> But that is a different beast. You are doing a 'getattr', and next
>> updating the attribute. What you cannot do on a cdef class is
>> 'setattr'. Python rules for extension types...
>>
>> BTW, such code creates reference cycles. 
> which are usually easily broken if one were to use a 
> weakref.WeakValueDictionar :)

... in which case the overhead is so high that you can just as well use a
plain Python class instead of an extension class.

[benchmark results left out as an exercise for the reader]

Stefan

From dalcinl at gmail.com  Wed Jun  3 02:22:26 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Jun 2009 21:22:26 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
Message-ID: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>

No time right now to look at this... All doctests fail on Py3.1rc1,
with errors like the traceback below. Are we needing a (trivial?) fix
or is something broken on Py3.1?

======================================================================
ERROR: compiling (cpp) and running withstat
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 345, in run
    doctest.DocTestSuite(self.module).run(result)
  File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 2226, in
DocTestSuite
    tests = test_finder.find(module, globs=globs, extraglobs=extraglobs)
  File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 820, in find
    source_lines = linecache.getlines(file, module.__dict__)
  File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 41, in getlines
    return updatecache(filename, module_globals)
  File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 130,
in updatecache
    lines = fp.readlines()
  File "/usr/local/python/3.1/lib/python3.1/codecs.py", line 300, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x94 in position
100: unexpected code byte


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Jun  3 09:00:13 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 3 Jun 2009 09:00:13 +0200 (CEST)
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
Message-ID: <cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> No time right now to look at this... All doctests fail on Py3.1rc1,
> with errors like the traceback below. Are we needing a (trivial?) fix
> or is something broken on Py3.1?
>
> ======================================================================
> ERROR: compiling (cpp) and running withstat
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "runtests.py", line 345, in run
>     doctest.DocTestSuite(self.module).run(result)
>   File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 2226, in
> DocTestSuite
>     tests = test_finder.find(module, globs=globs, extraglobs=extraglobs)
>   File "/usr/local/python/3.1/lib/python3.1/doctest.py", line 820, in find
>     source_lines = linecache.getlines(file, module.__dict__)
>   File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 41, in
> getlines
>     return updatecache(filename, module_globals)
>   File "/usr/local/python/3.1/lib/python3.1/linecache.py", line 130,
> in updatecache
>     lines = fp.readlines()
>   File "/usr/local/python/3.1/lib/python3.1/codecs.py", line 300, in
> decode
>     (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x94 in position
> 100: unexpected code byte

If all doctests fail like this, I doubt that there's much we can do about
this. I wonder why the UTF-8 codec is involved at all here, we are using
Unicode strings for "__doc__" everywhere in Py3 compatible tests. And why
does it look like linecache is reading from a file (even a filename!)
here? I can't look at the sources right now, but this feels awfully wrong
to me. I hope it's not trying to read the binary modules as text files...

I think we should file a bug report against Py3.1 and even bring this to
the attention of python-dev before it's too late.

Stefan


From collette at physics.ucla.edu  Wed Jun  3 09:07:18 2009
From: collette at physics.ucla.edu (Andrew Collette)
Date: Wed, 3 Jun 2009 00:07:18 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
Message-ID: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>

Hi,

I noticed there's an errant C99-style comment in the generated C code
for the method __Pyx_PyObject_Append:

return Py_None; // this is just to have an accurate signature

Unfortunately this breaks compilation with the -ansi option, which is
currently how my project is compiled on OS X.  This is with the
current public version of Cython (0.11.2).  As a temporary workaround
I'm using the += operator instead of append().

Andrew Collette

From robertwb at math.washington.edu  Wed Jun  3 09:27:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Jun 2009 00:27:46 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
In-Reply-To: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
References: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
Message-ID: <72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>

On Jun 3, 2009, at 12:07 AM, Andrew Collette wrote:

> Hi,
>
> I noticed there's an errant C99-style comment in the generated C code
> for the method __Pyx_PyObject_Append:
>
> return Py_None; // this is just to have an accurate signature
>
> Unfortunately this breaks compilation with the -ansi option, which is
> currently how my project is compiled on OS X.  This is with the
> current public version of Cython (0.11.2).  As a temporary workaround
> I'm using the += operator instead of append().

Strange--it looks like this was fixed months ago:

http://hg.cython.org/cython-devel/diff/8b6d42d16c99/Cython/Compiler/ 
ExprNodes.py

Maybe you need to touch the .pyx to re-generate the sources. It is  
gratifying to hear that is the only issue--should we be running our  
tests with this option? (Or is it gcc only?)

- Robert


From robertwb at math.washington.edu  Wed Jun  3 11:53:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Jun 2009 02:53:55 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>
	<C2F2563A-B960-422C-8E31-E5CD100209CA@math.washington.edu>
	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>
	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>
	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>
	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
Message-ID: <0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>

OK, I finally ran some simple benchmarks. To summarize, it looks like  
the non-naive algorithm is 4 times slower on my computer for doubles  
(see the code I used below).

-------- complex.pyx ----------

def time_div(double complex z, int N):
     cdef double complex a = 1
     cdef int i
     for i from 0 <= i < N:
         a = a/z
     return a

------ C code (toggled manually) --------

     static INLINE %(type)s %(type_name)s_div_fast(%(type)s a, %(type) 
s b) {
         %(type)s z;
         %(real_type)s denom = b.real*b.real + b.imag*b.imag;
         z.real = (a.real * b.real + a.imag * b.imag) / denom;
         z.imag = (a.imag * b.real - a.real * b.imag) / denom;
         return z;
     }

     static INLINE %(type)s %(type_name)s_div_slow(%(type)s a, %(type) 
s b) {
         %(type)s z;
         const %(real_type)s abs_breal = fabs(b.real);
         const %(real_type)s abs_bimag = fabs(b.imag);
         if (abs_breal >= abs_bimag) {
             const %(real_type)s ratio = b.imag / b.real;
             const %(real_type)s denom = b.real + b.imag * ratio;
             z.real = (a.real + a.imag * ratio) / denom;
             z.imag = (a.imag - a.real * ratio) / denom;
         }
         else {
             const %(real_type)s ratio = b.real / b.imag;
             const %(real_type)s denom = b.real * ratio + b.imag;
             z.real = (a.real * ratio + a.imag) / denom;
             z.imag = (a.imag * ratio - a.real) / denom;
         }
         return z;
     }

     #define %(type_name)s_div %(type_name)s_div_slow

------- session --------

In [1]: from complex import time_div; import math; a = math.e**1j
In [2]: time abs(time_div(a, 10**7))
CPU times: user 0.10 s, sys: 0.00 s, total: 0.10 s
Wall time: 0.10 s
Out[3]: 1.0000000002480893

vs.

In [1]: from complex import time_div; import math; a = math.e**1j
In [2]: time abs(time_div(a, 10**7))
CPU times: user 0.40 s, sys: 0.00 s, total: 0.40 s
Wall time: 0.40 s
Out[3]: 1.0000000000360101

------------------------------------

The difference in accumulated error after ten million divisions is:

sage: (1 - 1.0000000002480893) / (1 - 1.0000000000360101)
6.8894367097208654

 From before, relative errors for the naive vs. other algorithm,  
100000 runs, uniformly chosen in unit square (though nearly all  
distributions look basically the same):

naive
     better 26187
     avg 1.4940916064705992895601085724e-16
     worst 5.7659574333851360909896621025e-16
other
     better 63116
     avg 9.5414951065097745299683547276e-17
     worst 3.9584519821557590591785765975e-16

- Robert


From jonovik at gmail.com  Wed Jun  3 13:09:08 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Wed, 3 Jun 2009 13:09:08 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
Message-ID: <cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>

Dear Cythoneers,

I am simulating ordinary differential equation models with Pysundials
[1], a wrapper around the Sundials [2] suite of ODE solvers (written
in C). The ODE is specified by a Python function like:

def odefun(t, y, ydot, user_data):
? ?ydot[0] = ...
? ?ydot[1] = ...
? ?...
? ?return 0

corresponding to the Sundials convention [3]:

typedef int (*CVRhsFn)( realtype t, N_Vector y, N_Vector ydot, void *user_data);

After an inspiring workshop with Dag Sverre Seljebotn [4], I see that
Cython could routinely speed up my simulation work by a factor of
hundreds. My benchmark used Numpy arrays for y and ydot, and then
cythonizing the ODE function was straightforward. However, CVODE (the
Sundials solver that I use) requires that y and ydot be of type
N_Vector [5]. Pysundials wraps that type, but at the cost of
inefficient Python indexing.

Somewhere inside pysundials.cvode.NVector is an array of double
screaming to get out [6]. However, Dag Sverre told me that passing it
to Cython may require converting between ctypes and Cython pointers.
Moreover, the solver makes lots of calls to odefun(), and these would
still be Python-based. Therefore, it is probably best to bypass
Pysundials altogether and call CVODE directly.

Typical use of CVODE [7] involves manual memory allocation and lots of
routine steps. Pysundials faithfully replicates this somewhat
unpythonic interface, which could make it fairly straightforward to
convert my code (which includes some higher-level wrappers) to call
the CVODE C library directly.

I typically use ODE functions autogenerated from a model repository
[8]. Although so far I've been generating Python code [9], with Cython
I'd use C directly [10].

So, to summarize:
I would be very grateful for any help in getting started with calling
the Sundials libraries from Cython. I'm running Python 2.5.2 with
Numpy 1.1.0 under Linux:
-bash-3.2$ dmesg | head -1
Linux version 2.6.18-92.1.13.el5 (mockbuild at builder10.centos.org) (gcc
version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Sep 24 19:32:05
EDT 2008

Subtasks include:
* Installing Sundials from [2] (Done; test programs run successfully.
Also, Pysundials uses the same Sundials installation.)
* (Optional: Experienced Cythoneers might perhaps find it informative
to see how Pysundials wraps things; in that case, install from [1]
(and watch out for the CPATH gotcha mentioned on that page))
* Telling Cython where to find cvode.h, nvector.h, sundials.h (see [11])
* Importing anything from Sundials (see [12]). Currently "python
setup.py build_ext --inplace" complains about not finding nvector.h
[13]
* Allocating an NVector with Sundials (see [3, 7]).
* Handle the NVector from Python: exchange data with a Numpy array,
and pass the NVector to a C or Cython function.
* Compiling a C function (for the ODE right-hand side) to work with
CVODE. I would modify [10] to fit the calling convention [3].

Given this, I think the other steps [7] will follow fairly easily.
Despite the lengthiness of this post, I suspect that interfacing with
Sundials would be quite straightforward if I knew which buttons to
push. The benefits would carry over to all my work with cellular ODE
models, and would hopefully have some reuse value for others. Thanks
in advance for any pointers (pardon the pun).

Best regards,
Jon Olav Vik

--
Jon Olav Vik
jonovik at gmail.com
http://www.cigene.no/people/57-members/178-vik-jon-olav.html


[1] http://pysundials.sourceforge.net/
[2] https://computation.llnl.gov/casc/sundials/
[3] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/node5.html#SECTION00561000000000000000
[4] http://wiki.cython.org/talks/notur2009
[5] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/node7.html
[6]
In [25]: from pysundials import cvode
In [26]: y = cvode.NVector([3.14]) # state vector
In [38]: y.data.contents.content.contents.data[0]
Out[38]: 3.1400000000000001
In [39]: y.data.contents.content.contents.data[1]
Out[39]: 1.1297001364744253e-312
[7] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/node5.html#SECTION00540000000000000000
[8] http://www.cellml.org/models
[9] https://tracker.physiomeproject.org/show_bug.cgi?id=1514#c28
[10] http://www.cellml.org/models/fitzhugh_1961_version05/generateRawSource?lang=C
[11] First attempt at setup.py:
"""Run with e.g. python setup.py build_ext --inplace"""
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
? ?cmdclass = {'build_ext': build_ext},
? ?ext_modules = [
? ? ? ?Extension("hello",
? ? ? ? ? ?["hello.pyx"],
? ? ? ? ? ?include_dirs=['/site/VERSIONS/sundials-2.3.0/include/cvode',
? ? ? ? ? ? ? ?'/site/VERSIONS/sundials-2.3.0/include/nvector',
? ? ? ? ? ? ? ?'/site/VERSIONS/sundials-2.3.0/include/sundials'],
? ? ? ? ? ?library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
? ? ? ? ? ?libraries=['cvode']),
? ?]
)
[12] First attempt at hello.pyx:
cdef extern from "cvode.h":
? ?int CVodeSetFdata(void *cvode_mem, void *f_data)
[13] Failed attempt at building:
-bash-3.2$ python setup.py build_ext --inplace
running build_ext
building 'hello' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototy
? ? ? ? ? ? ? pes -fPIC -I/site/VERSIONS/sundials-2.3.0/include/cvode
-I/site/VERSIONS/sundial
? ? ? ? ? ? ? ? ? ? ? s-2.3.0/include/nvector
-I/site/VERSIONS/sundials-2.3.0/include/sundials -I/site

/VERSIONS/compython-2.5/Linux/include/python2.5 -c hello.c -o
build/temp.linux-x
? ? ? ? ? ? ? ? 86_64-2.5/hello.o
In file included from hello.c:135:
/site/VERSIONS/sundials-2.3.0/include/cvode/cvode.h:37:39: error:
sundials/sundi
? ? ? ? ? ? als_nvector.h: No such file or directory
In file included from hello.c:135:
/site/VERSIONS/sundials-2.3.0/include/cvode/cvode.h:177: error:
expected ')' bef
? ? ? ? ? ? ? ore 't'
...



-- 
Jon Olav Vik
jonovik at gmail.com
http://www.cigene.no/people/57-members/178-vik-jon-olav.html

From dagss at student.matnat.uio.no  Wed Jun  3 13:42:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Jun 2009 13:42:54 +0200
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>	<C2F2563A-B960-422C-8E31-E5CD100209CA@math.washington.edu>	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
Message-ID: <4A2661BE.8030605@student.matnat.uio.no>

Robert Bradshaw wrote:
>  From before, relative errors for the naive vs. other algorithm,  
> 100000 runs, uniformly chosen in unit square (though nearly all  
> distributions look basically the same):
>
> naive
>      better 26187
>      avg 1.4940916064705992895601085724e-16
>      worst 5.7659574333851360909896621025e-16
> other
>      better 63116
>      avg 9.5414951065097745299683547276e-17
>      worst 3.9584519821557590591785765975e-16
>
> - Robert
>   

This is not my speciality, but since the problem here is with things 
like fp cancellation on subtraction etc., wouldn't it be better to 
increase the odds of wildly different values?

Something like exp(uniform square)?

Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun  3 14:11:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Jun 2009 14:11:29 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
Message-ID: <4A266871.9070209@student.matnat.uio.no>

Jon Olav Vik wrote:
> Dear Cythoneers,
>   
Hi again! I'll be brief, suspecting that it's better with a quick answer 
now than putting it off until I don't remember getting back. Please get 
back with any obstacles that this doesn't solve.
> Subtasks include:
> * Installing Sundials from [2] (Done; test programs run successfully.
> Also, Pysundials uses the same Sundials installation.)
> * (Optional: Experienced Cythoneers might perhaps find it informative
> to see how Pysundials wraps things; in that case, install from [1]
> (and watch out for the CPATH gotcha mentioned on that page))
> * Telling Cython where to find cvode.h, nvector.h, sundials.h (see [11])
>   
I suspect the problem here is including "cvode", "nvector" etc. directly 
in include_path. Instead you should likely do

cdef extern from "cvode/....h":
    ...

and in setup.py simly have include_paths=[".../include"] rather than 
[".../include/cvode", ".../include/nvector", ...]. (This is how 
libraries are usually set up anyway.)

> * Importing anything from Sundials (see [12]). Currently "python
> setup.py build_ext --inplace" complains about not finding nvector.h
> [13]
> * Allocating an NVector with Sundials (see [3, 7]).
>   
Note that the call seems to allocate the ydot NVector for you (whether 
you still need to allocate other NVector's I didn't bother to check now; 
but it is likely straightforward).

The challenge here though is about function pointers.

Basically you need to define
cdef extern from ...:
   ctypedef double realtype
   ctypedef struct N_Vector:
      ...more or less copy definition from nvector header files...
   ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void 
*user_data)
   int CVodeInit(void* mem, CVRhsFn f, realtype t0, N_Vector y0)

Note that here CVRhsFn is a /type/, it is a pointer to a function, and 
CVodeInit is a function which takes such a pointer (a "callback"). So 
you'd do

cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
   ...

CVodeInit(..., your_rhs, t0, y0)

and now Sundials will know that your_rhs should be called.

Furthermore, user_data is for your use, for passing additional 
parameters to your_rhs. It can e.g. be a Python object (or Cython cdef 
class object) storing values your_rhs needs, as long as you cast it:

cdef class MyRhsProperties:
    ...

cdef MyRhsProperties props = ...
some_solve_function_taking_user_data(..., <void*>props, ...)

cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
     cdef MyRhsProperties props = <MyRhsProperties>user_data
     ...

Further refining this we get a typical pattern in these situations:

cdef class SundialsRhs:
    cdef int step(realtype t, N_vector y, N_vector ydot):
        ...

cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
     (<MyRhsProperties>user_data).step(t, y, ydot)

def foosolve(..., SundialsRhs rhs):
    some_solve_function_taking_user_data(..., <void*>rhs, ...)

which (for a small virtual dispatch penalty) allows you to subclass 
SundialsRhs to provide the callback function.

> * Handle the NVector from Python: exchange data with a Numpy array,
> and pass the NVector to a C or Cython function.
>   
Can you find the definition of NVector and paste it in an email if you 
don't succeed here? Having to fetch Sundials just to provide hints is a 
bit inconvenient...

Dag Sverre

From dalcinl at gmail.com  Wed Jun  3 18:08:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 3 Jun 2009 13:08:11 -0300
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
Message-ID: <e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>

On Wed, Jun 3, 2009 at 8:09 AM, Jon Olav Vik <jonovik at gmail.com> wrote:
>
> Somewhere inside pysundials.cvode.NVector is an array of double
> screaming to get out [6]. However, Dag Sverre told me that passing it
> to Cython may require converting between ctypes and Cython pointers.

if you have a 'ctypes' object , you can use ctypes.addressof() to get
a integer (ctypes uses PyLong_FromVoidPtr()) containing the underlying
pointer. Next, in Cython code you can use core Python C-API call
PyLong_AsVoidPtr() to get the C-side pointer.

Of course, if your odefun() is tiny, this will not alleviate the
overhead of the Python layer.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Jun  3 18:53:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 3 Jun 2009 18:53:24 +0200 (CEST)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
 differential equation models with CVODE
In-Reply-To: <e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>
Message-ID: <7626197aeb36199a01b1beaa0f266655.squirrel@webmail.uio.no>

Lisandro wrote:
> On Wed, Jun 3, 2009 at 8:09 AM, Jon Olav Vik <jonovik at gmail.com> wrote:
>>
>> Somewhere inside pysundials.cvode.NVector is an array of double
>> screaming to get out [6]. However, Dag Sverre told me that passing it
>> to Cython may require converting between ctypes and Cython pointers.
>
> if you have a 'ctypes' object , you can use ctypes.addressof() to get
> a integer (ctypes uses PyLong_FromVoidPtr()) containing the underlying
> pointer. Next, in Cython code you can use core Python C-API call
> PyLong_AsVoidPtr() to get the C-side pointer.
>
> Of course, if your odefun() is tiny, this will not alleviate the
> overhead of the Python layer.

Thanks, I wasn't aware of the addressof call. That's definitely convenient.

One thing we might want to do though is ship a ctypes.pxd for a bit faster
access directly to ctypes pointers.

Finally, while the topic is up, I personally have a wishlist item for
Cython which is automatic coercion between pointers and ctypes objects:

cdef int* i
print i # prints <ctypes.POINTER(ctypes.c_int)...> or somesuch

90% of my motivation for this behaviour is that then all cdef functions
would also be cpdef-able, and removing the need to explain that some
objects cannot be coerced to objects and back :-) Basically just more
language consistency.

Dag Sverre


From collette at physics.ucla.edu  Wed Jun  3 20:31:55 2009
From: collette at physics.ucla.edu (Andrew Collette)
Date: Wed, 3 Jun 2009 11:31:55 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
In-Reply-To: <72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>
References: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
	<72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>
Message-ID: <f125f5e0906031131nebf9677tca009911f4861bf5@mail.gmail.com>

Hi,

> Strange--it looks like this was fixed months ago:
>
> http://hg.cython.org/cython-devel/diff/8b6d42d16c99/Cython/Compiler/
> ExprNodes.py

It looks like for some reason Cython wasn't being upgraded correctly;
after sudo python setup.py install, cython --version still reported
0.10.3.  I had to manually delete the Cython directory in
site-packages and reinstall to get it to report 0.11.2.  But it works
now!

> Maybe you need to touch the .pyx to re-generate the sources. It is
> gratifying to hear that is the only issue--should we be running our
> tests with this option? (Or is it gcc only?)

I think all major compilers have a strict-ansi mode, although I'm sure
the switch is different for VC++.  I currently only use it with gcc on
OS X for compatibility with another package that looks for
__STRICT_ANSI__.  However, if Cython is intended to generate pure C89
code it might be worth adding this as a test.  At the moment it seems
this is already the case; my large-ish (9k line) Cython project now
compiles fine with -ansi.

Thanks!
Andrew

From robertwb at math.washington.edu  Wed Jun  3 21:11:22 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Jun 2009 12:11:22 -0700
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <7626197aeb36199a01b1beaa0f266655.squirrel@webmail.uio.no>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<e7ba66e40906030908n66b10ca0taaae44a3d8fb5bb8@mail.gmail.com>
	<7626197aeb36199a01b1beaa0f266655.squirrel@webmail.uio.no>
Message-ID: <C9BF6336-E87B-454B-B2B8-367A60E3057E@math.washington.edu>

On Jun 3, 2009, at 9:53 AM, Dag Sverre Seljebotn wrote:

> Lisandro wrote:
>> On Wed, Jun 3, 2009 at 8:09 AM, Jon Olav Vik <jonovik at gmail.com>  
>> wrote:
>>>
>>> Somewhere inside pysundials.cvode.NVector is an array of double
>>> screaming to get out [6]. However, Dag Sverre told me that  
>>> passing it
>>> to Cython may require converting between ctypes and Cython pointers.
>>
>> if you have a 'ctypes' object , you can use ctypes.addressof() to get
>> a integer (ctypes uses PyLong_FromVoidPtr()) containing the  
>> underlying
>> pointer. Next, in Cython code you can use core Python C-API call
>> PyLong_AsVoidPtr() to get the C-side pointer.
>>
>> Of course, if your odefun() is tiny, this will not alleviate the
>> overhead of the Python layer.
>
> Thanks, I wasn't aware of the addressof call. That's definitely  
> convenient.
>
> One thing we might want to do though is ship a ctypes.pxd for a bit  
> faster
> access directly to ctypes pointers.
>
> Finally, while the topic is up, I personally have a wishlist item for
> Cython which is automatic coercion between pointers and ctypes  
> objects:
>
> cdef int* i
> print i # prints <ctypes.POINTER(ctypes.c_int)...> or somesuch
>
> 90% of my motivation for this behaviour is that then all cdef  
> functions
> would also be cpdef-able, and removing the need to explain that some
> objects cannot be coerced to objects and back :-) Basically just more
> language consistency.

Sounds mostly like a good idea to me (though it would be exposing the  
messiness of raw C memory management to Python). I think there's a  
lot we could do with ctypes integration, at least for Python 2.5+.

- Robert



From jonovik at gmail.com  Thu Jun  4 00:30:55 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Wed, 3 Jun 2009 22:30:55 +0000 (UTC)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
Message-ID: <loom.20090603T221740-806@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:
> > * Telling Cython where to find cvode.h, nvector.h, sundials.h (see [11])
> >   
> I suspect the problem here is including "cvode", "nvector" etc. directly 
> in include_path. Instead you should likely do
> 
> cdef extern from "cvode/....h":
>     ...
> 
> and in setup.py simly have include_paths=[".../include"] rather than 
> [".../include/cvode", ".../include/nvector", ...]. (This is how 
> libraries are usually set up anyway.)

Thanks for the information; I'm not familiar with C libraries. Having looked in 
the /lib directory, I found that the library is called sundials_cvode, not just 
cvode. I modified setup.py to use "libraries=['sundials_cvode'])" and could 
then recompile without error. (I had to manually remove the build/ directory 
and hello.c, though.)

== Successful compilation ==
bash-3.2$ python setup.py build_ext --inplace
running build_ext
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/sundials-2.3.0/include -I/site/VERSIONS/
compython-2.5/Linux/include/python2.5 -c hello.c -o build/temp.linux-x86_64-2.5/
hello.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/hello.o -L/site/VERSIONS/
sundials-2.3.0/lib -L/site/VERSIONS/compython-2.5/Linux/lib -lsundials_cvode -
lpython2.5 -o hello.so

== setup.py ==
"""Run with e.g. python setup.py build_ext --inplace"""
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("hello",
            ["hello.pyx"],
            include_dirs=['/site/VERSIONS/sundials-2.3.0/include'],
            library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
            libraries=['sundials_cvode']),
    ]
)

== hello.pyx ==
cdef extern from "cvode/cvode.h":
   int CVodeSetFdata(void *cvode_mem, void *f_data)

> > * Importing anything from Sundials (see [12]). Currently "python
> > setup.py build_ext --inplace" complains about not finding nvector.h
> > [13]

==> Solved 8-)

I chose CVodeSetFdata for the above example because it didn't depend on any 
user-defined types. Now onto the next challenge...

> > * Allocating an NVector with Sundials (see [3, 7]).
> >   
> Note that the call seems to allocate the ydot NVector for you (whether 
> you still need to allocate other NVector's I didn't bother to check now; 
> but it is likely straightforward).

Yes, ydot is allocated by CVODE (though not initialized to zero, as I learned 
the hard way). I do need to allocate y, however.

As a start, I would like Cython to compile the following line of code:

y0 = N_VNew_Serial(N)

However, I'm unsure how much of the C declarations I need to duplicate (with 
ctypedef?). N_VNew_Serial returns an N_Vector:
include/nvector/nvector_serial.h:N_Vector N_VNew_Serial(long int vec_length);

The generic definition of an N_Vector is here:

== BEGIN include/sundials/sundials_nvector.h (excerpt) ==
#include <sundials/sundials_types.h>

/*
 * -----------------------------------------------------------------
 * Generic definition of N_Vector
 * -----------------------------------------------------------------
 */

/* Forward reference for pointer to N_Vector_Ops object */
typedef struct _generic_N_Vector_Ops *N_Vector_Ops;

/* Forward reference for pointer to N_Vector object */
typedef struct _generic_N_Vector *N_Vector;

/* Define array of N_Vectors */
typedef N_Vector *N_Vector_S;

/* Structure containing function pointers to vector operations  */  
struct _generic_N_Vector_Ops {
  N_Vector    (*nvclone)(N_Vector);
  N_Vector    (*nvcloneempty)(N_Vector);
  void        (*nvdestroy)(N_Vector);
  void        (*nvspace)(N_Vector, long int *, long int *);
  realtype*   (*nvgetarraypointer)(N_Vector);
  void        (*nvsetarraypointer)(realtype *, N_Vector);
  void        (*nvlinearsum)(realtype, N_Vector, realtype, N_Vector, N_Vector); 
  void        (*nvconst)(realtype, N_Vector);
  void        (*nvprod)(N_Vector, N_Vector, N_Vector);
  void        (*nvdiv)(N_Vector, N_Vector, N_Vector);
  void        (*nvscale)(realtype, N_Vector, N_Vector);
  void        (*nvabs)(N_Vector, N_Vector);
  void        (*nvinv)(N_Vector, N_Vector);
  void        (*nvaddconst)(N_Vector, realtype, N_Vector);
  realtype    (*nvdotprod)(N_Vector, N_Vector);
  realtype    (*nvmaxnorm)(N_Vector);
  realtype    (*nvwrmsnorm)(N_Vector, N_Vector);
  realtype    (*nvwrmsnormmask)(N_Vector, N_Vector, N_Vector);
  realtype    (*nvmin)(N_Vector);
  realtype    (*nvwl2norm)(N_Vector, N_Vector);
  realtype    (*nvl1norm)(N_Vector);
  void        (*nvcompare)(realtype, N_Vector, N_Vector);
  booleantype (*nvinvtest)(N_Vector, N_Vector);
  booleantype (*nvconstrmask)(N_Vector, N_Vector, N_Vector);
  realtype    (*nvminquotient)(N_Vector, N_Vector);
};

/*
 * -----------------------------------------------------------------
 * A vector is a structure with an implementation-dependent
 * 'content' field, and a pointer to a structure of vector
 * operations corresponding to that implementation.
 * -----------------------------------------------------------------
 */

struct _generic_N_Vector {
  void *content;
  struct _generic_N_Vector_Ops *ops;
};
== END include/sundials/sundials_nvector.h (excerpt) ==

(Pardon the long quote; unfortunately, I couldn't find a web repository for the 
Sundials source code. An earlier version is in this Ubuntu package, but note 
that the file hierarchy is different than the include/sundials/, include/
cvode/, etc.:
http://www.google.com/codesearch/p?hl=en#jVvm31OatEM/sundials/cvode/include/
cvode.h&q=nvector%20sundials%20cvode

(For all I know, the file hierarchy may be different in the source distribution 
and once it is installed as a library -- I'm really an absolute beginner in the 
C world.)

The abovementioned realtype (and booleantype) is defined in
include/sundials/sundials_types.h
from which I roughly quote some conditional defines:
#if defined(SUNDIALS_SINGLE_PRECISION)
typedef float realtype;
#elif defined(SUNDIALS_DOUBLE_PRECISION)
typedef double realtype;
#elif defined(SUNDIALS_EXTENDED_PRECISION)
typedef long double realtype;

As for the structs, I've tried and failed along these lines:
cdef extern from "sundials/sundials_nvector.h":
    ctypedef struct _generic_N_Vector
    ctypedef struct _generic_N_Vector *N_Vector
                                     ^
------------------------------------------------------------
/xanadu/home/jonvi/svn/trunk/cysundials/hello.pyx:8:38: Syntax error in struct 
or union definition

Do I really have to replicate the entire struct definition?


Thanks a lot for your help!

Regards,
Jon Olav



From dagss at student.matnat.uio.no  Thu Jun  4 09:25:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 09:25:04 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <loom.20090603T221740-806@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
Message-ID: <4A2776D0.7090405@student.matnat.uio.no>

Jon Olav Vik wrote:
> Yes, ydot is allocated by CVODE (though not initialized to zero, as I learned 
> the hard way). I do need to allocate y, however.
> 
> As a start, I would like Cython to compile the following line of code:
> 
> y0 = N_VNew_Serial(N)
> 
> However, I'm unsure how much of the C declarations I need to duplicate (with 
> ctypedef?). N_VNew_Serial returns an N_Vector:
> include/nvector/nvector_serial.h:N_Vector N_VNew_Serial(long int vec_length);
> 
> The generic definition of an N_Vector is here:
> 
> == BEGIN include/sundials/sundials_nvector.h (excerpt) ==
> #include <sundials/sundials_types.h>
> 
> /*
>  * -----------------------------------------------------------------
>  * Generic definition of N_Vector
>  * -----------------------------------------------------------------
>  */
> 
> /* Forward reference for pointer to N_Vector_Ops object */
> typedef struct _generic_N_Vector_Ops *N_Vector_Ops;
> 
> /* Forward reference for pointer to N_Vector object */
> typedef struct _generic_N_Vector *N_Vector;
> 
> /* Define array of N_Vectors */
> typedef N_Vector *N_Vector_S;
> 
> /* Structure containing function pointers to vector operations  */  
> struct _generic_N_Vector_Ops {
>   N_Vector    (*nvclone)(N_Vector);
>   N_Vector    (*nvcloneempty)(N_Vector);
>   void        (*nvdestroy)(N_Vector);
...
> 
> struct _generic_N_Vector {
>   void *content;
>   struct _generic_N_Vector_Ops *ops;
> };

> The abovementioned realtype (and booleantype) is defined in
> include/sundials/sundials_types.h
> from which I roughly quote some conditional defines:
> #if defined(SUNDIALS_SINGLE_PRECISION)
> typedef float realtype;
> #elif defined(SUNDIALS_DOUBLE_PRECISION)
> typedef double realtype;
> #elif defined(SUNDIALS_EXTENDED_PRECISION)
> typedef long double realtype;
> 
> As for the structs, I've tried and failed along these lines:
> cdef extern from "sundials/sundials_nvector.h":
>     ctypedef struct _generic_N_Vector
>     ctypedef struct _generic_N_Vector *N_Vector
>                                      ^
> ------------------------------------------------------------
> /xanadu/home/jonvi/svn/trunk/cysundials/hello.pyx:8:38: Syntax error in struct 
> or union definition
> 
> Do I really have to replicate the entire struct definition?

(Trivia: This is a lot like NumPy's source code.)

No, you just have to write "enough" for Cython to know what C code to 
output. Knowing what that is probably takes some C experience though; 
I'll try to help.

First, the #if-s cannot be expressed in Cython, but Cython is going in 
the direction of the type of external typedefs not to matter. I think 
you can safely do

cdef extern from ...:
     ctypedef double realtype

It should work also when Sundials is compiled with float or long double.

Then N_Vector etc. can probably be expressed like this:

cdef extern from ....:
     cdef struct _generic_N_Vector:
         void* content
         # not necesarry to declare ops unless you will be using
         # the contents of ops from Cython

     ctypedef _generic_N_Vector* N_vector

     N_Vector N_VNew_Serial(long int vec_length)

Then you will be able to do

cdef NVector v = N_VNew_Serial(10)
cdef realtype* v_data = <realtype*>v.content
v_data[0] = 34

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun  4 09:27:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 09:27:25 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <4A2776D0.7090405@student.matnat.uio.no>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
Message-ID: <4A27775D.9010801@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> First, the #if-s cannot be expressed in Cython, but Cython is going in 
> the direction of the type of external typedefs not to matter. I think 

I meant to say that the "type size" should matter as little as possible 
for external typedefs. int vs. unsigned vs. float still matters.

Also things are not 100% ready in this area in Cython. But if you know 
you are on a system where realtype is e.g. double, then just use double 
for now and we can discuss the details of this point again later...


-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Jun  4 09:35:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Jun 2009 00:35:45 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <4A2661BE.8030605@student.matnat.uio.no>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>	<C2F2563A-B960-422C-8E31-E5CD100209CA@math.washington.edu>	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
	<4A2661BE.8030605@student.matnat.uio.no>
Message-ID: <089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>

On Jun 3, 2009, at 4:42 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>  From before, relative errors for the naive vs. other algorithm,
>> 100000 runs, uniformly chosen in unit square (though nearly all
>> distributions look basically the same):
>>
>> naive
>>      better 26187
>>      avg 1.4940916064705992895601085724e-16
>>      worst 5.7659574333851360909896621025e-16
>> other
>>      better 63116
>>      avg 9.5414951065097745299683547276e-17
>>      worst 3.9584519821557590591785765975e-16
>>
>> - Robert
>>
>
> This is not my speciality, but since the problem here is with things
> like fp cancellation on subtraction etc., wouldn't it be better to
> increase the odds of wildly different values?
>
> Something like exp(uniform square)?

I did exp(X) + exp(X)*I where X was a uniform distribution on  
[-10,10] and various other distributions, all yielded very similar  
results (which was actually surprising to me).

Any thoughts on the 4x speed difference?

- Robert


From dagss at student.matnat.uio.no  Thu Jun  4 11:00:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 11:00:59 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <4A2776D0.7090405@student.matnat.uio.no>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
Message-ID: <4A278D4B.2040804@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Jon Olav Vik wrote:
>   
>> Yes, ydot is allocated by CVODE (though not initialized to zero, as I learned 
>> the hard way). I do need to allocate y, however.
>>
>> As a start, I would like Cython to compile the following line of code:
>>
>> y0 = N_VNew_Serial(N)
>>
>> However, I'm unsure how much of the C declarations I need to duplicate (with 
>> ctypedef?). N_VNew_Serial returns an N_Vector:
>> include/nvector/nvector_serial.h:N_Vector N_VNew_Serial(long int vec_length);
>>
>> The generic definition of an N_Vector is here:
>>
>> == BEGIN include/sundials/sundials_nvector.h (excerpt) ==
>> #include <sundials/sundials_types.h>
>>
>> /*
>>  * -----------------------------------------------------------------
>>  * Generic definition of N_Vector
>>  * -----------------------------------------------------------------
>>  */
>>
>> /* Forward reference for pointer to N_Vector_Ops object */
>> typedef struct _generic_N_Vector_Ops *N_Vector_Ops;
>>
>> /* Forward reference for pointer to N_Vector object */
>> typedef struct _generic_N_Vector *N_Vector;
>>
>> /* Define array of N_Vectors */
>> typedef N_Vector *N_Vector_S;
>>
>> /* Structure containing function pointers to vector operations  */  
>> struct _generic_N_Vector_Ops {
>>   N_Vector    (*nvclone)(N_Vector);
>>   N_Vector    (*nvcloneempty)(N_Vector);
>>   void        (*nvdestroy)(N_Vector);
>>     
> ...
>   
>> struct _generic_N_Vector {
>>   void *content;
>>   struct _generic_N_Vector_Ops *ops;
>> };
>>     
>
>   
>> The abovementioned realtype (and booleantype) is defined in
>> include/sundials/sundials_types.h
>> from which I roughly quote some conditional defines:
>> #if defined(SUNDIALS_SINGLE_PRECISION)
>> typedef float realtype;
>> #elif defined(SUNDIALS_DOUBLE_PRECISION)
>> typedef double realtype;
>> #elif defined(SUNDIALS_EXTENDED_PRECISION)
>> typedef long double realtype;
>>
>> As for the structs, I've tried and failed along these lines:
>> cdef extern from "sundials/sundials_nvector.h":
>>     ctypedef struct _generic_N_Vector
>>     ctypedef struct _generic_N_Vector *N_Vector
>>                                      ^
>> ------------------------------------------------------------
>> /xanadu/home/jonvi/svn/trunk/cysundials/hello.pyx:8:38: Syntax error in struct 
>> or union definition
>>
>> Do I really have to replicate the entire struct definition?
>>     
>
> (Trivia: This is a lot like NumPy's source code.)
>
> No, you just have to write "enough" for Cython to know what C code to 
> output. Knowing what that is probably takes some C experience though; 
> I'll try to help.
>
> First, the #if-s cannot be expressed in Cython, but Cython is going in 
> the direction of the type of external typedefs not to matter. I think 
> you can safely do
>
> cdef extern from ...:
>      ctypedef double realtype
>
> It should work also when Sundials is compiled with float or long double.
>
> Then N_Vector etc. can probably be expressed like this:
>
> cdef extern from ....:
>      cdef struct _generic_N_Vector:
>          void* content
>          # not necesarry to declare ops unless you will be using
>          # the contents of ops from Cython
>
>      ctypedef _generic_N_Vector* N_vector
>
>      N_Vector N_VNew_Serial(long int vec_length)
>
> Then you will be able to do
>
> cdef NVector v = N_VNew_Serial(10)
> cdef realtype* v_data = <realtype*>v.content
> v_data[0] = 34
>   
Make sure to also free/delete the vector though! (There must be a 
corresponding call there somewhere)

Dag Sverre

From barthelemy at crans.org  Thu Jun  4 11:38:20 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Thu, 4 Jun 2009 11:38:20 +0200
Subject: [Cython] running doctests in pyx files
Message-ID: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>

Hello,

I'm converting some python modules to cython ones. The original
function have doctests which I would obviously like to keep. However,
if I just let them in the pyx file (in def-ed and cpdef-ed functions
docstrings), doctest does not find them.

However, each function doctstring (myfunction.__doc__) seems fine.

Does-it mean that doctest parses the source file instead of using the
docstring from the python runtime? Or am I mistaken?

Any explanation and workaround is welcome !

a simple example is attached. I run it with python  2.6.2 (ubuntu
intrepid). Cython version 0.10.3.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hg.pyx
Type: application/octet-stream
Size: 1533 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090604/5e8f585a/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rundoctest.py
Type: text/x-python
Size: 104 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090604/5e8f585a/attachment.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: setup.py
Type: text/x-python
Size: 213 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090604/5e8f585a/attachment-0001.py 

From dagss at student.matnat.uio.no  Thu Jun  4 12:21:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 12:21:14 +0200
Subject: [Cython] running doctests in pyx files
In-Reply-To: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
Message-ID: <4A27A01A.70601@student.matnat.uio.no>

S?bastien Barth?lemy wrote:
> Hello,
>
> I'm converting some python modules to cython ones. The original
> function have doctests which I would obviously like to keep. However,
> if I just let them in the pyx file (in def-ed and cpdef-ed functions
> docstrings), doctest does not find them.
>
> However, each function doctstring (myfunction.__doc__) seems fine.
>
> Does-it mean that doctest parses the source file instead of using the
> docstring from the python runtime? Or am I mistaken?
>   
This is a problem with doctest, which uses inspect.is_function to check 
whether something is a function, which fails for Cython functions (which 
instead answer to inspect.is_builtin). It would be nice if someone would 
take this to the doctest list with a patch at some point...

There are various hackaround we could do in Cython, including wrapping 
every function in a dummy Python-interpreted function (speed loss 
though?) or have a directive to generate a __test__ dictionary. Best 
thing is probably to fix doctest but it takes ages before a patch there 
gets to our users even if it is accepted (Python 3.2?).

To get a solution right away, this is what I do to invoke testing on a 
module "cmbutils":

import cmbutils
import doctest
import inspect
cmbutils.__test__ = {}
for name in dir(cmbutils):
    value = getattr(cmbutils, name)
    if inspect.isbuiltin(value) and isinstance(value.__doc__, str):
        cmbutils.__test__[name] = value.__doc__
doctest.testmod(cmbutils)

You could also just set __test__ manually in the module, write a 
decorator which adds to __test__ for functions it decorates, etc.

Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun  4 12:22:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 04 Jun 2009 12:22:57 +0200
Subject: [Cython] running doctests in pyx files
In-Reply-To: <4A27A01A.70601@student.matnat.uio.no>
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
	<4A27A01A.70601@student.matnat.uio.no>
Message-ID: <4A27A081.8020807@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> S?bastien Barth?lemy wrote:
>> Hello,
>>
>> I'm converting some python modules to cython ones. The original
>> function have doctests which I would obviously like to keep. However,
>> if I just let them in the pyx file (in def-ed and cpdef-ed functions
>> docstrings), doctest does not find them.
>>
>> However, each function doctstring (myfunction.__doc__) seems fine.
>>
>> Does-it mean that doctest parses the source file instead of using the
>> docstring from the python runtime? Or am I mistaken?
>>   
> This is a problem with doctest, which uses inspect.is_function to 
> check whether something is a function, which fails for Cython 
> functions (which instead answer to inspect.is_builtin). It would be 
> nice if someone would take this to the doctest list with a patch at 
> some point...
>
> There are various hackaround we could do in Cython, including wrapping 
> every function in a dummy Python-interpreted function (speed loss 
> though?) or have a directive to generate a __test__ dictionary. Best 
> thing is probably to fix doctest but it takes ages before a patch 
> there gets to our users even if it is accepted (Python 3.2?).
>
> To get a solution right away, this is what I do to invoke testing on a 
> module "cmbutils":
>
> import cmbutils
> import doctest
> import inspect
> cmbutils.__test__ = {}
> for name in dir(cmbutils):
>    value = getattr(cmbutils, name)
>    if inspect.isbuiltin(value) and isinstance(value.__doc__, str):
>        cmbutils.__test__[name] = value.__doc__
> doctest.testmod(cmbutils)
>
> You could also just set __test__ manually in the module, write a 
> decorator which adds to __test__ for functions it decorates, etc.
BTW if you get that last solution to work, it would be nice if you could 
take the time to add it to the FAQ :-)

Dag Sverre

From dalcinl at gmail.com  Thu Jun  4 16:18:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 11:18:05 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>

On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> I think we should file a bug report against Py3.1 and even bring this to
> the attention of python-dev before it's too late.
>

Stefan, I saw your post in python-dev and corresponding issue in the
tracker, but I got confused. Is the solution know and the issue going
to be fixed?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Jun  4 16:27:43 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 11:27:43 -0300
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>
	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>
	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>
	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>
	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
	<4A2661BE.8030605@student.matnat.uio.no>
	<089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>
Message-ID: <e7ba66e40906040727q520c8fcemdd0c28129d73f501@mail.gmail.com>

On Thu, Jun 4, 2009 at 4:35 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Jun 3, 2009, at 4:42 AM, Dag Sverre Seljebotn wrote:
>
> Any thoughts on the 4x speed difference?
>

Any chance such difference could be related to the fabs() function
call? Could you test an explicit comparison?

const %(real_type)s abs_breal = b.real < 0 ? -b.real : b.real;
const %(real_type)s abs_bimag = b.imag < 0 ? -b.imag : b.imag;


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From barthelemy at crans.org  Thu Jun  4 17:37:19 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Thu, 4 Jun 2009 17:37:19 +0200
Subject: [Cython] running doctests in pyx files
In-Reply-To: <4A27A081.8020807@student.matnat.uio.no>
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
	<4A27A01A.70601@student.matnat.uio.no>
	<4A27A081.8020807@student.matnat.uio.no>
Message-ID: <78f7ab620906040837k67e1fb82ncb32e43de83ea93@mail.gmail.com>

Hello Dag,

thank you for your help,

>> This is a problem with doctest, which uses inspect.is_function to check
>> whether something is a function, which fails for Cython functions (which
>> instead answer to inspect.is_builtin). It would be nice if someone would
>> take this to the doctest list with a patch at some point...

I had a quick look at it but there is too much I do not understand in
both cython and doctest there to work on this now.

>> To get a solution right away, this is what I do to invoke testing on a
>> module "cmbutils":
>> ...

this works fine, as long as you're not doing "from somewhere import
something" in which case, "something" is tested too (and fails because
of bad context).

I added a _from_module() test (found in doctest.py) which seem to fix that.

> BTW if you get that last solution to work, it would be nice if you could
> take the time to add it to the FAQ :-)

Done.
http://wiki.cython.org/FAQ#Doctestskipstestsfromcythonfunctions.2Cwhy.3FIsthereaworkaround.3F

regards, and thank you again.

From stefan_ml at behnel.de  Thu Jun  4 18:29:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 4 Jun 2009 18:29:07 +0200 (CEST)
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
Message-ID: <7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel wrote:
>>
>> I think we should file a bug report against Py3.1 and even bring this to
>> the attention of python-dev before it's too late.
>>
>
> Stefan, I saw your post in python-dev and corresponding issue in the
> tracker, but I got confused. Is the solution know and the issue going
> to be fixed?

I didn't find the time to look into this any deeper. I'm not sure it will
get fixed, because a) the bug itself is a lot older than rc1, and b) the
comment sounded more like "the fix might be similar to what we did here".

I think we should still open a new bug. Given that you experienced the
problem, could you do that?

Stefan


From dalcinl at gmail.com  Thu Jun  4 19:27:22 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 14:27:22 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A2800E6.4030806@gmail.com>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<4A2800E6.4030806@gmail.com>
Message-ID: <e7ba66e40906041027j1281d86y154e7a0497a65a6e@mail.gmail.com>

On Thu, Jun 4, 2009 at 2:14 PM, David Martin <frostedcheerios at gmail.com> wrote:
>
> I've attached a patch for my addition of the "--mingw" option to
> runtests.py.

Looks good.

> There was a small issue regarding trailing backslashes in the
> INCLUDE_DIRS list, and aside from that, I couldn't get the mingw compiler to
> work correctly unless I emptied the INCLUDE_DIRS list (I run from a VS
> command prompt for our build system, so it will always have VS directories
> in my "include" env variable). It definitely shouldn't assign an empty list
> for INCLUDE_DIRS if --mingw is given, but it is the only way I could get it
> to work quickly. =)
>

I understand...

> If you know what needs to be done to fix those errors I can definitely help
> out. I've also attached the log containing the small amount of tests that
> are failing for me when using the MinGW compiler, which I believe are the
> same tests that are failing for you.
>

I'm tempted to either remove that getenv INCLUDE, or change the name.
That getenv was needed long ago for running numpy-specific tests, It
seems we could get rid of that hackery.

What other people on cython-dev think?



> -David
>
> --- runtests.py 2009-05-20 10:37:46.000000000 -0500
> +++ runtests222.py ? ? ?2009-06-04 11:48:07.000000000 -0500
> @@ -3,6 +3,7 @@
> ?import os, sys, re, shutil, unittest, doctest
>
> ?WITH_CYTHON = True
> +MINGW = False
>
> ?from distutils.dist import Distribution
> ?from distutils.core import Extension
> @@ -32,7 +33,9 @@
> ?# ? ?(2,4) : lambda x: x in ['run.set']
> ?}
>
> -INCLUDE_DIRS = [ d for d in os.getenv('INCLUDE', '').split(os.pathsep) if d
> ]
> +# Remove any trailing backslashes in the include paths because they cause
> +# gcc to incorrectly handle spaces in the paths that follow.
> +INCLUDE_DIRS = [ d.rstrip('\\') for d in os.getenv('INCLUDE',
> '').split(os.pathsep) if d ]
> ?CFLAGS = os.getenv('CFLAGS', '').split()
>
> ?class build_ext(_build_ext):
> @@ -277,6 +280,8 @@
> ? ? ? ? os.chdir(workdir)
> ? ? ? ? try:
> ? ? ? ? ? ? build_extension = build_ext(distutils_distro)
> + ? ? ? ? ? ?if MINGW:
> + ? ? ? ? ? ? ? ?build_extension.compiler = 'mingw32'
> ? ? ? ? ? ? build_extension.include_dirs = INCLUDE_DIRS[:]
> ? ? ? ? ? ? if incdir:
> ? ? ? ? ? ? ? ? build_extension.include_dirs.append(incdir)
> @@ -505,7 +510,11 @@
> ? ? ? ? ? ? ? ? ? ? ? help="do not run the file based tests")
> ? ? parser.add_option("--no-pyregr", dest="pyregr",
> ? ? ? ? ? ? ? ? ? ? ? action="store_false", default=True,
> - ? ? ? ? ? ? ? ? ? ? ?help="do not run the regression tests of CPython in
> tests/pyregr/")
> + ? ? ? ? ? ? ? ? ? ? ?help="do not run the regression tests of CPython in
> tests/pyregr/")
> + ? ?if sys.platform.startswith('win'):
> + ? ? ? ?parser.add_option("--mingw", dest="mingw",
> + ? ? ? ? ? ? ? ? ? ? ? ? action="store_true", default=False,
> + ? ? ? ? ? ? ? ? ? ? ? ? help="compile C/C++ code using the MinGW
> compilers")
> ? ? parser.add_option("--cython-only", dest="cython_only",
> ? ? ? ? ? ? ? ? ? ? ? action="store_true", default=False,
> ? ? ? ? ? ? ? ? ? ? ? help="only compile pyx to c, do not run C compiler or
> run the tests")
> @@ -547,6 +556,11 @@
> ? ? ? ? import coverage
> ? ? ? ? coverage.erase()
> ? ? ? ? coverage.start()
> +
> + ? ?if sys.platform.startswith('win'):
> + ? ? ? ?MINGW = options.mingw
> + ? ? ? ?if MINGW:
> + ? ? ? ? ? ?INCLUDE_DIRS = []
>
> ? ? WITH_CYTHON = options.with_cython
>
>
> Running tests against Cython 0.11.2
> Python 2.5.4 (r254:67916, Mar 25 2009, 13:58:21) [MSC v.1310 32 bit (Intel)]
>
> belchenko1.cpp: In function `void initbelchenko1()':
> belchenko1.cpp:430: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x22c): undefined reference to `p1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x230): undefined reference to `f1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x236): undefined reference to `p2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x23a): undefined reference to `f2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x240): undefined reference to `p3'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x244): undefined reference to `f3 at 0'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x24a): undefined reference to `p4'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.o:callingconvention.c:(.text+0
> x24e): undefined reference to `@f4 at 0'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x22c): undefined reference to `p1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x230): undefined reference to `f1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x236): undefined reference to `p2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x23a): undefined reference to `f2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x240): undefined reference to `p3'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x244): undefined reference to `f3 at 0'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x24a): undefined reference to `p4'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.o:callingconvention.cpp:(.te
> xt+0x24e): undefined reference to `@f4 at 0'
> collect2: ld returned 1 exit status
> cnamespec.c: In function `c_spam':
> cnamespec.c:343: warning: '__pyx_v_p' might be used uninitialized in this
> function
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\cnamespec.o:cnamespec.c:(.text+0xf):
> undefined r
> eference to `c_a'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\cnamespec.o:cnamespec.c:(.text+0x20):
> undefined
> reference to `c_b'
> collect2: ld returned 1 exit status
> cnamespec.cpp: In function `double c_spam(int, float)':
> cnamespec.cpp:343: warning: '__pyx_v_p' might be used uninitialized in this
> function
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cnamespec.o:cnamespec.cpp:(.text+0xf):
> undefin
> ed reference to `c_a'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cnamespec.o:cnamespec.cpp:(.text+0x20):
> undefi
> ned reference to `c_b'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercearraytoptr.o:coercearraytoptr.c:(.text+0xd
> ): undefined reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercearraytoptr.o:coercearraytoptr.c:(.text+0x1
> 8): undefined reference to `spam'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\coercearraytoptr.o:coercearraytoptr.cpp:(.text
> +0xd): undefined reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\coercearraytoptr.o:coercearraytoptr.cpp:(.text
> +0x18): undefined reference to `spam'
> collect2: ld returned 1 exit status
> complexbasetype.cpp: In function `void initcomplexbasetype()':
> complexbasetype.cpp:433: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp_exceptions_T265.cpp:
> (.text+0x1e6): undefined reference to `generic_error()'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp_exceptions_T265.cpp:
> (.text+0x25a): undefined reference to `specified_error()'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_t265.o:cpp_exceptions_T265.cpp:
> (.text+0x2e0): undefined reference to `dynamic_error()'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\declarations.o:declarations.c:(.text+0x5):
> undef
> ined reference to `ifnp'
> collect2: ld returned 1 exit status
> declarations.cpp: In function `void initdeclarations()':
> declarations.cpp:472: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\declarations.o:declarations.cpp:(.text+0x5):
> u
> ndefined reference to `ifnp'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.o:excvalcheck.c:(.text+0x7):
> undefin
> ed reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.o:excvalcheck.c:(.text+0x33):
> undefi
> ned reference to `grail'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.o:excvalcheck.c:(.text+0x64):
> undefi
> ned reference to `tomato'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.o:excvalcheck.cpp:(.text+0x7):
> und
> efined reference to `spam'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.o:excvalcheck.cpp:(.text+0x33):
> un
> defined reference to `grail'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.o:excvalcheck.cpp:(.text+0x64):
> un
> defined reference to `tomato'
> collect2: ld returned 1 exit status
> extern.cpp: In function `void initextern()':
> extern.cpp:440: warning: statement has no effect
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x19):
> undefined referenc
> e to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x21):
> undefined referenc
> e to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x26):
> undefined referenc
> e to `e1'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.o:nogil.c:(.text+0x2b):
> undefined referenc
> e to `e2'
> collect2: ld returned 1 exit status
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x1a):
> undefined refe
> rence to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x22):
> undefined refe
> rence to `g2'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x27):
> undefined refe
> rence to `e1()'
> E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.o:nogil.cpp:(.text+0x2c):
> undefined refe
> rence to `e2()'
> collect2: ld returned 1 exit status
>
> ======================================================================
> ERROR: compiling (c) callingconvention
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) callingconvention
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) cnamespec
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) cnamespec
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) coercearraytoptr
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) coercearraytoptr
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) cpp_exceptions_T265
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) declarations
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) declarations
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) excvalcheck
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) excvalcheck
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (c) nogil
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'gcc' failed with exit status 1
>
> ======================================================================
> ERROR: compiling (cpp) nogil
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "runtests.py", line 226, in runTest
> ? ?self.runCompileTest()
> ?File "runtests.py", line 230, in runCompileTest
> ? ?self.directory, self.expect_errors, self.annotate)
> ?File "runtests.py", line 336, in compile
> ? ?self.run_distutils(module, workdir, incdir)
> ?File "runtests.py", line 304, in run_distutils
> ? ?build_extension.run()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 290, in run
> ? ?self.build_extensions()
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 416, in
> build_extensions
> ? ?self.build_extension(ext)
> ?File "runtests.py", line 48, in build_extension
> ? ?_build_ext.build_extension(self, ext)
> ?File "E:\py\build\lib\distutils\command\build_ext.py", line 513, in
> build_extension
> ? ?target_lang=language)
> ?File "E:\py\build\lib\distutils\ccompiler.py", line 845, in
> link_shared_object
> ? ?extra_preargs, extra_postargs, build_temp, target_lang)
> ?File "E:\py\build\lib\distutils\cygwinccompiler.py", line 246, in link
> ? ?target_lang)
> ?File "E:\py\build\lib\distutils\unixccompiler.py", line 254, in link
> ? ?raise LinkError, msg
> LinkError: command 'g++' failed with exit status 1
>
> ----------------------------------------------------------------------
> Ran 1477 tests in 1347.264s
>
> FAILED (errors=13)
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Jun  4 20:22:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Jun 2009 11:22:55 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <e7ba66e40906040727q520c8fcemdd0c28129d73f501@mail.gmail.com>
References: <e7ba66e40905141556v2c0f8d1dgb4c2e85deb8e24d3@mail.gmail.com>
	<ac1e4c420905141631x787ff269u69ee61255cbd5915@mail.gmail.com>
	<e7ba66e40905141648k30a105cepb6c288298cbddd27@mail.gmail.com>
	<ac1e4c420905141812q243f7b5exf02459731e5c3b3a@mail.gmail.com>
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
	<e7ba66e40905142030t4a51b1b3o198fc2ceefff6b7b@mail.gmail.com>
	<05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>
	<0F01C86E-33E7-49EF-B134-A61AD616D005@math.washington.edu>
	<4A2661BE.8030605@student.matnat.uio.no>
	<089FF999-37AE-439E-8186-451604D5954F@math.washington.edu>
	<e7ba66e40906040727q520c8fcemdd0c28129d73f501@mail.gmail.com>
Message-ID: <B6C1A95B-6C67-459A-B504-06B5839B55B8@math.washington.edu>

On Jun 4, 2009, at 7:27 AM, Lisandro Dalcin wrote:

> On Thu, Jun 4, 2009 at 4:35 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Jun 3, 2009, at 4:42 AM, Dag Sverre Seljebotn wrote:
>>
>> Any thoughts on the 4x speed difference?
>>
>
> Any chance such difference could be related to the fabs() function
> call? Could you test an explicit comparison?
>
> const %(real_type)s abs_breal = b.real < 0 ? -b.real : b.real;
> const %(real_type)s abs_bimag = b.imag < 0 ? -b.imag : b.imag;

This is actually what I did first, and got the 4x slowdown, so I  
switched to fabs hoping it would be faster (despite only working for  
doubles for now), but it wasn't (on my machine at least).

- Robert


From dalcinl at gmail.com  Thu Jun  4 21:48:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 16:48:59 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A28174C.7010609@gmail.com>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	<e7ba66e40905271636u1b8f366coe5f0bef82350d600@mail.gmail.com>
	<4A2800E6.4030806@gmail.com>
	<e7ba66e40906041027j1281d86y154e7a0497a65a6e@mail.gmail.com>
	<4A28174C.7010609@gmail.com>
Message-ID: <e7ba66e40906041248w20c80aecm60e37f1edb9c6dbf@mail.gmail.com>

On Thu, Jun 4, 2009 at 3:49 PM, David Martin <frostedcheerios at gmail.com> wrote:
> Lisandro Dalcin wrote:
>
> I was mostly talking about the linker errors with "cdef extern" in the
> tests, but fixing the INCLUDE_DIRS issue would also be nice. If you know
> what needs to be done to fix the tests regarding cdef extern and any of the
> other failures in that log I sent you I can get to work on those.
>

Well, I'm not sure how to fix these tests without destroying what them
are trying to test. As quick fist fix would be to disable them for the
Win case.

BTW, there are things that should be improved in Cython. For example,

1) If we write "cdef extern void foo()", then Cython should emit a
prototype with DL_IMPORT
2) If we write "cdef public void bar(): pass", then Cython should emit
a prototype with DL_EXPORT

Currently, Cython used DL_EXPORT for both of them.

Of course, a key question about using DL_IMPORT... What would happen
if the 'void foo()' function is not actually in a DLL, but on a static
library, or some extra C file built together with the Cython-generated
C code to build-up the final extension module?





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonovik at gmail.com  Thu Jun  4 22:58:54 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Thu, 4 Jun 2009 20:58:54 +0000 (UTC)
Subject: [Cython]
	=?utf-8?q?Sundials_with_Cython=3F_Integrating/solving_or?=
	=?utf-8?q?dinary=09differential_equation_models_with_CVODE?=
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
Message-ID: <loom.20090604T194945-627@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:
> > Do I really have to replicate the entire struct definition?
> 
> No, you just have to write "enough" for Cython to know what C code to 
> output. Knowing what that is probably takes some C experience though; 
> I'll try to help.
> 
> First, the #if-s cannot be expressed in Cython, but Cython is going in 
> the direction of the type of external typedefs not to matter. I think 
> you can safely do
> 
> cdef extern from ...:
>      ctypedef double realtype
> 
> It should work also when Sundials is compiled with float or long double.
> 
> Then N_Vector etc. can probably be expressed like this:
> 
> cdef extern from ....:
>      cdef struct _generic_N_Vector:
>          void* content
>          # not necesarry to declare ops unless you will be using
>          # the contents of ops from Cython
> 
>      ctypedef _generic_N_Vector* N_vector
> 
>      N_Vector N_VNew_Serial(long int vec_length)
> 
> Then you will be able to do
> 
> cdef NVector v = N_VNew_Serial(10)
> cdef realtype* v_data = <realtype*>v.content
> v_data[0] = 34

Thank you so much! This seems to work well [1] after some adjustments, and I've 
even been able to print and modify the contents of an N_Vector.

* I found I had to add libraries=['sundials_cvode', 'sundials_nvecserial'] to 
get at the N_Vector functions. (Otherwise, the module compiled without error 
but failed on import with "ImportError: ./hello.so: undefined symbol: 
N_VNew_Serial".)
* I needed to cast v.content to <N_VectorContent_Serial>, not to <realtype*>. 
N_VectorContent_Serial is a struct with a .data field which apparently is what 
I'm after. [2]
* When trying to ctypedef N_VectorContent_Serial, I didn't realize it resided 
in another header file. Until I fetched it with 'cdef extern from "nvector/
nvector_serial.h"', I got these warnings (which I won't ignore again 8-):
hello.c:418: warning: implicit declaration of function 'N_VNew_Serial'
hello.c:418: warning: assignment makes pointer from integer without a cast
hello.c:436: warning: implicit declaration of function 'N_VPrint_Serial'
hello.c:476: warning: implicit declaration of function 'N_VDestroy_Serial'


Thanks a lot for your help so far!

Best regards,
Jon Olav



[1]
== compilation, running, output ==
bash-3.2$ rm -rf build hello.so hello.c && python setup.py build_ext --inplace 
&& python -c "import hello"
running build_ext
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/sundials-2.3.0/include -I/site/VERSIONS/
compython-2.5/Linux/include/python2.5 -c hello.c -o build/temp.linux-x86_64-2.5/
hello.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/hello.o -L/site/VERSIONS/
sundials-2.3.0/lib -L/site/VERSIONS/compython-2.5/Linux/lib -lsundials_cvode -
lsundials_nvecserial -lpython2.5 -o hello.so
1.1297001e-312
1.1297001e-312
2.3547898e-310

          0
          1
          2

== hello.pyx ==
"""Trying to modify an NVector"""

cdef extern from "sundials/sundials_types.h":
    ctypedef double realtype

cdef extern from "sundials/sundials_nvector.h":
    cdef struct _generic_N_Vector:
        void* content
    ctypedef _generic_N_Vector* N_Vector
    N_Vector N_VNew_Serial(long int vec_length)
    void N_VDestroy_Serial(N_Vector v)
    void N_VPrint_Serial(N_Vector v)

cdef extern from "nvector/nvector_serial.h":
    cdef struct _N_VectorContent_Serial:
        long int length
        realtype* data
    ctypedef _N_VectorContent_Serial* N_VectorContent_Serial


cdef N_Vector v = N_VNew_Serial(3)
cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
N_VPrint_Serial(v) # prints uninitialized floats
for i in range(3):
    v_data[i] = i
N_VPrint_Serial(v) # prints 0, 1, 2
N_VDestroy_Serial(v)

== setup.py ==
"""Run with e.g. python setup.py build_ext --inplace
Or, to remove intermediate files from previous builds:
rm -rf build hello.so hello.c && python setup.py build_ext --inplace && python -
c "import hello"
"""

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("hello",
            ["hello.pyx"],
            include_dirs=['/site/VERSIONS/sundials-2.3.0/include'],
            library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
            libraries=['sundials_cvode', 'sundials_nvecserial']),
    ]
)


[2]
This is what showed the problem in treating v.content as an array of double, 
"cdef realtype* v_data = <realtype*>v.content". Before I modified v_data[0], 
N_VPrint_Serial() would correctly print the (uninitialized) floats in the 
vector. After modifying v_data[0], it printed uninitialized floats ad infinitum.

Looking at the source code of N_VPrint_Serial() [3], I saw that it used the 
macros NV_LENGTH_S and NV_DATA_S, which accessed fields of the struct type 
N_VectorContent_Serial.

== excerpt from nvector_serial.h ==
#define NV_CONTENT_S(v)  ( (N_VectorContent_Serial)(v->content) )
#define NV_LENGTH_S(v)   ( NV_CONTENT_S(v)->length )
void N_VPrint_Serial(N_Vector x)
{
  long int i, N;
  realtype *xd;
  N  = NV_LENGTH_S(x);
  xd = NV_DATA_S(x);
  for (i=0; i < N; i++) {
    printf("%11.8g\n", *xd++);
  }
  printf("\n");
}


[3]
http://www.google.com/codesearch/p?hl=en#jVvm31OatEM/sundials/nvec_ser/
nvector_serial.c&q=nvector%2520sundials%2520cvode



From dalcinl at gmail.com  Thu Jun  4 23:49:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 18:49:25 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>

On Thu, Jun 4, 2009 at 1:29 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin wrote:
>> On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel wrote:
>>>
>>> I think we should file a bug report against Py3.1 and even bring this to
>>> the attention of python-dev before it's too late.
>>>
>>
>> Stefan, I saw your post in python-dev and corresponding issue in the
>> tracker, but I got confused. Is the solution know and the issue going
>> to be fixed?
>
> I didn't find the time to look into this any deeper. I'm not sure it will
> get fixed, because a) the bug itself is a lot older than rc1, and b) the
> comment sounded more like "the fix might be similar to what we did here".
>
> I think we should still open a new bug. Given that you experienced the
> problem, could you do that?
>

Done, http://bugs.python.org/issue6195, with patch, I've added Stefan
to the noisy list.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonovik at gmail.com  Fri Jun  5 00:34:12 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Thu, 4 Jun 2009 22:34:12 +0000 (UTC)
Subject: [Cython]
	=?utf-8?q?Sundials_with_Cython=3F_Integrating/solving_or?=
	=?utf-8?q?dinary=09differential_equation_models_with_CVODE?=
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
	<loom.20090604T194945-627@post.gmane.org>
Message-ID: <loom.20090604T222116-214@post.gmane.org>

Jon Olav Vik <jonovik at ...> writes:
> Thank you so much! This seems to work well [1] after some adjustments, and 
> I've even been able to print and modify the contents of an N_Vector.

Now I've had a go at another subtask:

> * Handle the NVector from Python: exchange data with a Numpy array,
> and pass the NVector to a C or Cython function.

The functions arr2nv() and nv2arr() work [1], at least within Cython. It can 
probably be done faster, but I don't expect frequent copying to/from N_Vectors.

I did try "cdef N_Vector arr2nv(np.ndarray[np.int_t] x):", based on
http://wiki.cython.org/tutorials/numpy#Efficientindexing
but I couldn't "cimport numpy as np" as required to get at np.int_t. Any ideas 
why I'd get "error: numpy/arrayobject.h: No such file or directory"? [2]

Regards,
Jon Olav


[1]
== compilation, running, output ==
bash-3.2$ rm -rf build hello.so hello.c && python setup.py build_ext --inplace 
&& python -c "import hello"
running build_ext
cythoning hello.pyx to hello.c
building 'hello' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/sundials-2.3.0/include -I/site/VERSIONS/
compython-2.5/Linux/include/python2.5 -c hello.c -o build/temp.linux-x86_64-2.5/
hello.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/hello.o -L/site/VERSIONS/
sundials-2.3.0/lib -L/site/VERSIONS/compython-2.5/Linux/lib -lsundials_cvode -
lsundials_nvecserial -lpython2.5 -o hello.so
1.1297001e-312
4.4472177e-316
1.5810101e-322

          0
          1
          2

[ 0.  1.  2.]
          0
          1
          2

== hello.pyx ==
"""Trying to modify an NVector"""

cdef extern from "sundials/sundials_types.h":
    ctypedef double realtype

cdef extern from "sundials/sundials_nvector.h":
    cdef struct _generic_N_Vector:
        void* content
    ctypedef _generic_N_Vector* N_Vector
    N_Vector N_VNew_Serial(long int vec_length)
    void N_VDestroy_Serial(N_Vector v)
    void N_VPrint_Serial(N_Vector v)

cdef extern from "nvector/nvector_serial.h":
    cdef struct _N_VectorContent_Serial:
        long int length
        realtype* data
    ctypedef _N_VectorContent_Serial* N_VectorContent_Serial

import numpy as np
cdef nv2arr(N_Vector v):
    cdef long int n = (<N_VectorContent_Serial>v.content).length
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    x = np.empty(n)
    for i in range(n):
        x[i] = v_data[i]
    return x

# cimport numpy as np
# cdef N_Vector arr2nv(np.ndarray[np.int_t] x):
cdef N_Vector arr2nv(x):
    cdef long int n = len(x)
    cdef N_Vector v = N_VNew_Serial(n)
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    for i in range(n):
        v_data[i] = x[i]
    return v

cdef N_Vector v = N_VNew_Serial(3)
cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
N_VPrint_Serial(v) # prints uninitialized floats
for i in range(3):
    v_data[i] = i
N_VPrint_Serial(v)
x = nv2arr(v)
print x
N_VDestroy_Serial(v)
v = arr2nv(x)
N_VPrint_Serial(v)
N_VDestroy_Serial(v)

[2]
"cimport numpy" fails inside my hello.pyx too; here follows a minimal example.

bash-3.2$ cat > cimp.pyx
cimport numpy
bash-3.2$ python -c "import pyximport; pyximport.install(build_dir='.'); import 
cimp"
./temp.linux-x86_64-2.5/pyrex/cimp.c:136:31: error: numpy/arrayobject.h: No 
such file or directory
./temp.linux-x86_64-2.5/pyrex/cimp.c:595: error: expected '=', ',', ';', 'asm' 
or '__attribute__' before '__pyx_t_5numpy_int8_t'
...
./temp.linux-x86_64-2.5/pyrex/cimp.c:639: error: expected '=', ',', ';', 'asm' 
or '__attribute__' before '__pyx_t_5numpy_complex_t'
./temp.linux-x86_64-2.5/pyrex/cimp.c:650: error: expected ')' before '*' token
./temp.linux-x86_64-2.5/pyrex/cimp.c: In function 
'__pyx_pf_5numpy_7ndarray___getbuffer__':
./temp.linux-x86_64-2.5/pyrex/cimp.c:737: error: 'PyArray_Descr' undeclared 
(first use in this function)
./temp.linux-x86_64-2.5/pyrex/cimp.c:737: error: (Each undeclared identifier is 
reported only once
./temp.linux-x86_64-2.5/pyrex/cimp.c:737: error: for each function it appears 
in.)
...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usit/titan/u1/jonvi/usr//lib/python2.5/site-packages/pyximport/
pyximport.py", line 288, in load_module
    self.pyxbuild_dir)
  File "/usit/titan/u1/jonvi/usr//lib/python2.5/site-packages/pyximport/
pyximport.py", line 154, in load_module
    raise ImportError("Building module failed: %s" % e)
ImportError: Building module failed: CompileError(DistutilsExecError("command 
'gcc' failed with exit status 1",),)



From dalcinl at gmail.com  Fri Jun  5 02:31:03 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 4 Jun 2009 21:31:03 -0300
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
In-Reply-To: <loom.20090604T222116-214@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
	<loom.20090604T194945-627@post.gmane.org>
	<loom.20090604T222116-214@post.gmane.org>
Message-ID: <e7ba66e40906041731v2d15672dy23fb82d492616b38@mail.gmail.com>

On Thu, Jun 4, 2009 at 7:34 PM, Jon Olav Vik <jonovik at gmail.com> wrote:
> Jon Olav Vik <jonovik at ...> writes:
>
> I did try "cdef N_Vector arr2nv(np.ndarray[np.int_t] x):", based on
> http://wiki.cython.org/tutorials/numpy#Efficientindexing
> but I couldn't "cimport numpy as np" as required to get at np.int_t. Any ideas
> why I'd get "error: numpy/arrayobject.h: No such file or directory"? [2]
>

Because the C compiler requires the include path to numpy headers. The
function numpy.get_include() returns it, however I do not remember if
pyximport has the fixes to let you pass such include path. Could you
try to do in your shell

$ export CFLAGS=-I`python -c 'import numpy; print numpy.get_include()'`

and next try to run your code?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonovik at gmail.com  Fri Jun  5 08:39:09 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Fri, 5 Jun 2009 06:39:09 +0000 (UTC)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090603T221740-806@post.gmane.org>
	<4A2776D0.7090405@student.matnat.uio.no>
	<loom.20090604T194945-627@post.gmane.org>
	<loom.20090604T222116-214@post.gmane.org>
	<e7ba66e40906041731v2d15672dy23fb82d492616b38@mail.gmail.com>
Message-ID: <loom.20090605T062635-632@post.gmane.org>

Lisandro Dalcin <dalcinl at ...> writes:

> On Thu, Jun 4, 2009 at 7:34 PM, Jon Olav Vik <jonovik at ...> wrote:
> > Jon Olav Vik <jonovik at ...> writes:
> >
> > I did try "cdef N_Vector arr2nv(np.ndarray[np.int_t] x):", based on
> > http://wiki.cython.org/tutorials/numpy#Efficientindexing
> > but I couldn't "cimport numpy as np" as required to get at np.int_t. Any 
ideas
> > why I'd get "error: numpy/arrayobject.h: No such file or directory"? [2]
> >
> 
> Because the C compiler requires the include path to numpy headers. The
> function numpy.get_include() returns it, however I do not remember if
> pyximport has the fixes to let you pass such include path. Could you
> try to do in your shell
> 
> $ export CFLAGS=-I`python -c 'import numpy; print numpy.get_include()'`
> 
> and next try to run your code?

The cimport statement now passes, though I'm not sure what to make of the "type-
punned" warnings [1]. "Defined but not used" doesn't sound too scary, though.

I remember now that we used numpy.get_include() in Dag Sverre's workshop. 
Adding "import numpy as np" and "include_dirs=['/site/VERSIONS/sundials-2.3.0/
include', np.get_include ()]," worked for my setup.py too (the "hello.pyx" 
example mentioned previously).

Thank you very much for your help. Later I'll have a go at actually running 
CVODE.

Best regards,
Jon Olav Vik


[1]
bash-3.2$ cat cimp.pyx
cimport numpy
print "cimported numpy"
bash-3.2$ python -c "import pyximport; pyximport.install(build_dir='.'); import 
cimp"
./temp.linux-x86_64-2.5/pyrex/cimp.c: In function '__Pyx_PyObject_IsTrue':
./temp.linux-x86_64-2.5/pyrex/cimp.c:3131: warning: dereferencing type-punned 
pointer will break strict-aliasing rules
./temp.linux-x86_64-2.5/pyrex/cimp.c:3132: warning: dereferencing type-punned 
pointer will break strict-aliasing rules
./temp.linux-x86_64-2.5/pyrex/cimp.c: At top level:
/site/VERSIONS/compython-2.5/Linux/lib/python2.5/site-packages/numpy/core/
include/numpy/__multiarray_api.h:959: warning: '_import_array' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:687: warning: '__pyx_k_3' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:688: warning: '__pyx_k_4' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:708: warning: '__pyx_k_24' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:714: warning: '__pyx_k_26' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:715: warning: '__pyx_k_27' defined but not 
used
./temp.linux-x86_64-2.5/pyrex/cimp.c:729: warning: 
'__pyx_pf_5numpy_7ndarray___getbuffer__' defined but not used
./temp.linux-x86_64-2.5/pyrex/cimp.c:1487: warning: 
'__pyx_pf_5numpy_7ndarray___releasebuffer__' defined but not used
cimported numpy



From jonovik at gmail.com  Fri Jun  5 15:41:41 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Fri, 5 Jun 2009 13:41:41 +0000 (UTC)
Subject: [Cython] Sundials with Cython? Integrating/solving ordinary
	differential equation models with CVODE
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>
	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>
	<4A266871.9070209@student.matnat.uio.no>
Message-ID: <loom.20090605T132731-705@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:
> The challenge here though is about function pointers.
> 
> Basically you need to define
> cdef extern from ...:
>    ctypedef double realtype
>    ctypedef struct N_Vector:
>       ...more or less copy definition from nvector header files...
>    ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void 
> *user_data)
>    int CVodeInit(void* mem, CVRhsFn f, realtype t0, N_Vector y0)
> 
> Note that here CVRhsFn is a /type/, it is a pointer to a function, and 
> CVodeInit is a function which takes such a pointer (a "callback"). So 
> you'd do
> 
> cdef int your_rhs(realtype t, N_Vector y, N_vector ydot, void* user_data):
>    ...
> 
> CVodeInit(..., your_rhs, t0, y0)
> 
> and now Sundials will know that your_rhs should be called.

I'm pleased to report that Sundials is finally doing my bidding from within 
Cython [1]! As I become more familiar with the cdef extern syntax, it's fairly 
easy to import new functions from Sundials/CVODE. I'll come back to efficient 
indexing [2] later.

If I may make a wish, I would like Cython to find more declarations 
automatically. For instance, having

cdef extern * from "cvode/cvode.h"

or, if that is impractical,

cdef extern from "cvode/cvode.h":
    CVodeCreate, CVRhsFn, CVodeMalloc, CVodeFree, CVode, CV_ADAMS, CV_BDF

instead of:

cdef extern from "cvode/cvode.h":
    void* CVodeCreate(int lmm, int iter)
    ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void *f_data)
    int CVodeMalloc(void *cvode_mem, CVRhsFn f, realtype t0, N_Vector y0,
        int itol, realtype reltol, void *abstol)
    void CVodeFree(void **cvode_mem)
    int CVode(void *cvode_mem, realtype tout, N_Vector yout, realtype *tret, 
        int itask)
    # constants
    ## /* lmm */
    DEF CV_ADAMS = 1
    DEF CV_BDF   = 2
    ...

I haven't benchmarked anything yet, but the relative ease of adapting the C 
tutorial for CVODE [3] is uplifting. As a last question, what is the cleanest 
way to transfer arrays of double between realtype* and Numpy vectors (one-
dimensional arrays)?

Thank you very much to all who helped me!

Best regards,
Jon Olav


[1]
== setup.py ==
"""Run with e.g. python setup.py build_ext --inplace
Or, to remove intermediate files from previous builds:
rm -rf build hello.so hello.c && python setup.py build_ext --inplace && python -
c "import hello"
"""
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy as np
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("hello",
            ["hello.pyx"],
            include_dirs=['/site/VERSIONS/sundials-2.3.0/include', 
np.get_include ()],
            library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
            libraries=['sundials_cvode', 'sundials_nvecserial']),
    ]
)

== hello.pyx ==
"""
Cythonizing simple usage of CVODE

Build and test with
rm -rf build hello.so hello.c && python setup.py build_ext --inplace && python -
c "import hello"
"""

cdef extern from "sundials/sundials_types.h":
    ctypedef double realtype

cdef extern from "sundials/sundials_nvector.h":
    cdef struct _generic_N_Vector:
        void* content
    ctypedef _generic_N_Vector* N_Vector
    N_Vector N_VNew_Serial(long int vec_length)
    void N_VDestroy_Serial(N_Vector v)
    void N_VPrint_Serial(N_Vector v)

cdef extern from "nvector/nvector_serial.h":
    cdef struct _N_VectorContent_Serial:
        long int length
        realtype* data
    ctypedef _N_VectorContent_Serial* N_VectorContent_Serial

cdef extern from "cvode/cvode.h":
    void* CVodeCreate(int lmm, int iter)
    ctypedef int (*CVRhsFn)(realtype t, N_Vector y, N_Vector ydot, void *f_data)
    int CVodeMalloc(void *cvode_mem, CVRhsFn f, realtype t0, N_Vector y0,
        int itol, realtype reltol, void *abstol)
    void CVodeFree(void **cvode_mem)
    int CVode(void *cvode_mem, realtype tout, N_Vector yout, realtype *tret, 
        int itask)
    # constants
    ## /* lmm */
    DEF CV_ADAMS = 1
    DEF CV_BDF   = 2
    ## /* iter */
    DEF CV_FUNCTIONAL = 1
    DEF CV_NEWTON     = 2
    ## /* itol */
    DEF CV_SS = 1
    DEF CV_SV = 2
    DEF CV_WF = 3
    ## /* itask */
    DEF CV_NORMAL         = 1
    DEF CV_ONE_STEP       = 2
    DEF CV_NORMAL_TSTOP   = 3
    DEF CV_ONE_STEP_TSTOP = 4
    ## /* CVODE return flags */
    DEF CV_SUCCESS = 0
    DEF CV_TSTOP_RETURN = 1
    DEF CV_ROOT_RETURN = 2

# how to find this stuff: 
# grep "CVDense(" /xanadu/site/common/VERSIONS/sundials-2.3.0/include/*/*
# /xanadu/site/common/VERSIONS/sundials-2.3.0/include/
# ... cvode/cvode_dense.h:int CVDense(void *cvode_mem, long int N);
cdef extern from "cvode/cvode_dense.h":
    int CVDense(void *cvode_mem, long int N)


import numpy as np
cdef nv2arr(N_Vector v):
    """Create new numpy array from N_Vector"""
    cdef long int n = (<N_VectorContent_Serial>v.content).length
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    x = np.empty(n)
    for i in range(n):
        x[i] = v_data[i]
    return x

cimport numpy as np
# cdef N_Vector arr2nv(np.ndarray[np.int_t] x):
cdef N_Vector arr2nv(x):
    """Create new N_Vector from numpy array"""
    cdef long int n = len(x)
    cdef N_Vector v = N_VNew_Serial(n)
    cdef realtype* v_data = (<N_VectorContent_Serial>v.content).data
    cdef long int i
    for i in range(n):
        v_data[i] = x[i]
    return v

# following https://computation.llnl.gov/casc/sundials/documentation/
# ... cv_guide/node5.html#SECTION00540000000000000000

# set problem dimensions
N = 2

# set vector of initial values
cdef N_Vector y0 = arr2nv(np.arange(N)) # N_VMake_Serial(N, ydata)

# create CVODE object
cdef void* cvode_mem = CVodeCreate(CV_BDF, CV_NEWTON)

## ODE right-hand side
cdef int vanderpol (realtype t, N_Vector y_nv, N_Vector ydot_nv, void *f_data):
    """van der Pol equation adapted from example_ode.py"""
    cdef realtype* y = (<N_VectorContent_Serial>y_nv.content).data
    cdef realtype* ydot = (<N_VectorContent_Serial>ydot_nv.content).data
    ydot[0] = y[1]
    ydot[1] = (1 - y[0] * y[0]) * y[1] - y[0] # eps[0] * ...
    return 0

### testing RHS
print "Current state:"
N_VPrint_Serial(y0) # 0, 1
cdef N_Vector ydot = arr2nv(np.zeros(N)) # N_VMake_Serial(N, ydata)
vanderpol(0.0, y0, ydot, <void*>None)
print "Rates of change:"
N_VPrint_Serial(ydot) # 1, 1

# initialize CVODE solver
# specify integration tolerances
# (note: Sundials 2.3 uses CVodeMalloc instead of CVodeInit + CVodeSStolerances)
cdef realtype abstol = 1e-4
flag = CVodeMalloc(cvode_mem, vanderpol, 0.0, y0, CV_SS, 1e-4, &abstol)
# if flag == CV_SUCCESS: print "CVodeMalloc returned CV_SUCCESS"

# set optional inputs

# attach linear solver module
flag = CVDense(cvode_mem, N)

# set linear solver optional inputs

# specify rootfinding problem

# advance solution in time
tmax = 150
y = np.zeros((tmax, N))
t = np.zeros((tmax,))
cdef realtype tout = 2.0
cdef realtype tret
cdef int i, j
cdef realtype* v_data = (<N_VectorContent_Serial>y0.content).data
for i in range(tmax):
    flag = CVode(cvode_mem, tout, y0, &tret, CV_ONE_STEP)
    if flag != CV_SUCCESS:
        break
    t[i] = tret
    for j in range(N):
        y[i,j] = v_data[j]

# get optional outputs

# deallocate memory for solution vector
N_VDestroy_Serial(ydot)
N_VDestroy_Serial(y0)

# free solver memory
CVodeFree(&cvode_mem)

# display results
import matplotlib as mpl
mpl.use("svg")
import pylab
pylab.plot(t, y)
pylab.savefig("cvode.svgz")

[2] http://docs.cython.org/docs/numpy_tutorial.html#efficient-indexing

[3] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/
node5.html#SECTION00540000000000000000


From barthelemy at crans.org  Fri Jun  5 17:46:32 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Fri, 5 Jun 2009 17:46:32 +0200
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
Message-ID: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>

Hello all,

I have a project involving cython files with the following file hierarchy.

/setup.py
/src/homogeneousmatrix.pyx
/src/homogeneousmatrix.pxd
/src/__init__.py
/src/...

However, I have some trouble writing a working  setup.py. With the
version below, generating src/homogeneousmatrix.c fails, apparently
because it does not use the corresponding pxd file, where some
necessary ctypedef are made.


I can compile everything with this setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    name='arboris',
    packages=['arboris',
 	          'arboris.robots'],
    package_dir = {'arboris':'src'},
    cmdclass = {'build_ext': build_ext},
    ext_package='arboris',
    ext_modules = [Extension("homogeneousmatrix",
                             ["src/homogeneousmatrix.pyx"])]
)


However, with the following one,  generating src/homogeneousmatrix.c
fails, apparently because cython does not find/use the corresponding
pxd file

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    name='arboris',
    packages=['arboris',
                  'arboris.robots'],
    package_dir = {'arboris':'src'},
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("arboris.homogeneousmatrix",
                             ["src/homogeneousmatrix.pyx"])]
)

I was wondering if this was an expected behaviour or a bug.

Have a nice week-end all.

From efiring at hawaii.edu  Fri Jun  5 21:38:46 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 05 Jun 2009 09:38:46 -1000
Subject: [Cython] support for numpy bool type
Message-ID: <4A297446.6030309@hawaii.edu>

In writing a cython extension to work with numpy masked arrays, I needed 
to work with the mask, which is dtype('bool').  Therefore I was 
expecting to be able to use np.bool_t, in analogy to np.float_t. 
Instead I had to use np.int8_t, and cast the input to np.int8 in a 
python wrapper.  Can bool_t be added to numpy.pxd?

Thanks.

Eric

From mattcbro at earthlink.net  Fri Jun  5 21:42:39 2009
From: mattcbro at earthlink.net (Matthew Bromberg)
Date: Fri, 05 Jun 2009 15:42:39 -0400
Subject: [Cython] Cython Memory Management
Message-ID: <4A29752F.7070901@earthlink.net>

I am not entirely clear how memory management works in cython for 
extension classes and their attributes/members.

I have a class in a .pxd file defined like,

cdef class rmat :
     """  Real matrix class binds to both vsipl and numpy ndarray.
     The ndarray object is returned via getarr.  The array is bound
     back to vsipl via admit """
     cdef vsip_mview_f *vm
     # this should be allowed but currently isn't
     # cdef np.ndarray[np.float32_t, ndim=2] arr
     # cdef vsip_block_f *block
     cdef object arr
     cpdef rmat parent
  ....


So apparently I can create an rmat object in python.  I have 
__cinit__()  and  __dealloc__() methods defined.

Question 1,  are references to an rmat object reference counted and 
garbage collected?
So far it seems like they are.

Question 2)  Are extension class members set to  other  extension 
classes or python objects,  such as  arr and parent  reference counted?
ie suppose I create a real matrix such as,
 >> m = rmat(2,3)
 >> m.arr = array1
 >> m.parent =  m0

The assignments to the class members are actually done inside a .pyx 
file so they are done via cython.  Do such assignments increment the 
refcounts and are things properly garbage collected when the reference 
counts go to zero?

Again it seems like this is true,  but it's not entirely obvious.  I 
might expect  member arr to be properly refcounted since it's explicitly 
declared as a python object,  but the parent member is declared as an 
extension type object.  In theory it could just treat it as a C pointer. 
What if I had declared it via a cdef instead of cpdef,  does that make 
any difference?

  I have some anecdotal evidence that it is properly garbage collected,  
but I don't know the rules or restrictions on this since the 
documentation only talks briefly about these issues in a couple of places.

Thanks in advance for your help,



From robertwb at math.washington.edu  Fri Jun  5 22:21:37 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Jun 2009 13:21:37 -0700
Subject: [Cython] Cython Memory Management
In-Reply-To: <4A29752F.7070901@earthlink.net>
References: <4A29752F.7070901@earthlink.net>
Message-ID: <644608A4-0148-4DC7-B861-1909C82A9648@math.washington.edu>

On Jun 5, 2009, at 12:42 PM, Matthew Bromberg wrote:

> I am not entirely clear how memory management works in cython for
> extension classes and their attributes/members.
>
> I have a class in a .pxd file defined like,
>
> cdef class rmat :
>      """  Real matrix class binds to both vsipl and numpy ndarray.
>      The ndarray object is returned via getarr.  The array is bound
>      back to vsipl via admit """
>      cdef vsip_mview_f *vm
>      # this should be allowed but currently isn't
>      # cdef np.ndarray[np.float32_t, ndim=2] arr
>      # cdef vsip_block_f *block
>      cdef object arr
>      cpdef rmat parent
>   ....

There's no such thing as a cpdef attribute, just cdef, cpdef only  
makes sense for functions.

> Question 1,  are references to an rmat object reference counted and
> garbage collected?
> So far it seems like they are.
>
> Question 2)  Are extension class members set to  other  extension
> classes or python objects,  such as  arr and parent  reference  
> counted?

Yes, anything that is an object (including subclasses thereof) is  
reference counted. You should never have to worry about reference  
counts unless you're casting objects to or from pointer types.

- Robert


From robertwb at math.washington.edu  Fri Jun  5 22:28:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Jun 2009 13:28:55 -0700
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
Message-ID: <9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>

On Jun 5, 2009, at 8:46 AM, S?bastien Barth?lemy wrote:

> Hello all,
>
> I have a project involving cython files with the following file  
> hierarchy.
>
> /setup.py
> /src/homogeneousmatrix.pyx
> /src/homogeneousmatrix.pxd
> /src/__init__.py
> /src/...
>
> However, I have some trouble writing a working  setup.py. With the
> version below, generating src/homogeneousmatrix.c fails, apparently
> because it does not use the corresponding pxd file, where some
> necessary ctypedef are made.
>
>
> I can compile everything with this setup.py:
>
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
>     name='arboris',
>     packages=['arboris',
>  	          'arboris.robots'],
>     package_dir = {'arboris':'src'},
>     cmdclass = {'build_ext': build_ext},
>     ext_package='arboris',
>     ext_modules = [Extension("homogeneousmatrix",
>                              ["src/homogeneousmatrix.pyx"])]
> )
>
>
> However, with the following one,  generating src/homogeneousmatrix.c
> fails, apparently because cython does not find/use the corresponding
> pxd file
>
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
>     name='arboris',
>     packages=['arboris',
>                   'arboris.robots'],
>     package_dir = {'arboris':'src'},
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("arboris.homogeneousmatrix",
>                              ["src/homogeneousmatrix.pyx"])]
> )
>
> I was wondering if this was an expected behaviour or a bug.

If you don't want to specify ext_package, you should probably lay out  
your sources as you would for normal .py files, e.g. put everything  
in the arboris package in an "arboris" directory rather than a "src"  
directory. Perhaps Cython should be inspecting package_dir?

- Robert


From jonovik at gmail.com  Sat Jun  6 00:57:20 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Sat, 6 Jun 2009 00:57:20 +0200
Subject: [Cython] Running doctests with Cython?
Message-ID: <cd166c030906051557m5387d0efwf191427db22e96a@mail.gmail.com>

I'm cythonizing an existing module with lots of doctests, which would be useful
in catching any errors or changes in behaviour that might result. However, I
have trouble running doctests in a cython-compiled module. The compiled module
offers the usual __doc__ strings (simple example follows below), yet python -m
fails with "No code object available". Is there a trick?

Thanks in advance for any help.

Best regards,
Jon Olav Vik


-bash-3.2$ cat cydoctest.pyx
def pyfun(x):
    """
    >>> pyfun(3)
    6
    """
    return x * 2

-bash-3.2$ rm -rf *.pyc build *.so *.c && python setup.py build_ext --inplace
running build_ext
cythoning cydoctest.pyx to cydoctest.c
building 'cydoctest' extension
creating build
creating build/temp.linux-x86_64-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/site/VERSIONS/compython-2.5/Linux/include/python2.5 -c
cydoctest.c -o build/temp.linux-x86_64-2.5/cydoctest.o
gcc -pthread -shared build/temp.linux-x86_64-2.5/cydoctest.o -L/site/VERSIONS/
compython-2.5/Linux/lib -lpython2.5 -o cydoctest.so

-bash-3.2$ python -c "import cydoctest; print cydoctest.pyfun.__doc__; print
cydoctest"
    >>> pyfun(3)
    6
<module 'cydoctest' from 'cydoctest.so'>

-bash-3.2$ python cydoctest.pyx -v
Trying:
    pyfun(3)
Expecting:
    6
ok
1 items had no tests:
    __main__
1 items passed all tests:
   1 tests in __main__.pyfun
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

-bash-3.2$ python -m cydoctest
Traceback (most recent call last):
  File "/site/VERSIONS/compython-2.5/Linux/lib/python2.5/runpy.py", line 90, in
run_module
    raise ImportError("No code object available for " + mod_name)
ImportError: No code object available for cydoctest

From glenn at tarbox.org  Sat Jun  6 06:00:45 2009
From: glenn at tarbox.org (Glenn Tarbox, PhD)
Date: Fri, 5 Jun 2009 21:00:45 -0700
Subject: [Cython] Trivial numpy test in sage notebook fails...
Message-ID: <f049b35c0906052100l41f5a77do73e755859b24f0dc@mail.gmail.com>

Here's the code which is the first cell of a sage notebook (sage 4.0)

%cython
import numpy as np
cimport numpy as np
DTYPE = np.int
ctypedef np.int_t DTYPE_t

Here's the error:

Traceback (most recent call last):    DTYPE = np.int
  File "/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py",
line 408, in cython_import_all
    create_local_c_file=create_local_c_file)
  File "/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py",
line 385, in cython_import
    create_local_c_file=create_local_c_file)
  File "/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/misc/cython.py",
line 367, in cython
    raise RuntimeError, "Error converting %s to C:\n%s\n%s"%(filename, log, err)
RuntimeError: Error converting
/home/tarbox/.sage/sage_notebook/worksheets/admin/89/code/sage23.spyx
to C:


Error converting Pyrex file to C:
------------------------------------------------------------
...

include "cdefs.pxi"
import numpy as np
cimport numpy as np
DTYPE = np.int
ctypedef np.int_t DTYPE_t        ^
------------------------------------------------------------

/home/tarbox/.sage/temp/hq2/12471/spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx_0.pyx:9:9:
'int_t' is not a type identifier


I tried with and without the "import numpy" directive thinking it may be
redundant...

This example comes from the "cython for numpy users" tutorial on the wiki
where it mentions that all numpy types have a corresponding _t type

There is indeed a numpy.int type... so I was wondering if the new work with
python buffers changed how things work.

-glenn

-- 
Glenn H. Tarbox, PhD ||  206-274-6919
http://www.tarbox.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090605/ccce6c30/attachment.htm 

From barthelemy at crans.org  Sat Jun  6 07:49:35 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Sat, 6 Jun 2009 07:49:35 +0200
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
Message-ID: <78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>

2009/6/5 Robert Bradshaw <robertwb at math.washington.edu>:
> On Jun 5, 2009, at 8:46 AM, S?bastien Barth?lemy wrote:
>> I have a project involving cython files with the following file
>> hierarchy.
>>
>> /setup.py
>> /src/homogeneousmatrix.pyx
>> /src/homogeneousmatrix.pxd
>> /src/__init__.py
>> /src/...

[...]

> If you don't want to specify ext_package, you should probably lay out
> your sources as you would for normal .py files, e.g. put everything
> in the arboris package in an "arboris" directory rather than a "src"
> directory.

I think my sources are currently laid out as for normal python source.
(while I agree renaming src into arboris would be nice.)


By the way, if I use

dir = {'arboris':'src'},
   ext_modules = [Extension("arboris.homogeneousmatrix",
                            ["src/homogeneousmatrix.pyx"])]

(that is without ext_package), cython -- as called by distutils --
does not use/find the pxd. But if I run cython directly from the
command line (cython src/homogeneousmatrix.pyx) it does. This
discrepancy sounds like a bug to me.


> Perhaps Cython should be inspecting package_dir?

I also think so, but I'm certainly not an expert. I also don't now why
distutils uses two different properties for packages (packages and
ext_package).

-- 
Sebastien

From dagss at student.matnat.uio.no  Sat Jun  6 09:05:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 06 Jun 2009 09:05:00 +0200
Subject: [Cython] Running doctests with Cython?
Message-ID: <3327123929.3069139@smtp.netcom.no>

This was just answered two days ago in another thread.

Dag Sverre Seljebotn
-----Original Message-----
From: Jon Olav Vik <jonovik at gmail.com>
Date: Saturday, Jun 6, 2009 12:58 am
Subject: [Cython] Running doctests with Cython?
To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net

I'm cythonizing an existing module with lots of doctests, which would be useful in catching any errors or changes in behaviour that might result. However, I have trouble running doctests in a cython-compiled module. The compiled module offers the usual __doc__ strings (simple example follows below), yet python -m fails with "No code object available". Is there a trick?
>
>Thanks in advance for any help.
>
>Best regards,
>Jon Olav Vik
>
>
>-bash-3.2$ cat cydoctest.pyx
>def pyfun(x):
>    """
>    >>> pyfun(3)
>    6
>    """
>    return x * 2
>
>-bash-3.2$ rm -rf *.pyc build *.so *.c && python setup.py build_ext --inplace running build_ext
>cythoning cydoctest.pyx to cydoctest.c
>building 'cydoctest' extension
>creating build
>creating build/temp.linux-x86_64-2.5
>gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict- prototypes -fPIC -I/site/VERSIONS/compython-2.5/Linux/include/python2.5 -c cydoctest.c -o build/temp.linux-x86_64-2.5/cydoctest.o
>gcc -pthread -shared build/temp.linux-x86_64-2.5/cydoctest.o -L/site/VERSIONS/ compython-2.5/Linux/lib -lpython2.5 -o cydoctest.so
>
>-bash-3.2$ python -c "import cydoctest; print cydoctest.pyfun.__doc__; print cydoctest"
>    >>> pyfun(3)
>    6
><module 'cydoctest' from 'cydoctest.so'>
>
>-bash-3.2$ python cydoctest.pyx -v
>Trying:
>    pyfun(3)
>Expecting:
>    6
>ok
>1 items had no tests:
>    __main__
>1 items passed all tests:
>   1 tests in __main__.pyfun
>1 tests in 2 items.
>1 passed and 0 failed.
>Test passed.
>
>-bash-3.2$ python -m cydoctest
>Traceback (most recent call last):
>  File "/site/VERSIONS/compython-2.5/Linux/lib/python2.5/runpy.py", line 90, in run_module
>    raise ImportError("No code object available for " + mod_name)
>ImportError: No code object available for cydoctest
>_______________________________________________
>Cython-dev mailing list
>Cython-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/cython-dev
>


From efiring at hawaii.edu  Sat Jun  6 09:08:46 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Fri, 05 Jun 2009 21:08:46 -1000
Subject: [Cython] support for numpy bool type
In-Reply-To: <4A297446.6030309@hawaii.edu>
References: <4A297446.6030309@hawaii.edu>
Message-ID: <4A2A15FE.8020201@hawaii.edu>

Eric Firing wrote:
> In writing a cython extension to work with numpy masked arrays, I needed 
> to work with the mask, which is dtype('bool').  Therefore I was 
> expecting to be able to use np.bool_t, in analogy to np.float_t. 
> Instead I had to use np.int8_t, and cast the input to np.int8 in a 
> python wrapper.  Can bool_t be added to numpy.pxd?
> 
> Thanks.
> 
> Eric


Attached is a diff that I think takes care of adding bool support.

Eric



-------------- next part --------------
A non-text attachment was scrubbed...
Name: cython_numpy_bool.diff
Type: text/x-patch
Size: 1722 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090605/5428c85b/attachment.bin 

From dagss at student.matnat.uio.no  Sat Jun  6 09:09:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: 06 Jun 2009 09:09:00 +0200
Subject: [Cython] Trivial numpy test in sage notebook fails...
Message-ID: <3327124144.3090608@smtp.netcom.no>

afaik this is a problem with the Sage notebook. Try to Cythonize from the command line, and if that works, take it to the Sage list.

Dag Sverre Seljebotn
-----Original Message-----
From: "Glenn Tarbox, PhD" <glenn at tarbox.org>
Date: Saturday, Jun 6, 2009 6:01 am
Subject: [Cython] Trivial numpy test in sage notebook fails...
To: Cython-dev at codespeak.netReply-To: cython-dev at codespeak.net

Here's the code which is the first cell of a sage notebook (sage 4.0)
>
>%cython
>import numpy as np
>cimport numpy as np
>DTYPE = np.int
>ctypedef np.int_t DTYPE_t
>
>Here's the error:
>
>Traceback (most recent call last):    DTYPE = np.int
>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py', line 408, in cython_import_all
>    create_local_c_file=create_local_c_file)
>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/server/support.py', line 385, in cython_import
>    create_local_c_file=create_local_c_file)
>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/sage/misc/cython.py', line 367, in cython
>    raise RuntimeError, 'Error converting %s to C:\n%s\n%s'%(filename, log, err)
>RuntimeError: Error converting /home/tarbox/.sage/sage_notebook/worksheets/admin/89/code/sage23.spyx to C:
>
>
>Error converting Pyrex file to C:
>------------------------------------------------------------
>...
>
>include 'cdefs.pxi'
>import numpy as np
>cimport numpy as np
>DTYPE = np.int
> ctypedef np.int_t DTYPE_t        ^
>------------------------------------------------------------
>
>/home/tarbox/.sage/temp/hq2/12471/spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx/_home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx_0.pyx:9:9: 'int_t' is not a type identifier 
>I tried with and without the 'import numpy' directive thinking it may be redundant...
>
>This example comes from the 'cython for numpy users' tutorial on the wiki where it mentions that all numpy types have a corresponding _t type
>
>There is indeed a numpy.int type... so I was wondering if the new work with python buffers changed how things work.
>
>-glenn
>
>-- 
>Glenn H. Tarbox, PhD || ?206-274-6919
>http://www.tarbox.org
>


From robertwb at math.washington.edu  Sat Jun  6 09:13:09 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 6 Jun 2009 00:13:09 -0700
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
	<78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
Message-ID: <880357DD-058E-417E-A247-E48735B98118@math.washington.edu>

On Jun 5, 2009, at 10:49 PM, S?bastien Barth?lemy wrote:

> 2009/6/5 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Jun 5, 2009, at 8:46 AM, S?bastien Barth?lemy wrote:
>>> I have a project involving cython files with the following file
>>> hierarchy.
>>>
>>> /setup.py
>>> /src/homogeneousmatrix.pyx
>>> /src/homogeneousmatrix.pxd
>>> /src/__init__.py
>>> /src/...
>
> [...]
>
>> If you don't want to specify ext_package, you should probably lay out
>> your sources as you would for normal .py files, e.g. put everything
>> in the arboris package in an "arboris" directory rather than a "src"
>> directory.
>
> I think my sources are currently laid out as for normal python source.
> (while I agree renaming src into arboris would be nice.)

Normally A/B/C.py corresponds to the module A.B.C. Is there any  
reason you can't rename src to arboris?

> By the way, if I use
>
> dir = {'arboris':'src'},
>    ext_modules = [Extension("arboris.homogeneousmatrix",
>                             ["src/homogeneousmatrix.pyx"])]
>
> (that is without ext_package), cython -- as called by distutils --
> does not use/find the pxd. But if I run cython directly from the
> command line (cython src/homogeneousmatrix.pyx) it does. This
> discrepancy sounds like a bug to me.

That is because it's actually creating a module  
"src.homogeneousmatrix" and it finds "homogeneousmatrix.pxd" just  
fine. Running Cython directly doesn't read setup.py at all.

- Robert

From barthelemy at crans.org  Sat Jun  6 09:34:47 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Sat, 6 Jun 2009 09:34:47 +0200
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <880357DD-058E-417E-A247-E48735B98118@math.washington.edu>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
	<78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
	<880357DD-058E-417E-A247-E48735B98118@math.washington.edu>
Message-ID: <78f7ab620906060034y6c9e399eo149fbb3d9a36cb1f@mail.gmail.com>

2009/6/6 Robert Bradshaw <robertwb at math.washington.edu>:
> On Jun 5, 2009, at 10:49 PM, S?bastien Barth?lemy wrote:
>> I think my sources are currently laid out as for normal python source.
>> (while I agree renaming src into arboris would be nice.)
>
> Normally A/B/C.py corresponds to the module A.B.C.

ok. I was thinking of "normal distutils way" which understands package_dir

> Is there any
> reason you can't rename src to arboris?

no, I did it indeed.

> is because it's actually creating a module
> "src.homogeneousmatrix" and it finds "homogeneousmatrix.pxd" just
> fine. Running Cython directly doesn't read setup.py at all.

thank you for the explanations. I'm almost fully enlighten. The only
dark spot left is the point of the distutil ext_package option.

-- 
Sebastien

From dagss at student.matnat.uio.no  Sat Jun  6 11:20:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 06 Jun 2009 11:20:10 +0200
Subject: [Cython] support for numpy bool type
In-Reply-To: <4A2A15FE.8020201@hawaii.edu>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
Message-ID: <4A2A34CA.8000702@student.matnat.uio.no>

Eric Firing wrote:
> Eric Firing wrote:
>> In writing a cython extension to work with numpy masked arrays, I 
>> needed to work with the mask, which is dtype('bool').  Therefore I 
>> was expecting to be able to use np.bool_t, in analogy to np.float_t. 
>> Instead I had to use np.int8_t, and cast the input to np.int8 in a 
>> python wrapper.  Can bool_t be added to numpy.pxd?
>>
>> Thanks.
>>
>> Eric
The reason it is not there is that AFAIK Cython has no support for an 
8-bit boolean type. I.e. you want

print arr[3] # should print "True", not 1.

and furthermore you want

arr[3] = obj # should mean arr[3] = bool(obj)

Which brings up the questions
1) I suppose the best thing would be to add support for a "bool" which 
would be a C99 _Bool if compiling on C99, and "unsigned char" with 
appropriate restrictions otherwise. Thoughts?
2) Is it better to keep ndarray[bool_t] disabled until this happens, or 
should it be enabled (as an "unsigned char" array) and then change its 
semantics later?

>
> Attached is a diff that I think takes care of adding bool support.

Thanks, I might apply it depending on the answers I get to the above 
questions.

Dag Sverre

From dagss at student.matnat.uio.no  Sat Jun  6 19:11:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 06 Jun 2009 19:11:28 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <loom.20090605T132731-705@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090605T132731-705@post.gmane.org>
Message-ID: <4A2AA340.2030004@student.matnat.uio.no>

Jon Olav Vik wrote:
> I haven't benchmarked anything yet, but the relative ease of adapting the C 
> tutorial for CVODE [3] is uplifting. As a last question, what is the cleanest 
> way to transfer arrays of double between realtype* and Numpy vectors (one-
> dimensional arrays)?
>   
Just a note: There is a nice and fairly canonical way of doing this, but 
it takes some typing and I'm close to a deadline for an exam project. 
I'll write it down on Tuesday or so if noone beats me to it.

Dag Sverre

From robertwb at math.washington.edu  Sat Jun  6 20:18:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 6 Jun 2009 11:18:51 -0700
Subject: [Cython] without ext_package, setup.py cannot find my pxd files
In-Reply-To: <78f7ab620906060034y6c9e399eo149fbb3d9a36cb1f@mail.gmail.com>
References: <78f7ab620906050846s58c124b8xee3a2a4c1f10cb86@mail.gmail.com>
	<9020D279-07B7-4F4A-A478-2A2ED3C2A0C4@math.washington.edu>
	<78f7ab620906052249v21f4edc7hab3753c4dc968293@mail.gmail.com>
	<880357DD-058E-417E-A247-E48735B98118@math.washington.edu>
	<78f7ab620906060034y6c9e399eo149fbb3d9a36cb1f@mail.gmail.com>
Message-ID: <13DE9389-6DDE-43E4-B9CB-2BA7D7AE6494@math.washington.edu>

On Jun 6, 2009, at 12:34 AM, S?bastien Barth?lemy wrote:

> 2009/6/6 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Jun 5, 2009, at 10:49 PM, S?bastien Barth?lemy wrote:
>>> I think my sources are currently laid out as for normal python  
>>> source.
>>> (while I agree renaming src into arboris would be nice.)
>>
>> Normally A/B/C.py corresponds to the module A.B.C.
>
> ok. I was thinking of "normal distutils way" which understands  
> package_dir
>
>> Is there any
>> reason you can't rename src to arboris?
>
> no, I did it indeed.
>
>> is because it's actually creating a module
>> "src.homogeneousmatrix" and it finds "homogeneousmatrix.pxd" just
>> fine. Running Cython directly doesn't read setup.py at all.
>
> thank you for the explanations.

You're welcome.

> I'm almost fully enlighten. The only
> dark spot left is the point of the distutil ext_package option.

That's a mystery to me too...

- Robert



From robertwb at math.washington.edu  Sat Jun  6 22:02:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 6 Jun 2009 13:02:11 -0700
Subject: [Cython] support for numpy bool type
In-Reply-To: <4A2A34CA.8000702@student.matnat.uio.no>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
Message-ID: <E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>

On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:

> Eric Firing wrote:
>> Eric Firing wrote:
>>> In writing a cython extension to work with numpy masked arrays, I
>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>> was expecting to be able to use np.bool_t, in analogy to np.float_t.
>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>
>>> Thanks.
>>>
>>> Eric
> The reason it is not there is that AFAIK Cython has no support for an
> 8-bit boolean type. I.e. you want
>
> print arr[3] # should print "True", not 1.
>
> and furthermore you want
>
> arr[3] = obj # should mean arr[3] = bool(obj)
>
> Which brings up the questions
> 1) I suppose the best thing would be to add support for a "bool" which
> would be a C99 _Bool if compiling on C99, and "unsigned char" with
> appropriate restrictions otherwise. Thoughts?

How would this relate to the current bint type? bint is an int  
because logical operations in C are ints, and also several operations  
in the Python/C API (and elsewhere) return true/false values as ints.

> 2) Is it better to keep ndarray[bool_t] disabled until this  
> happens, or
> should it be enabled (as an "unsigned char" array) and then change its
> semantics later?

I would be OK with allowing bool_t to be a synonym for int8_t, and  
then widening the semantics later (I don't see what would break,  
unless one was storing actual integer (rather than boolean) values).

- Robert


From efiring at hawaii.edu  Sat Jun  6 22:14:34 2009
From: efiring at hawaii.edu (Eric Firing)
Date: Sat, 06 Jun 2009 10:14:34 -1000
Subject: [Cython] support for numpy bool type
In-Reply-To: <E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
	<E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
Message-ID: <4A2ACE2A.8040906@hawaii.edu>

Robert Bradshaw wrote:
> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
> 
>> Eric Firing wrote:
>>> Eric Firing wrote:
>>>> In writing a cython extension to work with numpy masked arrays, I
>>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>>> was expecting to be able to use np.bool_t, in analogy to np.float_t.
>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>>
>>>> Thanks.
>>>>
>>>> Eric
>> The reason it is not there is that AFAIK Cython has no support for an
>> 8-bit boolean type. I.e. you want
>>
>> print arr[3] # should print "True", not 1.
>>
>> and furthermore you want
>>
>> arr[3] = obj # should mean arr[3] = bool(obj)
>>
>> Which brings up the questions
>> 1) I suppose the best thing would be to add support for a "bool" which
>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>> appropriate restrictions otherwise. Thoughts?
> 
> How would this relate to the current bint type? bint is an int  
> because logical operations in C are ints, and also several operations  
> in the Python/C API (and elsewhere) return true/false values as ints.
> 
>> 2) Is it better to keep ndarray[bool_t] disabled until this  
>> happens, or
>> should it be enabled (as an "unsigned char" array) and then change its
>> semantics later?
> 
> I would be OK with allowing bool_t to be a synonym for int8_t, and  
> then widening the semantics later (I don't see what would break,  
> unless one was storing actual integer (rather than boolean) values).

I think it should be uint8_t, not int8_t, because numpy booleans are 
stored as unsigned char.

Eric

> 
> - Robert
> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Sat Jun  6 22:22:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 6 Jun 2009 22:22:35 +0200 (CEST)
Subject: [Cython] support for numpy bool type
In-Reply-To: <E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
	<E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
Message-ID: <a236bdcbdd2d91e6838e52565565d5ac.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
>
>> Eric Firing wrote:
>>> Eric Firing wrote:
>>>> In writing a cython extension to work with numpy masked arrays, I
>>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>>> was expecting to be able to use np.bool_t, in analogy to np.float_t.
>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>>
>>>> Thanks.
>>>>
>>>> Eric
>> The reason it is not there is that AFAIK Cython has no support for an
>> 8-bit boolean type. I.e. you want
>>
>> print arr[3] # should print "True", not 1.
>>
>> and furthermore you want
>>
>> arr[3] = obj # should mean arr[3] = bool(obj)
>>
>> Which brings up the questions
>> 1) I suppose the best thing would be to add support for a "bool" which
>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>> appropriate restrictions otherwise. Thoughts?
>
> How would this relate to the current bint type? bint is an int
> because logical operations in C are ints, and also several operations
> in the Python/C API (and elsewhere) return true/false values as ints.

Well, bint is an int that has different semantics for conversion to/from
Python, and that seems to be needed here as well (e.g. a "bchar", which I
think is essentially C99 _Bool).

What happens if you do "cdef bint value = 4"? Is bool(4) called resulting
in 1 being placed in value? I think that's the wanted behaviour here...

Using bint directly seems to be right out though as one must access the
data using an 8-bit integer type. (Well, I could hard-code buffers to cast
back and forth between "bint" and "char", after the pointer dereference,
but that seems very unclean compared to introducing a new type).

Perhaps we should have "char bint", "short bint", "unsigned long long
bint" and so on :-) Really, the need for a seperate type for bools is a
strict Cython feature, because what it affects is Python object
conversion, so it doesn't hurt IMO that these are not in C. And size and
booleanness are orthogonal features. (But, this is a puristic approach
only, and in practice "bchar" would suffice.)

Dag Sverre


From jonovik at gmail.com  Sun Jun  7 04:06:38 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Sun, 7 Jun 2009 02:06:38 +0000 (UTC)
Subject: [Cython] running doctests in pyx files
References: <78f7ab620906040238va87122fna1cfc7a448da1ead@mail.gmail.com>
	<4A27A01A.70601@student.matnat.uio.no>
	<4A27A081.8020807@student.matnat.uio.no>
	<78f7ab620906040837k67e1fb82ncb32e43de83ea93@mail.gmail.com>
Message-ID: <loom.20090607T020207-45@post.gmane.org>

S?bastien Barth?lemy <barthelemy at ...> writes:

> >> This is a problem with doctest, which uses inspect.is_function to check
> >> whether something is a function, which fails for Cython functions (which
> >> instead answer to inspect.is_builtin). It would be nice if someone would
> >> take this to the doctest list with a patch at some point...
[...]
> http://wiki.cython.org/
FAQ#Doctestskipstestsfromcythonfunctions.2Cwhy.3FIsthereaworkaround.3F

Thanks for this, it was very useful. I've tweaked it into a reusable module 
allowing this:

python -c "import cydoctest, mymod; cydoctest.testmod(mymod)"

http://wiki.cython.org/FAQ#HowcanIrundoctestsinCythoncode.28pyxfiles.29.3F



From barthelemy at crans.org  Sun Jun  7 06:54:28 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Sun, 7 Jun 2009 06:54:28 +0200
Subject: [Cython] error(pos, "'%s.pxd' not found" % module_name)
Message-ID: <78f7ab620906062154g5e964644o5164e834bc935b3f@mail.gmail.com>

Hello all,

When I run "cython arboris/core/test.pyx" cython fails with the
attached msg.txt.

Apparently, the problem comes the following line (#8 in test.pyx)

from misc_c cimport asym, sin, cos

If I replace "cimport" by "import" it works fine.

However, there might be a bug in cython because msg.txt is sooo
unhelpful in the bad case.

Regards
-------------- next part --------------
Traceback (most recent call last):
  File "/usr/bin/cython", line 8, in <module>
    main(command_line = 1)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 733, in main
    result = compile(sources, options)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 710, in compile
    return compile_multiple(source, options)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 680, in compile_multiple
    result = run_pipeline(source, options)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 542, in run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 197, in run_pipeline
    data = phase(data)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 638, in __call__
    return super(AnalyseDeclarationsTransform, self).__call__(root)        
  File "Visitor.py", line 264, in Cython.Compiler.Visitor.CythonTransform.__call__ (/home/seb/Telechargements/Cython-0.11.2/Cython/Compiler/Visitor.c:4528)
  File "Visitor.py", line 250, in Cython.Compiler.Visitor.VisitorTransform.__call__ (/home/seb/Telechargements/Cython-0.11.2/Cython/Compiler/Visitor.c:4289)
  File "Visitor.py", line 45, in Cython.Compiler.Visitor.BasicVisitor.visit (/home/seb/Telechargements/Cython-0.11.2/Cython/Compiler/Visitor.c:1328)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 646, in visit_ModuleNode
    node.analyse_declarations(self.env_stack[-1])
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/ModuleNode.py", line 60, in analyse_declarations
    self.body.analyse_declarations(env)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py", line 358, in analyse_declarations
    stat.analyse_declarations(env)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py", line 4739, in analyse_declarations
    submodule_scope = env.context.find_module(name, relative_to = module_scope, pos = self.pos)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Main.py", line 260, in find_module
    error(pos, "'%s.pxd' not found" % module_name)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Errors.py", line 132, in error
    report_error(err)
  File "/home/seb/.local/lib/python2.6/site-packages/Cython/Compiler/Errors.py", line 121, in report_error
    line = "%s\n" % err
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 170: ordinal not in range(128)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.pyx
Type: application/octet-stream
Size: 2171 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: misc_c.pxd
Type: application/octet-stream
Size: 148 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0005.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test.pxd
Type: application/octet-stream
Size: 114 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0006.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: misc_c.pyx
Type: application/octet-stream
Size: 1074 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090607/f2204b2a/attachment-0007.obj 

From dagss at student.matnat.uio.no  Mon Jun  8 11:39:33 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 08 Jun 2009 11:39:33 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A159A96.9070907@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
Message-ID: <4A2CDC55.8000600@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> A decision is needed here within a few weeks; it would be great if any 
> core developers who don't care could respond to vote 0.

*bump*

Unless I hear something, I'm going to assume that people trust my 
opinion here and start implementing the solution below (it's kind of a 
blocker for Kurt's GSoC). Speak now or forever hold your peace...

Dag Sverre



> 
> The decision was made to use [] for templates. To reiterate the main 
> arguments:
> 
>   - It leaves the way open for exporting pre-instantiated templates to 
> Python in a /very/ natural fashion.
> 
>   - Templates are a bit like a collection of types anyway.
> 
> 
> However this creates some syntax issues:
> 
> A) Currently
> 
>    cdef object[int] arr = ...
> 
> can be used for PEP 3118 access on a generic object. While not a 
> technical problem (object will never become a template), it looks a bit 
> odd when introducing with a template feature.
> 
> B) How would a templated buffer type look like? MyType[int][float]?
> 
> Proposal:
> 
> The object[int] syntax is removed (are anyone using it?). This is 
> basically done by making any cdef class with the 
> __cythonbufferdefaults__ magic attribute become a "buffer template". 
> Such cdef classes cannot be templates. Any other class cannot be used as 
> a buffer-supporting object.
> 
> This removes functionality (much numeric code could care less about 
> whether the underlying object is an ndarray). As a substitute, the main 
> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted, 
> which is to use
> 
> cdef int[:] arr = ...
> 
> for generic access, without any object access (this means that for now 
> only indexing and getting the shape is allowed at all.)
> 
> I've been wondering a lot about whether this is the right way, and it 
> still seems so, even if it means that there will be (and likely continue 
> to be) two different ways of accessing a buffer; one through an explicit 
> "buffer type" and one through an "automatic template".
> 


-- 
Dag Sverre

From kwmsmith at gmail.com  Mon Jun  8 17:12:42 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 8 Jun 2009 10:12:42 -0500
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2CDC55.8000600@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
Message-ID: <c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>

On Mon, Jun 8, 2009 at 4:39 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> A decision is needed here within a few weeks; it would be great if any
>> core developers who don't care could respond to vote 0.
>
> *bump*
>
> Unless I hear something, I'm going to assume that people trust my
> opinion here and start implementing the solution below (it's kind of a
> blocker for Kurt's GSoC). Speak now or forever hold your peace...

The new syntax is fine with me, but you already knew that, and I'd be
presuming a lot to say I'm a core developer ;-)  I've some clarifying
questions below since my understanding of templates is fairly basic.

>>
>> The decision was made to use [] for templates. To reiterate the main
>> arguments:
>>
>> ? - It leaves the way open for exporting pre-instantiated templates to
>> Python in a /very/ natural fashion.
>>
>> ? - Templates are a bit like a collection of types anyway.
>>
>>
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>> ? ?cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?). This is
>> basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be used as
>> a buffer-supporting object.

'removed': so no one can use

cdef object[int] foo = ...

anymore?  Or it has different semantics (must be a 'buffer template'
cdef class)?  I presume all the current syntax for buffer indexing
will still be around for backwards compatibility.

>>
>> This removes functionality (much numeric code could care less about
>> whether the underlying object is an ndarray). As a substitute, the main
>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>> which is to use

What functionality is removed exactly?  The fast indexing with cdef
ints, and all buffer access is now through the object's __getitem__
and __setitem__ only?

>>
>> cdef int[:] arr = ...
>>
>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
>>
>> I've been wondering a lot about whether this is the right way, and it
>> still seems so, even if it means that there will be (and likely continue
>> to be) two different ways of accessing a buffer; one through an explicit
>> "buffer type" and one through an "automatic template".

Just to be clear: the "buffer type" is the cdef int[:], and the
'automatic template' is the cdef MyObj[int].  Both will be indexable,
with the first using default 'fast' indexing on the buffer itself, and
the second indexing through the MyObj.__getitem__.  Right?

Kurt

From ellisonbg.net at gmail.com  Mon Jun  8 20:05:33 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 8 Jun 2009 11:05:33 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A159A96.9070907@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
Message-ID: <6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>

Is there a reason the C++ <> syntax for templates was rejected?

Brian

On Thu, May 21, 2009 at 11:16 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> A decision is needed here within a few weeks; it would be great if any
> core developers who don't care could respond to vote 0.
>
> The decision was made to use [] for templates. To reiterate the main
> arguments:
>
> ?- It leaves the way open for exporting pre-instantiated templates to
> Python in a /very/ natural fashion.
>
> ?- Templates are a bit like a collection of types anyway.
>
>
> However this creates some syntax issues:
>
> A) Currently
>
> ? cdef object[int] arr = ...
>
> can be used for PEP 3118 access on a generic object. While not a
> technical problem (object will never become a template), it looks a bit
> odd when introducing with a template feature.
>
> B) How would a templated buffer type look like? MyType[int][float]?
>
> Proposal:
>
> The object[int] syntax is removed (are anyone using it?). This is
> basically done by making any cdef class with the
> __cythonbufferdefaults__ magic attribute become a "buffer template".
> Such cdef classes cannot be templates. Any other class cannot be used as
> a buffer-supporting object.
>
> This removes functionality (much numeric code could care less about
> whether the underlying object is an ndarray). As a substitute, the main
> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
> which is to use
>
> cdef int[:] arr = ...
>
> for generic access, without any object access (this means that for now
> only indexing and getting the shape is allowed at all.)
>
> I've been wondering a lot about whether this is the right way, and it
> still seems so, even if it means that there will be (and likely continue
> to be) two different ways of accessing a buffer; one through an explicit
> "buffer type" and one through an "automatic template".
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Mon Jun  8 20:35:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 8 Jun 2009 20:35:06 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>
References: <4A159A96.9070907@student.matnat.uio.no>
	<6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>
Message-ID: <82d1b76d4a698d9d26575117da01f1b8.squirrel@webmail.uio.no>

> Is there a reason the C++ <> syntax for templates was rejected?
>
> Brian

I can't seem to find the thread now, so this is from memory. There's the
points already mentioned (quoted below); I consider it extremely valuable
myself that templates written in Cython are potentially exportable to
Python using the same syntax.

Also it can be parsed more cleanly. E.g.

a<b>(c)

Is that two comparisons or calling the constructor of a template? No way
to know until you check the type of a and c, meaning the syntax is
ambiguous until you check that. (In Python this kind of thing is a
showstopper for a new syntax; Guido argues that a simple parser means a
language that's also simple to parse by humans).

(Even C++ struggles with this kind of thing; requiring you to write
"a<b<c> >", i.e. the extra, ugly space to avoid writing a right-shift
operator.)

In Cython we're not so picky about things like this and there are ways of
resolving ambiguous syntax later in the stage; but it does mean that if
Python gets generic types (which isn't entirely impossible), <> is out of
the question, while we could hope for Python using [] (Guido already used
it in mock demonstration syntax for the purpose).

> On Thu, May 21, 2009 at 11:16 AM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> A decision is needed here within a few weeks; it would be great if any
>> core developers who don't care could respond to vote 0.
>>
>> The decision was made to use [] for templates. To reiterate the main
>> arguments:
>>
>> ?- It leaves the way open for exporting pre-instantiated templates to
>> Python in a /very/ natural fashion.
>>
>> ?- Templates are a bit like a collection of types anyway.
>>
>>
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>> ? cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?). This is
>> basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be used as
>> a buffer-supporting object.
>>
>> This removes functionality (much numeric code could care less about
>> whether the underlying object is an ndarray). As a substitute, the main
>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>> which is to use
>>
>> cdef int[:] arr = ...
>>
>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
>>
>> I've been wondering a lot about whether this is the right way, and it
>> still seems so, even if it means that there will be (and likely continue
>> to be) two different ways of accessing a buffer; one through an explicit
>> "buffer type" and one through an "automatic template".
>>
>> --
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



From ellisonbg.net at gmail.com  Mon Jun  8 20:39:26 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 8 Jun 2009 11:39:26 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <82d1b76d4a698d9d26575117da01f1b8.squirrel@webmail.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<6ce0ac130906081105w75a3f4ecnae00057cacdaea2c@mail.gmail.com>
	<82d1b76d4a698d9d26575117da01f1b8.squirrel@webmail.uio.no>
Message-ID: <6ce0ac130906081139w7dffc38cxbc00403d3376aaf4@mail.gmail.com>

> I can't seem to find the thread now, so this is from memory. There's the
> points already mentioned (quoted below); I consider it extremely valuable
> myself that templates written in Cython are potentially exportable to
> Python using the same syntax.
>
> Also it can be parsed more cleanly. E.g.
>
> a<b>(c)
>
> Is that two comparisons or calling the constructor of a template? No way
> to know until you check the type of a and c, meaning the syntax is
> ambiguous until you check that. (In Python this kind of thing is a
> showstopper for a new syntax; Guido argues that a simple parser means a
> language that's also simple to parse by humans).

Yes, this makes sense.

> (Even C++ struggles with this kind of thing; requiring you to write
> "a<b<c> >", i.e. the extra, ugly space to avoid writing a right-shift
> operator.)

I agree that the C++ syntax is not necessarily the prettiest as well.

> In Cython we're not so picky about things like this and there are ways of
> resolving ambiguous syntax later in the stage; but it does mean that if
> Python gets generic types (which isn't entirely impossible), <> is out of
> the question, while we could hope for Python using [] (Guido already used
> it in mock demonstration syntax for the purpose).

Ahh, then this makes total sense.

Cheers,

Brian

>> On Thu, May 21, 2009 at 11:16 AM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> A decision is needed here within a few weeks; it would be great if any
>>> core developers who don't care could respond to vote 0.
>>>
>>> The decision was made to use [] for templates. To reiterate the main
>>> arguments:
>>>
>>> ?- It leaves the way open for exporting pre-instantiated templates to
>>> Python in a /very/ natural fashion.
>>>
>>> ?- Templates are a bit like a collection of types anyway.
>>>
>>>
>>> However this creates some syntax issues:
>>>
>>> A) Currently
>>>
>>> ? cdef object[int] arr = ...
>>>
>>> can be used for PEP 3118 access on a generic object. While not a
>>> technical problem (object will never become a template), it looks a bit
>>> odd when introducing with a template feature.
>>>
>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>
>>> Proposal:
>>>
>>> The object[int] syntax is removed (are anyone using it?). This is
>>> basically done by making any cdef class with the
>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>> Such cdef classes cannot be templates. Any other class cannot be used as
>>> a buffer-supporting object.
>>>
>>> This removes functionality (much numeric code could care less about
>>> whether the underlying object is an ndarray). As a substitute, the main
>>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>>> which is to use
>>>
>>> cdef int[:] arr = ...
>>>
>>> for generic access, without any object access (this means that for now
>>> only indexing and getting the shape is allowed at all.)
>>>
>>> I've been wondering a lot about whether this is the right way, and it
>>> still seems so, even if it means that there will be (and likely continue
>>> to be) two different ways of accessing a buffer; one through an explicit
>>> "buffer type" and one through an "automatic template".
>>>
>>> --
>>> Dag Sverre
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dalcinl at gmail.com  Mon Jun  8 20:41:36 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 8 Jun 2009 15:41:36 -0300
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2CDC55.8000600@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
Message-ID: <e7ba66e40906081141g1f03c334ua0e84fe9c7475ad1@mail.gmail.com>

On Mon, Jun 8, 2009 at 6:39 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> A decision is needed here within a few weeks; it would be great if any
>> core developers who don't care could respond to vote 0.
>
> *bump*
>
> Unless I hear something, I'm going to assume that people trust my
> opinion here and start implementing the solution below (it's kind of a
> blocker for Kurt's GSoC). Speak now or forever hold your peace...
>
> Dag Sverre
>

Perhaps you do not remember, but I've expressed my +1 for templates using  []

I'll just point that I like "MyType[int,float]" more than
"MyType[int][float]" ...

>
>
>>
>> The decision was made to use [] for templates. To reiterate the main
>> arguments:
>>
>> ? - It leaves the way open for exporting pre-instantiated templates to
>> Python in a /very/ natural fashion.
>>
>> ? - Templates are a bit like a collection of types anyway.
>>
>>
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>> ? ?cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?). This is
>> basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be used as
>> a buffer-supporting object.
>>
>> This removes functionality (much numeric code could care less about
>> whether the underlying object is an ndarray). As a substitute, the main
>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>> which is to use
>>
>> cdef int[:] arr = ...
>>
>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
>>
>> I've been wondering a lot about whether this is the right way, and it
>> still seems so, even if it means that there will be (and likely continue
>> to be) two different ways of accessing a buffer; one through an explicit
>> "buffer type" and one through an "automatic template".
>>
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Mon Jun  8 20:53:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 8 Jun 2009 20:53:51 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
	<c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>
Message-ID: <16477e0839ee8f2b42848ef748378222.squirrel@webmail.uio.no>

Kurt Smith wrote:
> On Mon, Jun 8, 2009 at 4:39 AM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Dag Sverre Seljebotn wrote:
> The new syntax is fine with me, but you already knew that, and I'd be
> presuming a lot to say I'm a core developer ;-)  I've some clarifying
> questions below since my understanding of templates is fairly basic.

To be clear, I definitely want input from *everybody* (users too!) I
didn't put it that well.

>>> However this creates some syntax issues:
>>>
>>> A) Currently
>>>
>>> ? ?cdef object[int] arr = ...
>>>
>>> can be used for PEP 3118 access on a generic object. While not a
>>> technical problem (object will never become a template), it looks a bit
>>> odd when introducing with a template feature.
>>>
>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>
>>> Proposal:
>>>
>>> The object[int] syntax is removed (are anyone using it?). This is
>>> basically done by making any cdef class with the
>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>> Such cdef classes cannot be templates. Any other class cannot be used
>>> as
>>> a buffer-supporting object.
>
> 'removed': so no one can use
>
> cdef object[int] foo = ...
>
> anymore?  Or it has different semantics (must be a 'buffer template'
> cdef class)?  I presume all the current syntax for buffer indexing
> will still be around for backwards compatibility.

Removed, so noone can use

cdef object[int] foo

any more. To get that exact behaviour, one has to do

cdef object foo
cdef int[:] foo_buf = foo

I think this is fair enough -- I'd like to see a real usecase for this
syntax if the new one is introduced?

ndarray[int] stays though.

>>> This removes functionality (much numeric code could care less about
>>> whether the underlying object is an ndarray). As a substitute, the main
>>> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
>>> which is to use
>
> What functionality is removed exactly?  The fast indexing with cdef
> ints, and all buffer access is now through the object's __getitem__
> and __setitem__ only?

Just removing object[type] by itself, without compensating, removes the
possibility of writing functions which work across multiple buffers
exporters. E.g. a single function which can handle both Image objects and
ndarray objects through the buffer API.


>
>>>
>>> cdef int[:] arr = ...
>>>
>>> for generic access, without any object access (this means that for now
>>> only indexing and getting the shape is allowed at all.)
>>>
>>> I've been wondering a lot about whether this is the right way, and it
>>> still seems so, even if it means that there will be (and likely
>>> continue
>>> to be) two different ways of accessing a buffer; one through an
>>> explicit
>>> "buffer type" and one through an "automatic template".
>
> Just to be clear: the "buffer type" is the cdef int[:], and the
> 'automatic template' is the cdef MyObj[int].  Both will be indexable,
> with the first using default 'fast' indexing on the buffer itself, and
> the second indexing through the MyObj.__getitem__.  Right?

No. I was clumsy here. If MyObj defines __cythonbufferdefaults__, Cython
treats it like ndarray is treated today. Otherwise, and when templates are
implemented, the "int" is used in whatever way the MyObj template uses it.

But what I mean is that in both cases, from the user's perspective one can
look at MyObj as a template of sorts.

However for writing more generic buffer code (where no knowledge of the
type of the object is sometimes not needed at all) a different kind of
concept is needed. object[int] used to provide this but I don't think many
used it.

Dag Sverre


From dagss at student.matnat.uio.no  Mon Jun  8 20:59:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 8 Jun 2009 20:59:12 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <e7ba66e40906081141g1f03c334ua0e84fe9c7475ad1@mail.gmail.com>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
	<e7ba66e40906081141g1f03c334ua0e84fe9c7475ad1@mail.gmail.com>
Message-ID: <d0c3953c4cf4539a5346a8f1d384882c.squirrel@webmail.uio.no>

Lisandro wrote:
> Perhaps you do not remember, but I've expressed my +1 for templates using
> []
>
> I'll just point that I like "MyType[int,float]" more than
> "MyType[int][float]" ...
>

I think I write too long emails :-)

Anyway, this wasn't about the template syntax (which is pretty much
settled I believe; Brian Granger just wanted to know why the decision went
that way), but about changing the PEP 3118 syntax, specifically, removing
the possibility of doing

cdef object[int] a

(is anybody even using it for real?) and replace it with

cdef int[:] a

with slightly different semantics. ndarray[int] stays as today.

Dag Sverre


From kwmsmith at gmail.com  Mon Jun  8 22:20:54 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Mon, 8 Jun 2009 15:20:54 -0500
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <16477e0839ee8f2b42848ef748378222.squirrel@webmail.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<4A2CDC55.8000600@student.matnat.uio.no>
	<c266a7930906080812t57bd86e4x792f63cc5c57f4f3@mail.gmail.com>
	<16477e0839ee8f2b42848ef748378222.squirrel@webmail.uio.no>
Message-ID: <c266a7930906081320s4b5c84d4t5751a25d9fde27bd@mail.gmail.com>

On Mon, Jun 8, 2009 at 1:53 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Kurt Smith wrote:
>> On Mon, Jun 8, 2009 at 4:39 AM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> Dag Sverre Seljebotn wrote:
>> The new syntax is fine with me, but you already knew that, and I'd be
>> presuming a lot to say I'm a core developer ;-) ?I've some clarifying
>> questions below since my understanding of templates is fairly basic.
>
> To be clear, I definitely want input from *everybody* (users too!) I
> didn't put it that well.
>
>>>> However this creates some syntax issues:
>>>>
>>>> A) Currently
>>>>
>>>> ? ?cdef object[int] arr = ...
>>>>
>>>> can be used for PEP 3118 access on a generic object. While not a
>>>> technical problem (object will never become a template), it looks a bit
>>>> odd when introducing with a template feature.
>>>>
>>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>>
>>>> Proposal:
>>>>
>>>> The object[int] syntax is removed (are anyone using it?). This is
>>>> basically done by making any cdef class with the
>>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>>> Such cdef classes cannot be templates. Any other class cannot be used
>>>> as
>>>> a buffer-supporting object.
>>
>> 'removed': so no one can use
>>
>> cdef object[int] foo = ...
>>
>> anymore? ?Or it has different semantics (must be a 'buffer template'
>> cdef class)? ?I presume all the current syntax for buffer indexing
>> will still be around for backwards compatibility.
>
> Removed, so noone can use
>
> cdef object[int] foo
>
> any more. To get that exact behaviour, one has to do
>
> cdef object foo
> cdef int[:] foo_buf = foo

I see now ;-)

>
> I think this is fair enough -- I'd like to see a real usecase for this
> syntax if the new one is introduced?
>
> ndarray[int] stays though.

A possible problem I envision with keeping 'ndarray[int]' around
exactly as is (although I'm exposing my ignorance of the proposed
template usage): if using bracket-templates becomes fairly common in
Cython, won't people then start to see 'ndarray[int]' as another
template expression and *not* a (magic) buffer statement?  Or is it
obvious since 'ndarray' is special?  If someone created their own cdef
class with __cythonbufferdefaults__, wouldn't it mimick the template
bracket expression/statement and be quite confusing?

>>
>> What functionality is removed exactly? ?The fast indexing with cdef
>> ints, and all buffer access is now through the object's __getitem__
>> and __setitem__ only?
>
> Just removing object[type] by itself, without compensating, removes the
> possibility of writing functions which work across multiple buffers
> exporters. E.g. a single function which can handle both Image objects and
> ndarray objects through the buffer API.
>
>
[snip]
>>
>> Just to be clear: the "buffer type" is the cdef int[:], and the
>> 'automatic template' is the cdef MyObj[int]. ?Both will be indexable,
>> with the first using default 'fast' indexing on the buffer itself, and
>> the second indexing through the MyObj.__getitem__. ?Right?
>
> No. I was clumsy here. If MyObj defines __cythonbufferdefaults__, Cython
> treats it like ndarray is treated today. Otherwise, and when templates are
> implemented, the "int" is used in whatever way the MyObj template uses it.
>
> But what I mean is that in both cases, from the user's perspective one can
> look at MyObj as a template of sorts.

I see.  Perhaps my above comment isn't such a concern, then.

Kurt

From robertwb at math.washington.edu  Tue Jun  9 09:06:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 00:06:50 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A159A96.9070907@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
Message-ID: <717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>

On May 21, 2009, at 11:16 AM, Dag Sverre Seljebotn wrote:

> A decision is needed here within a few weeks; it would be great if any
> core developers who don't care could respond to vote 0.

Sorry, I flagged this as something I wanted to reply to after reading  
your wiki pages on the topic, and then it scrolled off the top of my  
cython folder...

> However this creates some syntax issues:
>
> A) Currently
>
>    cdef object[int] arr = ...
>
> can be used for PEP 3118 access on a generic object. While not a
> technical problem (object will never become a template), it looks a  
> bit
> odd when introducing with a template feature.
>
> B) How would a templated buffer type look like? MyType[int][float]?
>
> Proposal:
>
> The object[int] syntax is removed (are anyone using it?).

It's a good idea to have a non-conflicting syntax like int[:], but I  
don't think we should be treating ndarray[int] and object[int] any  
differently, or removing backwards compatibility.

> This is basically done by making any cdef class with the
> __cythonbufferdefaults__ magic attribute become a "buffer template".
> Such cdef classes cannot be templates. Any other class cannot be  
> used as
> a buffer-supporting object.

Are you suggesting we require __cythonbufferdefaults__ for all buffer  
classes? __cythonbufferdefaults__ doesn't make sense for, e.g.,  
ndarrays.

> This removes functionality (much numeric code could care less about
> whether the underlying object is an ndarray). As a substitute, the  
> main
> idea in http://wiki.cython.org/enhancements/buffersyntax is accepted,
> which is to use
>
> cdef int[:] arr = ...
>
> for generic access, without any object access (this means that for now
> only indexing and getting the shape is allowed at all.)

I like the int[:] syntax, but I'm -1 on the raw buffer no-object- 
access idea. Having to write

cdef object foo = ...
cdef int[:] foo_buf = foo

seems clunky to me--I really like that in the current system one  
doesn't have to have to separate variables to access a buffer as a  
fast array or a Python object--this fits in well with the whole  
Cython philosophy. (I do think we should print warnings for people  
who write A[i][j] (which is slower from Python too, and should be  
obvious if cython -a is run).)

I would rather see int[:] be a type qualifier (like long,  
unsigned, ...) so one can write

cdef int[:] object foo

- Robert

From dagss at student.matnat.uio.no  Tue Jun  9 09:30:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 09:30:55 +0200
Subject: [Cython] Sundials with Cython? Integrating/solving
 ordinary	differential equation models with CVODE
In-Reply-To: <loom.20090605T132731-705@post.gmane.org>
References: <cd166c030906030401p2cf72047q80ea8f73b93d8293@mail.gmail.com>	<cd166c030906030409i54dfe4e4id07fbd4e49a79a66@mail.gmail.com>	<4A266871.9070209@student.matnat.uio.no>
	<loom.20090605T132731-705@post.gmane.org>
Message-ID: <4A2E0FAF.7050109@student.matnat.uio.no>

Jon Olav Vik wrote:
> If I may make a wish, I would like Cython to find more declarations 
> automatically. For instance, having
> 
> cdef extern * from "cvode/cvode.h"
> 
> or, if that is impractical,
> 
> cdef extern from "cvode/cvode.h":
>     CVodeCreate, CVRhsFn, CVodeMalloc, CVodeFree, CVode, CV_ADAMS, CV_BDF

Yes, we all would. If you find a decent CS student at your institution 
who would like to do this as his or her master thesis then drop us a 
note :-)

I think there's a script someone posted on the mailing list to try to 
autogenerate pxds, though I don't remember enough to find it -- anyone?

> I haven't benchmarked anything yet, but the relative ease of adapting the C 
> tutorial for CVODE [3] is uplifting. As a last question, what is the cleanest 
> way to transfer arrays of double between realtype* and Numpy vectors (one-
> dimensional arrays)?

OK here it is. Note that this sets up the ndarray to point to the *same* 
memory. Modifications to the resulting NumPy array will happen directly 
in the NVector!

Case A: If you are 100% sure that you do not call N_VDestroy_Serial 
before all references to the ndarray created are gone, you can simply do 
this:

cdef extern from "numpy/arrayobject.h":
     # These are simply missing from numpy.pxd due to laziness;
     # so declare manually for now
     void import_array()
     object PyArray_SimpleNewFromData(int nd,
                                      np.npy_intp* dims,
                                      int typenum,
                                      void* data)



import_array() # Needed because we call the raw NumPy C API below!

def some_func():
     cdef np.npy_intp* shape = [somehow_extract_length_of(some_nvector)]
     # Create ndarray. Assume realtype is double.
     assert sizeof(realtype) == sizeof(double)
     yourarray = PyArray_SimpleNewFromData(
       1, shape, np.NPY_DOUBLE,
       <void*>self.nvector.whatever_to_get_buffer_pointer)
     ...

But this is dangerous as the resulting yourarray will address 
nonallocated memory if you free the underlying NVector.

Case B: Proper memory handling:

First, create a C header file (workaround.h), because Cython cannot 
handle expressions of the form "foo(x) = y" so we need to "reformulate" 
the NumPy C API a bit:

#include <numpy/arrayobject.h>
#define PyArray_Set_BASE(arr, obj) PyArray_BASE(arr) = obj

Then, this Cython code should do the job. Note that we need to create a 
class because Python needs an object to associate with the memory, so 
that the memory gets freed at the right time.

cimport numpy as np

cdef extern from "workaround.h":
     void PyArray_Set_BASE(np.ndarray arr, object obj)

cdef extern from "numpy/arrayobject.h":
     # These are simply missing from numpy.pxd due to laziness;
     # so declare manually for now
     void import_array()
     object PyArray_SimpleNewFromData(int nd,
                                      np.npy_intp* dims,
                                      int typenum,
                                      void* data)



import_array() # Needed because we call the raw NumPy C API below!

cdef class CythonNVector:
     cdef N_Vector nvector
     def __init__(self, length):
         self.nvector = N_VNew_Serial(length)

     def __dealloc__(self):
         # Called when the object is being destroyed (normally there's
         # no need for this in Python, but here it is, as we need to
         # free C memory explicitly).
         #
         # Note: The "Python parts" of the object might already
         # be gone in this method -- only touch C attributes.
         N_VDestroy_Serial(self.nvector)

     cpdef toarray(self):
         cdef np.npy_intp* shape = \
[somehow_extract_length_of(self.nvector)]
         # Create ndarray. Assume realtype is double.
         assert sizeof(realtype) == sizeof(double)
         cdef np.ndarray result = PyArray_SimpleNewFromData(
   1, shape, np.NPY_DOUBLE,
   <void*>self.nvector.whatever_to_get_buffer_pointer)
         # Make the ndarray keep a reference to this object
         PyArray_Set_BASE(result, self)
         return result


Just modify this general scheme for your usecase (i.e. you can set 
nvector manually rather than allocate it the constructor, etc.)



-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun  9 09:46:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 09:46:48 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
References: <4A159A96.9070907@student.matnat.uio.no>
	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
Message-ID: <4A2E1368.4040701@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 21, 2009, at 11:16 AM, Dag Sverre Seljebotn wrote:
>> However this creates some syntax issues:
>>
>> A) Currently
>>
>>    cdef object[int] arr = ...
>>
>> can be used for PEP 3118 access on a generic object. While not a
>> technical problem (object will never become a template), it looks a  
>> bit
>> odd when introducing with a template feature.
>>
>> B) How would a templated buffer type look like? MyType[int][float]?
>>
>> Proposal:
>>
>> The object[int] syntax is removed (are anyone using it?).
> 
> It's a good idea to have a non-conflicting syntax like int[:], but I  
> don't think we should be treating ndarray[int] and object[int] any  
> differently, or removing backwards compatibility.

The idea here is that given templates the object hierarchy will look 
like this:

object
|
+-- SomeTemplate[int]
|
L-- ndarray[int]

My idea is then that it will look nicer if object doesn't default to 
being a "template" as well.

We could introduce a new decorator to make this point clear (at least, 
the following code will make clear how I'd prefer to think of it):

@cython.buffer.auto_buffer_template(
   defaults={"ndim":2, "mode":"strided"})
cdef class MyImage(object):
     ...

with the understanding that Cython transforms this into
cdef class MyImage[type, ndim=2, mode="strided"](object):
     ...

with an efficient __getitem__/__setitem__ implementation.

Making this behaviour the "default" template behaviour for a type, 
unless a template is explicitly requested, just seems more confusing.

I think virtually noone are using the object[...] at the moment, and 
better remove backwards compatability sooner than later.

But, it's certainly a point where pragmatism might go before purism.

>> This is basically done by making any cdef class with the
>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>> Such cdef classes cannot be templates. Any other class cannot be  
>> used as
>> a buffer-supporting object.
> 
> Are you suggesting we require __cythonbufferdefaults__ for all buffer  
> classes? __cythonbufferdefaults__ doesn't make sense for, e.g.,  
> ndarrays.

? They are used in numpy.pxd already (and if removed, mode would default 
to "full", causing an extra if-test per dimension per access...).

But I now like the above decorator -- but this would be the backwards 
compatability with most classes. Almost all serious uses must have set

cdef __cythonbufferdefaults__ = {"mode": "strided"}

anyway.

>> for generic access, without any object access (this means that for now
>> only indexing and getting the shape is allowed at all.)
> 
> I like the int[:] syntax, but I'm -1 on the raw buffer no-object- 
> access idea. Having to write
> 
> cdef object foo = ...
> cdef int[:] foo_buf = foo
> 
> seems clunky to me--I really like that in the current system one  
> doesn't have to have to separate variables to access a buffer as a  
> fast array or a Python object--this fits in well with the whole  
> Cython philosophy. (I do think we should print warnings for people  
> who write A[i][j] (which is slower from Python too, and should be  
> obvious if cython -a is run).)
> 
> I would rather see int[:] be a type qualifier (like long,  
> unsigned, ...) so one can write
> 
> cdef int[:] object foo

Hmm. I'm not against that -- would you then be ok with

cdef int[:] foo

not giving any object access at all? (The thing is I also want to use it 
to access arrays stored in C pointers etc.; + safe passing around of 
buffers in nogil mode!)

I guess I see it as getting hold of another interface -- I see it this 
way: Objects supporting PEP 3118 kind of "inherits from" int[:] through 
multiple inheritance.

(If doing something like this I'd rather like to adopt Guido's mock 
syntax for combining interfaces -- e.g. if a class implements both 
"Iterable" and "File" interfaces, one could do

cdef Iterable & File foo

Of course, Cython currently doesn't support multiple inheritance (even 
through Java interfaces), and we might not want to over-prepare for 
that. Meaning "cdef int[:] object foo" could work.)


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun  9 09:55:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 09:55:08 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2E1368.4040701@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
	<4A2E1368.4040701@student.matnat.uio.no>
Message-ID: <4A2E155C.1030701@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> seems clunky to me--I really like that in the current system one  
>> doesn't have to have to separate variables to access a buffer as a  
>> fast array or a Python object--this fits in well with the whole  
>> Cython philosophy. (I do think we should print warnings for people  
>> who write A[i][j] (which is slower from Python too, and should be  
>> obvious if cython -a is run).)
>>
>> I would rather see int[:] be a type qualifier (like long,  
>> unsigned, ...) so one can write
>>
>> cdef int[:] object foo
> 
> Hmm. I'm not against that -- would you then be ok with
> 
> cdef int[:] foo
> 
> not giving any object access at all? (The thing is I also want to use it 
> to access arrays stored in C pointers etc.; + safe passing around of 
> buffers in nogil mode!)

I really should reiterate a problem spot here: Efficient slicing.

cdef object[int] a = np.arange(10)
cdef object[int] b

b = a[5:] # made efficient
print b[0] # prints 5
print (<object>b)[0] # huh, prints 0?

Either that, or

print b.foo() # raises exception "NoneType has no attribute"?

When loosing the underlying object reference from the syntax, this 
problems disappears, as one won't expect "b.foo()" to work in the first 
place. Then

print (<object>b)[0] # prints 5

because b is coerced to a memoryview.


-- 
Dag Sverre

From robertwb at math.washington.edu  Tue Jun  9 10:31:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:31:53 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2E1368.4040701@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>
	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>
	<4A2E1368.4040701@student.matnat.uio.no>
Message-ID: <E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>

On Jun 9, 2009, at 12:46 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 21, 2009, at 11:16 AM, Dag Sverre Seljebotn wrote:
>>> However this creates some syntax issues:
>>>
>>> A) Currently
>>>
>>>    cdef object[int] arr = ...
>>>
>>> can be used for PEP 3118 access on a generic object. While not a
>>> technical problem (object will never become a template), it looks a
>>> bit
>>> odd when introducing with a template feature.
>>>
>>> B) How would a templated buffer type look like? MyType[int][float]?
>>>
>>> Proposal:
>>>
>>> The object[int] syntax is removed (are anyone using it?).
>>
>> It's a good idea to have a non-conflicting syntax like int[:], but I
>> don't think we should be treating ndarray[int] and object[int] any
>> differently, or removing backwards compatibility.
>
> The idea here is that given templates the object hierarchy will look
> like this:
>
> object
> |
> +-- SomeTemplate[int]
> |
> L-- ndarray[int]
>
> My idea is then that it will look nicer if object doesn't default to
> being a "template" as well.
>
> We could introduce a new decorator to make this point clear (at least,
> the following code will make clear how I'd prefer to think of it):
>
> @cython.buffer.auto_buffer_template(
>    defaults={"ndim":2, "mode":"strided"})
> cdef class MyImage(object):
>      ...
>
> with the understanding that Cython transforms this into
> cdef class MyImage[type, ndim=2, mode="strided"](object):
>      ...
>
> with an efficient __getitem__/__setitem__ implementation.
>
> Making this behaviour the "default" template behaviour for a type,
> unless a template is explicitly requested, just seems more confusing.

cdef class MyImage[...](object):

would always be a template, not a buffer.

> I think virtually noone are using the object[...] at the moment, and
> better remove backwards compatability sooner than later.

I see your point, but I think explicitly disallowing object is too  
inconsistent. It's also nice to not have to even cimport numpy to use  
efficient buffer access and be able to handle anything that exports a  
buffer interface.

> But, it's certainly a point where pragmatism might go before purism.
>
>>> This is basically done by making any cdef class with the
>>> __cythonbufferdefaults__ magic attribute become a "buffer template".
>>> Such cdef classes cannot be templates. Any other class cannot be
>>> used as
>>> a buffer-supporting object.
>>
>> Are you suggesting we require __cythonbufferdefaults__ for all buffer
>> classes? __cythonbufferdefaults__ doesn't make sense for, e.g.,
>> ndarrays.
>
> ? They are used in numpy.pxd already (and if removed, mode would  
> default
> to "full", causing an extra if-test per dimension per access...).

Oh, you're right. I was thinking ndim couldn't be set, but there are  
other things that can be.

> But I now like the above decorator -- but this would be the backwards
> compatability with most classes. Almost all serious uses must have set
>
> cdef __cythonbufferdefaults__ = {"mode": "strided"}
>
> anyway.

I'm OK with either the decorator of the __cythonbufferdefaults__.

>
>>> for generic access, without any object access (this means that  
>>> for now
>>> only indexing and getting the shape is allowed at all.)
>>
>> I like the int[:] syntax, but I'm -1 on the raw buffer no-object-
>> access idea. Having to write
>>
>> cdef object foo = ...
>> cdef int[:] foo_buf = foo
>>
>> seems clunky to me--I really like that in the current system one
>> doesn't have to have to separate variables to access a buffer as a
>> fast array or a Python object--this fits in well with the whole
>> Cython philosophy. (I do think we should print warnings for people
>> who write A[i][j] (which is slower from Python too, and should be
>> obvious if cython -a is run).)
>>
>> I would rather see int[:] be a type qualifier (like long,
>> unsigned, ...) so one can write
>>
>> cdef int[:] object foo
>
> Hmm. I'm not against that -- would you then be ok with
>
> cdef int[:] foo
>
> not giving any object access at all? (The thing is I also want to  
> use it
> to access arrays stored in C pointers etc.; + safe passing around of
> buffers in nogil mode!)

I could accept that, I trust that the details could be worked out  
nicely. Would be like a char* from an object, where the user is  
required to keep the object around long enough?

> I guess I see it as getting hold of another interface -- I see it this
> way: Objects supporting PEP 3118 kind of "inherits from" int[:]  
> through
> multiple inheritance.

I see "providing X api/interface" and "inheriting from" as two  
orthogonal concepts (though the two have historically been conflated  
due to statically linked languages such as C++ and Java).

> (If doing something like this I'd rather like to adopt Guido's mock
> syntax for combining interfaces -- e.g. if a class implements both
> "Iterable" and "File" interfaces, one could do
>
> cdef Iterable & File foo
>
> Of course, Cython currently doesn't support multiple inheritance (even
> through Java interfaces), and we might not want to over-prepare for
> that. Meaning "cdef int[:] object foo" could work.)

Agreed. Also, we don't want the syntax to suggest that any two types  
could be paired--there's something very different going on here.

- Robert



From robertwb at math.washington.edu  Tue Jun  9 10:38:58 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:38:58 -0700
Subject: [Cython] support for numpy bool type
In-Reply-To: <a236bdcbdd2d91e6838e52565565d5ac.squirrel@webmail.uio.no>
References: <4A297446.6030309@hawaii.edu> <4A2A15FE.8020201@hawaii.edu>
	<4A2A34CA.8000702@student.matnat.uio.no>
	<E59FC026-6EB0-4F70-92AC-BA04B9B9CB5D@math.washington.edu>
	<a236bdcbdd2d91e6838e52565565d5ac.squirrel@webmail.uio.no>
Message-ID: <88BEB4A6-11CA-4201-BDC9-A1CE4DA6C6CF@math.washington.edu>

On Jun 6, 2009, at 1:22 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
>>
>>> Eric Firing wrote:
>>>> Eric Firing wrote:
>>>>> In writing a cython extension to work with numpy masked arrays, I
>>>>> needed to work with the mask, which is dtype('bool').  Therefore I
>>>>> was expecting to be able to use np.bool_t, in analogy to  
>>>>> np.float_t.
>>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>>> python wrapper.  Can bool_t be added to numpy.pxd?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Eric
>>> The reason it is not there is that AFAIK Cython has no support  
>>> for an
>>> 8-bit boolean type. I.e. you want
>>>
>>> print arr[3] # should print "True", not 1.
>>>
>>> and furthermore you want
>>>
>>> arr[3] = obj # should mean arr[3] = bool(obj)
>>>
>>> Which brings up the questions
>>> 1) I suppose the best thing would be to add support for a "bool"  
>>> which
>>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>>> appropriate restrictions otherwise. Thoughts?
>>
>> How would this relate to the current bint type? bint is an int
>> because logical operations in C are ints, and also several operations
>> in the Python/C API (and elsewhere) return true/false values as ints.
>
> Well, bint is an int that has different semantics for conversion to/ 
> from
> Python,

That is exactly what it is.

> and that seems to be needed here as well (e.g. a "bchar", which I
> think is essentially C99 _Bool).

a C99 _Bool can only have two values, anything non-zero is converted  
to exactly 1.

> What happens if you do "cdef bint value = 4"? Is bool(4) called  
> resulting
> in 1 being placed in value? I think that's the wanted behaviour  
> here...

No, it retains the value 4. (This is one of the reasons I didn't call  
it a bool.)

> Using bint directly seems to be right out though as one must access  
> the
> data using an 8-bit integer type. (Well, I could hard-code buffers  
> to cast
> back and forth between "bint" and "char", after the pointer  
> dereference,
> but that seems very unclean compared to introducing a new type).
>
> Perhaps we should have "char bint", "short bint", "unsigned long long
> bint" and so on :-) Really, the need for a seperate type for bools  
> is a
> strict Cython feature, because what it affects is Python object
> conversion, so it doesn't hurt IMO that these are not in C. And  
> size and
> booleanness are orthogonal features. (But, this is a puristic approach
> only, and in practice "bchar" would suffice.)

Actually, that might work well. When I first introduced it, I saw no  
need to offer various sizes, but if one has arrays of them then it  
becomes important. And just when we thought we were simplifying the  
integer type system... :)

- Robert



From robertwb at math.washington.edu  Tue Jun  9 10:39:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:39:59 -0700
Subject: [Cython] Trivial numpy test in sage notebook fails...
In-Reply-To: <3327124144.3090608@smtp.netcom.no>
References: <3327124144.3090608@smtp.netcom.no>
Message-ID: <79017BF3-B11C-4378-A1C1-33572625AB28@math.washington.edu>

To conclude this thread, Sage ships its own numpy.pxd, which predates  
the buffer interface. We need to go in and remove this (there's a  
ticket in Sage to do this).

On Jun 6, 2009, at 12:09 AM, Dag Sverre Seljebotn wrote:

> afaik this is a problem with the Sage notebook. Try to Cythonize  
> from the command line, and if that works, take it to the Sage list.
>
> Dag Sverre Seljebotn
> -----Original Message-----
> From: "Glenn Tarbox, PhD" <glenn at tarbox.org>
> Date: Saturday, Jun 6, 2009 6:01 am
> Subject: [Cython] Trivial numpy test in sage notebook fails...
> To: Cython-dev at codespeak.netReply-To: cython-dev at codespeak.net
>
> Here's the code which is the first cell of a sage notebook (sage 4.0)
>>
>> %cython
>> import numpy as np
>> cimport numpy as np
>> DTYPE = np.int
>> ctypedef np.int_t DTYPE_t
>>
>> Here's the error:
>>
>> Traceback (most recent call last):    DTYPE = np.int
>>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/ 
>> sage/server/support.py', line 408, in cython_import_all
>>    create_local_c_file=create_local_c_file)
>>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/ 
>> sage/server/support.py', line 385, in cython_import
>>    create_local_c_file=create_local_c_file)
>>  File '/mnt/hdd/sage/sage-4.0/local/lib/python2.5/site-packages/ 
>> sage/misc/cython.py', line 367, in cython
>>    raise RuntimeError, 'Error converting %s to C:\n%s\n%s'% 
>> (filename, log, err)
>> RuntimeError: Error converting /home/tarbox/.sage/sage_notebook/ 
>> worksheets/admin/89/code/sage23.spyx to C:
>>
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>>
>> include 'cdefs.pxi'
>> import numpy as np
>> cimport numpy as np
>> DTYPE = np.int
>> ctypedef np.int_t DTYPE_t        ^
>> ------------------------------------------------------------
>>
>> /home/tarbox/.sage/temp/hq2/12471/spyx/ 
>> _home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx 
>> / 
>> _home_tarbox__sage_sage_notebook_worksheets_admin_89_code_sage23_spyx 
>> _0.pyx:9:9: 'int_t' is not a type identifier
>> I tried with and without the 'import numpy' directive thinking it  
>> may be redundant...
>>
>> This example comes from the 'cython for numpy users' tutorial on  
>> the wiki where it mentions that all numpy types have a  
>> corresponding _t type
>>
>> There is indeed a numpy.int type... so I was wondering if the new  
>> work with python buffers changed how things work.
>>
>> -glenn
>>
>> -- 
>> Glenn H. Tarbox, PhD ||  206-274-6919
>> http://www.tarbox.org
>>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Tue Jun  9 10:41:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:41:32 -0700
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
Message-ID: <3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>

On Jun 4, 2009, at 2:49 PM, Lisandro Dalcin wrote:

> On Thu, Jun 4, 2009 at 1:29 PM, Stefan Behnel <stefan_ml at behnel.de>  
> wrote:
>> Lisandro Dalcin wrote:
>>> On Wed, Jun 3, 2009 at 4:00 AM, Stefan Behnel wrote:
>>>>
>>>> I think we should file a bug report against Py3.1 and even bring  
>>>> this to
>>>> the attention of python-dev before it's too late.
>>>>
>>>
>>> Stefan, I saw your post in python-dev and corresponding issue in the
>>> tracker, but I got confused. Is the solution know and the issue  
>>> going
>>> to be fixed?
>>
>> I didn't find the time to look into this any deeper. I'm not sure  
>> it will
>> get fixed, because a) the bug itself is a lot older than rc1, and  
>> b) the
>> comment sounded more like "the fix might be similar to what we did  
>> here".
>>
>> I think we should still open a new bug. Given that you experienced  
>> the
>> problem, could you do that?
>>
>
> Done, http://bugs.python.org/issue6195, with patch, I've added Stefan
> to the noisy list.

Does the new patch there resolve the issue?

- Robert


From robertwb at math.washington.edu  Tue Jun  9 10:43:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 01:43:26 -0700
Subject: [Cython] C99 comment in __Pyx_PyObject_Append
In-Reply-To: <f125f5e0906031131nebf9677tca009911f4861bf5@mail.gmail.com>
References: <f125f5e0906030007n437dafe1x7892b027a39765d5@mail.gmail.com>
	<72E15573-DFD2-4D1E-ACF4-581470359819@math.washington.edu>
	<f125f5e0906031131nebf9677tca009911f4861bf5@mail.gmail.com>
Message-ID: <43627760-11C2-4014-87CE-07101AE45538@math.washington.edu>

On Jun 3, 2009, at 11:31 AM, Andrew Collette wrote:

> Hi,
>
>> Strange--it looks like this was fixed months ago:
>>
>> http://hg.cython.org/cython-devel/diff/8b6d42d16c99/Cython/Compiler/
>> ExprNodes.py
>
> It looks like for some reason Cython wasn't being upgraded correctly;
> after sudo python setup.py install, cython --version still reported
> 0.10.3.  I had to manually delete the Cython directory in
> site-packages and reinstall to get it to report 0.11.2.  But it works
> now!

I've seen this a couple of times before--not sure what the issue is.

>> Maybe you need to touch the .pyx to re-generate the sources. It is
>> gratifying to hear that is the only issue--should we be running our
>> tests with this option? (Or is it gcc only?)
>
> I think all major compilers have a strict-ansi mode, although I'm sure
> the switch is different for VC++.  I currently only use it with gcc on
> OS X for compatibility with another package that looks for
> __STRICT_ANSI__.  However, if Cython is intended to generate pure C89
> code it might be worth adding this as a test.  At the moment it seems
> this is already the case; my large-ish (9k line) Cython project now
> compiles fine with -ansi.

Cool. Yeah, I think we should add the flag to runtests.py if we can  
detect the compiler.

- Robert


From dagss at student.matnat.uio.no  Tue Jun  9 10:46:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 10:46:00 +0200
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>
References: <4A159A96.9070907@student.matnat.uio.no>	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>	<4A2E1368.4040701@student.matnat.uio.no>
	<E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>
Message-ID: <4A2E2148.4020204@student.matnat.uio.no>

Robert Bradshaw wrote:
>> Hmm. I'm not against that -- would you then be ok with
>>
>> cdef int[:] foo
>>
>> not giving any object access at all? (The thing is I also want to  
>> use it
>> to access arrays stored in C pointers etc.; + safe passing around of
>> buffers in nogil mode!)
> 
> I could accept that, I trust that the details could be worked out  
> nicely. Would be like a char* from an object, where the user is  
> required to keep the object around long enough?

With C pointers, yes. With buffers acquired from Python objects it would 
hold a reference, but I was thinking only acquisition/release would 
require GIL. I.e.

cdef int[:] a = obj # requires GIL
cdef int[:] b = a # does not require GIL -- extra "acquisition count"
# (remember that acquisition does have a speed penalty too, that
# really must go if one wants to use efficient slices inside
# numeric loops)

In some very contrived cases a release could be needed within a nogil 
block though...there are ways to resolve that (release queues or 
reacquiring the GIL temporarily). But users are unlikely to produce code 
requiring it.

I think I didn't manage to communicate my intentions regarding the other 
question with object[...] vs. templates, but now my daughter is awake so 
it'll have to wait :-)


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun  9 14:29:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 14:29:23 +0200
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
Message-ID: <4A2E55A3.5040509@student.matnat.uio.no>

I realize now that the recent discussion shouldn't be a discussion of 
syntax as such, but rather the overall strategy for numerics in Cython.

I think I finally know how to present the issue that I've been circling 
around during the spring reasonably clearly -- however I don't want to 
start either a) a thread where I have a longwinded start which nobody 
(understandably) bother to pick up, leaving us in the current state of 
indecision, or b) one of those 200-post threads spanning three weeks 
(which is too long and a lot of effort, summed up).

Can we try an IRC session? Do people have time? (Anybody who's 
interested, but at lest Robert and Kurt)? (If so I'll try to pose the 
question more properly on a single wiki screenful or so beforehand.)

-- 
Dag Sverre

From dalcinl at gmail.com  Tue Jun  9 15:52:38 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 9 Jun 2009 10:52:38 -0300
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
	<3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>
Message-ID: <e7ba66e40906090652v1d737514g61e4174269885c14@mail.gmail.com>

On Tue, Jun 9, 2009 at 5:41 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
>
> Does the new patch there resolve the issue?
>

Yes, it seems to fix the issue... Just tested from a fresh SVN update
with the latest patch applied.

BTW, I'm getting this failure in Py3k... we should handle this, though
not sure about what to do.

extdelslice.cpp:464: error: invalid conversion from ?int
(*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?
extsetslice.cpp:466: error: invalid conversion from ?int
(*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?


I'm also getting other failures, some are changes in exception
messages, other are related to division ....

This one is very nince :-)

<...> in complex_numbers_T305
Failed example:
    test_arithmetic(2j, 4j)
Expected:
    (-2j, 6j, -2j, (-8+0j), (0.5+0j))
Got:
    ((-0-2j), 6j, -2j, (-8+0j), (0.5+0j))


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jun  9 16:10:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 9 Jun 2009 16:10:30 +0200 (CEST)
Subject: [Cython] testsuite on Py3.1rc1 - everyting failing
In-Reply-To: <e7ba66e40906090652v1d737514g61e4174269885c14@mail.gmail.com>
References: <e7ba66e40906021722g568bd07aqcdc8a90ed30e0e3c@mail.gmail.com>
	<cd147c960994f6d4c4c737bb69d8df8c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906040718j14143ca0la08d3fb2842bc98e@mail.gmail.com>
	<7ace83e1bede226f86290fd80bd8d0fa.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<e7ba66e40906041449g260e578asa7b172164c8ac2cd@mail.gmail.com>
	<3A6F98F6-FDA3-43F3-9482-17C235F67DD9@math.washington.edu>
	<e7ba66e40906090652v1d737514g61e4174269885c14@mail.gmail.com>
Message-ID: <1a002aa5800dae922b2dde276fd291e3.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Lisandro Dalcin wrote:
> Yes, it seems to fix the issue... Just tested from a fresh SVN update
> with the latest patch applied.

Cool, well done.


> BTW, I'm getting this failure in Py3k... we should handle this, though
> not sure about what to do.
>
> extdelslice.cpp:464: error: invalid conversion from ?int
> (*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?
> extsetslice.cpp:466: error: invalid conversion from ?int
> (*)(PyObject*, Py_ssize_t, Py_ssize_t, PyObject*)? to ?void*?

I think a warning that __*slice__() isn't portable would be nice. Apart
from that, I wouldn't care too much. Maybe remove the test alltogether.


> I'm also getting other failures, some are changes in exception
> messages, other are related to division ....

Exception message changes are ugly. My take is: if Cython generates a
similar exception or compiler error, try to follow Py3 regarding the
message (if it makes sense, but there's usually a reason why they did it).
In our doctests, try to check as much of an exception as possible to avoid
false positives. But I agree that it's really tricky to check these things
in a portable way.


> This one is very nince :-)
>
> <...> in complex_numbers_T305
> Failed example:
>     test_arithmetic(2j, 4j)
> Expected:
>     (-2j, 6j, -2j, (-8+0j), (0.5+0j))
> Got:
>     ((-0-2j), 6j, -2j, (-8+0j), (0.5+0j))

Beautiful. :)

We do a lot of doctest string manipulation already, so that's just another
case IMHO.

Stefan


From kwmsmith at gmail.com  Tue Jun  9 18:04:03 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 9 Jun 2009 11:04:03 -0500
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <4A2E55A3.5040509@student.matnat.uio.no>
References: <4A2E55A3.5040509@student.matnat.uio.no>
Message-ID: <c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>

On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> I realize now that the recent discussion shouldn't be a discussion of
> syntax as such, but rather the overall strategy for numerics in Cython.
>
> I think I finally know how to present the issue that I've been circling
> around during the spring reasonably clearly -- however I don't want to
> start either a) a thread where I have a longwinded start which nobody
> (understandably) bother to pick up, leaving us in the current state of
> indecision, or b) one of those 200-post threads spanning three weeks
> (which is too long and a lot of effort, summed up).
>
> Can we try an IRC session? Do people have time? (Anybody who's
> interested, but at lest Robert and Kurt)? (If so I'll try to pose the
> question more properly on a single wiki screenful or so beforehand.)

I can make time -- I'll just sit around on #cython.  Send an email if
I'm not responding.

Kurt

From kwmsmith at gmail.com  Tue Jun  9 18:05:09 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 9 Jun 2009 11:05:09 -0500
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>
References: <4A2E55A3.5040509@student.matnat.uio.no>
	<c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>
Message-ID: <c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>

On Tue, Jun 9, 2009 at 11:04 AM, Kurt Smith<kwmsmith at gmail.com> wrote:
> On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> I realize now that the recent discussion shouldn't be a discussion of
>> syntax as such, but rather the overall strategy for numerics in Cython.
>>
>> I think I finally know how to present the issue that I've been circling
>> around during the spring reasonably clearly -- however I don't want to
>> start either a) a thread where I have a longwinded start which nobody
>> (understandably) bother to pick up, leaving us in the current state of
>> indecision, or b) one of those 200-post threads spanning three weeks
>> (which is too long and a lot of effort, summed up).
>>
>> Can we try an IRC session? Do people have time? (Anybody who's
>> interested, but at lest Robert and Kurt)? (If so I'll try to pose the
>> question more properly on a single wiki screenful or so beforehand.)
>
> I can make time -- I'll just sit around on #cython. ?Send an email if
> I'm not responding.

Unless, of course, you were thinking of later in the week, or
something ;-)  Just say the word.

>
> Kurt
>

From robertwb at math.washington.edu  Tue Jun  9 18:12:01 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 09:12:01 -0700
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>
References: <4A2E55A3.5040509@student.matnat.uio.no>
	<c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>
	<c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>
Message-ID: <421423AC-F52E-41AF-984B-CA91732BFDF4@math.washington.edu>

On Jun 9, 2009, at 9:05 AM, Kurt Smith wrote:

> On Tue, Jun 9, 2009 at 11:04 AM, Kurt Smith<kwmsmith at gmail.com> wrote:
>> On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> I realize now that the recent discussion shouldn't be a  
>>> discussion of
>>> syntax as such, but rather the overall strategy for numerics in  
>>> Cython.
>>>
>>> I think I finally know how to present the issue that I've been  
>>> circling
>>> around during the spring reasonably clearly -- however I don't  
>>> want to
>>> start either a) a thread where I have a longwinded start which  
>>> nobody
>>> (understandably) bother to pick up, leaving us in the current  
>>> state of
>>> indecision, or b) one of those 200-post threads spanning three weeks
>>> (which is too long and a lot of effort, summed up).
>>>
>>> Can we try an IRC session? Do people have time? (Anybody who's
>>> interested, but at lest Robert and Kurt)? (If so I'll try to pose  
>>> the
>>> question more properly on a single wiki screenful or so beforehand.)
>>
>> I can make time -- I'll just sit around on #cython.  Send an email if
>> I'm not responding.
>
> Unless, of course, you were thinking of later in the week, or
> something ;-)  Just say the word.

I'll be around (much) later tonight.

- Robert


From ondrej at certik.cz  Tue Jun  9 20:03:13 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Tue, 9 Jun 2009 12:03:13 -0600
Subject: [Cython] force cython to do local imports
Message-ID: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>

Hi,

I have the following problem -- I have a package hermes2d, with the structure:

hermes2d/__init__.py
hermes2d/_hermes2d.pyx + .pxd
hermes2d/_forms.pyx

and sometimes (depending on what the users tells cmake) I want to
build the .so libraries out of the tree and put their paths to
PYTHONPATH. Things just work as long as remove the file
hermes2d/__init__.py during the cython compilation. The reason is that
if I do:

cd hermes2d
cython _forms.pyx

it produces _forms.c with lines like:

  __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
__Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
__pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
(unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
{__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
__LINE__; goto __pyx_L1_error;}

Notice the import "hermes2d._hermes2d", which fails if _hermes2d.so is
out of the tree (e..g somewhere else on the filesystem, even if it's
in the search path!). If, however I remove the __init__.py, then
cython generates:

  __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
"LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
(unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
__pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
__pyx_L1_error;}

and everything just works, no matter where the file _hermes2d.so actually is.

Now my question is -- if I want those .so libraries not to be in the
tree, am I doing things the right way? E.g. I just have a line:

from _forms import *

in the hermes2d/forms.py

and then people just do "from hermes2d import forms" and things work,
no matter where the .so files are. But I need some way to force cython
to just pretend there is no __init__.py in there. Why is cython doing
this check at all? Does it have any advantages?

Thanks,
Ondrej

From seb.binet at gmail.com  Tue Jun  9 20:33:27 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Tue, 9 Jun 2009 20:33:27 +0200
Subject: [Cython] command to only parse cython statements ?
Message-ID: <200906092033.27437.binet@cern.ch>

hi there,

Just for fun and to see how cython works, I am trying to write a 
CythonInterpreter inheriting from the usual code.InteractiveConsole.

So far so good, I manage to compile cython-oneliners (leveraging 
pyximport.load_module) but then I'd need some help for multi-lines statements:
## ex:
cdef class Foo:
  pass
##

for this to work in a reasonnable timely fashion, I'd need to see if parsing 
the snippet of code is valid cython (without compiling).
(I guess I'd need to know if the parsing failed b/c of unsupported python 
constructs or if it is b/c the cython command is incomplete)

any hint ?

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From dagss at student.matnat.uio.no  Tue Jun  9 21:41:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 09 Jun 2009 21:41:14 +0200
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <421423AC-F52E-41AF-984B-CA91732BFDF4@math.washington.edu>
References: <4A2E55A3.5040509@student.matnat.uio.no>	<c266a7930906090904r388b3f5endf34cbfa90c7b3e6@mail.gmail.com>	<c266a7930906090905j6e60a548s684eb8e9ce73235a@mail.gmail.com>
	<421423AC-F52E-41AF-984B-CA91732BFDF4@math.washington.edu>
Message-ID: <4A2EBADA.3010600@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 9, 2009, at 9:05 AM, Kurt Smith wrote:
> 
>> On Tue, Jun 9, 2009 at 11:04 AM, Kurt Smith<kwmsmith at gmail.com> wrote:
>>> On Tue, Jun 9, 2009 at 7:29 AM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> I realize now that the recent discussion shouldn't be a  
>>>> discussion of
>>>> syntax as such, but rather the overall strategy for numerics in  
>>>> Cython.
>>>>
>>>> I think I finally know how to present the issue that I've been  
>>>> circling
>>>> around during the spring reasonably clearly -- however I don't  
>>>> want to
>>>> start either a) a thread where I have a longwinded start which  
>>>> nobody
>>>> (understandably) bother to pick up, leaving us in the current  
>>>> state of
>>>> indecision, or b) one of those 200-post threads spanning three weeks
>>>> (which is too long and a lot of effort, summed up).
>>>>
>>>> Can we try an IRC session? Do people have time? (Anybody who's
>>>> interested, but at lest Robert and Kurt)? (If so I'll try to pose  
>>>> the
>>>> question more properly on a single wiki screenful or so beforehand.)
>>> I can make time -- I'll just sit around on #cython.  Send an email if
>>> I'm not responding.
>> Unless, of course, you were thinking of later in the week, or
>> something ;-)  Just say the word.
> 
> I'll be around (much) later tonight.

I'm primarily available 2 more hours, but I guess that likely I'll catch 
you in my morning (10 hours from now).

(If it's the only option I'm at a point where I will get up in the 
middle of the night if it helps getting this stuff resolved though -- if 
so please respond with an estimate. Just sometime during the week is 
fine though.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 10 07:36:11 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 07:36:11 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <200906092033.27437.binet@cern.ch>
References: <200906092033.27437.binet@cern.ch>
Message-ID: <9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>

Sebastien Binet wrote:
> hi there,
>
> Just for fun and to see how cython works, I am trying to write a
> CythonInterpreter inheriting from the usual code.InteractiveConsole.
>
> So far so good, I manage to compile cython-oneliners (leveraging
> pyximport.load_module) but then I'd need some help for multi-lines
> statements:
> ## ex:
> cdef class Foo:
>   pass
> ##
>
> for this to work in a reasonnable timely fashion, I'd need to see if
> parsing
> the snippet of code is valid cython (without compiling).
> (I guess I'd need to know if the parsing failed b/c of unsupported python
> constructs or if it is b/c the cython command is incomplete)
>
> any hint ?

I don't think this is possible currently as Cython can generate errors at
any point in its pipeline. In time it might be possible to stop the
pipeline at an earlier stage, but I wouldn't expect great speedups here
still as parsing takes a significant part of the time (don't remember the
numbers though). Sorry.

Dag Svere


From robertwb at math.washington.edu  Wed Jun 10 08:03:12 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 23:03:12 -0700
Subject: [Cython] The future of the bracket syntax
In-Reply-To: <4A2E2148.4020204@student.matnat.uio.no>
References: <4A159A96.9070907@student.matnat.uio.no>	<717263C4-286F-4AE8-AFCE-711385B0975A@math.washington.edu>	<4A2E1368.4040701@student.matnat.uio.no>
	<E75D79A7-7EAB-4B68-AF88-17A2A86C3D55@math.washington.edu>
	<4A2E2148.4020204@student.matnat.uio.no>
Message-ID: <9378A591-507F-4D07-9838-B28152C4F6A0@math.washington.edu>

On Jun 9, 2009, at 1:46 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>> Hmm. I'm not against that -- would you then be ok with
>>>
>>> cdef int[:] foo
>>>
>>> not giving any object access at all? (The thing is I also want to
>>> use it
>>> to access arrays stored in C pointers etc.; + safe passing around of
>>> buffers in nogil mode!)
>>
>> I could accept that, I trust that the details could be worked out
>> nicely. Would be like a char* from an object, where the user is
>> required to keep the object around long enough?
>
> With C pointers, yes. With buffers acquired from Python objects it  
> would
> hold a reference, but I was thinking only acquisition/release would
> require GIL. I.e.
>
> cdef int[:] a = obj # requires GIL
> cdef int[:] b = a # does not require GIL -- extra "acquisition count"
> # (remember that acquisition does have a speed penalty too, that
> # really must go if one wants to use efficient slices inside
> # numeric loops)
>
> In some very contrived cases a release could be needed within a nogil
> block though...there are ways to resolve that (release queues or
> reacquiring the GIL temporarily). But users are unlikely to produce  
> code
> requiring it.

This sounds like a very good use case, and perhaps one could even  
assign (with a cast?) raw c pointers to it too (or maybe I'm being  
too ambitious there).

- Robert


From stefan_ml at behnel.de  Wed Jun 10 08:07:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 10 Jun 2009 08:07:23 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
References: <200906092033.27437.binet@cern.ch>
	<9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
Message-ID: <4A2F4D9B.9070502@behnel.de>

Hi,

Dag Sverre Seljebotn wote:
> Sebastien Binet wrote:
>> Just for fun and to see how cython works, I am trying to write a
>> CythonInterpreter inheriting from the usual code.InteractiveConsole.

Way cool. I always wanted something like that for quickly checking out code
snippets without going all the way through writing a module and
pyximporting it.


>> So far so good, I manage to compile cython-oneliners (leveraging
>> pyximport.load_module) but then I'd need some help for multi-lines
>> statements:
>> ## ex:
>> cdef class Foo:
>>   pass
>> ##
>>
>> for this to work in a reasonnable timely fashion, I'd need to see if
>> parsing the snippet of code is valid cython (without compiling).

The current parser isn't made for parsing things interactively, so I have
no idea if this is trivial or hard to do.

You might get away with checking for parser errors that indicate a missing
indentation after the last character that you passed. Every parse error
inside or before the last line would indicate a 'real' parser error.


>> (I guess I'd need to know if the parsing failed b/c of unsupported python
>> constructs or if it is b/c the cython command is incomplete)
> 
> I don't think this is possible currently as Cython can generate errors at
> any point in its pipeline.

This isn't the point here, though. All he needs is to know if a command is
complete or if he should keep reading stuff from the console.


> In time it might be possible to stop the
> pipeline at an earlier stage, but I wouldn't expect great speedups here
> still as parsing takes a significant part of the time (don't remember the
> numbers though).

Parse time really isn't a problem here. I think the main problem will be
that you need to load and reload tons of modules (again, not a timing
issue). Maybe giving the module a public C-API interface and accessing it
through ctypes would work better? But that's not something that needs
solving as a first step. I'm fine with loading a couple of thousand modules
in an interactive session for now. The main use case is testing stuff, so
restarting the interpreter when it gets too heavy isn't a huge problem IMHO.

Stefan

From robertwb at math.washington.edu  Wed Jun 10 08:13:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Jun 2009 23:13:24 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
Message-ID: <BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>

On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:

> Hi,
>
> I have the following problem -- I have a package hermes2d, with the  
> structure:
>
> hermes2d/__init__.py
> hermes2d/_hermes2d.pyx + .pxd
> hermes2d/_forms.pyx
>
> and sometimes (depending on what the users tells cmake) I want to
> build the .so libraries out of the tree and put their paths to
> PYTHONPATH. Things just work as long as remove the file
> hermes2d/__init__.py during the cython compilation. The reason is that
> if I do:
>
> cd hermes2d
> cython _forms.pyx
>
> it produces _forms.c with lines like:
>
>   __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}
>
> Notice the import "hermes2d._hermes2d", which fails if _hermes2d.so is
> out of the tree (e..g somewhere else on the filesystem, even if it's
> in the search path!). If, however I remove the __init__.py, then
> cython generates:
>
>   __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
> __pyx_L1_error;}
>
> and everything just works, no matter where the file _hermes2d.so  
> actually is.
>
> Now my question is -- if I want those .so libraries not to be in the
> tree, am I doing things the right way? E.g. I just have a line:
>
> from _forms import *
>
> in the hermes2d/forms.py
>
> and then people just do "from hermes2d import forms" and things work,
> no matter where the .so files are. But I need some way to force cython
> to just pretend there is no __init__.py in there. Why is cython doing
> this check at all? Does it have any advantages?

One advantage I see is that you may have two modules named _hermes2d  
in different packages that it might have to distinguish. (OK, rare  
with that name, but it could happen with more common names). Also,  
this way the module path is fully qualified as one would expect.

- Robert



From stefan at sun.ac.za  Wed Jun 10 15:58:28 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 10 Jun 2009 15:58:28 +0200
Subject: [Cython] How to build Cython extensions for Windows under Linux or
	OSX
Message-ID: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>

Hi all,

Yesterday, I had to build a Cython extension for a colleague using
Windows, so here are the steps required:

1. Install Wine (on OSX I used http://www.macports.org/)
2. Install Python 2.5 from the http://www.python.org .exe
3. Install mingw from http://www.mingw.org/.  Install the base system and g++.
4. Start the registry editor (wine regedit) and setup the system path:

 - Search for Environment and find the PATH key
 - Append to it c:\python25;c:\mingw\bin;c:\python25\scripts
 - Add a new key called "HOME" and set it to c:\windows\profiles\yourusername

5. Create a new file in ~/.wine/drive_c/windows/profiles/yourusername/
called pydistutils.cfg that contains

[build]
compiler=mingw32

This tells distutils not to look for MSVC7, but to use mingw as the
default compiler.

6. Install setuptools (not necessary, but convenient if you want to
install any other packages).

Get the .tar.gz from http://pypi.python.org/pypi/setuptools.
Unpack.
wine python setup.py install

7. Install Cython (either from source as in 6 or using `easy_install cython`)
8. Compile you extension (from your package source directory):

python setup.py bdist_wininst

This will only work if you have a setup.py file, similar to the one in

http://dip.sun.ac.za/~stefan/code/cython_demo.git

This step should then generate a .exe in the dist/ directory.

If your package requires NumPy to work, you'll have to install that
prior to step 8, by

1. Extracting the numpy source tree
2. python setup.py build -c mingw32
3. python setup.py install --skip-build

I hope someone else finds this recipe useful!

Regards
St?fan

From dalcinl at gmail.com  Wed Jun 10 16:30:15 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 10 Jun 2009 11:30:15 -0300
Subject: [Cython] How to build Cython extensions for Windows under Linux
	or OSX
In-Reply-To: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
References: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
Message-ID: <e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>

Just a couple of comments/questions regarding Windows, an OS I really
do not know well :-)...

1) Why did you need Wine?

2) Why did you use regedit to setup environment variables?
http://support.microsoft.com/kb/310519

3) Regarding setuptools... Why the exe installers at PyPI are not
enough for you?

4) Regarding numpy...  Why the exe installers are SourceForge are not
enough for you?


2009/6/10 St?fan van der Walt <stefan at sun.ac.za>:
> Hi all,
>
> Yesterday, I had to build a Cython extension for a colleague using
> Windows, so here are the steps required:
>
> 1. Install Wine (on OSX I used http://www.macports.org/)
> 2. Install Python 2.5 from the http://www.python.org .exe
> 3. Install mingw from http://www.mingw.org/. ?Install the base system and g++.
> 4. Start the registry editor (wine regedit) and setup the system path:
>
> ?- Search for Environment and find the PATH key
> ?- Append to it c:\python25;c:\mingw\bin;c:\python25\scripts
> ?- Add a new key called "HOME" and set it to c:\windows\profiles\yourusername
>
> 5. Create a new file in ~/.wine/drive_c/windows/profiles/yourusername/
> called pydistutils.cfg that contains
>
> [build]
> compiler=mingw32
>
> This tells distutils not to look for MSVC7, but to use mingw as the
> default compiler.
>
> 6. Install setuptools (not necessary, but convenient if you want to
> install any other packages).
>
> Get the .tar.gz from http://pypi.python.org/pypi/setuptools.
> Unpack.
> wine python setup.py install
>
> 7. Install Cython (either from source as in 6 or using `easy_install cython`)
> 8. Compile you extension (from your package source directory):
>
> python setup.py bdist_wininst
>
> This will only work if you have a setup.py file, similar to the one in
>
> http://dip.sun.ac.za/~stefan/code/cython_demo.git
>
> This step should then generate a .exe in the dist/ directory.
>
> If your package requires NumPy to work, you'll have to install that
> prior to step 8, by
>
> 1. Extracting the numpy source tree
> 2. python setup.py build -c mingw32
> 3. python setup.py install --skip-build
>
> I hope someone else finds this recipe useful!
>
> Regards
> St?fan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed Jun 10 17:08:04 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 10 Jun 2009 11:08:04 -0400
Subject: [Cython] How to build Cython extensions for Windows under Linux
	or OSX
In-Reply-To: <e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
References: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
	<e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
Message-ID: <7f014ea60906100808k3dca2698j889f04dbd2f4402c@mail.gmail.com>

I think because he was cross compiling these extensions for windows
while working in MacOSX or Linux

On Wed, Jun 10, 2009 at 10:30 AM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> Just a couple of comments/questions regarding Windows, an OS I really
> do not know well :-)...
>
> 1) Why did you need Wine?
>
> 2) Why did you use regedit to setup environment variables?
> http://support.microsoft.com/kb/310519
>
> 3) Regarding setuptools... Why the exe installers at PyPI are not
> enough for you?
>
> 4) Regarding numpy... ?Why the exe installers are SourceForge are not
> enough for you?
>
>
> 2009/6/10 St?fan van der Walt <stefan at sun.ac.za>:
>> Hi all,
>>
>> Yesterday, I had to build a Cython extension for a colleague using
>> Windows, so here are the steps required:
>>
>> 1. Install Wine (on OSX I used http://www.macports.org/)
>> 2. Install Python 2.5 from the http://www.python.org .exe
>> 3. Install mingw from http://www.mingw.org/. ?Install the base system and g++.
>> 4. Start the registry editor (wine regedit) and setup the system path:
>>
>> ?- Search for Environment and find the PATH key
>> ?- Append to it c:\python25;c:\mingw\bin;c:\python25\scripts
>> ?- Add a new key called "HOME" and set it to c:\windows\profiles\yourusername
>>
>> 5. Create a new file in ~/.wine/drive_c/windows/profiles/yourusername/
>> called pydistutils.cfg that contains
>>
>> [build]
>> compiler=mingw32
>>
>> This tells distutils not to look for MSVC7, but to use mingw as the
>> default compiler.
>>
>> 6. Install setuptools (not necessary, but convenient if you want to
>> install any other packages).
>>
>> Get the .tar.gz from http://pypi.python.org/pypi/setuptools.
>> Unpack.
>> wine python setup.py install
>>
>> 7. Install Cython (either from source as in 6 or using `easy_install cython`)
>> 8. Compile you extension (from your package source directory):
>>
>> python setup.py bdist_wininst
>>
>> This will only work if you have a setup.py file, similar to the one in
>>
>> http://dip.sun.ac.za/~stefan/code/cython_demo.git
>>
>> This step should then generate a .exe in the dist/ directory.
>>
>> If your package requires NumPy to work, you'll have to install that
>> prior to step 8, by
>>
>> 1. Extracting the numpy source tree
>> 2. python setup.py build -c mingw32
>> 3. python setup.py install --skip-build
>>
>> I hope someone else finds this recipe useful!
>>
>> Regards
>> St?fan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ondrej at certik.cz  Wed Jun 10 17:35:42 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 09:35:42 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
Message-ID: <85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>

On Wed, Jun 10, 2009 at 12:13 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:
>
>> Hi,
>>
>> I have the following problem -- I have a package hermes2d, with the
>> structure:
>>
>> hermes2d/__init__.py
>> hermes2d/_hermes2d.pyx + .pxd
>> hermes2d/_forms.pyx
>>
>> and sometimes (depending on what the users tells cmake) I want to
>> build the .so libraries out of the tree and put their paths to
>> PYTHONPATH. Things just work as long as remove the file
>> hermes2d/__init__.py during the cython compilation. The reason is that
>> if I do:
>>
>> cd hermes2d
>> cython _forms.pyx
>>
>> it produces _forms.c with lines like:
>>
>> ? __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
>> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
>> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
>> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
>> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
>> __LINE__; goto __pyx_L1_error;}
>>
>> Notice the import "hermes2d._hermes2d", which fails if _hermes2d.so is
>> out of the tree (e..g somewhere else on the filesystem, even if it's
>> in the search path!). If, however I remove the __init__.py, then
>> cython generates:
>>
>> ? __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
>> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
>> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
>> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
>> __pyx_L1_error;}
>>
>> and everything just works, no matter where the file _hermes2d.so
>> actually is.
>>
>> Now my question is -- if I want those .so libraries not to be in the
>> tree, am I doing things the right way? E.g. I just have a line:
>>
>> from _forms import *
>>
>> in the hermes2d/forms.py
>>
>> and then people just do "from hermes2d import forms" and things work,
>> no matter where the .so files are. But I need some way to force cython
>> to just pretend there is no __init__.py in there. Why is cython doing
>> this check at all? Does it have any advantages?
>
> One advantage I see is that you may have two modules named _hermes2d
> in different packages that it might have to distinguish. (OK, rare
> with that name, but it could happen with more common names). Also,
> this way the module path is fully qualified as one would expect.

Ok. So how do you propose I should fix the problem, if I need those
.so libraries at different place than the rest of the python source
files?

Ondrej

From seb.binet at gmail.com  Wed Jun 10 17:54:57 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Wed, 10 Jun 2009 17:54:57 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <4A2F4D9B.9070502@behnel.de>
References: <200906092033.27437.binet@cern.ch>
	<9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
	<4A2F4D9B.9070502@behnel.de>
Message-ID: <200906101754.57824.binet@cern.ch>

On Wednesday 10 June 2009 08:07:23 Stefan Behnel wrote:
> Hi,
>
> Dag Sverre Seljebotn wote:
> > Sebastien Binet wrote:
> >> Just for fun and to see how cython works, I am trying to write a
> >> CythonInterpreter inheriting from the usual code.InteractiveConsole.
>
> Way cool. I always wanted something like that for quickly checking out code
> snippets without going all the way through writing a module and
> pyximporting it.
>
> >> So far so good, I manage to compile cython-oneliners (leveraging
> >> pyximport.load_module) but then I'd need some help for multi-lines
> >> statements:
> >> ## ex:
> >> cdef class Foo:
> >>   pass
> >> ##
> >>
> >> for this to work in a reasonnable timely fashion, I'd need to see if
> >> parsing the snippet of code is valid cython (without compiling).
>
> The current parser isn't made for parsing things interactively, so I have
> no idea if this is trivial or hard to do.
>
> You might get away with checking for parser errors that indicate a missing
> indentation after the last character that you passed. Every parse error
> inside or before the last line would indicate a 'real' parser error.

thanks for the hint.
So I am now able to successfully enter at the cython prompt:
cython> cdef class Foo:
...              cdef int i

yeah!

the problem I get now is that I am out of luck if I want to actually do:
cdef class Foo:
  cdef int i
  cdef int j

as the class Foo is compiled just after I entered "cdef int i"...
I suppose I would have to know the indentation context of the last statement 
(if indent_lvl>0, gimme a new line)

hum... maybe I am not tackling this from the correct angle (perhaps that a 
simple buffer of strings filled while the last entry is not the_empty_string 
would do as I quick hack...)

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From robertwb at math.washington.edu  Wed Jun 10 18:01:33 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 10 Jun 2009 09:01:33 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
Message-ID: <574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>

On Jun 10, 2009, at 8:35 AM, Ondrej Certik wrote:

> On Wed, Jun 10, 2009 at 12:13 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:
>>
>>> Hi,
>>>
>>> I have the following problem -- I have a package hermes2d, with the
>>> structure:
>>>
>>> hermes2d/__init__.py
>>> hermes2d/_hermes2d.pyx + .pxd
>>> hermes2d/_forms.pyx
>>>
>>> and sometimes (depending on what the users tells cmake) I want to
>>> build the .so libraries out of the tree and put their paths to
>>> PYTHONPATH. Things just work as long as remove the file
>>> hermes2d/__init__.py during the cython compilation. The reason is  
>>> that
>>> if I do:
>>>
>>> cd hermes2d
>>> cython _forms.pyx
>>>
>>> it produces _forms.c with lines like:
>>>
>>>   __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
>>> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
>>> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
>>> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
>>> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
>>> __LINE__; goto __pyx_L1_error;}
>>>
>>> Notice the import "hermes2d._hermes2d", which fails if  
>>> _hermes2d.so is
>>> out of the tree (e..g somewhere else on the filesystem, even if it's
>>> in the search path!). If, however I remove the __init__.py, then
>>> cython generates:
>>>
>>>   __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
>>> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
>>> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
>>> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
>>> __pyx_L1_error;}
>>>
>>> and everything just works, no matter where the file _hermes2d.so
>>> actually is.
>>>
>>> Now my question is -- if I want those .so libraries not to be in the
>>> tree, am I doing things the right way? E.g. I just have a line:
>>>
>>> from _forms import *
>>>
>>> in the hermes2d/forms.py
>>>
>>> and then people just do "from hermes2d import forms" and things  
>>> work,
>>> no matter where the .so files are. But I need some way to force  
>>> cython
>>> to just pretend there is no __init__.py in there. Why is cython  
>>> doing
>>> this check at all? Does it have any advantages?
>>
>> One advantage I see is that you may have two modules named _hermes2d
>> in different packages that it might have to distinguish. (OK, rare
>> with that name, but it could happen with more common names). Also,
>> this way the module path is fully qualified as one would expect.
>
> Ok. So how do you propose I should fix the problem, if I need those
> .so libraries at different place than the rest of the python source
> files?

I don't know. A hack is to have your setup.py actually move the  
__init__.py temporarily--I'm sure there's a better longterm fix  
though. Just out of curiosity, why do you need to segregate the .so  
files?

- Robert



From robertwb at math.washington.edu  Wed Jun 10 18:02:56 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 10 Jun 2009 09:02:56 -0700
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <200906101754.57824.binet@cern.ch>
References: <200906092033.27437.binet@cern.ch>
	<9816207a627fe833db2adb328f4c68bc.squirrel@webmail.uio.no>
	<4A2F4D9B.9070502@behnel.de> <200906101754.57824.binet@cern.ch>
Message-ID: <D31AC5CD-8DA4-4460-B019-3FC1E646DBAA@math.washington.edu>

On Jun 10, 2009, at 8:54 AM, Sebastien Binet wrote:

> On Wednesday 10 June 2009 08:07:23 Stefan Behnel wrote:
>> Hi,
>>
>> Dag Sverre Seljebotn wote:
>>> Sebastien Binet wrote:
>>>> Just for fun and to see how cython works, I am trying to write a
>>>> CythonInterpreter inheriting from the usual  
>>>> code.InteractiveConsole.
>>
>> Way cool. I always wanted something like that for quickly checking  
>> out code
>> snippets without going all the way through writing a module and
>> pyximporting it.
>>
>>>> So far so good, I manage to compile cython-oneliners (leveraging
>>>> pyximport.load_module) but then I'd need some help for multi-lines
>>>> statements:
>>>> ## ex:
>>>> cdef class Foo:
>>>>   pass
>>>> ##
>>>>
>>>> for this to work in a reasonnable timely fashion, I'd need to  
>>>> see if
>>>> parsing the snippet of code is valid cython (without compiling).
>>
>> The current parser isn't made for parsing things interactively, so  
>> I have
>> no idea if this is trivial or hard to do.
>>
>> You might get away with checking for parser errors that indicate a  
>> missing
>> indentation after the last character that you passed. Every parse  
>> error
>> inside or before the last line would indicate a 'real' parser error.
>
> thanks for the hint.
> So I am now able to successfully enter at the cython prompt:
> cython> cdef class Foo:
> ...              cdef int i
>
> yeah!
>
> the problem I get now is that I am out of luck if I want to  
> actually do:
> cdef class Foo:
>   cdef int i
>   cdef int j
>
> as the class Foo is compiled just after I entered "cdef int i"...
> I suppose I would have to know the indentation context of the last  
> statement
> (if indent_lvl>0, gimme a new line)
>
> hum... maybe I am not tackling this from the correct angle (perhaps  
> that a
> simple buffer of strings filled while the last entry is not  
> the_empty_string
> would do as I quick hack...)

The way ipython does it is to require a new empty line to "leave" an  
indentation block--maybe you could do the same. (Or maybe you could  
even hook this in with ipython).

- Robert


From dagss at student.matnat.uio.no  Wed Jun 10 18:32:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 18:32:24 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
Message-ID: <4A2FE018.5060902@student.matnat.uio.no>

Thanks to chats with Kurt and Robert, I think I've managed to present my 
thoughts from a different and more constructive angle.

http://wiki.cython.org/enhancements/array

There's some time constraints in the picture for various reasons (e.g. 
Fortran GSoC development direction) so it would be great if this could 
be resolved one way or the other before too long. Part of that will 
likely happen on IRC.

Feedback from anybody (especially users) very welcome.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 10 18:35:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 18:35:12 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE018.5060902@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
Message-ID: <4A2FE0C0.9020004@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Thanks to chats with Kurt and Robert, I think I've managed to present my 
> thoughts from a different and more constructive angle.
> 
> http://wiki.cython.org/enhancements/array
> 
> There's some time constraints in the picture for various reasons (e.g. 
> Fortran GSoC development direction) so it would be great if this could 
> be resolved one way or the other before too long. Part of that will 
> likely happen on IRC.
> 
> Feedback from anybody (especially users) very welcome.
> 

Oh..: Please, let's discuss the high-level first (*if* something 
remotely looking like this is wanted at all) before discussing any of 
the lower-level stuff (syntax, implementation, feature-set etc.)

-- 
Dag Sverre

From cournape at gmail.com  Wed Jun 10 18:43:37 2009
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 11 Jun 2009 01:43:37 +0900
Subject: [Cython] How to build Cython extensions for Windows under Linux
	or OSX
In-Reply-To: <e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
References: <9457e7c80906100658o67a39e9etfcffd6e4256306d9@mail.gmail.com>
	<e7ba66e40906100730m2ef620f6rf53032455fd45466@mail.gmail.com>
Message-ID: <5b8d13220906100943y7e3f599dhaf2cb5155a69cbac@mail.gmail.com>

On Wed, Jun 10, 2009 at 11:30 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> Just a couple of comments/questions regarding Windows, an OS I really
> do not know well :-)...
>
> 1) Why did you need Wine?

Because then you don't have to use windows, of course ! :) The last
numpy binaries are built in wine, as a matter of fact.

David

From seb.binet at gmail.com  Wed Jun 10 19:20:01 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Wed, 10 Jun 2009 19:20:01 +0200
Subject: [Cython] command to only parse cython statements ?
In-Reply-To: <D31AC5CD-8DA4-4460-B019-3FC1E646DBAA@math.washington.edu>
References: <200906092033.27437.binet@cern.ch>
	<200906101754.57824.binet@cern.ch>
	<D31AC5CD-8DA4-4460-B019-3FC1E646DBAA@math.washington.edu>
Message-ID: <200906101920.01749.binet@cern.ch>

hi,

On Wednesday 10 June 2009 18:02:56 Robert Bradshaw wrote:
> > hum... maybe I am not tackling this from the correct angle (perhaps
> > that a
> > simple buffer of strings filled while the last entry is not
> > the_empty_string
> > would do as I quick hack...)
>
> The way ipython does it is to require a new empty line to "leave" an
> indentation block--maybe you could do the same. (Or maybe you could
> even hook this in with ipython).

attached is my hack. feel free to steal it...
there are probably many improvements to be performed.

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

-------------- next part --------------
A non-text attachment was scrubbed...
Name: icython.py
Type: text/x-python
Size: 5707 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090610/2839a547/attachment.py 

From ondrej at certik.cz  Wed Jun 10 19:41:58 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 11:41:58 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
Message-ID: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>

On Wed, Jun 10, 2009 at 10:01 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 10, 2009, at 8:35 AM, Ondrej Certik wrote:
>
>> On Wed, Jun 10, 2009 at 12:13 AM, Robert
>> Bradshaw<robertwb at math.washington.edu> wrote:
>>> On Jun 9, 2009, at 11:03 AM, Ondrej Certik wrote:
>>>
>>>> Hi,
>>>>
>>>> I have the following problem -- I have a package hermes2d, with the
>>>> structure:
>>>>
>>>> hermes2d/__init__.py
>>>> hermes2d/_hermes2d.pyx + .pxd
>>>> hermes2d/_forms.pyx
>>>>
>>>> and sometimes (depending on what the users tells cmake) I want to
>>>> build the .so libraries out of the tree and put their paths to
>>>> PYTHONPATH. Things just work as long as remove the file
>>>> hermes2d/__init__.py during the cython compilation. The reason is
>>>> that
>>>> if I do:
>>>>
>>>> cd hermes2d
>>>> cython _forms.pyx
>>>>
>>>> it produces _forms.c with lines like:
>>>>
>>>> ? __pyx_ptype_8hermes2d_9_hermes2d_LinSystem =
>>>> __Pyx_ImportType("hermes2d._hermes2d", "LinSystem", sizeof(struct
>>>> __pyx_obj_8hermes2d_9_hermes2d_LinSystem)); if
>>>> (unlikely(!__pyx_ptype_8hermes2d_9_hermes2d_LinSystem))
>>>> {__pyx_filename = __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno =
>>>> __LINE__; goto __pyx_L1_error;}
>>>>
>>>> Notice the import "hermes2d._hermes2d", which fails if
>>>> _hermes2d.so is
>>>> out of the tree (e..g somewhere else on the filesystem, even if it's
>>>> in the search path!). If, however I remove the __init__.py, then
>>>> cython generates:
>>>>
>>>> ? __pyx_ptype_9_hermes2d_LinSystem = __Pyx_ImportType("_hermes2d",
>>>> "LinSystem", sizeof(struct __pyx_obj_9_hermes2d_LinSystem)); if
>>>> (unlikely(!__pyx_ptype_9_hermes2d_LinSystem)) {__pyx_filename =
>>>> __pyx_f[1]; __pyx_lineno = 292; __pyx_clineno = __LINE__; goto
>>>> __pyx_L1_error;}
>>>>
>>>> and everything just works, no matter where the file _hermes2d.so
>>>> actually is.
>>>>
>>>> Now my question is -- if I want those .so libraries not to be in the
>>>> tree, am I doing things the right way? E.g. I just have a line:
>>>>
>>>> from _forms import *
>>>>
>>>> in the hermes2d/forms.py
>>>>
>>>> and then people just do "from hermes2d import forms" and things
>>>> work,
>>>> no matter where the .so files are. But I need some way to force
>>>> cython
>>>> to just pretend there is no __init__.py in there. Why is cython
>>>> doing
>>>> this check at all? Does it have any advantages?
>>>
>>> One advantage I see is that you may have two modules named _hermes2d
>>> in different packages that it might have to distinguish. (OK, rare
>>> with that name, but it could happen with more common names). Also,
>>> this way the module path is fully qualified as one would expect.
>>
>> Ok. So how do you propose I should fix the problem, if I need those
>> .so libraries at different place than the rest of the python source
>> files?
>
> I don't know. A hack is to have your setup.py actually move the
> __init__.py temporarily--I'm sure there's a better longterm fix
> though. Just out of curiosity, why do you need to segregate the .so
> files?

Because we use cmake to build the project (C++ and cython) and cmake
allows to build the project out of the tree, e.g. it leaves the .cpp
and .py files in the source dir and creates a build dir, where it
stores all the .o and .so files. If the project is just C++, it is
then possible to use the build directory as is, e.g. you don't have to
install it and it still works (of course if you want, you can also
install it).

With Cython, it doesn't work, because the .so files don't find each other.

One solution (besides moving __init__.py files away at each build) is
to just create an option to cython, that will disable this feature (or
pretend the __init__.py files are not there).

I am not convinced it's worth the pain though yet. So you think one
should never need to segregate the .so file in a Python + C++ project?

Ondrej

From ellisonbg.net at gmail.com  Wed Jun 10 19:58:14 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 10 Jun 2009 10:58:14 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
Message-ID: <6ce0ac130906101058k5ebe87c5q39c47a5cbd7eb022@mail.gmail.com>

> With Cython, it doesn't work, because the .so files don't find each other.
>
> One solution (besides moving __init__.py files away at each build) is
> to just create an option to cython, that will disable this feature (or
> pretend the __init__.py files are not there).
>
> I am not convinced it's worth the pain though yet. So you think one
> should never need to segregate the .so file in a Python + C++ project?

I think this should function like the --inplace option to distutils.
Once built, the .so's/.dylibs should be moved into the source tree if
the build is inplace.  Either that, or the .py files should all be
moved over to the out of place build location.  I think trying to run
things in a fragmented manner like you are describing is confusing and
is going to cause unneeded pain.

Cheers,

Brian

> Ondrej
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ellisonbg.net at gmail.com  Wed Jun 10 20:17:18 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 10 Jun 2009 11:17:18 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE0C0.9020004@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A2FE0C0.9020004@student.matnat.uio.no>
Message-ID: <6ce0ac130906101117o6e642454mccb923c26517170d@mail.gmail.com>

Dag,

I quickly glanced through the proposal and have two big picture questions:

* What will this make possible that is currently not possible?

* What will this make easier that is currently really difficult?

Cheers,

Brian

On Wed, Jun 10, 2009 at 9:35 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>> Thanks to chats with Kurt and Robert, I think I've managed to present my
>> thoughts from a different and more constructive angle.
>>
>> http://wiki.cython.org/enhancements/array
>>
>> There's some time constraints in the picture for various reasons (e.g.
>> Fortran GSoC development direction) so it would be great if this could
>> be resolved one way or the other before too long. Part of that will
>> likely happen on IRC.
>>
>> Feedback from anybody (especially users) very welcome.
>>
>
> Oh..: Please, let's discuss the high-level first (*if* something
> remotely looking like this is wanted at all) before discussing any of
> the lower-level stuff (syntax, implementation, feature-set etc.)
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From bpederse at gmail.com  Wed Jun 10 20:37:59 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 10 Jun 2009 11:37:59 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <guhfgd$pd5$1@ger.gmane.org>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
Message-ID: <e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>

On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
> Dag Sverre Seljebotn wrote:
>> Robert wrote:
>>> How to deal with Python's array.array directly - as with numpy.pxd
>>> is there a array.pxd ?
>>
>> Which Python version?
>
> mainly 2.6; and 2.3
>
>> Under Python 3 this should probably happen automatically, try:
>>
>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>
>> Not sure about Python 2.6+
>
> 2.6.2 didn't:
> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
> TypeError: 'array.array' does not have the buffer interface
>
> in Python "buffer(myarray)" also behaves strange
>
>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>> and filling in the Py_buffer struct).
>>
>> If somebody ends up doing this, please submit it for inclusion in
>> Python.
>
> I've put a array.pxd here:
> http://trac.cython.org/cython_trac/ticket/314

hi, what's the status on this? it seemed very useful.
if it needs tests, i could try to write some given some info on where
to start, what to
cover or maybe some cython-numpy tests to from which to crib?
thanks,
-brent


>
> --
> Cython direct interface to Python's array.array type (builtin module).
>
> ? ? * 1D contiguous data view
> ? ? * 2D views - contigious or [x,y] transposed/strided
> ? ? * tools for fast array creation, maximum C-speed and handiness
> ? ? * suitable as allround light weight auto-array within Cython
> code too
>
> ? ? See also: array_example.pyx and doc strings
> --
>
> Tests needed. just used it in 2.6 so far.
>
> Robert
>
>
> PS: For 2.3 a need arose for conditional compilation within the pxd.
> http://docs.cython.org/docs/language_basics.html#conditional-compilation
> doesn't seem to allow comparison against the python version so far? )
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Wed Jun 10 20:57:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 20:57:41 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
Message-ID: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>

Brian Granger wrote:
> Dag,
>
> I quickly glanced through the proposal and have two big picture questions:
>
> * What will this make possible that is currently not possible?

1) Efficient slices

2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
Koepke's project proposal)

3) Automatic contiguous copies in/out if a function is coded to work on
contiguous memory

Why can't this be done currently?

 * Cython contains no in-compiler support for NumPy, and so cannot know
how to create new underlying ndarray objects.

 * All optimizations need to hard-code semantics at compile-time. With
single-element indexing it seemed fair to assume the usual semantics of
zero-based indexing etc., but with slices tings get worse (which kind of
object is returned) and with arithmetic downright impossible (what does *
do again?)

That's not to say there's not other options:
1) We could hard-code support for NumPy only, and only allow ndarray and
not subclasses thereof.

2) We could invent some new protocol/syntax for defining compile-time
semantics for all relevant operations.

> * What will this make easier that is currently really difficult?

Well, nothing is really difficult. But passing, say, a PIL image into C
code and back again for processing (with, say, a contiguous C array) isn't
completely straightforward, but would be with this proposal.

Dag Sverre





From dagss at student.matnat.uio.no  Wed Jun 10 21:08:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 21:08:30 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
References: <guemg0$nc7$1@ger.gmane.org>
	<4A0AE4DF.20209@student.matnat.uio.no>	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
Message-ID: <4A3004AE.2030403@student.matnat.uio.no>

Brent Pedersen wrote:
> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert wrote:
>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>> is there a array.pxd ?
>>> Which Python version?
>> mainly 2.6; and 2.3
>>
>>> Under Python 3 this should probably happen automatically, try:
>>>
>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>
>>> Not sure about Python 2.6+
>> 2.6.2 didn't:
>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>     cdef object[float, ndim=1, mode="c"] b = pyarray
>> TypeError: 'array.array' does not have the buffer interface
>>
>> in Python "buffer(myarray)" also behaves strange
>>
>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>> and filling in the Py_buffer struct).
>>>
>>> If somebody ends up doing this, please submit it for inclusion in
>>> Python.
>> I've put a array.pxd here:
>> http://trac.cython.org/cython_trac/ticket/314
> 
> hi, what's the status on this? it seemed very useful.
> if it needs tests, i could try to write some given some info on where
> to start, what to
> cover or maybe some cython-numpy tests to from which to crib?
> thanks,
> -brent

Great!

The main issue with the patch as I see it is that it tries to hack on 
multi-dimensionality. That is very easily done simply by writing a 
subclass of array instead, and so doesn't belong in the pxd like this.

If you could just remove the multi-dimensional stuff from it I'd be 
happy to accept it. Accompanying tests are strongly preferred though.

There's a section on writing tests here:
http://wiki.cython.org/HackerGuide

I don't expect the numpy tests to be too useful (though they are in 
tests/run), just make sure the basics work with a couple of different 
datatypes. I'll be happy to suggest improvements if you make a first 
iteration anyway.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 10 21:19:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Jun 2009 21:19:24 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A3004AE.2030403@student.matnat.uio.no>
References: <guemg0$nc7$1@ger.gmane.org>	<4A0AE4DF.20209@student.matnat.uio.no>	<guhfgd$pd5$1@ger.gmane.org>	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
Message-ID: <4A30073C.5090106@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Brent Pedersen wrote:
>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert wrote:
>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>> is there a array.pxd ?
>>>> Which Python version?
>>> mainly 2.6; and 2.3
>>>
>>>> Under Python 3 this should probably happen automatically, try:
>>>>
>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>
>>>> Not sure about Python 2.6+
>>> 2.6.2 didn't:
>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>     cdef object[float, ndim=1, mode="c"] b = pyarray
>>> TypeError: 'array.array' does not have the buffer interface
>>>
>>> in Python "buffer(myarray)" also behaves strange
>>>
>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>> and filling in the Py_buffer struct).
>>>>
>>>> If somebody ends up doing this, please submit it for inclusion in
>>>> Python.
>>> I've put a array.pxd here:
>>> http://trac.cython.org/cython_trac/ticket/314
>> hi, what's the status on this? it seemed very useful.
>> if it needs tests, i could try to write some given some info on where
>> to start, what to
>> cover or maybe some cython-numpy tests to from which to crib?
>> thanks,
>> -brent
> 
> Great!
> 
> The main issue with the patch as I see it is that it tries to hack on 
> multi-dimensionality. That is very easily done simply by writing a 
> subclass of array instead, and so doesn't belong in the pxd like this.
> 
> If you could just remove the multi-dimensional stuff from it I'd be 
> happy to accept it. Accompanying tests are strongly preferred though.
> 
> There's a section on writing tests here:
> http://wiki.cython.org/HackerGuide
> 
> I don't expect the numpy tests to be too useful (though they are in 
> tests/run), just make sure the basics work with a couple of different 
> datatypes. I'll be happy to suggest improvements if you make a first 
> iteration anyway.
> 

Also it would be great if any patches could be created using Mercurial.

-- 
Dag Sverre

From stefan_ml at behnel.de  Wed Jun 10 08:14:51 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 10 Jun 2009 08:14:51 +0200
Subject: [Cython] IRC/Skype meeting on numerical strategy of Cython?
In-Reply-To: <4A2E55A3.5040509@student.matnat.uio.no>
References: <4A2E55A3.5040509@student.matnat.uio.no>
Message-ID: <4A2F4F5B.4080000@behnel.de>


Dag Sverre Seljebotn wrote:
> I realize now that the recent discussion shouldn't be a discussion of 
> syntax as such, but rather the overall strategy for numerics in Cython.
> 
> I think I finally know how to present the issue that I've been circling 
> around during the spring reasonably clearly -- however I don't want to 
> start either a) a thread where I have a longwinded start which nobody 
> (understandably) bother to pick up, leaving us in the current state of 
> indecision, or b) one of those 200-post threads spanning three weeks 
> (which is too long and a lot of effort, summed up).
> 
> Can we try an IRC session? Do people have time? (Anybody who's 
> interested, but at lest Robert and Kurt)? (If so I'll try to pose the 
> question more properly on a single wiki screenful or so beforehand.)

I would be interested, although numerics isn't my primary focus. However,
I'll hardly have time to schedule an IRC meeting, so I'll leave it up to
you and see if I can make it somehow.

Anyway, Dag, thanks for driving these things.

Stefan


From dalcinl at gmail.com  Wed Jun 10 23:37:06 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 10 Jun 2009 18:37:06 -0300
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE0C0.9020004@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A2FE0C0.9020004@student.matnat.uio.no>
Message-ID: <e7ba66e40906101437r2cea33ebxc0b3a26d3f3c122f@mail.gmail.com>

On Wed, Jun 10, 2009 at 1:35 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Dag Sverre Seljebotn wrote:
>
> Oh..: Please, let's discuss the high-level first (*if* something
> remotely looking like this is wanted at all) before discussing any of
> the lower-level stuff (syntax, implementation, feature-set etc.)
>

Definitely +1 for me... It looks really good...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From bpederse at gmail.com  Thu Jun 11 00:23:17 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 10 Jun 2009 15:23:17 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A3004AE.2030403@student.matnat.uio.no>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
Message-ID: <e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>

On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Brent Pedersen wrote:
>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert wrote:
>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>> is there a array.pxd ?
>>>> Which Python version?
>>> mainly 2.6; and 2.3
>>>
>>>> Under Python 3 this should probably happen automatically, try:
>>>>
>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>
>>>> Not sure about Python 2.6+
>>> 2.6.2 didn't:
>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
>>> TypeError: 'array.array' does not have the buffer interface
>>>
>>> in Python "buffer(myarray)" also behaves strange
>>>
>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>> and filling in the Py_buffer struct).
>>>>
>>>> If somebody ends up doing this, please submit it for inclusion in
>>>> Python.
>>> I've put a array.pxd here:
>>> http://trac.cython.org/cython_trac/ticket/314
>>
>> hi, what's the status on this? it seemed very useful.
>> if it needs tests, i could try to write some given some info on where
>> to start, what to
>> cover or maybe some cython-numpy tests to from which to crib?
>> thanks,
>> -brent
>
> Great!
>
> The main issue with the patch as I see it is that it tries to hack on
> multi-dimensionality. That is very easily done simply by writing a
> subclass of array instead, and so doesn't belong in the pxd like this.
>
> If you could just remove the multi-dimensional stuff from it I'd be
> happy to accept it. Accompanying tests are strongly preferred though.
>
> There's a section on writing tests here:
> http://wiki.cython.org/HackerGuide
>
> I don't expect the numpy tests to be too useful (though they are in
> tests/run), just make sure the basics work with a couple of different
> datatypes. I'll be happy to suggest improvements if you make a first
> iteration anyway.
>

thanks for the pointers,
i have a start on this, but what should i do with the arrayarray.h?
there doesnt seem to be any cases where a header is included with cython itself.

-brent


> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From bpederse at gmail.com  Thu Jun 11 02:00:16 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Wed, 10 Jun 2009 17:00:16 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
	<e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
Message-ID: <e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>

On Wed, Jun 10, 2009 at 3:23 PM, Brent Pedersen<bpederse at gmail.com> wrote:
> On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Brent Pedersen wrote:
>>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>> Robert wrote:
>>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>>> is there a array.pxd ?
>>>>> Which Python version?
>>>> mainly 2.6; and 2.3
>>>>
>>>>> Under Python 3 this should probably happen automatically, try:
>>>>>
>>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>>
>>>>> Not sure about Python 2.6+
>>>> 2.6.2 didn't:
>>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
>>>> TypeError: 'array.array' does not have the buffer interface
>>>>
>>>> in Python "buffer(myarray)" also behaves strange
>>>>
>>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>>> and filling in the Py_buffer struct).
>>>>>
>>>>> If somebody ends up doing this, please submit it for inclusion in
>>>>> Python.
>>>> I've put a array.pxd here:
>>>> http://trac.cython.org/cython_trac/ticket/314
>>>
>>> hi, what's the status on this? it seemed very useful.
>>> if it needs tests, i could try to write some given some info on where
>>> to start, what to
>>> cover or maybe some cython-numpy tests to from which to crib?
>>> thanks,
>>> -brent
>>
>> Great!
>>
>> The main issue with the patch as I see it is that it tries to hack on
>> multi-dimensionality. That is very easily done simply by writing a
>> subclass of array instead, and so doesn't belong in the pxd like this.
>>
>> If you could just remove the multi-dimensional stuff from it I'd be
>> happy to accept it. Accompanying tests are strongly preferred though.
>>
>> There's a section on writing tests here:
>> http://wiki.cython.org/HackerGuide
>>
>> I don't expect the numpy tests to be too useful (though they are in
>> tests/run), just make sure the basics work with a couple of different
>> datatypes. I'll be happy to suggest improvements if you make a first
>> iteration anyway.
>>
>
> thanks for the pointers,
> i have a start on this, but what should i do with the arrayarray.h?
> there doesnt seem to be any cases where a header is included with cython itself.
>
> -brent
>

here's a first iteration with tests for some feedback.
i'm still not sure what to do with the .h file. currently
the tests have to be run as:
INCLUDE=/path/to/cython-devel/Cython/Includes/ python runtests.py pyarray
-------------- next part --------------
A non-text attachment was scrubbed...
Name: array.patch
Type: text/x-diff
Size: 14963 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090610/31de2748/attachment-0001.bin 

From ondrej at certik.cz  Thu Jun 11 02:45:20 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 18:45:20 -0600
Subject: [Cython] problem with pure python mode
In-Reply-To: <12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
	<12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
Message-ID: <85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>

On Wed, Apr 29, 2009 at 4:00 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Apr 29, 2009, at 2:49 PM, Ondrej Certik wrote:
>
>>>>
>>>> Not sure--have you tried without the locals decorator in the Python
>>>> source? I hope this isn't a regression.
>>>
>>> Yes, as you can see, the locals is commented out. I'll try to bisect
>>> it to see which commit broke it.
>>
>> Ok, so it never worked. I am a bit confused, because I thought the
>> pure python mode worked fine for me before.
>>
>> In the current cython --- is there any way to keep a pure python code,
>> and create an accompanying pxd file that would annotate the cdef
>> functions and classes?
>
> This should work, that's how I implemented this a long time ago. If
> it's not working now it's because something broke (or, I'm not
> promising that it's completely bug-free, but it should be good--we
> use it to bootstrap Cython itself).

So I made it to work by observing how things are done in Cython itself.

If you do:

git clone git://github.com/certik/cython-test.git
cd cython-test
cython fact.pyx

then it produces nicely optimized fact.c file. However, if you do:

cython fact.py

it produces a fact.c, that compiles, but it is not optimized --- the
"i" and "r" variables are python objects, not ints, even though my
fact.pxd file contains:

import cython

@cython.locals(i=cython.int, r=cython.int)
cdef int factorial(int n)


Do you see anything wrong with my files? Is this supposed to work?

I think it is not implemented yet, because when I looked at the cython
sources, it doesn't work there either. This is the relevant part of
Scanning.pxd:

    @cython.locals(current_level=cython.long, new_level=cython.long)
    cpdef indentation_action(self, text)

and here is the declaration in the generated Scanning.c:

  PyObject *__pyx_v_new_level;


And consider for example the following chunk:

  /* "/home/ondrej/repos/cython-devel/Cython/Compiler/Scanning.py":398
 *         new_level = len(text)
 *         #print "Changing indent level from", current_level, "to",
new_level ###
 *         if new_level == current_level:             # <<<<<<<<<<<<<<
 *             return
 *         elif new_level > current_level:
 */
  __pyx_t_4 = PyObject_RichCompare(__pyx_v_new_level,
__pyx_v_current_level, Py_EQ); if (unlikely(!__pyx_t_4))
{__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno =
__LINE__; goto __pyx_L1_error;}
  __Pyx_GOTREF(__pyx_t_4);
  __pyx_t_3 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_3
< 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno
= __LINE__; goto __pyx_L1_error;}
  __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
  if (__pyx_t_3) {


both the new_level and current_level are declared as long, but above
you can see that Cython is calling lots of Python C/API methods...



Ondrej

From ondrej at certik.cz  Thu Jun 11 03:06:33 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Wed, 10 Jun 2009 19:06:33 -0600
Subject: [Cython] -unstable mergeback?
In-Reply-To: <204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
References: <4A14291F.2090806@student.matnat.uio.no>
	<C9D49322-800E-4409-A32D-299E10E64525@math.washington.edu>
	<4A20DED9.5010506@behnel.de>
	<204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
Message-ID: <85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>

On Sat, May 30, 2009 at 9:01 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On May 30, 2009, at 12:23 AM, Stefan Behnel wrote:
>
>> Hi,
>>
>> sorry, was away for a while.
>>
>> Robert Bradshaw schrieb:
>>> On May 20, 2009, at 9:00 AM, Dag Sverre Seljebotn wrote:
>>>
>>>> It seems that there's now a danger of -unstable becoming the main
>>>> development branch; I also think the dangerous things are completed
>>>> myself. If so, I think it should replace the current -devel. This
>>>> will
>>>> make a few tickets blockers for the next release though.
>>>
>>> If -unstable has stabilized, I think this makes sense. I envisioned -
>>> unstable as being for truly unstable things (e.g. closures, temp
>>> allocation rewrites) rather than the main development branch.
>>
>> Same from my side. There are a couple of regressions (failing
>> tests) in the
>> current -unstable that I couldn't easily fix right away. They need
>> some
>> more work before merging it back into -devel. BTW, I'd actually go the
>> opposite way once that's done: merge -devel up into -unstable, drop
>> -devel
>> and rename -unstable.
>
> I'd like to see if there's any major fallout trying to compile sage
> with -unstable before we make the switch though, but assuming things
> aren't catastrophic, let's do that.
>
>> Regarding the -closures branch, IMHO it now works well enough to
>> merge it
>> into cython-unstable. I'd still be happy if others could give it a try
>> first, maybe just run it on some Python code that uses nested
>> functions.
>
> I found an odd bug while demoing closures, so they're not ready yet,
> but I think they're close--certainly ready for other people to start
> trying them out. This is a feature that I would like people to really
> pound on before release.

The closure branch didn't compile our project (Compiler crash in GilCheck):

$ make
[ 74%] Built target hermes2d-real
[ 75%] Cython source

Error converting Pyrex file to C:
------------------------------------------------------------
...
    ctypedef scalar (*BiFormFnVol)(RealFunction *fu, RealFunction *fv,
            RefMap *ru, RefMap *rv)

    ctypedef scalar (*LiFormFnVol)(RealFunction *fv, RefMap *rv)

    cdef BiFormFnVol BF_SYM = <BiFormFnVol> 1
                             ^
------------------------------------------------------------

/home/ondrej/repos/hermes2d/python/hermes2d/_hermes2d.pxd:110:30:
Compiler crash in GilCheck

ModuleNode.body = StatListNode(_hermes2d.pxd:1:0)
StatListNode.stats[5] = CDefExternNode(_hermes2d.pxd:31:5,
    include_file = 'hermes2d.h')
CDefExternNode.body = StatListNode(_hermes2d.pxd:34:4)
StatListNode.stats[5] = SingleAssignmentNode(_hermes2d.pxd:110:9)
SingleAssignmentNode.rhs = TypecastNode(_hermes2d.pxd:110:30)
File 'ExprNodes.py', line 4006, in gil_check: TypecastNode(_hermes2d.pxd:110:30)

Compiler crash traceback from this point on:
  File "/home/ondrej/usr/lib/python/Cython/Compiler/ExprNodes.py",
line 4007, in gil_check
    if self.type.is_pyobject and self.is_temp:
AttributeError: 'NoneType' object has no attribute 'is_pyobject'
make[2]: *** [python/hermes2d/_forms.cpp] Error 1
make[1]: *** [python/hermes2d/CMakeFiles/_forms.dir/all] Error 2
make: *** [all] Error 2



But otherwise this works:


cdef int a(int x):
    def b(int n):
        cdef int r
        r = x + n
        return r
    return b(3)

Just curious question -- if I apply this patch:

$ git diff
diff --git a/fact.pyx b/fact.pyx
index 0cadb8d..518d3ba 100644
--- a/fact.pyx
+++ b/fact.pyx
@@ -7,7 +7,7 @@ cdef int factorial(int n):
     return r

 cdef int a(int x):
-    def b(int n):
+    cdef b(int n):
         cdef int r
         r = x + n
         return r


cython will complain:

Error converting Pyrex file to C:
------------------------------------------------------------
...
    for i in range(1, n + 1):
        r *= i
    return r

cdef int a(int x):
    cdef b(int n):
                ^
------------------------------------------------------------

/home/ondrej/repos/cython-test/fact.pyx:10:17: C function definition
not allowed here



I guess there is no way to make this work in C, right? Because I could
then pass the pointer to the function "b" to some external C program
and that function still needs to access my outer scope.


Ondrej

From robertwb at math.washington.edu  Thu Jun 11 09:28:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Jun 2009 00:28:07 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
Message-ID: <004A836B-BC00-4AA8-9DAA-DA99A737B526@math.washington.edu>


On Jun 10, 2009, at 10:41 AM, Ondrej Certik wrote:

[...]

>>> Ok. So how do you propose I should fix the problem, if I need those
>>> .so libraries at different place than the rest of the python source
>>> files?
>>
>> I don't know. A hack is to have your setup.py actually move the
>> __init__.py temporarily--I'm sure there's a better longterm fix
>> though. Just out of curiosity, why do you need to segregate the .so
>> files?
>
> Because we use cmake to build the project (C++ and cython) and cmake
> allows to build the project out of the tree, e.g. it leaves the .cpp
> and .py files in the source dir and creates a build dir, where it
> stores all the .o and .so files. If the project is just C++, it is
> then possible to use the build directory as is, e.g. you don't have to
> install it and it still works (of course if you want, you can also
> install it).
>
> With Cython, it doesn't work, because the .so files don't find each  
> other.
>
> One solution (besides moving __init__.py files away at each build) is
> to just create an option to cython, that will disable this feature (or
> pretend the __init__.py files are not there).
>
> I am not convinced it's worth the pain though yet. So you think one
> should never need to segregate the .so file in a Python + C++ project?

I don't see a reason why one should need to, but I don't claim  
there's never a valid reason either. More to the point, I don't see a  
quick or good workaround or sufficient motivation for a more complex/ 
disruptive one (but if you do, go ahead and look at it more).

- Robert



From robertwb at math.washington.edu  Thu Jun 11 09:34:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Jun 2009 00:34:32 -0700
Subject: [Cython] -unstable mergeback?
In-Reply-To: <85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
References: <4A14291F.2090806@student.matnat.uio.no>
	<C9D49322-800E-4409-A32D-299E10E64525@math.washington.edu>
	<4A20DED9.5010506@behnel.de>
	<204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
	<85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
Message-ID: <96690D04-7F5B-45DA-A7ED-535D6B9CEF5E@math.washington.edu>

On Jun 10, 2009, at 6:06 PM, Ondrej Certik wrote:

> On Sat, May 30, 2009 at 9:01 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On May 30, 2009, at 12:23 AM, Stefan Behnel wrote:
>>
>>> Hi,
>>>
>>> sorry, was away for a while.
>>>
>>> Robert Bradshaw schrieb:
>>>> On May 20, 2009, at 9:00 AM, Dag Sverre Seljebotn wrote:
>>>>
>>>>> It seems that there's now a danger of -unstable becoming the main
>>>>> development branch; I also think the dangerous things are  
>>>>> completed
>>>>> myself. If so, I think it should replace the current -devel. This
>>>>> will
>>>>> make a few tickets blockers for the next release though.
>>>>
>>>> If -unstable has stabilized, I think this makes sense. I  
>>>> envisioned -
>>>> unstable as being for truly unstable things (e.g. closures, temp
>>>> allocation rewrites) rather than the main development branch.
>>>
>>> Same from my side. There are a couple of regressions (failing
>>> tests) in the
>>> current -unstable that I couldn't easily fix right away. They need
>>> some
>>> more work before merging it back into -devel. BTW, I'd actually  
>>> go the
>>> opposite way once that's done: merge -devel up into -unstable, drop
>>> -devel
>>> and rename -unstable.
>>
>> I'd like to see if there's any major fallout trying to compile sage
>> with -unstable before we make the switch though, but assuming things
>> aren't catastrophic, let's do that.
>>
>>> Regarding the -closures branch, IMHO it now works well enough to
>>> merge it
>>> into cython-unstable. I'd still be happy if others could give it  
>>> a try
>>> first, maybe just run it on some Python code that uses nested
>>> functions.
>>
>> I found an odd bug while demoing closures, so they're not ready yet,
>> but I think they're close--certainly ready for other people to start
>> trying them out. This is a feature that I would like people to really
>> pound on before release.
>
> The closure branch didn't compile our project (Compiler crash in  
> GilCheck):

Thanks for the bug report.

>     for i in range(1, n + 1):
>         r *= i
>     return r
>
> cdef int a(int x):
>     cdef b(int n):
>                 ^
> ------------------------------------------------------------
>
> /home/ondrej/repos/cython-test/fact.pyx:10:17: C function definition
> not allowed here
>
> I guess there is no way to make this work in C, right? Because I could
> then pass the pointer to the function "b" to some external C program
> and that function still needs to access my outer scope.

A closure is more than just an function, it's a function with context  
(i.e. the x from the outer scope can be accessed). It's unclear what  
type "b" would have as a cdef function--wouldn't be impossible to do  
some mangling to make this appear to work in many cases (e.g. it  
could be a struct with the context as one field and function pointer  
in another), but it wouldn't be a (PyObject*)(int) for sure. C just  
doesn't have the concept of "bound" methods.

- Robert



From stefan_ml at behnel.de  Thu Jun 11 09:59:51 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 09:59:51 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
Message-ID: <4A30B977.9090306@behnel.de>


Ondrej Certik wrote:
> we use cmake to build the project (C++ and cython) and cmake
> allows to build the project out of the tree, e.g. it leaves the .cpp
> and .py files in the source dir and creates a build dir, where it
> stores all the .o and .so files.

Uhm, so the real problem is that cmake fails to take the source directory
structure into account when copying build artefacts over to the build
directory?

How is that something to 'fix' in Cython?


> If the project is just C++, it is
> then possible to use the build directory as is, e.g. you don't have to
> install it and it still works (of course if you want, you can also
> install it).

For testing, I build my extensions using distutils' "build_ext -i", which
creates the .so files inside the source tree (and leaves everything else
out of sight).

Stefan


From dagss at student.matnat.uio.no  Thu Jun 11 11:01:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 11:01:50 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>
References: <guemg0$nc7$1@ger.gmane.org>
	<4A0AE4DF.20209@student.matnat.uio.no>	<guhfgd$pd5$1@ger.gmane.org>	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>	<4A3004AE.2030403@student.matnat.uio.no>	<e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
	<e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>
Message-ID: <4A30C7FE.7080107@student.matnat.uio.no>

Brent Pedersen wrote:
> On Wed, Jun 10, 2009 at 3:23 PM, Brent Pedersen<bpederse at gmail.com> wrote:
>> On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>> Brent Pedersen wrote:
>>>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>>>> Dag Sverre Seljebotn wrote:
>>>>>> Robert wrote:
>>>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>>>> is there a array.pxd ?
>>>>>> Which Python version?
>>>>> mainly 2.6; and 2.3
>>>>>
>>>>>> Under Python 3 this should probably happen automatically, try:
>>>>>>
>>>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>>>
>>>>>> Not sure about Python 2.6+
>>>>> 2.6.2 didn't:
>>>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>>>     cdef object[float, ndim=1, mode="c"] b = pyarray
>>>>> TypeError: 'array.array' does not have the buffer interface
>>>>>
>>>>> in Python "buffer(myarray)" also behaves strange
>>>>>
>>>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>>>> and filling in the Py_buffer struct).
>>>>>>
>>>>>> If somebody ends up doing this, please submit it for inclusion in
>>>>>> Python.
>>>>> I've put a array.pxd here:
>>>>> http://trac.cython.org/cython_trac/ticket/314
>>>> hi, what's the status on this? it seemed very useful.
>>>> if it needs tests, i could try to write some given some info on where
>>>> to start, what to
>>>> cover or maybe some cython-numpy tests to from which to crib?
>>>> thanks,
>>>> -brent
>>> Great!
>>>
>>> The main issue with the patch as I see it is that it tries to hack on
>>> multi-dimensionality. That is very easily done simply by writing a
>>> subclass of array instead, and so doesn't belong in the pxd like this.
>>>
>>> If you could just remove the multi-dimensional stuff from it I'd be
>>> happy to accept it. Accompanying tests are strongly preferred though.
>>>
>>> There's a section on writing tests here:
>>> http://wiki.cython.org/HackerGuide
>>>
>>> I don't expect the numpy tests to be too useful (though they are in
>>> tests/run), just make sure the basics work with a couple of different
>>> datatypes. I'll be happy to suggest improvements if you make a first
>>> iteration anyway.
>>>
>> thanks for the pointers,
>> i have a start on this, but what should i do with the arrayarray.h?
>> there doesnt seem to be any cases where a header is included with cython itself.
>>
>> -brent
>>
> 
> here's a first iteration with tests for some feedback.
> i'm still not sure what to do with the .h file. currently

Hmm. If it wasn't for the #ifdefs it could have been ported to Cython 
code and inserted in the pxd I think.

Anyway, there's not much to do -- but you can have a look at runtests.py 
to make it add Cython/Includes to the C include path automatically?

I'm a bit uncertain about this though. Is the faster creation etc. 
really necesarry? It seems like adding a lot of complexity (basically 
one needs to keep up with all Python versions and verify that the header 
still works for each new Python release). So I'm not quite sure about 
it...thoughts? But if it is deemed important I think it can go in.

Tests seems fine -- I only glanced superficially, I just see that they 
are there and trust the submitter of the patch to know the subject 
matter better than me.

Finally, to properly support the PEP 3118, one should make sure that 
memory is not deallocated between a call to getbuffer and release. I.e. 
when the array is resized, one should not free the memory, but 
releasebuffer should do that.

I realize that this is likely impossible to do though (the array can be 
resized through Python calls, right?) so we'll just have to drop that 
and make a big warning not to resize while a buffer is acquired.

-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 16:45:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 16:45:50 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE018.5060902@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
Message-ID: <4A31189E.6000104@behnel.de>


Dag Sverre Seljebotn wrote:
> Thanks to chats with Kurt and Robert, I think I've managed to present my 
> thoughts from a different and more constructive angle.
> 
> http://wiki.cython.org/enhancements/array
> 
> There's some time constraints in the picture for various reasons (e.g. 
> Fortran GSoC development direction) so it would be great if this could 
> be resolved one way or the other before too long.

I'm mostly happy with the proposal. IMHO, this would also fit many use
cases of ticket 153

http://trac.cython.org/cython_trac/ticket/153

- I wonder why the "custom reference counting" is necessary, though. I
assume that the main advantage of doing it in Cython instead of CPython is
that we could allow memory allocation and ref-counting in nogil blocks, but
that would require us to add some other kind of synchronisation for arrays
that pass function barriers. I don't think that's worth it.

I doubt that many people would be surprised by getting a real Python object
when they use this feature, so why not just implement Cython arrays as
PyVarObjects?

- Another point is the "inout" modifier in the one example. Would that only
apply to C++ or would the semantics be the same in C? And: is it a general
modifier that would also apply to scalar variables (and how), or only to
arrays? What about pointers?

If it only applies to arrays, maybe putting it inside the array declaration
(e.g. as a kw arg) makes more sense?

- Should this
"""
Arithmetic (a + b turned into a loop on the arrays a and b)
"""

read

"""
Arithmetic (a + b returns a concatenated copy of a and b)
"""
?

Stefan

From bpederse at gmail.com  Thu Jun 11 16:48:36 2009
From: bpederse at gmail.com (Brent Pedersen)
Date: Thu, 11 Jun 2009 07:48:36 -0700
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A30C7FE.7080107@student.matnat.uio.no>
References: <guemg0$nc7$1@ger.gmane.org> <4A0AE4DF.20209@student.matnat.uio.no>
	<guhfgd$pd5$1@ger.gmane.org>
	<e183a99d0906101137k5123a784n70fbdef537c268fc@mail.gmail.com>
	<4A3004AE.2030403@student.matnat.uio.no>
	<e183a99d0906101523t3011fd0eqaf30a49b7b0eda3@mail.gmail.com>
	<e183a99d0906101700x79377319g6f92970a01f21eb6@mail.gmail.com>
	<4A30C7FE.7080107@student.matnat.uio.no>
Message-ID: <e183a99d0906110748r71d012e8q4d4ba306536d964a@mail.gmail.com>

On Thu, Jun 11, 2009 at 2:01 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Brent Pedersen wrote:
>> On Wed, Jun 10, 2009 at 3:23 PM, Brent Pedersen<bpederse at gmail.com> wrote:
>>> On Wed, Jun 10, 2009 at 12:08 PM, Dag Sverre
>>> Seljebotn<dagss at student.matnat.uio.no> wrote:
>>>> Brent Pedersen wrote:
>>>>> On Thu, May 14, 2009 at 9:05 AM, Robert<kxroberto at googlemail.com> wrote:
>>>>>> Dag Sverre Seljebotn wrote:
>>>>>>> Robert wrote:
>>>>>>>> How to deal with Python's array.array directly - as with numpy.pxd
>>>>>>>> is there a array.pxd ?
>>>>>>> Which Python version?
>>>>>> mainly 2.6; and 2.3
>>>>>>
>>>>>>> Under Python 3 this should probably happen automatically, try:
>>>>>>>
>>>>>>> cdef object[int, ndim=1, mode="c"] arr = yourarray
>>>>>>>
>>>>>>> Not sure about Python 2.6+
>>>>>> 2.6.2 didn't:
>>>>>> File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
>>>>>> ? ? cdef object[float, ndim=1, mode="c"] b = pyarray
>>>>>> TypeError: 'array.array' does not have the buffer interface
>>>>>>
>>>>>> in Python "buffer(myarray)" also behaves strange
>>>>>>
>>>>>>> For Python 2.5- an array.pxd must be written. It is not difficult, one
>>>>>>> simply follows the pattern in numpy.pxd (by implementing __getbuffer__
>>>>>>> and filling in the Py_buffer struct).
>>>>>>>
>>>>>>> If somebody ends up doing this, please submit it for inclusion in
>>>>>>> Python.
>>>>>> I've put a array.pxd here:
>>>>>> http://trac.cython.org/cython_trac/ticket/314
>>>>> hi, what's the status on this? it seemed very useful.
>>>>> if it needs tests, i could try to write some given some info on where
>>>>> to start, what to
>>>>> cover or maybe some cython-numpy tests to from which to crib?
>>>>> thanks,
>>>>> -brent
>>>> Great!
>>>>
>>>> The main issue with the patch as I see it is that it tries to hack on
>>>> multi-dimensionality. That is very easily done simply by writing a
>>>> subclass of array instead, and so doesn't belong in the pxd like this.
>>>>
>>>> If you could just remove the multi-dimensional stuff from it I'd be
>>>> happy to accept it. Accompanying tests are strongly preferred though.
>>>>
>>>> There's a section on writing tests here:
>>>> http://wiki.cython.org/HackerGuide
>>>>
>>>> I don't expect the numpy tests to be too useful (though they are in
>>>> tests/run), just make sure the basics work with a couple of different
>>>> datatypes. I'll be happy to suggest improvements if you make a first
>>>> iteration anyway.
>>>>
>>> thanks for the pointers,
>>> i have a start on this, but what should i do with the arrayarray.h?
>>> there doesnt seem to be any cases where a header is included with cython itself.
>>>
>>> -brent
>>>
>>
>> here's a first iteration with tests for some feedback.
>> i'm still not sure what to do with the .h file. currently
>
> Hmm. If it wasn't for the #ifdefs it could have been ported to Cython
> code and inserted in the pxd I think.
>
> Anyway, there's not much to do -- but you can have a look at runtests.py
> to make it add Cython/Includes to the C include path automatically?

i updated the patch to do that, but then what about when running outside
of tests? perhaps this is better as its own python module? robert?
i'm attaching updated patch here which adjusts the runtests.py hopefully
someone can add to the tests or suggest more. they're surely not complete
as i have only a superficial understanding of this.

-brent

>
> I'm a bit uncertain about this though. Is the faster creation etc.
> really necesarry? It seems like adding a lot of complexity (basically
> one needs to keep up with all Python versions and verify that the header
> still works for each new Python release). So I'm not quite sure about
> it...thoughts? But if it is deemed important I think it can go in.
>
> Tests seems fine -- I only glanced superficially, I just see that they
> are there and trust the submitter of the patch to know the subject
> matter better than me.
>
> Finally, to properly support the PEP 3118, one should make sure that
> memory is not deallocated between a call to getbuffer and release. I.e.
> when the array is resized, one should not free the memory, but
> releasebuffer should do that.
>
> I realize that this is likely impossible to do though (the array can be
> resized through Python calls, right?) so we'll just have to drop that
> and make a big warning not to resize while a buffer is acquired.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: array.export.patch
Type: text/x-patch
Size: 16652 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090611/58272a94/attachment-0001.bin 

From ondrej at certik.cz  Thu Jun 11 17:23:06 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 09:23:06 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A30B977.9090306@behnel.de>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
Message-ID: <85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>

On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Ondrej Certik wrote:
>> we use cmake to build the project (C++ and cython) and cmake
>> allows to build the project out of the tree, e.g. it leaves the .cpp
>> and .py files in the source dir and creates a build dir, where it
>> stores all the .o and .so files.
>
> Uhm, so the real problem is that cmake fails to take the source directory
> structure into account when copying build artefacts over to the build
> directory?

I think cmake is doing it right, it only builds binary things in the
binary directory and leaves all the sources (be it python or C++) in
the source directory.

>
> How is that something to 'fix' in Cython?

Consider this cython statement:

from hermes2d._hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
        int_grad_u_grad_v, int_v, malloc

This is essentially converted to a statement

py_module = PyImport_Import("hermes2d._hermes2d");      // Yes, I know
it's actually PyString in there and this is handled automatically by
__Pyx_ImportModule

Which is correct. Consider however this statement:

from _hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
        int_grad_u_grad_v, int_v, malloc

which is converted to:

py_module = PyImport_Import("hermes2d._hermes2d");

which I think is not right, because Cython changed my import path (and
what is worse, it depends on the positions of the __init__.py files in
my directory). All I am asking is if there is a really good reason,
why Cython mangles the import paths, why it doesn't leave it to the
user. Here is the documentation for PyImport_Import:

http://docs.python.org/c-api/import.html#PyImport_ImportModule

it says that "Changed in version 2.6: always use absolute imports".
What does it mean? Does it mean that one should always pass absolute
imports in it? If that is the case, then that would be a reason why
Cython does it. But I created a simple hack to Cython, which leaves
the paths as they are read from the .pyx file and it works for me in
python2.6 and does exactly what I want. I still need to polish the
patch, as it fails in some other cases, but essentially it's just a
change in search_include_directories() and couple other places.


>
>
>> If the project is just C++, it is
>> then possible to use the build directory as is, e.g. you don't have to
>> install it and it still works (of course if you want, you can also
>> install it).
>
> For testing, I build my extensions using distutils' "build_ext -i", which
> creates the .so files inside the source tree (and leaves everything else
> out of sight).

Right, I use the same approach with cmake (e.g. building in the tree)
and that works fine.

Some of my colleagues however like to work out of the tree (which also
has its advantages), so I am investigating if that is possible with
Cython (it works just fine with C++, where it links all the libraries
in the correct places, and when it installs, it changes the library
paths in the binaries).

Ondrej

From dagss at student.matnat.uio.no  Thu Jun 11 17:33:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 17:33:15 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31189E.6000104@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31189E.6000104@behnel.de>
Message-ID: <4A3123BB.5080603@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>   
>> Thanks to chats with Kurt and Robert, I think I've managed to present my 
>> thoughts from a different and more constructive angle.
>>
>> http://wiki.cython.org/enhancements/array
>>
>> There's some time constraints in the picture for various reasons (e.g. 
>> Fortran GSoC development direction) so it would be great if this could 
>> be resolved one way or the other before too long.
>>     
>
> I'm mostly happy with the proposal. IMHO, this would also fit many use
> cases of ticket 153
>   
Nice, and thanks for the feedback. I was mostly worried that this would 
be seen as too special-purpose for inclusion in mainline Cython.

I'll start with what I consider most important here:

> - Should this
> """
> Arithmetic (a + b turned into a loop on the arrays a and b)
> """
>
> read
>
> """
> Arithmetic (a + b returns a concatenated copy of a and b)
> """
> ?
>   
Well, that would upset all us numerical users a lot :-)

Doing arithmetic componentwise is one of the big ideas here. If + means 
concatenation you gain perhaps a convenient array type but it becomes 
much, *much* less useful for numerical purposes. I can see why one would 
argue against it though (not found in Python, array.array does 
concatenation etc.) so this is where I can only state what I'd 
personally find really useful. I feel pragmatism should win this one but 
you've got purism on your side.

PEP 3118 doesn't allow resizing buffers, which makes concatenation less 
natural to me. Though we could always return a new copy...

Anyway, say we want to make the usecases of both groups possible. Then 
the choice seems clear; if we use + for addition, one can make use of 
the full set +,-,/,*,** on arrays, while concatenation, being only one 
operation, could be moved into "cat(arr1, arr2)" or something. Using + 
for concatenation makes -,/,*,** useless...

I'm convinced that + would be used *a lot* more for componentwise 
addition than for concatenation.


> http://trac.cython.org/cython_trac/ticket/153
>
> - I wonder why the "custom reference counting" is necessary, though. I
> assume that the main advantage of doing it in Cython instead of CPython is
> that we could allow memory allocation and ref-counting in nogil blocks, but
> that would require us to add some other kind of synchronisation for arrays
> that pass function barriers. I don't think that's worth it.
>
> I doubt that many people would be surprised by getting a real Python object
> when they use this feature, so why not just implement Cython arrays as
> PyVarObjects?
>   
Yes, I've been thinking about this as well, and you are right that the 
proposal isn't quite there in that area.

I'll need to think more before I write more about this. Suffice to say 
now that there's many layers:

1) Memory container object. Definitely Python object.
2) Typed buffer acquisition, likely containing a single refcounted 
Py_buffer.
3) The actual variables, which will be structs with unpacked information 
from the Py_buffer.

Although I would like multithreading to be easier for numerical 
programming, here's an orthogonal solution: A new type of "nogil" 
section which releases the GIL but temporarily reacquires it on Python 
operations. On most code this would be slower, but it allows writing 
mostly nogil code but more easily take a slice or reassign an array here 
and there.

I need to reread that post on what issues Greg had with "with gil" though.

> - Another point is the "inout" modifier in the one example. Would that only
> apply to C++ or would the semantics be the same in C? And: is it a general
> modifier that would also apply to scalar variables (and how), or only to
> arrays? What about pointers?
>
> If it only applies to arrays, maybe putting it inside the array declaration
> (e.g. as a kw arg) makes more sense?
>   
Consider this:

cdef int[:,:] matrix
cdef func(inout matrix arg):
    cdef matrix var
    ...

So I'd like to avoid putting it in the brackets.

It comes from Fortran; there's no parallell in C or C++. I think it only 
applies to arrays, and it is necesarry to avoid unecessarry copying. It 
means that the contents of the array is copied into a new, contiguous 
array when entering the function, and then back again afterwards. This 
is just an instruction about what should happen to the *contents* of the 
array, so it's not in breach of Python semantics, and specifically would 
not make sense for scalars, and I think not for pointers either.


Dag Sverre


From dagss at student.matnat.uio.no  Thu Jun 11 17:37:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 17:37:30 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3123BB.5080603@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>
	<4A3123BB.5080603@student.matnat.uio.no>
Message-ID: <4A3124BA.70303@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:

(Re inout modifier:)
> Consider this:
>
> cdef int[:,:] matrix
> cdef func(inout matrix arg):
>     cdef matrix var
>     ...
>
> So I'd like to avoid putting it in the brackets.
>   
I made a quite critical typo there -- it should be

ctypedef int[:,:] matrix

That should make it more understandable...

Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 17:44:59 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 17:44:59 +0200
Subject: [Cython] -unstable mergeback?
In-Reply-To: <85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
References: <4A14291F.2090806@student.matnat.uio.no>	<C9D49322-800E-4409-A32D-299E10E64525@math.washington.edu>	<4A20DED9.5010506@behnel.de>	<204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>
	<85b5c3130906101806n25704bf4uc38e54c44db687cc@mail.gmail.com>
Message-ID: <4A31267B.7040105@behnel.de>

Hi,

Ondrej Certik wrote:
> The closure branch didn't compile our project (Compiler crash in GilCheck):
> 
> $ make
> [ 74%] Built target hermes2d-real
> [ 75%] Cython source
> 
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>     ctypedef scalar (*BiFormFnVol)(RealFunction *fu, RealFunction *fv,
>             RefMap *ru, RefMap *rv)
> 
>     ctypedef scalar (*LiFormFnVol)(RealFunction *fv, RefMap *rv)
> 
>     cdef BiFormFnVol BF_SYM = <BiFormFnVol> 1
>                              ^
> ------------------------------------------------------------
> 
> /home/ondrej/repos/hermes2d/python/hermes2d/_hermes2d.pxd:110:30:
> Compiler crash in GilCheck
> 
> ModuleNode.body = StatListNode(_hermes2d.pxd:1:0)
> StatListNode.stats[5] = CDefExternNode(_hermes2d.pxd:31:5,
>     include_file = 'hermes2d.h')
> CDefExternNode.body = StatListNode(_hermes2d.pxd:34:4)
> StatListNode.stats[5] = SingleAssignmentNode(_hermes2d.pxd:110:9)
> SingleAssignmentNode.rhs = TypecastNode(_hermes2d.pxd:110:30)
> File 'ExprNodes.py', line 4006, in gil_check: TypecastNode(_hermes2d.pxd:110:30)
> 
> Compiler crash traceback from this point on:
>   File "/home/ondrej/usr/lib/python/Cython/Compiler/ExprNodes.py",
> line 4007, in gil_check
>     if self.type.is_pyobject and self.is_temp:
> AttributeError: 'NoneType' object has no attribute 'is_pyobject'

Thanks for the report (and for testing!). This isn't related to closures,
though. I bet this also fails in cython-unstable, due to the gil-check
refactoring.

http://trac.cython.org/cython_trac/ticket/329

Stefan

From stefan_ml at behnel.de  Thu Jun 11 18:05:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 18:05:02 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3123BB.5080603@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>
	<4A3123BB.5080603@student.matnat.uio.no>
Message-ID: <4A312B2E.5020602@behnel.de>


Dag Sverre Seljebotn wrote:
> I'll start with what I consider most important here:
> 
>> - Should this
>> """
>> Arithmetic (a + b turned into a loop on the arrays a and b)
>> """
>>
>> read
>>
>> """
>> Arithmetic (a + b returns a concatenated copy of a and b)
>> """
>> ?
>>   
> Well, that would upset all us numerical users a lot :-)
> 
> Doing arithmetic componentwise is one of the big ideas here. If + means 
> concatenation you gain perhaps a convenient array type but it becomes 
> much, *much* less useful for numerical purposes. I can see why one would 
> argue against it though (not found in Python, array.array does 
> concatenation etc.) so this is where I can only state what I'd 
> personally find really useful. I feel pragmatism should win this one but 
> you've got purism on your side.
> 
> PEP 3118 doesn't allow resizing buffers, which makes concatenation less 
> natural to me. Though we could always return a new copy...
> 
> Anyway, say we want to make the usecases of both groups possible. Then 
> the choice seems clear; if we use + for addition, one can make use of 
> the full set +,-,/,*,** on arrays, while concatenation, being only one 
> operation, could be moved into "cat(arr1, arr2)" or something. Using + 
> for concatenation makes -,/,*,** useless...

Tuples support '+' and '*'. My immediate intuition is that an array type
should behave exactly like a (mutable, but fixed-size) tuple, except where
C dictates otherwise.

> I'm convinced that + would be used *a lot* more for componentwise
> addition than for concatenation.

I agree - at least in numerics. But if you want a mathematical vector type,
that's a different thing than a memory managed C array IMHO.

BTW, didn't we have this discussion twice already? That shows that at least
my intuition keeps clearly dictating otherwise, each time we get here.

Stefan


From ellisonbg.net at gmail.com  Thu Jun 11 18:23:14 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Thu, 11 Jun 2009 09:23:14 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
Message-ID: <6ce0ac130906110923y8d00f04i1ebbca1454bd3218@mail.gmail.com>

Dag,

Thanks for the clarifications.  I think this would be a really great
addition.  I often find myself needing fast-than-numpy arrays and this
seems like it would open the door to make that easier.  Also, being
able to freely move between the different types of arrays (numpy, pil,
pointers, etc.) would be great.

One more question?  At times I have used SLT vector template in Cython
and it works really well of you want a fast dynamically sized array
type.  Do you think it would be possible to have cython arrays be
interchangible with these as well?  I don't see why it wouldn't work
through the c++ "ponter" member that vector has.  That would be a
killer feature.

Cheers,

Brian



>> * What will this make possible that is currently not possible?
>
> 1) Efficient slices
>
> 2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
> Koepke's project proposal)
>
> 3) Automatic contiguous copies in/out if a function is coded to work on
> contiguous memory
>
> Why can't this be done currently?
>
> ?* Cython contains no in-compiler support for NumPy, and so cannot know
> how to create new underlying ndarray objects.
>
> ?* All optimizations need to hard-code semantics at compile-time. With
> single-element indexing it seemed fair to assume the usual semantics of
> zero-based indexing etc., but with slices tings get worse (which kind of
> object is returned) and with arithmetic downright impossible (what does *
> do again?)
>
> That's not to say there's not other options:
> 1) We could hard-code support for NumPy only, and only allow ndarray and
> not subclasses thereof.
>
> 2) We could invent some new protocol/syntax for defining compile-time
> semantics for all relevant operations.
>
>> * What will this make easier that is currently really difficult?
>
> Well, nothing is really difficult. But passing, say, a PIL image into C
> code and back again for processing (with, say, a contiguous C array) isn't
> completely straightforward, but would be with this proposal.
>
> Dag Sverre
>
>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Thu Jun 11 18:24:22 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 18:24:22 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
Message-ID: <4A312FB6.7000300@behnel.de>


Ondrej Certik wrote:
> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>> Ondrej Certik wrote:
>>> we use cmake to build the project (C++ and cython) and cmake
>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>> and .py files in the source dir and creates a build dir, where it
>>> stores all the .o and .so files.
>> Uhm, so the real problem is that cmake fails to take the source directory
>> structure into account when copying build artefacts over to the build
>> directory?
> 
> I think cmake is doing it right, it only builds binary things in the
> binary directory and leaves all the sources (be it python or C++) in
> the source directory.

Ok, I get it, so all that's missing here to make this work is put either
the __init__.py files next to the binaries or the binaries next to the
__init__.py files.


> Consider this cython statement:
> 
> from hermes2d._hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>         int_grad_u_grad_v, int_v, malloc
> 
> This is essentially converted to a statement
> 
> py_module = PyImport_Import("hermes2d._hermes2d");      // Yes, I know
> it's actually PyString in there and this is handled automatically by
> __Pyx_ImportModule
> 
> Which is correct. Consider however this statement:
> 
> from _hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>         int_grad_u_grad_v, int_v, malloc
> 
> which is converted to:
> 
> py_module = PyImport_Import("hermes2d._hermes2d");
> 
> which I think is not right, because Cython changed my import path (and
> what is worse, it depends on the positions of the __init__.py files in
> my directory). All I am asking is if there is a really good reason,
> why Cython mangles the import paths

Yes (although it doesn't "mangle" them). Cython needs to know the fully
qualified module names to correctly handle class cimports. Otherwise, the
type would be known with one name in the module that implements it and with
a different name in the module that cimports it. Greg would know this better.

BTW, this has been discussed before (on the Pyrex list, I guess).

Stefan

From dagss at student.matnat.uio.no  Thu Jun 11 18:32:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 18:32:28 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A312B2E.5020602@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>
	<4A312B2E.5020602@behnel.de>
Message-ID: <4A31319C.9090309@student.matnat.uio.no>

Stefan Behnel wrote:
> Tuples support '+' and '*'. My immediate intuition is that an array type
> should behave exactly like a (mutable, but fixed-size) tuple, except where
> C dictates otherwise.
>
>   
>> I'm convinced that + would be used *a lot* more for componentwise
>> addition than for concatenation.
>>     
>
> I agree - at least in numerics. But if you want a mathematical vector type,
> that's a different thing than a memory managed C array IMHO.
>
> BTW, didn't we have this discussion twice already? That shows that at least
> my intuition keeps clearly dictating otherwise, each time we get here.
>   
Hmm, don't remember previous discussions. Anyway there's not that much 
to *discuss* as such; there's obvious interests at conflict here and the 
case seems clear for both.

I'll try to put down the interesting points:

1) Concatenation is a slow operation with large overhead where one can 
after all just use Python objects or cdef methods with negligble 
overhead. Addition is a real usecase where Cython can provide something 
Python/NumPy can not provide (by reducing memory bus traffic for large 
expressions). I.e. concatenation can be done without this feature, while 
for efficient expression evaluation one would have to switch to another 
language -- there's NO way of getting it in Cython otherwise.

2) I'd have to reconsider how much work I'm willing to put into this 
given concatenation semantics. I am scratching my own itch here. If + is 
chosen for concatenation I might end up having to use Fortran anyway in 
some cases (where Python/NumPy isn't fast enough), and so my motivation 
would be mainly on getting Kurt's GSoC through rather than empower 
Cython in general in this area.

3) Sometimes no feature at all is much better than a wrong or 
unintuitive feature, so 2) should NOT decide this, and I'll deal just 
fine with a "no" nevertheless :-)

4) Looking at mailing list activity, the numerical crowd is a very 
significant user of Cython

5) But again, the numerical stuff has happened as a byproduct, I realize 
that neither you nor Robert need it, and that Cython's real goal lies 
elsewhere


Trying one last time for constructive solutions though:
- There's always compiler directives
- Or int[:,:,componentwise=True]

OTOH, clearly

rsq = add(mul(x, x), mul(y, y))

is rather out of the question :-)

Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 11 18:41:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 18:41:01 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <6ce0ac130906110923y8d00f04i1ebbca1454bd3218@mail.gmail.com>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
	<6ce0ac130906110923y8d00f04i1ebbca1454bd3218@mail.gmail.com>
Message-ID: <4A31339D.5030901@student.matnat.uio.no>

Brian Granger wrote:
> One more question?  At times I have used SLT vector template in Cython
> and it works really well of you want a fast dynamically sized array
> type.  Do you think it would be possible to have cython arrays be
> interchangible with these as well?  I don't see why it wouldn't work
> through the c++ "ponter" member that vector has.  That would be a
> killer feature.
>   
Perhaps afterwards (without support for resize it would be supported 
straight away), resizing seems like a natural extension. I think it 
doesn't affect design decisions now and so I'd rather deal with it later 
-- let's get this proposal even accepted first :-)

Dag Sverre

From ondrej at certik.cz  Thu Jun 11 18:49:47 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 10:49:47 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A312FB6.7000300@behnel.de>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
Message-ID: <85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>

On Thu, Jun 11, 2009 at 10:24 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Ondrej Certik wrote:
>> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>>> Ondrej Certik wrote:
>>>> we use cmake to build the project (C++ and cython) and cmake
>>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>>> and .py files in the source dir and creates a build dir, where it
>>>> stores all the .o and .so files.
>>> Uhm, so the real problem is that cmake fails to take the source directory
>>> structure into account when copying build artefacts over to the build
>>> directory?
>>
>> I think cmake is doing it right, it only builds binary things in the
>> binary directory and leaves all the sources (be it python or C++) in
>> the source directory.
>
> Ok, I get it, so all that's missing here to make this work is put either
> the __init__.py files next to the binaries or the binaries next to the
> __init__.py files.

Yes, but putting binaries next to __init__.py files essentially means
building in the tree (then I can just build in the tree from the
beginning, which I do, but other people prefer not to pollute the
build tree), and putting __init__.py files next to binaries doesn't
work for me, because I then have the same module twice in my import
path (once in the source directory, once in the binary directory) and
it clashes -- at least I didn't make it work.

>
>
>> Consider this cython statement:
>>
>> from hermes2d._hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>> ? ? ? ? int_grad_u_grad_v, int_v, malloc
>>
>> This is essentially converted to a statement
>>
>> py_module = PyImport_Import("hermes2d._hermes2d"); ? ? ?// Yes, I know
>> it's actually PyString in there and this is handled automatically by
>> __Pyx_ImportModule
>>
>> Which is correct. Consider however this statement:
>>
>> from _hermes2d cimport scalar, RealFunction, RefMap, WeakForm, \
>> ? ? ? ? int_grad_u_grad_v, int_v, malloc
>>
>> which is converted to:
>>
>> py_module = PyImport_Import("hermes2d._hermes2d");
>>
>> which I think is not right, because Cython changed my import path (and
>> what is worse, it depends on the positions of the __init__.py files in
>> my directory). All I am asking is if there is a really good reason,
>> why Cython mangles the import paths
>
> Yes (although it doesn't "mangle" them). Cython needs to know the fully
> qualified module names to correctly handle class cimports. Otherwise, the
> type would be known with one name in the module that implements it and with
> a different name in the module that cimports it. Greg would know this better.
>
> BTW, this has been discussed before (on the Pyrex list, I guess).

I found this:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/source_files.html

where it says:

"
If you don't do this //fully qualified module names//, you may find
that pickling doesn't work, among other problems. It also ensures that
the Pyrex compiler has the right idea about the layout of the module
namespace, which can be important when accessing extension types
defined in other modules.
"

so I guess it's not worthy to patch cython to do what I want. But I
still have a feeling that this is only an implementation detail in
cython, since Python itself can work this way, so I don't see why
Cython couldn't as well.


Ondrej

From dagss at student.matnat.uio.no  Thu Jun 11 18:53:32 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 18:53:32 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>	<4A30B977.9090306@behnel.de>	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
Message-ID: <4A31368C.7040507@student.matnat.uio.no>

Ondrej Certik wrote:
> On Thu, Jun 11, 2009 at 10:24 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>   
>> Ondrej Certik wrote:
>>     
>>> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>>>       
>>>> Ondrej Certik wrote:
>>>>         
>>>>> we use cmake to build the project (C++ and cython) and cmake
>>>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>>>> and .py files in the source dir and creates a build dir, where it
>>>>> stores all the .o and .so files.
>>>>>           
>>>> Uhm, so the real problem is that cmake fails to take the source directory
>>>> structure into account when copying build artefacts over to the build
>>>> directory?
>>>>         
>>> I think cmake is doing it right, it only builds binary things in the
>>> binary directory and leaves all the sources (be it python or C++) in
>>> the source directory.
>>>       
>> Ok, I get it, so all that's missing here to make this work is put either
>> the __init__.py files next to the binaries or the binaries next to the
>> __init__.py files.
>>     
>
> Yes, but putting binaries next to __init__.py files essentially means
> building in the tree (then I can just build in the tree from the
> beginning, which I do, but other people prefer not to pollute the
> build tree), and putting __init__.py files next to binaries doesn't
> work for me, because I then have the same module twice in my import
> path (once in the source directory, once in the binary directory) and
> it clashes -- at least I didn't make it work.
>   
This is the real problem I think. The concept of a build directory is 
that the result ends up there -- thus, copy over any pure .py files to 
the build directory, and only have that as the import path.

Really, that's how all out-of-tree builds work? Non-compiled files are 
copied.

I'm against any changes needed only because of a wierd import path setting.

Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 19:06:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 19:06:52 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31319C.9090309@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>
	<4A31319C.9090309@student.matnat.uio.no>
Message-ID: <4A3139AC.2050007@behnel.de>


Dag Sverre Seljebotn wrote:
> 5) But again, the numerical stuff has happened as a byproduct, I realize 
> that neither you nor Robert need it, and that Cython's real goal lies 
> elsewhere

It's less of a question of "who needs it", since there would be different
types of use cases for an array and vector type. This is about language
design, so the question is: what would users expect?

When I see a C array, I think of a memory block with a sequence of
identically typed items. I also think of a Python tuple because that
behaves very similar. I totally do not think of a math vector, because that
is a very (very!) special use case.

The examples you gave used "int" as a base type, simply because you were
thinking of numeric use cases. What about an array of pointers or structs?
What should adding two of those do?


> - There's always compiler directives

;) I actually thought about putting a mention of this with a big fat NO
into my last mail but then just stopped waiting for someone else to bring
this up.

A compiler directive would mean that

	a + b

would do entirely different things depending *not* on the types of a and b,
but on the state of the compiler. That's evil.


> - Or int[:,:,componentwise=True]

The main difference here (from a Python POV) is the implementation of
__add__ and friends, right? So the way I see it is that this should not be
an option. It's the entire *type* that should be different. If you want a
vector, use an explicit vector type. If you want a memory managed array,
use a standard array type. Don't mix the two, except after (explicit) coercion.

I haven't thought about how to cast this into a syntax yet, so this is just
my idea on how this should behave.

Stefan

From dagss at student.matnat.uio.no  Thu Jun 11 19:25:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 19:25:58 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3139AC.2050007@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de>
Message-ID: <4A313E26.7080006@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>   
>> 5) But again, the numerical stuff has happened as a byproduct, I realize 
>> that neither you nor Robert need it, and that Cython's real goal lies 
>> elsewhere
>>     
>
> It's less of a question of "who needs it", since there would be different
> types of use cases for an array and vector type. This is about language
> design, so the question is: what would users expect?
>   
Thanks again for supplying your thoughts in this thread.

The problem here is that the support must be built into the language, 
user-defined types won't do the trick (unless you have meta-programming 
capabilities on the level of LISP). So it's also a question of whether 
one can use Cython at all, or must use Fortran or C++ (the sad thing is 
that a language as horrible as Fortran will perhaps never be decently 
replaced, because all other languages consider math vectors a too 
special usecase.)

Anyway, thanks for clarifying the terms. So the question turns into: Is 
it possible to add a *vector* type to Cython? Or is that pulling it too 
far? (Choice of syntax aside.)

> When I see a C array, I think of a memory block with a sequence of
> identically typed items. I also think of a Python tuple because that
> behaves very similar. I totally do not think of a math vector, because that
> is a very (very!) special use case.
>   
Hmm. One can go with two different conclusions though
 a) You're ok with componentwise, but only given a different syntax
 b) Math vectors is too special a usecase to warrant native Cython support

Resolving this is my primary interest here, and we can deal with syntax 
later.

> The examples you gave used "int" as a base type, simply because you were
> thinking of numeric use cases. What about an array of pointers or structs?
> What should adding two of those do?
>   
Pointers: C semantics. I.e. you can add a pointer and an int array 
together to get new pointers, and so on. People are already doing 
exactly this with NumPy arrays (casting the pointers to integers and 
back though).

Structs: Ideally (for numericans) the arithmetic done field-wise. If you 
have a pointer in a struct it won't be allowed (unless, perhaps, one of 
the operands is another struct type with identical fields except that 
any pointer fields have been replaced with int fields. Yes, such things 
could be useful!)

But really I'm happy even if structs are dropped from this.

>> - There's always compiler directives
>>     
>
> ;) I actually thought about putting a mention of this with a big fat NO
> into my last mail but then just stopped waiting for someone else to bring
> this up.
>   
Well, but in reality, the substantial amount of numerical users would 
not consider putting it to anything but componentwise. Just consider it 
a nicer alternative to an outright fork NumCython :-) (Rest easy, I will 
not fork.)

Dag Sverre

From ondrej at certik.cz  Thu Jun 11 19:31:18 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 11:31:18 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A31368C.7040507@student.matnat.uio.no>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<BD5FAB7D-3C22-40A6-BA1D-032B716515D9@math.washington.edu>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
Message-ID: <85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>

On Thu, Jun 11, 2009 at 10:53 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Ondrej Certik wrote:
>> On Thu, Jun 11, 2009 at 10:24 AM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>>
>>> Ondrej Certik wrote:
>>>
>>>> On Thu, Jun 11, 2009 at 1:59 AM, Stefan Behnel wrote:
>>>>
>>>>> Ondrej Certik wrote:
>>>>>
>>>>>> we use cmake to build the project (C++ and cython) and cmake
>>>>>> allows to build the project out of the tree, e.g. it leaves the .cpp
>>>>>> and .py files in the source dir and creates a build dir, where it
>>>>>> stores all the .o and .so files.
>>>>>>
>>>>> Uhm, so the real problem is that cmake fails to take the source directory
>>>>> structure into account when copying build artefacts over to the build
>>>>> directory?
>>>>>
>>>> I think cmake is doing it right, it only builds binary things in the
>>>> binary directory and leaves all the sources (be it python or C++) in
>>>> the source directory.
>>>>
>>> Ok, I get it, so all that's missing here to make this work is put either
>>> the __init__.py files next to the binaries or the binaries next to the
>>> __init__.py files.
>>>
>>
>> Yes, but putting binaries next to __init__.py files essentially means
>> building in the tree (then I can just build in the tree from the
>> beginning, which I do, but other people prefer not to pollute the
>> build tree), and putting __init__.py files next to binaries doesn't
>> work for me, because I then have the same module twice in my import
>> path (once in the source directory, once in the binary directory) and
>> it clashes -- at least I didn't make it work.
>>
> This is the real problem I think. The concept of a build directory is
> that the result ends up there -- thus, copy over any pure .py files to
> the build directory, and only have that as the import path.
>
> Really, that's how all out-of-tree builds work? Non-compiled files are
> copied.

When I copy all the .py files, then it of course works. So maybe you
are right, that I should just copy (or rather link them) all the .py
files into the build dir and be done with this.

>
> I'm against any changes needed only because of a wierd import path setting.

I don't think it's weird.

As far as Cython is concerned, the only problem I can see is that it
changes all import paths to the fully qualified import paths. Python
allows both ways, Cython only the fully qualified way. As Stefan
explained, this is needed, because it will then not work properly at
the moment. But I am not convinced this cannot be fixed -- though it
may happen that the fix is too tedious and not worthy.

Ondrej

From kwmsmith at gmail.com  Thu Jun 11 19:42:44 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 11 Jun 2009 12:42:44 -0500
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A2FE018.5060902@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
Message-ID: <c266a7930906111042p182bbebi9f424d868038600c@mail.gmail.com>

On Wed, Jun 10, 2009 at 11:32 AM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Thanks to chats with Kurt and Robert, I think I've managed to present my
> thoughts from a different and more constructive angle.
>
> http://wiki.cython.org/enhancements/array
>
> There's some time constraints in the picture for various reasons (e.g.
> Fortran GSoC development direction) so it would be great if this could
> be resolved one way or the other before too long. Part of that will
> likely happen on IRC.
>
> Feedback from anybody (especially users) very welcome.

Nice proposal.  As mentioned in other comments, this would certainly
give Cython a strong numerical aspect and would make those using
Cython with numpy or external languages (ahem, Fortran ;-) quite
happy, I think.  The syntax seems to me nicely pythonic -- I have some
small comments on the stuff inside the brackets, but it'll wait until
(if and when) the proposal is accepted.

As an aside -- the array syntax & semantics within Fortran are, IMHO,
the one thing Fortran 90 got right and the main reason people *still*
write programs in it today.  The lack of a high-level array in
C-derivative languages is quite a hurdle for those who want
component-wise operations.  Pulling into Cython something that has
proven itself for a while now wouldn't be a bad thing.  But that's
from a numeric perspective, admittedly.

Kurt

From dalcinl at gmail.com  Thu Jun 11 20:01:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 11 Jun 2009 15:01:25 -0300
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3139AC.2050007@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31189E.6000104@behnel.de> <4A3123BB.5080603@student.matnat.uio.no>
	<4A312B2E.5020602@behnel.de> <4A31319C.9090309@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de>
Message-ID: <e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>

On Thu, Jun 11, 2009 at 2:06 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> When I see a C array, I think of a memory block with a sequence of
> identically typed items. I also think of a Python tuple because that
> behaves very similar. I totally do not think of a math vector, because that
> is a very (very!) special use case.
>

Well, for me it is not a matter about how do think of view an array,
but what do you want to do with an array. Could you tell me (apart
from char/wchar_t because of byte/unicode strings) how many times in
your life as a programmer did you need to concatenate an array of let
say integers or double precision floats ? That (I mean, concatenation)
is for me a very (very!) special use case for arrays...

Almost all APL's (array programming languages) I'm aware of do not
have concatenation semantics for a+b. Python is not an APL, but let me
ask... Why Cython should not be?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Jun 11 20:04:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:04:44 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A313E26.7080006@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>
	<4A313E26.7080006@student.matnat.uio.no>
Message-ID: <4A31473C.8020307@behnel.de>


Dag Sverre Seljebotn wrote:
> The problem here is that the support must be built into the language, 
> user-defined types won't do the trick (unless you have meta-programming 
> capabilities on the level of LISP).

I'm not arguing that the type shouldn't be a builtin. It's just a
difference if you require new syntax for a type or if you special case a
type in the compiler.


> Anyway, thanks for clarifying the terms. So the question turns into: Is 
> it possible to add a *vector* type to Cython? Or is that pulling it too 
> far? (Choice of syntax aside.)

I'm for having both. The vector type could become a subtype of an array
that overrides the numeric operations (although I wouldn't care too much if
that's too hard to implement).

We discussed the templating syntax a while ago. Even if that isn't there
yet, that doesn't mean we can't use its syntax.

	from cython cimport Vector
	cdef Vector[int] some_vector

would be the same as the equivalent "int[:]" array, except for arithmetic.


>> The examples you gave used "int" as a base type, simply because you were
>> thinking of numeric use cases. What about an array of pointers or structs?
>> What should adding two of those do?
>>   
> Pointers: C semantics. I.e. you can add a pointer and an int array 
> together to get new pointers, and so on. People are already doing 
> exactly this with NumPy arrays (casting the pointers to integers and 
> back though).

I actually meant adding two arrays of pointers.


> Structs: Ideally (for numericans) the arithmetic done field-wise.

I would expect that when I have a /vector/ of structs (and yes, that would
be a totally cool feature). An /array/ of structs should concatenate instead.

Stefan


From stefan_ml at behnel.de  Thu Jun 11 20:16:47 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:16:47 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>
	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>
	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>
	<e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>
Message-ID: <4A314A0F.5080505@behnel.de>


Lisandro Dalcin wrote:
> On Thu, Jun 11, 2009 at 2:06 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>> When I see a C array, I think of a memory block with a sequence of
>> identically typed items. I also think of a Python tuple because that
>> behaves very similar. I totally do not think of a math vector, because that
>> is a very (very!) special use case.
> 
> Well, for me it is not a matter about how do think of view an array,
> but what do you want to do with an array. Could you tell me (apart
> from char/wchar_t because of byte/unicode strings) how many times in
> your life as a programmer did you need to concatenate an array of let
> say integers or double precision floats ? That (I mean, concatenation)
> is for me a very (very!) special use case for arrays...

Have you never concatenated tuples? I consider this the usual chicken and
egg problem. If the functionality was available, people would use it.

Currently, I avoid C arrays in Cython wherever I can, simply because it's
really hard to deal with them (manual memory management, resizing, ...). If
Cython grew native array and vector types, I bet people would just jump on
them.

I'd even vote for allowing resizing of Cython's array type (or a subtype),
maybe through a resize() method that would call realloc() internally. That
would remove the need to keep C types in Python lists (or to do manual
memory management to achieve the same).

Stefan


From dalcinl at gmail.com  Thu Jun 11 20:31:31 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 11 Jun 2009 15:31:31 -0300
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>
	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
Message-ID: <e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>

On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>
> As far as Cython is concerned, the only problem I can see is that it
> changes all import paths to the fully qualified import paths. Python
> allows both ways, Cython only the fully qualified way. As Stefan
> explained, this is needed, because it will then not work properly at
> the moment. But I am not convinced this cannot be fixed -- though it
> may happen that the fix is too tedious and not worthy.
>

IMHO, the only "fix" Cython needs is support for relative imports...
Then you could write:

from ._hermes2d cimport what_you_need

This way, Ondrej would be able to easily write Cython code using
relative imports the right way...




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Jun 11 20:35:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 20:35:30 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31473C.8020307@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de>
Message-ID: <4A314E72.6000500@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Anyway, thanks for clarifying the terms. So the question turns into: Is 
>> it possible to add a *vector* type to Cython? Or is that pulling it too 
>> far? (Choice of syntax aside.)
> 
> I'm for having both. The vector type could become a subtype of an array
> that overrides the numeric operations (although I wouldn't care too much if
> that's too hard to implement).
> 
> We discussed the templating syntax a while ago. Even if that isn't there
> yet, that doesn't mean we can't use its syntax.
> 
> 	from cython cimport Vector
> 	cdef Vector[int] some_vector
> 
> would be the same as the equivalent "int[:]" array, except for arithmetic.

That answers most of my worries. I'll leave this to sink in for a few 
days; but I'll start brainstorming for shorter syntax alternatives which 
might make this discussion shorter :-)

Here's some starting points you all won't like:

int[:].cwise
int.cwise[:]

Also consider that "array" is already a Python builtin doing much of 
what Stefan seems to be after, at least Python side. array[int] should 
perhaps at least be an alias for what Stefan wants int[:] to be, at 
least in 1D, although Stefan's arguments from "expectation of random 
Python user" is still valid.

(BTW, this is how a 2D arrays look in Fortran, which is still a very 
dominant tool in numerics:

real, dimension(:,:) :: arr

So you can see where my inspiration with the [:] came from.)
-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 20:40:48 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:40:48 +0200
Subject: [Cython] force cython to do local imports
In-Reply-To: <e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>	<85b5c3130906100835u1794e18q8712ec2f6041870e@mail.gmail.com>	<574A7FC4-2672-4353-AF0C-60F3A2BF46C0@math.washington.edu>	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>	<4A30B977.9090306@behnel.de>	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>	<4A312FB6.7000300@behnel.de>	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>	<4A31368C.7040507@student.matnat.uio.no>	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
Message-ID: <4A314FB0.9090507@behnel.de>


Lisandro Dalcin wrote:
> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>> As far as Cython is concerned, the only problem I can see is that it
>> changes all import paths to the fully qualified import paths. Python
>> allows both ways, Cython only the fully qualified way. As Stefan
>> explained, this is needed, because it will then not work properly at
>> the moment. But I am not convinced this cannot be fixed -- though it
>> may happen that the fix is too tedious and not worthy.
>>
> 
> IMHO, the only "fix" Cython needs is support for relative imports...
> Then you could write:
> 
> from ._hermes2d cimport what_you_need
> 
> This way, Ondrej would be able to easily write Cython code using
> relative imports the right way...

I thought about this, too, and yes, that would be a nice feature. But it
doesn't help in this case. What Ondrej wanted was to be able to import the
module both from its package and as a plain import, i.e. when put outside
of any package. The above would not handle the latter case either.

Stefan

From dagss at student.matnat.uio.no  Thu Jun 11 20:53:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 20:53:42 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A314A0F.5080505@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<e7ba66e40906111101te921484x9f6952b43d629c08@mail.gmail.com>
	<4A314A0F.5080505@behnel.de>
Message-ID: <4A3152B6.6070206@student.matnat.uio.no>

Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>> On Thu, Jun 11, 2009 at 2:06 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>>> When I see a C array, I think of a memory block with a sequence of
>>> identically typed items. I also think of a Python tuple because that
>>> behaves very similar. I totally do not think of a math vector, because that
>>> is a very (very!) special use case.
>> Well, for me it is not a matter about how do think of view an array,
>> but what do you want to do with an array. Could you tell me (apart
>> from char/wchar_t because of byte/unicode strings) how many times in
>> your life as a programmer did you need to concatenate an array of let
>> say integers or double precision floats ? That (I mean, concatenation)
>> is for me a very (very!) special use case for arrays...
> 
> Have you never concatenated tuples? I consider this the usual chicken and
> egg problem. If the functionality was available, people would use it.

I can actually attest to this. In my primary language these days, R 
(very much a componentwise language), I find myself doing

cbind(arr1, arr2)

for concatenating column-wise,

rbind(arr1, arr2)

for concatenating row-wise, and

c(1, 2, 3, arr1)

for unpacking concatenation all the time. Only in situations where speed 
is not an issue though.

(Of course, I use componentwise + even more often -- but having 
available functions doing this even in the componentwise case would be 
nice).

BTW, all of Stefan's examples been one-dimensional, which doesn't quite 
fit with the CEP which has a strong many-dimensional emphasis (or at 
least that was the point). Would + only work for 1D, or would it always 
add along the first dimension (i.e. like rbind above)? How to add along 
other dimensions?

-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 11 20:52:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 20:52:00 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A314E72.6000500@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
Message-ID: <4A315250.7080606@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Anyway, thanks for clarifying the terms. So the question turns into: Is 
>>> it possible to add a *vector* type to Cython? Or is that pulling it too 
>>> far? (Choice of syntax aside.)
>> I'm for having both. The vector type could become a subtype of an array
>> that overrides the numeric operations (although I wouldn't care too much if
>> that's too hard to implement).
>>
>> We discussed the templating syntax a while ago. Even if that isn't there
>> yet, that doesn't mean we can't use its syntax.
>>
>> 	from cython cimport Vector
>> 	cdef Vector[int] some_vector
>>
>> would be the same as the equivalent "int[:]" array, except for arithmetic.
> 
> That answers most of my worries. I'll leave this to sink in for a few 
> days; but I'll start brainstorming for shorter syntax alternatives which 
> might make this discussion shorter :-)
> 
> Here's some starting points you all won't like:
> 
> int[:].cwise
> int.cwise[:]

Yes, you're right. :)

Honestly, what's so bad about making the vector type explicit? Even

	Vector[int[:]]
or
	Vector(int[:])

are better than a hard to read "cwise".


> Also consider that "array" is already a Python builtin doing much of 
> what Stefan seems to be after, at least Python side.

Well, yes, except for storing pointers and structs.


> (BTW, this is how a 2D arrays look in Fortran, which is still a very 
> dominant tool in numerics:
> 
> real, dimension(:,:) :: arr
> 
> So you can see where my inspiration with the [:] came from.)

I like the "int[:]" syntax. I just think that a vector should look
different in the code.

Stefan

From ondrej at certik.cz  Thu Jun 11 21:02:15 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 13:02:15 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <4A314FB0.9090507@behnel.de>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<85b5c3130906101041g1d9c32b4g1a90afbf78bcb849@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
	<4A314FB0.9090507@behnel.de>
Message-ID: <85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>

On Thu, Jun 11, 2009 at 12:40 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Lisandro Dalcin wrote:
>> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>>> As far as Cython is concerned, the only problem I can see is that it
>>> changes all import paths to the fully qualified import paths. Python
>>> allows both ways, Cython only the fully qualified way. As Stefan
>>> explained, this is needed, because it will then not work properly at
>>> the moment. But I am not convinced this cannot be fixed -- though it
>>> may happen that the fix is too tedious and not worthy.
>>>
>>
>> IMHO, the only "fix" Cython needs is support for relative imports...
>> Then you could write:
>>
>> from ._hermes2d cimport what_you_need
>>
>> This way, Ondrej would be able to easily write Cython code using
>> relative imports the right way...
>
> I thought about this, too, and yes, that would be a nice feature. But it
> doesn't help in this case. What Ondrej wanted was to be able to import the
> module both from its package and as a plain import, i.e. when put outside
> of any package. The above would not handle the latter case either.

I think it would. I have _forms.so and _hermes2d.so. The _hermes2d.so
is imported from _forms.so and if the import was relative, e.g. if
_forms.so was doing "import ._hermes2d", then it should work both in
the tree and out of the tree.

Ondrej

Ondrej

From dagss at student.matnat.uio.no  Thu Jun 11 21:07:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 21:07:15 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315250.7080606@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de>
Message-ID: <4A3155E3.908@student.matnat.uio.no>

I never keep a promise to shut up...

Stefan Behnel wrote:
> Honestly, what's so bad about making the vector type explicit? Even
> 
> 	Vector[int[:]]
> or
> 	Vector(int[:])
> 
> are better than a hard to read "cwise".

Being very concrete, the word Vector is not something I normally use in 
this situation and I would really prefer to keep it that way. If a is a 
"vector", I expect linear algebra behaviour -- "a.T * a" is a scalar and 
so on.

E.g. Sage has a Vector type with the mathematical meaning of the word 
vector. Cython code in Sage dealing with efficient numerical linear 
algebra could probably look quite horrible as a result if the word 
vector was chosen.

In every other programming language I know about with componentwise 
operations, the word "array" is used. "vector" is something you Coder 
People try to force down on us Numericals :-)

(I know that the word is used like this in CS circles, but I prefer only 
using the word in contexts where the meaning is crystal clear (as in 
"vectorized operations") myself.)

-- 
Dag Sverre

From ondrej at certik.cz  Thu Jun 11 21:07:50 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Jun 2009 13:07:50 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
	<4A314FB0.9090507@behnel.de>
	<85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>
Message-ID: <85b5c3130906111207s44d5a715qfe91081566a0b5c2@mail.gmail.com>

On Thu, Jun 11, 2009 at 1:02 PM, Ondrej Certik<ondrej at certik.cz> wrote:
> On Thu, Jun 11, 2009 at 12:40 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>>
>> Lisandro Dalcin wrote:
>>> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz> wrote:
>>>> As far as Cython is concerned, the only problem I can see is that it
>>>> changes all import paths to the fully qualified import paths. Python
>>>> allows both ways, Cython only the fully qualified way. As Stefan
>>>> explained, this is needed, because it will then not work properly at
>>>> the moment. But I am not convinced this cannot be fixed -- though it
>>>> may happen that the fix is too tedious and not worthy.
>>>>
>>>
>>> IMHO, the only "fix" Cython needs is support for relative imports...
>>> Then you could write:
>>>
>>> from ._hermes2d cimport what_you_need
>>>
>>> This way, Ondrej would be able to easily write Cython code using
>>> relative imports the right way...
>>
>> I thought about this, too, and yes, that would be a nice feature. But it
>> doesn't help in this case. What Ondrej wanted was to be able to import the
>> module both from its package and as a plain import, i.e. when put outside
>> of any package. The above would not handle the latter case either.
>
> I think it would. I have _forms.so and _hermes2d.so. The _hermes2d.so
> is imported from _forms.so and if the import was relative, e.g. if
> _forms.so was doing "import ._hermes2d", then it should work both in
> the tree and out of the tree.

Looking here, it should be easy to support relative imports:

http://docs.python.org/c-api/import.html

so unless there is some difficulty inside Cython, that I don't see yet
(e.g. the kind of thing Stefan was talking about), it should be
possible to support relative imports quite easily, I already
understand the part that takes the import path from .pyx file and then
makes it fully qualified --- so I would simply change that.

Ondrej

From stefan_ml at behnel.de  Thu Jun 11 21:18:26 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 21:18:26 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3155E3.908@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>
	<4A3155E3.908@student.matnat.uio.no>
Message-ID: <4A315882.4030305@behnel.de>


Dag Sverre Seljebotn wrote:
> I never keep a promise to shut up...
> 
> Stefan Behnel wrote:
>> Honestly, what's so bad about making the vector type explicit? Even
>>
>> 	Vector[int[:]]
>> or
>> 	Vector(int[:])
>>
>> are better than a hard to read "cwise".
> 
> Being very concrete, the word Vector is not something I normally use in 
> this situation and I would really prefer to keep it that way. If a is a 
> "vector", I expect linear algebra behaviour -- "a.T * a" is a scalar and 
> so on.
> 
> In every other programming language I know about with componentwise 
> operations, the word "array" is used. "vector" is something you Coder 
> People try to force down on us Numericals :-)
> 
> (I know that the word is used like this in CS circles, but I prefer only 
> using the word in contexts where the meaning is crystal clear (as in 
> "vectorized operations") myself.)

Ah, so all you really want is a SIMD data type, right? And I guess that's
how things work in Fortran?

Stefan

From stefan_ml at behnel.de  Thu Jun 11 21:27:57 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Jun 2009 21:27:57 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315882.4030305@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de>
Message-ID: <4A315ABD.3020806@behnel.de>


Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> In every other programming language I know about with componentwise 
>> operations, the word "array" is used. "vector" is something you Coder 
>> People try to force down on us Numericals :-)
>>
>> (I know that the word is used like this in CS circles, but I prefer only 
>> using the word in contexts where the meaning is crystal clear (as in 
>> "vectorized operations") myself.)
> 
> Ah, so all you really want is a SIMD data type, right?

... which would then also support parallel operations as in

	some_int_array *= 2

to multiply all entries by two, and

	some_float_array = math.sqrt(some_int_array)

to take the square-root of all entries in parallel?

Stefan


From dalcinl at gmail.com  Thu Jun 11 21:29:49 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 11 Jun 2009 16:29:49 -0300
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315882.4030305@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A312B2E.5020602@behnel.de> <4A31319C.9090309@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de> <4A313E26.7080006@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de> <4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de>
Message-ID: <e7ba66e40906111229o14a2342bxb7ab92b5c740693d@mail.gmail.com>

On Thu, Jun 11, 2009 at 4:18 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
>
> Ah, so all you really want is a SIMD data type, right?
>

Yes, and not only Dag, but almost all people doing numerics :-)

>
> And I guess that's
> how things work in Fortran?
>

Yes, and not only Fortran, but almost all APL langs out there:
Matlab/Octave/Scilab, R, IDL, D (though it uses c[]=a[]+b[])



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Jun 11 22:01:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 22:01:35 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A315ABD.3020806@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>
	<4A315ABD.3020806@behnel.de>
Message-ID: <4A31629F.8000608@student.matnat.uio.no>

Stefan Behnel wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> In every other programming language I know about with componentwise 
>>> operations, the word "array" is used. "vector" is something you Coder 
>>> People try to force down on us Numericals :-)
>>>
>>> (I know that the word is used like this in CS circles, but I prefer only 
>>> using the word in contexts where the meaning is crystal clear (as in 
>>> "vectorized operations") myself.)
>> Ah, so all you really want is a SIMD data type, right?

Right :-) Now linear algebra support in the compiler *would* be taking 
things quite overboard (and probably only slow things down, unless N is 
  really small, like 4x4 3D game matrices...), I'm actually against that.

> ... which would then also support parallel operations as in
> 
> 	some_int_array *= 2
> 
> to multiply all entries by two, and

That's actually an example in the CEP :-)

> 	some_float_array = math.sqrt(some_int_array)
> 
> to take the square-root of all entries in parallel?

Well, the issue here is function namespaces -- do we "misuse" the Python 
library like this? But the idea -- certainly. At least it should be easy 
to call C stdlib's sin in parallell on a double array.

And it should be possible to create custom functions to call in 
parallel, either by

cdef extern from "stdlib.h":
     double sin(double)
cdef myfunc(double x): ...
myfunc(sin(arr)) # SIMD, by convention, as we know they
                  # take single element

or, if people think that is too magic, by a decorator:

cdef extern from "stdlib.h":
     @cython.array.elementwise
     double sin(double)
@cython.array.elementwise
cdef myfunc(double x): ...

I suggest we deal with this particular issue much later though -- for 
now it's nice to have a roadmap and figure out the syntax for 
concatenateable vs. SIMD, but any actual work on componentwise 
operations will happen after everything else works.

-- 
Dag Sverre

From roland at utk.edu  Thu Jun 11 23:01:43 2009
From: roland at utk.edu (Roland Schulz)
Date: Thu, 11 Jun 2009 17:01:43 -0400
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31629F.8000608@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A3139AC.2050007@behnel.de> 
	<4A313E26.7080006@student.matnat.uio.no> <4A31473C.8020307@behnel.de> 
	<4A314E72.6000500@student.matnat.uio.no> <4A315250.7080606@behnel.de> 
	<4A3155E3.908@student.matnat.uio.no> <4A315882.4030305@behnel.de> 
	<4A315ABD.3020806@behnel.de> <4A31629F.8000608@student.matnat.uio.no>
Message-ID: <c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>

On Thu, Jun 11, 2009 at 4:01 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Stefan Behnel wrote:
> > Stefan Behnel wrote:
> >> Dag Sverre Seljebotn wrote:
> >>> In every other programming language I know about with componentwise
> >>> operations, the word "array" is used. "vector" is something you Coder
> >>> People try to force down on us Numericals :-)
> >>>
> >>> (I know that the word is used like this in CS circles, but I prefer
> only
> >>> using the word in contexts where the meaning is crystal clear (as in
> >>> "vectorized operations") myself.)
> >> Ah, so all you really want is a SIMD data type, right?
>
> Right :-) Now linear algebra support in the compiler *would* be taking
> things quite overboard (and probably only slow things down, unless N is
>  really small, like 4x4 3D game matrices...), I'm actually against that.
>


Well only if you do it directly in a hard-coded datatype.

Why not having a non numeric base class as Stefan proposed. And then
allowing the operator functions be handled by Eigen/Blitz. This would enable
very fast numerics, including SSE and fast linear algebra. And without
having it as tightly coupled in the main language. Similar in that expect to
weave, but with all the advantages of cython.

Component wise operations without optimization (thus collapsing d=a*b*c*d
into one loop instead of 3 and not using temporary arrays) does not give you
any speed-up over Numpy for vectorized code with large arrays.

For vectorized Numpy code the bottleneck is not the call from Python to C,
but the inefficient use of cache because of the temporary arrays.

And I don't think you need a full metalanguage to map the array class to
Eigen/Blitz. Though it would be still be couple somewhat to the main
language otherwise I agree you would need a metalanguage or full template
support. But it seems to me that there should be an optimum between full
template support and doing the numerics directly in a language specific
numeric array type. Sorry for being vage in this central point. I just don't
know the cython details here.

For the naming: why not calling it ndarray, nparray or narray? Even numpy
has two array types to allow different operator syntax. Hardcoding one into
the default array won't allow that in the future.

Roland



-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090611/cfcc7a41/attachment-0001.htm 

From dagss at student.matnat.uio.no  Thu Jun 11 23:22:43 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 23:22:43 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>
	<4A313E26.7080006@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
Message-ID: <4A3175A3.8010304@student.matnat.uio.no>

Thanks for your input!, you definitely know more about such computations 
than me.

Roland Schulz wrote:
> 
> 
> On Thu, Jun 11, 2009 at 4:01 PM, Dag Sverre Seljebotn 
> <dagss at student.matnat.uio.no <mailto:dagss at student.matnat.uio.no>> wrote:

>     Right :-) Now linear algebra support in the compiler *would* be taking
>     things quite overboard (and probably only slow things down, unless N is
>      really small, like 4x4 3D game matrices...), I'm actually against that.
> 
> 
> 
> Well only if you do it directly in a hard-coded datatype.
> 
> Why not having a non numeric base class as Stefan proposed. And then 
> allowing the operator functions be handled by Eigen/Blitz. This would 
> enable very fast numerics, including SSE and fast linear algebra. And 
> without having it as tightly coupled in the main language. Similar in 
> that expect to weave, but with all the advantages of cython.

I'll answer this below.

> 
> Component wise operations without optimization (thus collapsing 
> d=a*b*c*d into one loop instead of 3 and not using temporary arrays) 
> does not give you any speed-up over Numpy for vectorized code with large 
> arrays.
> 
> For vectorized Numpy code the bottleneck is not the call from Python to 
> C, but the inefficient use of cache because of the temporary arrays.

I don't know enough about this, but these two paragraphs seem slightly 
contradictory to me.

Anyway: Any builtin features as such in Cython can't really depend on 
Eigen/Blitz. (I know this is perhaps not what you are suggesting, but 
bear with me.)

As the CEP 517 says, I see this as a two-step process:

1) Naive C loops produced by Cython, which will be "good enough" for 
many cases, and which gives a reference spec and implementation which 
runs out of the box without library requirements.

2) Various plugins (with a seperate release process from Cython so that 
work in this area doesn't bog down Cython development), using e.g. 
Eigen/Blitz as backends, or run it on a GPU, in parallell using OpenMP 
(well, for heavy componentwise functions), etc. etc.

Whether a library could be produced in a mixture of C++ and Cython, 
perhaps adding more Cython features along the way, which implemented 
this directly using Eigen/Blitz instead... perhaps. It seems more 
difficult to get the smooth PEP 3118 integration, but it could perhaps 
be done.

> 
> And I don't think you need a full metalanguage to map the array class to 
> Eigen/Blitz. Though it would be still be couple somewhat to the main 
> language otherwise I agree you would need a metalanguage or full 
> template support. But it seems to me that there should be an optimum 
> between full template support and doing the numerics directly in a 
> language specific numeric array type. Sorry for being vage in this 
> central point. I just don't know the cython details here.

Well, the point here is that when you use Eigen/Blitz, what you do is 
use the metalanguage of C++. Which is much more powerful than anything 
we're likely to get in Cython. So in some sense you use C++ to do what 
you cannot do in Cython.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 11 23:32:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Jun 2009 23:32:41 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3175A3.8010304@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>
	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>
	<4A315ABD.3020806@behnel.de>	<4A31629F.8000608@student.matnat.uio.no>	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
Message-ID: <4A3177F9.5010007@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Thanks for your input!, you definitely know more about such computations 
> than me.
> 
> Roland Schulz wrote:
>> Component wise operations without optimization (thus collapsing 
>> d=a*b*c*d into one loop instead of 3 and not using temporary arrays) 
>> does not give you any speed-up over Numpy for vectorized code with large 
>> arrays.
>>
>> For vectorized Numpy code the bottleneck is not the call from Python to 
>> C, but the inefficient use of cache because of the temporary arrays.
> 
> I don't know enough about this, but these two paragraphs seem slightly 
> contradictory to me.

Or did you mean that

optimization == collapsing d=a*b*c*d into one loop instead of 3 and not 
using temporary arrays

?

It is definitely the plan of CEP 517 that

cdef int[:] a = ..., b = ..., c = ..., d = ...
d = a * b * c * d

turn into something very similar to

cdef size_t tmp1
cdef int[:] tmpresult = new array of right length
for tmp1 in range(a.shape[0]):
     tmpresult[tmp1] = a[tmp1] + b[tmp1] + c[tmp1] + d[tmp1]
d = tmpresult

although broadcasting should be supported and makes it more complicated 
(repeat arrays of length 1, if d has length 1 it must be reallocated -- 
and so on). With multidimensional this becomes more difficult, there's 
lots of ugly details concerning broadcasting and non-contiguous arrays 
(where the "innermost" dimension must be found at runtime...)

IMPORTANT NOTE: All of this is way ahead, all that is the question now 
is a coarse roadmap, and whether this is wanted at all or not.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Jun 12 08:44:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Jun 2009 23:44:53 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
Message-ID: <52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>

If anything, your proposal has at least been interesting enough to  
generate a lot of discussion. Would a fair summary be that you're  
proposing more full-featured wrapping of PEP 3118's bufferinfo struct  
(where all the semantics/operation logic are embedded in the Cython  
compiler and code emitted inline, rather than in an external class  
like an ndarray)? And essentially you want these to act as Fortran  
arrays, a native type in the language.

I am not opposed to this feature, despite it being off the path of  
being a good, optimizing Python compiler with easy glue to external  
code, as long as it doesn't intrude on the language too much. Mostly,  
this is because there is so much interest in using Cython in a  
numerical processing context. I do, however, have some reservations.

First, it sounds a bit like you're proposing to essentially re- 
implement NumPy. Would it just be slicing and arithmetic, or where  
would you draw the line before it doesn't really belong in a  
compiler, but rather a library. More below:

On Jun 10, 2009, at 11:57 AM, Dag Sverre Seljebotn wrote:

> Brian Granger wrote:
>> Dag,
>>
>> I quickly glanced through the proposal and have two big picture  
>> questions:
>>
>> * What will this make possible that is currently not possible?

This was originally my first question too, but you beat me to it.

> 1) Efficient slices

Is the inefficiency just in the object creation and Python indexing  
semantics? It's still O(1), right? Same with the other operations. (I  
guess there's also a question of result type.)

> 2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
> Koepke's project proposal)

Could you provide a bit more of a concrete example here? There is  
avoiding allocation of temporary arrays, is there more?

> 3) Automatic contiguous copies in/out if a function is coded to  
> work on
> contiguous memory
>
> Why can't this be done currently?
>
>  * Cython contains no in-compiler support for NumPy, and so cannot  
> know
> how to create new underlying ndarray objects.
>
>  * All optimizations need to hard-code semantics at compile-time. With
> single-element indexing it seemed fair to assume the usual  
> semantics of
> zero-based indexing etc., but with slices tings get worse (which  
> kind of
> object is returned) and with arithmetic downright impossible (what  
> does *
> do again?)
>
> That's not to say there's not other options:
> 1) We could hard-code support for NumPy only, and only allow  
> ndarray and
> not subclasses thereof.
>
> 2) We could invent some new protocol/syntax for defining compile-time
> semantics for all relevant operations.

Here I am torn--I don't like defining compile-time semantics because  
it goes against the whole OO style of inheritance (and feels even  
more remote than the very dynamic, late-binding Python runtime). I  
don't like option (1) either though.

Another an idea, have you thought of using NumPy as the backend? I.e.  
an int[:,:] is any bufferinfo--supporting object, but if one needs to  
be created you create it via an ndarray? This could (potentially)  
facilitate a lot more code reuse (especially for operations that are  
more complicated than a single loop over the data). (Might be messier  
than implementing int[:,:] directly though.) Suppose one develops a  
vectorized version of ndarrays, could that be a drop-in replacement?

The plug-in idea is very interesting, both from the perspective that  
it would allow one to play with different ways to operate on arrays,  
and also it could separate some of the array-processing logic out of  
the core compiler itself.

- Robert


P.S. In terms of componentwise operations vs. concatenate, I think a  
raw list type would be very useful too. int[:] has the advantage that  
it is easy to encode dimension and other information relevant to  
arrays. Memory-managed lists of ints, doubles, structs, etc. would be  
something I would very much like to see added, but via another syntax  
(e.g. [int] or int[]). Using the keyword "vector" or "array" is IMHO  
a bad idea, as it looks like an ordinary type, and bars a user from  
actually using that keyword as well. Syntax via punctuation makes it  
clear that something non-Pythonic is going on here.


P.P.S. Were you actually suggesting that when I have

cdef struct X:
     int a
     double b

I could do

cdef X x = ..., y = ..., z
z = x+y

?

From dagss at student.matnat.uio.no  Fri Jun 12 09:30:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 09:30:50 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
	<52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>
Message-ID: <4A32042A.20709@student.matnat.uio.no>

A quick answer to some of the points.

Robert Bradshaw wrote:
> First, it sounds a bit like you're proposing to essentially re- 
> implement NumPy. Would it just be slicing and arithmetic, or where  
> would you draw the line before it doesn't really belong in a  
> compiler, but rather a library. More below:

At this point, I should stress that what I'm after is a roadmap (how do 
I deal with requests coming up, and things like Hoyt's proposed project 
when that came up). I won't have time for doing more than the basic 
array stuff (to better facilitate ourself of Kurt's work) this time 
around -- which is still very useful for exchanging array data with 
C/Fortran code.

It would be "reimplementing" NumPy as far as the core array API goes, 
but the implementation would be totally different as we have a 
compilation stage. If anything, it is reimplementing Fortran, however 
Fortran desperately need more alternatives :-)

As for the library, the CEP scetches full interopability with the 
library, you would likely still do

arr = np.dot(arr,arr)

for matrix mul.

What to include? I think slicing, arithmetic (with "broadcasting", i.e 
dimensions of length 1 are repeated to make arrays conform), transpose 
(arr.T). No "sum()" member function or similar.

In addition perhaps high-level utilities for looping, such as 
"ndenumerate" etc:

cdef int[:,:] arr = ...
for (i, j), value in cython.array.ndenumerate(arr):
     ...

These would be quite nicely contained in cython.array though (and again 
have no direct NumPy link that's not horribly slow due to Python-level 
iterators).


> 
> On Jun 10, 2009, at 11:57 AM, Dag Sverre Seljebotn wrote:
> 
>> Brian Granger wrote:
>>> Dag,
>>>
>>> I quickly glanced through the proposal and have two big picture  
>>> questions:
>>>
>>> * What will this make possible that is currently not possible?
> 
> This was originally my first question too, but you beat me to it.
> 
>> 1) Efficient slices
> 
> Is the inefficiency just in the object creation and Python indexing  
> semantics? It's still O(1), right? Same with the other operations. (I  
> guess there's also a question of result type.)

Well, and then the resulting buffer is acquired and checked for data 
type. But essentially, yes.

This is purely a matter of notational convenience -- will it hurt to 
have that O(1) in your almost-inner-loop taking a slice of 50 elements, 
or do you pass "start", "end" around in your function?

>> 2) Leave the road open for memory bus friendlier arithmetic (re: Hoyt
>> Koepke's project proposal)
> 
> Could you provide a bit more of a concrete example here? There is  
> avoiding allocation of temporary arrays, is there more?

Most systems today are for simple functions like addition constrained by 
memory bus speed, not CPU speed. Consider

B = A + A + A + A + A + A

With NumPy, the data of A has to travel over the bus 6 times, with loop 
unrolling only 1, as it would turn into

B = new memory
for idx in ...
    B[idx] = A[idx] + A[idx] + ...

This can mean a dramatic speed increase as data doesn't have to enter 
the cache more than once.

(NumPy can't do this, but there is numexpr, which allow this:

B = numexpr.eval("A+A+...")

using bytecode interpreter and working in cache-sized blocks.)

>> 3) Automatic contiguous copies in/out if a function is coded to  
>> work on
>> contiguous memory
>>
>> Why can't this be done currently?
>>
>>  * Cython contains no in-compiler support for NumPy, and so cannot  
>> know
>> how to create new underlying ndarray objects.
>>
>>  * All optimizations need to hard-code semantics at compile-time. With
>> single-element indexing it seemed fair to assume the usual  
>> semantics of
>> zero-based indexing etc., but with slices tings get worse (which  
>> kind of
>> object is returned) and with arithmetic downright impossible (what  
>> does *
>> do again?)
>>
>> That's not to say there's not other options:
>> 1) We could hard-code support for NumPy only, and only allow  
>> ndarray and
>> not subclasses thereof.
>>
>> 2) We could invent some new protocol/syntax for defining compile-time
>> semantics for all relevant operations.
> 
> Here I am torn--I don't like defining compile-time semantics because  
> it goes against the whole OO style of inheritance (and feels even  
> more remote than the very dynamic, late-binding Python runtime). I  
> don't like option (1) either though.
> 
> Another an idea, have you thought of using NumPy as the backend? I.e.  
> an int[:,:] is any bufferinfo--supporting object, but if one needs to  
> be created you create it via an ndarray? This could (potentially)  
> facilitate a lot more code reuse (especially for operations that are  
> more complicated than a single loop over the data). (Might be messier  
> than implementing int[:,:] directly though.) Suppose one develops a  
> vectorized version of ndarrays, could that be a drop-in replacement?

? ndarrays are vectorized? But subclasses may not be.

I've thought about it. If arithmetic is implemented on arrays, the only 
thing that's seems to be reused this way is malloc/free of a memory area 
though. Better loose the dependency then.

> The plug-in idea is very interesting, both from the perspective that  
> it would allow one to play with different ways to operate on arrays,  
> and also it could separate some of the array-processing logic out of  
> the core compiler itself.

I can definitely see NumPy as being the first such plugin, i.e. it would 
force each array to exactly ndarray on arithmetic, to at least provide 
some arithmetic support right away.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Jun 12 10:02:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 12 Jun 2009 01:02:28 -0700
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A32042A.20709@student.matnat.uio.no>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>
	<52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>
	<4A32042A.20709@student.matnat.uio.no>
Message-ID: <5FFC2397-4769-49E3-9E00-04D22DDC4E1F@math.washington.edu>

On Jun 12, 2009, at 12:30 AM, Dag Sverre Seljebotn wrote:

> A quick answer to some of the points.
>
> Robert Bradshaw wrote:
>> First, it sounds a bit like you're proposing to essentially re-
>> implement NumPy. Would it just be slicing and arithmetic, or where
>> would you draw the line before it doesn't really belong in a
>> compiler, but rather a library. More below:
>
> At this point, I should stress that what I'm after is a roadmap  
> (how do
> I deal with requests coming up, and things like Hoyt's proposed  
> project
> when that came up). I won't have time for doing more than the basic
> array stuff (to better facilitate ourself of Kurt's work) this time
> around -- which is still very useful for exchanging array data with
> C/Fortran code.
>
> It would be "reimplementing" NumPy as far as the core array API goes,
> but the implementation would be totally different as we have a
> compilation stage. If anything, it is reimplementing Fortran, however
> Fortran desperately need more alternatives :-)
>
> As for the library, the CEP scetches full interopability with the
> library, you would likely still do
>
> arr = np.dot(arr,arr)
>
> for matrix mul.
>
> What to include? I think slicing, arithmetic (with "broadcasting", i.e
> dimensions of length 1 are repeated to make arrays conform), transpose
> (arr.T). No "sum()" member function or similar.

That sound reasonable.

> In addition perhaps high-level utilities for looping, such as
> "ndenumerate" etc:
>
> cdef int[:,:] arr = ...
> for (i, j), value in cython.array.ndenumerate(arr):
>      ...
>
> These would be quite nicely contained in cython.array though (and  
> again
> have no direct NumPy link that's not horribly slow due to Python-level
> iterators).

Something like this could certainly benefit from compile-time  
optimizations when the dimensions are fixed.

>
>
>>
>> On Jun 10, 2009, at 11:57 AM, Dag Sverre Seljebotn wrote:
>>
>>> Brian Granger wrote:
>>>> Dag,
>>>>
>>>> I quickly glanced through the proposal and have two big picture
>>>> questions:
>>>>
>>>> * What will this make possible that is currently not possible?
>>
>> This was originally my first question too, but you beat me to it.
>>
>>> 1) Efficient slices
>>
>> Is the inefficiency just in the object creation and Python indexing
>> semantics? It's still O(1), right? Same with the other operations. (I
>> guess there's also a question of result type.)
>
> Well, and then the resulting buffer is acquired and checked for data
> type. But essentially, yes.
>
> This is purely a matter of notational convenience -- will it hurt to
> have that O(1) in your almost-inner-loop taking a slice of 50  
> elements,
> or do you pass "start", "end" around in your function?

Your inner loop would typically be over largish arrays, so it  
probably wouldn't matter much, but I see your point.

>>> 2) Leave the road open for memory bus friendlier arithmetic (re:  
>>> Hoyt
>>> Koepke's project proposal)
>>
>> Could you provide a bit more of a concrete example here? There is
>> avoiding allocation of temporary arrays, is there more?
>
> Most systems today are for simple functions like addition  
> constrained by
> memory bus speed, not CPU speed.

Yep.

> Consider
>
> B = A + A + A + A + A + A
>
> With NumPy, the data of A has to travel over the bus 6 times, with  
> loop
> unrolling only 1, as it would turn into
>
> B = new memory
> for idx in ...
>     B[idx] = A[idx] + A[idx] + ...
>
> This can mean a dramatic speed increase as data doesn't have to enter
> the cache more than once.

Well, it's unlikely that you would have 6 copies of the same data to  
add, but it would save on store/load in the target array. The ability  
to combine multiple operations into a single pass is a powerful one.

> (NumPy can't do this, but there is numexpr, which allow this:
>
> B = numexpr.eval("A+A+...")
>
> using bytecode interpreter and working in cache-sized blocks.)

Didn't know about that, that's kind of neat. Have you looked at how  
it works for ideas?

>>> 3) Automatic contiguous copies in/out if a function is coded to
>>> work on
>>> contiguous memory
>>>
>>> Why can't this be done currently?
>>>
>>>  * Cython contains no in-compiler support for NumPy, and so cannot
>>> know
>>> how to create new underlying ndarray objects.
>>>
>>>  * All optimizations need to hard-code semantics at compile-time.  
>>> With
>>> single-element indexing it seemed fair to assume the usual
>>> semantics of
>>> zero-based indexing etc., but with slices tings get worse (which
>>> kind of
>>> object is returned) and with arithmetic downright impossible (what
>>> does *
>>> do again?)
>>>
>>> That's not to say there's not other options:
>>> 1) We could hard-code support for NumPy only, and only allow
>>> ndarray and
>>> not subclasses thereof.
>>>
>>> 2) We could invent some new protocol/syntax for defining compile- 
>>> time
>>> semantics for all relevant operations.
>>
>> Here I am torn--I don't like defining compile-time semantics because
>> it goes against the whole OO style of inheritance (and feels even
>> more remote than the very dynamic, late-binding Python runtime). I
>> don't like option (1) either though.
>>
>> Another an idea, have you thought of using NumPy as the backend? I.e.
>> an int[:,:] is any bufferinfo--supporting object, but if one needs to
>> be created you create it via an ndarray? This could (potentially)
>> facilitate a lot more code reuse (especially for operations that are
>> more complicated than a single loop over the data). (Might be messier
>> than implementing int[:,:] directly though.) Suppose one develops a
>> vectorized version of ndarrays, could that be a drop-in replacement?
>
> ? ndarrays are vectorized? But subclasses may not be.

I'm just thinking about taking advantage of improvements/ 
optimizations from ndarrays. But a lot of the stuff you're talking  
about is clearly and optimally O(n) so this is less of an issue  
(especially if you make it pluggable to support future parallelization).

> I've thought about it. If arithmetic is implemented on arrays, the  
> only
> thing that's seems to be reused this way is malloc/free of a memory  
> area
> though. Better loose the dependency then.

OK, makes sense. Wanted to know your thoughts at least.

- Robert


From dagss at student.matnat.uio.no  Fri Jun 12 11:37:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 11:37:28 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <5FFC2397-4769-49E3-9E00-04D22DDC4E1F@math.washington.edu>
References: <ebdab1bd76333f5d0ded0ba77228a525.squirrel@webmail.uio.no>	<52E3FF87-25CB-4F08-B424-422F855AAAF8@math.washington.edu>	<4A32042A.20709@student.matnat.uio.no>
	<5FFC2397-4769-49E3-9E00-04D22DDC4E1F@math.washington.edu>
Message-ID: <4A3221D8.6080000@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 12, 2009, at 12:30 AM, Dag Sverre Seljebotn wrote:
>
>   
>
>> In addition perhaps high-level utilities for looping, such as
>> "ndenumerate" etc:
>>
>> cdef int[:,:] arr = ...
>> for (i, j), value in cython.array.ndenumerate(arr):
>>      ...
>>
>> These would be quite nicely contained in cython.array though (and  
>> again
>> have no direct NumPy link that's not horribly slow due to Python-level
>> iterators).
>>     
>
> Something like this could certainly benefit from compile-time  
> optimizations when the dimensions are fixed.
>   
Actually, high-level iteration can be done quite fast when the ndims are 
not fixed too, but it's a long story (main idea is to have slow code 
extract the dimension with the lowest strides, and then have an inner 
for-loop set up over pointers/increments set up by the slower code). I'm 
thinking of "int[...]" syntax for what, which will allow all operations 
except item indexing.

>> Consider
>>
>> B = A + A + A + A + A + A
>>
>> With NumPy, the data of A has to travel over the bus 6 times, with  
>> loop
>> unrolling only 1, as it would turn into
>>
>> B = new memory
>> for idx in ...
>>     B[idx] = A[idx] + A[idx] + ...
>>
>> This can mean a dramatic speed increase as data doesn't have to enter
>> the cache more than once.
>>     
>
> Well, it's unlikely that you would have 6 copies of the same data to  
> add, but it would save on store/load in the target array. The ability  
> to combine multiple operations into a single pass is a powerful one.
>   
I just see that people do use numexpr to get around it.

>> (NumPy can't do this, but there is numexpr, which allow this:
>>
>> B = numexpr.eval("A+A+...")
>>
>> using bytecode interpreter and working in cache-sized blocks.)
>>     
>
> Didn't know about that, that's kind of neat. Have you looked at how  
> it works for ideas?
>   
1) Parses expression tree
2) Repeatedly executes expression tree on ideally-sized sub-chunks of 
the arrays

I.e. it balances array block size (and cache limit) vs. the time it 
takes to execute an expression tree, in order to stay within cache but 
divide the running time of the latter with a big constant factor. With 
Cython there's no need for this as the expression tree is in the format 
the CPU speaks.

Now, it might do SSE etc. as well, that's where I see the plugins, and 
code properly refactored from numexpr could fit in there. Long-term we 
might see the recent NumPy CorePy effort kick in -- CorePy can compile 
code into memory (for use with NumPy) or to assembly text file (for 
linking into Cython code through a plugin).

It would be neat to fold numexpr into Sage BTW :-), i.e.

x, y = var("x, y")
integrate(x**2 + y**2 + x, x)(somenumpyarray)

and run that through numexpr for more efficient evaluation. If I'm to do 
anything myself, I'm as likely to contribute that /before/ working on 
arithmetic in Cython *shrug*.

But even if dynamic expressions in Sage becomes a major mode of working 
with numerics, I still think there's always going to be room for Cython 
to provide honest, no-frills compiled code. And it is anyway likely 
needed to "embrace and extend" Fortran workflow that people are used to.

I wanted to ask you about fasteval in Sage -- I think it does something 
similar (with other types), but I can't seem to find any docs on it? 
Which source file should I look in?

Dag Sverre

From roland at utk.edu  Fri Jun 12 16:11:24 2009
From: roland at utk.edu (Roland Schulz)
Date: Fri, 12 Jun 2009 10:11:24 -0400
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3175A3.8010304@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de> 
	<4A314E72.6000500@student.matnat.uio.no> <4A315250.7080606@behnel.de> 
	<4A3155E3.908@student.matnat.uio.no> <4A315882.4030305@behnel.de> 
	<4A315ABD.3020806@behnel.de> <4A31629F.8000608@student.matnat.uio.no> 
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com> 
	<4A3175A3.8010304@student.matnat.uio.no>
Message-ID: <c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>

On Thu, Jun 11, 2009 at 5:22 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Thanks for your input!, you definitely know more about such computations
> than me.
>
> Roland Schulz wrote:
> >
> >
> > On Thu, Jun 11, 2009 at 4:01 PM, Dag Sverre Seljebotn
> > <dagss at student.matnat.uio.no <mailto:dagss at student.matnat.uio.no>>
> wrote:
>
> >     Right :-) Now linear algebra support in the compiler *would* be
> taking
> >     things quite overboard (and probably only slow things down, unless N
> is
> >      really small, like 4x4 3D game matrices...), I'm actually against
> that.
> >
> >
> >
> > Well only if you do it directly in a hard-coded datatype.
> >
> > Why not having a non numeric base class as Stefan proposed. And then
> > allowing the operator functions be handled by Eigen/Blitz. This would
> > enable very fast numerics, including SSE and fast linear algebra. And
> > without having it as tightly coupled in the main language. Similar in
> > that expect to weave, but with all the advantages of cython.
>
> I'll answer this below.
>
> >
> > Component wise operations without optimization (thus collapsing
> > d=a*b*c*d into one loop instead of 3 and not using temporary arrays)
> > does not give you any speed-up over Numpy for vectorized code with large
> > arrays.
> >
> > For vectorized Numpy code the bottleneck is not the call from Python to
> > C, but the inefficient use of cache because of the temporary arrays.
>
> I don't know enough about this, but these two paragraphs seem slightly
> contradictory to me.


I think it is important to see which features of this proposals are possible
but will only be included in the unforeseeable (=may be never) future and
what could/will be done soon.
Of course this depends on how much time there is so it is difficult to say.

By point is, if collapsing d=a*b*c*d into one loop is possible, but is not
clear when it will be added, this proposal won't speed up calculations over
vectorized numpy for the foreseeable furture.

Or putting it another way: Introducing only element-wise access won't help
performance by itself, so performance wise it is not a meaningful
intermediate step.

Also this optimization is not straight forward. If you look at
http://eigen.tuxfamily.org/index.php?title=Benchmark-August2008

you see that many libraries trying to do even simple Y+=alpha X do a bad job
performance wise.
And having to do all the required optimization in the cython compiler would
basicly mean rewriting the Frotran Compiler. And of course the language is
suboptimal but the compiler can be sometimes rather smart. And putting all
this in the Cython compiler sounds like awful a lot of work.


>
> Anyway: Any builtin features as such in Cython can't really depend on
> Eigen/Blitz. (I know this is perhaps not what you are suggesting, but
> bear with me.)
>
> As the CEP 517 says, I see this as a two-step process:
>
> 1) Naive C loops produced by Cython, which will be "good enough" for
> many cases, and which gives a reference spec and implementation which
> runs out of the box without library requirements.
>
> 2) Various plugins (with a seperate release process from Cython so that
> work in this area doesn't bog down Cython development), using e.g.
> Eigen/Blitz as backends, or run it on a GPU, in parallell using OpenMP
> (well, for heavy componentwise functions), etc. etc.


Why does it bog down Cython delvelopment if it is a library with ships with
cython? Also you can always require certain version of the external
components (and make it optional for complication for people not interested
in numerics).



>
> >
> > And I don't think you need a full metalanguage to map the array class to
> > Eigen/Blitz. Though it would be still be couple somewhat to the main
> > language otherwise I agree you would need a metalanguage or full
> > template support. But it seems to me that there should be an optimum
> > between full template support and doing the numerics directly in a
> > language specific numeric array type. Sorry for being vage in this
> > central point. I just don't know the cython details here.
>
> Well, the point here is that when you use Eigen/Blitz, what you do is
> use the metalanguage of C++. Which is much more powerful than anything
> we're likely to get in Cython. So in some sense you use C++ to do what
> you cannot do in Cython.


Exactly you have rather easy usage of Eigen/Blitz and only that code has to
be generated by Cython. Then the C++ compiler takes care of the expression
templates.

Roland


-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090612/ae40cdf2/attachment.htm 

From dagss at student.matnat.uio.no  Fri Jun 12 17:59:21 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 17:59:21 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
	<c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
Message-ID: <4A327B59.7010406@student.matnat.uio.no>

I think I agree more with you than you give me credit for...

Roland Schulz wrote:
> I think it is important to see which features of this proposals are 
> possible but will only be included in the unforeseeable (=may be never) 
> future and what could/will be done soon.
> Of course this depends on how much time there is so it is difficult to say.

OK I'll by crystal clear about this: In autumn,

    cdef int[:] a = ..., b = ...
    a = a + b

will be a syntax error. The main motivation for adding this type now is 
to make it easier to pass array data between Python objects (PEP 3118) 
and external libraries.

However, adding a new, native type is not something you do without 
thinking things through, and discussing possibilities for arithmetic 
operators was important to get the whole picture of "what kind of type" 
this is and what design and implementation considerations to take.

Also there was the issue of the whole direction of Cython in the area of 
numeric programming which has now largely been resolved. This also means 
that

cdef ndarray[int] a = ..., b = ...
a = a + b

will never be optimized -- because it is vital that such code allows 
arbitrary subclasses of ndarray to overload the operators.

> By point is, if collapsing d=a*b*c*d into one loop is possible, but is 
> not clear when it will be added, this proposal won't speed up 
> calculations over vectorized numpy for the foreseeable furture.
> 
> Or putting it another way: Introducing only element-wise access won't 
> help performance by itself, so performance wise it is not a meaningful 
> intermediate step.

I agree, what it is needed for is more convenient "raw" handling of PEP 
3118 without invoking NumPy to do it (and NumPy doesn't handle all the 
memory layouts that PEP 3118 handle).

BUT: The code needed to e.g. pass Python Imaging Library images to 
Fortran code is 50% of the way towards what it takes to implement naive 
componentwise operators.


> Also this optimization is not straight forward. If you look at
> http://eigen.tuxfamily.org/index.php?title=Benchmark-August2008
> 
> you see that many libraries trying to do even simple Y+=alpha X do a bad 
> job performance wise.
> And having to do all the required optimization in the cython compiler 
> would basicly mean rewriting the Frotran Compiler. And of course the 
> language is suboptimal but the compiler can be sometimes rather smart. 
> And putting all this in the Cython compiler sounds like awful a lot of work.

I completely agree!! -- I don't see Cython doing the optimizations 
itself, but make use of external libraries (like eigen) through plugins. 
The question is how this is facilitated!

I.e. you could probably define type after the C++ integration is more 
mature that allows you to do

     cdef eigen.vector a = ..., b = ... # no pun intended
     a = a + b # faster than anything else

but can you transparently convert those to and from PIL images, or pass 
them to a SciPy/NumPy function?

That's the issues that are really at stake here.

>     2) Various plugins (with a seperate release process from Cython so that
>     work in this area doesn't bog down Cython development), using e.g.
>     Eigen/Blitz as backends, or run it on a GPU, in parallell using OpenMP
>     (well, for heavy componentwise functions), etc. etc.
> 
> 
> Why does it bog down Cython delvelopment if it is a library with ships 
> with cython? Also you can always require certain version of the external 
> components (and make it optional for complication for people not 
> interested in numerics).

What I meant to say is that we must avoid situations like this:

Dag Sverre, year 2010: "Please wait with releasing Cython 1.3 until I'm 
done supporting the next version of Eigen"

So plugins have "their own release process" in the sense that Cython can 
be released at any time shipping a previous, stable version of the plugin.

But this is premature discussion ... :-)


>     Well, the point here is that when you use Eigen/Blitz, what you do is
>     use the metalanguage of C++. Which is much more powerful than anything
>     we're likely to get in Cython. So in some sense you use C++ to do what
>     you cannot do in Cython.
> 
> 
> Exactly you have rather easy usage of Eigen/Blitz and only that code has 
> to be generated by Cython. Then the C++ compiler takes care of the 
> expression templates.

Generating the code to use Eigen/Blitz and generating naive loops are 
probably about the same amount of work, and probably something like 70% 
overlap. I see no reason for not doing both, if only to avoid having to 
say "you need C++ to use this feature of the language".

Really, neither are rocket science, but all the "buffer interopability 
infrastructure" needs to be there first.

-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Jun 12 17:31:15 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Jun 2009 17:31:15 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A31629F.8000608@student.matnat.uio.no>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>	<4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
Message-ID: <4A3274C3.1050204@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>>> Ah, so all you really want is a SIMD data type, right?
> 
> Right :-) Now linear algebra support in the compiler *would* be taking 
> things quite overboard (and probably only slow things down, unless N is 
>   really small, like 4x4 3D game matrices...), I'm actually against that.
> 
>> ... which would then also support parallel operations as in
>>
>> 	some_int_array *= 2
>>
>> to multiply all entries by two, and
> 
> That's actually an example in the CEP :-)

Yes, and I skipped over it because that was a tiny section titled
"numerical calculations". It was far from obvious to me that that was the
main point of the proposal.


>> 	some_float_array = math.sqrt(some_int_array)
>>
>> to take the square-root of all entries in parallel?
> 
> Well, the issue here is function namespaces -- do we "misuse" the Python 
> library like this?

I was more thinking of math.h here, i.e. C functions.


> But the idea -- certainly. At least it should be easy 
> to call C stdlib's sin in parallell on a double array.
> 
> And it should be possible to create custom functions to call in 
> parallel, either by
> 
> cdef extern from "stdlib.h":
>      double sin(double)
> cdef myfunc(double x): ...
> myfunc(sin(arr)) # SIMD, by convention, as we know they
>                   # take single element

Are you sure you always know where the parallelism ends?


> or, if people think that is too magic, by a decorator:
> 
> cdef extern from "stdlib.h":
>      @cython.array.elementwise
>      double sin(double)
> @cython.array.elementwise
> cdef myfunc(double x): ...

Not if we can avoid it in any way.


> I suggest we deal with this particular issue much later though -- for 
> now it's nice to have a roadmap and figure out the syntax for 
> concatenateable vs. SIMD, but any actual work on componentwise 
> operations will happen after everything else works.

I don't have the impression that this proposal is accepted yet.

Stefan


From dagss at student.matnat.uio.no  Fri Jun 12 19:55:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Jun 2009 19:55:50 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3274C3.1050204@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>	<4A315ABD.3020806@behnel.de>	<4A31629F.8000608@student.matnat.uio.no>
	<4A3274C3.1050204@behnel.de>
Message-ID: <4A3296A6.4030404@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>>> Ah, so all you really want is a SIMD data type, right?
>> Right :-) Now linear algebra support in the compiler *would* be taking 
>> things quite overboard (and probably only slow things down, unless N is 
>>   really small, like 4x4 3D game matrices...), I'm actually against that.
>>
>>> ... which would then also support parallel operations as in
>>>
>>> 	some_int_array *= 2
>>>
>>> to multiply all entries by two, and
>> That's actually an example in the CEP :-)
> 
> Yes, and I skipped over it because that was a tiny section titled
> "numerical calculations". It was far from obvious to me that that was the
> main point of the proposal.

It was intended as a demonstration of what went before. Anyway, 
communication is hard. Sorry.

(When the dust has settled I'll revise the CEP.)

(I wouldn't say the main point of the proposal as such, but it would be 
the main reason why it would make sense to just as well make this a 
builtin feature and not just something in a library, due to 
implementation constraints.)

>> And it should be possible to create custom functions to call in 
>> parallel, either by
>>
>> cdef extern from "stdlib.h":
>>      double sin(double)
>> cdef myfunc(double x): ...
>> myfunc(sin(arr)) # SIMD, by convention, as we know they
>>                   # take single element
> 
> Are you sure you always know where the parallelism ends?

If a function takes "object", you have to assume it means to take the 
whole array and not its elements. I think that is it. (There are pretty 
standard rules for how to deal with multiple arguments etc. too).

>> or, if people think that is too magic, by a decorator:
>>
>> cdef extern from "stdlib.h":
>>      @cython.array.elementwise
>>      double sin(double)
>> @cython.array.elementwise
>> cdef myfunc(double x): ...
> 
> Not if we can avoid it in any way.

Well that's how you'd do it in Python (given the right elementwise 
decorator).

>> I suggest we deal with this particular issue much later though -- for 
>> now it's nice to have a roadmap and figure out the syntax for 
>> concatenateable vs. SIMD, but any actual work on componentwise 
>> operations will happen after everything else works.
> 
> I don't have the impression that this proposal is accepted yet.

I had that impression until now (except for the syntax bit, where you 
wanted to use int[:] for a list-style type. Or are you suggesting to use 
+ for multi-dimensional concatenation as well?)

I guess that was before we cleared up the SIMD misunderstanding though. 
So I'll revise my impression -- obviously, if you say so, there's still 
problems.

Rather than us discussing this I think it might be more constructive 
though if you comment on Robert's first email about this yesterday? He 
tends to put things in a different way from me.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Jun 13 07:33:38 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Jun 2009 07:33:38 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>
	<4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
	<c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
Message-ID: <4A333A32.4040904@behnel.de>


Roland Schulz wrote:
> I think it is important to see which features of this proposals are possible
> but will only be included in the unforeseeable (=may be never) future and
> what could/will be done soon.
> Of course this depends on how much time there is so it is difficult to say.
> 
> By point is, if collapsing d=a*b*c*d into one loop is possible, but is not
> clear when it will be added, this proposal won't speed up calculations over
> vectorized numpy for the foreseeable furture.
> [...]
> Why does it bog down Cython delvelopment if it is a library with ships with
> cython? Also you can always require certain version of the external
> components (and make it optional for complication for people not interested
> in numerics).

I think the main question then is: what would be the minimum required
change to the compiler that would enable providing these things as external
modules?

Stefan

From dagss at student.matnat.uio.no  Sat Jun 13 09:29:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 13 Jun 2009 09:29:44 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A333A32.4040904@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>
	<4A31473C.8020307@behnel.de> <4A314E72.6000500@student.matnat.uio.no>
	<4A315250.7080606@behnel.de> <4A3155E3.908@student.matnat.uio.no>
	<4A315882.4030305@behnel.de> <4A315ABD.3020806@behnel.de>
	<4A31629F.8000608@student.matnat.uio.no>
	<c93c21390906111401y2bff8173mfd7a4313581558f4@mail.gmail.com>
	<4A3175A3.8010304@student.matnat.uio.no>
	<c93c21390906120711m32abb4e7wcc26c6da2dc43bbb@mail.gmail.com>
	<4A333A32.4040904@behnel.de>
Message-ID: <afd1f743de6fa54a009317d1c95bc03a.squirrel@webmail.uio.no>

Stefan Behnel wrote:
> Roland Schulz wrote:
>> I think it is important to see which features of this proposals are
>> possible
>> but will only be included in the unforeseeable (=may be never) future
>> and
>> what could/will be done soon.
>> Of course this depends on how much time there is so it is difficult to
>> say.
>>
>> By point is, if collapsing d=a*b*c*d into one loop is possible, but is
>> not
>> clear when it will be added, this proposal won't speed up calculations
>> over
>> vectorized numpy for the foreseeable furture.
>> [...]
>> Why does it bog down Cython delvelopment if it is a library with ships
>> with
>> cython? Also you can always require certain version of the external
>> components (and make it optional for complication for people not
>> interested
>> in numerics).
>
> I think the main question then is: what would be the minimum required
> change to the compiler that would enable providing these things as
> external
> modules?

Provided that everything in the proposal is implemented in core Cython
(i.e. naive SIMD loops are implemented as a simple fallback mechanism
etc.), and it is done in a way using seperate node types (e.g
SIMDArithmeticNode), I was thinking to just have a pluggable pipeline, and
allow plugins to provide transforms and positions in the pipeline to plug
them in. (Oh, and plugin transforms could insert their own nodes too.)

This would require plugin developers to follow Cython node tree changes
(which one wouldn't promise to keep stable).

I'm not sure whether such a thing would work or whether there will be
requests of the form "can you just change this thing in this node before
it gets to our plugin transform". But I think that's a luxury problem --
if people do develop such plugins, it means more eyes on the Cython
codebase too.

Dag Sverre


From dagss at student.matnat.uio.no  Sat Jun 13 12:40:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 13 Jun 2009 12:40:26 +0200
Subject: [Cython] Proposal: Cython array type (CEP 517)
In-Reply-To: <4A3274C3.1050204@behnel.de>
References: <4A2FE018.5060902@student.matnat.uio.no>	<4A31189E.6000104@behnel.de>	<4A3123BB.5080603@student.matnat.uio.no>	<4A312B2E.5020602@behnel.de>	<4A31319C.9090309@student.matnat.uio.no>	<4A3139AC.2050007@behnel.de>	<4A313E26.7080006@student.matnat.uio.no>	<4A31473C.8020307@behnel.de>	<4A314E72.6000500@student.matnat.uio.no>	<4A315250.7080606@behnel.de>	<4A3155E3.908@student.matnat.uio.no>	<4A315882.4030305@behnel.de>	<4A315ABD.3020806@behnel.de>	<4A31629F.8000608@student.matnat.uio.no>
	<4A3274C3.1050204@behnel.de>
Message-ID: <4A33821A.9020107@student.matnat.uio.no>

Stefan Behnel wrote:
> I don't have the impression that this proposal is accepted yet.

I looked it over and here's way too much in the proposal that didn't 
communicate to people who are not already avid NumPy users. Sorry about 
that.

Within a few days I'll rewrite the CEP (perhaps split it in two), sum up 
the discussion, and we can start over properly then.

-- 
Dag Sverre

From darkgl0w at yahoo.com  Mon Jun 15 12:18:06 2009
From: darkgl0w at yahoo.com (Cristi Constantin)
Date: Mon, 15 Jun 2009 03:18:06 -0700 (PDT)
Subject: [Cython] support for numpy bool type / unicode type
Message-ID: <707613.24072.qm@web52108.mail.re2.yahoo.com>

Good day.
I was following this discussion too.
How about unicode_t ?
I am trying to make the numpy tutorial you have on Cython Wiki, but instead of np.int_t i need numpy.unicode_ type.
Do you have any idea how i could do that?

Check this python code:
import numpy as np
a=np.array([0,1,2],np.unicode_)
print a # array([u'0', u'1', u'2'], dtype='<U1')
a[0]=u'\u2588'
a[1]=u'\u00b6'
a[2]=u'\u2248'
a.astype('I')
# UnicodeEncodeError: 'decimal' codec can't encode character u'\u2588' in position 0: invalid decimal Unicode string
a.astype(np.uint32)
# Same unicode error.

What's the best "cast" i can do to have my Unicode Numpy Array?
Thank you in advance.

--- On Tue, 6/9/09, Robert Bradshaw <robertwb at math.washington.edu> wrote:

From: Robert Bradshaw <robertwb at math.washington.edu>
Subject: Re: [Cython] support for numpy bool type
To: cython-dev at codespeak.net
Date: Tuesday, June 9, 2009, 1:38 AM

On Jun 6, 2009, at 1:22 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jun 6, 2009, at 2:20 AM, Dag Sverre Seljebotn wrote:
>>
>>> Eric Firing wrote:
>>>> Eric Firing wrote:
>>>>> In writing a cython extension to work with numpy masked arrays, I
>>>>> needed to work with the mask, which is dtype('bool').? Therefore I
>>>>> was expecting to be able to use np.bool_t, in analogy to? 
>>>>> np.float_t.
>>>>> Instead I had to use np.int8_t, and cast the input to np.int8 in a
>>>>> python wrapper.? Can bool_t be added to numpy.pxd?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Eric
>>> The reason it is not there is that AFAIK Cython has no support? 
>>> for an
>>> 8-bit boolean type. I.e. you want
>>>
>>> print arr[3] # should print "True", not 1.
>>>
>>> and furthermore you want
>>>
>>> arr[3] = obj # should mean arr[3] = bool(obj)
>>>
>>> Which brings up the questions
>>> 1) I suppose the best thing would be to add support for a "bool"? 
>>> which
>>> would be a C99 _Bool if compiling on C99, and "unsigned char" with
>>> appropriate restrictions otherwise. Thoughts?
>>
>> How would this relate to the current bint type? bint is an int
>> because logical operations in C are ints, and also several operations
>> in the Python/C API (and elsewhere) return true/false values as ints.
>
> Well, bint is an int that has different semantics for conversion to/ 
> from
> Python,

That is exactly what it is.

> and that seems to be needed here as well (e.g. a "bchar", which I
> think is essentially C99 _Bool).

a C99 _Bool can only have two values, anything non-zero is converted? 
to exactly 1.

> What happens if you do "cdef bint value = 4"? Is bool(4) called? 
> resulting
> in 1 being placed in value? I think that's the wanted behaviour? 
> here...

No, it retains the value 4. (This is one of the reasons I didn't call? 
it a bool.)

> Using bint directly seems to be right out though as one must access? 
> the
> data using an 8-bit integer type. (Well, I could hard-code buffers? 
> to cast
> back and forth between "bint" and "char", after the pointer? 
> dereference,
> but that seems very unclean compared to introducing a new type).
>
> Perhaps we should have "char bint", "short bint", "unsigned long long
> bint" and so on :-) Really, the need for a seperate type for bools? 
> is a
> strict Cython feature, because what it affects is Python object
> conversion, so it doesn't hurt IMO that these are not in C. And? 
> size and
> booleanness are orthogonal features. (But, this is a puristic approach
> only, and in practice "bchar" would suffice.)

Actually, that might work well. When I first introduced it, I saw no? 
need to offer various sizes, but if one has arrays of them then it? 
becomes important. And just when we thought we were simplifying the? 
integer type system... :)

- Robert




      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090615/d0929800/attachment.htm 

From dagss at student.matnat.uio.no  Mon Jun 15 16:28:45 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 15 Jun 2009 16:28:45 +0200
Subject: [Cython] support for numpy bool type / unicode type
In-Reply-To: <707613.24072.qm@web52108.mail.re2.yahoo.com>
References: <707613.24072.qm@web52108.mail.re2.yahoo.com>
Message-ID: <4A365A9D.8040101@student.matnat.uio.no>

Cristi Constantin wrote:
> Good day.
> I was following this discussion too.
> How about unicode_t ?
> I am trying to make the numpy tutorial you have on Cython Wiki, but 
> instead of np.int_t i need numpy.unicode_ type.
> Do you have any idea how i could do that?
> 
> Check this python code:
> import numpy as np
> a=np.array([0,1,2],np.unicode_)
> print a # array([u'0', u'1', u'2'], dtype='<U1')
> a[0]=u'\u2588'
> a[1]=u'\u00b6'
> a[2]=u'\u2248'
> a.astype('I')
> # UnicodeEncodeError: 'decimal' codec can't encode character u'\u2588' 
> in position 0: invalid decimal Unicode string
> a.astype(np.uint32)
> # Same unicode error.
> 
> What's the best "cast" i can do to have my Unicode Numpy Array?
> Thank you in advance.

This is very badly supported in Cython, mostly because I have no idea 
what usecases people have for such arrays. Would you care to describe 
your usecase?

But if you want to modify Unicode 1-length strings as an integer type, 
you can do (in Cython)

cdef np.ndarray[some_int_type, cast=True] arr = a

Here, some_int_type needs to be an integer of the same size as 
"a.itemsize" would give you above -- I don't know how NumPy stored 
Unicode strings so I actually can't tell you the right one.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jun 15 18:12:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 15 Jun 2009 18:12:28 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
Message-ID: <4A3672EC.90002@student.matnat.uio.no>

Thanks to everybody who contributed to the discussion on a Cython array 
type last week! Here's a summary to attempt focusing the discussion.

There are now two CEPs:
- CEP 517, array type: http://wiki.cython.org/enhancements/array
- CEP 518, SIMD operations: http://wiki.cython.org/enhancements/simd

I mostly just added a "what does this facilitate"-section is added near 
the beginning of each, and the multidimensional aspect of the arrays has 
been emphasised. No need to reread it.

The story is this:

  a) I can only commit to implementing CEP 517 this time around

  b) The arrays themselves/CEP 517 can without loosing *too* much be 
done as a library (though I would prefer them as builtins myself). 
However SIMD/CEP 518 depends on building the arrays into the language, 
and switching approaches later would mean redoing some stuff. Therefore 
SIMD (and the numerical roadmap in general) keeps popping into the 
discussion.

  c) It would be nice to finally settle a roadmap for numerics in Cython 
(how far will the language be bent for the benefit of numerics), for 
instance to economize on discussions when questions pop up on the 
mailing list (and make plans). These two CEPs would create one such roadmap.

  d) CEP 518 isn't a lot of work (totally doable in a GSoC next 
year...though of course not at the expense of e.g. a full Python 
compatability GSoC or similar).

Key impressions from last week:

  a) Numerical users seem positive about all of this.
  b) Robert Bradshaw said:

I am not opposed to this feature, despite it being off the path of
being a good, optimizing Python compiler with easy glue to external
code, as long as it doesn't intrude on the language too much. Mostly,
this is because there is so much interest in using Cython in a
numerical processing context. I do, however, have some reservations.

[I believe the reservations were all taken care of.]

  c) I'm not sure about Stefan Behnel's current opinions about this.

-- 
Dag Sverre

From daniel.yarlett at gmail.com  Tue Jun 16 03:50:16 2009
From: daniel.yarlett at gmail.com (Daniel Yarlett)
Date: Tue, 16 Jun 2009 02:50:16 +0100
Subject: [Cython] Cython and Numpy
Message-ID: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>

Hi All,

I am porting some code which uses Cython and Numpy from Ubuntu to OS  
X, and have run into some problems getting the Cython compilation  
process working. I am using distutils with the following in my  
setup.py file:

> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
>    cmdclass = {'build_ext': build_ext},
>    ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"])]
> )

neuralRoutines.pyx imports and cimports numpy, amongst other things. I  
don't think the problem lies in there because it runs fine as it is  
under Ubuntu.

When I run "python setup.py build_ext --inplace" I get the following  
error messages:

> Yarletts-Computer:scorelineModels yarlett$ python setup.py build_ext  
> --inplace
> running build_ext
> building 'neuralRoutines' extension
> creating build
> creating build/temp.macosx-10.3-i386-2.6
> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - 
> fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/ 
> Frameworks/Python.framework/Versions/2.6/include/python2.6 -c  
> neuralRoutines.c -o build/temp.macosx-10.3-i386-2.6/neuralRoutines.o
> neuralRoutines.c:136:31: error: numpy/arrayobject.h: No such file or  
> directory
> neuralRoutines.c:136:31: error: numpy/arrayobject.h: No such file or  
> directory
> neuralRoutines.c:659: error: syntax error before  
> ?__pyx_t_5numpy_int8_t?neuralRoutines.c:659: error: syntax error  
> before ?__pyx_t_5numpy_int8_t?
> neuralRoutines.c:659: warning: data definition has no type or  
> storage class
> neuralRoutines.c:661: error: syntax error before  
> ?__pyx_t_5numpy_int16_t?
> neuralRoutines.c:661: warning: data definition has no type or  
> storage class
> neuralRoutines.c:663: error: syntax error before  
> ?__pyx_t_5numpy_int32_t?
> neuralRoutines.c:663: warning: data definition has no type or  
> storage class

It looks like arrayobject.h is not being found, so do I need to modify  
my setup.py file somehow? I'm afraid I'm not sure how to do this, so  
any help would be greatly appreciated.

When I search for arrayobject.h it can be found in the following  
places, amongst others, on my system:

> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.4.0.dev6728-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h

Many thanks,

Dan.

From darkgl0w at yahoo.com  Tue Jun 16 14:06:38 2009
From: darkgl0w at yahoo.com (Cristi Constantin)
Date: Tue, 16 Jun 2009 05:06:38 -0700 (PDT)
Subject: [Cython] Cython and Numpy
Message-ID: <357936.47971.qm@web52101.mail.re2.yahoo.com>

Good day.

You need to copy all .c files from your numpy installation (on windows it's "C:\Python25\Lib\site-packages\numpy\core\include\numpy\") into MingW\Include\numpy.

gcc can't find your arrayobject.h in python directory.

Have a nice day.

--- On Mon, 6/15/09, Daniel Yarlett <daniel.yarlett at gmail.com> wrote:

It looks like arrayobject.h is not being found, so do I need to modify? 
my setup.py file somehow? I'm afraid I'm not sure how to do this, so? 
any help would be greatly appreciated.

When I search for arrayobject.h it can be found in the following? 
places, amongst others, on my system:

> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy/core/include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site- 
> packages/numpy-1.4.0.dev6728-py2.5-macosx-10.3-i386.egg/numpy/core/ 
> include/numpy/arrayobject.h




      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090616/86965c96/attachment.htm 

From brett.calcott at gmail.com  Tue Jun 16 14:15:52 2009
From: brett.calcott at gmail.com (Brett Calcott)
Date: Tue, 16 Jun 2009 22:15:52 +1000
Subject: [Cython] Cython and Numpy
In-Reply-To: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
References: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
Message-ID: <c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>

Hi Daniel,


2009/6/16 Daniel Yarlett <daniel.yarlett at gmail.com>:
> ... SNIP ...
> It looks like arrayobject.h is not being found, so do I need to modify
> my setup.py file somehow? I'm afraid I'm not sure how to do this, so
> any help would be greatly appreciated.


This line in my .bashrc solves the same problem on my Mac

export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include

Or, I think you could add the same path to your definition of the
Extension() in setup.py. I think you can do this:

ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"],
        extra_compile_args=['-Ithe/path/you/want/here'],
)]

... but I haven't checked it.

>
> When I search for arrayobject.h it can be found in the following
> places, amongst others, on my system:
>
>> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
>> packages/numpy/core/include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-
>> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/
>> include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
>> packages/numpy/core/include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
>> packages/numpy-1.3.0.dev6308-py2.5-macosx-10.3-i386.egg/numpy/core/
>> include/numpy/arrayobject.h
>> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-
>> packages/numpy-1.4.0.dev6728-py2.5-macosx-10.3-i386.egg/numpy/core/
>> include/numpy/arrayobject.h

It looks like you've got a few versions installed. You'd best figure out
which one you are actually using, and delete the rest...

Brett

From dagss at student.matnat.uio.no  Tue Jun 16 15:07:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Jun 2009 15:07:48 +0200
Subject: [Cython] Cython and Numpy
In-Reply-To: <c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>
References: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
	<c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>
Message-ID: <4A379924.8040005@student.matnat.uio.no>

Brett Calcott wrote:
> Hi Daniel,
> 
> 
> 2009/6/16 Daniel Yarlett <daniel.yarlett at gmail.com>:
>> ... SNIP ...
>> It looks like arrayobject.h is not being found, so do I need to modify
>> my setup.py file somehow? I'm afraid I'm not sure how to do this, so
>> any help would be greatly appreciated.
> 
> 
> This line in my .bashrc solves the same problem on my Mac
> 
> export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include
> 
> Or, I think you could add the same path to your definition of the
> Extension() in setup.py. I think you can do this:
> 
> ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"],
>         extra_compile_args=['-Ithe/path/you/want/here'],
> )]
> 
> ... but I haven't checked it.

The usual way of doing this is

import numpy
...
setup(
    ...
    ext_modules = [Extension(..., include_path=[numpy.get_include()])]
)

No need to alter include paths etc. then.

(It would be great if somebody could take the time to add this question 
to the FAQ and/or the Cython/NumPy tutorial, it comes up a lot.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun 16 15:08:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Jun 2009 15:08:42 +0200
Subject: [Cython] Cython and Numpy
In-Reply-To: <4A379924.8040005@student.matnat.uio.no>
References: <9104B84B-2297-4693-91A0-D7E8186E259B@gmail.com>
	<c314d7d20906160515m3b8dadc2o9cc0fbcf7f98cc09@mail.gmail.com>
	<4A379924.8040005@student.matnat.uio.no>
Message-ID: <4A37995A.9040109@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Brett Calcott wrote:
>> Hi Daniel,
>>
>>
>> 2009/6/16 Daniel Yarlett <daniel.yarlett at gmail.com>:
>>> ... SNIP ...
>>> It looks like arrayobject.h is not being found, so do I need to modify
>>> my setup.py file somehow? I'm afraid I'm not sure how to do this, so
>>> any help would be greatly appreciated.
>>
>>
>> This line in my .bashrc solves the same problem on my Mac
>>
>> export 
>> C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include
>>
>> Or, I think you could add the same path to your definition of the
>> Extension() in setup.py. I think you can do this:
>>
>> ext_modules = [Extension("neuralRoutines", ["neuralRoutines.pyx"],
>>         extra_compile_args=['-Ithe/path/you/want/here'],
>> )]
>>
>> ... but I haven't checked it.
> 
> The usual way of doing this is
> 
> import numpy
> ...
> setup(
>    ...
>    ext_modules = [Extension(..., include_path=[numpy.get_include()])]
> )

Sorry, that should be include_dirs, not include_path.

-- 
Dag Sverre

From ellisonbg.net at gmail.com  Tue Jun 16 19:20:57 2009
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 16 Jun 2009 10:20:57 -0700
Subject: [Cython] Trivial numpy test in sage notebook fails...
In-Reply-To: <79017BF3-B11C-4378-A1C1-33572625AB28@math.washington.edu>
References: <3327124144.3090608@smtp.netcom.no>
	<79017BF3-B11C-4378-A1C1-33572625AB28@math.washington.edu>
Message-ID: <6ce0ac130906161020l80c746ev770b4520620262c0@mail.gmail.com>

I too need to use the numpy+cython stuff from the notebook.

> To conclude this thread, Sage ships its own numpy.pxd, which predates
> the buffer interface. We need to go in and remove this (there's a
> ticket in Sage to do this).

In the meantime, if I simply remove Sage's old numpy.pxd file will
this start working, or is it more complicated than that?

Cheers,

Brian

From robertwb at math.washington.edu  Wed Jun 17 06:45:41 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Jun 2009 21:45:41 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3672EC.90002@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
Message-ID: <12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>

On Jun 15, 2009, at 9:12 AM, Dag Sverre Seljebotn wrote:

> Thanks to everybody who contributed to the discussion on a Cython  
> array
> type last week! Here's a summary to attempt focusing the discussion.
>
> There are now two CEPs:
> - CEP 517, array type: http://wiki.cython.org/enhancements/array
> - CEP 518, SIMD operations: http://wiki.cython.org/enhancements/simd
>
> I mostly just added a "what does this facilitate"-section is added  
> near
> the beginning of each, and the multidimensional aspect of the  
> arrays has
> been emphasised. No need to reread it.

Looks good. I assume this supersedes http://wiki.cython.org/ 
enhancements/buffersyntax ; are there any other wiki pages that are  
made obsolete by these proposals?

> The story is this:
>
>   a) I can only commit to implementing CEP 517 this time around
>
>   b) The arrays themselves/CEP 517 can without loosing *too* much be
> done as a library (though I would prefer them as builtins myself).
> However SIMD/CEP 518 depends on building the arrays into the language,
> and switching approaches later would mean redoing some stuff.  
> Therefore
> SIMD (and the numerical roadmap in general) keeps popping into the
> discussion.
>
>   c) It would be nice to finally settle a roadmap for numerics in  
> Cython
> (how far will the language be bent for the benefit of numerics), for
> instance to economize on discussions when questions pop up on the
> mailing list (and make plans). These two CEPs would create one such  
> roadmap.
>
>   d) CEP 518 isn't a lot of work (totally doable in a GSoC next
> year...though of course not at the expense of e.g. a full Python
> compatability GSoC or similar).
>
> Key impressions from last week:
>
>   a) Numerical users seem positive about all of this.
>   b) Robert Bradshaw said:
>
> I am not opposed to this feature, despite it being off the path of
> being a good, optimizing Python compiler with easy glue to external
> code, as long as it doesn't intrude on the language too much. Mostly,
> this is because there is so much interest in using Cython in a
> numerical processing context. I do, however, have some reservations.
>
> [I believe the reservations were all taken care of.]

I still have some questions, but I am certainly in favor of something  
like this happening.

One thing that isn't quite clear is how exactly the reference  
counting/memory allocation is going to work. You give an example of  
explicitly creating an int[:,:] via int[:100,:100](). Would some kind  
of memoryview be created in the background? A string to hold the  
data? (This doesn't have to be decided now, just curious.) This is  
also needed for the copy "method," or any implicit copying that happens.

On a related note, it's still a bit unclear how these things can be  
passed around and stored. Are they just a Py_buffer + PyObject*? (I'm  
hoping you're thinking they can be passed around and stored with  
ease, with allocation either take care of by the corresponding object  
(which will clean up the memory when it gets collected) or if there  
is no object attached, the user needs to treat it as they would a raw  
pointer).

Until CEP 518, how would arithmetic happen? (Assuming I'm to lazy to  
write the loops myself) would I create two numpy objects to wrap  
them, add those objects, and then "unpack" the result. For large  
enough datasets, that's not too much overhead.

- Robert


From robertwb at math.washington.edu  Wed Jun 17 07:13:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Jun 2009 22:13:45 -0700
Subject: [Cython] force cython to do local imports
In-Reply-To: <85b5c3130906111207s44d5a715qfe91081566a0b5c2@mail.gmail.com>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<4A30B977.9090306@behnel.de>
	<85b5c3130906110823g6d2470e1t83640a96bd69cdad@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
	<4A314FB0.9090507@behnel.de>
	<85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>
	<85b5c3130906111207s44d5a715qfe91081566a0b5c2@mail.gmail.com>
Message-ID: <23671C20-F766-48C3-853E-A22ACDACBEA6@math.washington.edu>

On Jun 11, 2009, at 12:07 PM, Ondrej Certik wrote:

> On Thu, Jun 11, 2009 at 1:02 PM, Ondrej Certik<ondrej at certik.cz>  
> wrote:
>> On Thu, Jun 11, 2009 at 12:40 PM, Stefan  
>> Behnel<stefan_ml at behnel.de> wrote:
>>>
>>> Lisandro Dalcin wrote:
>>>> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz>  
>>>> wrote:
>>>>> As far as Cython is concerned, the only problem I can see is  
>>>>> that it
>>>>> changes all import paths to the fully qualified import paths.  
>>>>> Python
>>>>> allows both ways, Cython only the fully qualified way. As Stefan
>>>>> explained, this is needed, because it will then not work  
>>>>> properly at
>>>>> the moment. But I am not convinced this cannot be fixed --  
>>>>> though it
>>>>> may happen that the fix is too tedious and not worthy.
>>>>>
>>>>
>>>> IMHO, the only "fix" Cython needs is support for relative  
>>>> imports...
>>>> Then you could write:
>>>>
>>>> from ._hermes2d cimport what_you_need
>>>>
>>>> This way, Ondrej would be able to easily write Cython code using
>>>> relative imports the right way...
>>>
>>> I thought about this, too, and yes, that would be a nice feature.  
>>> But it
>>> doesn't help in this case. What Ondrej wanted was to be able to  
>>> import the
>>> module both from its package and as a plain import, i.e. when put  
>>> outside
>>> of any package. The above would not handle the latter case either.
>>
>> I think it would. I have _forms.so and _hermes2d.so. The _hermes2d.so
>> is imported from _forms.so and if the import was relative, e.g. if
>> _forms.so was doing "import ._hermes2d", then it should work both in
>> the tree and out of the tree.
>
> Looking here, it should be easy to support relative imports:
>
> http://docs.python.org/c-api/import.html
>
> so unless there is some difficulty inside Cython, that I don't see yet
> (e.g. the kind of thing Stefan was talking about), it should be
> possible to support relative imports quite easily, I already
> understand the part that takes the import path from .pyx file and then
> makes it fully qualified --- so I would simply change that.

Cython does support relative cimports (we use them all over in Sage).  
What it does not support is deciding a modules fully qualified name  
at runtime because it uses this information at compile time. (It is  
possible that this could be changed, I haven't thought of what the  
repercussions should be).

What I don't understand, in your setup, is how are you handling

foo.py
foo2.pyx
a/__init__.py
a/bar.pyx
a/bar.pxd

Now when foo.py does "from a.bar import *" or foo2.pyx does "from  
a.bar cimport *" how does this get resolved if the .so files are  
moved to the top level?

- Robert



From robertwb at math.washington.edu  Wed Jun 17 07:22:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Jun 2009 22:22:19 -0700
Subject: [Cython] problem with pure python mode
In-Reply-To: <85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
	<12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
	<85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>
Message-ID: <7B1321FB-58A6-4CA3-927A-54FA609A0336@math.washington.edu>

On Jun 10, 2009, at 5:45 PM, Ondrej Certik wrote:

> On Wed, Apr 29, 2009 at 4:00 PM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> On Apr 29, 2009, at 2:49 PM, Ondrej Certik wrote:
>>
>>>>>
>>>>> Not sure--have you tried without the locals decorator in the  
>>>>> Python
>>>>> source? I hope this isn't a regression.
>>>>
>>>> Yes, as you can see, the locals is commented out. I'll try to  
>>>> bisect
>>>> it to see which commit broke it.
>>>
>>> Ok, so it never worked. I am a bit confused, because I thought the
>>> pure python mode worked fine for me before.
>>>
>>> In the current cython --- is there any way to keep a pure python  
>>> code,
>>> and create an accompanying pxd file that would annotate the cdef
>>> functions and classes?
>>
>> This should work, that's how I implemented this a long time ago. If
>> it's not working now it's because something broke (or, I'm not
>> promising that it's completely bug-free, but it should be good--we
>> use it to bootstrap Cython itself).
>
> So I made it to work by observing how things are done in Cython  
> itself.
>
> If you do:
>
> git clone git://github.com/certik/cython-test.git
> cd cython-test
> cython fact.pyx
>
> then it produces nicely optimized fact.c file. However, if you do:
>
> cython fact.py
>
> it produces a fact.c, that compiles, but it is not optimized --- the
> "i" and "r" variables are python objects, not ints, even though my
> fact.pxd file contains:
>
> import cython
>
> @cython.locals(i=cython.int, r=cython.int)
> cdef int factorial(int n)
>
>
> Do you see anything wrong with my files? Is this supposed to work?
>
> I think it is not implemented yet, because when I looked at the cython
> sources, it doesn't work there either. This is the relevant part of
> Scanning.pxd:
>
>     @cython.locals(current_level=cython.long, new_level=cython.long)
>     cpdef indentation_action(self, text)
>
> and here is the declaration in the generated Scanning.c:
>
>   PyObject *__pyx_v_new_level;
>
>
> And consider for example the following chunk:
>
>   /* "/home/ondrej/repos/cython-devel/Cython/Compiler/Scanning.py":398
>  *         new_level = len(text)
>  *         #print "Changing indent level from", current_level, "to",
> new_level ###
>  *         if new_level == current_level:             # <<<<<<<<<<<<<<
>  *             return
>  *         elif new_level > current_level:
>  */
>   __pyx_t_4 = PyObject_RichCompare(__pyx_v_new_level,
> __pyx_v_current_level, Py_EQ); if (unlikely(!__pyx_t_4))
> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno =
> __LINE__; goto __pyx_L1_error;}
>   __Pyx_GOTREF(__pyx_t_4);
>   __pyx_t_3 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_3
> < 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno
> = __LINE__; goto __pyx_L1_error;}
>   __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
>   if (__pyx_t_3) {
>
>
> both the new_level and current_level are declared as long, but above
> you can see that Cython is calling lots of Python C/API methods...

Looks like a bug to me.

http://trac.cython.org/cython_trac/ticket/330

- Robert


From robertwb at math.washington.edu  Wed Jun 17 07:34:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Jun 2009 22:34:03 -0700
Subject: [Cython] Incorrect error message
In-Reply-To: <c314d7d20905190116r2c900372l7755aa2697b7001e@mail.gmail.com>
References: <c314d7d20905190116r2c900372l7755aa2697b7001e@mail.gmail.com>
Message-ID: <917B0025-20DA-4349-B948-0E02AF4ABE3D@math.washington.edu>

This is now http://trac.cython.org/cython_trac/ticket/332

On May 19, 2009, at 1:16 AM, Brett Calcott wrote:

> Hi,
>
> This following code gives an very misleading error message. It says
> there is something wrong with the definition of another_array (than
> ndarray is not a type identifier). If you comment out the definition
> of another_array, then it gives the correct (I assume) error message:
> that Buffer types can only be local.
>
> At least, I think that is the case.
>
> Cheers,
> Brett
>
> import numpy as np
> cimport numpy as np
> cimport cython
>
> cdef class Boodle:
>     cdef np.ndarray[double, ndim=1] an_array
>     def __cinit__(self):
>         cdef np.ndarray[double, ndim=1, mode="c"] \
>                 another_array = np.zeros(self.E, dtype=np.float64)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan at sun.ac.za  Wed Jun 17 12:11:02 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 17 Jun 2009 12:11:02 +0200
Subject: [Cython] Wiki pages on profiling and compiling Windows extensions
Message-ID: <9457e7c80906170311p654f6415y73a7df8385cac84b@mail.gmail.com>

Hi all,

I started two new wiki pages:

http://wiki.cython.org/Profiling
http://wiki.cython.org/BuildingWindowsInstaller

The profile page is basically a stub and should still be expanded.

Could we link to these from the wiki front page or from another visible page?

Thanks
St?fan

From dagss at student.matnat.uio.no  Wed Jun 17 18:57:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Jun 2009 18:57:12 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
Message-ID: <4A392068.6070800@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 15, 2009, at 9:12 AM, Dag Sverre Seljebotn wrote:
> 
>> Thanks to everybody who contributed to the discussion on a Cython  
>> array
>> type last week! Here's a summary to attempt focusing the discussion.
>>
>> There are now two CEPs:
>> - CEP 517, array type: http://wiki.cython.org/enhancements/array
>> - CEP 518, SIMD operations: http://wiki.cython.org/enhancements/simd
>>
>> I mostly just added a "what does this facilitate"-section is added  
>> near
>> the beginning of each, and the multidimensional aspect of the  
>> arrays has
>> been emphasised. No need to reread it.
> 
> Looks good. I assume this supersedes http://wiki.cython.org/ 
> enhancements/buffersyntax ; are there any other wiki pages that are  
> made obsolete by these proposals?

It is connected with http://wiki.cython.org/enhancements/arraytypes, 
although not all points are in (like conversion to list), and 
furthermore it should perhaps be made into your and Stefan's proposed 
type (which you called [int] or int[]) instead with + for concatenation.

BTW, those list-like types can likely share a lot of implementation with 
my proposed int[:]; the major change would be restricting it to 1D, 
different arithmetic behaviour, and, say, default coercion to list 
instead of memoryview (perhaps!, but let's not go there now). But it 
could still be PEP 3118-backed, coerceable from C pointers, etc., and 
share implementation for that. Basically it would be two different 
frontends to the same underlying type.

> I still have some questions, but I am certainly in favor of something  
> like this happening.

Yes, I didn't intend to solve all the questions now, just .

I suppose I'm still waiting for Stefan's opinion though, given his last 
comment last week. If positive, I think me and Kurt can do the main work 
with hammering out the details, though of course you can comment then as 
much (or little) as you want.

(I notice that I'm promoted to lead developer on the Cython front page 
-- thanks! -- but I don't take it for granted that it should be a case 
of majority vote, and at any rate I'd never push for something which 
would make Stefan less interested in the project.)

> One thing that isn't quite clear is how exactly the reference  
> counting/memory allocation is going to work. You give an example of  
> explicitly creating an int[:,:] via int[:100,:100](). Would some kind  
> of memoryview be created in the background? A string to hold the  
> data? (This doesn't have to be decided now, just curious.) This is  
> also needed for the copy "method," or any implicit copying that happens.
 >
 > On a related note, it's still a bit unclear how these things can be
 > passed around and stored. Are they just a Py_buffer + PyObject*? (I'm
 > hoping you're thinking they can be passed around and stored with
 > ease, with allocation either take care of by the corresponding object
 > (which will clean up the memory when it gets collected) or if there
 > is no object attached, the user needs to treat it as they would a raw
 > pointer).

I skipped over it as it is a long story. If you really are curious:

And int[:] is a pass-by-value struct, containing subslice info and a 
reference to an acquired view, which in turn is acquired from a 
memory-holding object.

This seems heavy but really is necesarry due to how PEP 3118 is. ("The 
only problem which can't be solved by another layer of indirection is 
too many layers of indirection.")

In detail:

There are three levels:
1) Memory-holding object. When Cython allocates, we need a new type 
(probably inheriting directly from object), which allocates memory and 
stores shape/stride information, and returns the right information in 
tp_getbuffer. Would be a 20-liner in Cython unless we want to go with 
PyVarObject for allocation in which case I suppose it is a 200-liner in C.

2) The acquired Py_buffer on that object. That would happen the same way 
as with every array; preferably by using memoryview, or if not, another 
custom Python type (need to backport memoryview anyway) or if we can't 
seem to avoid it a custom refcounted struct.

3) Accessing Py_buffer directly is too inefficient, so it must be 
unpacked in to a custom struct on the stack which basically holds 
shape/stride information and a reference to the Py_buffer-holding thing 
in 2). This is the actual variable/temporary type, and is passed-by-value.

When taking a slice, the struct in 3) is copied (while adjusting the 
shape/strides), increfing the view 2) in the process, and 2) holds on to 1).

Then there's fields and global variables which call for seperate 
decisions; probably the most consistent, and efficient in both speed and 
memory, is to store the structs of 3), although it is a bit 
counter-intuitive.

> Until CEP 518, how would arithmetic happen? (Assuming I'm to lazy to  
> write the loops myself) would I create two numpy objects to wrap  
> them, add those objects, and then "unpack" the result. For large  
> enough datasets, that's not too much overhead.

I suppose the real answer is that if you're too lazy to write the loops 
yourself, you'll be using ndarray[int] in the first place :-)

But yes, you can do

cdef int[:] a = ..., b = ...
a = np.array(a) + b

(Or, not 100% sure, but at least

a = np.array(a) + np.array(b)

will work).

Even before NumPy supports PEP 3118 this can work, as we can coerce 
int[:] to our own subclass of memoryview which implements NumPy's 
__toarray__ special function.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 17 19:03:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Jun 2009 19:03:51 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A392068.6070800@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
Message-ID: <4A3921F7.7030404@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> On a related note, it's still a bit unclear how these things can be
>> passed around and stored. Are they just a Py_buffer + PyObject*? (I'm
>> hoping you're thinking they can be passed around and stored with
>> ease, with allocation either take care of by the corresponding object
>> (which will clean up the memory when it gets collected) or if there
>> is no object attached, the user needs to treat it as they would a raw
>> pointer).

To really answer this, yes. Reference counting etc., it will just work 
like any other Python reference from the user's perspective, with the 
usual caveats of "<object>char_ptr".

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 17 19:08:37 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Jun 2009 19:08:37 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3921F7.7030404@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
	<4A3921F7.7030404@student.matnat.uio.no>
Message-ID: <4A392315.50305@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> On a related note, it's still a bit unclear how these things can be
>>> passed around and stored. Are they just a Py_buffer + PyObject*? (I'm
>>> hoping you're thinking they can be passed around and stored with
>>> ease, with allocation either take care of by the corresponding object
>>> (which will clean up the memory when it gets collected) or if there
>>> is no object attached, the user needs to treat it as they would a raw
>>> pointer).
> 
> To really answer this, yes. Reference counting etc., it will just work 
> like any other Python reference from the user's perspective, with the 
> usual caveats of "<object>char_ptr".
> 

Hmm. <object>char_ptr takes a copy I suppose though? So it's kind of 
different, but at any rate deallocating C storage before a wrapping 
array reference is destroyed is fatal. We could support

a = int[:,:](some_c_ptr, deallocator=stdlib.free)

though to transfer ownership.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 17 19:15:16 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Jun 2009 19:15:16 +0200
Subject: [Cython] Wiki pages on profiling and compiling Windows
	extensions
In-Reply-To: <9457e7c80906170311p654f6415y73a7df8385cac84b@mail.gmail.com>
References: <9457e7c80906170311p654f6415y73a7df8385cac84b@mail.gmail.com>
Message-ID: <4A3924A4.80405@student.matnat.uio.no>

St?fan van der Walt wrote:
> Hi all,
> 
> I started two new wiki pages:
> 
> http://wiki.cython.org/Profiling
> http://wiki.cython.org/BuildingWindowsInstaller
> 
> The profile page is basically a stub and should still be expanded.

Great!, thanks.

> Could we link to these from the wiki front page or from another visible page?

Sure, just add a link as you see fit, probably under Tips&Tricks on the 
wiki front page, and possibly referenced from the FAQ as well.


-- 
Dag Sverre

From stefan_ml at behnel.de  Wed Jun 17 20:41:38 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Jun 2009 20:41:38 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3672EC.90002@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
Message-ID: <4A3938E2.9020905@behnel.de>

Hi Dag,

thanks for writing this up (and for insisting :).

Dag Sverre Seljebotn wrote:
> Thanks to everybody who contributed to the discussion on a Cython array 
> type last week! Here's a summary to attempt focusing the discussion.
> 
> There are now two CEPs:
> - CEP 517, array type: http://wiki.cython.org/enhancements/array
> - CEP 518, SIMD operations: http://wiki.cython.org/enhancements/simd
> [...]
>   c) It would be nice to finally settle a roadmap for numerics in Cython 
> (how far will the language be bent for the benefit of numerics), for 
> instance to economize on discussions when questions pop up on the 
> mailing list (and make plans). These two CEPs would create one such roadmap.
> [...]
>   a) Numerical users seem positive about all of this.
>   b) Robert Bradshaw said:
>
> I am not opposed to this feature, despite it being off the path of
> being a good, optimizing Python compiler with easy glue to external
> code, as long as it doesn't intrude on the language too much. Mostly,
> this is because there is so much interest in using Cython in a
> numerical processing context. I do, however, have some reservations.
>
> [I believe the reservations were all taken care of.]
>
>   c) I'm not sure about Stefan Behnel's current opinions about this.

I'm not involved in numerics, so I can't comment on the feature impact.
>From my POV, every addition that doesn't 'degrade' (i.e. complicate) either
the language or the compiler code is just fine.

Every addition that requires major syntax support will almost always invoke
a discussion, as seen here. IMHO, getting a feature without syntax impact
is a lot better than yet another way of overloading and extending some
bracket syntax.

Regarding CEP 517, I think that a 'native', memory managed array type would
be very helpful in a fair lot of use cases. Plus, (most of) this can be
done in an external module. So that's a perfect new feature. Solving ticket
152 (PyVarObject support) first would be helpful to this end.

Regarding CEP 518, I do see the interest, and I do see why this cannot be
done externally. But I also see that there were doubts that this is enough
to be truly useful except for a number of cases. And it has some impact on
the language.

We had a discussion about parallelism in the language a while ago, which
lead to this write-up:

http://wiki.cython.org/enhancements/parallel

Nothing happened since then, as ticket 211 shows.

I like the idea of parallelism in the language in general, and I'm not
opposed to CEP 518. I'd just like to see it added in a way that clearly
separates it from the 'normal' array type.

What would be better for the intended use case: allow parallelism in the
language in general (i.e. parallel loops), or have a parallel type that
does some magic behind the scenes?

Stefan

From stefan_ml at behnel.de  Wed Jun 17 21:49:38 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Jun 2009 21:49:38 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A392068.6070800@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
Message-ID: <4A3948D2.3000003@behnel.de>

Hi Dag,

Dag Sverre Seljebotn wrote:
> those list-like types can likely share a lot of implementation with 
> my proposed int[:]; the major change would be restricting it to 1D, 
> different arithmetic behaviour, and, say, default coercion to list 
> instead of memoryview (perhaps!, but let's not go there now). But it 
> could still be PEP 3118-backed, coerceable from C pointers, etc., and 
> share implementation for that. Basically it would be two different 
> frontends to the same underlying type.

It would be great if those types could be implemented externally, i.e.
without new syntax but with an import. Some special casing in the compiler
may be ok (say, when Cython sees that the type defines some special method
or attribute), but a complete implementation inside the compiler sounds wrong.


> I suppose I'm still waiting for Stefan's opinion though, given his last 
> comment last week.

Yes, I admit I'm a little slow in responding lately. ;)


> I notice that I'm promoted to lead developer on the Cython front page 
> -- thanks! -- but I don't take it for granted that it should be a case 
> of majority vote, and at any rate I'd never push for something which 
> would make Stefan less interested in the project.

That's not the case. You long deserve being 'promoted' (or rather 'named')
lead developer and I appreciate the interest and support that we get from
the numerics area. I think it's worth making Cython a suitable language for
that area.


> There are three levels:
> 1) Memory-holding object. When Cython allocates, we need a new type 
> (probably inheriting directly from object), which allocates memory and 
> stores shape/stride information, and returns the right information in 
> tp_getbuffer. Would be a 20-liner in Cython unless we want to go with 
> PyVarObject for allocation in which case I suppose it is a 200-liner in C.

I'm all for keeping such an implementation in Cython, at least for now. We
can always optimise/reimplement later.


> 2) The acquired Py_buffer on that object.

Erm, buffers are not memory managed, right? They are just the plain structs
that I think of?

> That would happen the same way 
> as with every array; preferably by using memoryview, or if not, another 
> custom Python type (need to backport memoryview anyway) or if we can't 
> seem to avoid it a custom refcounted struct.

If we implement the object ourselves, what do we need memoryview for? Would
it not just implement the buffer protocol? Without any ref-counting? And
when coercing to an object, why not just return the object itself? It could
just have a C-level interface and a Python interface.

I also read in CEP 517 that resizing is not to be supported. Why not? It
could just fail with an exception when it notices that there are live
buffers on it.


> 3) Accessing Py_buffer directly is too inefficient, so it must be 
> unpacked in to a custom struct on the stack which basically holds 
> shape/stride information and a reference to the Py_buffer-holding thing 
> in 2). This is the actual variable/temporary type, and is passed-by-value.
> 
> When taking a slice, the struct in 3) is copied (while adjusting the 
> shape/strides), increfing the view 2) in the process, and 2) holds on to 1).

So a slice would be a view? Because if it was a copy, you'd need a separate
memory object to back it. A view would be easiest handled by the normal
memoryview(), though, right? And I find a copy much less surprising...

Stefan

From dagss at student.matnat.uio.no  Wed Jun 17 22:30:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Jun 2009 22:30:51 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3938E2.9020905@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no> <4A3938E2.9020905@behnel.de>
Message-ID: <4A39527B.5030702@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi Dag,
> 
> thanks for writing this up (and for insisting :).

It doesn't seem like I had much choice -- the main, concentrated period 
I have for Cython this summer, where I can work on totally new stuff, is 
about a week away, and ideally me and Kurt would have gone through some 
implementation details and work sharing by then. If a decision can't be 
reached very soon I fear I'll have to drop it anyway, making the 
discussion kind of moot.

> Every addition that requires major syntax support will almost always invoke
> a discussion, as seen here. IMHO, getting a feature without syntax impact
> is a lot better than yet another way of overloading and extending some
> bracket syntax.
> 
> Regarding CEP 517, I think that a 'native', memory managed array type would
> be very helpful in a fair lot of use cases. Plus, (most of) this can be
> done in an external module. So that's a perfect new feature. Solving ticket
> 152 (PyVarObject support) first would be helpful to this end.

...and adding support for creating custom PyVarObjects in Cython through 
special syntax perhaps.


> Regarding CEP 518, I do see the interest, and I do see why this cannot be
> done externally. But I also see that there were doubts that this is enough
> to be truly useful except for a number of cases. And it has some impact on
> the language.

It is useful in a lot of numerical cases. I just benchmarked: Doing 
sqrt(x*x+y*y) with a CEP 518 approach is 3 times faster on my machine 
than NumPy (using only whatever gcc does to my naive loop and a single 
thread).

It's the chicken and egg problem again: Currently Python/NumPy is seen 
as a MATLAB replacement, with this Python/NumPy could start to compete 
with FORTRAN (given some years of interest in Cython from the numerical 
community). These CEPs IMO scetch pretty much everything that make 
people still cling to FORTRAN, feature-wise.

Current standard practice is to write the heavy stuff in FORTRAN or C++ 
and then wrap it for use from Python; that's an entirely new "user 
segment" for whom Cython is currently not so much of an option.

> We had a discussion about parallelism in the language a while ago, which
> lead to this write-up:
> 
> http://wiki.cython.org/enhancements/parallel
> 
> Nothing happened since then, as ticket 211 shows.

Thread parallelism is IMO orthogonal. Even serial implementations of CEP 
518 would be very useful because of reduced memory bus traffic (re above 
3x speed increase). In some situations parallell implementations may not 
even help (if too little happen on the CPU vs. memory bandwidth); thus 
CEP 518 in serial is needed even to get to the point where memory 
bandwidth is offloaded.

In my own work I'd use pretty coarse-grained MPI and run the program on 
100 cluster nodes at the same time (each with only 4 cores); shared 
memory parallellism in the language thus buys me personally absolutely 
nothing. (Here each numerical user is different though, but I've been 
the one involved most heavily in Cython.)

> I like the idea of parallelism in the language in general, and I'm not
> opposed to CEP 518. I'd just like to see it added in a way that clearly
> separates it from the 'normal' array type.

Robert proposed that int[:,:] would be just that special numerics type, 
and that [int] or int[] would be a normal array type (perhaps even 
[[int]] for 2D). The int[:,:] notation has a lot of possibilities for 
embedding information about how each dimension is stored which is 
primarily useful in numerics -- specifying "int[:,::1]" so that it is 
known compile-time that the second dimension is contiguous can double 
the speed in an in-cache numerical situation.

Really, I think this is much about carving out a little room in the 
syntax where the significant numerical Cython community can play by 
themselves. int[:,:] doesn't seem to be something we'd have to tell 
non-numerics people about :-); and all the groundwork towards [int] 
would be done as a side-effect.

As Lisandro mentioned, for a numerical user, SIMD is the "normal" way of 
treating an array in practically every possible alternative language 
(and the reason C is out of the question for numerics).

> 
> What would be better for the intended use case: allow parallelism in the
> language in general (i.e. parallel loops), or have a parallel type that
> does some magic behind the scenes?

Assuming the intended use case is numerics, the latter one is pretty 
much the only viable option IMO. If one have to make a choice, but the 
former can be nice in some other situations.

As Kurt touched upon though, I don't think we can compete with 40 years 
of language development in the custom numerics languages. So I'm 
actually reluctant to have the discussion, and would hope to keep the 
discussion to whether we can "make a little corner in the syntax where 
the numerics people can play alone" and avoid any bikeshedding.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 17 23:28:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Jun 2009 23:28:12 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3948D2.3000003@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>
Message-ID: <4A395FEC.2060002@student.matnat.uio.no>

Stefan Behnel wrote:
>> There are three levels:
>> 1) Memory-holding object. When Cython allocates, we need a new type 
>> (probably inheriting directly from object), which allocates memory and 
>> stores shape/stride information, and returns the right information in 
>> tp_getbuffer. Would be a 20-liner in Cython unless we want to go with 
>> PyVarObject for allocation in which case I suppose it is a 200-liner in C.
> 
> I'm all for keeping such an implementation in Cython, at least for now. We
> can always optimise/reimplement later.

Sounds right. Would motive doing something about Cython support for 
PyVarObject too (perhaps a magic type you'd inherit from, which has a 
constructor with size and a void* field pointing to the allocated 
buffer? So #152 would get us 90% of the way.)

>> 2) The acquired Py_buffer on that object.
> 
> Erm, buffers are not memory managed, right? They are just the plain structs
> that I think of?

Yes, they need to be wrapped in an object of some kind.

> 
>> That would happen the same way 
>> as with every array; preferably by using memoryview, or if not, another 
>> custom Python type (need to backport memoryview anyway) or if we can't 
>> seem to avoid it a custom refcounted struct.
> 
> If we implement the object ourselves, what do we need memoryview for? Would
> it not just implement the buffer protocol? Without any ref-counting? And
> when coercing to an object, why not just return the object itself? It could
> just have a C-level interface and a Python interface.

I don't follow you quite. But some points I think are relevant:

a) The memory will often live in objects from 3rd party libraries 
accessed through PEP 3118. It seems cleanest to have Cython-allocated 
memory just be another such object, treated in the same fashion.

b) If you load 10 GB of data into memory in one array (yes some numerics 
users do that), copying is usually not an option; copying must always be 
easily predictable and user-managed.

c) When taking slices, you get a new view to the same data, however the 
original Py_buffer, likely stored in a memoryview object, cannot be 
modified by protocol. I.e. the new slice needs to be "stored" elsewhere. 
And the original object can't be told to reallocate/reslice.

> I also read in CEP 517 that resizing is not to be supported. Why not? It
> could just fail with an exception when it notices that there are live
> buffers on it.

It just seemed kind of useless if only Cython-allocated memory can be 
resized, and not C arrays or memory in other Python objects. But we 
could do it for Cython-allocated memory, perhaps by extending PEP 3118 
with some Cython-specific flags etc.

I'd rather leave it for later, but it would be cool to have. It's a 
matter of developer time vs. utility too.

>> 3) Accessing Py_buffer directly is too inefficient, so it must be 
>> unpacked in to a custom struct on the stack which basically holds 
>> shape/stride information and a reference to the Py_buffer-holding thing 
>> in 2). This is the actual variable/temporary type, and is passed-by-value.
>>
>> When taking a slice, the struct in 3) is copied (while adjusting the 
>> shape/strides), increfing the view 2) in the process, and 2) holds on to 1).
> 
> So a slice would be a view? Because if it was a copy, you'd need a separate
> memory object to back it. A view would be easiest handled by the normal
> memoryview(), though, right? And I find a copy much less surprising...

Sorry, another NumPy-ism which I forgot wasn't there in core Python. 
I'll fix the CEP at some point. Yes, slices are views and absolutely 
have to be. Slices being natural and efficient views to work with is 
kind of one of the main points. A NumPy example for you

arr[start:end:32, 3] += value * arr[start:end:32, 1]

is much more convenient than

for i in range(start, end, 32):
     arr[i, 3] += value * arr[i, 1]

(it may not look more convenient, but when you do it 50 times a day, 
with bigger and more complicated expressions, it really is)

Another point is that it is the only way of sanely manipulating very big 
arrays (which BTW can be memory mapped -- a single 10 GB array is rather 
common I think).

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun 17 23:39:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Jun 2009 23:39:08 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A395FEC.2060002@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no>
Message-ID: <4A39627C.7010607@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>>> There are three levels:
>>> 1) Memory-holding object. When Cython allocates, we need a new type 
>>> (probably inheriting directly from object), which allocates memory and 
>>> stores shape/stride information, and returns the right information in 
>>> tp_getbuffer. Would be a 20-liner in Cython unless we want to go with 
>>> PyVarObject for allocation in which case I suppose it is a 200-liner in C.
>> I'm all for keeping such an implementation in Cython, at least for now. We
>> can always optimise/reimplement later.
> 
> Sounds right. Would motive doing something about Cython support for 
> PyVarObject too (perhaps a magic type you'd inherit from, which has a 
> constructor with size and a void* field pointing to the allocated 
> buffer? So #152 would get us 90% of the way.)
> 
>>> 2) The acquired Py_buffer on that object.
>> Erm, buffers are not memory managed, right? They are just the plain structs
>> that I think of?
> 
> Yes, they need to be wrapped in an object of some kind.
> 
>>> That would happen the same way 
>>> as with every array; preferably by using memoryview, or if not, another 
>>> custom Python type (need to backport memoryview anyway) or if we can't 
>>> seem to avoid it a custom refcounted struct.
>> If we implement the object ourselves, what do we need memoryview for? Would
>> it not just implement the buffer protocol? Without any ref-counting? And
>> when coercing to an object, why not just return the object itself? It could
>> just have a C-level interface and a Python interface.
> 
> I don't follow you quite. But some points I think are relevant:
> 
> a) The memory will often live in objects from 3rd party libraries 
> accessed through PEP 3118. It seems cleanest to have Cython-allocated 
> memory just be another such object, treated in the same fashion.
> 
> b) If you load 10 GB of data into memory in one array (yes some numerics 
> users do that), copying is usually not an option; copying must always be 
> easily predictable and user-managed.
> 
> c) When taking slices, you get a new view to the same data, however the 
> original Py_buffer, likely stored in a memoryview object, cannot be 
> modified by protocol. I.e. the new slice needs to be "stored" elsewhere. 
> And the original object can't be told to reallocate/reslice.

I have a respectful request to make here: Can we just let whoever will 
implement this (i.e. me and to some degree Kurt) do whatever we feel is 
right regarding implementation, and keep the discussion to syntax impact 
and project impact of the CEPs?

For instance, I already had a long thread in here with Kurt working 
through a lot of the finer details here already (some of that is likely 
outdated, due in fact to something Stefan wrote, but still...).

While I realize that you and Robert has helped me avoid some poor design 
decisions in the past, something I'm really grateful for, it's not a 
scalable solution in the long run. Sometimes having to redo stuff later 
is easier than discussing everything on the mailing list.

Language semantics is something else, that needs to be discussed -- and 
I'm really glad the point about slice-as-a-view came up.


-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 18 08:23:26 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Jun 2009 08:23:26 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A395FEC.2060002@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no>
Message-ID: <4A39DD5E.4070401@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> I also read in CEP 517 that resizing is not to be supported. Why not? It
>> could just fail with an exception when it notices that there are live
>> buffers on it.
> 
> It just seemed kind of useless if only Cython-allocated memory can be 
> resized, and not C arrays or memory in other Python objects. But we 
> could do it for Cython-allocated memory, perhaps by extending PEP 3118 
> with some Cython-specific flags etc.

I don't understand what resizing has to do with PEP 3118 at all. You
obviously wouldn't resize a buffer view but the memory object itself. The
only link is that resizing would fail if you reduced the size of a memory
object while there is a live buffer view on it.


>>> 3) Accessing Py_buffer directly is too inefficient, so it must be 
>>> unpacked in to a custom struct on the stack which basically holds 
>>> shape/stride information and a reference to the Py_buffer-holding thing 
>>> in 2). This is the actual variable/temporary type, and is passed-by-value.
>>>
>>> When taking a slice, the struct in 3) is copied (while adjusting the 
>>> shape/strides), increfing the view 2) in the process, and 2) holds on to 1).
>> So a slice would be a view? Because if it was a copy, you'd need a separate
>> memory object to back it. A view would be easiest handled by the normal
>> memoryview(), though, right? And I find a copy much less surprising...
> 
> Sorry, another NumPy-ism which I forgot wasn't there in core Python. 
> I'll fix the CEP at some point. Yes, slices are views and absolutely 
> have to be. Slices being natural and efficient views to work with is 
> kind of one of the main points.

-1 on slices returning views for a sequence type. That is completely
unprecedented in Python and would only lead to confusing and buggy code.
What would you do to get a 'real' slice copy then? Requiring a user to do

	some_array[2:5][:]

looks totally wrong to me (besides not even working if the second slice
returned a view, too).

This *may* make more sense for a SIMD type, but even there I suggest not
using the slicing syntax for this. Make it a method call that returns a
slice view.

I get the impression by now that CEP 517 and CEP 518 are not related. They
describe completely different types, but only the array type makes sense in
the context of Cython. The SIMD type would introduce semantics that exist
neither in Python nor in C.

The only problem is that a SIMD data type requires Cython compiler support,
otherwise, you could just go and implement it as an external type. If you
see any way to implement this as an add-on rather than a compiler feature,
you'll have my blessing immediately. But I do not see this becoming a
smooth part of the language.

Yes, I'm being NIMCy here, but I really don't see this fit into the
language, sorry. I don't like dark corners in a programming language that
"you don't have to care about because you are not an XYZ user". That's ok
for a stdlib, but not for a language.

Stefan

From robertwb at math.washington.edu  Thu Jun 18 09:29:19 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Jun 2009 00:29:19 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A392315.50305@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
	<4A3921F7.7030404@student.matnat.uio.no>
	<4A392315.50305@student.matnat.uio.no>
Message-ID: <A8958E61-E37F-4D88-A9EB-0DBECAAAD97A@math.washington.edu>

On Jun 17, 2009, at 10:08 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert Bradshaw wrote:
>>>> On a related note, it's still a bit unclear how these things can be
>>>> passed around and stored. Are they just a Py_buffer + PyObject*?  
>>>> (I'm
>>>> hoping you're thinking they can be passed around and stored with
>>>> ease, with allocation either take care of by the corresponding  
>>>> object
>>>> (which will clean up the memory when it gets collected) or if there
>>>> is no object attached, the user needs to treat it as they would  
>>>> a raw
>>>> pointer).
>>
>> To really answer this, yes. Reference counting etc., it will just  
>> work
>> like any other Python reference from the user's perspective, with the
>> usual caveats of "<object>char_ptr".
>>
>
> Hmm. <object>char_ptr takes a copy I suppose though? So it's kind of
> different, but at any rate deallocating C storage before a wrapping
> array reference is destroyed is fatal. We could support
>
> a = int[:,:](some_c_ptr, deallocator=stdlib.free)
>
> though to transfer ownership.

This is starting to get messy, but I suppose not more so than http:// 
docs.python.org/c-api/cobject.html

I would also be OK with

a = <int[:50,:50]>some_c_ptr

Where the corresponding object is None (or NULL) and no automatic  
deallocation occurs when a is reclaimed. Another option is to accept  
PyCObject as well as buffer-supporting objects if the dimensions are  
given. These details could he hashed out later, the important point  
is that they would be passed around and stored as normal refcounted  
objects (and require the GIL, including for slicing).

- Robert


From dagss at student.matnat.uio.no  Thu Jun 18 09:39:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 09:39:01 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <A8958E61-E37F-4D88-A9EB-0DBECAAAD97A@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3921F7.7030404@student.matnat.uio.no>	<4A392315.50305@student.matnat.uio.no>
	<A8958E61-E37F-4D88-A9EB-0DBECAAAD97A@math.washington.edu>
Message-ID: <4A39EF15.7070608@student.matnat.uio.no>

Noticing we're all on here, how about IRC?

I'll hop in and see.

Dag Sverre

Robert Bradshaw wrote:
> On Jun 17, 2009, at 10:08 AM, Dag Sverre Seljebotn wrote:
> 
>> Dag Sverre Seljebotn wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert Bradshaw wrote:
>>>>> On a related note, it's still a bit unclear how these things can be
>>>>> passed around and stored. Are they just a Py_buffer + PyObject*?  
>>>>> (I'm
>>>>> hoping you're thinking they can be passed around and stored with
>>>>> ease, with allocation either take care of by the corresponding  
>>>>> object
>>>>> (which will clean up the memory when it gets collected) or if there
>>>>> is no object attached, the user needs to treat it as they would  
>>>>> a raw
>>>>> pointer).
>>> To really answer this, yes. Reference counting etc., it will just  
>>> work
>>> like any other Python reference from the user's perspective, with the
>>> usual caveats of "<object>char_ptr".
>>>
>> Hmm. <object>char_ptr takes a copy I suppose though? So it's kind of
>> different, but at any rate deallocating C storage before a wrapping
>> array reference is destroyed is fatal. We could support
>>
>> a = int[:,:](some_c_ptr, deallocator=stdlib.free)
>>
>> though to transfer ownership.
> 
> This is starting to get messy, but I suppose not more so than http:// 
> docs.python.org/c-api/cobject.html
> 
> I would also be OK with
> 
> a = <int[:50,:50]>some_c_ptr
> 
> Where the corresponding object is None (or NULL) and no automatic  
> deallocation occurs when a is reclaimed. Another option is to accept  
> PyCObject as well as buffer-supporting objects if the dimensions are  
> given. These details could he hashed out later, the important point  
> is that they would be passed around and stored as normal refcounted  
> objects (and require the GIL, including for slicing).
> 
> - Robert
> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Jun 18 09:46:34 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Jun 2009 00:46:34 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A39DD5E.4070401@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>
Message-ID: <0B902481-143E-4328-B854-24E8FAB6863D@math.washington.edu>

On Jun 17, 2009, at 11:23 PM, Stefan Behnel wrote:

>>> So a slice would be a view? Because if it was a copy, you'd need  
>>> a separate
>>> memory object to back it. A view would be easiest handled by the  
>>> normal
>>> memoryview(), though, right? And I find a copy much less  
>>> surprising...
>>
>> Sorry, another NumPy-ism which I forgot wasn't there in core Python.
>> I'll fix the CEP at some point. Yes, slices are views and absolutely
>> have to be. Slices being natural and efficient views to work with is
>> kind of one of the main points.
>
> -1 on slices returning views for a sequence type. That is completely
> unprecedented in Python and would only lead to confusing and buggy  
> code.
> What would you do to get a 'real' slice copy then? Requiring a user  
> to do
>
> 	some_array[2:5][:]
>
> looks totally wrong to me (besides not even working if the second  
> slice
> returned a view, too).
>
> This *may* make more sense for a SIMD type, but even there I  
> suggest not
> using the slicing syntax for this. Make it a method call that  
> returns a
> slice view.

This is what numpy does, and what makes slicing so powerful.

sage: import numpy
sage: A = numpy.zeros((2, 4))
sage: B = A[:,2:]
sage: B[1,1] = 100
sage: A
array([[   0.,    0.,    0.,    0.],
        [   0.,    0.,    0.,  100.]])

> I get the impression by now that CEP 517 and CEP 518 are not  
> related. They
> describe completely different types, but only the array type makes  
> sense in
> the context of Cython. The SIMD type would introduce semantics that  
> exist
> neither in Python nor in C.
>
> The only problem is that a SIMD data type requires Cython compiler  
> support,
> otherwise, you could just go and implement it as an external type.  
> If you
> see any way to implement this as an add-on rather than a compiler  
> feature,
> you'll have my blessing immediately. But I do not see this becoming a
> smooth part of the language.
>
> Yes, I'm being NIMCy here, but I really don't see this fit into the
> language, sorry. I don't like dark corners in a programming  
> language that
> "you don't have to care about because you are not an XYZ user".  
> That's ok
> for a stdlib, but not for a language.

I see both a native list and vector type as being worthwhile  
additions to the language. I also think parallelism needs to be  
supported at the language level, and specifying that the vector type  
may (in the future) be handled as a SIMD type is useful here.

The current buffer interface allows nice indexing, but we can't go  
much further without assuming (at compile time) the semantics of  
slicing, etc. In fact, we're already assuming that indexing happens  
as proscribed by the buffer interface (a reasonable, but easily  
violated, assumption). This new type would allow us to work with  
object[:,:] without worrying about what kind of object it was--the  
indexing, slicing, and eventually arithmetic would be defined as it  
is for ndarrays, also opening the possibilities for CEP 518.

I'd like to say it's syntactic sugar for something, but I'm at a loss  
to say what of (implying it's an actual new language feature). I  
would like it to be very easy to describe (from a high level), e.g.  
"It behaves just like an ndarray, but only supports a subset of  
operations."

- Robert


From dagss at student.matnat.uio.no  Thu Jun 18 09:53:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 09:53:58 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A39DD5E.4070401@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>
Message-ID: <4A39F296.6020907@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> I also read in CEP 517 that resizing is not to be supported. Why not? It
>>> could just fail with an exception when it notices that there are live
>>> buffers on it.
>> It just seemed kind of useless if only Cython-allocated memory can be 
>> resized, and not C arrays or memory in other Python objects. But we 
>> could do it for Cython-allocated memory, perhaps by extending PEP 3118 
>> with some Cython-specific flags etc.
> 
> I don't understand what resizing has to do with PEP 3118 at all. You
> obviously wouldn't resize a buffer view but the memory object itself. The
> only link is that resizing would fail if you reduced the size of a memory
> object while there is a live buffer view on it.

I've been struggling to find the right terms for this the entire spring 
-- perhaps the right term for CEP 517 is "typed memoryviews", although 
which can allocate new memory when necesarry.

What I mean is that this doesn't make sense:

import numpy as np
...
data = np.memmap("ten_gigabytes.dat", dtype=np.double)
cdef double[:] access_to_data = data
access_to_data.append(3)

because access_to_data ultimately accesses memory "on disk" in a memory 
mapped file, and we definitely can't append to file through PEP 3118.


> -1 on slices returning views for a sequence type. That is completely
> unprecedented in Python and would only lead to confusing and buggy code.
> What would you do to get a 'real' slice copy then? Requiring a user to do
> 
> 	some_array[2:5][:]
> 
> looks totally wrong to me (besides not even working if the second slice
> returned a view, too).

Yep, it wouldn't work, you need to do some_array[2:5].copy().

I'm really sorry about forgetting about this not being standard Python. 
I think you are wrong in it being unprecedented: memoryview seems to 
support similar behaviour (although I can't quite figure it out and 
there's no docs on it?)

Just think of it as "slicing up the view", the data belonging to another 
object... all int[:]s are really views.

All the 3rd party numeric libraries has been doing this for ages, which 
I'd count as at least some "precedence".

BUT, this was one of the big reasons NumPy would never make it into core 
Python IIRC.

> This *may* make more sense for a SIMD type, but even there I suggest not
> using the slicing syntax for this. Make it a method call that returns a
> slice view.

That would be too unwieldy:

x.slice((3, 4, 5), (1, 2, 3))[...] = 4

rather than

x[3:4:5, 1:2:3] = 4

> I get the impression by now that CEP 517 and CEP 518 are not related. They
> describe completely different types, but only the array type makes sense in
> the context of Cython. The SIMD type would introduce semantics that exist
> neither in Python nor in C.

The fault is mine in not describing CEP 517. Efficient slices (and I 
meant view-creating slices, though that wasn't written down 
unfortunately) was a core motivating factor in CEP 517. If you copy then 
slices can't really be efficient...


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 18 10:36:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 10:36:05 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A39F296.6020907@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
Message-ID: <4A39FC75.8010908@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> -1 on slices returning views for a sequence type. That is completely
>> unprecedented in Python and would only lead to confusing and buggy code.

> I'm really sorry about forgetting about this not being standard Python. 
> I think you are wrong in it being unprecedented: memoryview seems to 
> support similar behaviour (although I can't quite figure it out and 
> there's no docs on it?)

Here's an example with memoryview:

 >>> A = array.array('i', [0,1,2,3])
 >>> M1 = memoryview(A)
 >>> M2 = M1[2:4]
 >>> M2[0]
b'\x02\x00\x00\x00'
 >>> A[2] = 7
 >>> M2[0]
b'\x07\x00\x00\x00'


So as you see, the slice taken is a view. I mean int[:] to behave like this.

About confusing and buggy code, many years of lots of people using NumPy 
speaks otherwise. I'm only suggesting the same semantics as in NumPy. 
(NumPy arrays can also be backed by different storage coming from other 
libraries etc.)

I spoke with Robert in IRC and he'll try in a new mail to get back to 
the high-level matters at stake (I'm too close to the subject matter to 
succeed it appears), so I'll stop here.

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu Jun 18 10:42:20 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Jun 2009 01:42:20 -0700
Subject: [Cython] Cython array type: Summary
In-Reply-To: <4A39EF15.7070608@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3921F7.7030404@student.matnat.uio.no>	<4A392315.50305@student.matnat.uio.no>
	<A8958E61-E37F-4D88-A9EB-0DBECAAAD97A@math.washington.edu>
	<4A39EF15.7070608@student.matnat.uio.no>
Message-ID: <8F2EB9C1-1836-468D-AE54-D4AE32AEF3BF@math.washington.edu>

I'm partially responsible, but this thread keeps digressing into  
implementation details. As a high level view, what do people think about

1) Some kind of SIMD array type being added to the language.

2) The given proposal, which is essentially a type with syntax like  
int[:,:] that would behave just like a the widely-used numpy array,  
including slicing and (eventually) broadcasting, but the underlying  
memory may be held by something other than an numpy object (e.g. it  
could be any other buffer supporting object, or a raw pointer) and  
not all numpy operations would be supported right away (or ever).

While we're on that note, what about

3) Some kind of type for memory-managed lists of C types (where + is  
concatenation, append is supported, memory managed, and nice  
conversion to/from python lists).


I am in favor of all three, and obviously Dag is pushing for (1) and  
(2).

- Robert


From stefan_ml at behnel.de  Thu Jun 18 10:47:13 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Jun 2009 10:47:13 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A39F296.6020907@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
Message-ID: <5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Stefan Behnel wrote:
>>>> I also read in CEP 517 that resizing is not to be supported. Why not?
>>>> It
>>>> could just fail with an exception when it notices that there are live
>>>> buffers on it.
>>> It just seemed kind of useless if only Cython-allocated memory can be
>>> resized, and not C arrays or memory in other Python objects. But we
>>> could do it for Cython-allocated memory, perhaps by extending PEP 3118
>>> with some Cython-specific flags etc.
>>
>> I don't understand what resizing has to do with PEP 3118 at all. You
>> obviously wouldn't resize a buffer view but the memory object itself.
>> The
>> only link is that resizing would fail if you reduced the size of a
>> memory
>> object while there is a live buffer view on it.
>
> I've been struggling to find the right terms for this the entire spring
> -- perhaps the right term for CEP 517 is "typed memoryviews", although
> which can allocate new memory when necesarry.

You keep adding features as we keep going. ;)

Then we have three types:

1) a dynamic array type
- allocates memory on creation
- reallocates on (explicit) resizing, e.g. a .resize() method
- supports PEP 3118 (and disables shrinking with live buffers)
- returns a typed value on indexing
- returns a typed array copy on slicing
- behaves like a tuple otherwise

2) a typed memory view
- created on top of a buffer (or array)
- never allocates memory (for data, that is)
- creates a new view object on slicing
- behaves like an array otherwise

3) a SIMD memory view
- created on top of a buffer, array or memory view
- supports parallel per-item arithmetic
- behaves like a memory view otherwise

There is also the potential of optimising simple array use cases into an
optimised array type that uses dynamic C++ allocation (for C++ code
generation, that is) and doesn't support resizing, but that is out of
scope for this proposal for now.

Does this make sense?

Stefan


From dagss at student.matnat.uio.no  Thu Jun 18 11:00:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 11:00:50 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <4A3A0242.2010005@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Stefan Behnel wrote:
>>>>> I also read in CEP 517 that resizing is not to be supported. Why not?
>>>>> It
>>>>> could just fail with an exception when it notices that there are live
>>>>> buffers on it.
>>>> It just seemed kind of useless if only Cython-allocated memory can be
>>>> resized, and not C arrays or memory in other Python objects. But we
>>>> could do it for Cython-allocated memory, perhaps by extending PEP 3118
>>>> with some Cython-specific flags etc.
>>> I don't understand what resizing has to do with PEP 3118 at all. You
>>> obviously wouldn't resize a buffer view but the memory object itself.
>>> The
>>> only link is that resizing would fail if you reduced the size of a
>>> memory
>>> object while there is a live buffer view on it.
>> I've been struggling to find the right terms for this the entire spring
>> -- perhaps the right term for CEP 517 is "typed memoryviews", although
>> which can allocate new memory when necesarry.
> 
> You keep adding features as we keep going. ;)
> 
> Then we have three types:
> 
> 1) a dynamic array type
> - allocates memory on creation
> - reallocates on (explicit) resizing, e.g. a .resize() method
> - supports PEP 3118 (and disables shrinking with live buffers)
> - returns a typed value on indexing
> - returns a typed array copy on slicing
> - behaves like a tuple otherwise
> 
> 2) a typed memory view
> - created on top of a buffer (or array)
> - never allocates memory (for data, that is)
> - creates a new view object on slicing
> - behaves like an array otherwise

This last point is dangerous as we seem to disagree about what an array is.

> 
> 3) a SIMD memory view
> - created on top of a buffer, array or memory view
> - supports parallel per-item arithmetic
> - behaves like a memory view otherwise

Good summary. Starting from this: I want int[:,:] to be the combination 
of 2) and 3) (the catch being I can't promise to implement the 3-part 
myself this time around, but I think somebody will pick it up eventually).

An issue which is not covered here is that when doing

cdef your-type-3) a, b, c
# assign b and c
a = b + c

then new memory must be created as the target of "b + c", and then 3) 
will be hard-coded to allocate something (which could likely be 1), but 
there will be no references to it anywhere it except through 3), so no 
resize can happen).

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 18 11:12:34 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 11:12:34 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A0242.2010005@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
Message-ID: <4A3A0502.1030408@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Stefan Behnel wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>> Stefan Behnel wrote:
>>>>>> I also read in CEP 517 that resizing is not to be supported. Why not?
>>>>>> It
>>>>>> could just fail with an exception when it notices that there are live
>>>>>> buffers on it.
>>>>> It just seemed kind of useless if only Cython-allocated memory can be
>>>>> resized, and not C arrays or memory in other Python objects. But we
>>>>> could do it for Cython-allocated memory, perhaps by extending PEP 3118
>>>>> with some Cython-specific flags etc.
>>>> I don't understand what resizing has to do with PEP 3118 at all. You
>>>> obviously wouldn't resize a buffer view but the memory object itself.
>>>> The
>>>> only link is that resizing would fail if you reduced the size of a
>>>> memory
>>>> object while there is a live buffer view on it.
>>> I've been struggling to find the right terms for this the entire spring
>>> -- perhaps the right term for CEP 517 is "typed memoryviews", although
>>> which can allocate new memory when necesarry.
>> You keep adding features as we keep going. ;)
>>
>> Then we have three types:
>>
>> 1) a dynamic array type
>> - allocates memory on creation
>> - reallocates on (explicit) resizing, e.g. a .resize() method
>> - supports PEP 3118 (and disables shrinking with live buffers)
>> - returns a typed value on indexing
>> - returns a typed array copy on slicing
>> - behaves like a tuple otherwise
>>
>> 2) a typed memory view
>> - created on top of a buffer (or array)
>> - never allocates memory (for data, that is)
>> - creates a new view object on slicing
>> - behaves like an array otherwise
> 
> This last point is dangerous as we seem to disagree about what an array is.
> 
>> 3) a SIMD memory view
>> - created on top of a buffer, array or memory view
>> - supports parallel per-item arithmetic
>> - behaves like a memory view otherwise
> 
> Good summary. Starting from this: I want int[:,:] to be the combination 
> of 2) and 3) (the catch being I can't promise to implement the 3-part 
> myself this time around, but I think somebody will pick it up eventually).
> 
> An issue which is not covered here is that when doing
> 
> cdef your-type-3) a, b, c
> # assign b and c
> a = b + c
> 
> then new memory must be created as the target of "b + c", and then 3) 
> will be hard-coded to allocate something (which could likely be 1), but 
> there will be no references to it anywhere it except through 3), so no 
> resize can happen).

While I'm at it, another confusing thing is that I proposed

cdef int[:,:] a = int[:100,:100]()

to mean "create an object of the type in your 1), then assign it to a 
view of the type in 2)&3)".

I'm quite ready to pull that part of the proposal if it seems confusing, 
requiring one to do

cdef int[:,:] a = cython.somearray[int]((100,100))

instead.

-- 
Dag Sverre

From stefan_ml at behnel.de  Thu Jun 18 11:13:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Jun 2009 11:13:02 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A0242.2010005@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
Message-ID: <ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> we have three types:
>>
>> 1) a dynamic array type
>> - allocates memory on creation
>> - reallocates on (explicit) resizing, e.g. a .resize() method
>> - supports PEP 3118 (and disables shrinking with live buffers)
>> - returns a typed value on indexing
>> - returns a typed array copy on slicing
>> - behaves like a tuple otherwise
>>
>> 2) a typed memory view
>> - created on top of a buffer (or array)
>> - never allocates memory (for data, that is)
>> - creates a new view object on slicing
>> - behaves like an array otherwise
>
> This last point is dangerous as we seem to disagree about what an array
> is.

It's what I described under 1).


>> 3) a SIMD memory view
>> - created on top of a buffer, array or memory view
>> - supports parallel per-item arithmetic
>> - behaves like a memory view otherwise
>
> Good summary. Starting from this: I want int[:,:] to be the combination
> of 2) and 3)

You mean "3) and not 2)", right? Could you explain why you need a syntax
for this if it's only a view?

I understand that the compiler needs to see the type, but I don't
understand why you need a syntax other than a known type name that wraps
'something'.


> An issue which is not covered here is that when doing
>
> cdef your-type-3) a, b, c
> # assign b and c
> a = b + c
>
> then new memory must be created as the target of "b + c", and then 3)
> will be hard-coded to allocate something (which could likely be 1), but
> there will be no references to it anywhere it except through 3), so no
> resize can happen).

Well, the plus operation of the SIMD type would create a new array (or, if
it is smart enough, something close to the input types), do the
calculation, wrap the result in a new SIMD memory view and return that, so
that it gets assigned to 'a'. Did you mean the question of determining the
target type, or why do you consider this tricky?

Stefan


From dagss at student.matnat.uio.no  Thu Jun 18 15:48:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 15:48:41 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <4A3A45B9.2000008@student.matnat.uio.no>

Sorry about the medium-sized length, but I'd like this to be close to my 
last email on the subject. I'd just refer to Robert's mail, but I guess 
some more explanation about NumPy semantics is in order for the benefit 
of non-NumPy-users, so I've made a summary of that.

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> we have three types:
>>>
>>> 1) a dynamic array type
>>> - allocates memory on creation
>>> - reallocates on (explicit) resizing, e.g. a .resize() method
>>> - supports PEP 3118 (and disables shrinking with live buffers)
>>> - returns a typed value on indexing
>>> - returns a typed array copy on slicing
>>> - behaves like a tuple otherwise
>>>
>>> 2) a typed memory view
>>> - created on top of a buffer (or array)
>>> - never allocates memory (for data, that is)
>>> - creates a new view object on slicing
>>> - behaves like an array otherwise
>> This last point is dangerous as we seem to disagree about what an array
>> is.
> 
> It's what I described under 1).
> 
> 
>>> 3) a SIMD memory view
>>> - created on top of a buffer, array or memory view
>>> - supports parallel per-item arithmetic
>>> - behaves like a memory view otherwise
>> Good summary. Starting from this: I want int[:,:] to be the combination
>> of 2) and 3)
> 
> You mean "3) and not 2)", right? Could you explain why you need a syntax
> for this if it's only a view?

I suppose I meant some variation of 3) with some extra bullet points 
(slicing in particular). We need a syntax because SIMD operations must 
be handled as a special-case compile-time.

Robert put it well; what I want is the core NumPy array semantics on a 
view to any array memory -- builtin, so that it can be optimized 
compile-time. We need to return to that; trying to distill something 
else and more generic out of this seems to only bring confusion.

(This is about 1) and 2) in Robert's mail only though.)

I'll make a list of what we mean by NumPy semantics below. At the very 
bottom is some things which I think should *not* be included.

First:

1) Nobody is claming this is elegant or Pythonic. It is catering for a 
numerical special interest, nothing more nor less.

2) As Robert put it: He won't use it himself, but the rest of Cython 
indirectly benefits from all the Cython interest from the numerical users.

3) The proposed semantics below are really not up for in-detail 
discussion, what I'm really after is a "yes" or "no" -- I just don't 
have the time, and NumPy is the de facto standard for Python numerics 
and what everybody expects anyway. I don't want to invent something 
entirely new.

That said, here's a long list of what I mean with NumPy semantics, 
assuming both CEPs are implemented.

# make x a compile-time-optimizeable 2D view on memoryview(obj)
cdef int[:,:] x = obj

# make an unassigned 1D view
cdef int[:] y

# Indexing
x[2,3]

# Access shape, stride info, raw data pointer
x.shape
x.strides
x.data

# Slicing out new view of third row (in two ways)
y = x[2,:]
y = x[2,...]

# Now, modifying y modifies what x points to too.
# Make a copy so that y points to seperate memory:
y = y.copy()

# Indexing with None creates new, 1-length axis
x = y[None, :] # x.shape == (1, y.shape[0])
x = y[:, None] # x.shape == (y.shape[0], 1)

# Now, this:
x[0, 3] = 2
# modifies y[3] too.

# New view of exactly same data
x[:,:]
x[...]

# Set all entries in array 12
x[...] = 12

# Set only first row to 10
x[0, :] = 10

# Some ways of multiplying all elements with 2
x *= 2
x[...] *= 2
x[:,:] *= 2
x += x
x[...] += x

# A more complicated expression...allocates memory
x = stdmath.sqrt(x*x + x*(x+1)/(x+2))

# A more complicated expression...overwrites existing
# memory
x[...] = stdmath.sqrt(x*x + x*(x+1)/(x+2))

# Boolean operators
cdef bint[:,:] b # perhaps we could support 8-bit bool too
b = (x == 2)
# b is now an array the shape of x, containing True where x[i,j] == 2

# Get sum of elements
import numpy as np
np.sum(x)

# As for printing/coercion to Python object, that remains
# TBD. Either memoryview, or a pretty-printing subclass
# of memoryview, implementing NumPy's __toarray__ protocol
# as well for better compatability

Here's what I do NOT want to include from NumPy:

# Get sum and mean
x.sum()
x.mean()
# and so on, you have to do np.sum(x).

# "Fancy indexing" is a mess because the returned object
# (due to implementation constraints) is a copy, not a view,
# thus being inconsistent with the above. My stance is that
# this can go in when we can support treating it as a view,
# instead of following NumPy with making a copy. I have ideas
# for how to do this.

# Get the intersecting array of rows 1, 4 and 5 and
# colums 2 and 1
new_data_copy = x[[1,4,5], [2,1]]

# Set the same intersection to 0. This is where NumPy gets
# really inconsistent; making an exception specifically
# in __setitem__ for this case.
x[[1,4,5], [2,1]] = 0
# modified x

# If y has length 3, pick out element 0 and 4
y[[True, False, False, True]]

...and so on.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 18 15:54:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 15:54:02 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A45B9.2000008@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
Message-ID: <4A3A46FA.7080200@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Sorry about the medium-sized length, but I'd like this to be close to my 
> last email on the subject. I'd just refer to Robert's mail, but I guess 
> some more explanation about NumPy semantics is in order for the benefit 
> of non-NumPy-users, so I've made a summary of that.
> 
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Stefan Behnel wrote:
>>>> we have three types:
>>>>
>>>> 1) a dynamic array type
>>>> - allocates memory on creation
>>>> - reallocates on (explicit) resizing, e.g. a .resize() method
>>>> - supports PEP 3118 (and disables shrinking with live buffers)
>>>> - returns a typed value on indexing
>>>> - returns a typed array copy on slicing
>>>> - behaves like a tuple otherwise
>>>>
>>>> 2) a typed memory view
>>>> - created on top of a buffer (or array)
>>>> - never allocates memory (for data, that is)
>>>> - creates a new view object on slicing
>>>> - behaves like an array otherwise
>>> This last point is dangerous as we seem to disagree about what an array
>>> is.
>> It's what I described under 1).
>>
>>
>>>> 3) a SIMD memory view
>>>> - created on top of a buffer, array or memory view
>>>> - supports parallel per-item arithmetic
>>>> - behaves like a memory view otherwise
>>> Good summary. Starting from this: I want int[:,:] to be the combination
>>> of 2) and 3)
>> You mean "3) and not 2)", right? Could you explain why you need a syntax
>> for this if it's only a view?
> 
> I suppose I meant some variation of 3) with some extra bullet points 
> (slicing in particular). We need a syntax because SIMD operations must 
> be handled as a special-case compile-time.
> 
> Robert put it well; what I want is the core NumPy array semantics on a 
> view to any array memory -- builtin, so that it can be optimized 
> compile-time. We need to return to that; trying to distill something 
> else and more generic out of this seems to only bring confusion.
> 
> (This is about 1) and 2) in Robert's mail only though.)
> 
> I'll make a list of what we mean by NumPy semantics below. At the very 
> bottom is some things which I think should *not* be included.
> 
> First:
> 
> 1) Nobody is claming this is elegant or Pythonic. It is catering for a 
> numerical special interest, nothing more nor less.
> 
> 2) As Robert put it: He won't use it himself, but the rest of Cython 
> indirectly benefits from all the Cython interest from the numerical users.
> 
> 3) The proposed semantics below are really not up for in-detail 
> discussion, what I'm really after is a "yes" or "no" -- I just don't 
> have the time, and NumPy is the de facto standard for Python numerics 
> and what everybody expects anyway. I don't want to invent something 
> entirely new.
> 

> # Some ways of multiplying all elements with 2
> x *= 2
> x[...] *= 2
> x[:,:] *= 2
> x += x
> x[...] += x

OK I'll make an exception here -- I'm willing to discuss whether we 
should depart from NumPy semantics here and let

x2 = x
x *= 2

allocate new memory, so that x2 is not modified, being consistent with a 
direct transformation to "x = x * 2". One can always write

x[...] *= 2

if one wishes to modify original memory.

-- 
Dag Sverre

From dalcinl at gmail.com  Thu Jun 18 16:06:33 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 18 Jun 2009 11:06:33 -0300
Subject: [Cython] Cython array type: Summary
In-Reply-To: <8F2EB9C1-1836-468D-AE54-D4AE32AEF3BF@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
	<4A3921F7.7030404@student.matnat.uio.no>
	<4A392315.50305@student.matnat.uio.no>
	<A8958E61-E37F-4D88-A9EB-0DBECAAAD97A@math.washington.edu>
	<4A39EF15.7070608@student.matnat.uio.no>
	<8F2EB9C1-1836-468D-AE54-D4AE32AEF3BF@math.washington.edu>
Message-ID: <e7ba66e40906180706l52931043k29463f0ed494ad4c@mail.gmail.com>

On Thu, Jun 18, 2009 at 5:42 AM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> I'm partially responsible, but this thread keeps digressing into
> implementation details. As a high level view, what do people think about
>
> 1) Some kind of SIMD array type being added to the language.
>
> 2) The given proposal, which is essentially a type with syntax like
> int[:,:] that would behave just like a the widely-used numpy array,
> including slicing and (eventually) broadcasting, but the underlying
> memory may be held by something other than an numpy object (e.g. it
> could be any other buffer supporting object, or a raw pointer) and
> not all numpy operations would be supported right away (or ever).
>

+1

Any chance that the new Cython (nd)array play nice with Python lists,
like in numpy, were you can pass a list and get it 'coerced' to
numpy.ndarray?

> While we're on that note, what about
>
> 3) Some kind of type for memory-managed lists of C types (where + is
> concatenation, append is supported, memory managed, and nice
> conversion to/from python lists).
>

I assume that by 'list of C types' you mean an plain, malloced C
array, right? Then 'append' would involve a realloc() call, right? In
such case, I'm +1 on this...

Just looking at mpi4py's sources, I could take advantage of this in many places.

>
> I am in favor of all three, and obviously Dag is pushing for (1) and
> (2).
>

I am also in favor of all three.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu Jun 18 18:05:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Jun 2009 09:05:32 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A46FA.7080200@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A46FA.7080200@student.matnat.uio.no>
Message-ID: <F0E48C9B-AAF8-47C8-8512-DAA2EF0B6505@math.washington.edu>

On Jun 18, 2009, at 6:54 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> Sorry about the medium-sized length, but I'd like this to be close  
>> to my
>> last email on the subject. I'd just refer to Robert's mail, but I  
>> guess
>> some more explanation about NumPy semantics is in order for the  
>> benefit
>> of non-NumPy-users, so I've made a summary of that.
>>
>> Stefan Behnel wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Stefan Behnel wrote:
>>>>> we have three types:
>>>>>
>>>>> 1) a dynamic array type
>>>>> - allocates memory on creation
>>>>> - reallocates on (explicit) resizing, e.g. a .resize() method
>>>>> - supports PEP 3118 (and disables shrinking with live buffers)
>>>>> - returns a typed value on indexing
>>>>> - returns a typed array copy on slicing
>>>>> - behaves like a tuple otherwise
>>>>>
>>>>> 2) a typed memory view
>>>>> - created on top of a buffer (or array)
>>>>> - never allocates memory (for data, that is)
>>>>> - creates a new view object on slicing
>>>>> - behaves like an array otherwise
>>>> This last point is dangerous as we seem to disagree about what  
>>>> an array
>>>> is.
>>> It's what I described under 1).
>>>
>>>
>>>>> 3) a SIMD memory view
>>>>> - created on top of a buffer, array or memory view
>>>>> - supports parallel per-item arithmetic
>>>>> - behaves like a memory view otherwise
>>>> Good summary. Starting from this: I want int[:,:] to be the  
>>>> combination
>>>> of 2) and 3)
>>> You mean "3) and not 2)", right? Could you explain why you need a  
>>> syntax
>>> for this if it's only a view?
>>
>> I suppose I meant some variation of 3) with some extra bullet points
>> (slicing in particular). We need a syntax because SIMD operations  
>> must
>> be handled as a special-case compile-time.
>>
>> Robert put it well; what I want is the core NumPy array semantics  
>> on a
>> view to any array memory -- builtin, so that it can be optimized
>> compile-time. We need to return to that; trying to distill something
>> else and more generic out of this seems to only bring confusion.
>>
>> (This is about 1) and 2) in Robert's mail only though.)
>>
>> I'll make a list of what we mean by NumPy semantics below. At the  
>> very
>> bottom is some things which I think should *not* be included.
>>
>> First:
>>
>> 1) Nobody is claming this is elegant or Pythonic. It is catering  
>> for a
>> numerical special interest, nothing more nor less.
>>
>> 2) As Robert put it: He won't use it himself, but the rest of Cython
>> indirectly benefits from all the Cython interest from the  
>> numerical users.
>>
>> 3) The proposed semantics below are really not up for in-detail
>> discussion, what I'm really after is a "yes" or "no" -- I just don't
>> have the time, and NumPy is the de facto standard for Python numerics
>> and what everybody expects anyway. I don't want to invent something
>> entirely new.
>>
>
>> # Some ways of multiplying all elements with 2
>> x *= 2
>> x[...] *= 2
>> x[:,:] *= 2
>> x += x
>> x[...] += x
>
> OK I'll make an exception here -- I'm willing to discuss whether we
> should depart from NumPy semantics here and let
>
> x2 = x
> x *= 2
>
> allocate new memory, so that x2 is not modified, being consistent  
> with a
> direct transformation to "x = x * 2". One can always write
>
> x[...] *= 2
>
> if one wishes to modify original memory.

Or we disallow inplace operators on these types, to reduce ambiguity  
and state that there's no exceptions other than unsupported behavior.

- Robert



From robertwb at math.washington.edu  Thu Jun 18 18:06:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Jun 2009 09:06:53 -0700
Subject: [Cython] Cython array type: Summary
In-Reply-To: <e7ba66e40906180706l52931043k29463f0ed494ad4c@mail.gmail.com>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
	<4A3921F7.7030404@student.matnat.uio.no>
	<4A392315.50305@student.matnat.uio.no>
	<A8958E61-E37F-4D88-A9EB-0DBECAAAD97A@math.washington.edu>
	<4A39EF15.7070608@student.matnat.uio.no>
	<8F2EB9C1-1836-468D-AE54-D4AE32AEF3BF@math.washington.edu>
	<e7ba66e40906180706l52931043k29463f0ed494ad4c@mail.gmail.com>
Message-ID: <25B9EF20-EAE7-4923-98D1-B4CCBCCF0AEE@math.washington.edu>

On Jun 18, 2009, at 7:06 AM, Lisandro Dalcin wrote:

> On Thu, Jun 18, 2009 at 5:42 AM, Robert
> Bradshaw<robertwb at math.washington.edu> wrote:
>> I'm partially responsible, but this thread keeps digressing into
>> implementation details. As a high level view, what do people think  
>> about
>>
>> 1) Some kind of SIMD array type being added to the language.
>>
>> 2) The given proposal, which is essentially a type with syntax like
>> int[:,:] that would behave just like a the widely-used numpy array,
>> including slicing and (eventually) broadcasting, but the underlying
>> memory may be held by something other than an numpy object (e.g. it
>> could be any other buffer supporting object, or a raw pointer) and
>> not all numpy operations would be supported right away (or ever).
>>
>
> +1
>
> Any chance that the new Cython (nd)array play nice with Python lists,
> like in numpy, were you can pass a list and get it 'coerced' to
> numpy.ndarray?

Yes, I think that's in the plan.

>> While we're on that note, what about
>>
>> 3) Some kind of type for memory-managed lists of C types (where + is
>> concatenation, append is supported, memory managed, and nice
>> conversion to/from python lists).
>>
>
> I assume that by 'list of C types' you mean an plain, malloced C
> array, right? Then 'append' would involve a realloc() call, right? In
> such case, I'm +1 on this...

Yep.

> Just looking at mpi4py's sources, I could take advantage of this in  
> many places.
>
>>
>> I am in favor of all three, and obviously Dag is pushing for (1) and
>> (2).
>>
>
> I am also in favor of all three.

Thanks for the input.

- Robert


From ondrej at certik.cz  Thu Jun 18 18:40:00 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Jun 2009 10:40:00 -0600
Subject: [Cython] force cython to do local imports
In-Reply-To: <23671C20-F766-48C3-853E-A22ACDACBEA6@math.washington.edu>
References: <85b5c3130906091103y6da30c11kf9829320fdd93c6@mail.gmail.com>
	<4A312FB6.7000300@behnel.de>
	<85b5c3130906110949v4468da02scc7ed5ac67e9387d@mail.gmail.com>
	<4A31368C.7040507@student.matnat.uio.no>
	<85b5c3130906111031g2bc25336q871ae4115b815d87@mail.gmail.com>
	<e7ba66e40906111131j51298967oe9a51b632d496988@mail.gmail.com>
	<4A314FB0.9090507@behnel.de>
	<85b5c3130906111202x3d0255f9o859b410277f1cdfc@mail.gmail.com>
	<85b5c3130906111207s44d5a715qfe91081566a0b5c2@mail.gmail.com>
	<23671C20-F766-48C3-853E-A22ACDACBEA6@math.washington.edu>
Message-ID: <85b5c3130906180940o4191e470p9e1d057b8368eae3@mail.gmail.com>

On Tue, Jun 16, 2009 at 11:13 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 11, 2009, at 12:07 PM, Ondrej Certik wrote:
>
>> On Thu, Jun 11, 2009 at 1:02 PM, Ondrej Certik<ondrej at certik.cz>
>> wrote:
>>> On Thu, Jun 11, 2009 at 12:40 PM, Stefan
>>> Behnel<stefan_ml at behnel.de> wrote:
>>>>
>>>> Lisandro Dalcin wrote:
>>>>> On Thu, Jun 11, 2009 at 2:31 PM, Ondrej Certik<ondrej at certik.cz>
>>>>> wrote:
>>>>>> As far as Cython is concerned, the only problem I can see is
>>>>>> that it
>>>>>> changes all import paths to the fully qualified import paths.
>>>>>> Python
>>>>>> allows both ways, Cython only the fully qualified way. As Stefan
>>>>>> explained, this is needed, because it will then not work
>>>>>> properly at
>>>>>> the moment. But I am not convinced this cannot be fixed --
>>>>>> though it
>>>>>> may happen that the fix is too tedious and not worthy.
>>>>>>
>>>>>
>>>>> IMHO, the only "fix" Cython needs is support for relative
>>>>> imports...
>>>>> Then you could write:
>>>>>
>>>>> from ._hermes2d cimport what_you_need
>>>>>
>>>>> This way, Ondrej would be able to easily write Cython code using
>>>>> relative imports the right way...
>>>>
>>>> I thought about this, too, and yes, that would be a nice feature.
>>>> But it
>>>> doesn't help in this case. What Ondrej wanted was to be able to
>>>> import the
>>>> module both from its package and as a plain import, i.e. when put
>>>> outside
>>>> of any package. The above would not handle the latter case either.
>>>
>>> I think it would. I have _forms.so and _hermes2d.so. The _hermes2d.so
>>> is imported from _forms.so and if the import was relative, e.g. if
>>> _forms.so was doing "import ._hermes2d", then it should work both in
>>> the tree and out of the tree.
>>
>> Looking here, it should be easy to support relative imports:
>>
>> http://docs.python.org/c-api/import.html
>>
>> so unless there is some difficulty inside Cython, that I don't see yet
>> (e.g. the kind of thing Stefan was talking about), it should be
>> possible to support relative imports quite easily, I already
>> understand the part that takes the import path from .pyx file and then
>> makes it fully qualified --- so I would simply change that.
>
> Cython does support relative cimports (we use them all over in Sage).
> What it does not support is deciding a modules fully qualified name
> at runtime because it uses this information at compile time. (It is
> possible that this could be changed, I haven't thought of what the
> repercussions should be).
>
> What I don't understand, in your setup, is how are you handling
>
> foo.py
> foo2.pyx
> a/__init__.py
> a/bar.pyx
> a/bar.pxd
>
> Now when foo.py does "from a.bar import *" or foo2.pyx does "from
> a.bar cimport *" how does this get resolved if the .so files are
> moved to the top level?

This imho will fail and that is ok with me. If one wants to move .so
files to the toplevel (or pythonpath), one needs to use relative
imports, e.g. create:

_bar.pyx

and do "from _bar import stuff" in bar.py, which sits in a/bar.py.


Ondrej

From kwmsmith at gmail.com  Thu Jun 18 19:10:48 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Thu, 18 Jun 2009 12:10:48 -0500
Subject: [Cython] function argument datatype size not resolved at runtime
	when its an external ctypedef
Message-ID: <c266a7930906181010y24c9ebc4n4b454b522c80f9cd@mail.gmail.com>

In the failing example below, it seems that the size of an argument type for a
cdef function is not resolved at runtime when that argument type is an extern
ctypedef.  All the arguments to bfn() are therefore ints, which is incorrect.
It seems that this falls under ticket 303's domain, since the user shouldn't
care about the 'true' size of a datatype, whether its used inside a function or
in the function's argument list.

external_defs.h:

[snip]
typedef char CharTypedef;
typedef short ShortTypedef;
typedef int IntTypedef;
typedef long LongTypedef;
#if defined(T_LONGLONG)
typedef PY_LONG_LONG LongLongTypedef;
#else
typedef long LongLongTypedef;
#endif
[snip]

bug_T1000.pyx:

"""
>>> test_ints()
"""

cdef extern from "external_defs.h":
   ctypedef int CharTypedef
   ctypedef int ShortTypedef
   ctypedef int IntTypedef
   ctypedef int LongTypedef
   ctypedef int LongLongTypedef

def test_ints():
   cdef CharTypedef ct      = 2**(8*sizeof(CharTypedef)-1)-1 # 127
   cdef ShortTypedef st     = 2**(8*sizeof(ShortTypedef)-1)-1 # 32767
   cdef IntTypedef it       = 2**(8*sizeof(IntTypedef)-1)-1  # 2147483647
   cdef LongTypedef lt      = 2**(8*sizeof(LongTypedef)-1)-1 #
processor dependent...
   cdef LongLongTypedef llt = 2**(8*sizeof(LongLongTypedef)-1)-1 #
processor dependent...
   assert (ct, st, it, lt, llt) == bfn(ct, st, it, lt, llt)

def bfn(CharTypedef ct, ShortTypedef st, IntTypedef it, LongTypedef
lt, LongLongTypedef llt):
   return (ct, st, it, lt, llt)

8<-------------------------------------------------------------------------------------

Running the above gives:

======================================================================
FAIL: Doctest: bug_T1000
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
   raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for bug_T1000
 File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/c/bug_T1000.so",
line 164, in bug_T1000

----------------------------------------------------------------------
File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/c/bug_T1000.so",
line 166, in bug_T1000
Failed example:
   test_ints()
Exception raised:
   Traceback (most recent call last):
     File "/usr/lib/python2.6/doctest.py", line 1241, in __run
       compileflags, 1) in test.globs
     File "<doctest bug_T1000[0]>", line 1, in <module>
       test_ints()
     File "bug_T1000.pyx", line 18, in bug_T1000.test_ints (bug_T1000.c:525)
     File "bug_T1000.pyx", line 20, in bug_T1000.bfn (bug_T1000.c:647)
   OverflowError: value too large to convert to int


======================================================================
FAIL: Doctest: bug_T1000
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
   raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for bug_T1000
 File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/cpp/bug_T1000.so",
line 115, in bug_T1000

----------------------------------------------------------------------
File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/cpp/bug_T1000.so",
line 117, in bug_T1000
Failed example:
   test_ints()
Exception raised:
   Traceback (most recent call last):
     File "/usr/lib/python2.6/doctest.py", line 1241, in __run
       compileflags, 1) in test.globs
     File "<doctest bug_T1000[0]>", line 1, in <module>
       test_ints()
     File "bug_T1000.pyx", line 18, in bug_T1000.test_ints (bug_T1000.cpp:525)
     File "bug_T1000.pyx", line 20, in bug_T1000.bfn (bug_T1000.cpp:647)
   OverflowError: value too large to convert to int


----------------------------------------------------------------------
Ran 4 tests in 0.757s

FAILED (failures=2)

From stefan_ml at behnel.de  Thu Jun 18 19:24:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Jun 2009 19:24:56 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A45B9.2000008@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
Message-ID: <4A3A7868.6080001@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Sorry about the medium-sized length, but I'd like this to be close to my 
> last email on the subject. I'd just refer to Robert's mail, but I guess 
> some more explanation about NumPy semantics is in order for the benefit 
> of non-NumPy-users, so I've made a summary of that.

You honestly call that a summary? ;)

Anyway, thanks for doing that, that was pretty interesting to see.


> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> Stefan Behnel wrote:
>>>> we have three types:
>>>>
>>>> 1) a dynamic array type
>>>> - allocates memory on creation
>>>> - reallocates on (explicit) resizing, e.g. a .resize() method
>>>> - supports PEP 3118 (and disables shrinking with live buffers)
>>>> - returns a typed value on indexing
>>>> - returns a typed array copy on slicing
>>>> - behaves like a tuple otherwise
>>>>
>>>> 2) a typed memory view
>>>> - created on top of a buffer (or array)
>>>> - never allocates memory (for data, that is)
>>>> - creates a new view object on slicing
>>>> - behaves like an array otherwise
>>> This last point is dangerous as we seem to disagree about what an array
>>> is.
>> It's what I described under 1).
>>
>>>> 3) a SIMD memory view
>>>> - created on top of a buffer, array or memory view
>>>> - supports parallel per-item arithmetic
>>>> - behaves like a memory view otherwise
>>> Good summary. Starting from this: I want int[:,:] to be the combination
>>> of 2) and 3)
>> You mean "3) and not 2)", right? Could you explain why you need a syntax
>> for this if it's only a view?
> 
> I suppose I meant some variation of 3) with some extra bullet points 
> (slicing in particular).

Erm, I though you wanted slicing to return views here? When I wrote "memory
view" in 3), I meant the memory view defined in 2). You might want to
re-read my post...


> We need a syntax because SIMD operations must 
> be handled as a special-case compile-time.

What I'm asking is why this can't work:

	cdef SIMD some_simd_value = SIMD(array[20,20])

or, if you need to add parameters, what about

	cdef SIMD[20,20, stride="whatever"] some_simd_value = \
			SIMD(array[20,20])

I'm not actually repeating the parameters here, the RHS is for creating the
array object at runtime, the LHS is for declaring the type of the SIMD to
Cython at compile time. Some variation of this should work, I think, and
has the advantage that it's clear where view objects are created. It's just
a lot more explicit.


> 1) Nobody is claming this is elegant or Pythonic. It is catering for a 
> numerical special interest, nothing more nor less.

Memory managed arrays and typed memory views have a much wider applicability.


> That said, here's a long list of what I mean with NumPy semantics, 
> assuming both CEPs are implemented.
> 
> # make x a compile-time-optimizeable 2D view on memoryview(obj)
> cdef int[:,:] x = obj

That's exactly what I'm questioning. This hurts my eyes in an almost
perlish way, and it's totally unclear that this type has SIMD properties.
Given that we wanted to have a template syntax anyway, I'd largely prefer

	cdef SIMD[dim=2] x = obj

If we ever allow typedefs for Cython types, users can just define their own
int2D type or whatever makes sense for them.


> # Indexing
> x[2,3]
> 
> # Access shape, stride info, raw data pointer
> x.shape
> x.strides
> x.data

Sure.


> # Slicing out new view of third row (in two ways)
> y = x[2,:]
> y = x[2,...]

Shouldn't there be one way to do it?


> # Now, modifying y modifies what x points to too.
> # Make a copy so that y points to seperate memory:
> y = y.copy()

That's ok, assuming that the type specific behaviour of y is obvious.


> # Set all entries in array 12
> x[...] = 12

Now, *that* is a weird syntax. This is called broadcasting, right?


> # Set only first row to 10
> x[0, :] = 10

I like that, although it conflicts with the above syntax for broadcasting.
Given the example above, I would have expected

 x[0, ...] = 10


> # Some ways of multiplying all elements with 2
> x *= 2
> x[...] *= 2
> x[:,:] *= 2
> x += x
> x[...] += x

Supporting this efficiently should be easy as it's a pure type feature.


> # A more complicated expression...allocates memory
> x = stdmath.sqrt(x*x + x*(x+1)/(x+2))
> 
> # A more complicated expression...overwrites existing
> # memory
> x[...] = stdmath.sqrt(x*x + x*(x+1)/(x+2))

Supporting this efficiently obviously requires compiler support. Otherwise,
we'd always end up with a copy in between.


> # Boolean operators
> cdef bint[:,:] b # perhaps we could support 8-bit bool too
> b = (x == 2)
> # b is now an array the shape of x, containing True where x[i,j] == 2

That's awful, but I guess it gets better when you think about it long enough.


> # Get sum of elements
> import numpy as np
> np.sum(x)
> 
> # As for printing/coercion to Python object, that remains
> # TBD. Either memoryview, or a pretty-printing subclass
> # of memoryview, implementing NumPy's __toarray__ protocol
> # as well for better compatability

Fine.


> Here's what I do NOT want to include from NumPy:
> 
> # Get sum and mean
> x.sum()
> x.mean()
> # and so on, you have to do np.sum(x).

Sure.


> # "Fancy indexing" is a mess because the returned object
> # (due to implementation constraints) is a copy, not a view,
> # thus being inconsistent with the above. My stance is that
> # this can go in when we can support treating it as a view,
> # instead of following NumPy with making a copy. I have ideas
> # for how to do this.
> 
> # Get the intersecting array of rows 1, 4 and 5 and
> # colums 2 and 1
> new_data_copy = x[[1,4,5], [2,1]]
> 
> # Set the same intersection to 0. This is where NumPy gets
> # really inconsistent; making an exception specifically
> # in __setitem__ for this case.
> x[[1,4,5], [2,1]] = 0
> # modified x
> 
> # If y has length 3, pick out element 0 and 4
> y[[True, False, False, True]]

Right so.



From stefan_ml at behnel.de  Thu Jun 18 19:31:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Jun 2009 19:31:02 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A46FA.7080200@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A46FA.7080200@student.matnat.uio.no>
Message-ID: <4A3A79D6.2030601@behnel.de>


Dag Sverre Seljebotn wrote:
>> # Some ways of multiplying all elements with 2
>> x *= 2
>> x[...] *= 2
>> x[:,:] *= 2
>> x += x
>> x[...] += x
> 
> OK I'll make an exception here -- I'm willing to discuss whether we 
> should depart from NumPy semantics here and let
> 
> x2 = x
> x *= 2
> 
> allocate new memory, so that x2 is not modified, being consistent with a 
> direct transformation to "x = x * 2". One can always write
> 
> x[...] *= 2
> 
> if one wishes to modify original memory.

What's wrong with

	x = 2 * x2

for doing a copy ?

x *= 2

pretty clearly states that I want to modify x in place.

Stefan

From dagss at student.matnat.uio.no  Thu Jun 18 20:57:46 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 20:57:46 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A7868.6080001@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>
Message-ID: <4A3A8E2A.7040607@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> Sorry about the medium-sized length, but I'd like this to be close to my 
>> last email on the subject. I'd just refer to Robert's mail, but I guess 
>> some more explanation about NumPy semantics is in order for the benefit 
>> of non-NumPy-users, so I've made a summary of that.
> 
> You honestly call that a summary? ;)
> 
> Anyway, thanks for doing that, that was pretty interesting to see.

It wasn't perfect as we'll see. Anyway, the best source of this is 
simply to read NumPy docs, this is a good start:

http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html

>>> You mean "3) and not 2)", right? Could you explain why you need a syntax
>>> for this if it's only a view?
>> I suppose I meant some variation of 3) with some extra bullet points 
>> (slicing in particular).
> 
> Erm, I though you wanted slicing to return views here? When I wrote "memory
> view" in 3), I meant the memory view defined in 2). You might want to
> re-read my post...

I misunderstood what you referred to with memory view. I won't even go 
into details...

>> We need a syntax because SIMD operations must 
>> be handled as a special-case compile-time.
> 
> What I'm asking is why this can't work:
> 
> 	cdef SIMD some_simd_value = SIMD(array[20,20])
> 
> or, if you need to add parameters, what about
> 
> 	cdef SIMD[20,20, stride="whatever"] some_simd_value = \
> 			SIMD(array[20,20])
> 
> I'm not actually repeating the parameters here, the RHS is for creating the
> array object at runtime, the LHS is for declaring the type of the SIMD to
> Cython at compile time. Some variation of this should work, I think, and
> has the advantage that it's clear where view objects are created. It's just
> a lot more explicit.

OK, I misunderstood you again. Breaking down what you say in two:

1) Syntax of SIMD view type. You propose SIMD[something] instead of 
basetype[something] -- that can obviously be made to work. As all syntax 
it is a matter of taste. I'll certainly go for whatever majority vote 
says here, it's not very important to me, but I vote for 
basetype[something] as it looks a lot more attractive for potential 
numerical users IMO.

2) Acquisition. You could rewrite your example with my syntax like e.g. 
this:

obj = cython_array_type[int](20, 20) # create array object
cdef int[:,:] some_simd
some_simd = int[:,:](obj)

IIUC you would just switch out int[:,:] with SIMD[...] to get something 
close to what you want.

Hmm. I actually like the explicitness that you propose. But it is a 
matter of verbosity/repeating, as replacing line 3 with "some_simd = 
obj" could only mean one thing. I'm +0 on this issue.

In pure Python mode, one would definitely have to do "some_simd = 
int[:,:](obj)" though.


>> # Slicing out new view of third row (in two ways)
>> y = x[2,:]
>> y = x[2,...]
> 
> Shouldn't there be one way to do it?

Sorry: "..." means ":" repeated "as many times as necessary" (including 
0 times). So if you have a 5D array,

x[2, ..., 2]

would mean the same as

x[2,:,:,:,2]

See NumPy docs for further questions here.

>> # Now, modifying y modifies what x points to too.
>> # Make a copy so that y points to seperate memory:
>> y = y.copy()
> 
> That's ok, assuming that the type specific behaviour of y is obvious.
> 
> 
>> # Set all entries in array 12
>> x[...] = 12
> 
> Now, *that* is a weird syntax. This is called broadcasting, right?

Well, I explained ... above, beyond that I can only say that it makes 
perfect sense when you actually use it. It is basically "SIMD assignment".

Broadcasting is something else, and is rules for what happens when you 
use SIMD operators or functions on arrays of different shape or 
dimensionality; this can be allowed in more situations by repeating the 
smaller array following certain rules. Seems wierd but *really*, 
*really* useful.

http://docs.scipy.org/doc/numpy/reference/ufuncs.html#broadcasting


>> # Set only first row to 10
>> x[0, :] = 10
> 
> I like that, although it conflicts with the above syntax for broadcasting.
> Given the example above, I would have expected
> 
>  x[0, ...] = 10

I hope the explanation of ... solved this.

> 
> 
>> # Some ways of multiplying all elements with 2
>> x *= 2
>> x[...] *= 2
>> x[:,:] *= 2
>> x += x
>> x[...] += x
> 
> Supporting this efficiently should be easy as it's a pure type feature.

Yes, but if we do the below it is essentially just a special case which 
it makes no sense to treat differently (scalars, like 2, are 
"broadcasted" up to entire arrays, then the rest is the same).

>> # A more complicated expression...allocates memory
>> x = stdmath.sqrt(x*x + x*(x+1)/(x+2))
>>
>> # A more complicated expression...overwrites existing
>> # memory
>> x[...] = stdmath.sqrt(x*x + x*(x+1)/(x+2))
> 
> Supporting this efficiently obviously requires compiler support. Otherwise,
> we'd always end up with a copy in between.

Yep, that's what NumPy does, and how I get my 3x speedup.

>> # Boolean operators
>> cdef bint[:,:] b # perhaps we could support 8-bit bool too
>> b = (x == 2)
>> # b is now an array the shape of x, containing True where x[i,j] == 2
> 
> That's awful, but I guess it gets better when you think about it long enough.

Again, extremely useful. To check for equality of two arrays, you do

np.any(x == y) # there is an element such that x[i] == y[i]

or

np.all(x == y) # for all i, x[i] == y[i]


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 18 21:08:08 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 21:08:08 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A79D6.2030601@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A46FA.7080200@student.matnat.uio.no>
	<4A3A79D6.2030601@behnel.de>
Message-ID: <4A3A9098.9090906@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>>> # Some ways of multiplying all elements with 2
>>> x *= 2
>>> x[...] *= 2
>>> x[:,:] *= 2
>>> x += x
>>> x[...] += x
>> OK I'll make an exception here -- I'm willing to discuss whether we 
>> should depart from NumPy semantics here and let
>>
>> x2 = x
>> x *= 2
>>
>> allocate new memory, so that x2 is not modified, being consistent with a 
>> direct transformation to "x = x * 2". One can always write
>>
>> x[...] *= 2
>>
>> if one wishes to modify original memory.
> 
> What's wrong with
> 
> 	x = 2 * x2
> 
> for doing a copy ?
> 
> x *= 2
> 
> pretty clearly states that I want to modify x in place.

The problem is that if you do

y = x
x *= 2

then in current NumPy, y and x will still point to the same memory and 
reference the same values (in fact, be the exact same view object).

Usual Python semantics seems to imply that what NumPy *should* have done 
is let the latter line mean "x = x * 2", where a new array is allocated 
and the value of x*2 copied into the new array, so that x and y points 
to different memory after the operation.

BTW, something I forgot to say about the == operator is that a common 
pattern is to do

x[x == 2] = 0

to replace 2 with 0 everywhere in x, that's one of its primary uses.

To make it more clear:

xeq2mask = (x == 2) # array of booleans
x[xeq2mask] = 0

(That is connected with the boolean indexing thing, which I'm vary about 
letting into Cython, although an exception can safely be made in the 
case of __setitem__).

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 18 21:11:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Jun 2009 21:11:10 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A9098.9090906@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A46FA.7080200@student.matnat.uio.no>	<4A3A79D6.2030601@behnel.de>
	<4A3A9098.9090906@student.matnat.uio.no>
Message-ID: <4A3A914E.9050805@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>>> # Some ways of multiplying all elements with 2
>>>> x *= 2
>>>> x[...] *= 2
>>>> x[:,:] *= 2
>>>> x += x
>>>> x[...] += x
>>> OK I'll make an exception here -- I'm willing to discuss whether we 
>>> should depart from NumPy semantics here and let
>>>
>>> x2 = x
>>> x *= 2
>>>
>>> allocate new memory, so that x2 is not modified, being consistent with a 
>>> direct transformation to "x = x * 2". One can always write
>>>
>>> x[...] *= 2
>>>
>>> if one wishes to modify original memory.
>> What's wrong with
>>
>> 	x = 2 * x2
>>
>> for doing a copy ?
>>
>> x *= 2
>>
>> pretty clearly states that I want to modify x in place.
> 
> The problem is that if you do
> 
> y = x
> x *= 2
> 
> then in current NumPy, y and x will still point to the same memory and 
> reference the same values (in fact, be the exact same view object).
> 
> Usual Python semantics seems to imply that what NumPy *should* have done 
> is let the latter line mean "x = x * 2", where a new array is allocated 
> and the value of x*2 copied into the new array, so that x and y points 
> to different memory after the operation.

I stand corrected:

In [13]: a = [1,2,3]

In [14]: b = a

In [15]: a += [1,2,3]

In [16]: b
Out[16]: [1, 2, 3, 1, 2, 3]

OK, so this seems like a non-issue.


-- 
Dag Sverre

From roland at utk.edu  Thu Jun 18 21:15:04 2009
From: roland at utk.edu (Roland Schulz)
Date: Thu, 18 Jun 2009 15:15:04 -0400
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A914E.9050805@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<4A39F296.6020907@student.matnat.uio.no> 
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A46FA.7080200@student.matnat.uio.no> 
	<4A3A79D6.2030601@behnel.de> <4A3A9098.9090906@student.matnat.uio.no> 
	<4A3A914E.9050805@student.matnat.uio.no>
Message-ID: <c93c21390906181215pb949fadi41d36d9f8bebd8ca@mail.gmail.com>

On Thu, Jun 18, 2009 at 3:11 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Dag Sverre Seljebotn wrote:
> > Stefan Behnel wrote:
> >> Dag Sverre Seljebotn wrote:
>
>
> I stand corrected:
>
> In [13]: a = [1,2,3]
>
> In [14]: b = a
>
> In [15]: a += [1,2,3]
>
> In [16]: b
> Out[16]: [1, 2, 3, 1, 2, 3]
>
> OK, so this seems like a non-issue.


interestingly :

 >>> a=(1,2,3)
>>> b=a
>>> b+=(4,)
>>> a
(1, 2, 3)


Roland

>
>
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090618/dec8c517/attachment.htm 

From robert.kern at gmail.com  Thu Jun 18 21:52:02 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 18 Jun 2009 14:52:02 -0500
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <c93c21390906181215pb949fadi41d36d9f8bebd8ca@mail.gmail.com>
References: <4A3672EC.90002@student.matnat.uio.no>	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A46FA.7080200@student.matnat.uio.no>
	<4A3A79D6.2030601@behnel.de>
	<4A3A9098.9090906@student.matnat.uio.no>
	<4A3A914E.9050805@student.matnat.uio.no>
	<c93c21390906181215pb949fadi41d36d9f8bebd8ca@mail.gmail.com>
Message-ID: <h1e5t3$u91$1@ger.gmane.org>

On 2009-06-18 14:15, Roland Schulz wrote:
>
>
> On Thu, Jun 18, 2009 at 3:11 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no
> <mailto:dagss at student.matnat.uio.no>> wrote:
>
>     Dag Sverre Seljebotn wrote:
>      > Stefan Behnel wrote:
>      >> Dag Sverre Seljebotn wrote:
>
>
>     I stand corrected:
>
>     In [13]: a = [1,2,3]
>
>     In [14]: b = a
>
>     In [15]: a += [1,2,3]
>
>     In [16]: b
>     Out[16]: [1, 2, 3, 1, 2, 3]
>
>     OK, so this seems like a non-issue.
>
>
> interestingly :
>
>  >>> a=(1,2,3)
>  >>> b=a
>  >>> b+=(4,)
>  >>> a
> (1, 2, 3)

It's not that interesting. The augmented assignment operators "x ?= y" are 
defined to behave like "x = x ? y" except for *possible* in-place semantics. 
When the object is mutable, the operation is usually in-place. When the object 
is not mutable, then it is never in-place.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From dalcinl at gmail.com  Thu Jun 18 22:35:36 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 18 Jun 2009 17:35:36 -0300
Subject: [Cython] function argument datatype size not resolved at
	runtime when its an external ctypedef
In-Reply-To: <c266a7930906181010y24c9ebc4n4b454b522c80f9cd@mail.gmail.com>
References: <c266a7930906181010y24c9ebc4n4b454b522c80f9cd@mail.gmail.com>
Message-ID: <e7ba66e40906181335s77a8f2c4n6ec061664e0f2295@mail.gmail.com>

Kurt, I have a preliminary (working) patch for this... However, I'm
stacked about a mess issue in the codebase related to
'{from|to}_py_function' class members and the
'create_{from|to}_py_utility_code()' methods, all of them defined in
PyrexTypes, and used in other parts...

Until this mess (well, It seems a mess to me, though perhaps I'm
confused ) get fixed (I'm certainly in doubt about what to do), I
guess my patch will break something...


Robert, Have you anything to add? Can you imagine a possible way of
refactoring all that?


On Thu, Jun 18, 2009 at 2:10 PM, Kurt Smith<kwmsmith at gmail.com> wrote:
> In the failing example below, it seems that the size of an argument type for a
> cdef function is not resolved at runtime when that argument type is an extern
> ctypedef. ?All the arguments to bfn() are therefore ints, which is incorrect.
> It seems that this falls under ticket 303's domain, since the user shouldn't
> care about the 'true' size of a datatype, whether its used inside a function or
> in the function's argument list.
>
> external_defs.h:
>
> [snip]
> typedef char CharTypedef;
> typedef short ShortTypedef;
> typedef int IntTypedef;
> typedef long LongTypedef;
> #if defined(T_LONGLONG)
> typedef PY_LONG_LONG LongLongTypedef;
> #else
> typedef long LongLongTypedef;
> #endif
> [snip]
>
> bug_T1000.pyx:
>
> """
>>>> test_ints()
> """
>
> cdef extern from "external_defs.h":
> ? ctypedef int CharTypedef
> ? ctypedef int ShortTypedef
> ? ctypedef int IntTypedef
> ? ctypedef int LongTypedef
> ? ctypedef int LongLongTypedef
>
> def test_ints():
> ? cdef CharTypedef ct ? ? ?= 2**(8*sizeof(CharTypedef)-1)-1 # 127
> ? cdef ShortTypedef st ? ? = 2**(8*sizeof(ShortTypedef)-1)-1 # 32767
> ? cdef IntTypedef it ? ? ? = 2**(8*sizeof(IntTypedef)-1)-1 ?# 2147483647
> ? cdef LongTypedef lt ? ? ?= 2**(8*sizeof(LongTypedef)-1)-1 #
> processor dependent...
> ? cdef LongLongTypedef llt = 2**(8*sizeof(LongLongTypedef)-1)-1 #
> processor dependent...
> ? assert (ct, st, it, lt, llt) == bfn(ct, st, it, lt, llt)
>
> def bfn(CharTypedef ct, ShortTypedef st, IntTypedef it, LongTypedef
> lt, LongLongTypedef llt):
> ? return (ct, st, it, lt, llt)
>
> 8<-------------------------------------------------------------------------------------
>
> Running the above gives:
>
> ======================================================================
> FAIL: Doctest: bug_T1000
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
> ? raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for bug_T1000
> ?File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/c/bug_T1000.so",
> line 164, in bug_T1000
>
> ----------------------------------------------------------------------
> File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/c/bug_T1000.so",
> line 166, in bug_T1000
> Failed example:
> ? test_ints()
> Exception raised:
> ? Traceback (most recent call last):
> ? ? File "/usr/lib/python2.6/doctest.py", line 1241, in __run
> ? ? ? compileflags, 1) in test.globs
> ? ? File "<doctest bug_T1000[0]>", line 1, in <module>
> ? ? ? test_ints()
> ? ? File "bug_T1000.pyx", line 18, in bug_T1000.test_ints (bug_T1000.c:525)
> ? ? File "bug_T1000.pyx", line 20, in bug_T1000.bfn (bug_T1000.c:647)
> ? OverflowError: value too large to convert to int
>
>
> ======================================================================
> FAIL: Doctest: bug_T1000
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> ?File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
> ? raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for bug_T1000
> ?File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/cpp/bug_T1000.so",
> line 115, in bug_T1000
>
> ----------------------------------------------------------------------
> File "/home/ksmith/GSoC/cython-devel-303-reg/BUILD/run/cpp/bug_T1000.so",
> line 117, in bug_T1000
> Failed example:
> ? test_ints()
> Exception raised:
> ? Traceback (most recent call last):
> ? ? File "/usr/lib/python2.6/doctest.py", line 1241, in __run
> ? ? ? compileflags, 1) in test.globs
> ? ? File "<doctest bug_T1000[0]>", line 1, in <module>
> ? ? ? test_ints()
> ? ? File "bug_T1000.pyx", line 18, in bug_T1000.test_ints (bug_T1000.cpp:525)
> ? ? File "bug_T1000.pyx", line 20, in bug_T1000.bfn (bug_T1000.cpp:647)
> ? OverflowError: value too large to convert to int
>
>
> ----------------------------------------------------------------------
> Ran 4 tests in 0.757s
>
> FAILED (failures=2)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu Jun 18 23:21:13 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 18 Jun 2009 18:21:13 -0300
Subject: [Cython] function argument datatype size not resolved at
	runtime when its an external ctypedef
In-Reply-To: <e7ba66e40906181335s77a8f2c4n6ec061664e0f2295@mail.gmail.com>
References: <c266a7930906181010y24c9ebc4n4b454b522c80f9cd@mail.gmail.com>
	<e7ba66e40906181335s77a8f2c4n6ec061664e0f2295@mail.gmail.com>
Message-ID: <e7ba66e40906181421g28cc12f2u3909bbd8dfb851c8@mail.gmail.com>

On Thu, Jun 18, 2009 at 5:35 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
>
> Robert, Have you anything to add? Can you imagine a possible way of
> refactoring all that?
>

Robert, to put it clear... What I need to be sure of is the following...

At the point "some_type.{from|to}_py_function" attribute is accessed,
Do I have the guarantee that
"some_type.create_{from|to}_py_utility_code(env)" was previously
called in the chain? Could you make the the favor to check that? I got
really lost in the codebase trying to verify that ....



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Fri Jun 19 02:43:09 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 19 Jun 2009 12:43:09 +1200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A46FA.7080200@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A46FA.7080200@student.matnat.uio.no>
Message-ID: <4A3ADF1D.4080708@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> One can always write
> 
> x[...] *= 2
> 
> if one wishes to modify original memory.

Although unless some optimisation is done on this, it
won't be quite as efficient, since it will take a slice,
in-place modify the slice and then assign the slice
back to itself.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Fri Jun 19 03:27:03 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 19 Jun 2009 13:27:03 +1200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A9098.9090906@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A46FA.7080200@student.matnat.uio.no> <4A3A79D6.2030601@behnel.de>
	<4A3A9098.9090906@student.matnat.uio.no>
Message-ID: <4A3AE967.1010004@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> The problem is that if you do
> 
> y = x
> x *= 2
> 
> then in current NumPy, y and x will still point to the same memory and 
> reference the same values (in fact, be the exact same view object).
> 
> Usual Python semantics seems to imply that what NumPy *should* have done 
> is let the latter line mean "x = x * 2"

I don't follow that. Usual Python semantics for in-place
operations on mutable types also leaves x and y referring
to the same object. Given the way Numpy defines the *
operator, it's doing exactly what I would expect here.

-- 
Greg

From brett.calcott at gmail.com  Fri Jun 19 03:28:08 2009
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 19 Jun 2009 11:28:08 +1000
Subject: [Cython] Cython array type: Summary
In-Reply-To: <8F2EB9C1-1836-468D-AE54-D4AE32AEF3BF@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu> 
	<4A392068.6070800@student.matnat.uio.no>
	<4A3921F7.7030404@student.matnat.uio.no> 
	<4A392315.50305@student.matnat.uio.no>
	<A8958E61-E37F-4D88-A9EB-0DBECAAAD97A@math.washington.edu> 
	<4A39EF15.7070608@student.matnat.uio.no>
	<8F2EB9C1-1836-468D-AE54-D4AE32AEF3BF@math.washington.edu>
Message-ID: <c314d7d20906181828p1f15afc3v45f1247fefa6f1a0@mail.gmail.com>

>
> 1) Some kind of SIMD array type being added to the language.
>
> 2) The given proposal, which is essentially a type with syntax like
> int[:,:] that would behave just like a the widely-used numpy array,
> including slicing and (eventually) broadcasting, but the underlying
> memory may be held by something other than an numpy object (e.g. it
> could be any other buffer supporting object, or a raw pointer) and
> not all numpy operations would be supported right away (or ever).
>
> While we're on that note, what about
>
> 3) Some kind of type for memory-managed lists of C types (where + is
> concatenation, append is supported, memory managed, and nice
> conversion to/from python lists).
>
>
> I am in favor of all three, and obviously Dag is pushing for (1) and
> (2).
>

These additions would be fantastic. Any chance there would be support
for something like numpy's records?

Brett

From ondrej at certik.cz  Fri Jun 19 03:29:53 2009
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Jun 2009 19:29:53 -0600
Subject: [Cython] problem with pure python mode
In-Reply-To: <7B1321FB-58A6-4CA3-927A-54FA609A0336@math.washington.edu>
References: <85b5c3130904291155j59bc2335r66e446ca5f0951e6@mail.gmail.com>
	<B2F42569-AD37-473D-9809-F9C808DD3F47@math.washington.edu>
	<85b5c3130904291319p6da55380ra4c7ac937cdcdf10@mail.gmail.com>
	<85b5c3130904291449s305b00c6w17b8061a46fc730@mail.gmail.com>
	<12138492-389B-44AD-8AD1-8E43BDA3646F@math.washington.edu>
	<85b5c3130906101745r1ac9d4a0k7a955388c6322da5@mail.gmail.com>
	<7B1321FB-58A6-4CA3-927A-54FA609A0336@math.washington.edu>
Message-ID: <85b5c3130906181829u330f95edw6339c719cd724671@mail.gmail.com>

On Tue, Jun 16, 2009 at 11:22 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 10, 2009, at 5:45 PM, Ondrej Certik wrote:
>
>> On Wed, Apr 29, 2009 at 4:00 PM, Robert
>> Bradshaw<robertwb at math.washington.edu> wrote:
>>> On Apr 29, 2009, at 2:49 PM, Ondrej Certik wrote:
>>>
>>>>>>
>>>>>> Not sure--have you tried without the locals decorator in the
>>>>>> Python
>>>>>> source? I hope this isn't a regression.
>>>>>
>>>>> Yes, as you can see, the locals is commented out. I'll try to
>>>>> bisect
>>>>> it to see which commit broke it.
>>>>
>>>> Ok, so it never worked. I am a bit confused, because I thought the
>>>> pure python mode worked fine for me before.
>>>>
>>>> In the current cython --- is there any way to keep a pure python
>>>> code,
>>>> and create an accompanying pxd file that would annotate the cdef
>>>> functions and classes?
>>>
>>> This should work, that's how I implemented this a long time ago. If
>>> it's not working now it's because something broke (or, I'm not
>>> promising that it's completely bug-free, but it should be good--we
>>> use it to bootstrap Cython itself).
>>
>> So I made it to work by observing how things are done in Cython
>> itself.
>>
>> If you do:
>>
>> git clone git://github.com/certik/cython-test.git
>> cd cython-test
>> cython fact.pyx
>>
>> then it produces nicely optimized fact.c file. However, if you do:
>>
>> cython fact.py
>>
>> it produces a fact.c, that compiles, but it is not optimized --- the
>> "i" and "r" variables are python objects, not ints, even though my
>> fact.pxd file contains:
>>
>> import cython
>>
>> @cython.locals(i=cython.int, r=cython.int)
>> cdef int factorial(int n)
>>
>>
>> Do you see anything wrong with my files? Is this supposed to work?
>>
>> I think it is not implemented yet, because when I looked at the cython
>> sources, it doesn't work there either. This is the relevant part of
>> Scanning.pxd:
>>
>> ? ? @cython.locals(current_level=cython.long, new_level=cython.long)
>> ? ? cpdef indentation_action(self, text)
>>
>> and here is the declaration in the generated Scanning.c:
>>
>> ? PyObject *__pyx_v_new_level;
>>
>>
>> And consider for example the following chunk:
>>
>> ? /* "/home/ondrej/repos/cython-devel/Cython/Compiler/Scanning.py":398
>> ?* ? ? ? ? new_level = len(text)
>> ?* ? ? ? ? #print "Changing indent level from", current_level, "to",
>> new_level ###
>> ?* ? ? ? ? if new_level == current_level: ? ? ? ? ? ? # <<<<<<<<<<<<<<
>> ?* ? ? ? ? ? ? return
>> ?* ? ? ? ? elif new_level > current_level:
>> ?*/
>> ? __pyx_t_4 = PyObject_RichCompare(__pyx_v_new_level,
>> __pyx_v_current_level, Py_EQ); if (unlikely(!__pyx_t_4))
>> {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno =
>> __LINE__; goto __pyx_L1_error;}
>> ? __Pyx_GOTREF(__pyx_t_4);
>> ? __pyx_t_3 = __Pyx_PyObject_IsTrue(__pyx_t_4); if (unlikely(__pyx_t_3
>> < 0)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 398; __pyx_clineno
>> = __LINE__; goto __pyx_L1_error;}
>> ? __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
>> ? if (__pyx_t_3) {
>>
>>
>> both the new_level and current_level are declared as long, but above
>> you can see that Cython is calling lots of Python C/API methods...
>
> Looks like a bug to me.
>
> http://trac.cython.org/cython_trac/ticket/330

Do you have some idea where the problem is? You wrote "Parsing" in the
ticket. This pure Python mode is very important for me, so I tried to
get it fixed, but I need to invest more time to learn Cython internals
to produce something useful, hopefully over the weekend I'll find time
to dig into it.

Ondrej

From stefan_ml at behnel.de  Fri Jun 19 06:36:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 06:36:30 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3A8E2A.7040607@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no>
Message-ID: <4A3B15CE.2090104@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> OK, I misunderstood you again. Breaking down what you say in two:
> 
> 1) Syntax of SIMD view type. You propose SIMD[something] instead of 
> basetype[something] -- that can obviously be made to work. As all syntax 
> it is a matter of taste. I'll certainly go for whatever majority vote 
> says here, it's not very important to me, but I vote for 
> basetype[something] as it looks a lot more attractive for potential 
> numerical users IMO.
> 
> 2) Acquisition. You could rewrite your example with my syntax like e.g. 
> this:
> 
> obj = cython_array_type[int](20, 20) # create array object
> cdef int[:,:] some_simd
> some_simd = int[:,:](obj)
> 
> IIUC you would just switch out int[:,:] with SIMD[...] to get something 
> close to what you want.
> 
> Hmm. I actually like the explicitness that you propose. But it is a 
> matter of verbosity/repeating, as replacing line 3 with "some_simd = 
> obj" could only mean one thing. I'm +0 on this issue.
> 
> In pure Python mode, one would definitely have to do "some_simd = 
> int[:,:](obj)" though.

Explicitness was a major reason for my counterproposal, yes. And I think
the same would work in Python, where the SIMD type would be used to wrap a
buffer/array object.

The main reason was that I would prefer avoiding syntax overloading all
over the place. Type specialisation is a new concept that is expected to
use the [] syntax. We have the same syntax for buffers (which is also a
kind of specialisation). We will use the same syntax for the future array
type, which is at least closely related to buffers and should use C array
syntax anyway. Having the same syntax for memory views and SIMD views
sounds like an awful lot of things and a very unnecessary addition, since
the same can be achieved using the normal type instantiation syntax and
type specialisation on the SIMD/memory-view *type* (note the difference to
using that syntax on just *any* type). The [] syntax makes a lot of sense
for the new array type, it makes mostly sense for buffers, it was accepted
(IIRC) for type specialisation. IMHO, that's a long enough distance that we
go down that road. Going further would make the language less parseable to
humans.

I stripped the rest of your post with which I'm mostly fine. If we get a
new SIMD type, why not give it real SIMD semantics, even if they may seem a
bit weird at times.

Stefan


From robertwb at math.washington.edu  Fri Jun 19 09:06:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 00:06:51 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B15CE.2090104@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>
Message-ID: <D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>

On Jun 18, 2009, at 9:36 PM, Stefan Behnel wrote:

> Hi,
>
> Dag Sverre Seljebotn wrote:
>> OK, I misunderstood you again. Breaking down what you say in two:
>>
>> 1) Syntax of SIMD view type. You propose SIMD[something] instead of
>> basetype[something] -- that can obviously be made to work. As all  
>> syntax
>> it is a matter of taste. I'll certainly go for whatever majority vote
>> says here, it's not very important to me, but I vote for
>> basetype[something] as it looks a lot more attractive for potential
>> numerical users IMO.
>>
>> 2) Acquisition. You could rewrite your example with my syntax like  
>> e.g.
>> this:
>>
>> obj = cython_array_type[int](20, 20) # create array object
>> cdef int[:,:] some_simd
>> some_simd = int[:,:](obj)
>>
>> IIUC you would just switch out int[:,:] with SIMD[...] to get  
>> something
>> close to what you want.
>>
>> Hmm. I actually like the explicitness that you propose. But it is a
>> matter of verbosity/repeating, as replacing line 3 with "some_simd =
>> obj" could only mean one thing. I'm +0 on this issue.
>>
>> In pure Python mode, one would definitely have to do "some_simd =
>> int[:,:](obj)" though.
>
> Explicitness was a major reason for my counterproposal, yes. And I  
> think
> the same would work in Python, where the SIMD type would be used to  
> wrap a
> buffer/array object.
>
> The main reason was that I would prefer avoiding syntax overloading  
> all
> over the place. Type specialisation is a new concept that is  
> expected to
> use the [] syntax. We have the same syntax for buffers (which is  
> also a
> kind of specialisation). We will use the same syntax for the future  
> array
> type, which is at least closely related to buffers and should use C  
> array
> syntax anyway. Having the same syntax for memory views and SIMD views
> sounds like an awful lot of things and a very unnecessary addition,  
> since
> the same can be achieved using the normal type instantiation syntax  
> and
> type specialisation on the SIMD/memory-view *type* (note the  
> difference to
> using that syntax on just *any* type). The [] syntax makes a lot of  
> sense
> for the new array type, it makes mostly sense for buffers, it was  
> accepted
> (IIRC) for type specialisation. IMHO, that's a long enough distance  
> that we
> go down that road. Going further would make the language less  
> parseable to
> humans.

IIRC part of the motivation for int[:,:] syntax rather than  
special_name[int, ndim=2] was that the former is actually easier for  
humans to parse. Personally, it seems a natural extension of the int 
[50] syntax. It also makes it very clear to me that this is not some  
ordinary type implemented in some library, but a builtin type (like  
int[50]) that the compiler implements internally. This new type isn't  
really an object, it's a multi-dimensional set of ints, the same way  
int[10] or int* is a one-dimensional set of ints.

Note that just implementing an extension class Foo, and using the  
buffer syntax on Foo is insufficient (subtypes (which may override  
Foo's behavior) are disallowed, compile-time slicing and arithmetic  
can't be performed without special magic, an actual, potentially  
redundant, instance of Foo would have to be created). Since the SMID  
type can't be a cdef class, I think it's an advantage to not have it  
look like it is one.

- Robert


From stefan_ml at behnel.de  Fri Jun 19 09:28:05 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 09:28:05 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
Message-ID: <c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Jun 18, 2009, at 9:36 PM, Stefan Behnel wrote:
>> Having the same syntax
>> for memory views and SIMD views sounds like an awful lot of things
>> and a very unnecessary addition, since the same can be achieved using
>> the normal type instantiation syntax and type specialisation on the
>> SIMD/memory-view *type* (note the difference to using that syntax on
>> just *any* type). The [] syntax makes a lot of sense for the new
>> array type, it makes mostly sense for buffers, it was accepted
>> (IIRC) for type specialisation. IMHO, that's a long enough distance
>> that we go down that road. Going further would make the language
>> less parseable to humans.
>
> IIRC part of the motivation for int[:,:] syntax rather than
> special_name[int, ndim=2] was that the former is actually easier for
> humans to parse. Personally, it seems a natural extension of the int
> [50] syntax.

It behaves very different, though. For one, it's not a memory allocated
data object but a *view* on such a thing, just like the typed memory view.
What syntax would you propose for the memory view, and if none, why not?

> It also makes it very clear to me that this is not some
> ordinary type implemented in some library, but a builtin type (like
> int[50]) that the compiler implements internally.

That's an implementation detail, not a language 'feature'.

> This new type isn't
> really an object, it's a multi-dimensional set of ints, the same way
> int[10] or int* is a one-dimensional set of ints.

No, that would be the array type, which has no SIMD behaviour.

> Note that just implementing an extension class Foo, and using the
> buffer syntax on Foo is insufficient (subtypes (which may override
> Foo's behavior) are disallowed,

Fine, trying to do so is a compile time error then. Give it a good error
message saying you're "sorry but performance dictates an immutable type
here" and every user will be happy.

> compile-time slicing and arithmetic can't be performed without
> special magic, an actual, potentially redundant, instance of Foo
> would have to be created). Since the SMID type can't be a cdef
> class, I think it's an advantage to not have it look like it is one.

Details, details.

Stefan


From robertwb at math.washington.edu  Fri Jun 19 10:32:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 01:32:11 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>

On Jun 19, 2009, at 12:28 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Jun 18, 2009, at 9:36 PM, Stefan Behnel wrote:
>>> Having the same syntax
>>> for memory views and SIMD views sounds like an awful lot of things
>>> and a very unnecessary addition, since the same can be achieved  
>>> using
>>> the normal type instantiation syntax and type specialisation on the
>>> SIMD/memory-view *type* (note the difference to using that syntax on
>>> just *any* type). The [] syntax makes a lot of sense for the new
>>> array type, it makes mostly sense for buffers, it was accepted
>>> (IIRC) for type specialisation. IMHO, that's a long enough distance
>>> that we go down that road. Going further would make the language
>>> less parseable to humans.
>>
>> IIRC part of the motivation for int[:,:] syntax rather than
>> special_name[int, ndim=2] was that the former is actually easier for
>> humans to parse. Personally, it seems a natural extension of the int
>> [50] syntax.
>
> It behaves very different, though. For one, it's not a memory  
> allocated
> data object but a *view* on such a thing, just like the typed  
> memory view.
> What syntax would you propose for the memory view, and if none, why  
> not?

The proposal is to give our memory view types SIMD semantics, not  
have separate types with different semantics that one would have to  
cast between. Buffers and memory views are mostly abstractions and  
generalizations of ndarray that have been added to Python. What we  
want is to be able to treat these objects as one would treat numpy  
arrays, but with more room for optimization and more flexibility in  
the back end.

>> It also makes it very clear to me that this is not some
>> ordinary type implemented in some library, but a builtin type (like
>> int[50]) that the compiler implements internally.
>
> That's an implementation detail, not a language 'feature'.
>
>> This new type isn't
>> really an object, it's a multi-dimensional set of ints, the same way
>> int[10] or int* is a one-dimensional set of ints.
>
> No, that would be the array type, which has no SIMD behaviour.

int[], int[10], int*, and int[:] would be distinct types with  
different behaviors, but I think they have more in common (from a use  
point of view) than SomeType[int].

>> Note that just implementing an extension class Foo, and using the
>> buffer syntax on Foo is insufficient (subtypes (which may override
>> Foo's behavior) are disallowed,
>
> Fine, trying to do so is a compile time error then. Give it a good  
> error
> message saying you're "sorry but performance dictates an immutable  
> type
> here" and every user will be happy.

Perhaps we could add a final keyword to methods, but it would be  
messy to disallow Python subclasses, and still wouldn't allow the  
kind of optimization that can be done if the compiler knows that it's  
a SIMD type.

>> compile-time slicing and arithmetic can't be performed without
>> special magic, an actual, potentially redundant, instance of Foo
>> would have to be created). Since the SMID type can't be a cdef
>> class, I think it's an advantage to not have it look like it is one.
>
> Details, details.


I don't think it's just details, I think it's an important user  
interface issue. The syntax changes how one thinks about these  
things, and also impacts readability.

Trying to understand your point of view, what of the following are  
you against:

1) Any compiler/language support for memory views and SIMD---they  
should be a library requiring no support from the compiler or language.

2) A magic extension class that the compiler knows about and inlines/ 
emulates operations for optimization purposes, e.g. CythonArray[int,  
ndim=2].

3) Same as the above, CythonArray is really just a keyword--there's  
no actual extension class backing it since the compiler encapsulates  
its logic anyways.

4) Syntactic sugar like int[:,:] which means the same thing as  
CythonArray[int, ndim=2].

5) The syntactic sugar is the new syntax, rather than introducing the  
CythonArray keyword.

- Robert


From stefan_ml at behnel.de  Fri Jun 19 12:09:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 12:09:02 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>
Message-ID: <5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On Jun 19, 2009, at 12:28 AM, Stefan Behnel wrote:
>> Robert Bradshaw wrote:
>>> On Jun 18, 2009, at 9:36 PM, Stefan Behnel wrote:
>>>> Having the same syntax
>>>> for memory views and SIMD views sounds like an awful lot of things
>>>> and a very unnecessary addition, since the same can be achieved
>>>> using
>>>> the normal type instantiation syntax and type specialisation on the
>>>> SIMD/memory-view *type* (note the difference to using that syntax on
>>>> just *any* type). The [] syntax makes a lot of sense for the new
>>>> array type, it makes mostly sense for buffers, it was accepted
>>>> (IIRC) for type specialisation. IMHO, that's a long enough distance
>>>> that we go down that road. Going further would make the language
>>>> less parseable to humans.
>>>
>>> IIRC part of the motivation for int[:,:] syntax rather than
>>> special_name[int, ndim=2] was that the former is actually easier for
>>> humans to parse. Personally, it seems a natural extension of the int
>>> [50] syntax.
>>
>> It behaves very different, though. For one, it's not a memory
>> allocated data object but a *view* on such a thing, just like the
>> typed memory view.
>> What syntax would you propose for the memory view, and if none, why
>> not?
>
> The proposal is to give our memory view types SIMD semantics

That's where we disagree. I think there should be three types with
different behaviour: a dynamic array type that allocates memory and two
different view types with different (arithmetic etc.) behavior. I defined
all three of them in an earlier post.


> not
> have separate types with different semantics that one would have to
> cast between.

It's not a cast, it's more of a "create a view object for this buffer or
array". That's why I say it should look like an object creation. If Cython
optimises the object creation away or not, if it uses a different type, or
if it does exactly what the source indicates is an implementation detail.


> Buffers and memory views are mostly abstractions and
> generalizations of ndarray that have been added to Python.

Totally not. The first two are views, the latter is an object that
contains data. You can have tons of views, but only one data container.
Views can have different ways of modifying the container (such as a
sequence or SIMD way), but there is only one way per view type, and one
way per container.

> What we
> want is to be able to treat these objects as one would treat numpy
> arrays, but with more room for optimization and more flexibility in
> the back end.

We agree on that.

>>> It also makes it very clear to me that this is not some
>>> ordinary type implemented in some library, but a builtin type (like
>>> int[50]) that the compiler implements internally.
>>
>> That's an implementation detail, not a language 'feature'.
>>
>>> This new type isn't
>>> really an object, it's a multi-dimensional set of ints, the same way
>>> int[10] or int* is a one-dimensional set of ints.
>>
>> No, that would be the array type, which has no SIMD behaviour.
>
> int[], int[10], int*, and int[:] would be distinct types with
> different behaviors, but I think they have more in common (from a use
> point of view) than SomeType[int].

I already asked what syntax you wanted to have for a non-SIMD memory view.
Imagine we found a need for a third type of view one day. We can't keep
coming up with new ways of overriding syntax when we already have a
perfect way of instantiating *any* kind of view - simply by instantiating
its type.

>>> Note that just implementing an extension class Foo, and using the
>>> buffer syntax on Foo is insufficient (subtypes (which may override
>>> Foo's behavior) are disallowed,
>>
>> Fine, trying to do so is a compile time error then. Give it a good
>> error
>> message saying you're "sorry but performance dictates an immutable
>> type here" and every user will be happy.
>
> Perhaps we could add a final keyword to methods, but it would be
> messy to disallow Python subclasses, and still wouldn't allow the
> kind of optimization that can be done if the compiler knows that it's
> a SIMD type.

That's not what I meant. I meant to say that it would be a compile time
error to try to subtype the SIMD type at all. It's a special type for
which some things don't work, just like some things don't work with cdef
classes that work perfectly with Python classes (e.g. multiple
inheritance). That also leave a future path open if we ever figure out a
way to make subtyping possible (and meaningful).


>>> compile-time slicing and arithmetic can't be performed without
>>> special magic, an actual, potentially redundant, instance of Foo
>>> would have to be created). Since the SMID type can't be a cdef
>>> class, I think it's an advantage to not have it look like it is one.
>>
>> Details, details.
>
> I don't think it's just details, I think it's an important user
> interface issue. The syntax changes how one thinks about these
> things, and also impacts readability.
>
> Trying to understand your point of view, what of the following are
> you against:
>
> 1) Any compiler/language support for memory views and SIMD---they
> should be a library requiring no support from the compiler or language.

I think both the dynamic array type and the typed memory view type can be
implemented externally, while an optimised SIMD type requires compiler
support. I'm fine with all three, and I'm also ok with an array or memory
view type that benefits from some kind of compiler support (although I
think we should keep things out that can be kept out).


> 2) A magic extension class that the compiler knows about and inlines/
> emulates operations for optimization purposes, e.g. CythonArray[int,
> ndim=2].

Perfectly ok.


> 3) Same as the above, CythonArray is really just a keyword--there's
> no actual extension class backing it since the compiler encapsulates
> its logic anyways.

Details again, but if that's really required, well, then it's required.


> 4) Syntactic sugar like int[:,:] which means the same thing as
> CythonArray[int, ndim=2].

I'm against that, because I can't see why we should have syntactic sugar
for one type of view but not for another (or even a couple of other
types).


> 5) The syntactic sugar is the new syntax, rather than introducing the
> CythonArray keyword.

Same as above: adding a syntax for one type of view but not for another is
not a good way of evolving the language. We may add a syntax later on iff
we ever see that a) it's used very frequently and b) users complain about
the lengthy spelling and c) we don't have a better thing to throw a
specific syntax on. Until then, I'm against syntactic sugar for a very
special subset of a larger number of use cases.

Stefan


From dagss at student.matnat.uio.no  Fri Jun 19 12:16:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 12:16:25 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B15CE.2090104@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>
Message-ID: <4A3B6579.1040007@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> OK, I misunderstood you again. Breaking down what you say in two:
>>
>> 1) Syntax of SIMD view type. You propose SIMD[something] instead of 
>> basetype[something] -- that can obviously be made to work. As all syntax 
>> it is a matter of taste. I'll certainly go for whatever majority vote 
>> says here, it's not very important to me, but I vote for 
>> basetype[something] as it looks a lot more attractive for potential 
>> numerical users IMO.
>>
>> 2) Acquisition. You could rewrite your example with my syntax like e.g. 
>> this:
>>
>> obj = cython_array_type[int](20, 20) # create array object
>> cdef int[:,:] some_simd
>> some_simd = int[:,:](obj)
>>
>> IIUC you would just switch out int[:,:] with SIMD[...] to get something 
>> close to what you want.
>>
>> Hmm. I actually like the explicitness that you propose. But it is a 
>> matter of verbosity/repeating, as replacing line 3 with "some_simd = 
>> obj" could only mean one thing. I'm +0 on this issue.
>>
>> In pure Python mode, one would definitely have to do "some_simd = 
>> int[:,:](obj)" though.
> 
> Explicitness was a major reason for my counterproposal, yes. And I think
> the same would work in Python, where the SIMD type would be used to wrap a
> buffer/array object.

I'll let you and Robert go on with the syntax discussion without me.

However I have my doubts on whether you'll get anywhere, I maintain that 
this is likely a matter of taste in the end, with no objectively "right" 
or "wrong" decision.

Are you OK with leaving the issue to popular vote?

To recap, there's two decisions:
  a) Whether a view must be explicitly requested or is "coerced", i.e. 
whether one can do

     x = np.sin(x) # get view on result

or has to do

     x = SIMD_TYPE_SPECIFIER(np.sin(x))

  b) SIMD view type syntax


I strongly maintain that some information must be present per dimension 
-- I can post more details if you wish, but for clarity I will use 
"stridespec" as a placeholder below.

Furthermore there's the question of allowing per-dimension wraparound 
disabling.

Then, some options:

OPTION A:
   int[::stridespec,::stridespec]
   int[0::stridespec,::stridespec] -- default stridespec, but do not 
wraparound on first dim

Default stridespec possible.

OPTION B:
   SomeWord[int, ::stridespec, ::stridespec]
   SomeWord[int, 0::stridespec, ::stridespec]

Default stridespec possible. Looks odd to me with the type on the "first 
dimension".

OPTION C:
   SomeWord[int, (stridespec, stridespec)]
   SomeWord[int, (stridespec, stridespec), (False, True)]

Default stridespec not possible (unless perhaps on all strides at the 
same time: SomeWord[int])

If you agree with a popular vote deciding it, I suppose we should start 
a new thread and put it more nicely.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Jun 19 12:21:42 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 12:21:42 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B6579.1040007@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>
	<4A3B6579.1040007@student.matnat.uio.no>
Message-ID: <4A3B66B6.7030605@student.matnat.uio.no>

Corrections, corrections...

Dag Sverre Seljebotn wrote:
> Then, some options:
> 
> OPTION A:
>    int[::stridespec,::stridespec]
>    int[0::stridespec,::stridespec] -- default stridespec, but do not 
> wraparound on first dim

Forget "default stridespec" part of that sentence.

> Default stridespec possible.
> 
> OPTION B:
>    SomeWord[int, ::stridespec, ::stridespec]
>    SomeWord[int, 0::stridespec, ::stridespec]
> 
> Default stridespec possible. Looks odd to me with the type on the "first 
> dimension".
> 
> OPTION C:
>    SomeWord[int, (stridespec, stridespec)]
>    SomeWord[int, (stridespec, stridespec), (False, True)]
> 
> Default stridespec not possible (unless perhaps on all strides at the 
> same time: SomeWord[int])

No, you still need to know number of dimensions, so default stridespec 
not possible with this option. But "ndim=3" could be shorthand for 
(defaultstridespec, )*3.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Jun 19 12:28:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 12:28:01 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>
	<5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <4A3B6831.7000907@student.matnat.uio.no>

Stefan Behnel wrote:
> Robert Bradshaw wrote:
>> On Jun 19, 2009, at 12:28 AM, Stefan Behnel wrote:
>>> Robert Bradshaw wrote:
>>>> On Jun 18, 2009, at 9:36 PM, Stefan Behnel wrote:
>>>>> Having the same syntax
>>>>> for memory views and SIMD views sounds like an awful lot of things
>>>>> and a very unnecessary addition, since the same can be achieved
>>>>> using
>>>>> the normal type instantiation syntax and type specialisation on the
>>>>> SIMD/memory-view *type* (note the difference to using that syntax on
>>>>> just *any* type). The [] syntax makes a lot of sense for the new
>>>>> array type, it makes mostly sense for buffers, it was accepted
>>>>> (IIRC) for type specialisation. IMHO, that's a long enough distance
>>>>> that we go down that road. Going further would make the language
>>>>> less parseable to humans.
>>>> IIRC part of the motivation for int[:,:] syntax rather than
>>>> special_name[int, ndim=2] was that the former is actually easier for
>>>> humans to parse. Personally, it seems a natural extension of the int
>>>> [50] syntax.
>>> It behaves very different, though. For one, it's not a memory
>>> allocated data object but a *view* on such a thing, just like the
>>> typed memory view.
>>> What syntax would you propose for the memory view, and if none, why
>>> not?
>> The proposal is to give our memory view types SIMD semantics
> 
> That's where we disagree. I think there should be three types with
> different behaviour: a dynamic array type that allocates memory and two
> different view types with different (arithmetic etc.) behavior. I defined
> all three of them in an earlier post.

Just for clarification (I looked up your earlier post but didn't find 
it): Would the non-SIMD view type (your 2)) support more than one 
dimension? How would +, +=, append semantics be defined if there's more 
than one dimension?


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Jun 19 12:33:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 12:33:05 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B6831.7000907@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>	<5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3B6831.7000907@student.matnat.uio.no>
Message-ID: <4A3B6961.20201@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Robert Bradshaw wrote:
>>> On Jun 19, 2009, at 12:28 AM, Stefan Behnel wrote:
>>>> Robert Bradshaw wrote:
>>>>> On Jun 18, 2009, at 9:36 PM, Stefan Behnel wrote:
>>>>>> Having the same syntax
>>>>>> for memory views and SIMD views sounds like an awful lot of things
>>>>>> and a very unnecessary addition, since the same can be achieved
>>>>>> using
>>>>>> the normal type instantiation syntax and type specialisation on the
>>>>>> SIMD/memory-view *type* (note the difference to using that syntax on
>>>>>> just *any* type). The [] syntax makes a lot of sense for the new
>>>>>> array type, it makes mostly sense for buffers, it was accepted
>>>>>> (IIRC) for type specialisation. IMHO, that's a long enough distance
>>>>>> that we go down that road. Going further would make the language
>>>>>> less parseable to humans.
>>>>> IIRC part of the motivation for int[:,:] syntax rather than
>>>>> special_name[int, ndim=2] was that the former is actually easier for
>>>>> humans to parse. Personally, it seems a natural extension of the int
>>>>> [50] syntax.
>>>> It behaves very different, though. For one, it's not a memory
>>>> allocated data object but a *view* on such a thing, just like the
>>>> typed memory view.
>>>> What syntax would you propose for the memory view, and if none, why
>>>> not?
>>> The proposal is to give our memory view types SIMD semantics
>> That's where we disagree. I think there should be three types with
>> different behaviour: a dynamic array type that allocates memory and two
>> different view types with different (arithmetic etc.) behavior. I defined
>> all three of them in an earlier post.
> 
> Just for clarification (I looked up your earlier post but didn't find 
> it): Would the non-SIMD view type (your 2)) support more than one 
> dimension? How would +, +=, append semantics be defined if there's more 
> than one dimension?

Or...I suppose +, +=, append are only for the array, and the non-SIMD 
view can only do item indexing/slicing and has no arithmetic operators 
at all?

-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Jun 19 13:13:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 13:13:56 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B6831.7000907@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>
	<5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3B6831.7000907@student.matnat.uio.no>
Message-ID: <11ad2974e134aa36ce78f67964297362.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> Just for clarification (I looked up your earlier post but didn't find
> it): Would the non-SIMD view type (your 2)) support more than one
> dimension?

Didn't think about that, but I'd say why not.


> How would +, +=, append semantics be defined if there's more
> than one dimension?

Likely not at all, i.e. you'd get an error either at runtime or at compile
time (if doable). It would also be possible to consider a
multi-dimensional view a view on a recursive sequence instead (like tuples
of tuples), so that it would concatenate the first dimension. I'm open to
both, depending on what use cases we come up with (meaning: if we don't
currently care, this would be an error until further consideration).

Stefan


From stefan_ml at behnel.de  Fri Jun 19 13:32:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 13:32:52 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B6961.20201@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>
	<5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3B6831.7000907@student.matnat.uio.no>
	<4A3B6961.20201@student.matnat.uio.no>
Message-ID: <029a803d635a581e3a7806916334c8f7.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
>> Just for clarification (I looked up your earlier post but didn't find
>> it): Would the non-SIMD view type (your 2)) support more than one
>> dimension? How would +, +=, append semantics be defined if there's more
>> than one dimension?
>
> Or...I suppose +, +=, append are only for the array, and the non-SIMD
> view can only do item indexing/slicing and has no arithmetic operators
> at all?

That's also an option. However, given that a view can consist of any
subset of items in the underlying container, it would be nice to support
it at least for one dimension, so that you could do (for example)

    new_array = view_a[::2] + view_b[1::2]

efficiently (i.e. sort the odd elements behind the even ones).

I'm also not sure what type such a view should return when it needs to
create a new container. It cannot always be smart enough to create the
correct container type (e.g. for a view on a buffer). That might count as
an argument for not allowing any arithmetic at all. OTOH, there's a
similar case in PEP 3132 (extended iterable unpacking), and the decision
was to always return a list for the starred variable. If you really
require something different than a plain array, you can do the less
efficient

    new_array = EXPECTED_TYPE(view_a[::2]) + view_b[1::2]

assuming that the EXPECTED_TYPE supports this.

On a similar note, should adding views return a plain container, or should
the result get wrapped in the view type? I think that's what would happen
for SIMD operations, so I guess it would be right also for the typed
memory view to transform this:

    result = view_a[::2] + view_b[1::2]

into

    result = NEW_VIEW(NEW_ARRAY(view_a[::2] CONCAT view_b[1::2]))

But I think we're getting back into details before their time here. I'm
fine with disallowing this completely at the beginning, and to come up
with sensible semantics when we see what common use cases are.

Stefan


From dagss at student.matnat.uio.no  Fri Jun 19 13:39:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 13:39:55 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <029a803d635a581e3a7806916334c8f7.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>	<5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3B6831.7000907@student.matnat.uio.no>	<4A3B6961.20201@student.matnat.uio.no>
	<029a803d635a581e3a7806916334c8f7.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <4A3B790B.5040302@student.matnat.uio.no>

Stefan Behnel wrote:
> But I think we're getting back into details before their time here. I'm
> fine with disallowing this completely at the beginning, and to come up
> with sensible semantics when we see what common use cases are.

Yes, I was just trying to get a better grip on what you were thinking 
with two view types.

If there's no big usecase for arithmetic operators on them in the first 
place, I see no harm in having a 
i_really_know_how_simd_works_now_let_me_shoot_myself_in_the_foot 
directive enabling SIMD on views, enabling us to simplify and only have 
one view type (i.e. it wouldn't "switch" semantics, only allow 
semantics, which seems like a safer directive).


-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Jun 19 13:52:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 13:52:41 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B6579.1040007@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<4A3B6579.1040007@student.matnat.uio.no>
Message-ID: <359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> I'll let you and Robert go on with the syntax discussion without me.
>
> However I have my doubts on whether you'll get anywhere, I maintain that
> this is likely a matter of taste in the end, with no objectively "right"
> or "wrong" decision.
>
> Are you OK with leaving the issue to popular vote?

You mean as in the case of decorator support for Python? ;)

I think what this would mean is that all listening numerics people would
go "yes, I want a short syntax as in NumPy" and everyone else would go
"hmmm, I don't care, so I don't vote". I don't think that would give us a
meaningfull poll result.


> To recap, there's two decisions:
>   a) Whether a view must be explicitly requested or is "coerced", i.e.
> whether one can do
>
>      x = np.sin(x) # get view on result
>
> or has to do
>
>      x = SIMD_TYPE_SPECIFIER(np.sin(x))

Right, that's open (as I replied to your other mail). I think it depends
on how easy it is to get to the container when you have the view in your
hands. If you can easily unpack the container then there's not much of a
difference, but my assumption is that it's more convenient to get the type
back that you passed in, rather than a type that might behave different.

So my vote is for transforming

      x = np.sin(x) # get view on result

into

      x = SIMD_TYPE_SPECIFIER(np.sin(x))

internally. We might allow an exception when x is typed as a known
container result type of the operation, i.e.

      cdef array['some spec'] x = np.sin(x)

would not create an intermediate view, but write the result directly into
a newly created array. But that's an optimisation, not a requirement.


>   b) SIMD view type syntax
>
> I strongly maintain that some information must be present per dimension
> -- I can post more details if you wish, but for clarity I will use
> "stridespec" as a placeholder below.
>
> Furthermore there's the question of allowing per-dimension wraparound
> disabling.
>
> Then, some options:
>
> OPTION A:
>    int[::stridespec,::stridespec]
>    int[0::stridespec,::stridespec] -- default stridespec, but do not
> wraparound on first dim
>
> Default stridespec possible.
>
> OPTION B:
>    SomeWord[int, ::stridespec, ::stridespec]
>    SomeWord[int, 0::stridespec, ::stridespec]
>
> Default stridespec possible. Looks odd to me with the type on the "first
> dimension".
>
> OPTION C:
>    SomeWord[int, (stridespec, stridespec)]
>    SomeWord[int, (stridespec, stridespec), (False, True)]
>
> Default stridespec not possible (unless perhaps on all strides at the
> same time: SomeWord[int])

I can't really comment on that, as I never needed this option. I do agree
that this information is required, though. Maybe keyword arguments fix
this?

    SimdType[int, strides=[::stridespec,::stridespec]]

or something like that. Note that we need to smarten up the parser anyway
if we want to support templated types.

Stefan


From stefan_ml at behnel.de  Fri Jun 19 13:55:51 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 13:55:51 +0200 (CEST)
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3B790B.5040302@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de> <4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>
	<5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3B6831.7000907@student.matnat.uio.no>
	<4A3B6961.20201@student.matnat.uio.no>
	<029a803d635a581e3a7806916334c8f7.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3B790B.5040302@student.matnat.uio.no>
Message-ID: <4ad0acd3de6f89a640e94e919f13da54.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> But I think we're getting back into details before their time here. I'm
>> fine with disallowing this completely at the beginning, and to come up
>> with sensible semantics when we see what common use cases are.
>
> Yes, I was just trying to get a better grip on what you were thinking
> with two view types.
>
> If there's no big usecase for arithmetic operators on them in the first
> place, I see no harm in having a
> i_really_know_how_simd_works_now_let_me_shoot_myself_in_the_foot
> directive enabling SIMD on views, enabling us to simplify and only have
> one view type (i.e. it wouldn't "switch" semantics, only allow
> semantics, which seems like a safer directive).

Well, it would start switching semantics from the moment we start allowing
arithmetic operators on the memory view type, so I'm -1 on such an option
that would shoot back into our own foot in a few months to years.

Stefan


From stefan at sun.ac.za  Fri Jun 19 17:38:04 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Fri, 19 Jun 2009 17:38:04 +0200
Subject: [Cython] Wiki pages on profiling and compiling Windows
	extensions
In-Reply-To: <4A3924A4.80405@student.matnat.uio.no>
References: <9457e7c80906170311p654f6415y73a7df8385cac84b@mail.gmail.com> 
	<4A3924A4.80405@student.matnat.uio.no>
Message-ID: <9457e7c80906190838ufdf9fa2l3faee573218ec1d1@mail.gmail.com>

2009/6/17 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
>> Could we link to these from the wiki front page or from another visible page?
>
> Sure, just add a link as you see fit, probably under Tips&Tricks on the
> wiki front page, and possibly referenced from the FAQ as well.

Maybe you can give a hand -- I don't have the required permissions.

Thanks!
St?fan

From robertwb at math.washington.edu  Fri Jun 19 17:58:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 08:58:43 -0700
Subject: [Cython] Wiki pages on profiling and compiling Windows
	extensions
In-Reply-To: <9457e7c80906190838ufdf9fa2l3faee573218ec1d1@mail.gmail.com>
References: <9457e7c80906170311p654f6415y73a7df8385cac84b@mail.gmail.com>
	<4A3924A4.80405@student.matnat.uio.no>
	<9457e7c80906190838ufdf9fa2l3faee573218ec1d1@mail.gmail.com>
Message-ID: <094C0B31-60DB-446E-A704-E9F88E8D738F@math.washington.edu>

On Jun 19, 2009, at 8:38 AM, St?fan van der Walt wrote:

> 2009/6/17 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
>>> Could we link to these from the wiki front page or from another  
>>> visible page?
>>
>> Sure, just add a link as you see fit, probably under Tips&Tricks  
>> on the
>> wiki front page, and possibly referenced from the FAQ as well.
>
> Maybe you can give a hand -- I don't have the required permissions.

Sorry, the front page has higher restrictions due to it being a big  
spam target. I'm happy to add them.

- Robert


From dagss at student.matnat.uio.no  Fri Jun 19 18:08:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 18:08:59 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>
	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <4A3BB81B.3020107@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> I'll let you and Robert go on with the syntax discussion without me.
>>
>> However I have my doubts on whether you'll get anywhere, I maintain that
>> this is likely a matter of taste in the end, with no objectively "right"
>> or "wrong" decision.
>>
>> Are you OK with leaving the issue to popular vote?
> 
> You mean as in the case of decorator support for Python? ;)
> 
> I think what this would mean is that all listening numerics people would
> go "yes, I want a short syntax as in NumPy" and everyone else would go
> "hmmm, I don't care, so I don't vote". I don't think that would give us a
> meaningfull poll result.

You may be right in it not giving us new information, though I don't see 
why you can't give this premise more weight: The people who are actually 
likely to care either way are likely to disagree with you.

Anyway, I'll be really blunt.

I don't see another week of discussions changing anybody's opinion here, 
it's already been discussed and valid points have been raised on both 
sides. Humans have been in deadlocks like this before though; common 
solutions include

  - Meritocracy (whoever does the feature decides)
  - Rule be the elite (lead developer majority)
  - Democracy (mailing list majority)

Either way, your view about int[:,:] loose. Given this, I don't see why 
we have to drag on the discussion even further, unless this is *very* 
important to you. (As I said, I'd rather pull the whole thing than 
actually upset you.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Jun 19 18:17:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 18:17:09 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>
	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <4A3BBA05.8060706@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> I'll let you and Robert go on with the syntax discussion without me.
>>
>> However I have my doubts on whether you'll get anywhere, I maintain that
>> this is likely a matter of taste in the end, with no objectively "right"
>> or "wrong" decision.
>>
>> Are you OK with leaving the issue to popular vote?
> 
> You mean as in the case of decorator support for Python? ;)
> 
> I think what this would mean is that all listening numerics people would
> go "yes, I want a short syntax as in NumPy" and everyone else would go
> "hmmm, I don't care, so I don't vote". I don't think that would give us a
> meaningfull poll result.
> 
> 
>> To recap, there's two decisions:
>>   a) Whether a view must be explicitly requested or is "coerced", i.e.
>> whether one can do
>>
>>      x = np.sin(x) # get view on result
>>
>> or has to do
>>
>>      x = SIMD_TYPE_SPECIFIER(np.sin(x))
> 
> Right, that's open (as I replied to your other mail). I think it depends
> on how easy it is to get to the container when you have the view in your
> hands. If you can easily unpack the container then there's not much of a
> difference, but my assumption is that it's more convenient to get the type
> back that you passed in, rather than a type that might behave different.
> 
> So my vote is for transforming
> 
>       x = np.sin(x) # get view on result
> 
> into
> 
>       x = SIMD_TYPE_SPECIFIER(np.sin(x))
> 
> internally. We might allow an exception when x is typed as a known
> container result type of the operation, i.e.
> 
>       cdef array['some spec'] x = np.sin(x)
> 
> would not create an intermediate view, but write the result directly into
> a newly created array. But that's an optimisation, not a requirement.

I'm afraid my example was misleading; I meant np.sin to be a plain, 
object-taking (and thus non-SIMD) function. A clearer example is

     x = f()

vs.

     x = SIMD_TYPE_SPECIFIER(f())

however it hides how inconvenient it would be to use NumPy in this 
situation. Take "np.dot" -- it takes two array arguments, and return the 
linear algebra product (meaning non-SIMD) operations. Then it is

     x = np.dot(y, z)

vs.

     x = SIMD_TYPE_SPECIFIER(np.dot(y, z))

My vote is for the former though, just to save keystrokes, as that's 
something I do really often. But the latter should be supported as well 
and be the "written-out" version.

I don't see that this is much different from e.g. coercing to C types; 
constructing a new view is simply how Python objects are coerced to 
SIMD_TYPE_SPECIFIER.

-- 
Dag Sverre

From robertwb at math.washington.edu  Fri Jun 19 18:41:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 09:41:23 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<c4ec4aa76a11aa113a913ba4d5cac14b.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<CFFFBCF4-5905-46D4-ACBC-892C932529BB@math.washington.edu>
	<5d855f4e9647776e2b451d29970e5e48.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <35E816A6-F89B-4821-AA55-B93724C2417E@math.washington.edu>

On Jun 19, 2009, at 3:09 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>>
>> The proposal is to give our memory view types SIMD semantics
>
> That's where we disagree. I think there should be three types with
> different behaviour: a dynamic array type that allocates memory and  
> two
> different view types with different (arithmetic etc.) behavior. I  
> defined
> all three of them in an earlier post.

Ah, somehow I missed that this was an important point for you. I  
think we're all OK with a dynamic list type (that behaves like a  
python list of c types)--that would be really handy so most users  
will never even have to touch malloc. In terms of the two views, I  
see the SIMD type as a specialization (i.e extra features) of the  
"generic" memory view type, and have trouble seeing what it would  
provide that would make it incompatible with SIMD.

> I already asked what syntax you wanted to have for a non-SIMD  
> memory view.
> Imagine we found a need for a third type of view one day. We can't  
> keep
> coming up with new ways of overriding syntax when we already have a
> perfect way of instantiating *any* kind of view - simply by  
> instantiating
> its type.

We'll come to that when we need to. Probably just MemoryView[int,  
ndim=2] would be sufficient. But there is a large demand for an SIMD  
view type, and no one is coming up with a clear use for a strictly  
non-SIMD view type, which I think makes it special enough to merit  
special syntax. (The lack of native array types in other languages is  
one of the things that makes Fortran the language of choice for  
numerical programming.)

>> Trying to understand your point of view, what of the following are
>> you against:
>>
>> 1) Any compiler/language support for memory views and SIMD---they
>> should be a library requiring no support from the compiler or  
>> language.
>
> I think both the dynamic array type and the typed memory view type  
> can be
> implemented externally, while an optimised SIMD type requires compiler
> support. I'm fine with all three, and I'm also ok with an array or  
> memory
> view type that benefits from some kind of compiler support (although I
> think we should keep things out that can be kept out).
>
>
>> 2) A magic extension class that the compiler knows about and inlines/
>> emulates operations for optimization purposes, e.g. CythonArray[int,
>> ndim=2].
>
> Perfectly ok.
>
>
>> 3) Same as the above, CythonArray is really just a keyword--there's
>> no actual extension class backing it since the compiler encapsulates
>> its logic anyways.
>
> Details again, but if that's really required, well, then it's  
> required.
>
>
>> 4) Syntactic sugar like int[:,:] which means the same thing as
>> CythonArray[int, ndim=2].
>
> I'm against that, because I can't see why we should have syntactic  
> sugar
> for one type of view but not for another (or even a couple of other
> types).
>
>
>> 5) The syntactic sugar is the new syntax, rather than introducing the
>> CythonArray keyword.
>
> Same as above: adding a syntax for one type of view but not for  
> another is
> not a good way of evolving the language. We may add a syntax later  
> on iff
> we ever see that a) it's used very frequently and b) users complain  
> about
> the lengthy spelling and c) we don't have a better thing to throw a
> specific syntax on. Until then, I'm against syntactic sugar for a very
> special subset of a larger number of use cases.

I guess I see this as  the primary use case for memory views, but  
that's a difference of opinion. OK, then lets at least add this type  
without the syntactic sugar for now, and come back to it later.  
(Well, maybe not in the too distant future, but it's becoming like a  
bike-shedding thread right now.)

- Robert


From dalcinl at gmail.com  Fri Jun 19 18:44:34 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 19 Jun 2009 13:44:34 -0300
Subject: [Cython] ticket #333 (extern ctypedef integral <-> python object
	conversion), please review patch
Message-ID: <e7ba66e40906190944o27f4e319t920a1a94dbf63f14@mail.gmail.com>

I've finally managed to write a more or less working patch... Look at
the test, I've tried to cover many corner cases...

1) Dag & Kurt: I bet you will be happy.


2) Robert: your eyeballs needed, please comment on this:

a) Look at the changes outside PyrexTypes.pyx, I believe they make
sense, though I would like you to confirm that.

b) In the past, you raised some concerns about __int32 from ILP64
model... A possible (and suboptimal, no overflow-safe) way of handling
that is there, though "#if 0" disabled. I've tried to take advantage
of "_PyLong_{As/From}ByteArray()", but that (in particular, the "As"
one) is somewhat harder to use, as we should pass a PyLongObject type.




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Jun 19 18:47:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 09:47:45 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3BB81B.3020107@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>
	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3BB81B.3020107@student.matnat.uio.no>
Message-ID: <037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>

On Jun 19, 2009, at 9:08 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> I'll let you and Robert go on with the syntax discussion without me.
>>>
>>> However I have my doubts on whether you'll get anywhere, I  
>>> maintain that
>>> this is likely a matter of taste in the end, with no objectively  
>>> "right"
>>> or "wrong" decision.
>>>
>>> Are you OK with leaving the issue to popular vote?
>>
>> You mean as in the case of decorator support for Python? ;)
>>
>> I think what this would mean is that all listening numerics people  
>> would
>> go "yes, I want a short syntax as in NumPy" and everyone else  
>> would go
>> "hmmm, I don't care, so I don't vote". I don't think that would  
>> give us a
>> meaningfull poll result.
>
> You may be right in it not giving us new information, though I  
> don't see
> why you can't give this premise more weight:

Well, if there's no strong opposition to a feature but there are a  
bunch of people who like it, I'd take that as an indication that it's  
probably worth going in. Obviously, we are not in the "no strong  
opposition" from you which is why this thread has ballooned to almost  
50 messages.

> The people who are actually
> likely to care either way are likely to disagree with you.
>
> Anyway, I'll be really blunt.
>
> I don't see another week of discussions changing anybody's opinion  
> here,
> it's already been discussed and valid points have been raised on both
> sides. Humans have been in deadlocks like this before though; common
> solutions include
>
>   - Meritocracy (whoever does the feature decides)
>   - Rule be the elite (lead developer majority)
>   - Democracy (mailing list majority)
>
> Either way, your view about int[:,:] loose. Given this, I don't see  
> why
> we have to drag on the discussion even further, unless this is *very*
> important to you. (As I said, I'd rather pull the whole thing than
> actually upset you.)

Yes, that was blunt, but a valid point.

OK, how about we get the feature with the less controversial syntax,  
and then re-open the discussion if desired.

- Robert


From dagss at student.matnat.uio.no  Fri Jun 19 19:21:52 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 19:21:52 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>
	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>
Message-ID: <4A3BC930.4040007@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 19, 2009, at 9:08 AM, Dag Sverre Seljebotn wrote:
> 
>> Stefan Behnel wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> I'll let you and Robert go on with the syntax discussion without me.
>>>>
>>>> However I have my doubts on whether you'll get anywhere, I  
>>>> maintain that
>>>> this is likely a matter of taste in the end, with no objectively  
>>>> "right"
>>>> or "wrong" decision.
>>>>
>>>> Are you OK with leaving the issue to popular vote?
>>> You mean as in the case of decorator support for Python? ;)
>>>
>>> I think what this would mean is that all listening numerics people  
>>> would
>>> go "yes, I want a short syntax as in NumPy" and everyone else  
>>> would go
>>> "hmmm, I don't care, so I don't vote". I don't think that would  
>>> give us a
>>> meaningfull poll result.
>> You may be right in it not giving us new information, though I  
>> don't see
>> why you can't give this premise more weight:
> 
> Well, if there's no strong opposition to a feature but there are a  
> bunch of people who like it, I'd take that as an indication that it's  
> probably worth going in. Obviously, we are not in the "no strong  
> opposition" from you which is why this thread has ballooned to almost  
> 50 messages.
> 
>> The people who are actually
>> likely to care either way are likely to disagree with you.
>>
>> Anyway, I'll be really blunt.
>>
>> I don't see another week of discussions changing anybody's opinion  
>> here,
>> it's already been discussed and valid points have been raised on both
>> sides. Humans have been in deadlocks like this before though; common
>> solutions include
>>
>>   - Meritocracy (whoever does the feature decides)
>>   - Rule be the elite (lead developer majority)
>>   - Democracy (mailing list majority)
>>
>> Either way, your view about int[:,:] loose. Given this, I don't see  
>> why
>> we have to drag on the discussion even further, unless this is *very*
>> important to you. (As I said, I'd rather pull the whole thing than
>> actually upset you.)
> 
> Yes, that was blunt, but a valid point.
> 
> OK, how about we get the feature with the less controversial syntax,  
> and then re-open the discussion if desired.

Assuming you mean "simd[int, (stridespecs)]" is less controversial here:

My main issue with this solution is what precendence it creates for how 
project issues are settled. I don't like having a project culture where 
stamina in these long threads is what counts in the end -- I'd much 
rather have a clear, open voting process.

But hear this: If both Robert and Stefan agrees about a syntax, whatever 
the conclusion is, I won't say another word.

(That is, you need to get the stridespecs in, and I'll state now that 
I'd really, really prefer a Python-grammar-compatible syntax, unlike the 
alternative proposals until now. Getting SIMD in pure Python mode with 
the same type syntax is one of the neatest point in the whole int[:,:] 
syntax.)


-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Jun 19 20:07:22 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 20:07:22 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3BBA05.8060706@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3BBA05.8060706@student.matnat.uio.no>
Message-ID: <4A3BD3DA.9050700@behnel.de>


Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> So my vote is for transforming
>>
>>       x = np.sin(x) # get view on result
>>
>> into
>>
>>       x = SIMD_TYPE_SPECIFIER(np.sin(x))
>>
>> internally. We might allow an exception when x is typed as a known
>> container result type of the operation, i.e.
>>
>>       cdef array['some spec'] x = np.sin(x)
>>
>> would not create an intermediate view, but write the result directly into
>> a newly created array. But that's an optimisation, not a requirement.
> 
> I'm afraid my example was misleading; I meant np.sin to be a plain, 
> object-taking (and thus non-SIMD) function.

I understood that. The question was if the result of a compiler-generated
SIMD operation on a SIMD type should return a view or a container. My
answer was that it should return a SIMD view for consistency, but could
return a container on request, depending on the target type. So we'd
optimise away the coercion, in a way. That's more or less the same as the
optimisation that a calculation can be done in-place if you assign the
result to a participating object.

Stefan


From dagss at student.matnat.uio.no  Fri Jun 19 20:11:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 20:11:01 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3BD3DA.9050700@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BBA05.8060706@student.matnat.uio.no>
	<4A3BD3DA.9050700@behnel.de>
Message-ID: <4A3BD4B5.1000906@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> So my vote is for transforming
>>>
>>>       x = np.sin(x) # get view on result
>>>
>>> into
>>>
>>>       x = SIMD_TYPE_SPECIFIER(np.sin(x))
>>>
>>> internally. We might allow an exception when x is typed as a known
>>> container result type of the operation, i.e.
>>>
>>>       cdef array['some spec'] x = np.sin(x)
>>>
>>> would not create an intermediate view, but write the result directly into
>>> a newly created array. But that's an optimisation, not a requirement.
>> I'm afraid my example was misleading; I meant np.sin to be a plain, 
>> object-taking (and thus non-SIMD) function.
> 
> I understood that. The question was if the result of a compiler-generated
> SIMD operation on a SIMD type should return a view or a container. My
> answer was that it should return a SIMD view for consistency, but could
> return a container on request, depending on the target type. So we'd
> optimise away the coercion, in a way. That's more or less the same as the
> optimisation that a calculation can be done in-place if you assign the
> result to a participating object.

Thanks for explaining further. Sounds good!

-- 
Dag Sverre

From stefan_ml at behnel.de  Fri Jun 19 20:39:36 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Jun 2009 20:39:36 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3BC930.4040007@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>
	<4A3BC930.4040007@student.matnat.uio.no>
Message-ID: <4A3BDB68.5070402@behnel.de>


Dag Sverre Seljebotn wrote:
> [a couple of unnecessary words stripped]
> Robert Bradshaw wrote:
>> OK, how about we get the feature with the less controversial syntax,  
>> and then re-open the discussion if desired.
> 
> Assuming you mean "simd[int, (stridespecs)]" is less controversial here:
> 
> My main issue with this solution is what precendence it creates for how 
> project issues are settled. I don't like having a project culture where 
> stamina in these long threads is what counts in the end -- I'd much 
> rather have a clear, open voting process.
> 
> But hear this: If both Robert and Stefan agrees about a syntax, whatever 
> the conclusion is, I won't say another word.

Note that I raised a problem that you didn't solve so far, either. We
currently have a syntax for an array type (basically the C syntax) and a
syntax for the proposed SIMD type (int[:,:]). What we do not have is a
syntax for a memory view that does not have SIMD semantics, especially not
one that works in pure Python mode also. I assume that would use

  cdef MemoryView[int,int] v

in the original setup, which I would call inconsistent with all other
syntax we have for this.


> (That is, you need to get the stridespecs in, and I'll state now that 
> I'd really, really prefer a Python-grammar-compatible syntax, unlike the 
> alternative proposals until now. Getting SIMD in pure Python mode with 
> the same type syntax is one of the neatest point in the whole int[:,:] 
> syntax.)

Ok, so what about changing the template syntax then. Here's an idea. As I
said before, I'm a big fan of keyword arguments.

A template type (or generic type if you prefer) is a type that is
specialised by providing specific types for template parameters. These were
proposed to be specified as in

  cdef SomeTemplateType[int] x

Now, if we change that into allowing (something that looks like) a compile
time function call to return the type, we can do

  cdef SomeTemplateType(int, some_kw=True, other_kw=False) x

or, for better readability,

  ctypedef SomeTemplateType(int, some_kw=True, other_kw=False) SpecialType
  cdef SpecialType x

This will complicate the parser a bit in that we cannot easily distinguish
between a C function definition and a type declaration before having parsed
the signature and seeing a ':' or not. So I don't expect everyone to be
happy with this. However, if the goal is to provide a nicely readable way
for specialising types that also works in plain Python, then this is pretty
much as good as it can get. Because in plain Python, you can obviously do

  SpecialType = cython.typedef(
         SomeTemplateType(int, some_kw=True, other_kw=False))

  x = SpecialType(1,2,3)

Note also that the parser grammar can easily be disambiguated by requiring
a typedef for specialised types, just as in the Python code.

How does this impact the SIMD syntax then? Well, simple. You could do

  cdef mymethod(y):
      cdef SIMD(int, int, strides=[5,4]) x = y

or, if typedefs are required,

  ctypedef SIMD(int, int, strides=[5,4]) My2dSimdType

  cdef mymethod(y):
      cdef My2dSimdType x = y

Just an idea.

Stefan

From dagss at student.matnat.uio.no  Fri Jun 19 23:35:34 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 23:35:34 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3BDB68.5070402@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>
	<4A3BDB68.5070402@behnel.de>
Message-ID: <4A3C04A6.5070001@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> [a couple of unnecessary words stripped]
>> Robert Bradshaw wrote:
>>> OK, how about we get the feature with the less controversial syntax,  
>>> and then re-open the discussion if desired.
>> Assuming you mean "simd[int, (stridespecs)]" is less controversial here:
>>
>> My main issue with this solution is what precendence it creates for how 
>> project issues are settled. I don't like having a project culture where 
>> stamina in these long threads is what counts in the end -- I'd much 
>> rather have a clear, open voting process.
>>
>> But hear this: If both Robert and Stefan agrees about a syntax, whatever 
>> the conclusion is, I won't say another word.
> 
> Note that I raised a problem that you didn't solve so far, either. We
> currently have a syntax for an array type (basically the C syntax) and a
> syntax for the proposed SIMD type (int[:,:]). What we do not have is a
> syntax for a memory view that does not have SIMD semantics, especially not
> one that works in pure Python mode also. I assume that would use
> 
>   cdef MemoryView[int,int] v
> 
> in the original setup, which I would call inconsistent with all other
> syntax we have for this.
> 
> 
>> (That is, you need to get the stridespecs in, and I'll state now that 
>> I'd really, really prefer a Python-grammar-compatible syntax, unlike the 
>> alternative proposals until now. Getting SIMD in pure Python mode with 
>> the same type syntax is one of the neatest point in the whole int[:,:] 
>> syntax.)
> 
> Ok, so what about changing the template syntax then. Here's an idea. As I
> said before, I'm a big fan of keyword arguments.

<SNIP making this thread longer, and not even offering a comment to my 
blunt post>

I'm sorry, I'm pulling out of the whole thing. It just takes too much 
time on the mailing list. If somebody else want to finish the CEPs and 
implement it then I'd be thrilled though.

I suppose my real question was "can you just trust me on developing 
numerics and Cython further in this direction, I have these ideas I've 
been developing as an active numerics user and Cython developer over the 
past year".

I suppose the answer I got was "no, we have to do it all by comittee, 
and go through all the points in detail on the mailing list".

As it is, it looks like even if we could agree on a syntax, I'd still 
have to go to the mailing list for every little nick and cranny in the 
semantics, checking if it could be acceptable for both numerical and 
non-numerical use. I just don't have the time for that.

I think the real way forward for CEPs like these might be putting them 
on hold until we can meet in person -- if we need to design by comittee, 
let's be a real comittee. Emails just take too long, with days going by 
over simple misunderstandings. (Even if writing an email doesn't always 
take long, it is really testing for my patience to spend weeks before 
asking a question to seeing a resolution.)

The good news is that I can use what Cython time I have for better 
mentoring Kurt and fix more boring bugs.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Fri Jun 19 23:41:19 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Jun 2009 23:41:19 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C04A6.5070001@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>	<4A3BDB68.5070402@behnel.de>
	<4A3C04A6.5070001@student.matnat.uio.no>
Message-ID: <4A3C05FF.3010806@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn wrote:
>>> [a couple of unnecessary words stripped]
>>> Robert Bradshaw wrote:
>>>> OK, how about we get the feature with the less controversial syntax,  
>>>> and then re-open the discussion if desired.
>>> Assuming you mean "simd[int, (stridespecs)]" is less controversial here:
>>>
>>> My main issue with this solution is what precendence it creates for how 
>>> project issues are settled. I don't like having a project culture where 
>>> stamina in these long threads is what counts in the end -- I'd much 
>>> rather have a clear, open voting process.
>>>
>>> But hear this: If both Robert and Stefan agrees about a syntax, whatever 
>>> the conclusion is, I won't say another word.
>> Note that I raised a problem that you didn't solve so far, either. We
>> currently have a syntax for an array type (basically the C syntax) and a
>> syntax for the proposed SIMD type (int[:,:]). What we do not have is a
>> syntax for a memory view that does not have SIMD semantics, especially not
>> one that works in pure Python mode also. I assume that would use
>>
>>   cdef MemoryView[int,int] v
>>
>> in the original setup, which I would call inconsistent with all other
>> syntax we have for this.
>>
>>
>>> (That is, you need to get the stridespecs in, and I'll state now that 
>>> I'd really, really prefer a Python-grammar-compatible syntax, unlike the 
>>> alternative proposals until now. Getting SIMD in pure Python mode with 
>>> the same type syntax is one of the neatest point in the whole int[:,:] 
>>> syntax.)
>> Ok, so what about changing the template syntax then. Here's an idea. As I
>> said before, I'm a big fan of keyword arguments.
> 
> <SNIP making this thread longer, and not even offering a comment to my 
> blunt post>
> 
> I'm sorry, I'm pulling out of the whole thing. It just takes too much 
> time on the mailing list. If somebody else want to finish the CEPs and 
> implement it then I'd be thrilled though.
> 
> I suppose my real question was "can you just trust me on developing 
> numerics and Cython further in this direction, I have these ideas I've 
> been developing as an active numerics user and Cython developer over the 
> past year".
> 
> I suppose the answer I got was "no, we have to do it all by comittee, 
> and go through all the points in detail on the mailing list".
> 
> As it is, it looks like even if we could agree on a syntax, I'd still 
> have to go to the mailing list for every little nick and cranny in the 
> semantics, checking if it could be acceptable for both numerical and 
> non-numerical use. I just don't have the time for that.
> 
> I think the real way forward for CEPs like these might be putting them 
> on hold until we can meet in person -- if we need to design by comittee, 
> let's be a real comittee. Emails just take too long, with days going by 
> over simple misunderstandings. (Even if writing an email doesn't always 
> take long, it is really testing for my patience to spend weeks before 
> asking a question to seeing a resolution.)
> 
> The good news is that I can use what Cython time I have for better 
> mentoring Kurt and fix more boring bugs.
> 

While this is up, I just wanted to say that I'm sorry I wasted 
everybody's time with all those wierd proposals last spring, before my 
GSoC. I bet that was testing for your patience; I am truly grateful for 
the patience you showed me back then.

-- 
Dag Sverre

From greg.ewing at canterbury.ac.nz  Sat Jun 20 03:41:28 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 20 Jun 2009 13:41:28 +1200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
Message-ID: <4A3C3E48.4020108@canterbury.ac.nz>

Robert Bradshaw wrote:

> IIRC part of the motivation for int[:,:] syntax rather than  
> special_name[int, ndim=2] was that the former is actually easier for  
> humans to parse. Personally, it seems a natural extension of the int 
> [50] syntax.

I would be happier if it followed the C pattern more
closely by appearing on the right instead of the left
in a declaration, i.e.

   cdef int my_array[:,:]

As long as colons or ellipses are always used in this
form of declaration (am I right in thinking that's the
case?) then this is distinguishable from a plain array
declaration.

-- 
Greg

From dalcinl at gmail.com  Sat Jun 20 03:47:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 19 Jun 2009 22:47:11 -0300
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C3E48.4020108@canterbury.ac.nz>
References: <4A3672EC.90002@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<4A3C3E48.4020108@canterbury.ac.nz>
Message-ID: <e7ba66e40906191847m8bebdd3v792bcc30082a11d7@mail.gmail.com>

On Fri, Jun 19, 2009 at 10:41 PM, Greg Ewing<greg.ewing at canterbury.ac.nz> wrote:
> Robert Bradshaw wrote:
>
>> IIRC part of the motivation for int[:,:] syntax rather than
>> special_name[int, ndim=2] was that the former is actually easier for
>> humans to parse. Personally, it seems a natural extension of the int
>> [50] syntax.
>
> I would be happier if it followed the C pattern more
> closely by appearing on the right instead of the left
> in a declaration, i.e.
>
> ? cdef int my_array[:,:]
>
> As long as colons or ellipses are always used in this
> form of declaration (am I right in thinking that's the
> case?) then this is distinguishable from a plain array
> declaration.
>

BTW, that's also (one of) the Fortran pattern, the second below...

type, DIMENSION(bound) [,attr] :: name

type [,attr] :: name (bound)



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Sat Jun 20 06:03:45 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 21:03:45 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3BDB68.5070402@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>
	<4A3BC930.4040007@student.matnat.uio.no>
	<4A3BDB68.5070402@behnel.de>
Message-ID: <74E0AA94-BA44-4262-86AA-2C6B92624147@math.washington.edu>

On Jun 19, 2009, at 11:39 AM, Stefan Behnel wrote:

> Dag Sverre Seljebotn wrote:
>> [a couple of unnecessary words stripped]
>> Robert Bradshaw wrote:
>>> OK, how about we get the feature with the less controversial syntax,
>>> and then re-open the discussion if desired.
>>
>> Assuming you mean "simd[int, (stridespecs)]" is less controversial  
>> here:
>>
>> My main issue with this solution is what precendence it creates  
>> for how
>> project issues are settled. I don't like having a project culture  
>> where
>> stamina in these long threads is what counts in the end -- I'd much
>> rather have a clear, open voting process.
>>
>> But hear this: If both Robert and Stefan agrees about a syntax,  
>> whatever
>> the conclusion is, I won't say another word.

I don't think we agree yet, but it I think that can be postponed and  
discussing it now isn't getting anywhere quickly.

> Note that I raised a problem that you didn't solve so far, either. We
> currently have a syntax for an array type (basically the C syntax)  
> and a
> syntax for the proposed SIMD type (int[:,:]). What we do not have is a
> syntax for a memory view that does not have SIMD semantics,  
> especially not
> one that works in pure Python mode also. I assume that would use
>
>   cdef MemoryView[int,int] v
>
> in the original setup, which I would call inconsistent with all other
> syntax we have for this.

Nope, it's not on equal footing--I see little demand for a memory  
view that explicitly does not have SIMD semantics (not sure what it  
would have) but see a large demand for a memory view with such  
semantics.

- Robert


From robertwb at math.washington.edu  Sat Jun 20 06:07:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 21:07:27 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C3E48.4020108@canterbury.ac.nz>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no>
	<4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no>
	<4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no>
	<4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no>
	<4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<4A3C3E48.4020108@canterbury.ac.nz>
Message-ID: <A3C6A5D3-ABD9-4904-B880-62C0DC691E34@math.washington.edu>

On Jun 19, 2009, at 6:41 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> IIRC part of the motivation for int[:,:] syntax rather than
>> special_name[int, ndim=2] was that the former is actually easier for
>> humans to parse. Personally, it seems a natural extension of the int
>> [50] syntax.
>
> I would be happier if it followed the C pattern more
> closely by appearing on the right instead of the left
> in a declaration, i.e.
>
>    cdef int my_array[:,:]

This happens to be one of my pet peeves about C. Writing "int* a, b"  
to declare two variables of two types is, IMHO, just ugly (not to  
mention confusing to people who don't know C). We're stuck with it  
now, but I'd rather not propagate it.

> As long as colons or ellipses are always used in this
> form of declaration (am I right in thinking that's the
> case?) then this is distinguishable from a plain array
> declaration.

Yes.

- Robert

From robertwb at math.washington.edu  Sat Jun 20 06:07:52 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Jun 2009 21:07:52 -0700
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C04A6.5070001@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>
	<4A3BDB68.5070402@behnel.de>
	<4A3C04A6.5070001@student.matnat.uio.no>
Message-ID: <A9F11A44-060B-4862-B1BB-2471F9C764BE@math.washington.edu>

On Jun 19, 2009, at 2:35 PM, Dag Sverre Seljebotn wrote:

> I'm sorry, I'm pulling out of the whole thing. It just takes too much
> time on the mailing list. If somebody else want to finish the CEPs and
> implement it then I'd be thrilled though.

:(

> I suppose my real question was "can you just trust me on developing
> numerics and Cython further in this direction, I have these ideas I've
> been developing as an active numerics user and Cython developer  
> over the
> past year".

I can. Many of me questions (especially about details like  
recounting) were as more about curiosity of how you'd make it work  
than questioning your strategy. I don't think something this big  
should just be added, but if everyone's OK with "adding a new SIMD  
type, with new syntax" (which I am) then I have confidence that you  
can work out the details to make it the most useful to you and other  
numerics users.

There's been comparatively little debate about Kurt's project, so I'm  
not sure why this proposal was so much more controversial.

> I suppose the answer I got was "no, we have to do it all by comittee,
> and go through all the points in detail on the mailing list".
>
> As it is, it looks like even if we could agree on a syntax, I'd still
> have to go to the mailing list for every little nick and cranny in the
> semantics, checking if it could be acceptable for both numerical and
> non-numerical use. I just don't have the time for that.

Me either.

> I think the real way forward for CEPs like these might be putting them
> on hold until we can meet in person -- if we need to design by  
> comittee,
> let's be a real comittee. Emails just take too long, with days  
> going by
> over simple misunderstandings. (Even if writing an email doesn't  
> always
> take long, it is really testing for my patience to spend weeks before
> asking a question to seeing a resolution.)

I think language design is hard, and doesn't happen overnight, but we  
do need to come up with a new system for deciding this kind of thing.

Looking back (and into the future) here's a system that may have been  
more effective--at some point I should have thrown all the  
controversial issues on the wiki CEP page, listed the pros and cons  
of each, and allowed everyone to edit and expand. (Fortunately we're  
all mature enough here to not have edit wars.) This way the main  
points (just the data, not a talk page) would be listed in a  
succinct, non-redundant manner, at least much more so than a 50-long  
email thread. Then, once everyone feels that the CEP hits all  
relevant and important points, we should vote. Of course, this isn't  
a simple democracy--not everyone will vote (but we want to make a  
decision based on the input of those who at least care enough to hit  
reply rather than delete on their email client) and someone we've  
never heard of before doesn't have as much sway as someone who's been  
contributing for a long time, but hopefully consensus, or at least a  
clear majority, will become obvious.

> The good news is that I can use what Cython time I have for better
> mentoring Kurt and fix more boring bugs.

And hopefully more than that.

- Robert


From stefan_ml at behnel.de  Sat Jun 20 06:31:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 20 Jun 2009 06:31:31 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C04A6.5070001@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>	<4A3BDB68.5070402@behnel.de>
	<4A3C04A6.5070001@student.matnat.uio.no>
Message-ID: <4A3C6623.3060803@behnel.de>


Dag Sverre Seljebotn wrote:
> As it is, it looks like even if we could agree on a syntax, I'd still 
> have to go to the mailing list for every little nick and cranny in the 
> semantics, checking if it could be acceptable for both numerical and 
> non-numerical use. I just don't have the time for that.

Sorry, but I'm really trying to do exactly the opposite here. I would like
to have a syntax that is generic enough to support all we will need in a
clean and straight forward way, so that you can start hacking away without
having to ask back when you find details that sort-of don't really fit into
what we have decided on. If a normal type syntax is accepted, you can
design the type itself in whatever way, since you will just be
developing/reiterating/redesigning that (external looking) type and not
interfere with the language syntax.

Stefan

From greg.ewing at canterbury.ac.nz  Sat Jun 20 07:58:28 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 20 Jun 2009 17:58:28 +1200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <A3C6A5D3-ABD9-4904-B880-62C0DC691E34@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>
	<12C6BF25-7A80-432A-8A88-5C2C82526E2D@math.washington.edu>
	<4A392068.6070800@student.matnat.uio.no> <4A3948D2.3000003@behnel.de>
	<4A395FEC.2060002@student.matnat.uio.no> <4A39DD5E.4070401@behnel.de>
	<4A39F296.6020907@student.matnat.uio.no>
	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A0242.2010005@student.matnat.uio.no>
	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3A45B9.2000008@student.matnat.uio.no> <4A3A7868.6080001@behnel.de>
	<4A3A8E2A.7040607@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<D20022EC-6EF0-48D2-9447-E1EBFEEBB0A6@math.washington.edu>
	<4A3C3E48.4020108@canterbury.ac.nz>
	<A3C6A5D3-ABD9-4904-B880-62C0DC691E34@math.washington.edu>
Message-ID: <4A3C7A84.5030906@canterbury.ac.nz>

Robert Bradshaw wrote:

> This happens to be one of my pet peeves about C. Writing "int* a, b"  
> to declare two variables of two types is, IMHO, just ugly

That's because you're writing it in a perverse and
confusing way. It's really

   int *a, b

 > I'd rather not propagate it.

However, given that we're already modelling type
declarations on C, I think it's preferable to be
consistent about it. Following C in some places
but not others would be confusing.

-- 
Greg

From stefan_ml at behnel.de  Sat Jun 20 09:26:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 20 Jun 2009 09:26:27 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3672EC.90002@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>
Message-ID: <4A3C8F23.3080202@behnel.de>

Starting over here.

I looked through the CEPs and found CEP 506 about parametrised types which
describes pretty much what we need here. So, why not just use that?

http://wiki.cython.org/enhancements/typeparameters

It basically gives us parametrised types as 'type factories', and you can
pass parameters as supported by Python's call syntax, which implies that
this works just as well in pure Python mode. This allows Dag to design the
behaviour of the SIMD type in any way he sees fit and relieves us from the
need to overload the bracket syntax for one special type. It would also
work for the normal array type, so that we can easily distinguish it from a
C array declaration (in case we need to, which I consider quite possible).

I'm currently writing up CEP 519 to present my view on all this in a
somewhat structured way. I hope this will get us back to a point where
discussing makes sense again.

http://wiki.cython.org/enhancements/arraysandviews

Stefan

From dagss at student.matnat.uio.no  Sat Jun 20 10:08:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 20 Jun 2009 10:08:29 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C6623.3060803@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>	<4A3BDB68.5070402@behnel.de>	<4A3C04A6.5070001@student.matnat.uio.no>
	<4A3C6623.3060803@behnel.de>
Message-ID: <4A3C98FD.5080602@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> As it is, it looks like even if we could agree on a syntax, I'd still 
>> have to go to the mailing list for every little nick and cranny in the 
>> semantics, checking if it could be acceptable for both numerical and 
>> non-numerical use. I just don't have the time for that.
> 
> Sorry, but I'm really trying to do exactly the opposite here. I would like
> to have a syntax that is generic enough to support all we will need in a
> clean and straight forward way, so that you can start hacking away without
> having to ask back when you find details that sort-of don't really fit into
> what we have decided on. If a normal type syntax is accepted, you can
> design the type itself in whatever way, since you will just be
> developing/reiterating/redesigning that (external looking) type and not
> interfere with the language syntax.

The problem is just the time this all takes to get to that point -- 
every minute spent on emails is less time spent on actual coding. I'm 
not only wasting my time, but also your and Robert's time and ultimately 
making sure nothing is done about getting 0.12 out etc. etc.

About syntax, I should likely just have yielded and be over with it, but 
you didn't care to explain to my (when I pointed it out bluntly) why I 
should yield to a view that is in every way in miniority.

And then I woke up and asked myself if life is long enough for this kind 
of thing.


Well, threads (unfortunately) being an irresistible temptation, I'll say 
this last thing (in case it can help you understand my position):

- int[:,:] is just so much convenient and friendly for numerical users, 
and looks better for something I'd use that much

- I agree with Robert that Cython is one of those "transparent 
languages" where it should be possible to easily guess what kind of C 
code will be generated, and views seem too magic to be put alongside 
structs and extension types in the syntax.

- I too don't see what the point is with the non-SIMD view, it has no 
real usecase as it is just the SIMD view with less features. It may make 
you intellectually happier as SIMD is then "less integrated", but I 
think it will just serve to confuse users who will ask just that 
question: "Why are there two view types?" (And pretending it is 
inheritance or similar just seems dishonest, as there's so much magic 
things going on for SIMD, and won't can't inherit from the SIMD view type.)

Note: I'll try very, very hard to not make a reply to any reply you make 
to this last part. But I'd love a comment from you on how you think 
Cython should be managed and these long threads avoided etc., re: 
Robert's mail.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Jun 20 10:29:43 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 20 Jun 2009 10:29:43 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C98FD.5080602@student.matnat.uio.no>
References: <4A3672EC.90002@student.matnat.uio.no>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>	<4A3BDB68.5070402@behnel.de>	<4A3C04A6.5070001@student.matnat.uio.no>	<4A3C6623.3060803@behnel.de>
	<4A3C98FD.5080602@student.matnat.uio.no>
Message-ID: <4A3C9DF7.6030407@behnel.de>


Dag Sverre Seljebotn wrote:
> The problem is just the time this all takes to get to that point -- 
> every minute spent on emails is less time spent on actual coding. I'm 
> not only wasting my time, but also your and Robert's time and ultimately 
> making sure nothing is done about getting 0.12 out etc. etc.
> 
> About syntax, I should likely just have yielded and be over with it, but 
> you didn't care to explain to my (when I pointed it out bluntly) why I 
> should yield to a view that is in every way in miniority.

You know what? I think it's best if I just give in at this point. I'm not
the one driving this and I also won't be the one implementing it. If you
prefer going your way here, fine. Get the code working and get it
integrated into the language. Don't expect much brain food from my side,
just cut your way through it. You showed often enough that you can pull off
similarly sized things, so I'm sure you will manage to get this working.

Stefan

From dagss at student.matnat.uio.no  Sat Jun 20 10:46:46 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 20 Jun 2009 10:46:46 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3C9DF7.6030407@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>	<4A3BDB68.5070402@behnel.de>	<4A3C04A6.5070001@student.matnat.uio.no>	<4A3C6623.3060803@behnel.de>	<4A3C98FD.5080602@student.matnat.uio.no>
	<4A3C9DF7.6030407@behnel.de>
Message-ID: <4A3CA1F6.3000301@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn wrote:
>> The problem is just the time this all takes to get to that point -- 
>> every minute spent on emails is less time spent on actual coding. I'm 
>> not only wasting my time, but also your and Robert's time and ultimately 
>> making sure nothing is done about getting 0.12 out etc. etc.
>>
>> About syntax, I should likely just have yielded and be over with it, but 
>> you didn't care to explain to my (when I pointed it out bluntly) why I 
>> should yield to a view that is in every way in miniority.
> 
> You know what? I think it's best if I just give in at this point. I'm not
> the one driving this and I also won't be the one implementing it. If you
> prefer going your way here, fine. Get the code working and get it
> integrated into the language. Don't expect much brain food from my side,
> just cut your way through it. You showed often enough that you can pull off
> similarly sized things, so I'm sure you will manage to get this working.

Thanks for the confidence. I'll give it a few days and see how I feel 
about working on it when I've cooled down a bit (and talked over some of 
the details with Kurt and seen how much he gets room for in the GSoC -- 
hopefully by voice on Skype to give my fingers a break).

In a few weeks, when things are calmer, we should start a thread about 
how we can avoid similar long threads in the future (sometimes not 
having a feature is better than using so much time discussing it). 
Robert gave it a good start with the new CEP procedure, I still think 
there's more to say though.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Jun 20 11:11:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 20 Jun 2009 11:11:46 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <A9F11A44-060B-4862-B1BB-2471F9C764BE@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no>	<4A392068.6070800@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>	<4A3BDB68.5070402@behnel.de>	<4A3C04A6.5070001@student.matnat.uio.no>
	<A9F11A44-060B-4862-B1BB-2471F9C764BE@math.washington.edu>
Message-ID: <4A3CA7D2.7080805@behnel.de>


Robert Bradshaw wrote:
> There's been comparatively little debate about Kurt's project, so I'm  
> not sure why this proposal was so much more controversial.

My reaction emerged because Dag failed to provide enough information in the
CEP(s) to clear up the implications. He left them out because he knew what
he was talking about anyway, but I literally had to pull the details (even
goals!) out of his fingers one by one before they even received mention in
the CEP. And the more details I saw, the more I got the feeling that this
was just another application of parametrised types, as discussed several
times before. Solving a problem once is sometimes better than solving many
problems many times because the real problem seems too far away. Although
the current state of the discussion may mean that we at least end up with a
bird in the hand. Wouldn't be the first time.


> I think language design is hard, and doesn't happen overnight, but we
> do need to come up with a new system for deciding this kind of thing.
> 
> Looking back (and into the future) here's a system that may have been  
> more effective--at some point I should have thrown all the  
> controversial issues on the wiki CEP page, listed the pros and cons  
> of each, and allowed everyone to edit and expand. (Fortunately we're  
> all mature enough here to not have edit wars.)

And if everyone who brings in criticism is forced to either add it to the
CEP or even do some larger editing on it, I think you will quickly see who
is really interested and constructive. It will also break down the size of
e-mail threads, as a revised CEP is a good starting point for a new discussion.

> This way the main  
> points (just the data, not a talk page) would be listed in a  
> succinct, non-redundant manner, at least much more so than a 50-long  
> email thread.

Very true.

> Then, once everyone feels that the CEP hits all  
> relevant and important points, we should vote. Of course, this isn't 
> a simple democracy--not everyone will vote (but we want to make a  
> decision based on the input of those who at least care enough to hit  
> reply rather than delete on their email client) and someone we've  
> never heard of before doesn't have as much sway as someone who's been  
> contributing for a long time, but hopefully consensus, or at least a  
> clear majority, will become obvious.

I think this will become a lot simpler when there really is a mostly
complete CEP to point your fingers to. Agreeing that a CEP is complete
pretty much implies a decision what should be done with it. And if there
isn't a clear decision, well, then there's the need for a BD (not sure
about the FL).

Personally, I would still consider the current CEPs far from complete (not
feature-wise, rather in the sense that they lack discussion points and
alternatives), so rushing to get them accepted and implemented feels wrong
to me. But as I said, I'm out of that discussion anyway.

Stefan

From dagss at student.matnat.uio.no  Sat Jun 20 11:28:20 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 20 Jun 2009 11:28:20 +0200
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <4A3CA7D2.7080805@behnel.de>
References: <4A3672EC.90002@student.matnat.uio.no>	<4A3948D2.3000003@behnel.de>	<4A395FEC.2060002@student.matnat.uio.no>	<4A39DD5E.4070401@behnel.de>	<4A39F296.6020907@student.matnat.uio.no>	<5df61eea721755ea46ad8550f270aac2.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A0242.2010005@student.matnat.uio.no>	<ea24328e9cdac5cfe4e0222972b6d4ed.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3A45B9.2000008@student.matnat.uio.no>	<4A3A7868.6080001@behnel.de>	<4A3A8E2A.7040607@student.matnat.uio.no>	<4A3B15CE.2090104@behnel.de>	<4A3B6579.1040007@student.matnat.uio.no>	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<4A3BB81B.3020107@student.matnat.uio.no>	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>	<4A3BC930.4040007@student.matnat.uio.no>	<4A3BDB68.5070402@behnel.de>	<4A3C04A6.5070001@student.matnat.uio.no>	<A9F11A44-060B-4862-B1BB-2471F9C764BE@math.washington.edu>
	<4A3CA7D2.7080805@behnel.de>
Message-ID: <4A3CABB4.8090203@student.matnat.uio.no>

Stefan Behnel wrote:
> Robert Bradshaw wrote:
>> There's been comparatively little debate about Kurt's project, so I'm  
>> not sure why this proposal was so much more controversial.
> 
> My reaction emerged because Dag failed to provide enough information in the
> CEP(s) to clear up the implications. He left them out because he knew what
> he was talking about anyway, but I literally had to pull the details (even
> goals!) out of his fingers one by one before they even received mention in
> the CEP. And the more details I saw, the more I got the feeling that this

This critisicm is all true, and I hope I'm wiser until the next time I 
propose something.

To my defense it wasn't intentional, and it resulted from having gone 
over the details some times before on the mailing list in 
numerically-oriented threads which you weren't involved in. NumPy users 
understood what I was getting at much quicker -- but the CEP, as I 
presented it, definitely wasn't ready to be presented to people not 
familiar with NumPy, which means it wasn't ready for discussion by the 
Cython community at large -- we shouldn't be assuming NumPy familiarity 
in here.

It wasn't until very late I realized you hadn't used NumPy at all -- 
which of course was a major mistake from my side.

Apologies.

-- 
Dag Sverre

From kwmsmith at gmail.com  Sat Jun 20 20:42:24 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sat, 20 Jun 2009 13:42:24 -0500
Subject: [Cython] Cython array type: Summary, introducing CEP 518
In-Reply-To: <A9F11A44-060B-4862-B1BB-2471F9C764BE@math.washington.edu>
References: <4A3672EC.90002@student.matnat.uio.no> <4A3B15CE.2090104@behnel.de>
	<4A3B6579.1040007@student.matnat.uio.no>
	<359ad7838fe37d62c1f4bf05d4299cba.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4A3BB81B.3020107@student.matnat.uio.no>
	<037717D7-F62A-4713-95F6-DE9140519449@math.washington.edu>
	<4A3BC930.4040007@student.matnat.uio.no> <4A3BDB68.5070402@behnel.de>
	<4A3C04A6.5070001@student.matnat.uio.no>
	<A9F11A44-060B-4862-B1BB-2471F9C764BE@math.washington.edu>
Message-ID: <c266a7930906201142o25e6f62an687a04673320c7b5@mail.gmail.com>

On Fri, Jun 19, 2009 at 11:07 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:

>
> There's been comparatively little debate about Kurt's project, so I'm
> not sure why this proposal was so much more controversial.

If I may elbow in here a bit:

The original proposal tossed around was CEP 401, that introduced
fortran-specific syntax ('module', 'fortran' etc) and the new array
syntax.  It didn't get very far -- largely because the new fortran
keywords were  perceived as feature creep and not in keeping with
Cython's philosophy. The proposal was improved to making an external
fortran wrapper tool and excluding fortran-specific syntax -- and I
genuinely think it was an improvement -- it keeps things nicely
modular and the wrapper tool can evolve independently.  Ideally the
wrapper tool will get to the point of standing on its own feet.  The
project needs some way to pass arrays from Cython to external code,
which is why it is tied up with the new array syntax, etc.  As I
recall, the new buffer/array syntax discussion was postponed, and that
discussion has resulted in the current thread ;-)

Kurt

From dagss at student.matnat.uio.no  Sun Jun 21 00:04:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 21 Jun 2009 00:04:18 +0200
Subject: [Cython] Benevolent Dictator
Message-ID: <4A3D5CE2.20907@student.matnat.uio.no>

Me and Stefan Behnel has decided to nominate Robert Bradshaw as Cython's 
Benevolent Dictator (not necesarrily For Life), and he has accepted.

Knowing Robert, the Cython project will continue to stay open and 
largely consensus-driven. Yet the hope is that this change will allow us 
to move forward more quickly when there is confusion or conflict.

I must say I don't really expect any opposition to this change in 
Cython's project management structure, but please speak now (or forever 
hold your peace).

On behalf of the Benevolent Dictator nominating committee,
Dag Sverre

From kwmsmith at gmail.com  Sun Jun 21 02:58:38 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Sat, 20 Jun 2009 19:58:38 -0500
Subject: [Cython] Benevolent Dictator
In-Reply-To: <4A3D5CE2.20907@student.matnat.uio.no>
References: <4A3D5CE2.20907@student.matnat.uio.no>
Message-ID: <c266a7930906201758i6c7f8773p4497350339b71b30@mail.gmail.com>

On Sat, Jun 20, 2009 at 5:04 PM, Dag Sverre
Seljebotn<dagss at student.matnat.uio.no> wrote:
> Me and Stefan Behnel has decided to nominate Robert Bradshaw as Cython's
> Benevolent Dictator (not necesarrily For Life), and he has accepted.
>
> Knowing Robert, the Cython project will continue to stay open and
> largely consensus-driven. Yet the hope is that this change will allow us
> to move forward more quickly when there is confusion or conflict.
>
> I must say I don't really expect any opposition to this change in
> Cython's project management structure, but please speak now (or forever
> hold your peace).
>
> On behalf of the Benevolent Dictator nominating committee,
> Dag Sverre

+1 :-)

Kurt

From robertwb at math.washington.edu  Sun Jun 21 04:09:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 20 Jun 2009 19:09:51 -0700
Subject: [Cython] Benevolent Dictator
In-Reply-To: <c266a7930906201758i6c7f8773p4497350339b71b30@mail.gmail.com>
References: <4A3D5CE2.20907@student.matnat.uio.no>
	<c266a7930906201758i6c7f8773p4497350339b71b30@mail.gmail.com>
Message-ID: <9DF1266C-F232-4FB4-A98A-24336BE3D77B@math.washington.edu>

On Jun 20, 2009, at 5:58 PM, Kurt Smith wrote:

> On Sat, Jun 20, 2009 at 5:04 PM, Dag Sverre
> Seljebotn<dagss at student.matnat.uio.no> wrote:
>> Me and Stefan Behnel has decided to nominate Robert Bradshaw as  
>> Cython's
>> Benevolent Dictator (not necesarrily For Life), and he has accepted.
>>
>> Knowing Robert, the Cython project will continue to stay open and
>> largely consensus-driven. Yet the hope is that this change will  
>> allow us
>> to move forward more quickly when there is confusion or conflict.
>>
>> I must say I don't really expect any opposition to this change in
>> Cython's project management structure, but please speak now (or  
>> forever
>> hold your peace).
>>
>> On behalf of the Benevolent Dictator nominating committee,
>> Dag Sverre
>
> +1 :-)

Thanks. I'll do my best to provide dictatorial direction when we need  
it. Most of the time, however, my opinion is just my opinion, not a  
dictatorial pronouncement, open to challenge (or agreement) just like  
any other developer (or user for that matter).

- Robert


From vic.kelson at gmail.com  Sun Jun 21 12:53:02 2009
From: vic.kelson at gmail.com (Vic Kelson)
Date: Sun, 21 Jun 2009 06:53:02 -0400
Subject: [Cython] Benevolent Dictator
In-Reply-To: <4A3D5CE2.20907@student.matnat.uio.no>
Message-ID: <C663894E.7DB4%vic.kelson@gmail.com>

I for one welcome our new overlord. Congratulations, Robert, and THANKS!
--v


On 6/20/09 6:04 PM, "Dag Sverre Seljebotn" <dagss at student.matnat.uio.no>
wrote:

> Me and Stefan Behnel has decided to nominate Robert Bradshaw as Cython's
> Benevolent Dictator (not necesarrily For Life), and he has accepted.
> 
> Knowing Robert, the Cython project will continue to stay open and
> largely consensus-driven. Yet the hope is that this change will allow us
> to move forward more quickly when there is confusion or conflict.
> 
> I must say I don't really expect any opposition to this change in
> Cython's project management structure, but please speak now (or forever
> hold your peace).
> 
> On behalf of the Benevolent Dictator nominating committee,
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev



From daniel.yarlett at gmail.com  Mon Jun 22 00:28:12 2009
From: daniel.yarlett at gmail.com (Daniel Yarlett)
Date: Sun, 21 Jun 2009 23:28:12 +0100
Subject: [Cython] Cython class inheriting from Python class (defined in
	another file)
Message-ID: <B300FA2D-629C-4690-ADA2-12DC9D4DB903@gmail.com>

Hi All,

A fairly brief question: how does one get a Cython class to inherit  
from a Python class defined in another file? Say I have class A  
defined in foo.py, and I want to define a class, B, in a separate .pyx  
file, that inherits from A. In my .pyx file I currently have something  
like the following:

from foo import A
...
cdef class B(A):
	...

but I get an "'A' is not a type name" error when building. I found the  
following in the documentation.

> A complete definition of the base type must be available to Cython,  
> so if the base type is a built-in type, it must have been previously  
> declared as an extern extension type. If the base type is defined in  
> another Cython module, it must either be declared as an extern  
> extension type or imported using the cimport statement.

I guess instead of my import statement I need to have something like

cdef extern from "foo.py":
	...

at the top of my .pyx file, but can't figure out what type information  
I need to provide Cython. Apologies, as I'm sure the answer is  
obvious, but any help would be greatly appreciated.

Thanks,

Dan.
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090621/c85f1719/attachment.htm 

From jasone at canonware.com  Mon Jun 22 01:31:24 2009
From: jasone at canonware.com (Jason Evans)
Date: Sun, 21 Jun 2009 16:31:24 -0700
Subject: [Cython] Cython class inheriting from Python class (defined in
 another file)
In-Reply-To: <B300FA2D-629C-4690-ADA2-12DC9D4DB903@gmail.com>
References: <B300FA2D-629C-4690-ADA2-12DC9D4DB903@gmail.com>
Message-ID: <4A3EC2CC.1080902@canonware.com>

Daniel Yarlett wrote:
> A fairly brief question: how does one get a Cython class to inherit from 
> a Python class defined in another file? Say I have class A defined in 
> foo.py, and I want to define a class, B, in a separate .pyx file, that 
> inherits from A. In my .pyx file I currently have something like the 
> following:
> 
> from foo import A
> ...
> cdef class B(A):
> ...
> 

What about:

   from foo cimport A
            ^

Jason

From stefan_ml at behnel.de  Mon Jun 22 08:30:54 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 22 Jun 2009 08:30:54 +0200
Subject: [Cython] Cython class inheriting from Python class (defined in
 another file)
In-Reply-To: <B300FA2D-629C-4690-ADA2-12DC9D4DB903@gmail.com>
References: <B300FA2D-629C-4690-ADA2-12DC9D4DB903@gmail.com>
Message-ID: <4A3F251E.7020608@behnel.de>


Daniel Yarlett wrote:
> A fairly brief question: how does one get a Cython class to inherit from
> a Python class defined in another file?

This cannot work. cdef classes need a complete inheritance chain of cdef
classes.

Any reason you can't just use a normal Python subclass?

Stefan


From jonovik at gmail.com  Tue Jun 23 10:40:01 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Tue, 23 Jun 2009 08:40:01 +0000 (UTC)
Subject: [Cython] How to use my Cython extension class outside of its own
	module?
Message-ID: <loom.20090623T072004-228@post.gmane.org>

I have a Cython extension class that wraps the N_Vector type [1] from the 
Sundials solver suite for differential equations. (Thanks to Dag Sverre 
Seljebotn for guidance [2].) However, I have problems using the class outside 
of the module where it was defined (cynvector.pyx).

The following couple of trivial operations work when included at the end of 
cynvector.pyx:

cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector of length 2
cdef N_Vector p = v.nvector # alias to an attribute of the CythonNVector

(I've learned that both v and p need to have their type specified by 
"cdef ...", otherwise they default to generic Python objects and the "nvector" 
attribute isn't recognized.)

However, the same statements fail if I move them to another file, say 
test_cynvector.pyx:

from sundials cimport * # sundials.pxd contains type declaration for N_Vector
from cynvector import *
cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector of length 2
cdef N_Vector p = v.nvector # alias to an attribute of the CythonNVector

fails with:

cdef CythonNVector v = CythonNVector(2)
    ^
'CythonNVector' is not a type identifier

Why isn't type CythonNVector recognized outside of cynvector.pyx? A similar 
error results if I try "from cynvector cimport CythonNVector"; then I get "Name 
'CythonNVector' not declared in module 'cynvector'".


Thanks in advance for any help.

[1] https://computation.llnl.gov/casc/sundials/documentation/cv_guide/node7.html
[2] http://article.gmane.org/gmane.comp.python.cython.devel/6556


Here is the essence of cynvector.pyx, based on [2]:

== cynvector.pyx ==
cimport numpy as np
from sundials cimport * # sundials.pxd declares N_Vector and friends

cdef extern from "pyarray_set_base.h":
    void PyArray_Set_BASE(np.ndarray arr, object obj)

cdef extern from "numpy/arrayobject.h":
    void import_array()
    object PyArray_SimpleNewFromData(int nd, np.npy_intp* dims, 
        int typenum, void* data)

import_array()

cdef class CythonNVector:
    cdef N_Vector nvector
    def __init__(self, length):
        self.nvector = N_VNew_Serial(length)

    def __dealloc__(self):
        N_VDestroy_Serial(self.nvector)

    cpdef toarray(self):
        cdef np.npy_intp* shape =  <np.npy_intp *>(
            &(<N_VectorContent_Serial>(self.nvector).content).length)
        assert sizeof(realtype) == sizeof(double)
        cdef np.ndarray result = PyArray_SimpleNewFromData(1, shape, 
            np.NPY_DOUBLE, 
            <void*>((<N_VectorContent_Serial>(self.nvector).content).data))
        PyArray_Set_BASE(result, self)
        return result

# Testing: These statements do work at the end of cynvector.pyx.
cdef CythonNVector v = CythonNVector(2)
cdef N_Vector p = v.nvector



From robertwb at math.washington.edu  Tue Jun 23 10:47:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 23 Jun 2009 01:47:11 -0700
Subject: [Cython] How to use my Cython extension class outside of its
	own module?
In-Reply-To: <loom.20090623T072004-228@post.gmane.org>
References: <loom.20090623T072004-228@post.gmane.org>
Message-ID: <A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>

On Jun 23, 2009, at 1:40 AM, Jon Olav Vik wrote:

> I have a Cython extension class that wraps the N_Vector type [1]  
> from the
> Sundials solver suite for differential equations. (Thanks to Dag  
> Sverre
> Seljebotn for guidance [2].) However, I have problems using the  
> class outside
> of the module where it was defined (cynvector.pyx).
>
> The following couple of trivial operations work when included at  
> the end of
> cynvector.pyx:
>
> cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector  
> of length 2
> cdef N_Vector p = v.nvector # alias to an attribute of the  
> CythonNVector
>
> (I've learned that both v and p need to have their type specified by
> "cdef ...", otherwise they default to generic Python objects and  
> the "nvector"
> attribute isn't recognized.)
>
> However, the same statements fail if I move them to another file, say
> test_cynvector.pyx:
>
> from sundials cimport * # sundials.pxd contains type declaration  
> for N_Vector
> from cynvector import *
> cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector  
> of length 2
> cdef N_Vector p = v.nvector # alias to an attribute of the  
> CythonNVector
>
> fails with:
>
> cdef CythonNVector v = CythonNVector(2)
>     ^
> 'CythonNVector' is not a type identifier
>
> Why isn't type CythonNVector recognized outside of cynvector.pyx? A  
> similar
> error results if I try "from cynvector cimport CythonNVector"; then  
> I get "Name
> 'CythonNVector' not declared in module 'cynvector'".

What does your cynvector.pxd look like? You probably need to declare  
CythonNVector there so other modules can cimport it.

>
>
> Thanks in advance for any help.
>
> [1] https://computation.llnl.gov/casc/sundials/documentation/ 
> cv_guide/node7.html
> [2] http://article.gmane.org/gmane.comp.python.cython.devel/6556
>
>
> Here is the essence of cynvector.pyx, based on [2]:
>
> == cynvector.pyx ==
> cimport numpy as np
> from sundials cimport * # sundials.pxd declares N_Vector and friends
>
> cdef extern from "pyarray_set_base.h":
>     void PyArray_Set_BASE(np.ndarray arr, object obj)
>
> cdef extern from "numpy/arrayobject.h":
>     void import_array()
>     object PyArray_SimpleNewFromData(int nd, np.npy_intp* dims,
>         int typenum, void* data)
>
> import_array()
>
> cdef class CythonNVector:
>     cdef N_Vector nvector
>     def __init__(self, length):
>         self.nvector = N_VNew_Serial(length)
>
>     def __dealloc__(self):
>         N_VDestroy_Serial(self.nvector)
>
>     cpdef toarray(self):
>         cdef np.npy_intp* shape =  <np.npy_intp *>(
>             &(<N_VectorContent_Serial>(self.nvector).content).length)
>         assert sizeof(realtype) == sizeof(double)
>         cdef np.ndarray result = PyArray_SimpleNewFromData(1, shape,
>             np.NPY_DOUBLE,
>             <void*>((<N_VectorContent_Serial> 
> (self.nvector).content).data))
>         PyArray_Set_BASE(result, self)
>         return result
>
> # Testing: These statements do work at the end of cynvector.pyx.
> cdef CythonNVector v = CythonNVector(2)
> cdef N_Vector p = v.nvector
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Tue Jun 23 10:53:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 23 Jun 2009 10:53:44 +0200
Subject: [Cython] How to use my Cython extension class outside of its
 own module?
In-Reply-To: <A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>
References: <loom.20090623T072004-228@post.gmane.org>
	<A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>
Message-ID: <4A409818.2090801@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 23, 2009, at 1:40 AM, Jon Olav Vik wrote:
> 
>> I have a Cython extension class that wraps the N_Vector type [1]  
>> from the
>> Sundials solver suite for differential equations. (Thanks to Dag  
>> Sverre
>> Seljebotn for guidance [2].) However, I have problems using the  
>> class outside
>> of the module where it was defined (cynvector.pyx).
>>
>> The following couple of trivial operations work when included at  
>> the end of
>> cynvector.pyx:
>>
>> cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector  
>> of length 2
>> cdef N_Vector p = v.nvector # alias to an attribute of the  
>> CythonNVector
>>
>> (I've learned that both v and p need to have their type specified by
>> "cdef ...", otherwise they default to generic Python objects and  
>> the "nvector"
>> attribute isn't recognized.)
>>
>> However, the same statements fail if I move them to another file, say
>> test_cynvector.pyx:
>>
>> from sundials cimport * # sundials.pxd contains type declaration  
>> for N_Vector
>> from cynvector import *
>> cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector  
>> of length 2
>> cdef N_Vector p = v.nvector # alias to an attribute of the  
>> CythonNVector
>>
>> fails with:
>>
>> cdef CythonNVector v = CythonNVector(2)
>>     ^
>> 'CythonNVector' is not a type identifier
>>
>> Why isn't type CythonNVector recognized outside of cynvector.pyx? A  
>> similar
>> error results if I try "from cynvector cimport CythonNVector"; then  
>> I get "Name
>> 'CythonNVector' not declared in module 'cynvector'".
> 
> What does your cynvector.pxd look like? You probably need to declare  
> CythonNVector there so other modules can cimport it.

Actually there probably is no -- it says import, not cimport.

So you need a cynvector.pxd as well, and cimport in in addition to the 
normal import.

(There's a wishlist item for automatically deducing this from pyx files, 
but it isn't there yet.)

http://docs.cython.org/docs/sharing_declarations.html


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue Jun 23 10:56:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 23 Jun 2009 10:56:14 +0200
Subject: [Cython] How to use my Cython extension class outside of its
 own module?
In-Reply-To: <4A409818.2090801@student.matnat.uio.no>
References: <loom.20090623T072004-228@post.gmane.org>	<A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>
	<4A409818.2090801@student.matnat.uio.no>
Message-ID: <4A4098AE.40605@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> On Jun 23, 2009, at 1:40 AM, Jon Olav Vik wrote:
>>
>>> I have a Cython extension class that wraps the N_Vector type [1]  
>>> from the
>>> Sundials solver suite for differential equations. (Thanks to Dag  
>>> Sverre
>>> Seljebotn for guidance [2].) However, I have problems using the  
>>> class outside
>>> of the module where it was defined (cynvector.pyx).
>>>
>>> The following couple of trivial operations work when included at  
>>> the end of
>>> cynvector.pyx:
>>>
>>> cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector  
>>> of length 2
>>> cdef N_Vector p = v.nvector # alias to an attribute of the  
>>> CythonNVector
>>>
>>> (I've learned that both v and p need to have their type specified by
>>> "cdef ...", otherwise they default to generic Python objects and  
>>> the "nvector"
>>> attribute isn't recognized.)
>>>
>>> However, the same statements fail if I move them to another file, say
>>> test_cynvector.pyx:
>>>
>>> from sundials cimport * # sundials.pxd contains type declaration  
>>> for N_Vector
>>> from cynvector import *
>>> cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector  
>>> of length 2
>>> cdef N_Vector p = v.nvector # alias to an attribute of the  
>>> CythonNVector
>>>
>>> fails with:
>>>
>>> cdef CythonNVector v = CythonNVector(2)
>>>     ^
>>> 'CythonNVector' is not a type identifier
>>>
>>> Why isn't type CythonNVector recognized outside of cynvector.pyx? A  
>>> similar
>>> error results if I try "from cynvector cimport CythonNVector"; then  
>>> I get "Name
>>> 'CythonNVector' not declared in module 'cynvector'".
>> What does your cynvector.pxd look like? You probably need to declare  
>> CythonNVector there so other modules can cimport it.
> 
> Actually there probably is no -- it says import, not cimport.
> 
> So you need a cynvector.pxd as well, and cimport in in addition to the 
> normal import.
> 
> (There's a wishlist item for automatically deducing this from pyx files, 
> but it isn't there yet.)
> 
> http://docs.cython.org/docs/sharing_declarations.html
> 

BTW as reported the necesarry pxd files in my tutorial examples are 
missing from the upload and I haven't got around to uploading them yet; 
in case you looked at the tutorial for inspiration and got confused by that.

-- 
Dag Sverre

From jonovik at gmail.com  Tue Jun 23 12:26:57 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Tue, 23 Jun 2009 10:26:57 +0000 (UTC)
Subject: [Cython] How to use my Cython extension class outside of its
	own module?
References: <loom.20090623T072004-228@post.gmane.org>
	<A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>
	<4A409818.2090801@student.matnat.uio.no>
Message-ID: <loom.20090623T091132-146@post.gmane.org>

Dag Sverre Seljebotn <dagss at ...> writes:
> Robert Bradshaw wrote:
> > On Jun 23, 2009, at 1:40 AM, Jon Olav Vik wrote:
> >> Why isn't type CythonNVector recognized outside of cynvector.pyx? A  
> >> similar error results if I try "from cynvector cimport CythonNVector"; 
> >> then I get "Name 'CythonNVector' not declared in module 'cynvector'".
> > 
> > What does your cynvector.pxd look like? You probably need to declare  
> > CythonNVector there so other modules can cimport it.
> 
> Actually there probably is no -- it says import, not cimport.
> 
> So you need a cynvector.pxd as well, and cimport in in addition to the 
> normal import.
> 
> (There's a wishlist item for automatically deducing this from pyx files, 
> but it isn't there yet.)
> 
> http://docs.cython.org/docs/sharing_declarations.html

Thanks, I've got it working now. (I did use a pxd file for the Sundials 
declarations, but didn't realize that I needed one for my Cython module too.)

== test_cynvector.pyx ==
from cynvector cimport *
from sundials cimport *
cdef CythonNVector v = CythonNVector(2) # allocates empty N_Vector of length 2
cdef N_Vector p = v.nvector # alias to an attribute of the CythonNVector
print v.toarray() # uninitialized memory, but a success nevertheless...

As it turns out, I only needed the cimport for my example. To both import and 
cimport the module, however, it seems that I cannot use *:

from cynvector import *
from cynvector cimport * # TypeError: Cannot overwrite C type CythonNVector

Importing and then cimporting the module under the same name works fine, though:

import cynvector
cimport cynvector
cimport sundials
cdef cynvector.CythonNVector v = cynvector.CythonNVector(2)
cdef sundials.N_Vector p = v.nvector

== cynvector.pxd ==
"""Declarations for Cython bridge between Numpy array and Sundials N_Vector"""
from sundials cimport * # sundials.pxd defines N_Vector
cdef class CythonNVector:
    cdef N_Vector nvector
    cpdef toarray(self)

== cynvector.pyx ==
cimport numpy as np
from sundials cimport * # sundials.pxd declares N_Vector and friends

cdef extern from "pyarray_set_base.h":
    void PyArray_Set_BASE(np.ndarray arr, object obj)

cdef extern from "numpy/arrayobject.h":
    void import_array()
    object PyArray_SimpleNewFromData(int nd, np.npy_intp* dims, 
        int typenum, void* data)

import_array()

cdef class CythonNVector:
    cdef N_Vector nvector
    def __init__(self, length):
        self.nvector = N_VNew_Serial(length)

    def __dealloc__(self):
        N_VDestroy_Serial(self.nvector)

    cpdef toarray(self):
        cdef np.npy_intp* shape =  <np.npy_intp *>(
            &(<N_VectorContent_Serial>(self.nvector).content).length)
        assert sizeof(realtype) == sizeof(double)
        cdef np.ndarray result = PyArray_SimpleNewFromData(1, shape, 
            np.NPY_DOUBLE, 
            <void*>((<N_VectorContent_Serial>(self.nvector).content).data))
        PyArray_Set_BASE(result, self)
        return result

== setup_test_cynvector.py ==
"""Compile and test with
rm -rf build test_cynvector.so test_cynvector.c && \
python setup_test_cynvector.py build_ext --inplace && \
python -c "import test_cynvector"
"""

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy as np

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("test_cynvector",
            ["test_cynvector.pyx"],
            include_dirs=['/site/VERSIONS/sundials-2.3.0/include', 
np.get_include ()],
            library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'],
            libraries=['sundials_cvode', 'sundials_nvecserial']),
    ]
)



From jonovik at gmail.com  Tue Jun 23 13:33:32 2009
From: jonovik at gmail.com (Jon Olav Vik)
Date: Tue, 23 Jun 2009 11:33:32 +0000 (UTC)
Subject: [Cython]
	=?utf-8?q?How_to_use_my_Cython_extension_class_outside_o?=
	=?utf-8?q?f_its=09own_module=3F?=
References: <loom.20090623T072004-228@post.gmane.org>
	<A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>
	<4A409818.2090801@student.matnat.uio.no>
	<loom.20090623T091132-146@post.gmane.org>
Message-ID: <loom.20090623T111927-627@post.gmane.org>

Jon Olav Vik <jonovik at ...> writes:
> == cynvector.pxd ==
> from sundials cimport * # sundials.pxd defines N_Vector
> cdef class CythonNVector:
>     cdef N_Vector nvector
>     cpdef toarray(self)
> == cynvector.pyx ==
[...]
> cdef class CythonNVector:
>     cdef N_Vector nvector # <-- ERROR
[...]

Gotcha: When I copied declarations from cynvector.pyx to cynvector.pxd, I 
should have removed the original "cdef N_Vector nvector" from the .pyx file. As 
it was, I got a "'nvector' redeclared" error when re-compiling my example. The 
reason I didn't see it at first is that I compiled cynvector.so just from 
cynvector.pyx first, before creating cynvector.pxd.

To summarize: "Formal" declarations of object attributes in filename.pxd are 
automatically available in filename.pyx and should not be repeated there. 
(Functions and class definitions _are_ repeated, for implementation.)



From dagss at student.matnat.uio.no  Tue Jun 23 13:44:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 23 Jun 2009 13:44:51 +0200
Subject: [Cython] How to use my Cython extension class outside of its
 own module?
In-Reply-To: <loom.20090623T111927-627@post.gmane.org>
References: <loom.20090623T072004-228@post.gmane.org>	<A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>	<4A409818.2090801@student.matnat.uio.no>	<loom.20090623T091132-146@post.gmane.org>
	<loom.20090623T111927-627@post.gmane.org>
Message-ID: <4A40C033.1010302@student.matnat.uio.no>

Jon Olav Vik wrote:
> Jon Olav Vik <jonovik at ...> writes:
>> == cynvector.pxd ==
>> from sundials cimport * # sundials.pxd defines N_Vector
>> cdef class CythonNVector:
>>     cdef N_Vector nvector
>>     cpdef toarray(self)
>> == cynvector.pyx ==
> [...]
>> cdef class CythonNVector:
>>     cdef N_Vector nvector # <-- ERROR
> [...]
> 
> Gotcha: When I copied declarations from cynvector.pyx to cynvector.pxd, I 
> should have removed the original "cdef N_Vector nvector" from the .pyx file. As 
> it was, I got a "'nvector' redeclared" error when re-compiling my example. The 
> reason I didn't see it at first is that I compiled cynvector.so just from 
> cynvector.pyx first, before creating cynvector.pxd.
> 
> To summarize: "Formal" declarations of object attributes in filename.pxd are 
> automatically available in filename.pyx and should not be repeated there. 
> (Functions and class definitions _are_ repeated, for implementation.)

This confused me a lot too when I met it the first time. Could we change 
this? That is, I think we should start silently accept/ignore repeating 
the declaration of cdef class fields (and module global variables?) in 
the pyx -- this should be backwards compatible.

-- 
Dag Sverre

From robertwb at math.washington.edu  Wed Jun 24 18:10:23 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 24 Jun 2009 09:10:23 -0700
Subject: [Cython] How to use my Cython extension class outside of its
	own module?
In-Reply-To: <4A40C033.1010302@student.matnat.uio.no>
References: <loom.20090623T072004-228@post.gmane.org>	<A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>	<4A409818.2090801@student.matnat.uio.no>	<loom.20090623T091132-146@post.gmane.org>
	<loom.20090623T111927-627@post.gmane.org>
	<4A40C033.1010302@student.matnat.uio.no>
Message-ID: <097294B3-6A63-4E7E-B8A2-C7869AADEE7C@math.washington.edu>

On Jun 23, 2009, at 4:44 AM, Dag Sverre Seljebotn wrote:

> Jon Olav Vik wrote:
>> Jon Olav Vik <jonovik at ...> writes:
>>> == cynvector.pxd ==
>>> from sundials cimport * # sundials.pxd defines N_Vector
>>> cdef class CythonNVector:
>>>     cdef N_Vector nvector
>>>     cpdef toarray(self)
>>> == cynvector.pyx ==
>> [...]
>>> cdef class CythonNVector:
>>>     cdef N_Vector nvector # <-- ERROR
>> [...]
>>
>> Gotcha: When I copied declarations from cynvector.pyx to  
>> cynvector.pxd, I
>> should have removed the original "cdef N_Vector nvector" from  
>> the .pyx file. As
>> it was, I got a "'nvector' redeclared" error when re-compiling my  
>> example. The
>> reason I didn't see it at first is that I compiled cynvector.so  
>> just from
>> cynvector.pyx first, before creating cynvector.pxd.
>>
>> To summarize: "Formal" declarations of object attributes in  
>> filename.pxd are
>> automatically available in filename.pyx and should not be repeated  
>> there.
>> (Functions and class definitions _are_ repeated, for implementation.)
>
> This confused me a lot too when I met it the first time. Could we  
> change
> this? That is, I think we should start silently accept/ignore  
> repeating
> the declaration of cdef class fields (and module global variables?) in
> the pyx -- this should be backwards compatible.

I'm not opposed to such a change. Errors, of course, on non-matching  
and extra fields.

- Robert


From dalcinl at gmail.com  Thu Jun 25 01:15:08 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 24 Jun 2009 20:15:08 -0300
Subject: [Cython] How to use my Cython extension class outside of its
	own module?
In-Reply-To: <097294B3-6A63-4E7E-B8A2-C7869AADEE7C@math.washington.edu>
References: <loom.20090623T072004-228@post.gmane.org>
	<A9B11B0B-F9C9-4303-8BDA-1FA2BA75F47D@math.washington.edu>
	<4A409818.2090801@student.matnat.uio.no>
	<loom.20090623T091132-146@post.gmane.org>
	<loom.20090623T111927-627@post.gmane.org>
	<4A40C033.1010302@student.matnat.uio.no>
	<097294B3-6A63-4E7E-B8A2-C7869AADEE7C@math.washington.edu>
Message-ID: <e7ba66e40906241615r4b48c6c2y302912e05f2b3144@mail.gmail.com>

On Wed, Jun 24, 2009 at 1:10 PM, Robert
Bradshaw<robertwb at math.washington.edu> wrote:
> On Jun 23, 2009, at 4:44 AM, Dag Sverre Seljebotn wrote:
>
>> Jon Olav Vik wrote:
>>> Jon Olav Vik <jonovik at ...> writes:
>>>> == cynvector.pxd ==
>>>> from sundials cimport * # sundials.pxd defines N_Vector
>>>> cdef class CythonNVector:
>>>> ? ? cdef N_Vector nvector
>>>> ? ? cpdef toarray(self)
>>>> == cynvector.pyx ==
>>> [...]
>>>> cdef class CythonNVector:
>>>> ? ? cdef N_Vector nvector # <-- ERROR
>>> [...]
>>>
>>> Gotcha: When I copied declarations from cynvector.pyx to
>>> cynvector.pxd, I
>>> should have removed the original "cdef N_Vector nvector" from
>>> the .pyx file. As
>>> it was, I got a "'nvector' redeclared" error when re-compiling my
>>> example. The
>>> reason I didn't see it at first is that I compiled cynvector.so
>>> just from
>>> cynvector.pyx first, before creating cynvector.pxd.
>>>
>>> To summarize: "Formal" declarations of object attributes in
>>> filename.pxd are
>>> automatically available in filename.pyx and should not be repeated
>>> there.
>>> (Functions and class definitions _are_ repeated, for implementation.)
>>
>> This confused me a lot too when I met it the first time. Could we
>> change
>> this? That is, I think we should start silently accept/ignore
>> repeating
>> the declaration of cdef class fields (and module global variables?) in
>> the pyx -- this should be backwards compatible.
>
> I'm not opposed to such a change. Errors, of course, on non-matching
> and extra fields.
>

I?m not opposed to. However, I think that the pyx file should declare
nothing, or it should repeat ALL the declaration on the pxd, with
MATCHING TYPES and even in the SAME ORDER... If not, you just open the
door to more confussion and possible bugs (when the pxd is
cimported)...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From evlutte at gmail.com  Fri Jun 26 19:59:02 2009
From: evlutte at gmail.com (evlutte at gmail.com)
Date: Fri, 26 Jun 2009 17:59:02 +0000
Subject: [Cython] Cython and numpy question(s)
Message-ID: <000325575b1e754820046d44189d@google.com>

I'm working on a module that will hopefully eventually go to the SAGE  
project. Here's a snippet of code:


import numpy as np
cimport numpy as np
import math
#cimport math
import cmath
#cimport cmath
from math import pi
from cmath import exp

cdef double PI = pi
cdef double TWOPI = 2*PI
cdef I = complex(0,1)

class Riemann_Map():

def __init__(self,f,fprime,a,N, init_full = False):
self.a = a
self.N = N
self.tk = np.array(np.arange(N)*TWOPI/N, dtype = np.float64)
cdef np.ndarray[np.complex128, ndim = 1] cp = np.zeros(N,dtype =  
np.complex128)

As far as I can tell from the cython/numpy tutorial, that last line should  
be legit, but I get this error:
"complex128 is not a type identifier"

Because I'm just learning cython, I'll probably be coming back with more  
questions.

Thanks for any help you can provide,

Ethan Van Andel
Because I'm ver
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090626/fbf448a7/attachment-0001.htm 

From dagss at student.matnat.uio.no  Fri Jun 26 20:11:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 26 Jun 2009 20:11:28 +0200
Subject: [Cython] Cython and numpy question(s)
In-Reply-To: <000325575b1e754820046d44189d@google.com>
References: <000325575b1e754820046d44189d@google.com>
Message-ID: <4A450F50.80801@student.matnat.uio.no>

evlutte at gmail.com wrote:
> I'm working on a module that will hopefully eventually go to the SAGE 
> project. Here's a snippet of code:
> 
> 
> 
> As far as I can tell from the cython/numpy tutorial, that last line 
> should be legit, but I get this error:
> "complex128 is not a type identifier"

This is a problem with Sage, which ships a different numpy.pxd from 
Cython, so that cimport numpy does something else than in the tutorial. 
There's a thread on sage-dev from some weeks ago, started by Glenn 
Tarbox, where the appropriate Sage ticket is noted down.

-- 
Dag Sverre

From dsurviver at gmail.com  Fri Jun 26 22:31:24 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Fri, 26 Jun 2009 17:31:24 -0300
Subject: [Cython]  Vote for constructor method of C++ classes
Message-ID: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>

Hi, all.
For my GSoC work with C++ Support, I need to know if we'll use the
Python '__init__' special method or use C++ way, with the constructor
with the same name of the class:

class Foo
{
public:
    Foo() { } //constructor
}

For this, we need your vote.

Know that the choose of the syntax will not change our work to
implement. Just choose wich will be better for you :)

So, shich do you prefer? (I'm neutral on it)

-- 
- Danilo Freitas

From dsurviver at gmail.com  Fri Jun 26 22:32:00 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Fri, 26 Jun 2009 17:32:00 -0300
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
Message-ID: <45239150906261332s4c478f5agf049bf846c9ed652@mail.gmail.com>

> So, shich do you prefer? (I'm neutral on it)

*which :)


-- 
- Danilo Freitas

From dagss at student.matnat.uio.no  Fri Jun 26 23:14:01 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 26 Jun 2009 23:14:01 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
Message-ID: <4A453A19.5040305@student.matnat.uio.no>

Danilo Freitas wrote:
> Hi, all.
> For my GSoC work with C++ Support, I need to know if we'll use the
> Python '__init__' special method or use C++ way, with the constructor
> with the same name of the class:
> 
> class Foo
> {
> public:
>     Foo() { } //constructor
> }
> 
> For this, we need your vote.
> 
> Know that the choose of the syntax will not change our work to
> implement. Just choose wich will be better for you :)
> 
> So, shich do you prefer? (I'm neutral on it)

Well, what is the syntax going to be for other operators? I.e. I would 
think you could have either

cdef extern cppclass Foo: # or whatever
     Foo(int a, int b)
     Foo operator+(Foo other)

or

cdef extern cppclass Foo:
     def __init__(self, int a, int b)
     def Foo __add__(self, Foo other)

or

cdef extern cppclass Foo:
     cdef __init__(self, int a, int b)
     cdef Foo __add__(self, Foo other)

but I wouldn't feel happy about a mixture of those.

-- 
Dag Sverre

From dsurviver at gmail.com  Sat Jun 27 00:12:18 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Fri, 26 Jun 2009 19:12:18 -0300
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A453A19.5040305@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no>
Message-ID: <45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>

> Well, what is the syntax going to be for other operators? I.e. I would
> think you could have either

For now, we still didn't think on it. It may be another question for
vote. We could expand this thread for voting also on the syntax for
the other operators (like the constructor, the work should be the
same).

> cdef extern cppclass Foo: # or whatever
>     Foo(int a, int b)
>     Foo operator+(Foo other)
>
> or
>
> cdef extern cppclass Foo:
>     def __init__(self, int a, int b)
>     def Foo __add__(self, Foo other)
>
> or
>
> cdef extern cppclass Foo:
>     cdef __init__(self, int a, int b)
>     cdef Foo __add__(self, Foo other)
>
> but I wouldn't feel happy about a mixture of those.

Yep. Allowing different syntax for constructor and operators would be
weird. Defining a 'default' is better.

To let people know better here, the syntax for defining C++ classes is
(using an example file for tests):

cdef extern from "foo.h" namespace something: #namespace is optional

    cdef cppclass Foo:
        int do_something(int)

    cdef cppclass Foo2(Foo): #inheritance allowed
        pass

    cdef Foo *make_Foo():
        pass

    cdef Foo2 *make_Foo2()

cdef Foo *a = make_Foo()
cdef Foo2 *b = make_Foo2()


We can declare methods inside the class without the cdef, so, taking
this to constructors and operators should be allowed too.
Also, objects declared inside a C++ class are taken as C types (Python
objects not allowed)
So, a C++ class won't have any Python object as member. We'll just get
the already existing syntax.
Robert may explain better this part of the project.

-- 
- Danilo Freitas

From stefan_ml at behnel.de  Sat Jun 27 07:42:22 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Jun 2009 07:42:22 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>
	<45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>
Message-ID: <4A45B13E.7030302@behnel.de>

Hi,

Danilo Freitas wrote:
> To let people know better here, the syntax for defining C++ classes is
> (using an example file for tests):
> 
> cdef extern from "foo.h" namespace something: #namespace is optional

I would prefer seeing from that header line that this block deals with C++.
Enforcing that wouldn't hurt users, but improve readability, and make it
easier to expand the block content to potential future syntax extensions
that aren't allowed for C definitions.


>     cdef cppclass Foo:
>         int do_something(int)

Fine with me, but what about overloaded signatures? That also impacts the
constructor.


>     cdef cppclass Foo2(Foo): #inheritance allowed
>         pass

Sure.


>     cdef Foo *make_Foo():
>         pass
> 
>     cdef Foo2 *make_Foo2()

Would that be a regular factory function or a way to write the constructor?


> We can declare methods inside the class without the cdef, so, taking
> this to constructors and operators should be allowed too.

ok


> Also, objects declared inside a C++ class are taken as C types (Python
> objects not allowed)

Why not? They'd map to a PyObject*, which Cython code would ref-count its
access to. If you change it from C++ code, you'd just be on your own, as
usual. Same for the C++ destructor, which Cython won't generate for you.
Call it an advanced feature.

Stefan


From stefan_ml at behnel.de  Sat Jun 27 07:48:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Jun 2009 07:48:27 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A453A19.5040305@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no>
Message-ID: <4A45B2AB.9060609@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Danilo Freitas wrote:
>> For my GSoC work with C++ Support, I need to know if we'll use the
>> Python '__init__' special method or use C++ way, with the constructor
>> with the same name of the class:
>>
>> class Foo
>> {
>> public:
>>     Foo() { } //constructor
>> }
> 
> Well, what is the syntax going to be for other operators? I.e. I would 
> think you could have either
> 
> cdef extern cppclass Foo: # or whatever
>      Foo(int a, int b)
>      Foo operator+(Foo other)
> 
> or
> 
> cdef extern cppclass Foo:
>      def __init__(self, int a, int b)
>      def Foo __add__(self, Foo other)
> 
> or
> 
> cdef extern cppclass Foo:
>      cdef __init__(self, int a, int b)
>      cdef Foo __add__(self, Foo other)
> 
> but I wouldn't feel happy about a mixture of those.

+1 for XOR. Given that the semantics are not necessarily what one would
expect from a Python __add__, and given that we need a way to support
overloaded signatures anyway, I'm +0.5 for a syntax that differs from
Python, such as the above "operator+". Developers who use this feature must
be familiar with C++ syntax anyway, so keeping the declarations close to
C++ is a plus (pun intended).

Stefan

From dagss at student.matnat.uio.no  Sat Jun 27 09:02:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Jun 2009 09:02:57 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A45B2AB.9060609@behnel.de>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>
	<4A45B2AB.9060609@behnel.de>
Message-ID: <4A45C421.4090005@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> Danilo Freitas wrote:
>>> For my GSoC work with C++ Support, I need to know if we'll use the
>>> Python '__init__' special method or use C++ way, with the constructor
>>> with the same name of the class:
>>>
>>> class Foo
>>> {
>>> public:
>>>     Foo() { } //constructor
>>> }
>> Well, what is the syntax going to be for other operators? I.e. I would 
>> think you could have either
>>
>> cdef extern cppclass Foo: # or whatever
>>      Foo(int a, int b)
>>      Foo operator+(Foo other)
>>
>> or
>>
>> cdef extern cppclass Foo:
>>      def __init__(self, int a, int b)
>>      def Foo __add__(self, Foo other)
>>
>> or
>>
>> cdef extern cppclass Foo:
>>      cdef __init__(self, int a, int b)
>>      cdef Foo __add__(self, Foo other)
>>
>> but I wouldn't feel happy about a mixture of those.
> 
> +1 for XOR. Given that the semantics are not necessarily what one would
> expect from a Python __add__, and given that we need a way to support
> overloaded signatures anyway, I'm +0.5 for a syntax that differs from
> Python, such as the above "operator+". Developers who use this feature must
> be familiar with C++ syntax anyway, so keeping the declarations close to
> C++ is a plus (pun intended).

I'm not so sure. This has already been discussed, but I'll recap.

C++'s operator[] vary much from Python. There's no seperate setitem 
operator, instead you use the "getitem" operator and the general C++ 
support for assigning to the result of a function call (i.e. in C++ you 
can do a.foo(b) = c, so getitem is enough to support a[b] = c).

Also there's the host of operators which can't really be invoked from 
Cython (at least easily/without magic methods to call them explicitly).

Now, there are two approaches:

A) Support the full C++ semantics. In order to get a working setitem, 
you also need to support at least operator= (non-trivial, but I assume 
it would actually be OK here...) and C++ references.

This leads to C++ constructor syntax.

B) Define operators in terms of "what C++ code is output". So, we have 
__setitem__ on C++ classes, and simply define it as "declaring the 
signature of the entire C++ stub a[b] = c".

This approach leads to __init__, IMO.

Note that B pushes the C++/Cython semantics gap to the point "before" 
the cppclass decl is written, i.e. the user has to do the mapping in his 
or her head. Approach A allows C++ syntax, but requires Cython to decide 
how to deal with C++ semantics in each case.

I was a heavy supporter of B earlier but I'm a little bit less sure now.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Jun 27 09:07:21 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Jun 2009 09:07:21 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>
	<45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>
Message-ID: <4A45C529.60803@student.matnat.uio.no>

Danilo Freitas wrote:
>> Well, what is the syntax going to be for other operators? I.e. I would
>> think you could have either
> 
> For now, we still didn't think on it. It may be another question for
> vote. We could expand this thread for voting also on the syntax for
> the other operators (like the constructor, the work should be the
> same).

It's a good idea to treat issues seperately. Unfortunately I'm uncertain 
about whether it is possible in this case -- in seems to be too strongly 
connected with operators in my opinion though. OTOH, if operators are 
treated first, the conclusion for constructors will come automatically IMO.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Jun 27 09:19:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Jun 2009 09:19:54 +0200
Subject: [Cython] On memory view syntax
Message-ID: <4A45C81A.8050304@student.matnat.uio.no>

It seems that me and Kurt are full speed ahead implementing memory view, 
after all.

Regarding syntax (not only the big issue of int[:], but also lots of 
smaller details in it), I think I'm going to take this stance:

In the first Cython release this ends up in (0.12 perhaps) this feature 
is marked as "experimental". Even using it will output a warning:

Warning: The memory view type is experimental, and its syntax may change 
in future Cython versions. Do not write too much code with it.

(A compiler directive can turn off the warning for Kurt's use.)

This seems to allow us getting the necessary experience developing and 
using it before fixing anything.

For those who are still following, I'd love some input on whether 
creative use of & seems acceptable. Basically on each axis we need to 
specify two closely connected, but orthogonal attributes:

Access mode: direct, ptr, full
Packing mode: contig, strided, follow

These will have lots of defaults and so on so you don't normally use 
them, but I'll deal with the situation where you do (for power users, 
which can tuck it away in a ctypedef). My visually preffered alternative is

cdef int[::contig & ptr, ::strided & direct]

although it would be possible to do e.g.

cdef int[::(contig, ptr), ::(strided, direct)]

instead.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Jun 27 09:33:45 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Jun 2009 09:33:45 +0200
Subject: [Cython] On memory view syntax
In-Reply-To: <4A45C81A.8050304@student.matnat.uio.no>
References: <4A45C81A.8050304@student.matnat.uio.no>
Message-ID: <4A45CB59.1020305@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> It seems that me and Kurt are full speed ahead implementing memory view, 
> after all.
> 
> Regarding syntax (not only the big issue of int[:], but also lots of 
> smaller details in it), I think I'm going to take this stance:
> 
> In the first Cython release this ends up in (0.12 perhaps) this feature 
> is marked as "experimental". Even using it will output a warning:
> 
> Warning: The memory view type is experimental, and its syntax may change 
> in future Cython versions. Do not write too much code with it.
> 
> (A compiler directive can turn off the warning for Kurt's use.)
> 
> This seems to allow us getting the necessary experience developing and 
> using it before fixing anything.
> 
> For those who are still following, I'd love some input on whether 
> creative use of & seems acceptable. Basically on each axis we need to 
> specify two closely connected, but orthogonal attributes:
> 
> Access mode: direct, ptr, full
> Packing mode: contig, strided, follow
> 
> These will have lots of defaults and so on so you don't normally use 
> them, but I'll deal with the situation where you do (for power users, 
> which can tuck it away in a ctypedef). My visually preffered alternative is
> 
> cdef int[::contig & ptr, ::strided & direct]
> 
> although it would be possible to do e.g.
> 
> cdef int[::(contig, ptr), ::(strided, direct)]
> 
> instead.
> 

Hmm. I just realize that if this was C, you'd have those as flags in 
different bits and use |, so add to that list:

cdef int[::contig | ptr, ::strided | direct]

But I like & better, visually, and this isn't a low-level C feature. *shrug*

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Sat Jun 27 10:02:45 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Jun 2009 10:02:45 +0200
Subject: [Cython] On CythonUtilityCode
Message-ID: <4A45D225.9090304@student.matnat.uio.no>

Just a quick note that in the gsoc-kurt branch I've implemented this:

from UtilityCode import CythonUtilityCode

...
    env.use_utility_code(foo) # only on env. currently, not code.


foo = CythonUtilityCode(u"""

class __Pyx_UtilClass:
     pass

cdef int __Pyx_UtilityFunction(int arg):
     return arg * 2

""")

Now, this will cause __Pyx_UtilityFunction to be dumped into the final C 
file, and the type descriptor to be called __Pyx_UtilClass (i.e. name 
mangling is dropped in some important cases).

The details might change around a bit, this is just a note to say I've 
done it, so that noone else starts on something similar.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sat Jun 27 16:57:29 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Jun 2009 16:57:29 +0200
Subject: [Cython] On CythonUtilityCode
In-Reply-To: <4A45D225.9090304@student.matnat.uio.no>
References: <4A45D225.9090304@student.matnat.uio.no>
Message-ID: <4A463359.6060101@behnel.de>

Hi,

thanks for doing that.

Dag Sverre Seljebotn wrote:
> Just a quick note that in the gsoc-kurt branch I've implemented this:
> 
> from UtilityCode import CythonUtilityCode
> 
> ...
>     env.use_utility_code(foo) # only on env. currently, not code.

That's too bad, because 'code' is where it belongs.

I assume the environment is required to define the utility code? In that
case, what about splitting it up, so that 'env' can be used to define
(parse/analyse/compile) the utility code snippet, but 'code' provides the
functionality to actually use it? (so that it won't turn up in the code if
the code generation decides not to need it)


> foo = CythonUtilityCode(u"""
> 
> class __Pyx_UtilClass:
>      pass
> 
> cdef int __Pyx_UtilityFunction(int arg):
>      return arg * 2
> 
> """)
> 
> Now, this will cause __Pyx_UtilityFunction to be dumped into the final C 
> file, and the type descriptor to be called __Pyx_UtilClass (i.e. name 
> mangling is dropped in some important cases).

Sounds hairy, but I think that makes sense. You have to know the name to
call the function after all. We could provide the utility code with a way
to mangle names, though. The code generator could then call into it,
instead of knowing the mangled name, so that

	my_utility_code.mangle("UtilityFunction")

would return the above "__Pyx_UtilityFunction", or whatever the compiler
decided to use as name.

Stefan


From dagss at student.matnat.uio.no  Sat Jun 27 17:28:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Jun 2009 17:28:10 +0200
Subject: [Cython] On CythonUtilityCode
In-Reply-To: <4A463359.6060101@behnel.de>
References: <4A45D225.9090304@student.matnat.uio.no>
	<4A463359.6060101@behnel.de>
Message-ID: <4A463A8A.1020004@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> thanks for doing that.
> 
> Dag Sverre Seljebotn wrote:
>> Just a quick note that in the gsoc-kurt branch I've implemented this:
>>
>> from UtilityCode import CythonUtilityCode
>>
>> ...
>>     env.use_utility_code(foo) # only on env. currently, not code.
> 
> That's too bad, because 'code' is where it belongs.
> 
> I assume the environment is required to define the utility code? In that
> case, what about splitting it up, so that 'env' can be used to define
> (parse/analyse/compile) the utility code snippet, but 'code' provides the
> functionality to actually use it? (so that it won't turn up in the code if
> the code generation decides not to need it)

I definitely see this as "one step along the road"; in the end I really 
want to be able to use this on code. Not now though.

(It's really just about how ModuleNode is currently used to generate 
code -- I just copy the nodes in there before the code generation 
starts, similar to how inline code in pxd files are handled. This is a 
hack, I definitely see ModuleNode refactored at some point.)

Perfect is the enemy of good.

>> foo = CythonUtilityCode(u"""
>>
>> class __Pyx_UtilClass:
>>      pass
>>.
>> cdef int __Pyx_UtilityFunction(int arg):
>>      return arg * 2
>>
>> """)
>>
>> Now, this will cause __Pyx_UtilityFunction to be dumped into the final C 
>> file, and the type descriptor to be called __Pyx_UtilClass (i.e. name 
>> mangling is dropped in some important cases).
> 
> Sounds hairy, but I think that makes sense. You have to know the name to

What prompted me to it is that in the current C utility code, it works 
the same way. I would really like name mangling and what kind of method 
was used to generate utility code to be orthogonal concepts.

I added a "prefix" keyword argument BTW, above one could pass 
prefix="__Pyx_" and then define "cdef UtilityFunction" to get the same 
result.

> call the function after all. We could provide the utility code with a way
> to mangle names, though. The code generator could then call into it,
> instead of knowing the mangled name, so that
> 
> 	my_utility_code.mangle("UtilityFunction")
> 
> would return the above "__Pyx_UtilityFunction", or whatever the compiler
> decided to use as name.

Hmm. I'll think about it and see.

-- 
Dag Sverre

From robertwb at math.washington.edu  Sat Jun 27 20:01:03 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 27 Jun 2009 11:01:03 -0700
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A45B13E.7030302@behnel.de>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>
	<45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>
	<4A45B13E.7030302@behnel.de>
Message-ID: <6ADF23BE-4021-4143-891F-57DF9CFB95DE@math.washington.edu>

On Jun 26, 2009, at 10:42 PM, Stefan Behnel wrote:

> Hi,
>
> Danilo Freitas wrote:
>> To let people know better here, the syntax for defining C++  
>> classes is
>> (using an example file for tests):
>>
>> cdef extern from "foo.h" namespace something: #namespace is optional
>
> I would prefer seeing from that header line that this block deals  
> with C++.
> Enforcing that wouldn't hurt users, but improve readability, and  
> make it
> easier to expand the block content to potential future syntax  
> extensions
> that aren't allowed for C definitions.

I'm not sure what the gain is here. Either the whole file is C++, or  
it's not. (For instance, the new operator would be invalid anywhere  
in the file if it's not C++). You don't have some blocks that are and  
some blocks that are not.

>
>
>>     cdef cppclass Foo:
>>         int do_something(int)
>
> Fine with me, but what about overloaded signatures? That also  
> impacts the
> constructor.

We will support them just as we support overloaded methods.

>>     cdef Foo *make_Foo():
>>         pass
>>
>>     cdef Foo2 *make_Foo2()
>
> Would that be a regular factory function or a way to write the  
> constructor?

Those are just functions for testing C++ class manipulation before  
constructor syntax had been settled on. (Of course, they're perfectly  
valid as factory functions as written.)

>> Also, objects declared inside a C++ class are taken as C types  
>> (Python
>> objects not allowed)
>
> Why not? They'd map to a PyObject*, which Cython code would ref- 
> count its
> access to. If you change it from C++ code, you'd just be on your  
> own, as
> usual. Same for the C++ destructor, which Cython won't generate for  
> you.
> Call it an advanced feature.


We are not defining C++ classes here, just declaring externally- 
defined classes. It is unclear what the semantics of an object member  
would be. Would it there be a guarantee that it is set to a non-NULL  
increfed value on construction? Does the dectructor have a contract  
to decref it? It's probably better to require a specific cast here to  
signal from this point on refcounting has to be done manually (either  
by the user, or by the C++ library author).

I'm not opposed to methods that take/return objects (e.g. when a new  
reference is returned, it should be an object), as there's no  
"lifetime" to worry about.

- Robert


From robertwb at math.washington.edu  Sat Jun 27 20:36:11 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 27 Jun 2009 11:36:11 -0700
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A45C421.4090005@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>
	<4A45B2AB.9060609@behnel.de>
	<4A45C421.4090005@student.matnat.uio.no>
Message-ID: <65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>

On Jun 27, 2009, at 12:02 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Hi,
>>
>> Dag Sverre Seljebotn wrote:
>>> Danilo Freitas wrote:
>>>> For my GSoC work with C++ Support, I need to know if we'll use the
>>>> Python '__init__' special method or use C++ way, with the  
>>>> constructor
>>>> with the same name of the class:
>>>>
>>>> class Foo
>>>> {
>>>> public:
>>>>     Foo() { } //constructor
>>>> }
>>> Well, what is the syntax going to be for other operators? I.e. I  
>>> would
>>> think you could have either
>>>
>>> cdef extern cppclass Foo: # or whatever
>>>      Foo(int a, int b)
>>>      Foo operator+(Foo other)
>>>
>>> or
>>>
>>> cdef extern cppclass Foo:
>>>      def __init__(self, int a, int b)
>>>      def Foo __add__(self, Foo other)
>>>
>>> or
>>>
>>> cdef extern cppclass Foo:
>>>      cdef __init__(self, int a, int b)
>>>      cdef Foo __add__(self, Foo other)
>>>
>>> but I wouldn't feel happy about a mixture of those.

You are right that operators and constructors are intimately  
connected syntaxwise, and I agree we should do the same for both.  
Constructors are more basic (you can't get far without those) but  
operators are more nuanced, so they should be decided together.

>> +1 for XOR. Given that the semantics are not necessarily what one  
>> would
>> expect from a Python __add__, and given that we need a way to support
>> overloaded signatures anyway, I'm +0.5 for a syntax that differs from
>> Python, such as the above "operator+". Developers who use this  
>> feature must
>> be familiar with C++ syntax anyway, so keeping the declarations  
>> close to
>> C++ is a plus (pun intended).
>
> I'm not so sure. This has already been discussed, but I'll recap.
>
> C++'s operator[] vary much from Python. There's no seperate setitem
> operator, instead you use the "getitem" operator and the general C++
> support for assigning to the result of a function call (i.e. in C++  
> you
> can do a.foo(b) = c, so getitem is enough to support a[b] = c).
>
> Also there's the host of operators which can't really be invoked from
> Cython (at least easily/without magic methods to call them  
> explicitly).
>
> Now, there are two approaches:
>
> A) Support the full C++ semantics. In order to get a working setitem,
> you also need to support at least operator= (non-trivial, but I assume
> it would actually be OK here...) and C++ references.
>
> This leads to C++ constructor syntax.
>
> B) Define operators in terms of "what C++ code is output". So, we have
> __setitem__ on C++ classes, and simply define it as "declaring the
> signature of the entire C++ stub a[b] = c".
>
> This approach leads to __init__, IMO.
>
> Note that B pushes the C++/Cython semantics gap to the point "before"
> the cppclass decl is written, i.e. the user has to do the mapping  
> in his
> or her head. Approach A allows C++ syntax, but requires Cython to  
> decide
> how to deal with C++ semantics in each case.
>
> I was a heavy supporter of B earlier but I'm a little bit less sure  
> now.

I'm also leaning towards B. For one thing, it only takes one person  
to write the pxd, and then many people go and read/use it. (Think of  
the utility of numpy.pxd.) Lots of these people are probably more  
fluent in Python than C++ (a natural bias--if it were the opposite,  
they'd be more likely to write against the C++ library directly). It  
also frees us from the constraint of having to understand and emulate  
the C++ operators exactly. It avoids the complexities of dealing with  
references and operator= before having anything up and useful. Also,  
if we're ever going to support operator overloading on anything but C+ 
+ classes (e.g. user-defined structs, or fast pathways for cdef class  
arithmetic--not saying we definitely should do that now but we might  
one day (CEP 503)) it's nice to have a non-C++ notation. There are  
advantages for C++ style operator declaration as well though.

This is one of the milestones for Danilo's GSoC midterm evaluation  
(July 13) so preferably we can settle this sooner rather than later.

- Robert


From dagss at student.matnat.uio.no  Sun Jun 28 00:02:46 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 28 Jun 2009 00:02:46 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>	<4A45B2AB.9060609@behnel.de>	<4A45C421.4090005@student.matnat.uio.no>
	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
Message-ID: <4A469706.8050509@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 27, 2009, at 12:02 AM, Dag Sverre Seljebotn wrote:
>> Now, there are two approaches:
>>
>> A) Support the full C++ semantics. In order to get a working setitem,
>> you also need to support at least operator= (non-trivial, but I assume
>> it would actually be OK here...) and C++ references.
>>
>> This leads to C++ constructor syntax.
>>
>> B) Define operators in terms of "what C++ code is output". So, we have
>> __setitem__ on C++ classes, and simply define it as "declaring the
>> signature of the entire C++ stub a[b] = c".
>>
>> This approach leads to __init__, IMO.
>>
>> Note that B pushes the C++/Cython semantics gap to the point "before"
>> the cppclass decl is written, i.e. the user has to do the mapping  
>> in his
>> or her head. Approach A allows C++ syntax, but requires Cython to  
>> decide
>> how to deal with C++ semantics in each case.
>>
>> I was a heavy supporter of B earlier but I'm a little bit less sure  
>> now.
> 
> I'm also leaning towards B. For one thing, it only takes one person  
> to write the pxd, and then many people go and read/use it. (Think of  
> the utility of numpy.pxd.) Lots of these people are probably more  
> fluent in Python than C++ (a natural bias--if it were the opposite,  
> they'd be more likely to write against the C++ library directly). It  
> also frees us from the constraint of having to understand and emulate  
> the C++ operators exactly. It avoids the complexities of dealing with  
> references and operator= before having anything up and useful. Also,  
> if we're ever going to support operator overloading on anything but C+ 
> + classes (e.g. user-defined structs, or fast pathways for cdef class  
> arithmetic--not saying we definitely should do that now but we might  
> one day (CEP 503)) it's nice to have a non-C++ notation. There are  
> advantages for C++ style operator declaration as well though.
> 
> This is one of the milestones for Danilo's GSoC midterm evaluation  
> (July 13) so preferably we can settle this sooner rather than later.

We already did a lot of discussion on this; I'm rather confident that 
you know all the issues involved by now. As far as I'm concerned you two 
(Danilo & Robert) can just make a decision.

That said, I believe I still have something to contribute here though, 
which follows. But it is going rather deep into operators and not being 
focused on constructors (can't be helped). And if you think it is taking 
the discussion off the path then you certainly don't need to compose a 
detailed answer, see above.

EXAMPLE:

Looking at operator++ (e.g. the need to do it++ on some STL iterators) 
here are the options I see:

i) Take A from above, and introduce cython.inc. So declare as operator++ 
and use as cython.inc(it).

ii) Take B from above, and introduce cython.inc. One then needs to 
invent __inc__, which is not in Python, for this purpose.

iii) What was originally my proposal, which is to somehow do

cdef cppclass MyIterator:
     def next(self):
         # somehow manage to express that you want ++
         # invoked on self. My proposal was just arbitrary inline
         # cpp code, which would thus allow mapping ANY kind of wierd
         # C++ semantics into Cython semantics, through defining custom
         # methods. And I actually consider the feature much neater than
         # all the current hacks that's going on --
         # let's make hacking easy! So:
         cython.c("++self")

Note: For i) and ii), += 1 can be made a synonym from cython.inc; that 
is quite besides the point though.

DISCUSSION:

The problem I have with B, as I think you think of it, is that it seems 
to limit what you are able to do with C++ libraries, since only a 
"Cython subset" is available.

With an ability to define new, arbitrary, Cython-specific methods with 
inline C++ code though, one ensures that one can always make a map from 
C++ to Cython semantics.

But without such an ability, I'm tempted to prefer A instead; which at 
least would make it easier to, eventually, bring the full set of C++ 
semantics into Cython (without inventing wierd things like __inc__).

It's just the idea of being able to wrap "C++ libraries if they are not 
too strange" I have some issues with.

-- 
Dag Sverre

From greg.ewing at canterbury.ac.nz  Sun Jun 28 03:07:19 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 28 Jun 2009 13:07:19 +1200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A45C421.4090005@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no> <4A45B2AB.9060609@behnel.de>
	<4A45C421.4090005@student.matnat.uio.no>
Message-ID: <4A46C247.6080008@canterbury.ac.nz>

My opinion is that C++ features should use C++ syntax,
or something close to it.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Sun Jun 28 03:26:00 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 28 Jun 2009 13:26:00 +1200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <6ADF23BE-4021-4143-891F-57DF9CFB95DE@math.washington.edu>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no>
	<45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>
	<4A45B13E.7030302@behnel.de>
	<6ADF23BE-4021-4143-891F-57DF9CFB95DE@math.washington.edu>
Message-ID: <4A46C6A8.50109@canterbury.ac.nz>

Robert Bradshaw wrote:

> We are not defining C++ classes here, just declaring externally- 
> defined classes. It is unclear what the semantics of an object member  
> would be.

As long as python-valued members are disallowed in
a C struct, they should also be disallowed in a
C++ class, for the same reasons.

-- 
Greg

From robertwb at math.washington.edu  Sun Jun 28 11:06:29 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 28 Jun 2009 02:06:29 -0700
Subject: [Cython] [sage-devel] Using Cython / Numpy with Sage...
In-Reply-To: <f049b35c0906070734h517ef290s1ce36e86cdb6a91f@mail.gmail.com>
References: <f049b35c0906070734h517ef290s1ce36e86cdb6a91f@mail.gmail.com>
Message-ID: <9FBE9A57-A19F-4EB1-B475-3A5678419C8C@math.washington.edu>

The tickets: http://trac.sagemath.org/sage_trac/ticket/4571 and  
http://trac.cython.org/cython_trac/ticket/339 . Also, http:// 
trac.sagemath.org/sage_trac/ticket/6438 to support complex numbers in  
Sage.

- Robert


On Jun 7, 2009, at 7:34 AM, Glenn Tarbox, PhD wrote:

> This has become a two part question / issue:
> The use of Cython / Numpy in the notebook
> The use of Cython / Numpy on the command line
> Cython itself seems to work fine with the "Hello World" example, so  
> the core installation is consistent with the base cython package  
> (at least with my admittedly trivial tests)
>
> Using Cython with Numpy, however, doesn't appear to work "outta the  
> box"
>
> First, I tried the following in the notebook:
>
> %cython
> import numpy as np
> cimport numpy as np
> DTYPE = np.int
> ctypedef np.int_t DTYPE_t
>
> Here's the error:
>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> Traceback (most recent call last): cimport numpy as np
> File "/mnt/hdd/sage/sage-4.0.1/local/lib/python2.5/site-packages/ 
> sage/server/support.py", line 408, in cython_import_all
> create_local_c_file=create_local_c_file)
> File "/mnt/hdd/sage/sage-4.0.1/local/lib/python2.5/site-packages/ 
> sage/server/support.py", line 385, in cython_import
> create_local_c_file=create_local_c_file)
> File "/mnt/hdd/sage/sage-4.0.1/local/lib/python2.5/site-packages/ 
> sage/misc/cython.py", line 367, in cython
> raise RuntimeError, "Error converting %s to C:\n%s\n%s"%(filename,  
> log, err)
> RuntimeError: Error converting /home/tarbox/.sage/sage_notebook/ 
> worksheets/admin/89/code/sage4.spyx to C:
>
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> include "cdefs.pxi"
> #from __future__ import division
> import numpy as np
> cimport numpy as np
> DTYPE = np.int
> ctypedef np.int_t DTYPE_t ^
> ------------------------------------------------------------
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>
> I posted a question to the cython-dev list and Dag suggested trying  
> a straight Cythonize first...
>
> Trying an equivalent Cython NumPy test file yields the following  
> error:
>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
> tarbox at tarbox-laptop:$ python setup.py build_ext --inplace
> running build_ext
> building 'testnumpy' extension
> gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict- 
> prototypes -fPIC -I/home/tarbox/projects/sage/sage-4.0.1/local/ 
> include/python2.5 -c testnumpy.c -o build/temp.linux-i686-2.5/ 
> testnumpy.o
> testnumpy.c:133:31: error: numpy/arrayobject.h: No such file or  
> directory
> testnumpy.c:390: error: expected ?=?, ?,?, ?;?, ?asm? or  
> ?__attribute__? before ?__pyx_t_5numpy_int8_t?
>
> etc.... (usual big spew after an include file isn't found and every  
> symbol isn't found)
>
> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>
>
> setup.py is pulled directly from the cython tutorial docs:
>
> tarbox at puget:$ cat setup.py
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("testnumpy", ["testnumpy.pyx"])]
> )
>
> The numpy include directories were problematic previously (an extra  
> "core" directory level recently introduced) although Trac indicates  
> that issue was fixed and I'm "pretty sure" I first saw the include  
> problem in the notebook which doesn't occur.
>
> It appears that the include fix might not be general the command  
> line is throwing a similar error to what I believe I saw in earlier  
> (<=4.0) versions of Sage.
>
> I'll put a ticket up on Trac but wanted to make sure I'm not just  
> being dense or if there are other scripts I should be using.
>
> -glenn
>
> -- 
> Glenn H. Tarbox, PhD ||  206-274-6919
> http://www.tarbox.org
>
> --~--~---------~--~----~------------~-------~--~----~
> To post to this group, send email to sage-devel at googlegroups.com
> To unsubscribe from this group, send email to sage-devel- 
> unsubscribe at googlegroups.com
> For more options, visit this group at http://groups.google.com/ 
> group/sage-devel
> URLs: http://www.sagemath.org
> -~----------~----~----~----~------~----~------~--~---
>


From robertwb at math.washington.edu  Sun Jun 28 11:11:09 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 28 Jun 2009 02:11:09 -0700
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A469706.8050509@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>	<4A45B2AB.9060609@behnel.de>	<4A45C421.4090005@student.matnat.uio.no>
	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
	<4A469706.8050509@student.matnat.uio.no>
Message-ID: <8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>

On Jun 27, 2009, at 3:02 PM, Dag Sverre Seljebotn wrote:

> It's just the idea of being able to wrap "C++ libraries if they are  
> not
> too strange" I have some issues with.

I have issues with this approach too, so no worries about selling  
short on that front :). (Wrapping strange libraries may be non- 
trivial, but hopefully no more so than using them.) Also, the final  
milestone of the project is to be able to wrap all of STL, which  
although it doesn't cover everything, does quite a bit.

- Robert


From robertwb at math.washington.edu  Sun Jun 28 11:13:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 28 Jun 2009 02:13:08 -0700
Subject: [Cython] On CythonUtilityCode
In-Reply-To: <4A45D225.9090304@student.matnat.uio.no>
References: <4A45D225.9090304@student.matnat.uio.no>
Message-ID: <944B5237-C63F-4F3B-9523-E611BEED74ED@math.washington.edu>

On Jun 27, 2009, at 1:02 AM, Dag Sverre Seljebotn wrote:

> Just a quick note that in the gsoc-kurt branch I've implemented this:
>
> from UtilityCode import CythonUtilityCode
>
> ...
>     env.use_utility_code(foo) # only on env. currently, not code.
>
>
> foo = CythonUtilityCode(u"""
>
> class __Pyx_UtilClass:
>      pass
>
> cdef int __Pyx_UtilityFunction(int arg):
>      return arg * 2
>
> """)
>
> Now, this will cause __Pyx_UtilityFunction to be dumped into the  
> final C
> file, and the type descriptor to be called __Pyx_UtilClass (i.e. name
> mangling is dropped in some important cases).
>
> The details might change around a bit, this is just a note to say I've
> done it, so that noone else starts on something similar.

I'm not opposed to this.

Eventually we may want to consider grouping such code into a central  
Cython module, or at least .h file (for multi-file compilation) as  
there is a lot of redundancy between modules with all the utility  
code produced.

- Robert



From seb.binet at gmail.com  Sun Jun 28 16:11:06 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Sun, 28 Jun 2009 16:11:06 +0200
Subject: [Cython] unwrap C/C++ bindings
Message-ID: <200906281611.06958.binet@cern.ch>

hi there,

say I have a set of C++ libraries for which python bindings have already been 
generated (with Reflex [1])
using these bindings is quite easy but then users complain that their code is 
slower than in C++ (d'oh!)

so I was playing with the idea to use python+reflex introspections 
capabilities to automatically generate a cython module out of some python 
code:

##
def do_silly_stuff():
  import ROOT
  h = ROOT.THistogram(...)
  for i in xrange(100):
   h.Fill(ROOT.TRandom())
  return h

(ROOT is the python module which gives access to these C++ libraries.)

my goal is to bypass the python wrapper and call directly into either the C++ 
function/method or its C-stub equivalent.

I see 2 ways:
 - automatically generate a .pyx/.pxd file containing the definition of the 
wrapped classes and stuff
 - tab into the cython parser to help him recognise that ROOT.THistogram is a 
C++ statement and that there is no need to go through the python C-api or any 
other mambo-jumbo...

any advice on how to achieve that ? (which way is simpler and faster to be 
done, code to start with,...)

cheers,
sebastien.

[1] http://root.cern.ch/drupal/content/reflex
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From dagss at student.matnat.uio.no  Sun Jun 28 20:28:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 28 Jun 2009 20:28:12 +0200
Subject: [Cython] unwrap C/C++ bindings
In-Reply-To: <200906281611.06958.binet@cern.ch>
References: <200906281611.06958.binet@cern.ch>
Message-ID: <4A47B63C.1030100@student.matnat.uio.no>

Sebastien Binet wrote:
> hi there,
> 
> say I have a set of C++ libraries for which python bindings have already been 
> generated (with Reflex [1])
> using these bindings is quite easy but then users complain that their code is 
> slower than in C++ (d'oh!)
> 
> so I was playing with the idea to use python+reflex introspections 
> capabilities to automatically generate a cython module out of some python 
> code:
> 
> ##
> def do_silly_stuff():
>   import ROOT
>   h = ROOT.THistogram(...)
>   for i in xrange(100):
>    h.Fill(ROOT.TRandom())
>   return h
> 
> (ROOT is the python module which gives access to these C++ libraries.)
> 
> my goal is to bypass the python wrapper and call directly into either the C++ 
> function/method or its C-stub equivalent.
> 
> I see 2 ways:
>  - automatically generate a .pyx/.pxd file containing the definition of the 
> wrapped classes and stuff
>  - tab into the cython parser to help him recognise that ROOT.THistogram is a 
> C++ statement and that there is no need to go through the python C-api or any 
> other mambo-jumbo...
> 
> any advice on how to achieve that ? (which way is simpler and faster to be 
> done, code to start with,...)

First of all, this will likely be much easier if you wait until after 
summer, when there will be more C++ support in Cython.

Using Reflex to automatically generate pyx/pxd files make sense, 
shouldn't be too much work (especially after the summer with more C++ 
support) and would be very useful. But one must then write Cython, not 
Python, code in order to get the speedup. I.e.

cimport ROOT
def do_silly_stuff():
     cdef Root.THistogram h = ROOT.THistogram(...)
     ...

and so on; that is, one must explicitly type variables to get the C++ 
speedup.

The second option is not really an option -- ROOT is a Python module, 
and there's no way of finding out "what is beneath" and call it directly 
(except for the existing mechanism of a) writing pxds and b) using types).

Anything close to the "automatic speedup" you seem to want (that is, 
without adding types manually in client code) would require adding 
automatic type inference to Cython. This is something there's a lot of 
interest in, but it is also a significant amount of work.

We're certainly very enthusiastic about any efforts in this direction 
(and willing to assist with planning and guidance); but I don't think 
there's any shortcuts, unfortunately.

-- 
Dag Sverre

From stefan_ml at behnel.de  Sun Jun 28 20:47:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 28 Jun 2009 20:47:41 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <6ADF23BE-4021-4143-891F-57DF9CFB95DE@math.washington.edu>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>	<45239150906261512u27e265faw669dba01d1aaf55@mail.gmail.com>	<4A45B13E.7030302@behnel.de>
	<6ADF23BE-4021-4143-891F-57DF9CFB95DE@math.washington.edu>
Message-ID: <4A47BACD.6080906@behnel.de>

Hi,

Robert Bradshaw wrote:
> On Jun 26, 2009, at 10:42 PM, Stefan Behnel wrote:
>> Danilo Freitas wrote:
>>> cdef extern from "foo.h" namespace something: #namespace is optional
>> I would prefer seeing from that header line that this block deals  
>> with C++.
>> Enforcing that wouldn't hurt users, but improve readability, and  
>> make it
>> easier to expand the block content to potential future syntax  
>> extensions that aren't allowed for C definitions.
> 
> I'm not sure what the gain is here. Either the whole file is C++, or  
> it's not.

For one, the parser would always know that it is starting to parse a C++
block *before* it happens to stumble into a class definition. Then, having
a clear indication at the top of a .pxd file that the following are C++
definitions makes it clear to a reader that it cannot be cimported from C code.


>>> Also, objects declared inside a C++ class are taken as C types  
>>> (Python objects not allowed)
>> Why not? They'd map to a PyObject*, which Cython code would ref- 
>> count its
>> access to. If you change it from C++ code, you'd just be on your  
>> own, as
>> usual. Same for the C++ destructor, which Cython won't generate for  
>> you. Call it an advanced feature.
> 
> It is unclear what the semantics of an object member 
> would be. Would it there be a guarantee that it is set to a non-NULL  
> increfed value on construction? Does the dectructor have a contract  
> to decref it? It's probably better to require a specific cast here to  
> signal from this point on refcounting has to be done manually (either  
> by the user, or by the C++ library author).

Good points. Let's forbid that for the time being.

Stefan


From dagss at student.matnat.uio.no  Mon Jun 29 14:45:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 29 Jun 2009 14:45:24 +0200
Subject: [Cython] On CythonUtilityCode
In-Reply-To: <944B5237-C63F-4F3B-9523-E611BEED74ED@math.washington.edu>
References: <4A45D225.9090304@student.matnat.uio.no>
	<944B5237-C63F-4F3B-9523-E611BEED74ED@math.washington.edu>
Message-ID: <4A48B764.8060404@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 27, 2009, at 1:02 AM, Dag Sverre Seljebotn wrote:
> 
>> Just a quick note that in the gsoc-kurt branch I've implemented this:
>>
>> from UtilityCode import CythonUtilityCode
>>
>> ...
>>     env.use_utility_code(foo) # only on env. currently, not code.
>>
>>
>> foo = CythonUtilityCode(u"""
>>
>> class __Pyx_UtilClass:
>>      pass
>>
>> cdef int __Pyx_UtilityFunction(int arg):
>>      return arg * 2
>>
>> """)
>>
>> Now, this will cause __Pyx_UtilityFunction to be dumped into the  
>> final C
>> file, and the type descriptor to be called __Pyx_UtilClass (i.e. name
>> mangling is dropped in some important cases).
>>
>> The details might change around a bit, this is just a note to say I've
>> done it, so that noone else starts on something similar.
> 
> I'm not opposed to this.
> 
> Eventually we may want to consider grouping such code into a central  
> Cython module, or at least .h file (for multi-file compilation) as  
> there is a lot of redundancy between modules with all the utility  
> code produced.

The same thoughts did indeed occur to me. However it is strictly 
orthogonal to this -- utility code written in C has the same problem. 
And we can just set up another pipeline to have Cython dump such an .h 
file from existing utility code declarations. (Another completely 
orthogonal thing would be to move utility code to resource files for 
easier editing.)

As for whether it should be done -- do you know if e.g. Sage builds are 
IO bound? If not then a header file itself won't help, you need to 
change the build (seperately compiled .c file, or preprocessed header 
files).

A "Cython standard library" Python module is an interesting thought, 
however I think the versioning problems are too big at the moment.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jun 29 15:29:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 29 Jun 2009 15:29:57 +0200
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>	<4A453A19.5040305@student.matnat.uio.no>	<4A45B2AB.9060609@behnel.de>	<4A45C421.4090005@student.matnat.uio.no>	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>	<4A469706.8050509@student.matnat.uio.no>
	<8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
Message-ID: <4A48C1D5.80204@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 27, 2009, at 3:02 PM, Dag Sverre Seljebotn wrote:
> 
>> It's just the idea of being able to wrap "C++ libraries if they are  
>> not
>> too strange" I have some issues with.
> 
> I have issues with this approach too, so no worries about selling  
> short on that front :). (Wrapping strange libraries may be non- 
> trivial, but hopefully no more so than using them.) Also, the final  
> milestone of the project is to be able to wrap all of STL, which  
> although it doesn't cover everything, does quite a bit.

Sure, was just pointing out that a given path might lead you to need to 
define __inc__ or __deref__ and so on in order to be able to support all 
of STL (in fact __deref__ is a much better example; as the return type 
of "*it" must be declared, and "it[0]" may not be legal).

I'd rather prefer C++ syntax over __inc__ or __deref__ (i.e. if we 
didn't manage to "get rid of" the C++ semantics in the first place). I 
haven't found an alternative to inline C++ code OR supporting only a 
subset, so if you're still against inline C++ code I'm inclined to give 
a +1 to C++ syntax, also on the constructor.

But I don't really worry, and will leave it at this.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jun 29 17:06:31 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 29 Jun 2009 17:06:31 +0200
Subject: [Cython] Users list revisited
In-Reply-To: <4A20E3CC.2040508@behnel.de>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>	<7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>	<33DA9D7A-EB99-4248-AB6F-5BECEC637AB4@math.washington.edu>
	<4A20E3CC.2040508@behnel.de>
Message-ID: <4A48D877.8010001@student.matnat.uio.no>

Stefan Behnel wrote:
> Robert Bradshaw wrote:
>> On May 27, 2009, at 1:00 PM, Chris Colbert wrote:
>>
>>> I agree, I think a Cython-users is a great idea.
>>>
>>> You all have a been a great help to me in learning Cython, but I  
>>> feel horrible having to ask such newbie questions all the time, and  
>>> fear I will one day wear out my welcome.
>> No worries, you won't be a newbie forever, and then you can pay it  
>> back by answering other people's questions :).
>>
>>> So +1 for a Cython-users list.
>> +1 from me too, for all the reasons listed below. Stefan, do you want  
>> to set up one at codespeak.net for consistency?
> 
> Asking there right now.

Got an answer yet?

If it was negative, perhaps we could ask PSF? It appears to me that 
python.org hosts a few 3rd party libraries not in core, I don't get the 
impression that core inclusion is strictly necesarry to have a mailing 
list hosted there.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Mon Jun 29 18:10:26 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 29 Jun 2009 18:10:26 +0200
Subject: [Cython] Surprise: A bool isn't a bool
In-Reply-To: <4f74ca650906290836t4c2a6f27i45b5b436abe4951c@mail.gmail.com>
References: <mailman.42053.1246289710.27370.cython-dev@codespeak.net>
	<4f74ca650906290836t4c2a6f27i45b5b436abe4951c@mail.gmail.com>
Message-ID: <4A48E772.8030603@student.matnat.uio.no>

Bjarke Roune wrote:
> There seems to be no better way to report issues in Cython than send
> it to someone on the project (I suggest a better way be made that
> doesn't require signing up for something). Here you go.

It is to avoid spam (both on trac and the mailing list). Anyway, thanks 
for reporting. See below.

> On Mon, Jun 29, 2009 at 5:35 PM, <cython-dev-owner at codespeak.net> wrote:
>> You are not allowed to post to this mailing list, and your message has
>> been automatically rejected.  If you think that your messages are
>> being rejected in error, contact the mailing list owner at
>> cython-dev-owner at codespeak.net.
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Bjarke Roune <bjarke at daimi.au.dk>
>> To: cython-dev at codespeak.net
>> Date: Mon, 29 Jun 2009 17:31:16 +0200
>> Subject: Surprise: A bool isn't a bool
>> I was wrapping a C++ function in Cython, and ran into a mysterious
>> segmentation fault, and the cause turned out to be quite a trap for
>> beginning Cython programmers wrapping C++ libraries. Let's say there
>> was a function like this:
>>
>> bool foo() {
>>    return true;
>> }
>>
>> I then declared it in Cython as
>>
>> cdef extern from "bar.h":
>>    bool foo()
>>
>> and I used the function in code such as
>>
>> if (foo()):
>>    print "baz"
>>
>> This crashed my program with a segmentation violation and no explanation.
>>
>> After a while of looking through my C++ code trying to find the
>> pointer error, I realized that a C++ bool is not a Cython bool, and
>> that since a Cython bool is an object, mismatching the two can lead to
>> a memory error. I replaced bool with bint and everything was fine
>> (though I wonder if Cython ever puts anything other than 0 and 1 into
>> a bint, since only 0 and 1 are valid values for a C++ bool).
>>
>> It is surprising and confusing to a beginning Cython user such as me
>> that a bool isn't a bool, and if there is any way to issue some sort
>> of warning about mismatching a C++ bool with a Cython bool, I think
>> that would be a good idea.

Indeed. The question is whether it is possible to find such a way. 
Perhaps disallow "bool" in cdef extern functions? Although that would 
add an unecesarry typecheck if a C library in fact did return a Python 
boolean object... (like many C functions in Python's standard library 
might).


-- 
Dag Sverre

From robertwb at math.washington.edu  Mon Jun 29 18:12:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Jun 2009 09:12:08 -0700
Subject: [Cython] On CythonUtilityCode
In-Reply-To: <4A48B764.8060404@student.matnat.uio.no>
References: <4A45D225.9090304@student.matnat.uio.no>
	<944B5237-C63F-4F3B-9523-E611BEED74ED@math.washington.edu>
	<4A48B764.8060404@student.matnat.uio.no>
Message-ID: <D0A6A516-6596-4870-820B-02B1B4BB8356@math.washington.edu>

On Jun 29, 2009, at 5:45 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Jun 27, 2009, at 1:02 AM, Dag Sverre Seljebotn wrote:
>>
>>> Just a quick note that in the gsoc-kurt branch I've implemented  
>>> this:
>>>
>>> from UtilityCode import CythonUtilityCode
>>>
>>> ...
>>>     env.use_utility_code(foo) # only on env. currently, not code.
>>>
>>>
>>> foo = CythonUtilityCode(u"""
>>>
>>> class __Pyx_UtilClass:
>>>      pass
>>>
>>> cdef int __Pyx_UtilityFunction(int arg):
>>>      return arg * 2
>>>
>>> """)
>>>
>>> Now, this will cause __Pyx_UtilityFunction to be dumped into the
>>> final C
>>> file, and the type descriptor to be called __Pyx_UtilClass (i.e.  
>>> name
>>> mangling is dropped in some important cases).
>>>
>>> The details might change around a bit, this is just a note to say  
>>> I've
>>> done it, so that noone else starts on something similar.
>>
>> I'm not opposed to this.
>>
>> Eventually we may want to consider grouping such code into a central
>> Cython module, or at least .h file (for multi-file compilation) as
>> there is a lot of redundancy between modules with all the utility
>> code produced.
>
> The same thoughts did indeed occur to me. However it is strictly
> orthogonal to this -- utility code written in C has the same problem.

Yes, completely orthogonal, but it's a related idea that I had in my  
head so I figured I'd throw it out there.

> And we can just set up another pipeline to have Cython dump such an .h
> file from existing utility code declarations. (Another completely
> orthogonal thing would be to move utility code to resource files for
> easier editing.)
>
> As for whether it should be done -- do you know if e.g. Sage builds  
> are
> IO bound?

Not sure, but that's certainly a big component.

> If not then a header file itself won't help, you need to
> change the build (seperately compiled .c file, or preprocessed header
> files).

Even if it's IO bound, writing to a single .h file rather than to all  
300 .c files should make a difference. Same with the .c compiler  
reading (the single .h file will be "hot").

> A "Cython standard library" Python module is an interesting thought,
> however I think the versioning problems are too big at the moment.


I'm certainly viewing something re-built every time alongside the .c  
files to skirt that issue (in the short term at least).

- Robert


From robertwb at math.washington.edu  Mon Jun 29 18:18:41 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Jun 2009 09:18:41 -0700
Subject: [Cython] Surprise: A bool isn't a bool
In-Reply-To: <4A48E772.8030603@student.matnat.uio.no>
References: <mailman.42053.1246289710.27370.cython-dev@codespeak.net>
	<4f74ca650906290836t4c2a6f27i45b5b436abe4951c@mail.gmail.com>
	<4A48E772.8030603@student.matnat.uio.no>
Message-ID: <9D849050-1DBC-4F82-ABC1-CF77F17D7068@math.washington.edu>

On Jun 29, 2009, at 9:10 AM, Dag Sverre Seljebotn wrote:

> Bjarke Roune wrote:
>> There seems to be no better way to report issues in Cython than send
>> it to someone on the project (I suggest a better way be made that
>> doesn't require signing up for something). Here you go.
>
> It is to avoid spam (both on trac and the mailing list). Anyway,  
> thanks
> for reporting. See below.

Sorry. Yes, we only do this because it's been by far the most  
effective way of killing spam.

>
>> On Mon, Jun 29, 2009 at 5:35 PM, <cython-dev-owner at codespeak.net>  
>> wrote:
>>> You are not allowed to post to this mailing list, and your  
>>> message has
>>> been automatically rejected.  If you think that your messages are
>>> being rejected in error, contact the mailing list owner at
>>> cython-dev-owner at codespeak.net.
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Bjarke Roune <bjarke at daimi.au.dk>
>>> To: cython-dev at codespeak.net
>>> Date: Mon, 29 Jun 2009 17:31:16 +0200
>>> Subject: Surprise: A bool isn't a bool
>>> I was wrapping a C++ function in Cython, and ran into a mysterious
>>> segmentation fault, and the cause turned out to be quite a trap for
>>> beginning Cython programmers wrapping C++ libraries. Let's say there
>>> was a function like this:
>>>
>>> bool foo() {
>>>    return true;
>>> }
>>>
>>> I then declared it in Cython as
>>>
>>> cdef extern from "bar.h":
>>>    bool foo()
>>>
>>> and I used the function in code such as
>>>
>>> if (foo()):
>>>    print "baz"
>>>
>>> This crashed my program with a segmentation violation and no  
>>> explanation.
>>>
>>> After a while of looking through my C++ code trying to find the
>>> pointer error, I realized that a C++ bool is not a Cython bool, and
>>> that since a Cython bool is an object, mismatching the two can  
>>> lead to
>>> a memory error. I replaced bool with bint and everything was fine
>>> (though I wonder if Cython ever puts anything other than 0 and 1  
>>> into
>>> a bint, since only 0 and 1 are valid values for a C++ bool).

Yes, Cython can put any int value into a bint--it's only  
distinguishing feature is conversion to/from Python objects. However,  
the C++ compiler should turn it into 0/1 whenever you assign it to/ 
use it as a C++ bool.

>>> It is surprising and confusing to a beginning Cython user such as me
>>> that a bool isn't a bool, and if there is any way to issue some sort
>>> of warning about mismatching a C++ bool with a Cython bool, I think
>>> that would be a good idea.
>
> Indeed. The question is whether it is possible to find such a way.
> Perhaps disallow "bool" in cdef extern functions? Although that would
> add an unecesarry typecheck if a C library in fact did return a Python
> boolean object... (like many C functions in Python's standard library
> might).

Perhaps at least a warning would be useful.

- Robert



From dagss at student.matnat.uio.no  Mon Jun 29 18:29:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 29 Jun 2009 18:29:10 +0200
Subject: [Cython] [Fwd: Re: Surprise: A bool isn't a bool]
Message-ID: <4A48EBD6.2080706@student.matnat.uio.no>



-------- Original Message --------
Subject: Re: Surprise: A bool isn't a bool
Date: Mon, 29 Jun 2009 18:26:58 +0200
From: Bjarke Roune <bjarke at daimi.au.dk>
To: Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
References: <mailman.42053.1246289710.27370.cython-dev at codespeak.net> 
  <4f74ca650906290836t4c2a6f27i45b5b436abe4951c at mail.gmail.com>	 
<4A48E772.8030603 at student.matnat.uio.no>

> Indeed. The question is whether it is possible to find such a way. Perhaps
> disallow "bool" in cdef extern functions? Although that would add an
> unecesarry typecheck if a C library in fact did return a Python boolean
> object... (like many C functions in Python's standard library might).
>
Perhaps let there be a switch to turn the error or warning off, and
some kind of instruction in a Cython file that turns it off for only
that file.

Cheers
Bjarke

-- 
Dag Sverre

From eslaught at ucsd.edu  Mon Jun 29 20:13:10 2009
From: eslaught at ucsd.edu (Elliott Slaughter)
Date: Mon, 29 Jun 2009 11:13:10 -0700
Subject: [Cython] Compiling with MinGW
Message-ID: <42c0ab790906291113n127feec0k52b8af3faa2be7ae@mail.gmail.com>

Hi,

I just tried to install Cython with Python 2.5.4 (from the binary installer)
on Windows. Since I don't have MSVC installed, I get this message:

C:\...\Cython-0.11.2>python setup.py install
...
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible
binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin
installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

But the -c option isn't actually recognized by setup.py:

C:\...\Cython-0.11.2>python setup.py -c mingw32 install
...
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option -c not recognized

And setup.py --help doesn't give any relevant suggestions.

I would appreciate it if someone could point me in the right direction.

Thanks.

-- 
Elliott Slaughter

"Don't worry about what anybody else is going to do. The best way to predict
the future is to invent it." - Alan Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090629/8b9ab730/attachment.htm 

From dalcinl at gmail.com  Mon Jun 29 20:24:04 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 29 Jun 2009 15:24:04 -0300
Subject: [Cython] Compiling with MinGW
In-Reply-To: <42c0ab790906291113n127feec0k52b8af3faa2be7ae@mail.gmail.com>
References: <42c0ab790906291113n127feec0k52b8af3faa2be7ae@mail.gmail.com>
Message-ID: <e7ba66e40906291124w63b13577j345f73cecb086d70@mail.gmail.com>

Build&Install using EXACTLY this:

python setup.py build -c mingw32 install



On Mon, Jun 29, 2009 at 3:13 PM, Elliott Slaughter<eslaught at ucsd.edu> wrote:
> Hi,
>
> I just tried to install Cython with Python 2.5.4 (from the binary installer)
> on Windows. Since I don't have MSVC installed, I get this message:
>
> C:\...\Cython-0.11.2>python setup.py install
> ...
> error: Python was built with Visual Studio 2003;
> extensions must be built with a compiler than can generate compatible
> binaries.
> Visual Studio 2003 was not found on this system. If you have Cygwin
> installed,
> you can try compiling with MingW32, by passing "-c mingw32" to setup.py.
>
> But the -c option isn't actually recognized by setup.py:
>
> C:\...\Cython-0.11.2>python setup.py -c mingw32 install
> ...
> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
> ?? or: setup.py --help [cmd1 cmd2 ...]
> ?? or: setup.py --help-commands
> ?? or: setup.py cmd --help
>
> error: option -c not recognized
>
> And setup.py --help doesn't give any relevant suggestions.
>
> I would appreciate it if someone could point me in the right direction.
>
> Thanks.
>
> --
> Elliott Slaughter
>
> "Don't worry about what anybody else is going to do. The best way to predict
> the future is to invent it." - Alan Kay
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From elliottslaughter at gmail.com  Mon Jun 29 20:37:14 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Mon, 29 Jun 2009 11:37:14 -0700
Subject: [Cython] Compiling with MinGW
In-Reply-To: <e7ba66e40906291124w63b13577j345f73cecb086d70@mail.gmail.com>
References: <42c0ab790906291113n127feec0k52b8af3faa2be7ae@mail.gmail.com>
	<e7ba66e40906291124w63b13577j345f73cecb086d70@mail.gmail.com>
Message-ID: <42c0ab790906291137h26b0a593o123b58896b7198d0@mail.gmail.com>

Thanks.

Now gcc seems to fail on paths with backslashes in them.

C:\bin\Cython-0.11.2>python setup.py build -c mingw32
...
building 'Cython.Plex.Scanners' extension
gcc -mno-cygwin -mdll -O -Wall -IC:\bin\Python25\include
-IC:\bin\Python25\PC -c
 C:\bin\Cython-0.11.2\Cython\Plex\Scanners.c -o
c:\bin\cython-0.11.2\cython\plex
\scanners.o
error: command 'gcc' failed: No such file or directory

Suggestions?

On Mon, Jun 29, 2009 at 11:24 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> Build&Install using EXACTLY this:
>
> python setup.py build -c mingw32 install
>
>
>
> On Mon, Jun 29, 2009 at 3:13 PM, Elliott Slaughter<eslaught at ucsd.edu>
> wrote:
> > Hi,
> >
> > I just tried to install Cython with Python 2.5.4 (from the binary
> installer)
> > on Windows. Since I don't have MSVC installed, I get this message:
> >
> > C:\...\Cython-0.11.2>python setup.py install
> > ...
> > error: Python was built with Visual Studio 2003;
> > extensions must be built with a compiler than can generate compatible
> > binaries.
> > Visual Studio 2003 was not found on this system. If you have Cygwin
> > installed,
> > you can try compiling with MingW32, by passing "-c mingw32" to setup.py.
> >
> > But the -c option isn't actually recognized by setup.py:
> >
> > C:\...\Cython-0.11.2>python setup.py -c mingw32 install
> > ...
> > usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
> >    or: setup.py --help [cmd1 cmd2 ...]
> >    or: setup.py --help-commands
> >    or: setup.py cmd --help
> >
> > error: option -c not recognized
> >
> > And setup.py --help doesn't give any relevant suggestions.
> >
> > I would appreciate it if someone could point me in the right direction.
> >
> > Thanks.
> >
> > --
> > Elliott Slaughter
> >
> > "Don't worry about what anybody else is going to do. The best way to
> predict
> > the future is to invent it." - Alan Kay
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Elliott Slaughter

"Don't worry about what anybody else is going to do. The best way to predict
the future is to invent it." - Alan Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090629/669290b6/attachment.htm 

From sccolbert at gmail.com  Mon Jun 29 20:59:42 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 29 Jun 2009 14:59:42 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	<e7ba66e40905271633w4f2cc25ej4ae8962c942a0d75@mail.gmail.com>
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	<e7ba66e40905272229ka61ccbbleb23adabe3ce160e@mail.gmail.com>
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
	<7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
	<7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>
Message-ID: <7f014ea60906291159s1577649ax5344e6fabab0906c@mail.gmail.com>

Now that i've had a few minutes to test this on another platform, I
can confirm it wasnt a bug in Cython.

The wrapper builds and executes fine under ubuntu 9.04 64bit. So the
problem I was having was either a MinGW or Windows issue. Either way,
i'm not going to worry about it.

Cheers,

Chris

On Thu, May 28, 2009 at 10:50 PM, Chris Colbert<sccolbert at gmail.com> wrote:
> Just to clarify, I've never used keyword arguments with this function.
> Chris
>
> On Thu, May 28, 2009 at 10:46 PM, Chris Colbert <sccolbert at gmail.com> wrote:
>>
>> Robert,
>> That's correct. Of course I had to remove all the checks related to
>> Keyword args in the body of the function before it would execute. So
>> essentially, cleaning the code so that it only accepts positional arguments
>> fixes the crashing problem.
>> Chris
>>
>> On Thu, May 28, 2009 at 6:40 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>>
>>> On May 28, 2009, at 9:16 AM, Chris Colbert wrote:
>>>
>>> > Alright, I "fixed" the problem by hacking the Cython generated C file.
>>> >
>>> > I removed everything in the IF clause that determined whether the
>>> > Python Arguments were keyword argument or not and left only the
>>> > PyTuple_GET_ITEM statements.
>>> >
>>> > I then changed the PyMethodDef line like so:
>>> >
>>> > old entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
>>> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS|METH_KEYWORDS, __Pyx_DOCSTR
>>> > (0)},
>>> > new entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
>>> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS, __Pyx_DOCSTR(0)},
>>> >
>>> >
>>> > Removing METH_KEYWORDS was essential. Even though the extension
>>> > compiled when it was present, the program still crashed.
>>> >
>>> > Without it, it works fine for all integers 0,1,2,3.
>>>
>>> Very strange. So to make sure I understand correctly, it was crashing
>>> with METH_KEYWORDS even if you weren't using keyword arguments?
>>>
>>> - Robert
>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>

From seb.binet at gmail.com  Mon Jun 29 21:11:10 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Mon, 29 Jun 2009 21:11:10 +0200
Subject: [Cython] unwrap C/C++ bindings
In-Reply-To: <4A47B63C.1030100@student.matnat.uio.no>
References: <200906281611.06958.binet@cern.ch>
	<4A47B63C.1030100@student.matnat.uio.no>
Message-ID: <200906292111.11008.binet@cern.ch>

Dag,

> > say I have a set of C++ libraries for which python bindings have already
> > been generated (with Reflex [1])
> > using these bindings is quite easy but then users complain that their
> > code is slower than in C++ (d'oh!)
> >
> > so I was playing with the idea to use python+reflex introspections
> > capabilities to automatically generate a cython module out of some python
> > code:
> >
> > ##
> > def do_silly_stuff():
> >   import ROOT
> >   h = ROOT.THistogram(...)
> >   for i in xrange(100):
> >    h.Fill(ROOT.TRandom())
> >   return h
> >
> > (ROOT is the python module which gives access to these C++ libraries.)
> >
> > my goal is to bypass the python wrapper and call directly into either the
> > C++ function/method or its C-stub equivalent.
> >
> > I see 2 ways:
> >  - automatically generate a .pyx/.pxd file containing the definition of
> > the wrapped classes and stuff
> >  - tab into the cython parser to help him recognise that ROOT.THistogram
> > is a C++ statement and that there is no need to go through the python
> > C-api or any other mambo-jumbo...
> >
> > any advice on how to achieve that ? (which way is simpler and faster to
> > be done, code to start with,...)
>
> First of all, this will likely be much easier if you wait until after
> summer, when there will be more C++ support in Cython.

yep, I figured that much.
there is still the ability to call in the C-stubs the Reflex library is 
generating (C++ isn't a model of portability...) so I'll go that way in 
waiting for the end of the gsoc.

> The second option is not really an option -- ROOT is a Python module,
> and there's no way of finding out "what is beneath" and call it directly
> (except for the existing mechanism of a) writing pxds and b) using types).
as you can see I have no understanding of how the cython parser works, but I 
was under the impression that when cython was compiling python code to C using 
the python C-api it knew somehow how to translate e.g.
def foo():
 t = (1,2,3)
 return t

into the appropriate PyTuple_xyz calls... and I had the wild dream I could 
hook myself in there.

> Anything close to the "automatic speedup" you seem to want (that is,
> without adding types manually in client code) would require adding
> automatic type inference to Cython. This is something there's a lot of
> interest in, but it is also a significant amount of work.
>
> We're certainly very enthusiastic about any efforts in this direction
> (and willing to assist with planning and guidance); but I don't think
> there's any shortcuts, unfortunately.
type inference and/or type annotations are indeed a holy grail in python to 
which I am sympathetic but rather clueless as to how to even start :)
somehow I think unladen-swallow will give us some tools to do it.

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From dsurviver at gmail.com  Mon Jun 29 21:28:22 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Mon, 29 Jun 2009 16:28:22 -0300
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4A48C1D5.80204@student.matnat.uio.no>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no> <4A45B2AB.9060609@behnel.de>
	<4A45C421.4090005@student.matnat.uio.no>
	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
	<4A469706.8050509@student.matnat.uio.no>
	<8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
	<4A48C1D5.80204@student.matnat.uio.no>
Message-ID: <45239150906291228k1f8c7ebdtaaa4b13b742403d3@mail.gmail.com>

Hi,

sorry for my delay.  I'm still with internet problems at home :(


I'm a little inclined for a C++ syntax. I think the operators
declaration on C++ syntax are more clearer. I always thought things
like __add__, __getitem__, __setitem__ weird.

It would also be clearer for the parser. When it see 'operator', it
already knows what is to come, and check if the operator is in an
"operator list".

I think maybe we could also define operators outside a C++ class
declaring block (like C++).

Allowing the complete C++ classes on Cython, bringing STL to it will
be a simpler step.

From dalcinl at gmail.com  Mon Jun 29 21:45:04 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 29 Jun 2009 16:45:04 -0300
Subject: [Cython] Compiling with MinGW
In-Reply-To: <42c0ab790906291137h26b0a593o123b58896b7198d0@mail.gmail.com>
References: <42c0ab790906291113n127feec0k52b8af3faa2be7ae@mail.gmail.com>
	<e7ba66e40906291124w63b13577j345f73cecb086d70@mail.gmail.com>
	<42c0ab790906291137h26b0a593o123b58896b7198d0@mail.gmail.com>
Message-ID: <e7ba66e40906291245p5eb16165vb42d5879fa856fcc@mail.gmail.com>

Mmm... No! I bet you do not have C:\MinGW\bin in your %PATH%, the "No
such file or directory" message is related to "gcc" command not being
found...

Set your %PATH% appropriately before, and next build & install:

set PATH=C:\MinGW\bin;%PATH%
python setup.py build -c mingw32 install


PS: If you still have problems, you should ask Microsoft for further
help.... (Just joking :-), keep coming!)

-

On Mon, Jun 29, 2009 at 3:37 PM, Elliott
Slaughter<elliottslaughter at gmail.com> wrote:
> Thanks.
>
> Now gcc seems to fail on paths with backslashes in them.
>
> C:\bin\Cython-0.11.2>python setup.py build -c mingw32
> ...
> building 'Cython.Plex.Scanners' extension
> gcc -mno-cygwin -mdll -O -Wall -IC:\bin\Python25\include
> -IC:\bin\Python25\PC -c
> ?C:\bin\Cython-0.11.2\Cython\Plex\Scanners.c -o
> c:\bin\cython-0.11.2\cython\plex
> \scanners.o
> error: command 'gcc' failed: No such file or directory
>
> Suggestions?
>
> On Mon, Jun 29, 2009 at 11:24 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>
>> Build&Install using EXACTLY this:
>>
>> python setup.py build -c mingw32 install
>>
>>
>>
>> On Mon, Jun 29, 2009 at 3:13 PM, Elliott Slaughter<eslaught at ucsd.edu>
>> wrote:
>> > Hi,
>> >
>> > I just tried to install Cython with Python 2.5.4 (from the binary
>> > installer)
>> > on Windows. Since I don't have MSVC installed, I get this message:
>> >
>> > C:\...\Cython-0.11.2>python setup.py install
>> > ...
>> > error: Python was built with Visual Studio 2003;
>> > extensions must be built with a compiler than can generate compatible
>> > binaries.
>> > Visual Studio 2003 was not found on this system. If you have Cygwin
>> > installed,
>> > you can try compiling with MingW32, by passing "-c mingw32" to setup.py.
>> >
>> > But the -c option isn't actually recognized by setup.py:
>> >
>> > C:\...\Cython-0.11.2>python setup.py -c mingw32 install
>> > ...
>> > usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
>> > ?? or: setup.py --help [cmd1 cmd2 ...]
>> > ?? or: setup.py --help-commands
>> > ?? or: setup.py cmd --help
>> >
>> > error: option -c not recognized
>> >
>> > And setup.py --help doesn't give any relevant suggestions.
>> >
>> > I would appreciate it if someone could point me in the right direction.
>> >
>> > Thanks.
>> >
>> > --
>> > Elliott Slaughter
>> >
>> > "Don't worry about what anybody else is going to do. The best way to
>> > predict
>> > the future is to invent it." - Alan Kay
>> >
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
>
> --
> Elliott Slaughter
>
> "Don't worry about what anybody else is going to do. The best way to predict
> the future is to invent it." - Alan Kay
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Mon Jun 29 21:51:23 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 29 Jun 2009 16:51:23 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60906291159s1577649ax5344e6fabab0906c@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<e7ba66e40905271633w4f2cc25ej4ae8962c942a0d75@mail.gmail.com>
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	<e7ba66e40905272229ka61ccbbleb23adabe3ce160e@mail.gmail.com>
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
	<7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
	<7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>
	<7f014ea60906291159s1577649ax5344e6fabab0906c@mail.gmail.com>
Message-ID: <e7ba66e40906291251w31821d74sdb014538113461ea@mail.gmail.com>

On Mon, Jun 29, 2009 at 3:59 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>
> The wrapper builds and executes fine under ubuntu 9.04 64bit. So the
> problem I was having was either a MinGW or Windows issue.
>

Really strange... The Cython testsuite and my own projects do work
just fine with MinGW (and yes, I do test for kw args)...

> Either way,
> i'm not going to worry about it.
>

Just to be sure... if you look at all the OpenCV headers, can you see
"__stdcall" somewhere?


> Cheers,
>
> Chris
>
> On Thu, May 28, 2009 at 10:50 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>> Just to clarify, I've never used keyword arguments with this function.
>> Chris
>>
>> On Thu, May 28, 2009 at 10:46 PM, Chris Colbert <sccolbert at gmail.com> wrote:
>>>
>>> Robert,
>>> That's correct. Of course I had to remove all the checks related to
>>> Keyword args in the body of the function before it would execute. So
>>> essentially, cleaning the code so that it only accepts positional arguments
>>> fixes the crashing problem.
>>> Chris
>>>
>>> On Thu, May 28, 2009 at 6:40 PM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>>>
>>>> On May 28, 2009, at 9:16 AM, Chris Colbert wrote:
>>>>
>>>> > Alright, I "fixed" the problem by hacking the Cython generated C file.
>>>> >
>>>> > I removed everything in the IF clause that determined whether the
>>>> > Python Arguments were keyword argument or not and left only the
>>>> > PyTuple_GET_ITEM statements.
>>>> >
>>>> > I then changed the PyMethodDef line like so:
>>>> >
>>>> > old entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
>>>> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS|METH_KEYWORDS, __Pyx_DOCSTR
>>>> > (0)},
>>>> > new entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
>>>> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS, __Pyx_DOCSTR(0)},
>>>> >
>>>> >
>>>> > Removing METH_KEYWORDS was essential. Even though the extension
>>>> > compiled when it was present, the program still crashed.
>>>> >
>>>> > Without it, it works fine for all integers 0,1,2,3.
>>>>
>>>> Very strange. So to make sure I understand correctly, it was crashing
>>>> with METH_KEYWORDS even if you weren't using keyword arguments?
>>>>
>>>> - Robert
>>>>
>>>> _______________________________________________
>>>> Cython-dev mailing list
>>>> Cython-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From elliottslaughter at gmail.com  Mon Jun 29 22:10:28 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Mon, 29 Jun 2009 13:10:28 -0700
Subject: [Cython] Compiling with MinGW
In-Reply-To: <e7ba66e40906291245p5eb16165vb42d5879fa856fcc@mail.gmail.com>
References: <42c0ab790906291113n127feec0k52b8af3faa2be7ae@mail.gmail.com>
	<e7ba66e40906291124w63b13577j345f73cecb086d70@mail.gmail.com>
	<42c0ab790906291137h26b0a593o123b58896b7198d0@mail.gmail.com>
	<e7ba66e40906291245p5eb16165vb42d5879fa856fcc@mail.gmail.com>
Message-ID: <42c0ab790906291310n734cf6dfw68c861da5c44dc0c@mail.gmail.com>

On Mon, Jun 29, 2009 at 12:45 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> Mmm... No! I bet you do not have C:\MinGW\bin in your %PATH%, the "No
> such file or directory" message is related to "gcc" command not being
> found...
>
> Set your %PATH% appropriately before, and next build & install:
>
> set PATH=C:\MinGW\bin;%PATH%
> python setup.py build -c mingw32 install
>

Actually, it was on PATH, but cygwin had made gcc.exe.lnk a shortcut to
gcc-3.exe, which cmd.exe couldn't find. Copying the file made everything
work properly.

Thanks for the help.

PS: If you still have problems, you should ask Microsoft for further
> help.... (Just joking :-), keep coming!)
>

I just wish cygwin would play nicer with native Windows programs....

-
>
> On Mon, Jun 29, 2009 at 3:37 PM, Elliott
> Slaughter<elliottslaughter at gmail.com> wrote:
> > Thanks.
> >
> > Now gcc seems to fail on paths with backslashes in them.
> >
> > C:\bin\Cython-0.11.2>python setup.py build -c mingw32
> > ...
> > building 'Cython.Plex.Scanners' extension
> > gcc -mno-cygwin -mdll -O -Wall -IC:\bin\Python25\include
> > -IC:\bin\Python25\PC -c
> >  C:\bin\Cython-0.11.2\Cython\Plex\Scanners.c -o
> > c:\bin\cython-0.11.2\cython\plex
> > \scanners.o
> > error: command 'gcc' failed: No such file or directory
> >
> > Suggestions?
> >
> > On Mon, Jun 29, 2009 at 11:24 AM, Lisandro Dalcin <dalcinl at gmail.com>
> wrote:
> >>
> >> Build&Install using EXACTLY this:
> >>
> >> python setup.py build -c mingw32 install
> >>
> >>
> >>
> >> On Mon, Jun 29, 2009 at 3:13 PM, Elliott Slaughter<eslaught at ucsd.edu>
> >> wrote:
> >> > Hi,
> >> >
> >> > I just tried to install Cython with Python 2.5.4 (from the binary
> >> > installer)
> >> > on Windows. Since I don't have MSVC installed, I get this message:
> >> >
> >> > C:\...\Cython-0.11.2>python setup.py install
> >> > ...
> >> > error: Python was built with Visual Studio 2003;
> >> > extensions must be built with a compiler than can generate compatible
> >> > binaries.
> >> > Visual Studio 2003 was not found on this system. If you have Cygwin
> >> > installed,
> >> > you can try compiling with MingW32, by passing "-c mingw32" to
> setup.py.
> >> >
> >> > But the -c option isn't actually recognized by setup.py:
> >> >
> >> > C:\...\Cython-0.11.2>python setup.py -c mingw32 install
> >> > ...
> >> > usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
> >> >    or: setup.py --help [cmd1 cmd2 ...]
> >> >    or: setup.py --help-commands
> >> >    or: setup.py cmd --help
> >> >
> >> > error: option -c not recognized
> >> >
> >> > And setup.py --help doesn't give any relevant suggestions.
> >> >
> >> > I would appreciate it if someone could point me in the right
> direction.
> >> >
> >> > Thanks.
> >> >
> >> > --
> >> > Elliott Slaughter
> >> >
> >> > "Don't worry about what anybody else is going to do. The best way to
> >> > predict
> >> > the future is to invent it." - Alan Kay
> >> >
> >> > _______________________________________________
> >> > Cython-dev mailing list
> >> > Cython-dev at codespeak.net
> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Lisandro Dalc?n
> >> ---------------
> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> Tel/Fax: +54-(0)342-451.1594
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> >
> > --
> > Elliott Slaughter
> >
> > "Don't worry about what anybody else is going to do. The best way to
> predict
> > the future is to invent it." - Alan Kay
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Elliott Slaughter

"Don't worry about what anybody else is going to do. The best way to predict
the future is to invent it." - Alan Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090629/1969303b/attachment.htm 

From dalcinl at gmail.com  Mon Jun 29 22:56:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 29 Jun 2009 17:56:39 -0300
Subject: [Cython] Compiling with MinGW
In-Reply-To: <42c0ab790906291310n734cf6dfw68c861da5c44dc0c@mail.gmail.com>
References: <42c0ab790906291113n127feec0k52b8af3faa2be7ae@mail.gmail.com>
	<e7ba66e40906291124w63b13577j345f73cecb086d70@mail.gmail.com>
	<42c0ab790906291137h26b0a593o123b58896b7198d0@mail.gmail.com>
	<e7ba66e40906291245p5eb16165vb42d5879fa856fcc@mail.gmail.com>
	<42c0ab790906291310n734cf6dfw68c861da5c44dc0c@mail.gmail.com>
Message-ID: <e7ba66e40906291356t20fbe904v299451af77aca2aa@mail.gmail.com>

Ah!! Sorry for the confusion... I'm not using GCC from Cygwin, just a
basic MinGW install (from MSYS).

-

On Mon, Jun 29, 2009 at 5:10 PM, Elliott
Slaughter<elliottslaughter at gmail.com> wrote:
> On Mon, Jun 29, 2009 at 12:45 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>
>> Mmm... No! I bet you do not have C:\MinGW\bin in your %PATH%, the "No
>> such file or directory" message is related to "gcc" command not being
>> found...
>>
>> Set your %PATH% appropriately before, and next build & install:
>>
>> set PATH=C:\MinGW\bin;%PATH%
>> python setup.py build -c mingw32 install
>
> Actually, it was on PATH, but cygwin had made gcc.exe.lnk a shortcut to
> gcc-3.exe, which cmd.exe couldn't find. Copying the file made everything
> work properly.
>
> Thanks for the help.
>
>> PS: If you still have problems, you should ask Microsoft for further
>> help.... (Just joking :-), keep coming!)
>
> I just wish cygwin would play nicer with native Windows programs....
>
>> -
>>
>> On Mon, Jun 29, 2009 at 3:37 PM, Elliott
>> Slaughter<elliottslaughter at gmail.com> wrote:
>> > Thanks.
>> >
>> > Now gcc seems to fail on paths with backslashes in them.
>> >
>> > C:\bin\Cython-0.11.2>python setup.py build -c mingw32
>> > ...
>> > building 'Cython.Plex.Scanners' extension
>> > gcc -mno-cygwin -mdll -O -Wall -IC:\bin\Python25\include
>> > -IC:\bin\Python25\PC -c
>> > ?C:\bin\Cython-0.11.2\Cython\Plex\Scanners.c -o
>> > c:\bin\cython-0.11.2\cython\plex
>> > \scanners.o
>> > error: command 'gcc' failed: No such file or directory
>> >
>> > Suggestions?
>> >
>> > On Mon, Jun 29, 2009 at 11:24 AM, Lisandro Dalcin <dalcinl at gmail.com>
>> > wrote:
>> >>
>> >> Build&Install using EXACTLY this:
>> >>
>> >> python setup.py build -c mingw32 install
>> >>
>> >>
>> >>
>> >> On Mon, Jun 29, 2009 at 3:13 PM, Elliott Slaughter<eslaught at ucsd.edu>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > I just tried to install Cython with Python 2.5.4 (from the binary
>> >> > installer)
>> >> > on Windows. Since I don't have MSVC installed, I get this message:
>> >> >
>> >> > C:\...\Cython-0.11.2>python setup.py install
>> >> > ...
>> >> > error: Python was built with Visual Studio 2003;
>> >> > extensions must be built with a compiler than can generate compatible
>> >> > binaries.
>> >> > Visual Studio 2003 was not found on this system. If you have Cygwin
>> >> > installed,
>> >> > you can try compiling with MingW32, by passing "-c mingw32" to
>> >> > setup.py.
>> >> >
>> >> > But the -c option isn't actually recognized by setup.py:
>> >> >
>> >> > C:\...\Cython-0.11.2>python setup.py -c mingw32 install
>> >> > ...
>> >> > usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
>> >> > ?? or: setup.py --help [cmd1 cmd2 ...]
>> >> > ?? or: setup.py --help-commands
>> >> > ?? or: setup.py cmd --help
>> >> >
>> >> > error: option -c not recognized
>> >> >
>> >> > And setup.py --help doesn't give any relevant suggestions.
>> >> >
>> >> > I would appreciate it if someone could point me in the right
>> >> > direction.
>> >> >
>> >> > Thanks.
>> >> >
>> >> > --
>> >> > Elliott Slaughter
>> >> >
>> >> > "Don't worry about what anybody else is going to do. The best way to
>> >> > predict
>> >> > the future is to invent it." - Alan Kay
>> >> >
>> >> > _______________________________________________
>> >> > Cython-dev mailing list
>> >> > Cython-dev at codespeak.net
>> >> > http://codespeak.net/mailman/listinfo/cython-dev
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Lisandro Dalc?n
>> >> ---------------
>> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> >> Tel/Fax: +54-(0)342-451.1594
>> >> _______________________________________________
>> >> Cython-dev mailing list
>> >> Cython-dev at codespeak.net
>> >> http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>> >
>> > --
>> > Elliott Slaughter
>> >
>> > "Don't worry about what anybody else is going to do. The best way to
>> > predict
>> > the future is to invent it." - Alan Kay
>> >
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
>
> --
> Elliott Slaughter
>
> "Don't worry about what anybody else is going to do. The best way to predict
> the future is to invent it." - Alan Kay
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Tue Jun 30 00:08:48 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 29 Jun 2009 18:08:48 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <e7ba66e40906291251w31821d74sdb014538113461ea@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	<e7ba66e40905272229ka61ccbbleb23adabe3ce160e@mail.gmail.com>
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
	<7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
	<7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>
	<7f014ea60906291159s1577649ax5344e6fabab0906c@mail.gmail.com>
	<e7ba66e40906291251w31821d74sdb014538113461ea@mail.gmail.com>
Message-ID: <7f014ea60906291508x6ff79de3o409570e31cb976d2@mail.gmail.com>

>>
>
> Just to be sure... if you look at all the OpenCV headers, can you see
> "__stdcall" somewhere?
>
>

here's the results of a grep:

brucewayne at broo:/usr/local/include/opencv$ grep -ir "__stdcall" *
cxtypes.h:    #define CV_STDCALL __stdcall
highgui.h:    #define CV_STDCALL __stdcall
ml.h:    #define CV_STDCALL __stdcall


Chris

From dalcinl at gmail.com  Tue Jun 30 02:23:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 29 Jun 2009 21:23:27 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60906291508x6ff79de3o409570e31cb976d2@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<e7ba66e40905272229ka61ccbbleb23adabe3ce160e@mail.gmail.com>
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
	<7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
	<7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>
	<7f014ea60906291159s1577649ax5344e6fabab0906c@mail.gmail.com>
	<e7ba66e40906291251w31821d74sdb014538113461ea@mail.gmail.com>
	<7f014ea60906291508x6ff79de3o409570e31cb976d2@mail.gmail.com>
Message-ID: <e7ba66e40906291723u6a1591e0t9ee18ce9b5ee60b1@mail.gmail.com>

Now send me the results of the cmd line below,  :-)

$ grep -ir "CV_STDCALL" *


On Mon, Jun 29, 2009 at 7:08 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>>
>>
>> Just to be sure... if you look at all the OpenCV headers, can you see
>> "__stdcall" somewhere?
>>
>>
>
> here's the results of a grep:
>
> brucewayne at broo:/usr/local/include/opencv$ grep -ir "__stdcall" *
> cxtypes.h: ? ?#define CV_STDCALL __stdcall
> highgui.h: ? ?#define CV_STDCALL __stdcall
> ml.h: ? ?#define CV_STDCALL __stdcall
>
>
> Chris
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Jun 30 02:24:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 29 Jun 2009 21:24:39 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <e7ba66e40906291723u6a1591e0t9ee18ce9b5ee60b1@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
	<7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
	<7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>
	<7f014ea60906291159s1577649ax5344e6fabab0906c@mail.gmail.com>
	<e7ba66e40906291251w31821d74sdb014538113461ea@mail.gmail.com>
	<7f014ea60906291508x6ff79de3o409570e31cb976d2@mail.gmail.com>
	<e7ba66e40906291723u6a1591e0t9ee18ce9b5ee60b1@mail.gmail.com>
Message-ID: <e7ba66e40906291724r31ccd131hedde00107a3279d3@mail.gmail.com>

Of better, please feel free to send me a tar.gz/zip containing the
full headers of OpenCV. Send them to <dalcinl at gmail.com> ...


On Mon, Jun 29, 2009 at 9:23 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> Now send me the results of the cmd line below, ?:-)
>
> $ grep -ir "CV_STDCALL" *
>
>
> On Mon, Jun 29, 2009 at 7:08 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>>>
>>>
>>> Just to be sure... if you look at all the OpenCV headers, can you see
>>> "__stdcall" somewhere?
>>>
>>>
>>
>> here's the results of a grep:
>>
>> brucewayne at broo:/usr/local/include/opencv$ grep -ir "__stdcall" *
>> cxtypes.h: ? ?#define CV_STDCALL __stdcall
>> highgui.h: ? ?#define CV_STDCALL __stdcall
>> ml.h: ? ?#define CV_STDCALL __stdcall
>>
>>
>> Chris
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Tue Jun 30 02:58:30 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 29 Jun 2009 20:58:30 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <e7ba66e40906291724r31ccd131hedde00107a3279d3@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
	<7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
	<7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>
	<7f014ea60906291159s1577649ax5344e6fabab0906c@mail.gmail.com>
	<e7ba66e40906291251w31821d74sdb014538113461ea@mail.gmail.com>
	<7f014ea60906291508x6ff79de3o409570e31cb976d2@mail.gmail.com>
	<e7ba66e40906291723u6a1591e0t9ee18ce9b5ee60b1@mail.gmail.com>
	<e7ba66e40906291724r31ccd131hedde00107a3279d3@mail.gmail.com>
Message-ID: <7f014ea60906291758g60449cfcm9a699c78a53f3f9b@mail.gmail.com>

how about both? :)

brucewayne at broo:/usr/local/include/opencv$ grep -ir "CV_STDCALL" *
cxcore.h:typedef IplImage* (CV_STDCALL* Cv_iplCreateImageHeader)
cxcore.h:typedef void (CV_STDCALL* Cv_iplAllocateImageData)(IplImage*,int,int);
cxcore.h:typedef void (CV_STDCALL* Cv_iplDeallocate)(IplImage*,int);
cxcore.h:typedef IplROI* (CV_STDCALL* Cv_iplCreateROI)(int,int,int,int,int);
cxcore.h:typedef IplImage* (CV_STDCALL* Cv_iplCloneImage)(const IplImage*);
cxtypes.h:    #define CV_STDCALL __stdcall
cxtypes.h:    #define CV_STDCALL
highgui.h:    #define CV_STDCALL __stdcall
highgui.h:    #define CV_STDCALL
ml.h:    #define CV_STDCALL __stdcall
ml.h:    #define CV_STDCALL


and you got mail!

Chris

On Mon, Jun 29, 2009 at 8:24 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
> Of better, please feel free to send me a tar.gz/zip containing the
> full headers of OpenCV. Send them to <dalcinl at gmail.com> ...
>
>
> On Mon, Jun 29, 2009 at 9:23 PM, Lisandro Dalcin<dalcinl at gmail.com> wrote:
>> Now send me the results of the cmd line below, ?:-)
>>
>> $ grep -ir "CV_STDCALL" *
>>
>>
>> On Mon, Jun 29, 2009 at 7:08 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>>>>
>>>>
>>>> Just to be sure... if you look at all the OpenCV headers, can you see
>>>> "__stdcall" somewhere?
>>>>
>>>>
>>>
>>> here's the results of a grep:
>>>
>>> brucewayne at broo:/usr/local/include/opencv$ grep -ir "__stdcall" *
>>> cxtypes.h: ? ?#define CV_STDCALL __stdcall
>>> highgui.h: ? ?#define CV_STDCALL __stdcall
>>> ml.h: ? ?#define CV_STDCALL __stdcall
>>>
>>>
>>> Chris
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Tue Jun 30 04:17:50 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Jun 2009 19:17:50 -0700
Subject: [Cython] [Fwd: Re: Surprise: A bool isn't a bool]
In-Reply-To: <4A48EBD6.2080706@student.matnat.uio.no>
References: <4A48EBD6.2080706@student.matnat.uio.no>
Message-ID: <BF576354-872A-4C6F-B8BB-EA35F54C070A@math.washington.edu>

On Jun 29, 2009, at 9:29 AM, Dag Sverre Seljebotn wrote:

> -------- Original Message --------
> Subject: Re: Surprise: A bool isn't a bool
> Date: Mon, 29 Jun 2009 18:26:58 +0200
> From: Bjarke Roune <bjarke at daimi.au.dk>
> To: Dag Sverre Seljebotn <dagss at student.matnat.uio.no>
> References: <mailman.42053.1246289710.27370.cython-dev at codespeak.net>
>   <4f74ca650906290836t4c2a6f27i45b5b436abe4951c at mail.gmail.com>	
> <4A48E772.8030603 at student.matnat.uio.no>
>
>> Indeed. The question is whether it is possible to find such a way.  
>> Perhaps
>> disallow "bool" in cdef extern functions? Although that would add an
>> unecesarry typecheck if a C library in fact did return a Python  
>> boolean
>> object... (like many C functions in Python's standard library might).
>>
> Perhaps let there be a switch to turn the error or warning off, and
> some kind of instruction in a Cython file that turns it off for only
> that file.

Actually, I'm not really sure why we have that as a builtin type at  
all--it doesn't really do any good and leads to confusion (especially  
if we introduce a real bool type later).

- Robert


From robertwb at math.washington.edu  Tue Jun 30 04:20:58 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Jun 2009 19:20:58 -0700
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <45239150906291228k1f8c7ebdtaaa4b13b742403d3@mail.gmail.com>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no>
	<4A45B2AB.9060609@behnel.de>
	<4A45C421.4090005@student.matnat.uio.no>
	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
	<4A469706.8050509@student.matnat.uio.no>
	<8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
	<4A48C1D5.80204@student.matnat.uio.no>
	<45239150906291228k1f8c7ebdtaaa4b13b742403d3@mail.gmail.com>
Message-ID: <4CB4349B-BBE7-4AA4-88B2-4B432DF4AFE0@math.washington.edu>

On Jun 29, 2009, at 12:28 PM, Danilo Freitas wrote:

> Hi,
>
> sorry for my delay.  I'm still with internet problems at home :(
>
>
> I'm a little inclined for a C++ syntax. I think the operators
> declaration on C++ syntax are more clearer. I always thought things
> like __add__, __getitem__, __setitem__ weird.
>
> It would also be clearer for the parser. When it see 'operator', it
> already knows what is to come, and check if the operator is in an
> "operator list".
>
> I think maybe we could also define operators outside a C++ class
> declaring block (like C++).

 >>> import this

> Allowing the complete C++ classes on Cython, bringing STL to it will
> be a simpler step.

Of course this does make references and reference semantics necessary  
before we get operator overloading in. Did you make a list of the  
pros/cons of each the two options?

- Robert


From stefan_ml at behnel.de  Tue Jun 30 12:25:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 30 Jun 2009 12:25:07 +0200
Subject: [Cython] OT: Sage build
In-Reply-To: <D0A6A516-6596-4870-820B-02B1B4BB8356@math.washington.edu>
References: <4A45D225.9090304@student.matnat.uio.no>	<944B5237-C63F-4F3B-9523-E611BEED74ED@math.washington.edu>	<4A48B764.8060404@student.matnat.uio.no>
	<D0A6A516-6596-4870-820B-02B1B4BB8356@math.washington.edu>
Message-ID: <4A49E803.3060905@behnel.de>


Robert Bradshaw wrote:
> On Jun 29, 2009, at 5:45 AM, Dag Sverre Seljebotn wrote:
>> do you know if e.g. Sage builds are IO bound?
> 
> Not sure, but that's certainly a big component.

Now that you mention it, how do you parallelise the builds anyway?
(assuming that you do that)

Stefan


From wstein at gmail.com  Tue Jun 30 12:42:42 2009
From: wstein at gmail.com (William Stein)
Date: Tue, 30 Jun 2009 12:42:42 +0200
Subject: [Cython] OT: Sage build
In-Reply-To: <4A49E803.3060905@behnel.de>
References: <4A45D225.9090304@student.matnat.uio.no>
	<944B5237-C63F-4F3B-9523-E611BEED74ED@math.washington.edu> 
	<4A48B764.8060404@student.matnat.uio.no>
	<D0A6A516-6596-4870-820B-02B1B4BB8356@math.washington.edu> 
	<4A49E803.3060905@behnel.de>
Message-ID: <85e81ba30906300342o62bc9fa6m3530a391a3f9e370@mail.gmail.com>

On Tue, Jun 30, 2009 at 12:25 PM, Stefan Behnel<stefan_ml at behnel.de> wrote:
>
> Robert Bradshaw wrote:
>> On Jun 29, 2009, at 5:45 AM, Dag Sverre Seljebotn wrote:
>>> do you know if e.g. Sage builds are IO bound?
>>
>> Not sure, but that's certainly a big component.
>
> Now that you mention it, how do you parallelise the builds anyway?
> (assuming that you do that)
>
> Stefan

I wrote half the code to do this (=build Sage's Cython library in
parallel), so I'll comment briefly.  I've cc'd Craig Citro who wrote
the other half, so maybe he can comment.   The Cython part of the Sage
library consists of 237 distinct Cython modules.  We build these
modules in parallel in two stages:

  (1) Run Cython on all modules that need to be regenerated (as
determined by our automatic dependency checking system) in parallel.
This is done in parallel using the standard Python-2.6 multiprocessing
module.

  (2) One (1) is completely done, run gcc in parallel, again I think
using multiprocessing.  This is done by modifying Python's distutils.

The Sage project's standard developer build machine has 24 cores, so
having the above is a huge time saver.

For those using Sage who want to use the above, just do "export
MAKE='make -j12'" say, to tell "sage -br" to run up to 12 processes at
once during the build process.

 -- William

>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

From barthelemy at crans.org  Tue Jun 30 16:21:36 2009
From: barthelemy at crans.org (=?ISO-8859-1?Q?S=E9bastien_Barth=E9lemy?=)
Date: Tue, 30 Jun 2009 16:21:36 +0200
Subject: [Cython] ImportError: ./pycdd.so: undefined symbol: __gmpq_init
Message-ID: <78f7ab620906300721s39b3b82fl659881c598614dbe@mail.gmail.com>

Hello,

after a months-long pause, I'm giving a new try to libcdd wrappers,
and have a few related questions.

first, is it possible to put compile-time definitions (DEF) outside
the file? As an option to cython or to the Cython.Distutil? It would
be useful to reflect #ifdef option in cython from the build script and
avoid some redundancy.

second, is there a way to acces the C file descriptor of python file
types from cython? I could not find it in cython documentation nor in
python C-API reference.

Third, ... well..., it does not work, and I don't understand why. When
I import my module, it fails with the following traceback.

>>> import pycdd
ImportError                               Traceback (most recent call last)
/home/seb/Devel/pypolyhedra/pycdd/<ipython console> in <module>()
ImportError: ./pycdd.so: undefined symbol: __gmpq_init

any idea of what might happen ?

here is the compile log.

$ LANG=C python setup.py build_ext --inplace && ipython test.py
running build_ext
cythoning pygmp.pyx to pygmp.c
building 'pygmp' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c pygmp.c -o
build/temp.linux-i686-2.6/pygmp.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
build/temp.linux-i686-2.6/pygmp.o -lgmp -o pygmp.so
cythoning pycdd.pyx to pycdd.c
building 'pycdd' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -Icddlib-094f/lib-src-gmp
-I/usr/include/python2.6 -c pycdd.c -o
build/temp.linux-i686-2.6/pycdd.o
pycdd.c: In function '__pyx_pf_5pycdd_13CddPolyhedron_get_hrep':
pycdd.c:1061: warning: passing argument 1 of
'__pyx_convert__to_py_mpq_t' from incompatible pointer type
pycdd.c:1168: warning: passing argument 1 of
'__pyx_convert__to_py_mpq_t' from incompatible pointer type
pycdd.c: In function '__pyx_f_5pycdd__hrep_to_cdd_matrix':
pycdd.c:1533: warning: passing argument 1 of '__gmpq_set' from
incompatible pointer type
pycdd.c:1595: warning: passing argument 1 of '__gmpq_neg' from
incompatible pointer type
pycdd.c: In function '__pyx_convert__to_py_mpq_t':
pycdd.c:2353: warning: label 'bad' defined but not used
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
build/temp.linux-i686-2.6/pycdd.o -Lcddlib-094f/lib-src-gmp -lcddgmp
-o pycdd.so

The program is available in there:
 http://perso.crans.org/barthelemy/pycdd.tar.gz
 http://perso.crans.org/barthelemy/pycdd.git/

but I'was expecting to get it a little bit further before asking for a review.

Thank you for any help !

From robertwb at math.washington.edu  Tue Jun 30 17:55:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Jun 2009 08:55:57 -0700
Subject: [Cython] OT: Sage build
In-Reply-To: <4A49E803.3060905@behnel.de>
References: <4A45D225.9090304@student.matnat.uio.no>	<944B5237-C63F-4F3B-9523-E611BEED74ED@math.washington.edu>	<4A48B764.8060404@student.matnat.uio.no>
	<D0A6A516-6596-4870-820B-02B1B4BB8356@math.washington.edu>
	<4A49E803.3060905@behnel.de>
Message-ID: <69BE1C8E-F570-4BD8-BA93-2FD8C6DC320E@math.washington.edu>

On Jun 30, 2009, at 3:25 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> On Jun 29, 2009, at 5:45 AM, Dag Sverre Seljebotn wrote:
>>> do you know if e.g. Sage builds are IO bound?
>>
>> Not sure, but that's certainly a big component.
>
> Now that you mention it, how do you parallelise the builds anyway?
> (assuming that you do that)

We walk through the list of extensions in setup.py before passing  
them onto distutils (running cython on them manually, and replacing  
the .pyx files in the source list with .c(pp)). We then do our own  
dependancy checking, and divide them up into n lists, and fork n  
threads in Python.

(I didn't write the forking part, so that's just the big picture.)  
It's all in http://hg.sagemath.org/sage-main/file/d2d8c2f97d32/setup.py

- Robert
  

From robertwb at math.washington.edu  Tue Jun 30 18:00:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Jun 2009 09:00:40 -0700
Subject: [Cython] ImportError: ./pycdd.so: undefined symbol: __gmpq_init
In-Reply-To: <78f7ab620906300721s39b3b82fl659881c598614dbe@mail.gmail.com>
References: <78f7ab620906300721s39b3b82fl659881c598614dbe@mail.gmail.com>
Message-ID: <27D29499-6609-4A59-9AB6-880493B4A797@math.washington.edu>

On Jun 30, 2009, at 7:21 AM, S?bastien Barth?lemy wrote:

> Hello,
>
> after a months-long pause, I'm giving a new try to libcdd wrappers,
> and have a few related questions.
>
> first, is it possible to put compile-time definitions (DEF) outside
> the file? As an option to cython or to the Cython.Distutil? It would
> be useful to reflect #ifdef option in cython from the build script and
> avoid some redundancy.

No, that's not possible.

>
> second, is there a way to acces the C file descriptor of python file
> types from cython? I could not find it in cython documentation nor in
> python C-API reference.

Yes, it's in the Python/C API refernce under "objects."

>
> Third, ... well..., it does not work, and I don't understand why. When
> I import my module, it fails with the following traceback.
>
>>>> import pycdd
> ImportError                               Traceback (most recent  
> call last)
> /home/seb/Devel/pypolyhedra/pycdd/<ipython console> in <module>()
> ImportError: ./pycdd.so: undefined symbol: __gmpq_init
>
> any idea of what might happen ?

My guess would be that you're declaring something wrong (given the  
amount of warnings below).

>
> here is the compile log.
>
> $ LANG=C python setup.py build_ext --inplace && ipython test.py
> running build_ext
> cythoning pygmp.pyx to pygmp.c
> building 'pygmp' extension
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c pygmp.c -o
> build/temp.linux-i686-2.6/pygmp.o
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-i686-2.6/pygmp.o -lgmp -o pygmp.so
> cythoning pycdd.pyx to pycdd.c
> building 'pycdd' extension
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -Icddlib-094f/lib-src-gmp
> -I/usr/include/python2.6 -c pycdd.c -o
> build/temp.linux-i686-2.6/pycdd.o
> pycdd.c: In function '__pyx_pf_5pycdd_13CddPolyhedron_get_hrep':
> pycdd.c:1061: warning: passing argument 1 of
> '__pyx_convert__to_py_mpq_t' from incompatible pointer type
> pycdd.c:1168: warning: passing argument 1 of
> '__pyx_convert__to_py_mpq_t' from incompatible pointer type
> pycdd.c: In function '__pyx_f_5pycdd__hrep_to_cdd_matrix':
> pycdd.c:1533: warning: passing argument 1 of '__gmpq_set' from
> incompatible pointer type
> pycdd.c:1595: warning: passing argument 1 of '__gmpq_neg' from
> incompatible pointer type
> pycdd.c: In function '__pyx_convert__to_py_mpq_t':
> pycdd.c:2353: warning: label 'bad' defined but not used
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-i686-2.6/pycdd.o -Lcddlib-094f/lib-src-gmp -lcddgmp
> -o pycdd.so
>
> The program is available in there:
>  http://perso.crans.org/barthelemy/pycdd.tar.gz
>  http://perso.crans.org/barthelemy/pycdd.git/
>
> but I'was expecting to get it a little bit further before asking  
> for a review.
>
> Thank you for any help !
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Tue Jun 30 19:15:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 30 Jun 2009 19:15:48 +0200
Subject: [Cython] ImportError: ./pycdd.so: undefined symbol: __gmpq_init
In-Reply-To: <27D29499-6609-4A59-9AB6-880493B4A797@math.washington.edu>
References: <78f7ab620906300721s39b3b82fl659881c598614dbe@mail.gmail.com>
	<27D29499-6609-4A59-9AB6-880493B4A797@math.washington.edu>
Message-ID: <4A4A4844.7030003@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Jun 30, 2009, at 7:21 AM, S?bastien Barth?lemy wrote:
> 
>> Hello,
>>
>> after a months-long pause, I'm giving a new try to libcdd wrappers,
>> and have a few related questions.
>>
>> first, is it possible to put compile-time definitions (DEF) outside
>> the file? As an option to cython or to the Cython.Distutil? It would
>> be useful to reflect #ifdef option in cython from the build script and
>> avoid some redundancy.
> 
> No, that's not possible.
> 
>> second, is there a way to acces the C file descriptor of python file
>> types from cython? I could not find it in cython documentation nor in
>> python C-API reference.
> 
> Yes, it's in the Python/C API refernce under "objects."

Be aware though at in some compilation environments this will fail 
though (e.g. Windows, with MinGW as compiler for Cython module but 
Visual C-compiled Python).


-- 
Dag Sverre

From dsurviver at gmail.com  Tue Jun 30 20:21:57 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Tue, 30 Jun 2009 15:21:57 -0300
Subject: [Cython] Vote for constructor method of C++ classes
In-Reply-To: <4CB4349B-BBE7-4AA4-88B2-4B432DF4AFE0@math.washington.edu>
References: <45239150906261331s3b460866i8716daaf11146ced@mail.gmail.com>
	<4A453A19.5040305@student.matnat.uio.no> <4A45B2AB.9060609@behnel.de>
	<4A45C421.4090005@student.matnat.uio.no>
	<65D8D186-BF46-4414-B60C-19B545F91DEA@math.washington.edu>
	<4A469706.8050509@student.matnat.uio.no>
	<8548E308-527E-4845-8647-A3B23AA97EA5@math.washington.edu>
	<4A48C1D5.80204@student.matnat.uio.no>
	<45239150906291228k1f8c7ebdtaaa4b13b742403d3@mail.gmail.com>
	<4CB4349B-BBE7-4AA4-88B2-4B432DF4AFE0@math.washington.edu>
Message-ID: <45239150906301121r337df296t98de619def26f086@mail.gmail.com>

>  >>> import this

Didn't know that. Cool :)

> Of course this does make references and reference semantics necessary
> before we get operator overloading in. Did you make a list of the
> pros/cons of each the two options?

Still not. But I can do it right now. Pros/cons are for now just in my mind.
In some minutes I can do it and post on the mailing list.


-- 
- Danilo Freitas

From elliottslaughter at gmail.com  Tue Jun 30 22:37:56 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Tue, 30 Jun 2009 13:37:56 -0700
Subject: [Cython] Search paths for definition files
Message-ID: <42c0ab790906301337u7f8c4f9rdcbf60f99a4f74b9@mail.gmail.com>

Hi,

The manual states that "the Cython compiler searches for a file called
modulename.pxd along the search path for include files, as specified by -I
command line options" [1].

I'm wondering if there is any way to get this behavior with distutils (i.e.
from inside a setup.py script). The build_ext command doens't seem to
support using the -I option in the same way that Cython does.

$ cython -Idir foo.pyx

$ python setup_foo.py build_ext -c mingw32 -i
running build_ext
error: file 'bar.pxd' does not exist

$ mv dir/bar.pxd .

$ python setup_foo.py build_ext -c mingw32 -i
running build_ext
building 'foo' extension
... compiles successfully ...

$ cat foo.pyx
cimport bar

$ cat bar.pxd

$ cat setup_foo.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension('foo', ['foo.pyx'])]
)

This is with Python 2.5.4, Cython 0.11.2, and MinGW from Cygwin on Windows.

Thanks.

[1]
http://docs.cython.org/docs/sharing_declarations.html#search-paths-for-definition-files

-- 
Elliott Slaughter

"Don't worry about what anybody else is going to do. The best way to predict
the future is to invent it." - Alan Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090630/fd71d121/attachment.htm 

From dalcinl at gmail.com  Tue Jun 30 23:43:00 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 30 Jun 2009 18:43:00 -0300
Subject: [Cython] Search paths for definition files
In-Reply-To: <42c0ab790906301337u7f8c4f9rdcbf60f99a4f74b9@mail.gmail.com>
References: <42c0ab790906301337u7f8c4f9rdcbf60f99a4f74b9@mail.gmail.com>
Message-ID: <e7ba66e40906301443p7aeecec2o587584449fec6db7@mail.gmail.com>

On Tue, Jun 30, 2009 at 5:37 PM, Elliott
Slaughter<elliottslaughter at gmail.com> wrote:
> Hi,
>
> The manual states that "the Cython compiler searches for a file called
> modulename.pxd along the search path for include files, as specified by -I
> command line options" [1].
>

A good thing about open source projects is that the best
docummentation are the actual sources :-)
http://hg.cython.org/cython-devel/file/tip/Cython/Distutils/extension.py

Try this:

from Cython.Distutils.extension import Extension
...
Extension('foo', ['foo.pyx'], pyrex_include_dirs=['dir1', 'dir2', 'dir3'])


Other way (not recommended, not portable for POSIX/Windows):
add a setup.cfg file alongside your setup.py file, and write inside it
(use ";" to separate on Windows):

[build_ext]
pyrex_include_dirs = dir1:dir2:dir3



> I'm wondering if there is any way to get this behavior with distutils (i.e.
> from inside a setup.py script). The build_ext command doens't seem to
> support using the -I option in the same way that Cython does.
>
> $ cython -Idir foo.pyx
>
> $ python setup_foo.py build_ext -c mingw32 -i
> running build_ext
> error: file 'bar.pxd' does not exist
>
> $ mv dir/bar.pxd .
>
> $ python setup_foo.py build_ext -c mingw32 -i
> running build_ext
> building 'foo' extension
> ... compiles successfully ...
>
> $ cat foo.pyx
> cimport bar
>
> $ cat bar.pxd
>
> $ cat setup_foo.py
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> setup(
> ??? cmdclass = {'build_ext': build_ext},
> ??? ext_modules = [Extension('foo', ['foo.pyx'])]
> )
>
> This is with Python 2.5.4, Cython 0.11.2, and MinGW from Cygwin on Windows.
>
> Thanks.
>
> [1]
> http://docs.cython.org/docs/sharing_declarations.html#search-paths-for-definition-files
>
> --
> Elliott Slaughter
>
> "Don't worry about what anybody else is going to do. The best way to predict
> the future is to invent it." - Alan Kay
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From elliottslaughter at gmail.com  Tue Jun 30 23:55:24 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Tue, 30 Jun 2009 14:55:24 -0700
Subject: [Cython] Search paths for definition files
In-Reply-To: <e7ba66e40906301443p7aeecec2o587584449fec6db7@mail.gmail.com>
References: <42c0ab790906301337u7f8c4f9rdcbf60f99a4f74b9@mail.gmail.com>
	<e7ba66e40906301443p7aeecec2o587584449fec6db7@mail.gmail.com>
Message-ID: <42c0ab790906301455i9e14fb0y84e56a10435fe04a@mail.gmail.com>

On Tue, Jun 30, 2009 at 2:43 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> On Tue, Jun 30, 2009 at 5:37 PM, Elliott
> Slaughter<elliottslaughter at gmail.com> wrote:
> > Hi,
> >
> > The manual states that "the Cython compiler searches for a file called
> > modulename.pxd along the search path for include files, as specified by
> -I
> > command line options" [1].
> >
>
> A good thing about open source projects is that the best
> docummentation are the actual sources :-)
> http://hg.cython.org/cython-devel/file/tip/Cython/Distutils/extension.py
>
> Try this:
>
> from Cython.Distutils.extension import Extension
> ...
> Extension('foo', ['foo.pyx'], pyrex_include_dirs=['dir1', 'dir2', 'dir3'])
>

Thanks.

-- 
Elliott Slaughter

"Don't worry about what anybody else is going to do. The best way to predict
the future is to invent it." - Alan Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090630/42602db9/attachment.htm