From stefan_ml at behnel.de  Sun Jun  1 14:10:11 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 01 Jun 2008 14:10:11 +0200
Subject: [Cython] about let users have more control on garbage collection
In-Reply-To: <e7ba66e40805311449n2151aaf0s2266fb5d9344e9b1@mail.gmail.com>
References: <e7ba66e40805300740qb59c01nc60d72d07e44a276@mail.gmail.com>	<48418CC9.2030103@behnel.de>
	<e7ba66e40805311449n2151aaf0s2266fb5d9344e9b1@mail.gmail.com>
Message-ID: <484291A3.5090909@behnel.de>

Hi,

Lisandro Dalcin wrote:
> cdef class Mat: # <- proxy class
>    cdef PetscMat # <- PETSc object, a pointer to a refcounted struct.
> 
> I need to store arbitrary Python objects inside the PETSc object, I'm
> actually storing a dictionary. This dictionary have to survive until
> the PETSc object is deallocated, but that deallocation do not
> necesarily occurs when the instance of my proxy class is deallocated,
> because the PETSc object can still 'live' (that is, a referece being
> owned) inside other PETSc objects.

Can't you work with singleton proxies? That's what we do in lxml. There's
never more than one Python Element proxy for an XML node struct. We use a
factory function to build a proxy for a struct and keep a back pointer from
the struct to the class, so that we can reuse an already existing proxy. All
the Python state is kept in attributes of the proxy object, and the struct can
be freed when the proxy is garbage collected by Python.

Stefan


From fperez.net at gmail.com  Sun Jun  1 22:18:29 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Jun 2008 13:18:29 -0700
Subject: [Cython] Cython tutorial at Scipy 2008?
Message-ID: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>

Hi folks,

I'm in charge of co-organizing (with Travis Oliphant) the tutorial
sessions at Scipy 2008, to be held in August of this year at Caltech:

http://www.scipy.org/SciPy2008
http://www.scipy.org/SciPy2008/Tutorials

Any chance someone from the Cython gang might come and be willing to
give a cython tutorial?  We don't have  too many details sorted out
yet, so we're just feeling things  out...

Cheers,

f

From wstein at gmail.com  Sun Jun  1 22:23:46 2008
From: wstein at gmail.com (William Stein)
Date: Sun, 1 Jun 2008 13:23:46 -0700
Subject: [Cython] Cython tutorial at Scipy 2008?
In-Reply-To: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>
References: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>
Message-ID: <85e81ba30806011323h3c7bac75ua046616e5f6d54a8@mail.gmail.com>

On Sun, Jun 1, 2008 at 1:18 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi folks,
>
> I'm in charge of co-organizing (with Travis Oliphant) the tutorial
> sessions at Scipy 2008, to be held in August of this year at Caltech:
>
> http://www.scipy.org/SciPy2008
> http://www.scipy.org/SciPy2008/Tutorials
>
> Any chance someone from the Cython gang might come and be willing to
> give a cython tutorial?  We don't have  too many details sorted out
> yet, so we're just feeling things  out...
>

Robert Bradshaw and I will likely both be in nearby San Diego then,
in which case one or both of us could potentially come.

William

From fperez.net at gmail.com  Mon Jun  2 02:46:15 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Jun 2008 17:46:15 -0700
Subject: [Cython] Cython tutorial at Scipy 2008?
In-Reply-To: <85e81ba30806011323h3c7bac75ua046616e5f6d54a8@mail.gmail.com>
References: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>
	<85e81ba30806011323h3c7bac75ua046616e5f6d54a8@mail.gmail.com>
Message-ID: <db6b5ecc0806011746s739bff3ci1cf2b65c8df403fd@mail.gmail.com>

On Sun, Jun 1, 2008 at 1:23 PM, William Stein <wstein at gmail.com> wrote:
> On Sun, Jun 1, 2008 at 1:18 PM, Fernando Perez <fperez.net at gmail.com> wrote:
>> Hi folks,
>>
>> I'm in charge of co-organizing (with Travis Oliphant) the tutorial
>> sessions at Scipy 2008, to be held in August of this year at Caltech:
>>
>> http://www.scipy.org/SciPy2008
>> http://www.scipy.org/SciPy2008/Tutorials
>>
>> Any chance someone from the Cython gang might come and be willing to
>> give a cython tutorial?  We don't have  too many details sorted out
>> yet, so we're just feeling things  out...
>>
>
> Robert Bradshaw and I will likely both be in nearby San Diego then,
> in which case one or both of us could potentially come.

Great.  I had also listed a Sage tutorial as a potential topic, so as
long as some of you are around, perhaps you could pitch in for that
too?  Right now we're just getting feelers for both who could teach
what, and what people want to see.  It's a bit of a matchmaking game,
but there's no point in offering something if we don't have anyone to
teach it.

Note that in addition to the 2 days of tutorials and 2 days of
conference, the weekend is reserved for dev sprints.  It would be very
nice to have some joint work on sage/cython/numpy/scipy take place.
Many of the usual suspects will be there.

Cheers,

f

From wstein at gmail.com  Mon Jun  2 03:46:36 2008
From: wstein at gmail.com (William Stein)
Date: Sun, 1 Jun 2008 18:46:36 -0700
Subject: [Cython] Cython tutorial at Scipy 2008?
In-Reply-To: <db6b5ecc0806011746s739bff3ci1cf2b65c8df403fd@mail.gmail.com>
References: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>
	<85e81ba30806011323h3c7bac75ua046616e5f6d54a8@mail.gmail.com>
	<db6b5ecc0806011746s739bff3ci1cf2b65c8df403fd@mail.gmail.com>
Message-ID: <85e81ba30806011846id8eb8b2w40828096229f3d6@mail.gmail.com>

On Sun, Jun 1, 2008 at 5:46 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Sun, Jun 1, 2008 at 1:23 PM, William Stein <wstein at gmail.com> wrote:
>> On Sun, Jun 1, 2008 at 1:18 PM, Fernando Perez <fperez.net at gmail.com> wrote:
>>> Hi folks,
>>>
>>> I'm in charge of co-organizing (with Travis Oliphant) the tutorial
>>> sessions at Scipy 2008, to be held in August of this year at Caltech:
>>>
>>> http://www.scipy.org/SciPy2008
>>> http://www.scipy.org/SciPy2008/Tutorials
>>>
>>> Any chance someone from the Cython gang might come and be willing to
>>> give a cython tutorial?  We don't have  too many details sorted out
>>> yet, so we're just feeling things  out...
>>>
>>
>> Robert Bradshaw and I will likely both be in nearby San Diego then,
>> in which case one or both of us could potentially come.
>
> Great.  I had also listed a Sage tutorial as a potential topic, so as
> long as some of you are around, perhaps you could pitch in for that
> too?  Right now we're just getting feelers for both who could teach

I am certainly enthusiastic about doing a Sage tutorial.

> what, and what people want to see.  It's a bit of a matchmaking game,
> but there's no point in offering something if we don't have anyone to
> teach it.
>
> Note that in addition to the 2 days of tutorials and 2 days of
> conference, the weekend is reserved for dev sprints.  It would be very
> nice to have some joint work on sage/cython/numpy/scipy take place.
> Many of the usual suspects will be there.
>

Excellent.

William

From fperez.net at gmail.com  Mon Jun  2 04:05:15 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Sun, 1 Jun 2008 19:05:15 -0700
Subject: [Cython] Cython tutorial at Scipy 2008?
In-Reply-To: <85e81ba30806011846id8eb8b2w40828096229f3d6@mail.gmail.com>
References: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>
	<85e81ba30806011323h3c7bac75ua046616e5f6d54a8@mail.gmail.com>
	<db6b5ecc0806011746s739bff3ci1cf2b65c8df403fd@mail.gmail.com>
	<85e81ba30806011846id8eb8b2w40828096229f3d6@mail.gmail.com>
Message-ID: <db6b5ecc0806011905k31092481v4dfc282703244057@mail.gmail.com>

On Sun, Jun 1, 2008 at 6:46 PM, William Stein <wstein at gmail.com> wrote:

> I am certainly enthusiastic about doing a Sage tutorial.

Ok, thanks for the info.  We'll get in touch with you when as soon as
we know a bit more regarding schedule, etc.

Cheers,

f

From ondrej at certik.cz  Mon Jun  2 13:47:39 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 2 Jun 2008 13:47:39 +0200
Subject: [Cython] SWIG and Cython
In-Reply-To: <e7ba66e40805301443q7e483ecu6df52375d12589e0@mail.gmail.com>
References: <96de71860805301247t5c6e53e8x953734e26c1340de@mail.gmail.com>
	<e7ba66e40805301443q7e483ecu6df52375d12589e0@mail.gmail.com>
Message-ID: <85b5c3130806020447g608bc3cfr104b1d8dfb5c1232@mail.gmail.com>

On Fri, May 30, 2008 at 11:43 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> Well, In the case of wrapping large, legacy C++ libraries, then the
> choice is not easy, it depend on many factors. I'll try to elaborate
>
> For something like MPI and PETSc in case for mpi4py and petsc4py,
> which are both being converted to Cython, the option is really clear
> for me: just use Cython, do not bother with SWIG.
>
> Why use Cython? Their native API's are just C (well, MPI has a
> standard C++ API, but not much more featured than the C one). Wrapping
> a C API and providing just C-ish functions at the Python level is just
> unpythonic and painfull to use that plain C. You really need a trully
> OO Python API, if not, you Python level API is just much crap than the
> C one, hard to use, perhaps the only you alleviate is error checking.
> Building a OO API targeting Python through using Cython and calling a
> your legacy library C API is just easy, fast, convenient than using
> SWIG and next use the SWIG generated code to implement a
> better-looking, high-level, pythonic API for Python-level consumption.
>
> In the case of a large C++ API its depend on how much of that API it
> makes sense to expose to Python. Let suppose we have an API with a lot
> of classes and a lot of methods (let say 10,20,30 classes, with
> 10,10,30 methods each). Then wrapping the full API in the current
> status of Cython, that is going to be a real pain, to much manual.
> BUT, if you are going to use only a few of your classes, and only
> expose a few of your methods, or even want to expose new methods, more
> convenient to use in the Python side, then, again I will definitelly
> choose Cython.
>
> My personal experience in going from SWIG to Cython is worth to take
> into account:
>
> Implementing a really good, robust, full featured, pythonic access to
> full PETSc functionalities for petsc4py took me about two years of
> carefull writting of typemaps, implementing Python extension types BY
> HAND, and writting a log or C code inside custom typemaps, with Python
> and Numpy  C API calls plus PETSc calls.
>
> Then, after the success with Cython for the mpi4py case, I started the
> process of rewritting petsc4py from scratch, working at night, let say
> from 9:00 PM to until 1:00 AM or 2:00 AM. In about 10 days, yes, JUST
> 10 DAYS of night work, I'm near to have implemented all the features
> of the previous implementation, and the new implementation is actually
> bether and more featured....
>
>
> In short, if you really, really, really want to wrapp the FULL C++
> library, then the Cython way will be harder than the SWIG way. But I
> really doubt that you really need to access the FULL C++ API. But one
> thing that you sould take for granted: the Cython way will let you
> write better-looking, more easily maintenable source code.
>
> That's all, folks!

Thanks for sharing this, it is a very useful info!

Ondrej

From dalcinl at gmail.com  Mon Jun  2 16:58:10 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 2 Jun 2008 11:58:10 -0300
Subject: [Cython] about let users have more control on garbage collection
In-Reply-To: <484291A3.5090909@behnel.de>
References: <e7ba66e40805300740qb59c01nc60d72d07e44a276@mail.gmail.com>
	<48418CC9.2030103@behnel.de>
	<e7ba66e40805311449n2151aaf0s2266fb5d9344e9b1@mail.gmail.com>
	<484291A3.5090909@behnel.de>
Message-ID: <e7ba66e40806020758l7f64780cwf7ec46caf9bd8e25@mail.gmail.com>

On 6/1/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
> Can't you work with singleton proxies? That's what we do in lxml. There's
>  never more than one Python Element proxy for an XML node struct.

I'll take a look, but I believe such approach, even if possible, would
complicate a lot my implementation.

Stefan, iff it is no much work for you, could you point me a link to
the actual lxml code implementing all this machinery?




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Mon Jun  2 20:43:08 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 2 Jun 2008 11:43:08 -0700
Subject: [Cython] SWIG and Cython
In-Reply-To: <e7ba66e40805301520y145dcc2dp511a813f70cd91dd@mail.gmail.com>
References: <96de71860805301247t5c6e53e8x953734e26c1340de@mail.gmail.com>
	<e7ba66e40805301443q7e483ecu6df52375d12589e0@mail.gmail.com>
	<d75825660805301451s3bb6823dn7422738f0bc11e2c@mail.gmail.com>
	<e7ba66e40805301520y145dcc2dp511a813f70cd91dd@mail.gmail.com>
Message-ID: <D117F337-56AD-4711-81E1-D96BA0B3B757@math.washington.edu>

On May 30, 2008, at 3:20 PM, Lisandro Dalcin wrote:

> On 5/30/08, rahul garg <whycode at gmail.com> wrote:
>> I am a little curious about ctypes. Have you looked at ctypes and  
>> what
>> advantages/disadvantages do you think it has over SWIG and Cython for
>> wrapping the libraries you are interested in? Of course ctypes  
>> doesnt work
>> for C++ but any other specific problems?
>
> Well, a C library API is not just functions and data exported by the
> linker. It is also #define'd macros, typedef'd stuff, custom C structs
> with many slots. If you want to write a full featured wrapper to a
> complex library, the ctypes way is going to be even more painfull than
> SWIG or Cython.
>
> Suppose you want to access, in the MPI case, MPI_COMM_WORLD. Look at
> the header files of major's MPI out there (MPICH2, OpenMPI, HP MPI,
> IBM MPI): in all cases MPI_COMM_WORLD is a #defined thing, in some
> case an integer, in other a pointer. And that applies to all MPI
> predefined handles (operations, datatypes, etc,) and other stuff. You
> just cannot access and manage a full C API in ctypes; at least, I do
> not know how to handle that.
>
> For example, the new Cython-based mpi4py can be build against Python
> version ranging from 2.3 to 3.0 (yes, it is Py3K ready!), and with all
> major MPI's out there, either if they are MPI-1 or MPI-2
> implementation, in Linux, Mac OS X, and Windows. I'll never buy the
> ctypes way until someone can show me that this degree of portability
> is possible. Even if someone can do this with ctypes, I guess such a
> code would not be easy to understand and maintain.
>
> For the case of petsc4py, the same comments apply. PETSc C API do have
> macros. Furthermore, it's API changes from release to release (PETSc
> developers does not bother from backward compatibility issues).  But
> petsc4py, either the former SWIG-based or the newer Cython-based, can
> be used with petsc-2.3.2, petsc-2.3.3, or the latest development copy.
> All this with a layer of compatibility functions and macros. How in
> the hell a serious developer is going to handle all that with ctypes??
>
> Please note that I do not have nothing against ctypes. It is a
> wonderful and very good tool, but you cannot use it in all scenarios.
> However, if the libraries are all C functions, and they do not use
> macros in any way, then ctypes is still a good option, no compilation
> needed, pure python code. Of course, be prepared to get a segfault
> from time to time, not because of a ctypes bug, but because of your
> bugs.

Very good summary. Also, I would like to note that if you're actually  
using (rather than just wrapping) a C library then ctypes will be  
vastly slower.

- Robert




From robertwb at math.washington.edu  Mon Jun  2 21:22:27 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 2 Jun 2008 12:22:27 -0700
Subject: [Cython] Cython tutorial at Scipy 2008?
In-Reply-To: <db6b5ecc0806011746s739bff3ci1cf2b65c8df403fd@mail.gmail.com>
References: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>
	<85e81ba30806011323h3c7bac75ua046616e5f6d54a8@mail.gmail.com>
	<db6b5ecc0806011746s739bff3ci1cf2b65c8df403fd@mail.gmail.com>
Message-ID: <3C70C136-EBE6-432E-80BF-FCEA66DBDEFD@math.washington.edu>

On Jun 1, 2008, at 5:46 PM, Fernando Perez wrote:

> On Sun, Jun 1, 2008 at 1:23 PM, William Stein <wstein at gmail.com>  
> wrote:
>> On Sun, Jun 1, 2008 at 1:18 PM, Fernando Perez  
>> <fperez.net at gmail.com> wrote:
>>> Hi folks,
>>>
>>> I'm in charge of co-organizing (with Travis Oliphant) the tutorial
>>> sessions at Scipy 2008, to be held in August of this year at  
>>> Caltech:
>>>
>>> http://www.scipy.org/SciPy2008
>>> http://www.scipy.org/SciPy2008/Tutorials
>>>
>>> Any chance someone from the Cython gang might come and be willing to
>>> give a cython tutorial?  We don't have  too many details sorted out
>>> yet, so we're just feeling things  out...
>>>
>>
>> Robert Bradshaw and I will likely both be in nearby San Diego then,
>> in which case one or both of us could potentially come.
>
> Great.  I had also listed a Sage tutorial as a potential topic, so as
> long as some of you are around, perhaps you could pitch in for that
> too?  Right now we're just getting feelers for both who could teach
> what, and what people want to see.  It's a bit of a matchmaking game,
> but there's no point in offering something if we don't have anyone to
> teach it.
>
> Note that in addition to the 2 days of tutorials and 2 days of
> conference, the weekend is reserved for dev sprints.  It would be very
> nice to have some joint work on sage/cython/numpy/scipy take place.
> Many of the usual suspects will be there.

Yes, I'll be around and would be willing to help out with something  
like that.

- Robert



From robertwb at math.washington.edu  Mon Jun  2 21:43:52 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 2 Jun 2008 12:43:52 -0700
Subject: [Cython] Py3K: recent rename PyString -> PyBytes
In-Reply-To: <48418AFE.2080303@behnel.de>
References: <e7ba66e40805261204q633b8242o8eaf866783f967ac@mail.gmail.com>	<483B26B0.6060409@behnel.de>	<e7ba66e40805270643l2292a50aj1a52afa08a2c53fb@mail.gmail.com>	<483E536F.2080005@behnel.de>	<e7ba66e40805290806n31715ad9n8fdbaa2a220f78c3@mail.gmail.com>	<483ECBE3.6010200@behnel.de>	<e7ba66e40805290913k877851cr4914b0fd2160f38c@mail.gmail.com>	<D8FCB956-288E-4E43-85CE-CDF082B9FA24@math.washington.edu>	<483F0D0F.2080002@behnel.de>	<F159489B-2B95-4145-9E75-1A8803A35BDF@math.washington.edu>	<483FE3D8.7020203@behnel.de>
	<1D4FE830-2596-4050-8FE4-4F5753893A57@math.washington.edu>
	<48418AFE.2080303@behnel.de>
Message-ID: <322897BC-78CB-484D-BF55-EE3EB15AFBB6@math.washington.edu>

On May 31, 2008, at 10:29 AM, Stefan Behnel wrote:

> Hi,
>
> Robert Bradshaw wrote:
>> On May 30, 2008, at 4:24 AM, Stefan Behnel wrote:
>>> Also, code like
>>>
>>>                 code.putln('"%s",' % name)
>>>
>>> may not work in all cases. If we can't be sure it's a pure ASCII
>>> identifier
>>> (i.e. it was parsed as an identifier by the current Cython parser),
>>> the name
>>> has to be encoded as UTF-8 first and escaped using
>>> Utils.escape_byte_string().
>>
>> On this note, what happens when a cdef variable (like above) has non-
>> ASCII characters in it?
>
> Not sure what you mean here.
>
> Do you mean in its name? That can't currently happen as the scanner  
> only
> allows pure ASCII alphanumeric identifiers.

Yep, that's what I meant (and, in fact, that's what the code above is  
using).

> I don't think we should support
> PEP 3131 in Cython, the mapping to C identifiers would become too  
> obscure.
>
> http://www.python.org/dev/peps/pep-3131/

Anything we don't support is a bug.

> Keyword arguments are a different thing, but allowing non-ASCII  
> keywords in
> function signatures will require us to write our own  
> ParseTupleAndKeywords()
> to keep up compatibility with Py2, so I don't find that worth  
> supporting
> either (for now). Although having our own PTAK() where you could  
> simply pass
> Python strings as acceptable keyword list would be helpful in  
> general...

We wouldn't have to back-port this to Py2, it would be an error in  
this case (maybe at C compile time there would an error raised if non- 
ascii identifiers are used).

- Robert


From robertwb at math.washington.edu  Mon Jun  2 21:46:22 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 2 Jun 2008 12:46:22 -0700
Subject: [Cython] Problems with example
In-Reply-To: <FE08AFB9-0731-4A12-8A30-782AAF98F252@mac.com>
References: <A7131774-1FEE-4588-9E0A-603122937CF9@mac.com>
	<FE08AFB9-0731-4A12-8A30-782AAF98F252@mac.com>
Message-ID: <841A23EA-3C4D-4D96-989A-773D855D038F@math.washington.edu>

On May 31, 2008, at 8:43 AM, Tommy Grav wrote:

> I found the solution
>
> gcc -dynamiclib -undefined dynamic_lookup -single_module -o c1.so c1.o
>
> Note that python will not find libraries called *.dylib in your  
> work directory,
> while *.so works fine. Does any know why?

Glad you were able to figure out a solution. No idea why .so files  
are required--this sounds like a Python issue so perhaps someone  
there would know.

>
> Cheers
>   Tommy
>
> On May 30, 2008, at 9:22 PM, Tommy Grav wrote:
>
>> Hi everyone.
>>
>> I am on a Powerbook running 10.5.3 and trying to go through the  
>> example on
>> http://www.perrygeo.net/wordpress/?p=116 , which fails. Can anyone
>> tell me what I am doing wrong?
>>
>> I create the c1.pxd
>> and run
>>
>> cython c1.pyx
>> gcc -c -fPIC -I/usr/include/python2.5 cl.c
>> gcc -dynamiclib -o c1.dylib -dylib c1.o
>> Undefined symbols:
>>   "_PyDict_New", referenced from:
>>       ___Pyx_Import in c1.o
>>
>> ld: symbol(s) not found
>> collect2: ld returned 1 exit status
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From king at mathematik.uni-jena.de  Mon Jun  2 21:51:31 2008
From: king at mathematik.uni-jena.de (Simon King)
Date: Mon, 2 Jun 2008 21:51:31 +0200 (CEST)
Subject: [Cython] Problem with cdef long
Message-ID: <Pine.LNX.4.64.0806022141180.19015@mpc739.mati.uni-jena.de>

Dear Cython team,

hopefully the following really is a cython-question, not a Sage-question.

Write a file Problem.pyx:

ctypedef struct Term_t:
    long coef

cdef class Term:
    cdef Term_t Data
    def __init__(self, c):
        self.Data.coef = c
    def coefficient(self):
        return self.Data.coef


Start Sage and do
 sage: attach Problem.pyx
 sage: T=Term(3)
 sage: type(T.coefficient())

Then the result is 
 <type 'int'>
and *not* <type 'long'>!

Why is it of type 'int' although coef is defined 'long' in Term_t? How can 
i avoid this automatic down-grading of coef?

I really want coef to be of type 'long', since 'int' isn't good enough in 
my application.

Yours
	Simon
 

From fperez.net at gmail.com  Mon Jun  2 21:51:02 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 2 Jun 2008 12:51:02 -0700
Subject: [Cython] Cython tutorial at Scipy 2008?
In-Reply-To: <3C70C136-EBE6-432E-80BF-FCEA66DBDEFD@math.washington.edu>
References: <db6b5ecc0806011318w155fe4b2l457e43015ab2fe6a@mail.gmail.com>
	<85e81ba30806011323h3c7bac75ua046616e5f6d54a8@mail.gmail.com>
	<db6b5ecc0806011746s739bff3ci1cf2b65c8df403fd@mail.gmail.com>
	<3C70C136-EBE6-432E-80BF-FCEA66DBDEFD@math.washington.edu>
Message-ID: <db6b5ecc0806021251s5a97794brc997d605930fce2e@mail.gmail.com>

On Mon, Jun 2, 2008 at 12:22 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:

> Yes, I'll be around and would be willing to help out with something
> like that.

Great, I've put you guys in as potential presenters.  I'll let you
know once I have more details.

Cheers,

f

From robertwb at math.washington.edu  Mon Jun  2 22:00:32 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 2 Jun 2008 13:00:32 -0700
Subject: [Cython] Problem with cdef long
In-Reply-To: <Pine.LNX.4.64.0806022141180.19015@mpc739.mati.uni-jena.de>
References: <Pine.LNX.4.64.0806022141180.19015@mpc739.mati.uni-jena.de>
Message-ID: <7DB5F9C2-332B-4C3D-ACC6-A18B8576772D@math.washington.edu>

On Jun 2, 2008, at 12:51 PM, Simon King wrote:

> Dear Cython team,
>
> hopefully the following really is a cython-question, not a Sage- 
> question.
>
> Write a file Problem.pyx:
>
> ctypedef struct Term_t:
>     long coef
>
> cdef class Term:
>     cdef Term_t Data
>     def __init__(self, c):
>         self.Data.coef = c
>     def coefficient(self):
>         return self.Data.coef
>
>
> Start Sage and do
>  sage: attach Problem.pyx
>  sage: T=Term(3)
>  sage: type(T.coefficient())
>
> Then the result is
>  <type 'int'>
> and *not* <type 'long'>!
>
> Why is it of type 'int' although coef is defined 'long' in Term_t?

I think your confusion here is over the difference between C ints/ 
longs/etc. and Python ints/longs. The Python "int" type is a Python  
object that wraps a C long. Python "long" type is an arbitrary- 
precision integer. Your Term.coefficient function returns a Python  
object, so it takes the self.Data.coef (which is a C long) and wraps  
it in a Python object (of type "int").

> How can i avoid this automatic down-grading of coef?
>
> I really want coef to be of type 'long', since 'int' isn't good  
> enough in
> my application.

When you say "int" isn't good enough, do you mean you need arbitrary  
precision? Because there isn't a (simple) C type that will give you  
that (you would have to use mpz_t or something like that).

- Robert


From stefan_ml at behnel.de  Tue Jun  3 13:48:15 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 3 Jun 2008 13:48:15 +0200 (CEST)
Subject: [Cython] Next release
Message-ID: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Hi,

the Python betas have been postponed by a week, to June 11:

http://permalink.gmane.org/gmane.comp.python.python-3000.devel/13681

That's close to dev1, however, once "cython-release" is updated and
released, we can always apply selected changes to it to keep releasing
micro versions from there if we notice any incompatible changes in
Py3/2.6. So we don't really have to wait for the final beta releases.

I wouldn't mind a release next monday.

BTW, what's the next version anyway? Has anyone taken a look at the latest
Pyrex changes? Is there anything left that's worth merging in so that we
can release a true 0.9.8?

Stefan


From stefan_ml at behnel.de  Mon Jun  2 21:13:47 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 02 Jun 2008 21:13:47 +0200
Subject: [Cython] about let users have more control on garbage collection
In-Reply-To: <e7ba66e40806020758l7f64780cwf7ec46caf9bd8e25@mail.gmail.com>
References: <e7ba66e40805300740qb59c01nc60d72d07e44a276@mail.gmail.com>	<48418CC9.2030103@behnel.de>	<e7ba66e40805311449n2151aaf0s2266fb5d9344e9b1@mail.gmail.com>	<484291A3.5090909@behnel.de>
	<e7ba66e40806020758l7f64780cwf7ec46caf9bd8e25@mail.gmail.com>
Message-ID: <4844466B.8030406@behnel.de>

Hi,

Lisandro Dalcin wrote:
> On 6/1/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>> Can't you work with singleton proxies? That's what we do in lxml. There's
>>  never more than one Python Element proxy for an XML node struct.
> 
> I'll take a look, but I believe such approach, even if possible, would
> complicate a lot my implementation.

Not sure. A factory function for creating proxies is a good idea in general,
and once that's in place, you can do all sorts of weird things in there.


> Stefan, iff it is no much work for you, could you point me a link to
> the actual lxml code implementing all this machinery?

Ah, asking about the deep magic, right? ;)

Here's the _elementFactory() function:

http://codespeak.net/svn/lxml/trunk/src/lxml/lxml.etree.pyx

and a bit more of the proxy machinery is in here:

http://codespeak.net/svn/lxml/trunk/src/lxml/proxy.pxi

But that truly is black magic...

Stefan


From robertwb at math.washington.edu  Tue Jun  3 23:13:53 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 3 Jun 2008 14:13:53 -0700
Subject: [Cython] Next release
In-Reply-To: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: <18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>

On Jun 3, 2008, at 4:48 AM, Stefan Behnel wrote:

> Hi,
>
> the Python betas have been postponed by a week, to June 11:
>
> http://permalink.gmane.org/gmane.comp.python.python-3000.devel/13681

:-( Oh well...

> That's close to dev1, however, once "cython-release" is updated and
> released, we can always apply selected changes to it to keep releasing
> micro versions from there if we notice any incompatible changes in
> Py3/2.6. So we don't really have to wait for the final beta releases.

Sure.

> I wouldn't mind a release next monday.

Sounds like a good target to me.

> BTW, what's the next version anyway? Has anyone taken a look at the  
> latest
> Pyrex changes? Is there anything left that's worth merging in so  
> that we
> can release a true 0.9.8?

I looked at all the Pyrex changes and incorporated everything we  
didn't already have except for the GIL and dependancy tracking stuff.  
Someone who actually deals a lot with multi-threaded code should  
probably comment on the GIL stuff, and the dependancy tracking stuff  
looks mostly good though I'd like to know what you think before  
blindly merging it over.

- Robert


From stefan_ml at behnel.de  Tue Jun  3 23:06:46 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 03 Jun 2008 23:06:46 +0200
Subject: [Cython] Py3K: recent rename PyString -> PyBytes
In-Reply-To: <48418AFE.2080303@behnel.de>
References: <e7ba66e40805261204q633b8242o8eaf866783f967ac@mail.gmail.com>	<483B26B0.6060409@behnel.de>	<e7ba66e40805270643l2292a50aj1a52afa08a2c53fb@mail.gmail.com>	<483E536F.2080005@behnel.de>	<e7ba66e40805290806n31715ad9n8fdbaa2a220f78c3@mail.gmail.com>	<483ECBE3.6010200@behnel.de>	<e7ba66e40805290913k877851cr4914b0fd2160f38c@mail.gmail.com>	<D8FCB956-288E-4E43-85CE-CDF082B9FA24@math.washington.edu>	<483F0D0F.2080002@behnel.de>	<F159489B-2B95-4145-9E75-1A8803A35BDF@math.washington.edu>	<483FE3D8.7020203@behnel.de>	<1D4FE830-2596-4050-8FE4-4F5753893A57@math.washington.edu>
	<48418AFE.2080303@behnel.de>
Message-ID: <4845B266.4040905@behnel.de>

Hi,

Robert Bradshaw wrote:
> Stefan Behnel wrote:
>>> On this note, what happens when a cdef variable (like above) has non- 
>>> ASCII characters in it?
>> 
>> Do you mean in its name? That can't currently happen as the scanner only
>> allows pure ASCII alphanumeric identifiers.
>
> Yep, that's what I meant (and, in fact, that's what the code above is
> using).

I know, I just wanted to mention it to be sure you're aware of it. If you say
not supporting PEP 3131 is a bug, then even this may stop working one day.

I was actually considering opening C source output files with a plain ASCII
codec, just to make sure we get an error if we accidentally output any
non-ASCII characters into the C source. I couldn't do that yet, one reason
being that we currently write names as unicode strings but string literals as
encoded byte strings. That will have to be fixed when migrating Cython itself
to Py3, which is strict about the type of output streams (text vs. byte streams).

I consider encoding at the very end to be the right thing to do, even more so
the more unicode support we enable in various places. Maybe the CCodeWriter
should get dedicated methods for writing byte string literals, unicode string
literals and literal identifier names, so that we end up with a single place
to handle output encodings. You'd then pass in the source code input encoding
on creation and it would just do the right thing, depending on the method
through which the code content came in.

Or, instead of doing the string formatting outside of the call, we could pass
all values in as "*args" and do the formatting in the code writer, where we
could then convert EncodedString instances amongst the arguments, for example.

I'm not sure yet what's the right way of handling this...


>> Keyword arguments are a different thing, but allowing non-ASCII keywords in
>> function signatures will require us to write our own ParseTupleAndKeywords()
>> to keep up compatibility with Py2
>
> We wouldn't have to back-port this to Py2, it would be an error in
> this case (maybe at C compile time there would an error raised if non-
> ascii identifiers are used).

That's a good point. In Py2, keyword arguments *must* be byte strings.
Although ASCII is not enforced, you can only pass non-ASCII keywords using
"**dict" and accept them using "**kwargs", so it would be ok IMHO if we just
generated an "#if Py2 #error" directive when we find non-ASCII keywords in the
signature.

Stefan



From bblais at gmail.com  Wed Jun  4 15:41:00 2008
From: bblais at gmail.com (Brian Blais)
Date: Wed, 4 Jun 2008 09:41:00 -0400
Subject: [Cython] accessing an array of ints
Message-ID: <814FE4F9-2AEB-4FCA-A020-178062EAE2AD@gmail.com>

Hello,

If I have an array of ints, as in numpy:

a=numpy.zeros(5,int)

and I call a cython function like:

myfun(a)

and in the cython I have:

cpdef myfun(c_numpy.ndarray A):

    # stuff here


how can I access the data pointer?  I tried:

    cdef int *ap=<int *> A.data

but that doesn't seem to work (it works for double *, but not int  
*).  I must be missing something really simple.


          thanks,

              Brian Blais



From languitar at semipol.de  Wed Jun  4 16:57:53 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Wed, 04 Jun 2008 16:57:53 +0200
Subject: [Cython] char* and NULL in log statements
Message-ID: <4846AD71.50407@semipol.de>

Hi,

I've just noticed that using NULL values in char* is a bad idea when
converting them to a python string. This results in a segfault:

cdef void doSomething(char *string):
    print string

def doIt():
    cdef char* string = NULL
    doSomething(NULL)

*Wouldn't it be a good idea to automatically convert them to None?*

For example I have a lot of debug messages that simply indicate which
function is currently called and with which arguments. At the moment
this is always 4 extra lines to check for the NULL value, which could
even be semantically correct for the rest of that function:

cdef void doSomething(char *string):
    if string == NULL:
        printString = None
    else:
        printString = string
    print printString
    # now I want to use NULL...

def doIt():
    cdef char* string = NULL
    doSomething(NULL)

Even if I use a custom function for this (which isn't easy to manage in
formatted strings), this will be or is a common cause for bugs. I don't
want to know the times that I have forgotten to check this...

- Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080604/9a939503/attachment.pgp 

From dagss at student.matnat.uio.no  Wed Jun  4 17:48:51 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Jun 2008 17:48:51 +0200
Subject: [Cython] accessing an array of ints
In-Reply-To: <814FE4F9-2AEB-4FCA-A020-178062EAE2AD@gmail.com>
References: <814FE4F9-2AEB-4FCA-A020-178062EAE2AD@gmail.com>
Message-ID: <4846B963.3000301@student.matnat.uio.no>

Brian Blais wrote:
> Hello,
>
> If I have an array of ints, as in numpy:
>
> a=numpy.zeros(5,int)
>
> and I call a cython function like:
>
> myfun(a)
>
> and in the cython I have:
>
> cpdef myfun(c_numpy.ndarray A):
>
>     # stuff here
>
>
> how can I access the data pointer?  I tried:
>
>     cdef int *ap=<int *> A.data
>
> but that doesn't seem to work (it works for double *, but not int  
> *).  I must be missing something really simple.
>   
1) Be aware that the "int" is really two different types (the latter one 
being a C "int", the former an identifier for numpy). Better use 
"numpy.int32" for numpy.zeros and a datatype that you know is 32 bit int 
in the cdef. I think that what you have written will fail on 64- bit 
machines but I'm not sure (could imagine that NumPy would choose int64 
as the default "int", while the C compiler would stick with 32 bit 
*shrug*). There are some npy_int32 etc. typedefs/defines in the NumPy 
header files that are better used for C types rather than plain "int".

2) It will still only work if you have exactly the array created by zero 
above. I.e., if you instead pass something like

numpy.zeros(10, int)[::-2]

for instance, then the buffer isn't modified -- instead, something 
called "strides" is set in the struct which tells you to iterator "-2" 
for every step along the 1st dimension. Same goes for 
numpy.zeros((5,5,5), int)[1,:,2] and so on.

So
a) Either make absolutely sure you have a C-contiguous array
or
b) Assign ap as you do, but access it like this:

a[i * A.strides[0]]

rather than a[i]. (a[i*A.strides[0] + j*A.strides[1]] will give you 2D 
access and so on)

Does this answer your question?

Dag Sverre

From stefan_ml at behnel.de  Wed Jun  4 19:00:10 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jun 2008 19:00:10 +0200
Subject: [Cython] remaining Pyrex changes (was: Next release)
In-Reply-To: <18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
Message-ID: <4846CA1A.2060706@behnel.de>

Hi,

Robert Bradshaw wrote:
> I looked at all the Pyrex changes and incorporated everything we  
> didn't already have

Nice! I noticed you added a couple of things, but I lost track of what was in
and what wasn't. Glad to hear most things are there already.


> except for the GIL and dependancy tracking stuff.
> Someone who actually deals a lot with multi-threaded code should  
> probably comment on the GIL stuff,

I'll take a look at it. At first glance, it seems to be mostly bug fixes and a
couple of simplifications, so I don't expect big surprises.


> and the dependancy tracking stuff  
> looks mostly good though I'd like to know what you think before  
> blindly merging it over.

That looks just fine to me. I did a mostly manual merge and it seems to work
for me.

I also merged (or rather reimplemented) the package resolution stuff that
Pyrex does when searching for include/cimport files. This implies that when
you cimport a .pxd from a package directory such as "package/module.pxd",
there must be a "package/__init__.py[x]" in the search path. Cython didn't
care about that before, but I think it makes sense to restrict package
directory imports to the way Python would do them. That said, both imports
from package directories and those from qualified dotted file names
("package.module.pxd") should work (lxml uses the latter).

Robert, I don't know what naming convention you use in Sage, so could you test
that the changes didn't break it?

Stefan


From stefan_ml at behnel.de  Wed Jun  4 19:47:08 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jun 2008 19:47:08 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846AD71.50407@semipol.de>
References: <4846AD71.50407@semipol.de>
Message-ID: <4846D51C.2030607@behnel.de>

Hi,

Johannes Wienke wrote:
> I've just noticed that using NULL values in char* is a bad idea when
> converting them to a python string.

Every pointer can potentially be NULL. If Cython added NULL checks to
everything that might be NULL, you'd get pretty inefficient code with loads of
expensive conditionals. You should take care to write robust code yourself.


> *Wouldn't it be a good idea to automatically convert them to None?*

What would be gained from having to check for None instead of having to check
for NULL? Checking for NULL, referencing None and then checking for None is
definitely more expensive than a straight check for NULL in your code.

If you *really* want None, then you can use something like this:

    cdef inline stringOrNone(char* value):
         if value is NULL: return None
         return value

and wrap all your char*->byte string conversions explicitly in a call to this
function. Remember, explicit is better than implicit.

Stefan


From languitar at semipol.de  Wed Jun  4 20:33:56 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Wed, 04 Jun 2008 20:33:56 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846D51C.2030607@behnel.de>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
Message-ID: <4846E014.5080300@semipol.de>

Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
> Johannes Wienke wrote:
>> I've just noticed that using NULL values in char* is a bad idea when
>> converting them to a python string.
> 
> Every pointer can potentially be NULL. If Cython added NULL checks to
> everything that might be NULL, you'd get pretty inefficient code with loads of
> expensive conditionals. You should take care to write robust code yourself.

To my mind only char pointers would need this extra behavior as they
have a somewhat special role in C because of the absence a string type.

>> *Wouldn't it be a good idea to automatically convert them to None?*
> 
> What would be gained from having to check for None instead of having to check
> for NULL? Checking for NULL, referencing None and then checking for None is
> definitely more expensive than a straight check for NULL in your code.

None is automatically converted to Python strings, so you can use it in
every string or print statement without troubles. This conversion should
also only happen if the char* needs to be converted to a python string.
Than it is in your own control to do this. In all cases where the string
is not NULL this would only cost one little comparison with NULL.

Moreover I think it's more desirable to have code that is less error
prone (a program dying because of segfault is a serious bug) than the
tiny speed up by wasting _one_ comparison with NULL. As someone else on
this list said before: cython is all about writing C without writing
C... And if I am not writing C, or at least try to minimize the part, I
don't want to waste my time with ugly C memory management stuff.

> If you *really* want None, then you can use something like this:
> 
>     cdef inline stringOrNone(char* value):
>          if value is NULL: return None
>          return value

That's exactly what I'm doing now but that's error-prone as you have to
do this manually and can forget it.

- Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080604/be1436ce/attachment-0001.pgp 

From martin at martincmartin.com  Wed Jun  4 21:38:22 2008
From: martin at martincmartin.com (Martin C. Martin)
Date: Wed, 04 Jun 2008 15:38:22 -0400
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846D51C.2030607@behnel.de>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
Message-ID: <4846EF2E.1000605@martincmartin.com>



Stefan Behnel wrote:
> Hi,
> 
> Johannes Wienke wrote:
>> I've just noticed that using NULL values in char* is a bad idea when
>> converting them to a python string.
> 
> Every pointer can potentially be NULL. If Cython added NULL checks to
> everything that might be NULL, you'd get pretty inefficient code with loads of
> expensive conditionals. You should take care to write robust code yourself.

Actually, one of the lessons of JVM optimizations is that NULL checks 
are only a single instruction when they're not non-null.  You can tell 
the compiler to predict that the "non-null" will be taken, and when it 
is, there's no branch penalty.

However, I agree that it's against the spirit of C/C++, and hence 
Cython, to automatically check all pointers for null.

>> *Wouldn't it be a good idea to automatically convert them to None?*
> 
> What would be gained from having to check for None instead of having to check
> for NULL? Checking for NULL, referencing None and then checking for None is
> definitely more expensive than a straight check for NULL in your code.
> 
> If you *really* want None, then you can use something like this:
> 
>     cdef inline stringOrNone(char* value):
>          if value is NULL: return None
>          return value
> 
> and wrap all your char*->byte string conversions explicitly in a call to this
> function. Remember, explicit is better than implicit.
> 
> Stefan
> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

From dalcinl at gmail.com  Wed Jun  4 21:45:52 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Jun 2008 16:45:52 -0300
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846E014.5080300@semipol.de>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de>
Message-ID: <e7ba66e40806041245y40f4490h7d8ea6f14d9e155c@mail.gmail.com>

Johannes, I think you are right. The char* -> str conversion should
return None if the pointer is NULL. This is even consistent with the
form Py_BuildValue() works in Python C API.

As a side note, the mapping of NULL pointers to None is what SWIG does
in almost all cases. I would not follow that approach in Cython for
all cases, but the char* -> str is special enough.

Stefan, are you completelly sure that the performance implications of
checking for NULL pointers is this case are noticeable enough as to do
not follow the safe path? IMHO, I would avoid the chances for
segfaults.


On 6/4/08, Johannes Wienke <languitar at semipol.de> wrote:
> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>
> > Johannes Wienke wrote:
>  >> I've just noticed that using NULL values in char* is a bad idea when
>  >> converting them to a python string.
>  >
>  > Every pointer can potentially be NULL. If Cython added NULL checks to
>  > everything that might be NULL, you'd get pretty inefficient code with loads of
>  > expensive conditionals. You should take care to write robust code yourself.
>
>
> To my mind only char pointers would need this extra behavior as they
>  have a somewhat special role in C because of the absence a string type.
>
>
>  >> *Wouldn't it be a good idea to automatically convert them to None?*
>  >
>  > What would be gained from having to check for None instead of having to check
>  > for NULL? Checking for NULL, referencing None and then checking for None is
>  > definitely more expensive than a straight check for NULL in your code.
>
>
> None is automatically converted to Python strings, so you can use it in
>  every string or print statement without troubles. This conversion should
>  also only happen if the char* needs to be converted to a python string.
>  Than it is in your own control to do this. In all cases where the string
>  is not NULL this would only cost one little comparison with NULL.
>
>  Moreover I think it's more desirable to have code that is less error
>  prone (a program dying because of segfault is a serious bug) than the
>  tiny speed up by wasting _one_ comparison with NULL. As someone else on
>  this list said before: cython is all about writing C without writing
>  C... And if I am not writing C, or at least try to minimize the part, I
>  don't want to waste my time with ugly C memory management stuff.
>
>
>  > If you *really* want None, then you can use something like this:
>  >
>  >     cdef inline stringOrNone(char* value):
>  >          if value is NULL: return None
>  >          return value
>
>
> That's exactly what I'm doing now but that's error-prone as you have to
>  do this manually and can forget it.
>
>
>  - Johannes
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>
>
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Jun  4 21:54:12 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jun 2008 21:54:12 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846E014.5080300@semipol.de>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de>
Message-ID: <4846F2E4.302@behnel.de>

Hi,

Johannes Wienke wrote:
> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
> 
> To my mind only char pointers would need this extra behavior as they
> have a somewhat special role in C because of the absence a string type.

Then why None and not ''? And why None and not a ValueError?


> None is automatically converted to Python strings, so you can use it in
> every string or print statement without troubles.

Not on my side of the cable:

    Python 2.5.1 (r251:54863, Mar  7 2008, 04:10:12)
    [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> "abc" + None
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: cannot concatenate 'str' and 'NoneType' objects

And the following will still crash, even under your proposal:

    cdef char* s = NULL
    py_s = s
    print PyString_GET_SIZE(py_s)

And would you also want this to work:

    cdef char* s = None


> don't want to waste my time with ugly C memory management stuff.

This is not about memory management at all. This is about making sure your
code handles corner cases correctly. Where does the NULL pointer come from
anyway? Is it maybe an error return of a function that you didn't catch? You
wouldn't want Cython to find your off-by-one errors also, would you?


>> If you *really* want None, then you can use something like this:
>>
>>     cdef inline stringOrNone(char* value):
>>          if value is NULL: return None
>>          return value
> 
> That's exactly what I'm doing now but that's error-prone as you have to
> do this manually and can forget it.

Why? Isn't it good practice to guard your code against anticipated invalid
data? Don't you ask yourself from time to time when you write Python code:
"can this variable be None?" So why not ask "can this pointer be NULL" in Cython?

I think it's an advantage that you get a straight crash when your code
contains the obvious bug that you forgot to test a NULL pointer for a NULL
value, instead of an automatic coercing to None, which might propagate through
your application and induce hard to track down bugs elsewhere in your program.
Usually, C value handling happens within a limited scope (e.g. a function),
while Python values tend to cross API boundaries into user code.

Stefan


From dalcinl at gmail.com  Wed Jun  4 22:08:42 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Jun 2008 17:08:42 -0300
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846F2E4.302@behnel.de>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de> <4846F2E4.302@behnel.de>
Message-ID: <e7ba66e40806041308p36f88499we48c728c2dcea1a5@mail.gmail.com>

On 6/4/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Then why None and not ''? And why None and not a ValueError?

Well, I would not object ValueError. That would be far better than a segfault.

>  > None is automatically converted to Python strings, so you can use it in
>  > every string or print statement without troubles.
>
> Not on my side of the cable:
>     TypeError: cannot concatenate 'str' and 'NoneType' objects

I believe Johannes was actually talking about str(None) -> "None"

>  And the following will still crash, even under your proposal:
>
>     cdef char* s = NULL
>     py_s = s
>     print PyString_GET_SIZE(py_s)

So now I definitely believe that the 'py_s = s' line should generate a
ValueError.

>  And would you also want this to work:
>
>     cdef char* s = None

I do not believe  Johannes asked for this. That should never be valid,
moreover, that should by a Cython compile-time syntax error (I believe
it is, right?)

>  > don't want to waste my time with ugly C memory management stuff.
> This is not about memory management at all. This is about making sure your
>  code handles corner cases correctly. Where does the NULL pointer come from
>  anyway? Is it maybe an error return of a function that you didn't catch? You
>  wouldn't want Cython to find your off-by-one errors also, would you?

So the correct way of handling it is, again, raise ValueError. A
segfault should never be an option.

>
>  I think it's an advantage that you get a straight crash when your code
>  contains the obvious bug that you forgot to test a NULL pointer for a NULL

I really do not see the advantage of a crash. Python has exceptions!!


In short, would it make sense to generate ValueError for  the char* ->
str conversion? Honestly, the performance issues are going to be
really small.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From languitar at semipol.de  Wed Jun  4 22:38:16 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Wed, 04 Jun 2008 22:38:16 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846F2E4.302@behnel.de>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de> <4846F2E4.302@behnel.de>
Message-ID: <4846FD38.3080106@semipol.de>

Hi,

Am 06/04/2008 09:54 PM schrieb Stefan Behnel:
> Johannes Wienke wrote:
>> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>>
>> To my mind only char pointers would need this extra behavior as they
>> have a somewhat special role in C because of the absence a string type.
> 
> Then why None and not ''? And why None and not a ValueError?

Because None has for python the same meaning as NULL in C. ValueError
would be another possibility. Nevertheless if NULL can be a legal value
for the rest of that function, this would be as awkward to handle as the
explicit check for NULL if I only want a safe print statement.

>> None is automatically converted to Python strings, so you can use it in
>> every string or print statement without troubles.
> 
> Not on my side of the cable:
> 
>     Python 2.5.1 (r251:54863, Mar  7 2008, 04:10:12)
>     [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> "abc" + None
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     TypeError: cannot concatenate 'str' and 'NoneType' objects

Maybe I wasn't precise enough. None can be displayed using str() and can
therefore be used without worries in format strings. Moreover a python
exception is a _much_ better error indicator than a segfault without any
stack trace.

> And the following will still crash, even under your proposal:
> 
>     cdef char* s = NULL
>     py_s = s
>     print PyString_GET_SIZE(py_s)

I've never used the python API, so I have no clue how it is done there.

> And would you also want this to work:
> 
>     cdef char* s = None

For my purpose that's not necessary but maybe someone else will need
this... Well, I don't know if that would be a good idea. Looks strange
to give an explicit C variable an explicit python value. But that's only
what I think at first sight.

>> don't want to waste my time with ugly C memory management stuff.
> 
> This is not about memory management at all. This is about making sure your
> code handles corner cases correctly. Where does the NULL pointer come from
> anyway? Is it maybe an error return of a function that you didn't catch? You
> wouldn't want Cython to find your off-by-one errors also, would you?

If I access memory that is not memory I own or that points to somewhere
undefined this is a kind of memory management, to my mind. Maybe that's
a strong personal bias as my preferred language is Java. But in pure
Python there are also rarely any cases where you have to worry about
directly working on the memory.

Of course I want to check for NULL values and I'm a fan of defensive
programming but life could be much easier if another source for
segfaults is removed. And that's absolutely in the spirit of defensive
programming (have a look at the chapter in Code Complete).

I could understand that this is a bad idea to implement if there are any
other reasons than speed. Is there any semantic problem that could arise
for current code? I can't think of anything.

>>> If you *really* want None, then you can use something like this:
>>>
>>>     cdef inline stringOrNone(char* value):
>>>          if value is NULL: return None
>>>          return value
>> That's exactly what I'm doing now but that's error-prone as you have to
>> do this manually and can forget it.
> 
> Why? Isn't it good practice to guard your code against anticipated invalid
> data? Don't you ask yourself from time to time when you write Python code:
> "can this variable be None?" So why not ask "can this pointer be NULL" in Cython?

Of course this good practice and I totally agree that you have to guard
code from potential error values. But this is some extra work you have
to do as the programmer and programmers make mistakes and forget things
and so on... So it would be really defensive if the language helps
programmers to avoid such bugs. Especially because they result in ugly
segfaults. And as Cython is about annotating python with C features I
would claim that the main thinking is done in python and there it
wouldn't be such a problem to use None.

For example this works like a charm in python and is a typical logger
pattern:

def foo(maybeNone):
    logger.debug("Calling foo, argument: '%s'." % maybeNone)
    # other stuff

> I think it's an advantage that you get a straight crash when your code
> contains the obvious bug that you forgot to test a NULL pointer for a NULL
> value, instead of an automatic coercing to None, which might propagate through
> your application and induce hard to track down bugs elsewhere in your program.
> Usually, C value handling happens within a limited scope (e.g. a function),
> while Python values tend to cross API boundaries into user code.

But what is a straight crash useful for, if it gives you nearly no cue
where the error occurred? A python stack trace is so much more useful,
even if it is much longer. And maybe it's even more robust as the error
may somehow be corrected a few levels above the actually corrupt
function call.


Just as another thought I can give you some background why I have to
constantly log method calls:

For my project I provide an already existing API to old plugins written
in C. So I have simply no control about how they call my API. For some
of these plugins I even don't have the source code. The easiest way to
track the control flow is to log method calls that print out the
arguments the given to the function.

- Johanne

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080604/83978cc9/attachment.pgp 

From languitar at semipol.de  Wed Jun  4 22:45:12 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Wed, 04 Jun 2008 22:45:12 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <e7ba66e40806041308p36f88499we48c728c2dcea1a5@mail.gmail.com>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de> <4846F2E4.302@behnel.de>
	<e7ba66e40806041308p36f88499we48c728c2dcea1a5@mail.gmail.com>
Message-ID: <4846FED8.3050303@semipol.de>

Hi,

Am 06/04/2008 10:08 PM schrieb Lisandro Dalcin:
> In short, would it make sense to generate ValueError for  the char* ->
> str conversion? Honestly, the performance issues are going to be
> really small.

Just take a look at my reply to Stefan. In summary: ValueError to my
mind is hard to use as I can do a lot of stuff with None in Python. I
don't think that's an error case. In C, NULL is a special case exactly
as None is in python. So why not use None?

- Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080604/6204b919/attachment.pgp 

From dagss at student.matnat.uio.no  Wed Jun  4 23:36:00 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Jun 2008 23:36:00 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846FD38.3080106@semipol.de>
References: <4846AD71.50407@semipol.de>
	<4846D51C.2030607@behnel.de>	<4846E014.5080300@semipol.de>
	<4846F2E4.302@behnel.de> <4846FD38.3080106@semipol.de>
Message-ID: <48470AC0.7040901@student.matnat.uio.no>

Johannes Wienke wrote:
> Hi,
> 
> Am 06/04/2008 09:54 PM schrieb Stefan Behnel:
>> Johannes Wienke wrote:
>>> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>>>
>>> To my mind only char pointers would need this extra behavior as they
>>> have a somewhat special role in C because of the absence a string type.
>> Then why None and not ''? And why None and not a ValueError?
> 
> Because None has for python the same meaning as NULL in C. ValueError
> would be another possibility. Nevertheless if NULL can be a legal value
> for the rest of that function, this would be as awkward to handle as the
> explicit check for NULL if I only want a safe print statement.

I have to agree with Stefan (not for performance reasons, but for not 
bogging down the language with too much magic). In Cython you do have to 
care about whether you are dealing with a C pointer or a Python object.

Blurring it in this specific case doesn't seem like a good idea, to me 
it seems to simply encourage bad habits. (For instance, what kind of 
behaviour are you assuming for non-ASCII values in that automatic char* 
to str conversion?)

char* is usually used to call into legacy C code, if you need to print 
them I'd argue that in most cases you are converting to char* one step 
too early. But if you really need to print char* directly,

print "%s ... %s" % (cb2str(a), cb2str(b))

is *a lot* better than % (a, b) because

a) it is explicit what's going on (conversion of a byte buffer to a 
string is *not* trivial and should not be transparent anyway)
b) you can change cb2str to use the charset you are assuming
c) your code can be made to work in Python 3
d) A function taking a char* may very well be taking unprintable 
characters, perhaps cb2str should really hex-encode the data instead

and so on.

Raising ValueError (or similar) as a generic feature for "any C pointer 
which coerces automatically to Python objects" (which currently is char* 
only?) seems like an good feature (though depending on a decision being 
reached on the char* coercion business then that might be deprecated 
anyway, and then it may not be worth the effort).

However, Cython cannot be Java because C is not Java. It will always be 
possible to do (<int*>0)[0] and segfault; that's part of the deal when 
writing C.

Cython can very well work safely; simply use Python strings!; avoid all 
pointers, etc. Java doesn't have a char* either. The Cython equivalent 
to java.lang.String is a Python str, not a char*!

-- 
Dag Sverre

From languitar at semipol.de  Wed Jun  4 23:42:37 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Wed, 04 Jun 2008 23:42:37 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <48470AC0.7040901@student.matnat.uio.no>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de> <4846F2E4.302@behnel.de>
	<4846FD38.3080106@semipol.de>
	<48470AC0.7040901@student.matnat.uio.no>
Message-ID: <48470C4D.5060605@semipol.de>

Hi,

Am 06/04/2008 11:36 PM schrieb Dag Sverre Seljebotn:
> Johannes Wienke wrote:
>> Hi,
>>
>> Am 06/04/2008 09:54 PM schrieb Stefan Behnel:
>>> Johannes Wienke wrote:
>>>> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>>>>
>>>> To my mind only char pointers would need this extra behavior as they
>>>> have a somewhat special role in C because of the absence a string type.
>>> Then why None and not ''? And why None and not a ValueError?
>> Because None has for python the same meaning as NULL in C. ValueError
>> would be another possibility. Nevertheless if NULL can be a legal value
>> for the rest of that function, this would be as awkward to handle as the
>> explicit check for NULL if I only want a safe print statement.
 >
> char* is usually used to call into legacy C code, if you need to print 
> them I'd argue that in most cases you are converting to char* one step 
> too early. But if you really need to print char* directly,

Well but for my purposes the legacy code calls into cython.

> print "%s ... %s" % (cb2str(a), cb2str(b))

What is that function? I have never seen that before?

Conversion problems are of course a much better reason not to implement 
this than speed reasons.

[...]

> However, Cython cannot be Java because C is not Java. It will always be 
> possible to do (<int*>0)[0] and segfault; that's part of the deal when 
> writing C.

That's true. But that's a much more obvious bug.

> Cython can very well work safely; simply use Python strings!; avoid all 
> pointers, etc. Java doesn't have a char* either. The Cython equivalent 
> to java.lang.String is a Python str, not a char*!

But I can't, because the pugins I have to wrap work in C and there may 
be other situations where an existing C API requires callbacks and so on.

- Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080604/d66957fb/attachment.pgp 

From dagss at student.matnat.uio.no  Wed Jun  4 23:55:43 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Jun 2008 23:55:43 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <48470C4D.5060605@semipol.de>
References: <4846AD71.50407@semipol.de>
	<4846D51C.2030607@behnel.de>	<4846E014.5080300@semipol.de>
	<4846F2E4.302@behnel.de>	<4846FD38.3080106@semipol.de>	<48470AC0.7040901@student.matnat.uio.no>
	<48470C4D.5060605@semipol.de>
Message-ID: <48470F5F.4010209@student.matnat.uio.no>

Johannes Wienke wrote:
> Hi,
> 
> Am 06/04/2008 11:36 PM schrieb Dag Sverre Seljebotn:
>> Johannes Wienke wrote:
>>> Hi,
>>>
>>> Am 06/04/2008 09:54 PM schrieb Stefan Behnel:
>>>> Johannes Wienke wrote:
>>>>> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>>>>>
>>>>> To my mind only char pointers would need this extra behavior as they
>>>>> have a somewhat special role in C because of the absence a string 
>>>>> type.
>>>> Then why None and not ''? And why None and not a ValueError?
>>> Because None has for python the same meaning as NULL in C. ValueError
>>> would be another possibility. Nevertheless if NULL can be a legal value
>>> for the rest of that function, this would be as awkward to handle as the
>>> explicit check for NULL if I only want a safe print statement.
>  >
>> char* is usually used to call into legacy C code, if you need to print 
>> them I'd argue that in most cases you are converting to char* one step 
>> too early. But if you really need to print char* directly,
> 
> Well but for my purposes the legacy code calls into cython.
> 
>> print "%s ... %s" % (cb2str(a), cb2str(b))
> 
> What is that function? I have never seen that before?

I just renamed what Stefan already provided you (I think you may have 
missed his point though, go reread his first post). Here, I'll 
reimplement it for you:

cdef inline object cb2str(char* bytebuf):
     if bytebuf == NULL:
         return "None"
     else:
         return bytebuf

Then, for Python 3 compatability and nice logging of any binary data you 
are passed, you can switch to

cdef inline object cb2str(char* bytebuf):
     if bytebuf == NULL:
         return u"None"
     else:
         try:
             return unicode(bytebuf, "iso-8859-1")
         except:
             # have some code to return a hex-formatted string instead,
             # as the char* doesn't contain latin data...

And so on. The point is: Outputting a char* to a logfile is a 
potentially complex task with associated business logic. Stick it in a 
function.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed Jun  4 23:59:42 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 04 Jun 2008 23:59:42 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4846E014.5080300@semipol.de>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de>
Message-ID: <4847104E.6010003@student.matnat.uio.no>

Johannes Wienke wrote:
> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>> If you *really* want None, then you can use something like this:
>>
>>     cdef inline stringOrNone(char* value):
>>          if value is NULL: return None
>>          return value
> 
> That's exactly what I'm doing now but that's error-prone as you have to
> do this manually and can forget it.

(Sorry about telling you to reread, I can see that you commented on it 
already.)

Well, in my mind, this is a reason for supporting Stefan in removing 
auto-coercion of char* to Python strings altogether (that is suggested 
once down in those unicode discussion threads, right Stefan?). Then you 
would get a nice compiler error when you forget it, and it won't be 
error-prone.

-- 
Dag Sverre

From languitar at semipol.de  Thu Jun  5 00:02:53 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Thu, 05 Jun 2008 00:02:53 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4847104E.6010003@student.matnat.uio.no>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de>
	<4847104E.6010003@student.matnat.uio.no>
Message-ID: <4847110D.1090101@semipol.de>

Am 06/04/2008 11:59 PM schrieb Dag Sverre Seljebotn:
> Johannes Wienke wrote:
>> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>>> If you *really* want None, then you can use something like this:
>>>
>>>     cdef inline stringOrNone(char* value):
>>>          if value is NULL: return None
>>>          return value
>> That's exactly what I'm doing now but that's error-prone as you have to
>> do this manually and can forget it.
> 
> (Sorry about telling you to reread, I can see that you commented on it 
> already.)
> 
> Well, in my mind, this is a reason for supporting Stefan in removing 
> auto-coercion of char* to Python strings altogether (that is suggested 
> once down in those unicode discussion threads, right Stefan?). Then you 
> would get a nice compiler error when you forget it, and it won't be 
> error-prone.

Now with the context of encoding issues I can see the point. Things you 
don't have to think of to often as Java developer. ;) Removing the 
conversion then would be a good idea. Compiler warnings are of course 
the best bug prevention. On the other hand this really handy... Hard to 
decide.

- Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080605/6d167a1b/attachment.pgp 

From dagss at student.matnat.uio.no  Thu Jun  5 00:06:26 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 05 Jun 2008 00:06:26 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <48470F5F.4010209@student.matnat.uio.no>
References: <4846AD71.50407@semipol.de>	<4846D51C.2030607@behnel.de>	<4846E014.5080300@semipol.de>	<4846F2E4.302@behnel.de>	<4846FD38.3080106@semipol.de>	<48470AC0.7040901@student.matnat.uio.no>	<48470C4D.5060605@semipol.de>
	<48470F5F.4010209@student.matnat.uio.no>
Message-ID: <484711E2.9000709@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Then, for Python 3 compatability and nice logging of any binary data you 
> are passed, you can switch to
> 
> cdef inline object cb2str(char* bytebuf):
>      if bytebuf == NULL:
>          return u"None"
>      else:
>          try:
>              return unicode(bytebuf, "iso-8859-1")
>          except:
>              # have some code to return a hex-formatted string instead,
>              # as the char* doesn't contain latin data...
> 
> And so on. The point is: Outputting a char* to a logfile is a 
> potentially complex task with associated business logic. Stick it in a 
> function.

To expand on that: For your purposes, outputting "None" is probably 
wrong anyway. What happens if the client library passes a char* 
containing the string "None"? How do you distinguish it?

I.e. something like this is probably better (cb means "charbuf" btw):

cdef inline object cbrepr(char* bytebuf):
     if bytebuf == NULL:
         return u'None'
     else:
         try:
             return u'"%s"' % unicode(bytebuf, "iso-8859-1")
         except:
             return u'binary data:%s' % your_hex_dump_func(bytebuf)


(But now we're straying rather far from Cython and into general 
programming...)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun  5 00:10:38 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 05 Jun 2008 00:10:38 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4847110D.1090101@semipol.de>
References: <4846AD71.50407@semipol.de>
	<4846D51C.2030607@behnel.de>	<4846E014.5080300@semipol.de>	<4847104E.6010003@student.matnat.uio.no>
	<4847110D.1090101@semipol.de>
Message-ID: <484712DE.9080904@student.matnat.uio.no>

Johannes Wienke wrote:
> Am 06/04/2008 11:59 PM schrieb Dag Sverre Seljebotn:
>> Johannes Wienke wrote:
>>> Am 06/04/2008 07:47 PM schrieb Stefan Behnel:
>>>> If you *really* want None, then you can use something like this:
>>>>
>>>>     cdef inline stringOrNone(char* value):
>>>>          if value is NULL: return None
>>>>          return value
>>> That's exactly what I'm doing now but that's error-prone as you have to
>>> do this manually and can forget it.
>>
>> (Sorry about telling you to reread, I can see that you commented on it 
>> already.)
>>
>> Well, in my mind, this is a reason for supporting Stefan in removing 
>> auto-coercion of char* to Python strings altogether (that is suggested 
>> once down in those unicode discussion threads, right Stefan?). Then 
>> you would get a nice compiler error when you forget it, and it won't 
>> be error-prone.
> 
> Now with the context of encoding issues I can see the point. Things you 
> don't have to think of to often as Java developer. ;) Removing the 
> conversion then would be a good idea. Compiler warnings are of course 
> the best bug prevention. On the other hand this really handy... Hard to 
> decide.

To further expand on this point (for the purposes on the ongoing 
month-long argument about char* behaviour on this mailing list): If you 
were doing the same thing in Java (i.e. interfacing with a C library), I 
can tell you that you *would* have to care about encoding issues :-) 
There's no way Java would have let you create a string without 
specifying the encoding somewhere.

-- 
Dag Sverre

From dalcinl at gmail.com  Thu Jun  5 01:36:38 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 4 Jun 2008 20:36:38 -0300
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <4847104E.6010003@student.matnat.uio.no>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de> <4847104E.6010003@student.matnat.uio.no>
Message-ID: <e7ba66e40806041636n7fa498e3s623c059b75d0149@mail.gmail.com>

On 6/4/08, Dag Sverre Seljebotn <dagss at student.matnat.uio.no> wrote:
>  Well, in my mind, this is a reason for supporting Stefan in removing
>  auto-coercion of char* to Python strings altogether (that is suggested
>  once down in those unicode discussion threads, right Stefan?).

Well, I believe that is the right approach. However, what would be the
way to generate a byte string from a char* pointer?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu Jun  5 04:06:54 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 05 Jun 2008 14:06:54 +1200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <e7ba66e40806041245y40f4490h7d8ea6f14d9e155c@mail.gmail.com>
References: <4846AD71.50407@semipol.de> <4846D51C.2030607@behnel.de>
	<4846E014.5080300@semipol.de>
	<e7ba66e40806041245y40f4490h7d8ea6f14d9e155c@mail.gmail.com>
Message-ID: <48474A3E.4050505@canterbury.ac.nz>

Lisandro Dalcin wrote:
> Stefan, are you completelly sure that the performance implications of
> checking for NULL pointers is this case are noticeable enough as to do
> not follow the safe path?

Also keep in mind that this is only going to be done
when you're about to construct a Python string, which
is a fairly expensive operation.

-- 
Greg

From stefan_ml at behnel.de  Thu Jun  5 08:17:58 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jun 2008 08:17:58 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <484711E2.9000709@student.matnat.uio.no>
References: <4846AD71.50407@semipol.de>	<4846D51C.2030607@behnel.de>	<4846E014.5080300@semipol.de>	<4846F2E4.302@behnel.de>	<4846FD38.3080106@semipol.de>	<48470AC0.7040901@student.matnat.uio.no>	<48470C4D.5060605@semipol.de>	<48470F5F.4010209@student.matnat.uio.no>
	<484711E2.9000709@student.matnat.uio.no>
Message-ID: <48478516.6070107@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> For your purposes, outputting "None" is probably 
> wrong anyway. What happens if the client library passes a char* 
> containing the string "None"? How do you distinguish it?

Yes, that's a valid point (and you're right that Johannes pretty much missed
my point). It would be weird if this worked:

    cdef char* s = NULL
    py_s = s
    assert py_s is None

but this didn't:

    cdef char* s
    py_s = None
    s = py_s

So if both would work, you might really end up with situations where you pass
None into a C function as a NULL pointer. Sounds pretty dangerous to me.

Even if we don't allow the second bit, the first case would already blur the
border between NULL pointers and Python objects, which currently is very well
separated. I think it would be a step backwards to cut into that.

Stefan

From stefan_ml at behnel.de  Thu Jun  5 08:39:21 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jun 2008 08:39:21 +0200
Subject: [Cython] char* and NULL in log statements
In-Reply-To: <e7ba66e40806041636n7fa498e3s623c059b75d0149@mail.gmail.com>
References: <4846AD71.50407@semipol.de>
	<4846D51C.2030607@behnel.de>	<4846E014.5080300@semipol.de>
	<4847104E.6010003@student.matnat.uio.no>
	<e7ba66e40806041636n7fa498e3s623c059b75d0149@mail.gmail.com>
Message-ID: <48478A19.2030607@behnel.de>

Hi,

Lisandro Dalcin wrote:
> On 6/4/08, Dag Sverre Seljebotn <dagss at student.matnat.uio.no> wrote:
>>  Well, in my mind, this is a reason for supporting Stefan in removing
>>  auto-coercion of char* to Python strings altogether (that is suggested
>>  once down in those unicode discussion threads, right Stefan?).
> 
> Well, I believe that is the right approach. However, what would be the
> way to generate a byte string from a char* pointer?

The Python equivalent of a C char* is a byte string ("bytes" or "bytearray" in
Py3). I totally support auto-coercion between byte strings and char*. I'm just
opposed to coercing a unicode object to a char*, as that tends to be an easy
source of bugs rather than something that makes your life easier.

My current favourite are file names. lxml deals with two types of path names:
URLs and filesystem paths. Both are UTF-8 encoded when coming from an XML file
and users commonly pass byte strings in Py2 and unicode strings in Py3. UTF-8
encoded URLs are fine in the case of libxml2, but to access a file on the
local file system, the file name must always use the local file system
encoding, which is often an ISO encoding or stuff like cp1252 (IIRC). So it is
actually pretty involved to encode a file path (once you know that it actually
*is* a file path and not a URL), or to decode a user provided byte string path
into a unicode string, e.g. to print it in an error message (which must use
the encoding of the output device!).

Encodings can really, really become a complex matter...

Stefan

From stefan_ml at behnel.de  Thu Jun  5 15:05:17 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jun 2008 15:05:17 +0200
Subject: [Cython] nogil checking
Message-ID: <4847E48D.2070103@behnel.de>

Hi,

I merged most of the nogil checking code from Pyrex into Cython, but I noticed
that it will break a lot of code. It makes Cython pretty strict about what is
allowed in a nogil function and what isn't. It even checks function pointer
assignments for "nogil" matches, so you really have to take care when
declaring and assigning callback functions.

The problem is that it's not always possible to fix such code, as "nogil" is
more about semantics than about syntax. In lxml, I use the same (SAX2)
callback function struct in a number of places, and I sometimes release the
GIL when passing it into libxml2 and sometimes I keep holding it, depending on
what I consider faster. The related callback functions are designed to handle
exactly their specific case, so some are declared "with GIL" and others do not
have a declaration as they know the GIL will be held when they get called. So
this requires an explicit cast now. And a cast always holds the risk of
shadowing real bugs.

Here's an example. Say, we have three functions that implement a callback:

    cdef void c1():
        # Python stuff
    cdef void c2() with GIL:
        # Python stuff
    cdef void c3() nogil:
        # C stuff

For the following callback function type

    ctypedef void (*callback)()

all of the three functions should be assignable, as we know we hold the GIL
when it's called, whereas for

    ctypedef void (*callback)() nogil

only c2 and c3 should work, as we know we do not hold it. Pyrex currently
distinguishes "nogil" and "with GIL" functions generally from normal cdef
functions, so c2 and c3 cannot be assigned to the first callback pointer. I
consider that a bug in Pyrex. Below is a very simple patch that fixes the
problem above. It's a bit hackish in that it's not restricted to an
assignment, but it seems to work for me.

Stefan


diff -r d69b3342f623 Cython/Compiler/PyrexTypes.py
--- a/Cython/Compiler/PyrexTypes.py     Thu Jun 05 12:20:27 2008 +0200
+++ b/Cython/Compiler/PyrexTypes.py     Thu Jun 05 14:09:21 2008 +0200
@@ -695,7 +695,7 @@ class CFuncType(CType):
             return 0
         if not self.same_calling_convention_as(other_type):
             return 0
-        if self.nogil != other_type.nogil:
+        if self.nogil and not other_type.nogil:
             return 0
         return 1



From dalcinl at gmail.com  Thu Jun  5 16:24:17 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 5 Jun 2008 11:24:17 -0300
Subject: [Cython] Context.extract_module_name() not being used in
	Context.compile() at Main.py
Message-ID: <e7ba66e40806050724u16326e1ex1bb1ae03862c8ccb@mail.gmail.com>

Stefan, could you review if the one-line patch I've attached is right?
I believe you forgot to do that.

Without this patch, Cython just does not work (at Cython compilation
time) for me, not easy to figure out why, but I guess *.pxd's are not
being found.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: find_module.patch
Type: application/octet-stream
Size: 621 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080605/5bc17a4f/attachment.obj 

From dalcinl at gmail.com  Thu Jun  5 17:21:58 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 5 Jun 2008 12:21:58 -0300
Subject: [Cython] GCC warnings about uninitialized variables
Message-ID: <e7ba66e40806050821y56e85863hc6b950b40f681d9c@mail.gmail.com>

GCC complains about uninitialized variables when compiling the
generated C sources in some cases like the below:

cdef int CHKERR(int ierr) except -1:
    if ierr==0: return 0
    raise RuntimeError

cdef int obj2int(object ob) except *:
    return ob

def foo(a):
    cdef int i = obj2int(a)
    CHKERR(i)

The warning is something like :

retval.c: In function '__pyx_pf_6retval_foo':
retval.c:244: warning: '__pyx_r' may be used uninitialized in this function


Then I noticed that Cython does not initializes '__pyx_r' to NULL and
that's the source of the problem. IMHO, we should initialize it to
NULL. Iff it even happens that (because of a bug) Cython generates bad
code, It's far easier to discover it from a SystemError exception
(because function returned NULL and an exception was not raised) than
a segfault because of the return is just garbage.

Whould a patch for this be accepted (well, it would be just one line
patch) ?? Iff it is accepted, what do you prefer for the generted C
code:

PyObject __pyx_r = 0;

or

PyObject __pyx_r = NULL;

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Jun  5 21:18:24 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jun 2008 21:18:24 +0200
Subject: [Cython] Context.extract_module_name() not being used in
 Context.compile() at Main.py
In-Reply-To: <e7ba66e40806050724u16326e1ex1bb1ae03862c8ccb@mail.gmail.com>
References: <e7ba66e40806050724u16326e1ex1bb1ae03862c8ccb@mail.gmail.com>
Message-ID: <48483C00.4040302@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Stefan, could you review if the one-line patch I've attached is right?

Doesn't

  module_name = full_module_name or self.extract_module_name(source, options)

work for you? "full_module_name" is something that is extracted from
distutils, so it's the best guess we can start with if it's provided.

Stefan


From dalcinl at gmail.com  Thu Jun  5 22:46:06 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 5 Jun 2008 17:46:06 -0300
Subject: [Cython] Context.extract_module_name() not being used in
	Context.compile() at Main.py
In-Reply-To: <48483C00.4040302@behnel.de>
References: <e7ba66e40806050724u16326e1ex1bb1ae03862c8ccb@mail.gmail.com>
	<48483C00.4040302@behnel.de>
Message-ID: <e7ba66e40806051346g15c8069du84e78adba40ccbb2@mail.gmail.com>

On 6/5/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Doesn't
>
>   module_name = full_module_name or self.extract_module_name(source, options)
>
>  work for you?

It's not working, but I believe I definitelly catched the problem...

> "full_module_name" is something that is extracted from
>  distutils, so it's the best guess we can start with if it's provided.

But then what are those lines near the begining of  Context.compile(),
pasted below ?

        if full_module_name is None:
            full_module_name, _ = os.path.splitext(source)
            full_module_name = re.sub(r'[\\/]', '.', full_module_name)
            full_module_name = re.sub(r'[^\w.]', '_', full_module_name)

Iff I comment-out those lines, then all works with your latest patch.
And now I believe they should be removed. It seems that all the
previous hackery was in fact what was letting Cython manage modules
better than Pyrex (which required *.pyx files with full dotted names
for package/module management, righ?). In short, the lines above seems
to be obsolete...



>
>  Stefan
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Fri Jun  6 05:48:08 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 06 Jun 2008 15:48:08 +1200
Subject: [Cython] nogil checking
In-Reply-To: <4847E48D.2070103@behnel.de>
References: <4847E48D.2070103@behnel.de>
Message-ID: <4848B378.4040103@canterbury.ac.nz>

Stefan Behnel wrote:

>     cdef void c1():
>     cdef void c2() with GIL:
>     cdef void c3() nogil:
> 
> For the following callback function type
> 
>     ctypedef void (*callback)()
> 
> all of the three functions should be assignable

Yes, it's being too strict at the moment. I'll
see about fixing it.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Jun  6 06:09:39 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 06 Jun 2008 16:09:39 +1200
Subject: [Cython] Context.extract_module_name() not being used in
 Context.compile() at Main.py
In-Reply-To: <e7ba66e40806050724u16326e1ex1bb1ae03862c8ccb@mail.gmail.com>
References: <e7ba66e40806050724u16326e1ex1bb1ae03862c8ccb@mail.gmail.com>
Message-ID: <4848B883.1060904@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Without this patch, Cython just does not work (at Cython compilation
> time) for me, not easy to figure out why, but I guess *.pxd's are not
> being found.

You may need to put __init__.py or __init__.pyx files in
your source directories so that they will be recognised
as packages.

-- 
Greg

From stefan_ml at behnel.de  Fri Jun  6 07:46:48 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Jun 2008 07:46:48 +0200
Subject: [Cython] Context.extract_module_name() not being used
 in	Context.compile() at Main.py
In-Reply-To: <e7ba66e40806051346g15c8069du84e78adba40ccbb2@mail.gmail.com>
References: <e7ba66e40806050724u16326e1ex1bb1ae03862c8ccb@mail.gmail.com>	<48483C00.4040302@behnel.de>
	<e7ba66e40806051346g15c8069du84e78adba40ccbb2@mail.gmail.com>
Message-ID: <4848CF48.7070400@behnel.de>

Hi,

Lisandro Dalcin wrote:
>         if full_module_name is None:
>             full_module_name, _ = os.path.splitext(source)
>             full_module_name = re.sub(r'[\\/]', '.', full_module_name)
>             full_module_name = re.sub(r'[^\w.]', '_', full_module_name)
> 
> Iff I comment-out those lines, then all works with your latest patch.

another good catch, thanks!

Stefan


From stefan_ml at behnel.de  Fri Jun  6 08:33:25 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Jun 2008 08:33:25 +0200
Subject: [Cython] GCC warnings about uninitialized variables
In-Reply-To: <e7ba66e40806050821y56e85863hc6b950b40f681d9c@mail.gmail.com>
References: <e7ba66e40806050821y56e85863hc6b950b40f681d9c@mail.gmail.com>
Message-ID: <4848DA35.506@behnel.de>

Hi,

Lisandro Dalcin wrote:
> GCC complains about uninitialized variables when compiling the
> generated C sources in some cases like the below:
> 
> cdef int CHKERR(int ierr) except -1:
>     if ierr==0: return 0
>     raise RuntimeError
> 
> cdef int obj2int(object ob) except *:
>     return ob
> 
> def foo(a):
>     cdef int i = obj2int(a)
>     CHKERR(i)

I added that as test case under tests/run/exceptionpropagation.pyx.


> The warning is something like :
> 
> retval.c: In function '__pyx_pf_6retval_foo':
> retval.c:244: warning: '__pyx_r' may be used uninitialized in this function

GCC points to the wrong function here (maybe due to inlining), it actually
means obj2int(). Looking at the generated code:

"""
static  int __pyx_f_20exceptionpropagation_obj2int(PyObject *__pyx_v_ob) {
  int __pyx_r;
  int __pyx_1;

  __pyx_1 = __pyx_PyInt_int(__pyx_v_ob);
             if (unlikely((__pyx_1 == (int)-1) {...; goto __pyx_L1;}
  __pyx_r = __pyx_1;
  goto __pyx_L0;

  __pyx_r = 0;            /* sadly, this is unused code ... */
  goto __pyx_L0;
  __pyx_L1:;
  __Pyx_AddTraceback("exceptionpropagation.obj2int");
  __pyx_L0:;
  return __pyx_r;
}
"""

In foo(), Cython correctly generates

"""
  __pyx_r = Py_None; Py_INCREF(Py_None); /* initialisation in normal case */
  goto __pyx_L0;
  __pyx_L1:;
  __Pyx_AddTraceback("exceptionpropagation.foo");
  __pyx_r = NULL;                     /* initialisation in exception case */
  __pyx_L0:;
  return __pyx_r;
"""

I'm not sure how to fix this (I've never even used "except *"). Maybe an
initialisation in exactly that case makes sense?

Stefan


From stefan_ml at behnel.de  Fri Jun  6 17:14:59 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Jun 2008 17:14:59 +0200
Subject: [Cython] GCC warnings about uninitialized variables
In-Reply-To: <4848DA35.506@behnel.de>
References: <e7ba66e40806050821y56e85863hc6b950b40f681d9c@mail.gmail.com>
	<4848DA35.506@behnel.de>
Message-ID: <48495473.5060807@behnel.de>


Stefan Behnel wrote:
> Lisandro Dalcin wrote:
>> GCC complains about uninitialized variables when compiling the
>> generated C sources in some cases like the below:
>>
>> cdef int CHKERR(int ierr) except -1:
>>     if ierr==0: return 0
>>     raise RuntimeError
>>
>> cdef int obj2int(object ob) except *:
>>     return ob
>>
>> def foo(a):
>>     cdef int i = obj2int(a)
>>     CHKERR(i)
> 
> I added that as test case under tests/run/exceptionpropagation.pyx.

The following works for me (it's in cython-devel).

Stefan


# HG changeset patch
# User Stefan Behnel <scoder at users.berlios.de>
# Date 1212765117 -7200
# Node ID 7a3fa433aaf4da1c2e2f655facd8889658820e8c
# Parent  e92098251a9d38ef700c78bfb6ed593f685ba563
fix return value setting for 'except *' functions

diff -r e92098251a9d -r 7a3fa433aaf4 Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py  Fri Jun 06 08:31:11 2008 +0200
+++ b/Cython/Compiler/Nodes.py  Fri Jun 06 17:11:57 2008 +0200
@@ -956,6 +956,8 @@ class FuncDefNode(StatNode, BlockNode):
             exc_check = self.caller_will_check_exceptions()
             if err_val is not None or exc_check:
                 code.putln('__Pyx_AddTraceback("%s");' %
self.entry.qualified_name)
+                if err_val is None and self.return_type.default_value:
+                    err_val = self.return_type.default_value
                 if err_val is not None:
                     code.putln(
                         "%s = %s;" % (


From dalcinl at gmail.com  Fri Jun  6 18:11:16 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 6 Jun 2008 13:11:16 -0300
Subject: [Cython] GCC warnings about uninitialized variables
In-Reply-To: <48495473.5060807@behnel.de>
References: <e7ba66e40806050821y56e85863hc6b950b40f681d9c@mail.gmail.com>
	<4848DA35.506@behnel.de> <48495473.5060807@behnel.de>
Message-ID: <e7ba66e40806060911w3c8c9672web52cc95ae6d62c3@mail.gmail.com>

This is not working for me in other cases, I'll try to make equivalent
examples and then I'll come back.

On 6/6/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
>  Stefan Behnel wrote:
>  > Lisandro Dalcin wrote:
>  >> GCC complains about uninitialized variables when compiling the
>  >> generated C sources in some cases like the below:
>  >>
>  >> cdef int CHKERR(int ierr) except -1:
>  >>     if ierr==0: return 0
>  >>     raise RuntimeError
>  >>
>  >> cdef int obj2int(object ob) except *:
>  >>     return ob
>  >>
>  >> def foo(a):
>  >>     cdef int i = obj2int(a)
>  >>     CHKERR(i)
>  >
>  > I added that as test case under tests/run/exceptionpropagation.pyx.
>
>
> The following works for me (it's in cython-devel).
>
>  Stefan
>
>
>  # HG changeset patch
>  # User Stefan Behnel <scoder at users.berlios.de>
>  # Date 1212765117 -7200
>  # Node ID 7a3fa433aaf4da1c2e2f655facd8889658820e8c
>  # Parent  e92098251a9d38ef700c78bfb6ed593f685ba563
>  fix return value setting for 'except *' functions
>
>  diff -r e92098251a9d -r 7a3fa433aaf4 Cython/Compiler/Nodes.py
>  --- a/Cython/Compiler/Nodes.py  Fri Jun 06 08:31:11 2008 +0200
>  +++ b/Cython/Compiler/Nodes.py  Fri Jun 06 17:11:57 2008 +0200
>  @@ -956,6 +956,8 @@ class FuncDefNode(StatNode, BlockNode):
>              exc_check = self.caller_will_check_exceptions()
>              if err_val is not None or exc_check:
>                  code.putln('__Pyx_AddTraceback("%s");' %
>  self.entry.qualified_name)
>  +                if err_val is None and self.return_type.default_value:
>  +                    err_val = self.return_type.default_value
>                  if err_val is not None:
>                      code.putln(
>                          "%s = %s;" % (
>
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Jun  6 22:03:08 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 06 Jun 2008 22:03:08 +0200
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <4846CA1A.2060706@behnel.de>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
	<4846CA1A.2060706@behnel.de>
Message-ID: <484997FC.7010207@behnel.de>

Hi,

Stefan Behnel wrote:
> Robert Bradshaw wrote:
>> I looked at all the Pyrex changes and incorporated everything we  
>> didn't already have
>> except for the GIL and dependancy tracking stuff.

I finished merging both now and also the parser cleanup (it now has a parser
context object instead of tons of pass-through keyword arguments, that was so
over-due...). After adapting lxml to the stricter nogil checking (and applying
the temporary (?) fix I posted), all seems to work just fine for me.

I think the next big test is Sage then. Anyone? :)

Robert, if I'm not mistaken, this means that the next Cython release will be
0.9.8.15, right? Could you double-check that all Pyrex changes are in now? We
seem to be missing at least the forward declaration simplifications, Pyrex
changesets 71/74/76. I think they'd be nice to have.

We also have to check that we added all new test cases from Pyrex, at least
into the tests/compile and tests/errors directories. I'll have close to no
time this weekend, but since this is a simple task, maybe someone else could
take a start here? More tests means a better release on Monday.

Stefan


From robertwb at math.washington.edu  Fri Jun  6 22:10:11 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 6 Jun 2008 13:10:11 -0700
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <484997FC.7010207@behnel.de>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
	<4846CA1A.2060706@behnel.de> <484997FC.7010207@behnel.de>
Message-ID: <FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>

On Jun 6, 2008, at 1:03 PM, Stefan Behnel wrote:

> Hi,
>
> Stefan Behnel wrote:
>> Robert Bradshaw wrote:
>>> I looked at all the Pyrex changes and incorporated everything we
>>> didn't already have
>>> except for the GIL and dependancy tracking stuff.
>
> I finished merging both now and also the parser cleanup (it now has  
> a parser
> context object instead of tons of pass-through keyword arguments,  
> that was so
> over-due...). After adapting lxml to the stricter nogil checking  
> (and applying
> the temporary (?) fix I posted), all seems to work just fine for me.
>
> I think the next big test is Sage then. Anyone? :)

I plan on doing that later tonight after I get back... I've compiled  
sage recently with cython-devel, so don't expect any big surprises.

> Robert, if I'm not mistaken, this means that the next Cython  
> release will be
> 0.9.8.15, right? Could you double-check that all Pyrex changes are  
> in now? We
> seem to be missing at least the forward declaration  
> simplifications, Pyrex
> changesets 71/74/76. I think they'd be nice to have.

Our circular import stuff is a superset of this, but maybe I'm  
mistaken here. We could support this too if you want.

> We also have to check that we added all new test cases from Pyrex,  
> at least
> into the tests/compile and tests/errors directories. I'll have  
> close to no
> time this weekend, but since this is a simple task, maybe someone  
> else could
> take a start here? More tests means a better release on Monday.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Sat Jun  7 08:04:04 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 07 Jun 2008 08:04:04 +0200
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>	<4846CA1A.2060706@behnel.de>
	<484997FC.7010207@behnel.de>
	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
Message-ID: <484A24D4.7050901@behnel.de>

Hi,

Robert Bradshaw wrote:
> On Jun 6, 2008, at 1:03 PM, Stefan Behnel wrote:
>> We seem to be missing at least the forward declaration  
>> simplifications, Pyrex
>> changesets 71/74/76. I think they'd be nice to have.
> 
> Our circular import stuff is a superset of this

Partly, yes. However, it's also a way to simplify declarations. From what I
understood so far, it basically restricts class forward declarations to saying
"this is a class", without repeating things like base classes and maybe even
the [] renaming of exported types. That allows you to keep all of that in just
one place in your source.

Stefan

From languitar at semipol.de  Sun Jun  8 01:24:11 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Sun, 08 Jun 2008 01:24:11 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
	interpreter
Message-ID: <484B189B.8030402@semipol.de>

Hi,

maybe I've just got a lack of understanding or I don't know...

I've got a class that in general looks like this:

cpdef class Foo:
	cdef char* myData

	cdef void setData(Foo self, char *data)
		self.myData = data

	cpdef doSomething(Foo self)
		print self.myData

The problem is that working with myData directly from Cython is no
problem but calling doSomething from the python interpreter causes a
segfault because self.myData then points to NULL. I've also observed
that self in setData points to a different address than self in
doSomething if it is called from the interpreter.

What's the problem with this approach?

Thanks
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080608/ea2a702d/attachment.pgp 

From dagss at student.matnat.uio.no  Sun Jun  8 01:48:07 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 8 Jun 2008 01:48:07 +0200 (CEST)
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <484B189B.8030402@semipol.de>
References: <484B189B.8030402@semipol.de>
Message-ID: <64259.193.157.229.67.1212882487.squirrel@webmail.uio.no>

Johannes Wienke wrote:
> Hi,
>
> maybe I've just got a lack of understanding or I don't know...
>
> I've got a class that in general looks like this:
>
> cpdef class Foo:
> 	cdef char* myData
>
> 	cdef void setData(Foo self, char *data)
> 		self.myData = data
>
> 	cpdef doSomething(Foo self)
> 		print self.myData
>
> The problem is that working with myData directly from Cython is no
> problem but calling doSomething from the python interpreter causes a
> segfault because self.myData then points to NULL. I've also observed
> that self in setData points to a different address than self in
> doSomething if it is called from the interpreter.

Can you provide a full running session of what you try to do? I.e. can you
confirm that you do something like this:

Py> import foo
Py> a = foo.Foo()
Py> a.setData("aaaa")
Py> a.doSomething()

and it still crashes? (I'm guessing so...and in that case, I don't have a
clue, wait for somebody else to respond :-) ).

(But do double-check that the "self" argument is typed correctly. What
happens if you try to assign "self" to a cdef-ed local variable, i.e.

cdef Foo x = self
print x.myData

?)

Dag Sverre


From robertwb at math.washington.edu  Sun Jun  8 02:00:41 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Jun 2008 17:00:41 -0700
Subject: [Cython] Accessing C attributes from Python functions in the
	interpreter
In-Reply-To: <484B189B.8030402@semipol.de>
References: <484B189B.8030402@semipol.de>
Message-ID: <DDF9B460-E7B4-4B20-B873-59FEB9F470C2@math.washington.edu>

On Jun 7, 2008, at 4:24 PM, Johannes Wienke wrote:

> Hi,
>
> maybe I've just got a lack of understanding or I don't know...
>
> I've got a class that in general looks like this:
>
> cpdef class Foo:
> 	cdef char* myData
>
> 	cdef void setData(Foo self, char *data)
> 		self.myData = data
>
> 	cpdef doSomething(Foo self)
> 		print self.myData
>
> The problem is that working with myData directly from Cython is no
> problem but calling doSomething from the python interpreter causes a
> segfault because self.myData then points to NULL. I've also observed
> that self in setData points to a different address than self in
> doSomething if it is called from the interpreter.
>
> What's the problem with this approach?


Where are you setting your myData? If you do

     Foo().doSomething()

then it will segfault because the data isn't set yet. If you have

     def set_data_from_python(self, py_string):
         self.myData = py_string

Then the conversion will happen, but myData is set to point to the  
inside of py_string (i.e. no copying is actually done) and so the  
instant py_string gets deallocated the actual char* gets reclaimed.

Hopefully this helps, though the code snippet above doesn't crash by  
itself.

- Robert
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080607/1de1ce27/attachment.pgp 

From greg.ewing at canterbury.ac.nz  Sun Jun  8 01:55:50 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 08 Jun 2008 11:55:50 +1200
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <484A24D4.7050901@behnel.de>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
	<4846CA1A.2060706@behnel.de> <484997FC.7010207@behnel.de>
	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
	<484A24D4.7050901@behnel.de>
Message-ID: <484B2006.3070308@canterbury.ac.nz>

Robert Bradshaw wrote:

> Our circular import stuff is a superset of this

What does Cython do differently in this area?

-- 
Greg

From robertwb at math.washington.edu  Sun Jun  8 02:05:28 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 7 Jun 2008 17:05:28 -0700
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <484B2006.3070308@canterbury.ac.nz>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
	<4846CA1A.2060706@behnel.de> <484997FC.7010207@behnel.de>
	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
	<484A24D4.7050901@behnel.de> <484B2006.3070308@canterbury.ac.nz>
Message-ID: <8D0A9ECF-786C-4D15-9E4F-DE04AD16849C@math.washington.edu>


On Jun 7, 2008, at 4:55 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> Our circular import stuff is a superset of this
>
> What does Cython do differently in this area?

In a.pyx you can do "from b cimport B" and from b.pyx you can do  
"from a cimport A" without any problems.

- Robert



From greg.ewing at canterbury.ac.nz  Sun Jun  8 02:31:43 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 08 Jun 2008 12:31:43 +1200
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <8D0A9ECF-786C-4D15-9E4F-DE04AD16849C@math.washington.edu>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
	<4846CA1A.2060706@behnel.de> <484997FC.7010207@behnel.de>
	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
	<484A24D4.7050901@behnel.de> <484B2006.3070308@canterbury.ac.nz>
	<8D0A9ECF-786C-4D15-9E4F-DE04AD16849C@math.washington.edu>
Message-ID: <484B286F.3050200@canterbury.ac.nz>

Robert Bradshaw wrote:

> In a.pyx you can do "from b cimport B" and from b.pyx you can do  
> "from a cimport A" without any problems.

There's never been any problem with that in Pyrex, as
far as I know.

The problems occur when .pxd files cimport from each
other, not .pyx files. That's what the recently
added forward declaration features are addressing.

Does Cython have a different way of handling circular
cimports among .pxd files? If so, how does it work?

-- 
Greg

From gfurnish at gfurnish.net  Sun Jun  8 02:48:18 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sat, 7 Jun 2008 17:48:18 -0700
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <484B286F.3050200@canterbury.ac.nz>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
	<4846CA1A.2060706@behnel.de> <484997FC.7010207@behnel.de>
	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
	<484A24D4.7050901@behnel.de> <484B2006.3070308@canterbury.ac.nz>
	<8D0A9ECF-786C-4D15-9E4F-DE04AD16849C@math.washington.edu>
	<484B286F.3050200@canterbury.ac.nz>
Message-ID: <8f8f8530806071748l29495fcct602d4f6c5c390ce@mail.gmail.com>

Classes already get forward declared; we thus just essentially run a
dependency sorting algorithm on the classes to make sure they are
output in the correct order and this gives us circular imports in pxd
files.

On Sat, Jun 7, 2008 at 5:31 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Robert Bradshaw wrote:
>
>> In a.pyx you can do "from b cimport B" and from b.pyx you can do
>> "from a cimport A" without any problems.
>
> There's never been any problem with that in Pyrex, as
> far as I know.
>
> The problems occur when .pxd files cimport from each
> other, not .pyx files. That's what the recently
> added forward declaration features are addressing.
>
> Does Cython have a different way of handling circular
> cimports among .pxd files? If so, how does it work?
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From greg.ewing at canterbury.ac.nz  Sun Jun  8 03:02:29 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 08 Jun 2008 13:02:29 +1200
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <8f8f8530806071748l29495fcct602d4f6c5c390ce@mail.gmail.com>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>
	<4846CA1A.2060706@behnel.de> <484997FC.7010207@behnel.de>
	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
	<484A24D4.7050901@behnel.de> <484B2006.3070308@canterbury.ac.nz>
	<8D0A9ECF-786C-4D15-9E4F-DE04AD16849C@math.washington.edu>
	<484B286F.3050200@canterbury.ac.nz>
	<8f8f8530806071748l29495fcct602d4f6c5c390ce@mail.gmail.com>
Message-ID: <484B2FA5.6070502@canterbury.ac.nz>

Gary Furnish wrote:
> Classes already get forward declared; we thus just essentially run a
> dependency sorting algorithm on the classes to make sure they are
> output in the correct order and this gives us circular imports in pxd
> files.

You mean if a cimport references something that's
not defined yet, it's assumed to be a class?

That's not necessarily correct -- it could be a
struct or union, or a typedef referring to just
about any type.

-- 
Greg

From gfurnish at gfurnish.net  Sun Jun  8 03:34:18 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sat, 7 Jun 2008 18:34:18 -0700
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <484B2FA5.6070502@canterbury.ac.nz>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<4846CA1A.2060706@behnel.de> <484997FC.7010207@behnel.de>
	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>
	<484A24D4.7050901@behnel.de> <484B2006.3070308@canterbury.ac.nz>
	<8D0A9ECF-786C-4D15-9E4F-DE04AD16849C@math.washington.edu>
	<484B286F.3050200@canterbury.ac.nz>
	<8f8f8530806071748l29495fcct602d4f6c5c390ce@mail.gmail.com>
	<484B2FA5.6070502@canterbury.ac.nz>
Message-ID: <8f8f8530806071834w929683dtff1f0d453b343ad6@mail.gmail.com>

No, I mean we basically run a sort on the vtabs, after everything is
imported.  This plus a sort on (something else) is sufficient to make
circular pxd imports work.

On Sat, Jun 7, 2008 at 6:02 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Gary Furnish wrote:
>> Classes already get forward declared; we thus just essentially run a
>> dependency sorting algorithm on the classes to make sure they are
>> output in the correct order and this gives us circular imports in pxd
>> files.
>
> You mean if a cimport references something that's
> not defined yet, it's assumed to be a class?
>
> That's not necessarily correct -- it could be a
> struct or union, or a typedef referring to just
> about any type.
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From greg.ewing at canterbury.ac.nz  Sun Jun  8 06:20:13 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 08 Jun 2008 16:20:13 +1200
Subject: [Cython] GCC warnings about uninitialized variables
In-Reply-To: <e7ba66e40806050821y56e85863hc6b950b40f681d9c@mail.gmail.com>
References: <e7ba66e40806050821y56e85863hc6b950b40f681d9c@mail.gmail.com>
Message-ID: <484B5DFD.70209@canterbury.ac.nz>

Lisandro Dalcin wrote:
> GCC complains about uninitialized variables when compiling the
> generated C sources in some cases like the below:
> 
> cdef int obj2int(object ob) except *:
>     return ob

As an aside, it's probably more efficient to declare
a function like that as something like

   cdef int obj2int(object ob) except? -1:

or some other unlikely value in place of -1. Then
callers will only call PyError_Occurred if that
particular value is returned, rather than every time.

The only time you should really have to use except *
is with a function returning void.

-- 
Greg


From stefan_ml at behnel.de  Sun Jun  8 07:44:24 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jun 2008 07:44:24 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <484B189B.8030402@semipol.de>
References: <484B189B.8030402@semipol.de>
Message-ID: <484B71B8.90504@behnel.de>

Hi,

as a side-note:

Johannes Wienke wrote:
> cpdef class Foo:
> 	cdef char* myData
> 
> 	cdef void setData(Foo self, char *data)
> 		self.myData = data

no need to declare the type of self here, you can just write

 	cdef void setData(self, char *data)
 		self.myData = data

Stefan

From stefan_ml at behnel.de  Sun Jun  8 08:21:27 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jun 2008 08:21:27 +0200
Subject: [Cython] remaining Pyrex changes
In-Reply-To: <8f8f8530806071748l29495fcct602d4f6c5c390ce@mail.gmail.com>
References: <54075.194.114.62.69.1212493695.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<18257ABF-C06F-4C78-A4D8-D35267426525@math.washington.edu>	<4846CA1A.2060706@behnel.de>
	<484997FC.7010207@behnel.de>	<FB51F4F4-2F99-4D33-916F-9842B0FF958B@math.washington.edu>	<484A24D4.7050901@behnel.de>
	<484B2006.3070308@canterbury.ac.nz>	<8D0A9ECF-786C-4D15-9E4F-DE04AD16849C@math.washington.edu>	<484B286F.3050200@canterbury.ac.nz>
	<8f8f8530806071748l29495fcct602d4f6c5c390ce@mail.gmail.com>
Message-ID: <484B7A67.1030204@behnel.de>

Hi,

Gary Furnish wrote:
> Classes already get forward declared; we thus just essentially run a
> dependency sorting algorithm on the classes to make sure they are
> output in the correct order and this gives us circular imports in pxd
> files.

the code for that is in the method sort_type_hierarchy() in ModuleNode.py.

Stefan


From languitar at semipol.de  Sun Jun  8 10:04:48 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Sun, 08 Jun 2008 10:04:48 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <484B71B8.90504@behnel.de>
References: <484B189B.8030402@semipol.de> <484B71B8.90504@behnel.de>
Message-ID: <484B92A0.5010202@semipol.de>

Hi,

Am 06/08/2008 07:44 AM schrieb Stefan Behnel:
> as a side-note:
> 
> Johannes Wienke wrote:
>> cpdef class Foo:
>> 	cdef char* myData
>>
>> 	cdef void setData(Foo self, char *data)
>> 		self.myData = data
> 
> no need to declare the type of self here, you can just write
> 
>  	cdef void setData(self, char *data)
>  		self.myData = data

Yes, I know. This was only one thing I tried to find the bug.... but
without luck. I will try to break it down to a working example today.

Johannes



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080608/7f7d46b5/attachment.pgp 

From languitar at semipol.de  Sun Jun  8 11:10:08 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Sun, 08 Jun 2008 11:10:08 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <64259.193.157.229.67.1212882487.squirrel@webmail.uio.no>
References: <484B189B.8030402@semipol.de>
	<64259.193.157.229.67.1212882487.squirrel@webmail.uio.no>
Message-ID: <484BA1F0.60209@semipol.de>

Am 06/08/2008 01:48 AM schrieb Dag Sverre Seljebotn:
> Johannes Wienke wrote:
>> Hi,
>>
>> maybe I've just got a lack of understanding or I don't know...
>>
>> I've got a class that in general looks like this:
>>
>> cpdef class Foo:
>> 	cdef char* myData
>>
>> 	cdef void setData(Foo self, char *data)
>> 		self.myData = data
>>
>> 	cpdef doSomething(Foo self)
>> 		print self.myData
>>
>> The problem is that working with myData directly from Cython is no
>> problem but calling doSomething from the python interpreter causes a
>> segfault because self.myData then points to NULL. I've also observed
>> that self in setData points to a different address than self in
>> doSomething if it is called from the interpreter.
> 
> Can you provide a full running session of what you try to do?

Ok, I have found the problem but have absolutely no clue how to solve this:

My code has to run inside Reinteract
(http://fishsoup.net/software/reinteract/). At the bottom of the site
the author explains the system Reinteract uses: "However, if reinteract
detects that a statement modifies a variable, then it makes a shallow
copy of the variable using copy.copy()"

This call to copy seems to be the problem because C data structures seem
to be forgotten while copying. Here's a little test case:

cdef extern from "string.h":
    char *strcpy(char *dest, char *src)
    long strlen(char *s)

cdef extern from "stdlib.h":
    void *calloc(long nmemb, long size)

cpdef class Foo:

    cdef char *arg

    def fillArg(self, string):
        s = string
        self.arg = <char *>calloc(strlen(s), sizeof(char))
        strcpy(self.arg, s)

    def tellArg(self):
        if self.arg == NULL:
            print "I don't have an arg"
        else:
            print self.arg



>>> from ship import test
>>> f = test.Foo()
>>> f.fillArg("Hey")
>>> f.tellArg()
Hey
>>> import copy
>>> fCopy = copy.copy(f)
>>> fCopy.tellArg()
I don't have an arg

In my case I simply didn't check that arg was not NULL and that resulted
in the segfault.

But is there a way to convince copy to copy also the C declarations?

Thanks in advance
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080608/c3caae25/attachment-0001.pgp 

From dagss at student.matnat.uio.no  Sun Jun  8 11:16:15 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 8 Jun 2008 11:16:15 +0200 (CEST)
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <484BA1F0.60209@semipol.de>
References: <484B189B.8030402@semipol.de>
	<64259.193.157.229.67.1212882487.squirrel@webmail.uio.no>
	<484BA1F0.60209@semipol.de>
Message-ID: <64326.193.157.229.67.1212916575.squirrel@webmail.uio.no>

Johannes Wienke wrote:
>
> But is there a way to convince copy to copy also the C declarations?
>

Google for "python copy module". The copy module docs then will state that
it uses the pickling API. So see the pyrex docs for pickling and try
that...

Dag Sverre


From languitar at semipol.de  Sun Jun  8 12:02:41 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Sun, 08 Jun 2008 12:02:41 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <64326.193.157.229.67.1212916575.squirrel@webmail.uio.no>
References: <484B189B.8030402@semipol.de>
	<64259.193.157.229.67.1212882487.squirrel@webmail.uio.no>
	<484BA1F0.60209@semipol.de>
	<64326.193.157.229.67.1212916575.squirrel@webmail.uio.no>
Message-ID: <484BAE41.4070108@semipol.de>

Am 06/08/2008 11:16 AM schrieb Dag Sverre Seljebotn:
> Johannes Wienke wrote:
>> But is there a way to convince copy to copy also the C declarations?
>>
> 
> Google for "python copy module". The copy module docs then will state that
> it uses the pickling API. So see the pyrex docs for pickling and try
> that...

Thanks for that hint, but where exactly do I find something about
pickling in the Pyrex docs?

Furthermore, wouldn't it be worth to mention this in the Cython docs
(preferably the Sphinx documentation)?

Johannes


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080608/f13b6556/attachment.pgp 

From dagss at student.matnat.uio.no  Sun Jun  8 12:57:44 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 8 Jun 2008 12:57:44 +0200 (CEST)
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <484BAE41.4070108@semipol.de>
References: <484B189B.8030402@semipol.de>
	<64259.193.157.229.67.1212882487.squirrel@webmail.uio.no>
	<484BA1F0.60209@semipol.de>
	<64326.193.157.229.67.1212916575.squirrel@webmail.uio.no>
	<484BAE41.4070108@semipol.de>
Message-ID: <64409.193.157.229.67.1212922664.squirrel@webmail.uio.no>

Johannes Wienke wrote:
> Am 06/08/2008 11:16 AM schrieb Dag Sverre Seljebotn:
>> Johannes Wienke wrote:
>>> But is there a way to convince copy to copy also the C declarations?
>>>
>>
>> Google for "python copy module". The copy module docs then will state
>> that
>> it uses the pickling API. So see the pyrex docs for pickling and try
>> that...
>
> Thanks for that hint, but where exactly do I find something about
> pickling in the Pyrex docs?
>

I'm sorry, I think I've misunderstood something myself here. It looks like
the pickling API is a generic Python feature towards extension classes,
and not really a Cython (or Pyrex) feature as such. So the documentation
for it can be found in the Python pickling module docs.

Also there's this thread from not long ago:

http://thread.gmane.org/gmane.comp.python.cython.devel/1896

Dag Sverre


From greg.ewing at canterbury.ac.nz  Sun Jun  8 13:44:31 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 08 Jun 2008 23:44:31 +1200
Subject: [Cython] ANN: Pyrex 0.9.8.3
Message-ID: <484BC61F.8040407@canterbury.ac.nz>

Pyrex 0.9.8.3 is now available:

   http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/

Compiling multiple .pyx files in one go works properly now, and
can be substantially faster if you have a lot of modules that
cimport from each other. I had to rearrange various things to
make this work, so I hope I haven't broken anything.

The compatibility rules for nogil function pointers have
been fixed, so you can assign a nogil function to a function
pointer that isn't declared nogil (but not the other way around).

Plus a few other bug fixes listed in CHANGES.


What is Pyrex?
--------------

Pyrex is a language for writing Python extension modules.
It lets you freely mix operations on Python and C data, with
all Python reference counting and error checking handled
automatically.

From stefan_ml at behnel.de  Sun Jun  8 14:59:28 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 08 Jun 2008 14:59:28 +0200
Subject: [Cython] Profiling Cython
Message-ID: <484BD7B0.80504@behnel.de>

Hi,

I did a tiny bit of profiling on Cython compiling lxml.etree. Here are the
numbers I get:

"""
         3127018 function calls (2777951 primitive calls) in 25.128 CPU seconds

   Ordered by: internal time, call count
   List reduced from 1035 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   119144    4.579    0.000    4.671    0.000 Scanners.py:148(run_machine_inlined)
    18316    1.586    0.000    1.586    0.000 codecs.py:371(read)
   119144    1.074    0.000    5.746    0.000 Scanners.py:109(scan_a_token)
    88362    1.023    0.000    7.976    0.000 Scanners.py:88(read)
    29680    0.673    0.000    1.659    0.000 Code.py:103(mark_pos)
       77    0.588    0.008    0.588    0.008 posixpath.py:168(exists)
    88362    0.538    0.000    8.514    0.000 Scanning.py:397(next)
23318/2985    0.495    0.000    0.517    0.000 Nodes.py:155(end_pos)
    70446    0.442    0.000    0.591    0.000 Code.py:62(put)
90657/10562    0.351    0.000    4.549    0.000 Parsing.py:59(p_binop_expr)
[...]
"""

So the major headache here is Scanners.py in Plex. The method at the top-rank
is a huge function. According to the comments, it's the result of inlining a
couple of method calls that originally lead to slow code, and it looks heavily
profiled already.

Assuming that further optimisation attempts were rather futile, I just
compiled the module with Cython. The first (obvious) result is that the
internal calls disappear from the profile log, as they are now internal C
calls. The call that remains is Scanner.next(), which originally took an
accumulated 8.5 seconds. In the compiled version, it's down to just over 5
seconds, that's more than 40 percent faster.

"""
         2595627 function calls (2246560 primitive calls) in 18.681 CPU seconds

   Ordered by: internal time, call count
   List reduced from 1028 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    88362    4.246    0.000    5.041    0.000 Scanning.py:397(next)
    29680    0.673    0.000    1.632    0.000 Code.py:103(mark_pos)
    70446    0.439    0.000    0.586    0.000 Code.py:62(put)
90657/10562    0.335    0.000    3.228    0.000 Parsing.py:59(p_binop_expr)
23318/2985    0.316    0.000    0.338    0.000 Nodes.py:155(end_pos)
    65791    0.295    0.000    0.903    0.000 Code.py:47(putln)
    29677    0.289    0.000    0.915    0.000 Code.py:93(file_contents)
     4724    0.287    0.000    0.292    0.000 Symtab.py:532(allocate_temp)
    88070    0.247    0.000    0.247    0.000 ExprNodes.py:192(subexpr_nodes)
    52071    0.232    0.000    0.232    0.000 Nodes.py:82(__init__)
[...]
"""

In total, I get an improvement of 12% in compilation time. That makes me think
that it's actually worth putting the compilation into Cython's own setup.py,
and installing the compiled Scanning module next to the Python one (Python
prefers C extensions on import). Here's a patch, what do you think?

Stefan


-------------- next part --------------
A non-text attachment was scrubbed...
Name: compile-scanner-on-install.patch
Type: text/x-patch
Size: 1744 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080608/c251a481/attachment.bin 

From languitar at semipol.de  Mon Jun  9 18:27:56 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Mon, 09 Jun 2008 18:27:56 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <64409.193.157.229.67.1212922664.squirrel@webmail.uio.no>
References: <484B189B.8030402@semipol.de>
	<64259.193.157.229.67.1212882487.squirrel@webmail.uio.no>
	<484BA1F0.60209@semipol.de>
	<64326.193.157.229.67.1212916575.squirrel@webmail.uio.no>
	<484BAE41.4070108@semipol.de>
	<64409.193.157.229.67.1212922664.squirrel@webmail.uio.no>
Message-ID: <484D5A0C.3020803@semipol.de>

Am 06/08/2008 12:57 PM schrieb Dag Sverre Seljebotn:
> Johannes Wienke wrote:
>> Am 06/08/2008 11:16 AM schrieb Dag Sverre Seljebotn:
>>> Johannes Wienke wrote:
>>>> But is there a way to convince copy to copy also the C declarations?
>>>>
>>> Google for "python copy module". The copy module docs then will state
>>> that
>>> it uses the pickling API. So see the pyrex docs for pickling and try
>>> that...
>> Thanks for that hint, but where exactly do I find something about
>> pickling in the Pyrex docs?
>>
> 
> I'm sorry, I think I've misunderstood something myself here. It looks like
> the pickling API is a generic Python feature towards extension classes,
> and not really a Cython (or Pyrex) feature as such. So the documentation
> for it can be found in the Python pickling module docs.
> 
> Also there's this thread from not long ago:
> 
> http://thread.gmane.org/gmane.comp.python.cython.devel/1896

Thank you for the help.

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080609/6cdd0d3f/attachment.pgp 

From dalcinl at gmail.com  Tue Jun 10 16:25:40 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Jun 2008 11:25:40 -0300
Subject: [Cython] about those new *.dep files
Message-ID: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>

I've notices that now Cython generates *.dep files alongside the C
sources. It is strictly needed that those file are left around if not
explicitely requested for the user?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jun 10 18:17:21 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jun 2008 18:17:21 +0200
Subject: [Cython] about those new *.dep files
In-Reply-To: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>
References: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>
Message-ID: <484EA911.4090201@behnel.de>

Hi,

Lisandro Dalcin wrote:
> I've notices that now Cython generates *.dep files alongside the C
> sources. It is strictly needed that those file are left around if not
> explicitely requested for the user?

they store the dependencies (.pxd and .pxi files) of the compiled C file, so
users will likely be happy about them.

I think they still missing from to the .hgignore, though.

Stefan

From dalcinl at gmail.com  Tue Jun 10 19:02:59 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Jun 2008 14:02:59 -0300
Subject: [Cython] about those new *.dep files
In-Reply-To: <484EA911.4090201@behnel.de>
References: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>
	<484EA911.4090201@behnel.de>
Message-ID: <e7ba66e40806101002k66711a9cm5a726843ffaae2f4@mail.gmail.com>

On 6/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> they store the dependencies (.pxd and .pxi files) of the compiled C file, so
>  users will likely be happy about them.

Granted. I asked just in case this was an oversight after the last merges.

>  I think they still missing from to the .hgignore, though.


Could you also add the 'build/' directory to .hgignore ? I always use
Cython installed on my home directory (just to be sure that the
install is never broken)

BTW, could you check in Python 3.0 that the MANIFEST file is properly
generated from MANIFEST.in ? I believe the 'recursive-include'
directive is broken in Py3 distutils (however, my last test was about
a week ago against Py3 SVN, perhaps it's now fixed).





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jun 10 19:55:41 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jun 2008 19:55:41 +0200
Subject: [Cython] about those new *.dep files
In-Reply-To: <e7ba66e40806101002k66711a9cm5a726843ffaae2f4@mail.gmail.com>
References: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>	<484EA911.4090201@behnel.de>
	<e7ba66e40806101002k66711a9cm5a726843ffaae2f4@mail.gmail.com>
Message-ID: <484EC01D.2020001@behnel.de>

Hi,

Lisandro Dalcin wrote:
> BTW, could you check in Python 3.0 that the MANIFEST file is properly
> generated from MANIFEST.in ? I believe the 'recursive-include'
> directive is broken in Py3 distutils (however, my last test was about
> a week ago against Py3 SVN, perhaps it's now fixed).

Since you can't even run Cython under Py3, I don't see why it should be a
problem if you can't package it properly under that system.

Stefan


From stefan_ml at behnel.de  Tue Jun 10 20:29:22 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jun 2008 20:29:22 +0200
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
Message-ID: <484EC802.4010105@behnel.de>

http://www.python.org/dev/peps/pep-3121/

To be implemented one day. I don't even think it would be too hard to use such
a per-module instance state struct in general, even when not in Py3. That
would keep us from generating tons of Py3 specific code to implement this.

Stefan

From dalcinl at gmail.com  Tue Jun 10 20:31:32 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Jun 2008 15:31:32 -0300
Subject: [Cython] about those new *.dep files
In-Reply-To: <484EC01D.2020001@behnel.de>
References: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>
	<484EA911.4090201@behnel.de>
	<e7ba66e40806101002k66711a9cm5a726843ffaae2f4@mail.gmail.com>
	<484EC01D.2020001@behnel.de>
Message-ID: <e7ba66e40806101131xaae5a0v7751241cf0d49922@mail.gmail.com>

On 6/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Since you can't even run Cython under Py3, I don't see why it should be a
>  problem if you can't package it properly under that system.
>

But this is going to be problem any future Py3 package. This issue has
nothing to do with Cython, but a bug somewere in Py3 distutils. I'm
just abusing of you and asking for your help to confirm the problem
:-). If you do not have the time, never mind and forget about this
request...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Jun 10 20:43:35 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Jun 2008 15:43:35 -0300
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <484EC802.4010105@behnel.de>
References: <484EC802.4010105@behnel.de>
Message-ID: <e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>

Furthermore, we can 'abuse' of this new feature to 'export' C API's
from Cython-generated extension modules to other third-party
Cython-generated modules, thus avoiding at all the need of relying on
dynamic linking or dlopen hackery. Just some more good stuff for the
near future ...


On 6/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> http://www.python.org/dev/peps/pep-3121/
>
>  To be implemented one day. I don't even think it would be too hard to use such
>  a per-module instance state struct in general, even when not in Py3. That
>  would keep us from generating tons of Py3 specific code to implement this.
>
>  Stefan
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jun 10 20:47:28 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jun 2008 20:47:28 +0200
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>
References: <484EC802.4010105@behnel.de>
	<e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>
Message-ID: <484ECC40.6090506@behnel.de>


Lisandro Dalcin wrote:
> Furthermore, we can 'abuse' of this new feature to 'export' C API's
> from Cython-generated extension modules to other third-party
> Cython-generated modules, thus avoiding at all the need of relying on
> dynamic linking or dlopen hackery. Just some more good stuff for the
> near future ...

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/external.html#C_API_Declarations

Stefan


From dalcinl at gmail.com  Tue Jun 10 21:31:46 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Jun 2008 16:31:46 -0300
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <484ECC40.6090506@behnel.de>
References: <484EC802.4010105@behnel.de>
	<e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>
	<484ECC40.6090506@behnel.de>
Message-ID: <e7ba66e40806101231s7af419b0x7671c5d045bd8200@mail.gmail.com>

Ups!... I didn't know this was already available! I've just reviewed
the generated code, and the implementation of all this seems just
perfect and amazing!!

But I would like to know something. In the example available in the
link you pointed me we have:

cdef api void activate(Vehicle *v):
   .....

and then in the generated 'delorean_api.h' we have

static void (*activate)(struct Vehicle *);

Then the question is: Why the C name of the function pointer
'activate' is not mangled, using something like this (as it is
similarly done for C type objects):

static void (*__pyx_cfun_8delorean_activate)(struct Vehicle *);
#define activate __pyx_cfun_8delorean_activate





On 6/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
>  Lisandro Dalcin wrote:
>  > Furthermore, we can 'abuse' of this new feature to 'export' C API's
>  > from Cython-generated extension modules to other third-party
>  > Cython-generated modules, thus avoiding at all the need of relying on
>  > dynamic linking or dlopen hackery. Just some more good stuff for the
>  > near future ...
>
>
> http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/external.html#C_API_Declarations
>
>
>  Stefan
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jun 10 21:37:47 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 10 Jun 2008 21:37:47 +0200
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <e7ba66e40806101231s7af419b0x7671c5d045bd8200@mail.gmail.com>
References: <484EC802.4010105@behnel.de>	<e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>	<484ECC40.6090506@behnel.de>
	<e7ba66e40806101231s7af419b0x7671c5d045bd8200@mail.gmail.com>
Message-ID: <484ED80B.5010004@behnel.de>

Hi,

Lisandro Dalcin wrote:
> cdef api void activate(Vehicle *v):
>    .....
> 
> and then in the generated 'delorean_api.h' we have
> 
> static void (*activate)(struct Vehicle *);
> 
> Then the question is: Why the C name of the function pointer
> 'activate' is not mangled, using something like this (as it is
> similarly done for C type objects):
> 
> static void (*__pyx_cfun_8delorean_activate)(struct Vehicle *);
> #define activate __pyx_cfun_8delorean_activate

That's not the case for types at all. If you say

    cdef public class _Element [ type LxmlElementType, object LxmlElement ]:

then the names appear in the C file unchanged.

Stefan

From simon at arrowtheory.com  Tue Jun 10 21:40:35 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Tue, 10 Jun 2008 15:40:35 -0400
Subject: [Cython] cimport a plain c int ?
Message-ID: <20080610154035.8192c05f.simon@arrowtheory.com>


Is it possible to export a module level c integer to another pyrex module using
the cimport mechanism ?

Simon.

From dalcinl at gmail.com  Tue Jun 10 22:10:39 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 10 Jun 2008 17:10:39 -0300
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <484ED80B.5010004@behnel.de>
References: <484EC802.4010105@behnel.de>
	<e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>
	<484ECC40.6090506@behnel.de>
	<e7ba66e40806101231s7af419b0x7671c5d045bd8200@mail.gmail.com>
	<484ED80B.5010004@behnel.de>
Message-ID: <e7ba66e40806101310u7c43fd83qbff3375d8954f072@mail.gmail.com>

Perhaps I'm not being clear enough... perhaps and example will help.

Put this lines in a 'foo.pyx'  file anc cythonize it

# ---------------
cdef public class Foo [object PyFoo, type PyFoo_Type]:
    pass

cdef api void bar():
    pass
# --------------------

Then you will see that an 'foo_api.h' header is generated. Look at it
at the very begining, and you will see


static PyTypeObject *__pyx_ptype_3foo_Foo;
#define PyFoo_Type (*__pyx_ptype_3foo_Foo)

static void (*bar)(void);


Then, in the API header, Cython declares the type object with a
mangled C name, and then a #define.

Then, why isn't the same approach used for the C name 'bar' for the
function pointer? Does this make sense in order to 'protect' the
Cython-generated definition to interact badly with stuff that could be
got #include'd before the point "xxxx_api.h" is #include'd ??





On 6/10/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>
>  Lisandro Dalcin wrote:
>  > cdef api void activate(Vehicle *v):
>  >    .....
>  >
>  > and then in the generated 'delorean_api.h' we have
>  >
>  > static void (*activate)(struct Vehicle *);
>  >
>  > Then the question is: Why the C name of the function pointer
>  > 'activate' is not mangled, using something like this (as it is
>  > similarly done for C type objects):
>  >
>  > static void (*__pyx_cfun_8delorean_activate)(struct Vehicle *);
>  > #define activate __pyx_cfun_8delorean_activate
>
>
> That's not the case for types at all. If you say
>
>     cdef public class _Element [ type LxmlElementType, object LxmlElement ]:
>
>  then the names appear in the C file unchanged.
>
>
>  Stefan
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Jun 11 00:45:03 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 10 Jun 2008 15:45:03 -0700
Subject: [Cython] cimport a plain c int ?
In-Reply-To: <20080610154035.8192c05f.simon@arrowtheory.com>
References: <20080610154035.8192c05f.simon@arrowtheory.com>
Message-ID: <8DC9A6E2-5D61-4374-8825-99C625ECBF26@math.washington.edu>

Not yet.

On Jun 10, 2008, at 12:40 PM, Simon Burton wrote:

>
> Is it possible to export a module level c integer to another pyrex  
> module using
> the cimport mechanism ?
>
> Simon.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From greg.ewing at canterbury.ac.nz  Wed Jun 11 03:12:39 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 11 Jun 2008 13:12:39 +1200
Subject: [Cython] about those new *.dep files
In-Reply-To: <484EA911.4090201@behnel.de>
References: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>
	<484EA911.4090201@behnel.de>
Message-ID: <484F2687.6020103@canterbury.ac.nz>

Stefan Behnel wrote:

> they store the dependencies (.pxd and .pxi files) of the compiled C file, so
> users will likely be happy about them.

If you're worried about this, you could generate them
only when dependency tracking is being done, so users
who never use the dependency tracking features would
never see them.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Wed Jun 11 09:53:58 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 11 Jun 2008 19:53:58 +1200
Subject: [Cython] ANN: Pyrex 0.9.8.4
Message-ID: <484F8496.3070006@canterbury.ac.nz>

Pyrex 0.9.8.4 is now available:

   http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/

This version fixes a bug introduced by the last change
to unsigned integer indexing.


What is Pyrex?
--------------

Pyrex is a language for writing Python extension modules.
It lets you freely mix operations on Python and C data, with
all Python reference counting and error checking handled
automatically.

From jonas at MIT.EDU  Wed Jun 11 15:40:13 2008
From: jonas at MIT.EDU (Eric Jonas)
Date: Wed, 11 Jun 2008 09:40:13 -0400
Subject: [Cython] Faster Pickle possible?
Message-ID: <1213191613.8633.10.camel@convolution>

I assume most of us are using Cython because normal python is too slow
for our particular operation. I've been writing a compiler in python
which generates large graphs with ~600k nodes, and I'd like to
serialize/checkpoint these to disk for later stages of the compiler
pipeline to use. However, at the moment, it takes ~60 seconds to
serialize the resulting graph to disk with cPickle. 

Have any Cython users found better/faster ways of serializing to disk? I
could potentially use the "marshal" module, but my understanding is that
it only works for built-in types, and of course, that the root of my
class hierarchy I have four cython classes. I'd love to avoid writing my
own serialize/deserialize methods, if at all possible. 

I can't be the only person in this position, 

Thanks, 
			...Eric




From dalcinl at gmail.com  Wed Jun 11 17:13:25 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Jun 2008 12:13:25 -0300
Subject: [Cython] about those new *.dep files
In-Reply-To: <484F2687.6020103@canterbury.ac.nz>
References: <e7ba66e40806100725x5b2c99ehe12418174885a339@mail.gmail.com>
	<484EA911.4090201@behnel.de> <484F2687.6020103@canterbury.ac.nz>
Message-ID: <e7ba66e40806110813i3ade58bcxe45ef21f8da67e28@mail.gmail.com>

On 6/10/08, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Stefan Behnel wrote:
>
>  > they store the dependencies (.pxd and .pxi files) of the compiled C file, so
>  > users will likely be happy about them.
>
>
> If you're worried about this, you could generate them
>  only when dependency tracking is being done, so users
>  who never use the dependency tracking features would
>  never see them.

That's precisely what I meant. If the user do not ask for it (like in
my particular case),  *.dep files should not be generated. Of course,
if they are genereated, that is not an issue at all.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Jun 11 17:35:27 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Jun 2008 12:35:27 -0300
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <1213191613.8633.10.camel@convolution>
References: <1213191613.8633.10.camel@convolution>
Message-ID: <e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>

Eric, I've tried hard in the past to speedup pickling for
communication general Python object via MPI (for my project mpi4py).
I'm now reimplementing mpi4py from scratch using Cython, but I've not
yet reimplemented my pickle machinery.

Could you try to output your pickles to a cStringIO instance (instead
of a normal file instance) and tell me if you can get some speedup? If
you get some speedup, then perhaps we can hack a bit to make it even
faster for your use case...

Of course, the other way would be to define from scratch a custom
'format'  for saving your data, but I anticipate that that would be
really a lot of work, and I even doubt you can get more speedup that
with cPickle if you still want to be have to serialize arbitrary
Python data...


On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:
> I assume most of us are using Cython because normal python is too slow
>  for our particular operation. I've been writing a compiler in python
>  which generates large graphs with ~600k nodes, and I'd like to
>  serialize/checkpoint these to disk for later stages of the compiler
>  pipeline to use. However, at the moment, it takes ~60 seconds to
>  serialize the resulting graph to disk with cPickle.
>
>  Have any Cython users found better/faster ways of serializing to disk? I
>  could potentially use the "marshal" module, but my understanding is that
>  it only works for built-in types, and of course, that the root of my
>  class hierarchy I have four cython classes. I'd love to avoid writing my
>  own serialize/deserialize methods, if at all possible.
>
>  I can't be the only person in this position,
>
>  Thanks,
>                         ...Eric
>
>
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonas at MIT.EDU  Wed Jun 11 19:06:11 2008
From: jonas at MIT.EDU (Eric Jonas)
Date: Wed, 11 Jun 2008 13:06:11 -0400
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>
References: <1213191613.8633.10.camel@convolution>
	<e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>
Message-ID: <1213203972.8633.18.camel@convolution>

Lisandro, 
   Profile tells me that pickle is taking 70 seconds with a cStringIO
object and 76 seconds with a regular file object. Is that in line with
what you've seen ? I'm not entirely sure what pickle does that takes so
long, but I have a feeling it's spending a lot of time walking various
circular references -- this is a pretty dense graph i'm trying to
pickle, although (to avoid the recursion limit) I do try and break most
of the graph's references before-hand. 

		...Eric



On Wed, 2008-06-11 at 12:35 -0300, Lisandro Dalcin wrote:
> Eric, I've tried hard in the past to speedup pickling for
> communication general Python object via MPI (for my project mpi4py).
> I'm now reimplementing mpi4py from scratch using Cython, but I've not
> yet reimplemented my pickle machinery.
> 
> Could you try to output your pickles to a cStringIO instance (instead
> of a normal file instance) and tell me if you can get some speedup? If
> you get some speedup, then perhaps we can hack a bit to make it even
> faster for your use case...
> 
> Of course, the other way would be to define from scratch a custom
> 'format'  for saving your data, but I anticipate that that would be
> really a lot of work, and I even doubt you can get more speedup that
> with cPickle if you still want to be have to serialize arbitrary
> Python data...
> 
> 
> On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:
> > I assume most of us are using Cython because normal python is too slow
> >  for our particular operation. I've been writing a compiler in python
> >  which generates large graphs with ~600k nodes, and I'd like to
> >  serialize/checkpoint these to disk for later stages of the compiler
> >  pipeline to use. However, at the moment, it takes ~60 seconds to
> >  serialize the resulting graph to disk with cPickle.
> >
> >  Have any Cython users found better/faster ways of serializing to disk? I
> >  could potentially use the "marshal" module, but my understanding is that
> >  it only works for built-in types, and of course, that the root of my
> >  class hierarchy I have four cython classes. I'd love to avoid writing my
> >  own serialize/deserialize methods, if at all possible.
> >
> >  I can't be the only person in this position,
> >
> >  Thanks,
> >                         ...Eric
> >
> >
> >
> >  _______________________________________________
> >  Cython-dev mailing list
> >  Cython-dev at codespeak.net
> >  http://codespeak.net/mailman/listinfo/cython-dev
> >
> 
> 


From stefan_ml at behnel.de  Wed Jun 11 14:46:35 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 11 Jun 2008 14:46:35 +0200
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <e7ba66e40806101310u7c43fd83qbff3375d8954f072@mail.gmail.com>
References: <484EC802.4010105@behnel.de>	<e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>	<484ECC40.6090506@behnel.de>	<e7ba66e40806101231s7af419b0x7671c5d045bd8200@mail.gmail.com>	<484ED80B.5010004@behnel.de>
	<e7ba66e40806101310u7c43fd83qbff3375d8954f072@mail.gmail.com>
Message-ID: <484FC92B.7060403@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Then, why isn't the same approach used for the C name 'bar' for the
> function pointer? Does this make sense in order to 'protect' the
> Cython-generated definition to interact badly with stuff that could be
> got #include'd before the point "xxxx_api.h" is #include'd ??

I think you just have to take care what you make public.

Stefan


From stefan_ml at behnel.de  Wed Jun 11 19:44:53 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 11 Jun 2008 19:44:53 +0200
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <1213191613.8633.10.camel@convolution>
References: <1213191613.8633.10.camel@convolution>
Message-ID: <48500F15.5090104@behnel.de>

Hi,

Eric Jonas wrote:
> I assume most of us are using Cython because normal python is too slow
> for our particular operation. I've been writing a compiler in python
> which generates large graphs with ~600k nodes, and I'd like to
> serialize/checkpoint these to disk for later stages of the compiler
> pipeline to use. However, at the moment, it takes ~60 seconds to
> serialize the resulting graph to disk with cPickle. 

You can implement your own pickle mechanism for a specific object. We do that
in lxml.objectify, where you can pickle an XML element into a serialised XML
byte stream. If you can find a way to efficiently walk your graph at the C
level and serialise that by hand step by step, you may end up being much
faster than pickle's introspection machinery.

Stefan


From dalcinl at gmail.com  Wed Jun 11 20:23:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Jun 2008 15:23:57 -0300
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <1213203972.8633.18.camel@convolution>
References: <1213191613.8633.10.camel@convolution>
	<e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>
	<1213203972.8633.18.camel@convolution>
Message-ID: <e7ba66e40806111123s62a21c25i4d3095caa28356ad@mail.gmail.com>

On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:
> Lisandro,
>    Profile tells me that pickle is taking 70 seconds with a cStringIO
>  object and 76 seconds with a regular file object. Is that in line with
>  what you've seen ?

Well, as my use case is to serialize object and send them to the
wires, I never use file objects. Anyway, your result is the expected
one. So the cStringIO appoach will not help you too much.

> I'm not entirely sure what pickle does that takes so
>  long, but I have a feeling it's spending a lot of time walking various
>  circular references -- this is a pretty dense graph i'm trying to
>  pickle, although (to avoid the recursion limit) I do try and break most
>  of the graph's references before-hand.

Well, you said your graph has ~600k nodes. That's quite a lot of nodes
to traverse for cPickle.

I believe that you will need to follow Stefan's suggestion, that is,
you will need to implement a custom way for data persistence. Hope you
do not have general Python object attributes in your base classes!.


>  On Wed, 2008-06-11 at 12:35 -0300, Lisandro Dalcin wrote:
>  > Eric, I've tried hard in the past to speedup pickling for
>  > communication general Python object via MPI (for my project mpi4py).
>  > I'm now reimplementing mpi4py from scratch using Cython, but I've not
>  > yet reimplemented my pickle machinery.
>  >
>  > Could you try to output your pickles to a cStringIO instance (instead
>  > of a normal file instance) and tell me if you can get some speedup? If
>  > you get some speedup, then perhaps we can hack a bit to make it even
>  > faster for your use case...
>  >
>  > Of course, the other way would be to define from scratch a custom
>  > 'format'  for saving your data, but I anticipate that that would be
>  > really a lot of work, and I even doubt you can get more speedup that
>  > with cPickle if you still want to be have to serialize arbitrary
>  > Python data...
>  >
>  >
>  > On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:
>  > > I assume most of us are using Cython because normal python is too slow
>  > >  for our particular operation. I've been writing a compiler in python
>  > >  which generates large graphs with ~600k nodes, and I'd like to
>  > >  serialize/checkpoint these to disk for later stages of the compiler
>  > >  pipeline to use. However, at the moment, it takes ~60 seconds to
>  > >  serialize the resulting graph to disk with cPickle.
>  > >
>  > >  Have any Cython users found better/faster ways of serializing to disk? I
>  > >  could potentially use the "marshal" module, but my understanding is that
>  > >  it only works for built-in types, and of course, that the root of my
>  > >  class hierarchy I have four cython classes. I'd love to avoid writing my
>  > >  own serialize/deserialize methods, if at all possible.
>  > >
>  > >  I can't be the only person in this position,
>  > >
>  > >  Thanks,
>  > >                         ...Eric
>  > >
>  > >
>  > >
>  > >  _______________________________________________
>  > >  Cython-dev mailing list
>  > >  Cython-dev at codespeak.net
>  > >  http://codespeak.net/mailman/listinfo/cython-dev
>  > >
>  >
>  >
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jonas at MIT.EDU  Wed Jun 11 20:31:54 2008
From: jonas at MIT.EDU (Eric Jonas)
Date: Wed, 11 Jun 2008 14:31:54 -0400
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <e7ba66e40806111123s62a21c25i4d3095caa28356ad@mail.gmail.com>
References: <1213191613.8633.10.camel@convolution>
	<e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>
	<1213203972.8633.18.camel@convolution>
	<e7ba66e40806111123s62a21c25i4d3095caa28356ad@mail.gmail.com>
Message-ID: <1213209114.8633.26.camel@convolution>

Lisandro, 
    One more question: what did you mean by 
> you will need to implement a custom way for data persistence. Hope you
> do not have general Python object attributes in your base classes!.

I've got cython base classes and I want to pickle subclasses of those
cython base classes. But those subclasses could have just about anything
as an attribute. Is that still going to screw me? 

Thanks again, 
			...Eric


> 
> 
> >  On Wed, 2008-06-11 at 12:35 -0300, Lisandro Dalcin wrote:
> >  > Eric, I've tried hard in the past to speedup pickling for
> >  > communication general Python object via MPI (for my project mpi4py).
> >  > I'm now reimplementing mpi4py from scratch using Cython, but I've not
> >  > yet reimplemented my pickle machinery.
> >  >
> >  > Could you try to output your pickles to a cStringIO instance (instead
> >  > of a normal file instance) and tell me if you can get some speedup? If
> >  > you get some speedup, then perhaps we can hack a bit to make it even
> >  > faster for your use case...
> >  >
> >  > Of course, the other way would be to define from scratch a custom
> >  > 'format'  for saving your data, but I anticipate that that would be
> >  > really a lot of work, and I even doubt you can get more speedup that
> >  > with cPickle if you still want to be have to serialize arbitrary
> >  > Python data...
> >  >
> >  >
> >  > On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:
> >  > > I assume most of us are using Cython because normal python is too slow
> >  > >  for our particular operation. I've been writing a compiler in python
> >  > >  which generates large graphs with ~600k nodes, and I'd like to
> >  > >  serialize/checkpoint these to disk for later stages of the compiler
> >  > >  pipeline to use. However, at the moment, it takes ~60 seconds to
> >  > >  serialize the resulting graph to disk with cPickle.
> >  > >
> >  > >  Have any Cython users found better/faster ways of serializing to disk? I
> >  > >  could potentially use the "marshal" module, but my understanding is that
> >  > >  it only works for built-in types, and of course, that the root of my
> >  > >  class hierarchy I have four cython classes. I'd love to avoid writing my
> >  > >  own serialize/deserialize methods, if at all possible.
> >  > >
> >  > >  I can't be the only person in this position,
> >  > >
> >  > >  Thanks,
> >  > >                         ...Eric
> >  > >
> >  > >
> >  > >
> >  > >  _______________________________________________
> >  > >  Cython-dev mailing list
> >  > >  Cython-dev at codespeak.net
> >  > >  http://codespeak.net/mailman/listinfo/cython-dev
> >  > >
> >  >
> >  >
> >
> >  _______________________________________________
> >  Cython-dev mailing list
> >  Cython-dev at codespeak.net
> >  http://codespeak.net/mailman/listinfo/cython-dev
> >
> 
> 


From raoul.hecky at gmail.com  Wed Jun 11 21:24:27 2008
From: raoul.hecky at gmail.com (raoul)
Date: Wed, 11 Jun 2008 21:24:27 +0200
Subject: [Cython] Export conversion mechanism to external callers
Message-ID: <200806112124.27635.raoul.hecky@gmail.com>

Hello,

Cython is pretty new to me (and even python), and i want to do something not 
so simple.

Here is the context:
I have a standard c++ application using efl (enlightenment foundation 
libraries), and I want to extend my app with some python plugins. And these 
plugins needs to access some efl data to be able to draw on the same canvas 
than my main application.

To achieve that goal, i'm using boost.python to embedd python, that part is 
working great. 
On the python side, there is already a python binding for efl (python-efl). 
These bindings are written using Cython. This is why i'm asking here...

So, my python plugins will use python-efl to play with efl, but I need to pass 
some data to python plugins like canvas pointer, drawing objects, ... These 
data are Efl specific C struct.
I read the pyx/pxi files from python-efl, and there are some conversion 
fonction in there, to be able to pass efl struct to python objects. My 
question is, if there a way to be able to reuse that part of code from 
python-efl in my c++ application, so that i can give my python plugins Efl 
struct, and these python modules could use them as python-efl objects?

Someone on #edevelop told me that there is a way to export the needed 
functions to external callers, something like a keyword using Cython?

Perhaps someone here could probably point me to a solution, or even give me 
another direction to look.

Thanks in advance,

-- 
------------------------
Raoul Hecky


From robertwb at math.washington.edu  Wed Jun 11 21:32:10 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Jun 2008 12:32:10 -0700
Subject: [Cython] Export conversion mechanism to external callers
In-Reply-To: <200806112124.27635.raoul.hecky@gmail.com>
References: <200806112124.27635.raoul.hecky@gmail.com>
Message-ID: <9A34C2FC-962E-4BD8-9BA3-E02881F3B188@math.washington.edu>

It sounds like what you're looking for is the "public" keyword.

See http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/ 
Doc/Manual/external.html#PublicDecls

On Jun 11, 2008, at 12:24 PM, raoul wrote:

> Hello,
>
> Cython is pretty new to me (and even python), and i want to do  
> something not
> so simple.
>
> Here is the context:
> I have a standard c++ application using efl (enlightenment foundation
> libraries), and I want to extend my app with some python plugins.  
> And these
> plugins needs to access some efl data to be able to draw on the  
> same canvas
> than my main application.
>
> To achieve that goal, i'm using boost.python to embedd python, that  
> part is
> working great.
> On the python side, there is already a python binding for efl  
> (python-efl).
> These bindings are written using Cython. This is why i'm asking  
> here...
>
> So, my python plugins will use python-efl to play with efl, but I  
> need to pass
> some data to python plugins like canvas pointer, drawing  
> objects, ... These
> data are Efl specific C struct.
> I read the pyx/pxi files from python-efl, and there are some  
> conversion
> fonction in there, to be able to pass efl struct to python objects. My
> question is, if there a way to be able to reuse that part of code from
> python-efl in my c++ application, so that i can give my python  
> plugins Efl
> struct, and these python modules could use them as python-efl objects?
>
> Someone on #edevelop told me that there is a way to export the needed
> functions to external callers, something like a keyword using Cython?
>
> Perhaps someone here could probably point me to a solution, or even  
> give me
> another direction to look.
>
> Thanks in advance,
>
> -- 
> ------------------------
> Raoul Hecky
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Wed Jun 11 22:43:20 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 11 Jun 2008 17:43:20 -0300
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <1213209114.8633.26.camel@convolution>
References: <1213191613.8633.10.camel@convolution>
	<e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>
	<1213203972.8633.18.camel@convolution>
	<e7ba66e40806111123s62a21c25i4d3095caa28356ad@mail.gmail.com>
	<1213209114.8633.26.camel@convolution>
Message-ID: <e7ba66e40806111343r37275374y544ca31310851593@mail.gmail.com>

On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:

> I've got cython base classes and I want to pickle subclasses of those
>  cython base classes. But those subclasses could have just about anything
>  as an attribute. Is that still going to screw me?

Yes, if your 'nodes' can have arbitrary Python attibutes, then
implementing yourself a general mechanism ala pickle protocol is going
to be a nightmare for you.

Perhaps you can take a look at lxml, as Stefan pointed out? Perhaps
you can borrow some good ideas.

Have you your code available somewhere to take a look at your base
classes you want to pickle? Or perhaps you can send me a private mail
with some file attached? Perhaps the only way to make a clever
suggestion is to look at the actual code ;-)


>
>                         ...Eric
>
>
>
>  >
>  >
>  > >  On Wed, 2008-06-11 at 12:35 -0300, Lisandro Dalcin wrote:
>  > >  > Eric, I've tried hard in the past to speedup pickling for
>  > >  > communication general Python object via MPI (for my project mpi4py).
>  > >  > I'm now reimplementing mpi4py from scratch using Cython, but I've not
>  > >  > yet reimplemented my pickle machinery.
>  > >  >
>  > >  > Could you try to output your pickles to a cStringIO instance (instead
>  > >  > of a normal file instance) and tell me if you can get some speedup? If
>  > >  > you get some speedup, then perhaps we can hack a bit to make it even
>  > >  > faster for your use case...
>  > >  >
>  > >  > Of course, the other way would be to define from scratch a custom
>  > >  > 'format'  for saving your data, but I anticipate that that would be
>  > >  > really a lot of work, and I even doubt you can get more speedup that
>  > >  > with cPickle if you still want to be have to serialize arbitrary
>  > >  > Python data...
>  > >  >
>  > >  >
>  > >  > On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:
>  > >  > > I assume most of us are using Cython because normal python is too slow
>  > >  > >  for our particular operation. I've been writing a compiler in python
>  > >  > >  which generates large graphs with ~600k nodes, and I'd like to
>  > >  > >  serialize/checkpoint these to disk for later stages of the compiler
>  > >  > >  pipeline to use. However, at the moment, it takes ~60 seconds to
>  > >  > >  serialize the resulting graph to disk with cPickle.
>  > >  > >
>  > >  > >  Have any Cython users found better/faster ways of serializing to disk? I
>  > >  > >  could potentially use the "marshal" module, but my understanding is that
>  > >  > >  it only works for built-in types, and of course, that the root of my
>  > >  > >  class hierarchy I have four cython classes. I'd love to avoid writing my
>  > >  > >  own serialize/deserialize methods, if at all possible.
>  > >  > >
>  > >  > >  I can't be the only person in this position,
>  > >  > >
>  > >  > >  Thanks,
>  > >  > >                         ...Eric
>  > >  > >
>  > >  > >
>  > >  > >
>  > >  > >  _______________________________________________
>  > >  > >  Cython-dev mailing list
>  > >  > >  Cython-dev at codespeak.net
>  > >  > >  http://codespeak.net/mailman/listinfo/cython-dev
>  > >  > >
>  > >  >
>  > >  >
>  > >
>  > >  _______________________________________________
>  > >  Cython-dev mailing list
>  > >  Cython-dev at codespeak.net
>  > >  http://codespeak.net/mailman/listinfo/cython-dev
>  > >
>  >
>  >
>
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Jun 11 23:56:46 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 11 Jun 2008 23:56:46 +0200
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <e7ba66e40806111343r37275374y544ca31310851593@mail.gmail.com>
References: <1213191613.8633.10.camel@convolution>	<e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>	<1213203972.8633.18.camel@convolution>	<e7ba66e40806111123s62a21c25i4d3095caa28356ad@mail.gmail.com>	<1213209114.8633.26.camel@convolution>
	<e7ba66e40806111343r37275374y544ca31310851593@mail.gmail.com>
Message-ID: <48504A1E.3010701@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On 6/11/08, Eric Jonas <jonas at mit.edu> wrote:
> 
>> I've got cython base classes and I want to pickle subclasses of those
>>  cython base classes. But those subclasses could have just about anything
>>  as an attribute. Is that still going to screw me?
> 
> Yes, if your 'nodes' can have arbitrary Python attibutes, then
> implementing yourself a general mechanism ala pickle protocol is going
> to be a nightmare for you.

Not really, you just have each class implement a polymorphic 
serialize/deserialize method, and invoke it to do the real stuff. 
Perhaps a bit more hassle perhaps but hardly a "nightmare".

You can't really get around having a polymorphic method for every class 
to get top speed (dictionary creation, lookups and traversal, which any 
generic mechanism would use, is much slower).

I.e., if you want C speed, you need to write C-like static code and use 
more static algorithms. There's no way around that (except for a macro 
facility, which we don't have).

-- 
Dag Sverre

From ondrej at certik.cz  Thu Jun 12 01:51:40 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 12 Jun 2008 01:51:40 +0200
Subject: [Cython] Profiling Cython
In-Reply-To: <484BD7B0.80504@behnel.de>
References: <484BD7B0.80504@behnel.de>
Message-ID: <85b5c3130806111651q1b074c4by78db19bd6f006260@mail.gmail.com>

On Sun, Jun 8, 2008 at 2:59 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> I did a tiny bit of profiling on Cython compiling lxml.etree. Here are the
> numbers I get:
>
> """
>         3127018 function calls (2777951 primitive calls) in 25.128 CPU seconds
>
>   Ordered by: internal time, call count
>   List reduced from 1035 to 20 due to restriction <20>
>
>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>   119144    4.579    0.000    4.671    0.000 Scanners.py:148(run_machine_inlined)
>    18316    1.586    0.000    1.586    0.000 codecs.py:371(read)
>   119144    1.074    0.000    5.746    0.000 Scanners.py:109(scan_a_token)
>    88362    1.023    0.000    7.976    0.000 Scanners.py:88(read)
>    29680    0.673    0.000    1.659    0.000 Code.py:103(mark_pos)
>       77    0.588    0.008    0.588    0.008 posixpath.py:168(exists)
>    88362    0.538    0.000    8.514    0.000 Scanning.py:397(next)
> 23318/2985    0.495    0.000    0.517    0.000 Nodes.py:155(end_pos)
>    70446    0.442    0.000    0.591    0.000 Code.py:62(put)
> 90657/10562    0.351    0.000    4.549    0.000 Parsing.py:59(p_binop_expr)
> [...]
> """
>
> So the major headache here is Scanners.py in Plex. The method at the top-rank
> is a huge function. According to the comments, it's the result of inlining a
> couple of method calls that originally lead to slow code, and it looks heavily
> profiled already.
>
> Assuming that further optimisation attempts were rather futile, I just
> compiled the module with Cython. The first (obvious) result is that the
> internal calls disappear from the profile log, as they are now internal C
> calls. The call that remains is Scanner.next(), which originally took an
> accumulated 8.5 seconds. In the compiled version, it's down to just over 5
> seconds, that's more than 40 percent faster.
>
> """
>         2595627 function calls (2246560 primitive calls) in 18.681 CPU seconds
>
>   Ordered by: internal time, call count
>   List reduced from 1028 to 20 due to restriction <20>
>
>   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>    88362    4.246    0.000    5.041    0.000 Scanning.py:397(next)
>    29680    0.673    0.000    1.632    0.000 Code.py:103(mark_pos)
>    70446    0.439    0.000    0.586    0.000 Code.py:62(put)
> 90657/10562    0.335    0.000    3.228    0.000 Parsing.py:59(p_binop_expr)
> 23318/2985    0.316    0.000    0.338    0.000 Nodes.py:155(end_pos)
>    65791    0.295    0.000    0.903    0.000 Code.py:47(putln)
>    29677    0.289    0.000    0.915    0.000 Code.py:93(file_contents)
>     4724    0.287    0.000    0.292    0.000 Symtab.py:532(allocate_temp)
>    88070    0.247    0.000    0.247    0.000 ExprNodes.py:192(subexpr_nodes)
>    52071    0.232    0.000    0.232    0.000 Nodes.py:82(__init__)
> [...]
> """
>
> In total, I get an improvement of 12% in compilation time. That makes me think
> that it's actually worth putting the compilation into Cython's own setup.py,
> and installing the compiled Scanning module next to the Python one (Python
> prefers C extensions on import). Here's a patch, what do you think?

Maybe it'd be cool if one could instruct cython using comments, or
strings, so that one has one file, that works in regular python, and
you can use it to compile itself, but more optimized, than your patch
does. Otherwise, how can you improve the cythonized code (not having
to maintain the pure python and cython codes side by side)?

Ondrej

From robertwb at math.washington.edu  Thu Jun 12 02:06:32 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Jun 2008 17:06:32 -0700
Subject: [Cython] Profiling Cython
In-Reply-To: <85b5c3130806111651q1b074c4by78db19bd6f006260@mail.gmail.com>
References: <484BD7B0.80504@behnel.de>
	<85b5c3130806111651q1b074c4by78db19bd6f006260@mail.gmail.com>
Message-ID: <B77E5EDF-2A96-4D6B-848B-62CF51F4366F@math.washington.edu>

On Jun 11, 2008, at 4:51 PM, Ondrej Certik wrote:

> On Sun, Jun 8, 2008 at 2:59 PM, Stefan Behnel <stefan_ml at behnel.de>  
> wrote:
>> Hi,
>>
>> I did a tiny bit of profiling on Cython compiling lxml.etree. Here  
>> are the
>> numbers I get:
>>
>> """
>>         3127018 function calls (2777951 primitive calls) in 25.128  
>> CPU seconds
>>
>>   Ordered by: internal time, call count
>>   List reduced from 1035 to 20 due to restriction <20>
>>
>>   ncalls  tottime  percall  cumtime  percall filename:lineno 
>> (function)
>>   119144    4.579    0.000    4.671    0.000 Scanners.py:148 
>> (run_machine_inlined)
>>    18316    1.586    0.000    1.586    0.000 codecs.py:371(read)
>>   119144    1.074    0.000    5.746    0.000 Scanners.py:109 
>> (scan_a_token)
>>    88362    1.023    0.000    7.976    0.000 Scanners.py:88(read)
>>    29680    0.673    0.000    1.659    0.000 Code.py:103(mark_pos)
>>       77    0.588    0.008    0.588    0.008 posixpath.py:168(exists)
>>    88362    0.538    0.000    8.514    0.000 Scanning.py:397(next)
>> 23318/2985    0.495    0.000    0.517    0.000 Nodes.py:155(end_pos)
>>    70446    0.442    0.000    0.591    0.000 Code.py:62(put)
>> 90657/10562    0.351    0.000    4.549    0.000 Parsing.py:59 
>> (p_binop_expr)
>> [...]
>> """
>>
>> So the major headache here is Scanners.py in Plex. The method at  
>> the top-rank
>> is a huge function. According to the comments, it's the result of  
>> inlining a
>> couple of method calls that originally lead to slow code, and it  
>> looks heavily
>> profiled already.
>>
>> Assuming that further optimisation attempts were rather futile, I  
>> just
>> compiled the module with Cython. The first (obvious) result is  
>> that the
>> internal calls disappear from the profile log, as they are now  
>> internal C
>> calls. The call that remains is Scanner.next(), which originally  
>> took an
>> accumulated 8.5 seconds. In the compiled version, it's down to  
>> just over 5
>> seconds, that's more than 40 percent faster.
>>
>> """
>>         2595627 function calls (2246560 primitive calls) in 18.681  
>> CPU seconds
>>
>>   Ordered by: internal time, call count
>>   List reduced from 1028 to 20 due to restriction <20>
>>
>>   ncalls  tottime  percall  cumtime  percall filename:lineno 
>> (function)
>>    88362    4.246    0.000    5.041    0.000 Scanning.py:397(next)
>>    29680    0.673    0.000    1.632    0.000 Code.py:103(mark_pos)
>>    70446    0.439    0.000    0.586    0.000 Code.py:62(put)
>> 90657/10562    0.335    0.000    3.228    0.000 Parsing.py:59 
>> (p_binop_expr)
>> 23318/2985    0.316    0.000    0.338    0.000 Nodes.py:155(end_pos)
>>    65791    0.295    0.000    0.903    0.000 Code.py:47(putln)
>>    29677    0.289    0.000    0.915    0.000 Code.py:93 
>> (file_contents)
>>     4724    0.287    0.000    0.292    0.000 Symtab.py:532 
>> (allocate_temp)
>>    88070    0.247    0.000    0.247    0.000 ExprNodes.py:192 
>> (subexpr_nodes)
>>    52071    0.232    0.000    0.232    0.000 Nodes.py:82(__init__)
>> [...]
>> """
>>
>> In total, I get an improvement of 12% in compilation time. That  
>> makes me think
>> that it's actually worth putting the compilation into Cython's own  
>> setup.py,
>> and installing the compiled Scanning module next to the Python one  
>> (Python
>> prefers C extensions on import). Here's a patch, what do you think?

I was looking at this patch a bit earlier and have to say this is  
very neat.

> Maybe it'd be cool if one could instruct cython using comments, or
> strings, so that one has one file, that works in regular python, and
> you can use it to compile itself, but more optimized, than your patch
> does. Otherwise, how can you improve the cythonized code (not having
> to maintain the pure python and cython codes side by side)?


Hiding semantic information in comments and strings is something we  
want to avoid. However we have discussed doing this via decorators  
(which wouldn't do anything in Python, but would act as compiler  
hints to Cython). Functoin annotations ( http://www.python.org/dev/ 
peps/pep-3107/ ) and type inference could make a huge difference here  
as well.

- Robert



From robertwb at math.washington.edu  Thu Jun 12 02:32:56 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Jun 2008 17:32:56 -0700
Subject: [Cython] Cython 0.9.8 Release candidate
Message-ID: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>

You can download it at http://cython.org/Cython-0.9.8.tar.gz to try  
it out, or pull from http://hg.cython.org/cython-devel/

This release catches us up with all of the features of Pyrex 0.9.8 as  
well as numerous independent improvements.

- Robert


From greg.ewing at canterbury.ac.nz  Thu Jun 12 02:36:17 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 12 Jun 2008 12:36:17 +1200
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <e7ba66e40806101310u7c43fd83qbff3375d8954f072@mail.gmail.com>
References: <484EC802.4010105@behnel.de>
	<e7ba66e40806101143j7617b07dqb09672d0952e0c29@mail.gmail.com>
	<484ECC40.6090506@behnel.de>
	<e7ba66e40806101231s7af419b0x7671c5d045bd8200@mail.gmail.com>
	<484ED80B.5010004@behnel.de>
	<e7ba66e40806101310u7c43fd83qbff3375d8954f072@mail.gmail.com>
Message-ID: <48506F81.5030705@canterbury.ac.nz>

Lisandro Dalcin wrote:

> static PyTypeObject *__pyx_ptype_3foo_Foo;
> #define PyFoo_Type (*__pyx_ptype_3foo_Foo)

That's an implementation detail. The #define is there so that
the same code can be generated to refer to the type object
whether it's statically declared in the same module or
imported from another module.

There's no need for the #define in the case of a function,
because C automatically dereferences a function pointer if
you try to call it.

Having the function name #defined to a mangled name would
do no good, because if the name it's exported under is going
to clash with anything, the #defined name would clash in
the same way.

When you export an api for a C function, you're putting its
name in the C global namespace, and it's your responsibility
to ensure that it doesn't clash with anything.

-- 
Greg

From JSpurny at seznam.cz  Tue Jun  3 14:38:18 2008
From: JSpurny at seznam.cz (=?iso-8859-2?Q?Jan=20Spurn=FD?=)
Date: Tue, 03 Jun 2008 14:38:18 +0200 (CEST)
Subject: [Cython] public class methods
Message-ID: <976.1085-10028-1446183896-1212496695@seznam.cz>

Hi,

I just found cython and I'm trying to speedup my programs with it... I have several classes which are bottlenecks of my applications and I want to make them as fast as possible while keeping the same interface as old python classes had..

I would like to use cdef-ed methods in python
so I'm trying to do something like this:

a.pyx:
----------------------------------------------------
cdef class A:
    cdef public void a(A self):
        print 'method A::a'
        return
----------------------------------------------------

pa.py
----------------------------------------------------
from a import A

xa = A()
xa.a()
----------------------------------------------------

and it says:
----------------------------------------------------
python pa.py 
Traceback (most recent call last):
  File "pa.py", line 4, in <module>
    xa.a()
AttributeError: 'a.A' object has no attribute 'a'

----------------------------------------------------

thanks for any suggestions..

Jan Spurny

From jwienke at techfak.uni-bielefeld.de  Sun Jun  8 02:05:00 2008
From: jwienke at techfak.uni-bielefeld.de (Johannes Wienke)
Date: Sun, 08 Jun 2008 02:05:00 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <DDF9B460-E7B4-4B20-B873-59FEB9F470C2@math.washington.edu>
References: <484B189B.8030402@semipol.de>
	<DDF9B460-E7B4-4B20-B873-59FEB9F470C2@math.washington.edu>
Message-ID: <484B222C.9060205@techfak.uni-bielefeld.de>

Am 06/08/2008 02:00 AM schrieb Robert Bradshaw:
> Where are you setting your myData? If you do
> 
>     Foo().doSomething()
> 
> then it will segfault because the data isn't set yet. If you have
> 
>     def set_data_from_python(self, py_string):
>         self.myData = py_string
> 
> Then the conversion will happen, but myData is set to point to the 
> inside of py_string (i.e. no copying is actually done) and so the 
> instant py_string gets deallocated the actual char* gets reclaimed.

That's not the problem. My data is set by an external library that I 
have opened before. I've also double-checked that it is set. But somehow 
in another self than the one used when calling the data.

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080608/f9004674/attachment.pgp 

From robertwb at math.washington.edu  Thu Jun 12 03:01:44 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Jun 2008 18:01:44 -0700
Subject: [Cython] [Pyrex] ANN: Pyrex 0.9.7.1
In-Reply-To: <4829582C.4000905@users.sourceforge.net>
References: <48294510.5060407@canterbury.ac.nz>
	<4829582C.4000905@users.sourceforge.net>
Message-ID: <CDF81BDE-6D9E-4C3B-A2A0-0C04C51662D7@math.washington.edu>

FYI, I just tried this out and it does work in the latest Cython.  
(Probably fixed in Pyrex by now too.)

- Robert


On May 13, 2008, at 1:58 AM, S?bastien Sabl? wrote:

> Hi Greg,
>
> I tried both Pyrex 0.9.7 and 0.9.7.1, but I have an error which didn't
> occurred with previous versions:
> TypeError: 'dict' object does not support item assignment"
>
> The code where the problem happens looks like this:
>
> cdef reflect_bind(wrapped, void *addr, object o):
>      wrapped[<int>addr] = o
>
>
> By the way, thanks for Pyrex, it is a great tool; we are using it to
> migrate a big C application to Python and Pyrex is making our life  
> a lot
> easier.
>
> Thanks in advance
>
> --
> S?bastien Sabl?
>
>
> Greg Ewing a ?crit :
>> Pyrex 0.9.7.1 is now available:
>>
>>    http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/
>>
>> This version fixes a bug in the new integer indexing
>> optimisation which causes indexing of a non-sequence type
>> with a C int to fail with a TypeError.
>>
>> What is Pyrex?
>> --------------
>>
>> Pyrex is a language for writing Python extension modules.
>> It lets you freely mix operations on Python and C data, with
>> all Python reference counting and error checking handled
>> automatically.
>>
>> _______________________________________________
>> Pyrex mailing list
>> Pyrex at lists.copyleft.no
>> http://lists.copyleft.no/mailman/listinfo/pyrex
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Thu Jun 12 03:12:03 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Jun 2008 18:12:03 -0700
Subject: [Cython] public class methods
In-Reply-To: <976.1085-10028-1446183896-1212496695@seznam.cz>
References: <976.1085-10028-1446183896-1212496695@seznam.cz>
Message-ID: <A9FDFD3A-361C-4ADB-9F5F-B7D8AD830666@math.washington.edu>

On Jun 3, 2008, at 5:38 AM, Jan Spurn? wrote:

> Hi,
>
> I just found cython and I'm trying to speedup my programs with  
> it... I have several classes which are bottlenecks of my  
> applications and I want to make them as fast as possible while  
> keeping the same interface as old python classes had..

Sounds like a perfect fit for Cython.

> I would like to use cdef-ed methods in python
> so I'm trying to do something like this:
>
> a.pyx:
> ----------------------------------------------------
> cdef class A:
>     cdef public void a(A self):
>         print 'method A::a'
>         return
> ----------------------------------------------------
>
> pa.py
> ----------------------------------------------------
> from a import A
>
> xa = A()
> xa.a()
> ----------------------------------------------------
>
> and it says:
> ----------------------------------------------------
> python pa.py
> Traceback (most recent call last):
>   File "pa.py", line 4, in <module>
>     xa.a()
> AttributeError: 'a.A' object has no attribute 'a'
>
> ----------------------------------------------------
>
> thanks for any suggestions..

The problem here is that cdef methods cannot be called from Python-- 
they are literally C functions only visible from C. You need either  
normal "def" functions (which still get compiled to fast C) or  
"cpdef" functions that are a hybred of cdef and def. What you  
probably want to do is

cdef class A:
     cpdef a(self):
         # do stuff

which will compile the method a() into fast c code but still be  
accessible from Python, and if you call it from Cython then it avoids  
Python calling conventions. Note that "self" is automatically of type  
A, and the public keyword is something completely separate (used for  
interfacing with outside C code).

- Robert


From ondrej at certik.cz  Thu Jun 12 03:41:58 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 12 Jun 2008 03:41:58 +0200
Subject: [Cython] Profiling Cython
In-Reply-To: <B77E5EDF-2A96-4D6B-848B-62CF51F4366F@math.washington.edu>
References: <484BD7B0.80504@behnel.de>
	<85b5c3130806111651q1b074c4by78db19bd6f006260@mail.gmail.com>
	<B77E5EDF-2A96-4D6B-848B-62CF51F4366F@math.washington.edu>
Message-ID: <85b5c3130806111841t7d55b7e4sdfdb34a12d123b57@mail.gmail.com>

On Thu, Jun 12, 2008 at 2:06 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Jun 11, 2008, at 4:51 PM, Ondrej Certik wrote:
>
>> On Sun, Jun 8, 2008 at 2:59 PM, Stefan Behnel <stefan_ml at behnel.de>
>> wrote:
>>> Hi,
>>>
>>> I did a tiny bit of profiling on Cython compiling lxml.etree. Here
>>> are the
>>> numbers I get:
>>>
>>> """
>>>         3127018 function calls (2777951 primitive calls) in 25.128
>>> CPU seconds
>>>
>>>   Ordered by: internal time, call count
>>>   List reduced from 1035 to 20 due to restriction <20>
>>>
>>>   ncalls  tottime  percall  cumtime  percall filename:lineno
>>> (function)
>>>   119144    4.579    0.000    4.671    0.000 Scanners.py:148
>>> (run_machine_inlined)
>>>    18316    1.586    0.000    1.586    0.000 codecs.py:371(read)
>>>   119144    1.074    0.000    5.746    0.000 Scanners.py:109
>>> (scan_a_token)
>>>    88362    1.023    0.000    7.976    0.000 Scanners.py:88(read)
>>>    29680    0.673    0.000    1.659    0.000 Code.py:103(mark_pos)
>>>       77    0.588    0.008    0.588    0.008 posixpath.py:168(exists)
>>>    88362    0.538    0.000    8.514    0.000 Scanning.py:397(next)
>>> 23318/2985    0.495    0.000    0.517    0.000 Nodes.py:155(end_pos)
>>>    70446    0.442    0.000    0.591    0.000 Code.py:62(put)
>>> 90657/10562    0.351    0.000    4.549    0.000 Parsing.py:59
>>> (p_binop_expr)
>>> [...]
>>> """
>>>
>>> So the major headache here is Scanners.py in Plex. The method at
>>> the top-rank
>>> is a huge function. According to the comments, it's the result of
>>> inlining a
>>> couple of method calls that originally lead to slow code, and it
>>> looks heavily
>>> profiled already.
>>>
>>> Assuming that further optimisation attempts were rather futile, I
>>> just
>>> compiled the module with Cython. The first (obvious) result is
>>> that the
>>> internal calls disappear from the profile log, as they are now
>>> internal C
>>> calls. The call that remains is Scanner.next(), which originally
>>> took an
>>> accumulated 8.5 seconds. In the compiled version, it's down to
>>> just over 5
>>> seconds, that's more than 40 percent faster.
>>>
>>> """
>>>         2595627 function calls (2246560 primitive calls) in 18.681
>>> CPU seconds
>>>
>>>   Ordered by: internal time, call count
>>>   List reduced from 1028 to 20 due to restriction <20>
>>>
>>>   ncalls  tottime  percall  cumtime  percall filename:lineno
>>> (function)
>>>    88362    4.246    0.000    5.041    0.000 Scanning.py:397(next)
>>>    29680    0.673    0.000    1.632    0.000 Code.py:103(mark_pos)
>>>    70446    0.439    0.000    0.586    0.000 Code.py:62(put)
>>> 90657/10562    0.335    0.000    3.228    0.000 Parsing.py:59
>>> (p_binop_expr)
>>> 23318/2985    0.316    0.000    0.338    0.000 Nodes.py:155(end_pos)
>>>    65791    0.295    0.000    0.903    0.000 Code.py:47(putln)
>>>    29677    0.289    0.000    0.915    0.000 Code.py:93
>>> (file_contents)
>>>     4724    0.287    0.000    0.292    0.000 Symtab.py:532
>>> (allocate_temp)
>>>    88070    0.247    0.000    0.247    0.000 ExprNodes.py:192
>>> (subexpr_nodes)
>>>    52071    0.232    0.000    0.232    0.000 Nodes.py:82(__init__)
>>> [...]
>>> """
>>>
>>> In total, I get an improvement of 12% in compilation time. That
>>> makes me think
>>> that it's actually worth putting the compilation into Cython's own
>>> setup.py,
>>> and installing the compiled Scanning module next to the Python one
>>> (Python
>>> prefers C extensions on import). Here's a patch, what do you think?
>
> I was looking at this patch a bit earlier and have to say this is
> very neat.
>
>> Maybe it'd be cool if one could instruct cython using comments, or
>> strings, so that one has one file, that works in regular python, and
>> you can use it to compile itself, but more optimized, than your patch
>> does. Otherwise, how can you improve the cythonized code (not having
>> to maintain the pure python and cython codes side by side)?
>
>
> Hiding semantic information in comments and strings is something we
> want to avoid. However we have discussed doing this via decorators
> (which wouldn't do anything in Python, but would act as compiler
> hints to Cython). Functoin annotations ( http://www.python.org/dev/
> peps/pep-3107/ ) and type inference could make a huge difference here
> as well.

Yes, that's even a better solution.

Ondrej

From gfurnish at gfurnish.net  Thu Jun 12 04:55:59 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Wed, 11 Jun 2008 19:55:59 -0700
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
Message-ID: <8f8f8530806111955u4feafb3dp7d728f3a965039cf@mail.gmail.com>

This completely breaks the symbolics rewrite in Sage, and requires a
removal of site-packages/cython and reinstall of old version to fix
the damage.  (Specifically it seems to break module level cpdef
functions in some of the files in the circular def stuff, but I'm not
convinced that module level cpdef functions arn't broke on their own)

On Wed, Jun 11, 2008 at 5:32 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> You can download it at http://cython.org/Cython-0.9.8.tar.gz to try
> it out, or pull from http://hg.cython.org/cython-devel/
>
> This release catches us up with all of the features of Pyrex 0.9.8 as
> well as numerous independent improvements.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Thu Jun 12 05:05:03 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Jun 2008 20:05:03 -0700
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <8f8f8530806111955u4feafb3dp7d728f3a965039cf@mail.gmail.com>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
	<8f8f8530806111955u4feafb3dp7d728f3a965039cf@mail.gmail.com>
Message-ID: <BD0E2782-BE60-4711-A411-2810E1E5737A@math.washington.edu>

Thanks for the report. That's what release candidates are for, right :-)

I compiled Sage a couple of revisions back, and it worked fine, and  
did another sage -ba when I sent out this email (and just now  
checked, it has the same problem with module-level cpdef functions).  
I'm going to try to sort this out and I'll get back to you so we can  
be sure it works with your symbolics.

As for nixing site-packages/cython I'm not sure what would cause that...

- Robert

On Jun 11, 2008, at 7:55 PM, Gary Furnish wrote:

> This completely breaks the symbolics rewrite in Sage, and requires a
> removal of site-packages/cython and reinstall of old version to fix
> the damage.  (Specifically it seems to break module level cpdef
> functions in some of the files in the circular def stuff, but I'm not
> convinced that module level cpdef functions arn't broke on their own)
>
> On Wed, Jun 11, 2008 at 5:32 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> You can download it at http://cython.org/Cython-0.9.8.tar.gz to try
>> it out, or pull from http://hg.cython.org/cython-devel/
>>
>> This release catches us up with all of the features of Pyrex 0.9.8 as
>> well as numerous independent improvements.
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From gfurnish at gfurnish.net  Thu Jun 12 05:18:29 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Wed, 11 Jun 2008 20:18:29 -0700
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <BD0E2782-BE60-4711-A411-2810E1E5737A@math.washington.edu>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
	<8f8f8530806111955u4feafb3dp7d728f3a965039cf@mail.gmail.com>
	<BD0E2782-BE60-4711-A411-2810E1E5737A@math.washington.edu>
Message-ID: <8f8f8530806112018l5279ddfaod5f76764039efe13@mail.gmail.com>

Well it looked like there was a file in site-packages that was causing
the problem that was not there in the last revision.

On Wed, Jun 11, 2008 at 8:05 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Thanks for the report. That's what release candidates are for, right :-)
>
> I compiled Sage a couple of revisions back, and it worked fine, and
> did another sage -ba when I sent out this email (and just now
> checked, it has the same problem with module-level cpdef functions).
> I'm going to try to sort this out and I'll get back to you so we can
> be sure it works with your symbolics.
>
> As for nixing site-packages/cython I'm not sure what would cause that...
>
> - Robert
>
> On Jun 11, 2008, at 7:55 PM, Gary Furnish wrote:
>
>> This completely breaks the symbolics rewrite in Sage, and requires a
>> removal of site-packages/cython and reinstall of old version to fix
>> the damage.  (Specifically it seems to break module level cpdef
>> functions in some of the files in the circular def stuff, but I'm not
>> convinced that module level cpdef functions arn't broke on their own)
>>
>> On Wed, Jun 11, 2008 at 5:32 PM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>> You can download it at http://cython.org/Cython-0.9.8.tar.gz to try
>>> it out, or pull from http://hg.cython.org/cython-devel/
>>>
>>> This release catches us up with all of the features of Pyrex 0.9.8 as
>>> well as numerous independent improvements.
>>>
>>> - Robert
>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Thu Jun 12 05:38:08 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 11 Jun 2008 20:38:08 -0700
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <8f8f8530806112018l5279ddfaod5f76764039efe13@mail.gmail.com>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
	<8f8f8530806111955u4feafb3dp7d728f3a965039cf@mail.gmail.com>
	<BD0E2782-BE60-4711-A411-2810E1E5737A@math.washington.edu>
	<8f8f8530806112018l5279ddfaod5f76764039efe13@mail.gmail.com>
Message-ID: <1C2FE7CF-0898-4841-8435-115389D3F848@math.washington.edu>

Module-level cpdef functions fixed, new patch and tar up at the same  
place.

On Jun 11, 2008, at 8:18 PM, Gary Furnish wrote:

> Well it looked like there was a file in site-packages that was causing
> the problem that was not there in the last revision.

Odd...any suggestions on how to fix this?

>
> On Wed, Jun 11, 2008 at 8:05 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> Thanks for the report. That's what release candidates are for,  
>> right :-)
>>
>> I compiled Sage a couple of revisions back, and it worked fine, and
>> did another sage -ba when I sent out this email (and just now
>> checked, it has the same problem with module-level cpdef functions).
>> I'm going to try to sort this out and I'll get back to you so we can
>> be sure it works with your symbolics.
>>
>> As for nixing site-packages/cython I'm not sure what would cause  
>> that...
>>
>> - Robert
>>
>> On Jun 11, 2008, at 7:55 PM, Gary Furnish wrote:
>>
>>> This completely breaks the symbolics rewrite in Sage, and requires a
>>> removal of site-packages/cython and reinstall of old version to fix
>>> the damage.  (Specifically it seems to break module level cpdef
>>> functions in some of the files in the circular def stuff, but I'm  
>>> not
>>> convinced that module level cpdef functions arn't broke on their  
>>> own)
>>>
>>> On Wed, Jun 11, 2008 at 5:32 PM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>>> You can download it at http://cython.org/Cython-0.9.8.tar.gz to try
>>>> it out, or pull from http://hg.cython.org/cython-devel/
>>>>
>>>> This release catches us up with all of the features of Pyrex  
>>>> 0.9.8 as
>>>> well as numerous independent improvements.
>>>>
>>>> - Robert
>>>>
>>>> _______________________________________________
>>>> Cython-dev mailing list
>>>> Cython-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Thu Jun 12 07:44:18 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 12 Jun 2008 07:44:18 +0200
Subject: [Cython] Accessing C attributes from Python functions in the
 interpreter
In-Reply-To: <484B222C.9060205@techfak.uni-bielefeld.de>
References: <484B189B.8030402@semipol.de>	<DDF9B460-E7B4-4B20-B873-59FEB9F470C2@math.washington.edu>
	<484B222C.9060205@techfak.uni-bielefeld.de>
Message-ID: <4850B7B2.8000603@behnel.de>

Hi,

Johannes Wienke wrote:
> My data is set by an external library that I
> have opened before. I've also double-checked that it is set. But somehow
> in another self than the one used when calling the data.

I see two possibilities:

a) It's not the same object. Try to make sure that you don't re-create it
somewhere else. (e.g. put a __cinit__ method in there and print "I'm alive!")

b) You call the method from C code or otherwise by hand and pass the wrong
instance as self. Print the type of the object that self points to when the
method is called.

Stefan


From stefan_ml at behnel.de  Thu Jun 12 08:20:58 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 12 Jun 2008 08:20:58 +0200
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
Message-ID: <4850C04A.7000806@behnel.de>

Hi Robert,

Robert Bradshaw wrote:
> You can download it at http://cython.org/Cython-0.9.8.tar.gz to try  
> it out, or pull from http://hg.cython.org/cython-devel/
> 
> This release catches us up with all of the features of Pyrex 0.9.8 as  
> well as numerous independent improvements.

Thanks for the hard work. Works perfectly for me.

Stefan


From stefan_ml at behnel.de  Thu Jun 12 08:28:52 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 12 Jun 2008 08:28:52 +0200
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <484EC802.4010105@behnel.de>
References: <484EC802.4010105@behnel.de>
Message-ID: <4850C224.3060206@behnel.de>

Hi,

Stefan Behnel wrote:
> http://www.python.org/dev/peps/pep-3121/
> 
> To be implemented one day.

... or rather earlier. What I didn't realise when I read the PEP was that this
was supposed to *replace* the current API, not to complement it. This means
that Py_InitModule4() is now gone, and Cython modules won't work on Py3beta
any more. Great...

I'll dig into this to see if I can come up with a quick fix, although I doubt
it. In any case, I do not consider this release critical. It's fine if
everything compiles nicely on Py2.6 for now.

Stefan

From stefan_ml at behnel.de  Thu Jun 12 09:35:51 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 12 Jun 2008 09:35:51 +0200
Subject: [Cython] PEP 3121 on extension module initialisation in Py3
In-Reply-To: <4850C224.3060206@behnel.de>
References: <484EC802.4010105@behnel.de> <4850C224.3060206@behnel.de>
Message-ID: <4850D1D7.4020301@behnel.de>

Hi again,

Stefan Behnel wrote:
>> http://www.python.org/dev/peps/pep-3121/
> What I didn't realise when I read the PEP was that this
> was supposed to *replace* the current API, not to complement it. This means
> that Py_InitModule4() is now gone, and Cython modules won't work on Py3beta
> any more.

I worked around this in rev 719. It doesn't currently use any local per-module
instance state or garbage collection support, the PyModuleDef struct is only
used to pass on the information that was there already. But this can be built
upon later on. It compiles and works for me, that's what counts.

Stefan

From stefan_ml at behnel.de  Thu Jun 12 11:02:12 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 12 Jun 2008 11:02:12 +0200
Subject: [Cython] Type casting issues and strict aliasing
Message-ID: <4850E614.1090806@behnel.de>

Hi,

I've run into a couple of strict aliasing issues with type casts for the vtab
struct. Can anyone with more C experience than me comment on these?

For the "r_jiba1" test case in

http://hg.cython.org/cython-devel/file/e21391d5f23a/tests/run/r_jiba1.pyx

Cython currently generates this in the tp_new function of the Norwegian class:

  *(struct __pyx_vtabstruct_7r_jiba1_Norwegian **)&p->__pyx_base.__pyx_vtab =
        __pyx_vtabptr_7r_jiba1_Norwegian;

which results in this strict aliasing warning with gcc 4.1:

  r_jiba1.c: In function '__pyx_tp_new_7r_jiba1_Norwegian':
  r_jiba1.c:596: warning: dereferencing type-punned pointer will break
          strict-aliasing rules

It goes away when I change it to this (patch is attached):

  p->__pyx_base.__pyx_vtab =
   (struct __pyx_vtabstruct_7r_jiba1_Parrot*)__pyx_vtabptr_7r_jiba1_Norwegian;

Parrot is the base type here. Since these changes usually tend to break some
compiler somewhere, I hesitate to apply the patch right away, though...


Similar things happen when the C methods are initialised in the module init
function. Currently, we have:

  *(void(**)(void))&__pyx_vtable_7r_jiba1_Parrot.describe =
                    (void(*)(void))__pyx_f_7r_jiba1_6Parrot_describe;
  *(void(**)(void))&__pyx_vtable_7r_jiba1_Norwegian.__pyx_base.describe =
                    (void(*)(void))__pyx_f_7r_jiba1_9Norwegian_describe;

and the resulting warning is:

  r_jiba1.c: In function 'PyInit_r_jiba1':
  r_jiba1.c:800: warning: dereferencing type-punned pointer will break
          strict-aliasing rules
  r_jiba1.c:807: warning: dereferencing type-punned pointer will break
          strict-aliasing rules

The problem here is mostly the "self" argument, which has the type of the
concrete class for each subtype. The code that generates this in the
generate_exttype_vtable_init_code() function in ModuleNode.py has the type
casts hard-coded. What would be the right thing to do here? Cast one of the
functions (rhs?) to the type of the other?

Any help is appreciated.

Thanks,
Stefan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: vtab-typecast.patch
Type: text/x-patch
Size: 930 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080612/06757f3d/attachment-0001.bin 

From dalcinl at gmail.com  Thu Jun 12 15:33:25 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Jun 2008 10:33:25 -0300
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <4850E614.1090806@behnel.de>
References: <4850E614.1090806@behnel.de>
Message-ID: <e7ba66e40806120633w7142e5e3td172321376df9059@mail.gmail.com>

On 6/12/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>  Cython currently generates this in the tp_new function of the Norwegian class:
>
>   *(struct __pyx_vtabstruct_7r_jiba1_Norwegian **)&p->__pyx_base.__pyx_vtab =
>         __pyx_vtabptr_7r_jiba1_Norwegian;
>
>  which results in this strict aliasing warning with gcc 4.1:
>
>   r_jiba1.c: In function '__pyx_tp_new_7r_jiba1_Norwegian':
>   r_jiba1.c:596: warning: dereferencing type-punned pointer will break
>           strict-aliasing rules

I do not see way this is wrong, but anyway, the generated code is
confusing, you have a lhs cast and a dereferencing in the same line.

>  It goes away when I change it to this (patch is attached):
>
>   p->__pyx_base.__pyx_vtab =
>    (struct __pyx_vtabstruct_7r_jiba1_Parrot*)__pyx_vtabptr_7r_jiba1_Norwegian;
>

And this is right, and much clear. In fact, it is equivalent to the following:

PyTypeObject * tp = &PyList_Type;
PyObject *p = (PyObject *) tp.

So I do not expect that to NEVER break. If after the patch all tests
pass, I would say you have to push it.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Jun 12 16:43:34 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 12 Jun 2008 16:43:34 +0200
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <e7ba66e40806120633w7142e5e3td172321376df9059@mail.gmail.com>
References: <4850E614.1090806@behnel.de>
	<e7ba66e40806120633w7142e5e3td172321376df9059@mail.gmail.com>
Message-ID: <48513616.8040203@behnel.de>

Hi,

Lisandro Dalcin wrote:
> If after the patch all tests pass, I would say you have to push it.

Ok, thanks, that's what I did. However, that was the easy one. I'm having a
pretty hard time figuring out what to do about the second problem.

Stefan

From stefan_ml at behnel.de  Thu Jun 12 21:39:49 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 12 Jun 2008 21:39:49 +0200
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <4850E614.1090806@behnel.de>
References: <4850E614.1090806@behnel.de>
Message-ID: <48517B85.6030900@behnel.de>

Hi,

regarding the second issue below, a simple cast to a (void*) works for me:

  __pyx_vtable_7r_jiba1_Norwegian.__pyx_base.describe =
                             (void*)__pyx_f_7r_jiba1_9Norwegian_describe;

Again, this is with gcc 4.1 and Py3.0 SVN. Are there any comments on this one?
What would other compilers say about this?

Stefan


Stefan Behnel wrote:
> Similar things happen when the C methods are initialised in the module init
> function. Currently, we have:
> 
>   *(void(**)(void))&__pyx_vtable_7r_jiba1_Parrot.describe =
>                     (void(*)(void))__pyx_f_7r_jiba1_6Parrot_describe;
>   *(void(**)(void))&__pyx_vtable_7r_jiba1_Norwegian.__pyx_base.describe =
>                     (void(*)(void))__pyx_f_7r_jiba1_9Norwegian_describe;
> 
> and the resulting warning is:
> 
>   r_jiba1.c: In function 'PyInit_r_jiba1':
>   r_jiba1.c:800: warning: dereferencing type-punned pointer will break
>           strict-aliasing rules
>   r_jiba1.c:807: warning: dereferencing type-punned pointer will break
>           strict-aliasing rules
> 
> The problem here is mostly the "self" argument, which has the type of the
> concrete class for each subtype. The code that generates this in the
> generate_exttype_vtable_init_code() function in ModuleNode.py has the type
> casts hard-coded. What would be the right thing to do here? Cast one of the
> functions (rhs?) to the type of the other?


From dalcinl at gmail.com  Thu Jun 12 22:40:19 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Jun 2008 17:40:19 -0300
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <48517B85.6030900@behnel.de>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
Message-ID: <e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>

On 6/12/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>  regarding the second issue below, a simple cast to a (void*) works for me:

No, No... I believe casting funtion pointers to void* pointers is not
valid C99 code. In short, (void*) is a pointer to 'data',
void(*)(void) is a pointer to 'code'.

Make sure you pass all the following flags to GCC:

 -Wall -pedantic -std=c99


>   __pyx_vtable_7r_jiba1_Norwegian.__pyx_base.describe =
>
>                              (void*)__pyx_f_7r_jiba1_9Norwegian_describe;
>
>  Again, this is with gcc 4.1 and Py3.0 SVN. Are there any comments on this one?
>  What would other compilers say about this?

Well, in this case, the fix is not so easy :-( . I can imagine of a
hackery that could work, but this requires that you can generate the C
types for both functions, that is, you can write (here unmangled for
clarity)

union {
 void (*fp_base)(struct Parrot*);
 void (*fp)(struct Norwegian*);
} describe_Parrot_Norwegian

And then, after introducing a new C scope (that is, beteen braces {}
), use the following:

{
  describe_Parrot_Norwegian tmp;
  tmp.fp = Norwegian_describe
  vtable_Parrot.describe = tmp.fp_base;
}

Vile but cute hackery using type punning ;-) . I do not know of any
other form that would actually works. The C standard even says that
the result of that is undefined, but I expect that would not be the
case in almost all the compilers out there. Check GCC info pages, they
say that.


>
>  Stefan Behnel wrote:
>  > Similar things happen when the C methods are initialised in the module init
>  > function. Currently, we have:
>  >
>  >   *(void(**)(void))&__pyx_vtable_7r_jiba1_Parrot.describe =
>  >                     (void(*)(void))__pyx_f_7r_jiba1_6Parrot_describe;
>  >   *(void(**)(void))&__pyx_vtable_7r_jiba1_Norwegian.__pyx_base.describe =
>  >                     (void(*)(void))__pyx_f_7r_jiba1_9Norwegian_describe;
>  >
>  > and the resulting warning is:
>  >
>  >   r_jiba1.c: In function 'PyInit_r_jiba1':
>  >   r_jiba1.c:800: warning: dereferencing type-punned pointer will break
>  >           strict-aliasing rules
>  >   r_jiba1.c:807: warning: dereferencing type-punned pointer will break
>  >           strict-aliasing rules
>  >
>  > The problem here is mostly the "self" argument, which has the type of the
>  > concrete class for each subtype. The code that generates this in the
>  > generate_exttype_vtable_init_code() function in ModuleNode.py has the type
>  > casts hard-coded. What would be the right thing to do here? Cast one of the
>  > functions (rhs?) to the type of the other?
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From cwitty at newtonlabs.com  Fri Jun 13 00:08:29 2008
From: cwitty at newtonlabs.com (Carl Witty)
Date: Thu, 12 Jun 2008 15:08:29 -0700
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
Message-ID: <ac1e4c420806121508m4c7c14c2s331df86a8d531708@mail.gmail.com>

On Thu, Jun 12, 2008 at 1:40 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 6/12/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Hi,
>>
>>  regarding the second issue below, a simple cast to a (void*) works for me:
>
> No, No... I believe casting funtion pointers to void* pointers is not
> valid C99 code. In short, (void*) is a pointer to 'data',
> void(*)(void) is a pointer to 'code'.
>
> Make sure you pass all the following flags to GCC:
>
>  -Wall -pedantic -std=c99
>
>
>>   __pyx_vtable_7r_jiba1_Norwegian.__pyx_base.describe =
>>
>>                              (void*)__pyx_f_7r_jiba1_9Norwegian_describe;
>>
>>  Again, this is with gcc 4.1 and Py3.0 SVN. Are there any comments on this one?
>>  What would other compilers say about this?
>
> Well, in this case, the fix is not so easy :-( . I can imagine of a
> hackery that could work, but this requires that you can generate the C
> types for both functions, that is, you can write (here unmangled for
> clarity)
>
> union {
>  void (*fp_base)(struct Parrot*);
>  void (*fp)(struct Norwegian*);
> } describe_Parrot_Norwegian
>
> And then, after introducing a new C scope (that is, beteen braces {}
> ), use the following:
>
> {
>  describe_Parrot_Norwegian tmp;
>  tmp.fp = Norwegian_describe
>  vtable_Parrot.describe = tmp.fp_base;
> }
>
> Vile but cute hackery using type punning ;-) . I do not know of any
> other form that would actually works. The C standard even says that
> the result of that is undefined, but I expect that would not be the
> case in almost all the compilers out there. Check GCC info pages, they
> say that.

I don't think the solution is nearly so complicated.  Can't you just
cast the function to the correct type?

   vtable_Parrot.describe = (void (*)(struct Parrot *))Norwegian_describe;

Carl

From dalcinl at gmail.com  Fri Jun 13 01:14:10 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 12 Jun 2008 20:14:10 -0300
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <ac1e4c420806121508m4c7c14c2s331df86a8d531708@mail.gmail.com>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
	<ac1e4c420806121508m4c7c14c2s331df86a8d531708@mail.gmail.com>
Message-ID: <e7ba66e40806121614o101f5065oc2f5ad1351d3b3a1@mail.gmail.com>

On 6/12/08, Carl Witty <cwitty at newtonlabs.com> wrote:
> I don't think the solution is nearly so complicated.  Can't you just
>  cast the function to the correct type?
>
>    vtable_Parrot.describe = (void (*)(struct Parrot *))Norwegian_describe;

Well, the type of 'Norwegian_describe' is acctually 'void (*)(struct
*Norwegian)'. If those types aren't aliasables, then that will break
strict aliasing rules. What I do not know if in this case the funtions
types can alias each other (I guess they cannot).

In the end, I do not expect any compiler to generate bad code for this
particular case, but perhaps we still get the nasty GCC warnings.
IMHO, all Cython-generated code should compile free of warnings...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Fri Jun 13 03:28:43 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 13 Jun 2008 13:28:43 +1200
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <48517B85.6030900@behnel.de>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
Message-ID: <4851CD4B.40906@canterbury.ac.nz>

Stefan Behnel wrote:
> Hi,
> 
> regarding the second issue below, a simple cast to a (void*) works for me:
> 
>   __pyx_vtable_7r_jiba1_Norwegian.__pyx_base.describe =
>                              (void*)__pyx_f_7r_jiba1_9Norwegian_describe;

I suspect that won't work when compiling as C++, because
you can only implicitly cast *to* void * in C++, not the
other way.

I think to fix this properly you'll have to figure out
what the actual type of the LHS is and cast to that.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Jun 13 03:35:34 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 13 Jun 2008 13:35:34 +1200
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
Message-ID: <4851CEE6.9010207@canterbury.ac.nz>

Lisandro Dalcin wrote:

> I believe casting funtion pointers to void* pointers is not
> valid C99 code. In short, (void*) is a pointer to 'data',
> void(*)(void) is a pointer to 'code'.

Yeah, I discovered that too when trying to compile with
-pedantic.

Does anyone know what the correct way to spell "pointer to
any function" is in C99? It seems like just as legitimate
a notion to me as "pointer to any data", but I can't find
a way of expressing it.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri Jun 13 03:52:11 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 13 Jun 2008 13:52:11 +1200
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <e7ba66e40806121614o101f5065oc2f5ad1351d3b3a1@mail.gmail.com>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
	<ac1e4c420806121508m4c7c14c2s331df86a8d531708@mail.gmail.com>
	<e7ba66e40806121614o101f5065oc2f5ad1351d3b3a1@mail.gmail.com>
Message-ID: <4851D2CB.7040900@canterbury.ac.nz>

Lisandro Dalcin wrote:

> IMHO, all Cython-generated code should compile free of warnings...

That's a nice ideal, but it seems to be increasingly
difficult to achieve with the combination of (a) more
and more pedantic little rules being added to the C
standard, (b) the implementers of GCC insisting on
warning about all of them, and (c) people insisting
on compiling Pyrex/Cython generated code with every
possible warning turned on.

-- 
Greg


From gfurnish at gfurnish.net  Fri Jun 13 04:56:01 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Thu, 12 Jun 2008 19:56:01 -0700
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <4850C04A.7000806@behnel.de>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>
	<4850C04A.7000806@behnel.de>
Message-ID: <8f8f8530806121956p2c6e27d1jcfdbab7cd6e175a9@mail.gmail.com>

This seems to break padics:

from __future__ import with_statement
                                    ^
------------------------------------------------------------

/home/gfurnish/sage-3.0.2-symbolics/devel/sage-symbolics/sage/rings/padics/padic_generic_element.pyx:19:37:
future feature with_statement is not defined


On Wed, Jun 11, 2008 at 11:20 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi Robert,
>
> Robert Bradshaw wrote:
>> You can download it at http://cython.org/Cython-0.9.8.tar.gz to try
>> it out, or pull from http://hg.cython.org/cython-devel/
>>
>> This release catches us up with all of the features of Pyrex 0.9.8 as
>> well as numerous independent improvements.
>
> Thanks for the hard work. Works perfectly for me.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From cwitty at newtonlabs.com  Fri Jun 13 05:20:42 2008
From: cwitty at newtonlabs.com (Carl Witty)
Date: Thu, 12 Jun 2008 20:20:42 -0700
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <e7ba66e40806121614o101f5065oc2f5ad1351d3b3a1@mail.gmail.com>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
	<ac1e4c420806121508m4c7c14c2s331df86a8d531708@mail.gmail.com>
	<e7ba66e40806121614o101f5065oc2f5ad1351d3b3a1@mail.gmail.com>
Message-ID: <ac1e4c420806122020j172ec706qdc0374d6820b1c4a@mail.gmail.com>

On Thu, Jun 12, 2008 at 4:14 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 6/12/08, Carl Witty <cwitty at newtonlabs.com> wrote:
>> I don't think the solution is nearly so complicated.  Can't you just
>>  cast the function to the correct type?
>>
>>    vtable_Parrot.describe = (void (*)(struct Parrot *))Norwegian_describe;
>
> Well, the type of 'Norwegian_describe' is acctually 'void (*)(struct
> *Norwegian)'. If those types aren't aliasables, then that will break
> strict aliasing rules. What I do not know if in this case the funtions
> types can alias each other (I guess they cannot).

No; you only break aliasing rules if you read or write a value through
a pointer of the wrong type.  The above code doesn't do that.

Carl

From cwitty at newtonlabs.com  Fri Jun 13 05:23:31 2008
From: cwitty at newtonlabs.com (Carl Witty)
Date: Thu, 12 Jun 2008 20:23:31 -0700
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <4851CEE6.9010207@canterbury.ac.nz>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
	<4851CEE6.9010207@canterbury.ac.nz>
Message-ID: <ac1e4c420806122023o5906cda1m5bf723d2d039c3df@mail.gmail.com>

On Thu, Jun 12, 2008 at 6:35 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Does anyone know what the correct way to spell "pointer to
> any function" is in C99? It seems like just as legitimate
> a notion to me as "pointer to any data", but I can't find
> a way of expressing it.

I think you can just pick some arbitrary function pointer type, like
(void (*)(void)).

Carl

From stefan_ml at behnel.de  Fri Jun 13 08:22:17 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 13 Jun 2008 08:22:17 +0200
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <8f8f8530806121956p2c6e27d1jcfdbab7cd6e175a9@mail.gmail.com>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>	<4850C04A.7000806@behnel.de>
	<8f8f8530806121956p2c6e27d1jcfdbab7cd6e175a9@mail.gmail.com>
Message-ID: <48521219.8090407@behnel.de>

Hi,

Gary Furnish wrote:
> This seems to break padics:
> 
> from __future__ import with_statement
>                                     ^
> ------------------------------------------------------------
> 
> /home/gfurnish/sage-3.0.2-symbolics/devel/sage-symbolics/sage/rings/padics/padic_generic_element.pyx:19:37:
> future feature with_statement is not defined

That line shouldn't be there. Cython doesn't support the with statement anyway
(even if "with nogil" looks similar).

Stefan


From stefan_ml at behnel.de  Fri Jun 13 08:41:30 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 13 Jun 2008 08:41:30 +0200
Subject: [Cython] Cython 0.9.8 released
Message-ID: <4852169A.1050706@behnel.de>

Hi all,

Cython 0.9.8 has been released. A lot of work by Robert, Greg and others
(inluding myself) went into this release to fix many bugs and add tons of
great features.

This release features

- everything that was done in Pyrex 0.9.8.x that wasn't in Cython already
- faster compilation
- more optimisations
- stricter warnings about potential coding problems (e.g. when using nogil)
- support for Python 2.6
- __future__ imports for unicode literals

Due to very recent changes in the Python code base, this release does not yet
compile code for Python 3.0beta1, but it's well prepared so that when 3.0beta1
finally comes out, we can provide a quick update to make your code compile
with anything from Python 2.3 to 3.0.

Have fun,
Stefan

From dalcinl at gmail.com  Fri Jun 13 16:06:02 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 13 Jun 2008 11:06:02 -0300
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <ac1e4c420806122020j172ec706qdc0374d6820b1c4a@mail.gmail.com>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
	<ac1e4c420806121508m4c7c14c2s331df86a8d531708@mail.gmail.com>
	<e7ba66e40806121614o101f5065oc2f5ad1351d3b3a1@mail.gmail.com>
	<ac1e4c420806122020j172ec706qdc0374d6820b1c4a@mail.gmail.com>
Message-ID: <e7ba66e40806130706g79123c45yf6f9a4de095ca57@mail.gmail.com>

OK, your are right, that code should be definitely safe, even if GCC
generares a spurious warnings.

On 6/13/08, Carl Witty <cwitty at newtonlabs.com> wrote:
> On Thu, Jun 12, 2008 at 4:14 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>  > On 6/12/08, Carl Witty <cwitty at newtonlabs.com> wrote:
>  >> I don't think the solution is nearly so complicated.  Can't you just
>  >>  cast the function to the correct type?
>  >>
>  >>    vtable_Parrot.describe = (void (*)(struct Parrot *))Norwegian_describe;
>  >
>  > Well, the type of 'Norwegian_describe' is acctually 'void (*)(struct
>  > *Norwegian)'. If those types aren't aliasables, then that will break
>  > strict aliasing rules. What I do not know if in this case the funtions
>  > types can alias each other (I guess they cannot).
>
>
> No; you only break aliasing rules if you read or write a value through
>  a pointer of the wrong type.  The above code doesn't do that.
>
>
>  Carl
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From languitar at semipol.de  Fri Jun 13 17:46:17 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Fri, 13 Jun 2008 17:46:17 +0200
Subject: [Cython] assert statements in Cython
Message-ID: <48529649.9010600@semipol.de>

Hi,

is there anything special about using assert statements in Cython? I've
noticed that something like assert foo != NULL results in returning
None, if the method containing this statement really gets foo containing
NULL.

Regards
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080613/ea1c3418/attachment.pgp 

From robertwb at math.washington.edu  Fri Jun 13 18:24:41 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 13 Jun 2008 09:24:41 -0700
Subject: [Cython] Cython 0.9.8 Release candidate
In-Reply-To: <48521219.8090407@behnel.de>
References: <64C7C180-D8F1-4B00-9E36-69E3A85F84F1@math.washington.edu>	<4850C04A.7000806@behnel.de>
	<8f8f8530806121956p2c6e27d1jcfdbab7cd6e175a9@mail.gmail.com>
	<48521219.8090407@behnel.de>
Message-ID: <FF851B1D-BF34-4BB5-BB44-BA25FB017072@math.washington.edu>

On Jun 12, 2008, at 11:22 PM, Stefan Behnel wrote:

> Hi,
>
> Gary Furnish wrote:
>> This seems to break padics:
>>
>> from __future__ import with_statement
>>                                     ^
>> ------------------------------------------------------------
>>
>> /home/gfurnish/sage-3.0.2-symbolics/devel/sage-symbolics/sage/ 
>> rings/padics/padic_generic_element.pyx:19:37:
>> future feature with_statement is not defined
>
> That line shouldn't be there. Cython doesn't support the with  
> statement anyway
> (even if "with nogil" looks similar).

Yes, there were a couple of changes I had to make to the Sage  
codebase (such as this one, which shouldn't have been valid before  
and isn't used) and they're posted as a patch.

- Robert



From ndbecker2 at gmail.com  Fri Jun 13 19:28:26 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 13 Jun 2008 13:28:26 -0400
Subject: [Cython] self tests?
Message-ID: <g2uann$8k1$1@ger.gmane.org>

Is there a set of self-tests that can be run on cython?  How do I run them? 
Are they all expected to pass?

I'd like to add this to the automatic checking for building rpms on fedora.


From robertwb at math.washington.edu  Fri Jun 13 19:32:01 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 13 Jun 2008 10:32:01 -0700
Subject: [Cython] self tests?
In-Reply-To: <g2uann$8k1$1@ger.gmane.org>
References: <g2uann$8k1$1@ger.gmane.org>
Message-ID: <9D4F8335-736D-4531-AAD5-23A0FA802139@math.washington.edu>

On Jun 13, 2008, at 10:28 AM, Neal Becker wrote:

> Is there a set of self-tests that can be run on cython?  How do I  
> run them?
> Are they all expected to pass?

Yes, see the runtests files at the root directory. They should all pass.

>
> I'd like to add this to the automatic checking for building rpms on  
> fedora.

Great idea.

- Robert


From jonas at MIT.EDU  Fri Jun 13 20:04:44 2008
From: jonas at MIT.EDU (Eric Jonas)
Date: Fri, 13 Jun 2008 14:04:44 -0400
Subject: [Cython] Faster Pickle possible?
In-Reply-To: <48504A1E.3010701@student.matnat.uio.no>
References: <1213191613.8633.10.camel@convolution>
	<e7ba66e40806110835y1ed888fbm627f9748329882c5@mail.gmail.com>
	<1213203972.8633.18.camel@convolution>
	<e7ba66e40806111123s62a21c25i4d3095caa28356ad@mail.gmail.com>
	<1213209114.8633.26.camel@convolution>
	<e7ba66e40806111343r37275374y544ca31310851593@mail.gmail.com>
	<48504A1E.3010701@student.matnat.uio.no>
Message-ID: <1213380284.6373.3.camel@convolution>

> I.e., if you want C speed, you need to write C-like static code and use 
> more static algorithms. There's no way around that (except for a macro 
> facility, which we don't have).

Per the advice on comp.lang.python, I disabled GC pre-pickle and then
re-enabled it post-pickle. It turned something that was O(n^2) into O(n)
and cut down my run times from 1000s to 80s. 

The people said: 


> I'm somewhat shocked -- is the occasionally-abysmal behavior of the
> GC documented anywhere?

"I don't think so.  It's somewhat of an FAQ on the list, though.  The
question tends to arise when someone tries to construct a large list
of non-trivial objects, which takes quadratic time because object
allocation regularly triggers GC, which traverses the growing list."

	
			...Eric


From stefan_ml at behnel.de  Fri Jun 13 20:14:51 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 13 Jun 2008 20:14:51 +0200
Subject: [Cython] assert statements in Cython
In-Reply-To: <48529649.9010600@semipol.de>
References: <48529649.9010600@semipol.de>
Message-ID: <4852B91B.2080605@behnel.de>


Johannes Wienke wrote:
> is there anything special about using assert statements in Cython? I've
> noticed that something like assert foo != NULL results in returning
> None, if the method containing this statement really gets foo containing
> NULL.

Would you have some example code?

Stefan


From stefan_ml at behnel.de  Fri Jun 13 20:21:31 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 13 Jun 2008 20:21:31 +0200
Subject: [Cython] Python betas delayed again
Message-ID: <4852BAAB.7010904@behnel.de>

Hi,

just a quick note that the first beta versions of Python 2.6 and 3.0 have been
delayed by another week. They are now expected on June 18. This is mainly due
to stability and build problems caused by last-minute commits.

I would still consider the C API stable by now, as the remaining time is
dedicated to fixing release critical bugs only.

Stefan

From ndbecker2 at gmail.com  Fri Jun 13 23:15:11 2008
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 13 Jun 2008 17:15:11 -0400
Subject: [Cython] cython-0.9.8 installs into python arch-specific dir?
Message-ID: <g2uo0t$k7c$1@ger.gmane.org>

It seems running
%{__python} setup.py build


%{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT

will install into arch-specific python dir?  The previous cython didn't do
this I believe?

e.g.:
/usr/lib64/python2.5/site-packages/...

It looks like all the installed code is pure python - so it should be in non
arch-specific python dir.

Was this changed, and is this intentional?


From robertwb at math.washington.edu  Sat Jun 14 01:06:43 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 13 Jun 2008 16:06:43 -0700
Subject: [Cython] cython-0.9.8 installs into python arch-specific dir?
In-Reply-To: <g2uo0t$k7c$1@ger.gmane.org>
References: <g2uo0t$k7c$1@ger.gmane.org>
Message-ID: <80EB438C-6167-48AC-8384-B9D4C2FBC92B@math.washington.edu>

On Jun 13, 2008, at 2:15 PM, Neal Becker wrote:

> It seems running
> %{__python} setup.py build
>
>
> %{__python} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT
>
> will install into arch-specific python dir?  The previous cython  
> didn't do
> this I believe?
>
> e.g.:
> /usr/lib64/python2.5/site-packages/...
>
> It looks like all the installed code is pure python - so it should  
> be in non
> arch-specific python dir.
>
> Was this changed, and is this intentional?

Yes, Cython now uses itself to compile scanning.py (for speed) if  
possible. Eventually we hope to bootstrap even more. However, all the  
pure python code is still there and will work fine.

- Robert



From greg.ewing at canterbury.ac.nz  Sat Jun 14 03:05:41 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 14 Jun 2008 13:05:41 +1200
Subject: [Cython] Type casting issues and strict aliasing
In-Reply-To: <ac1e4c420806122023o5906cda1m5bf723d2d039c3df@mail.gmail.com>
References: <4850E614.1090806@behnel.de> <48517B85.6030900@behnel.de>
	<e7ba66e40806121340i5270d005h1169b1bb2f3677a2@mail.gmail.com>
	<4851CEE6.9010207@canterbury.ac.nz>
	<ac1e4c420806122023o5906cda1m5bf723d2d039c3df@mail.gmail.com>
Message-ID: <48531965.50503@canterbury.ac.nz>

Carl Witty wrote:
> I think you can just pick some arbitrary function pointer type, like
> (void (*)(void)).

I suppose so, but it's not quite the same thing, since
you wouldn't get the ability to assign any function to
one of these without casting.

-- 
Greg

From jwienke at techfak.uni-bielefeld.de  Sat Jun 14 11:53:40 2008
From: jwienke at techfak.uni-bielefeld.de (Johannes Wienke)
Date: Sat, 14 Jun 2008 11:53:40 +0200
Subject: [Cython] assert statements in Cython
In-Reply-To: <4852B91B.2080605@behnel.de>
References: <48529649.9010600@semipol.de> <4852B91B.2080605@behnel.de>
Message-ID: <48539524.6070900@techfak.uni-bielefeld.de>

Am 06/13/2008 08:14 PM schrieb Stefan Behnel:
> Johannes Wienke wrote:
>> is there anything special about using assert statements in Cython? I've
>> noticed that something like assert foo != NULL results in returning
>> None, if the method containing this statement really gets foo containing
>> NULL.
> 
> Would you have some example code?

Well, can't reproduce it. :\ Maybe I forgot an except clause and didn't
see the warning about the ignored exception or something like that.

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080614/38cb3aee/attachment.pgp 

From robertwb at math.washington.edu  Tue Jun 17 02:19:55 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Jun 2008 17:19:55 -0700
Subject: [Cython] Dev days status
Message-ID: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>

These last couple of days we have appreciated having Dag Sverre  
Seljebotn and Dan Gindikin (the creator of Pex) around, and have had  
a lot of discussions (if not yet a lot of code). Personally, I have  
been wrapped up in other sage-related projects as well. A good  
summary of the discussion can be found here http://wiki.cython.org/ 
enhancements/pex . We have also decided to change the direction of  
Dag's NumPy GSoC project a bit--to really get what we want it had  
become clear that a better approach will provide direct support for  
buffers http://wiki.cython.org/enhancements/buffer to provide fast  
NumPy array access.

The biggest proposals for change are:

- Having an except [uncommon value]? by default on all implemented  
functions.
- Phasing out cimport, letting import be cimport whenever the  
relevant header files are found.
- Using the .pyx file for "cimport" when a .pxd file is unavailable  
(basically auto-generation of pxd files).

These last two have the disadvantage that the dependancy tree could  
become much more connected, so we would provide a way to disable  
them. Thoughts?

- Robert


From rebirth at orcon.net.nz  Tue Jun 17 04:35:45 2008
From: rebirth at orcon.net.nz (David McNab)
Date: Tue, 17 Jun 2008 14:35:45 +1200
Subject: [Cython] initialised cdef class variables
Message-ID: <1213670145.11944.22.camel@rebirth.conscious.co.nz>

Hi,

I've been trying to convert my 'PyGene' genetic programming/algorithms
package to Cython, and have hit a major obstacle.

With PyGene, configuration is largely done by subclassing and overriding
class attributes.

But with Cython(/Pyrex), this doesn't seem to be possible. If, in a cdef
class, I declare writable class attributes with initialisers, such as:

  cdef class Foo:

     cdef int bar = 5

the class attribute 'bar' is not getting initialised.

Would it be possible to have class attribute initialiser support in a
future release of Cython?

Cheers
David




From robertwb at math.washington.edu  Tue Jun 17 07:59:15 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 16 Jun 2008 22:59:15 -0700
Subject: [Cython] initialised cdef class variables
In-Reply-To: <1213670145.11944.22.camel@rebirth.conscious.co.nz>
References: <1213670145.11944.22.camel@rebirth.conscious.co.nz>
Message-ID: <3A5C38B3-3536-42C3-B7EB-5404A6DB28E2@math.washington.edu>

On Jun 16, 2008, at 7:35 PM, David McNab wrote:

> Hi,
>
> I've been trying to convert my 'PyGene' genetic programming/algorithms
> package to Cython, and have hit a major obstacle.
>
> With PyGene, configuration is largely done by subclassing and  
> overriding
> class attributes.
>
> But with Cython(/Pyrex), this doesn't seem to be possible. If, in a  
> cdef
> class, I declare writable class attributes with initialisers, such as:
>
>   cdef class Foo:
>
>      cdef int bar = 5
>
> the class attribute 'bar' is not getting initialised.
>
> Would it be possible to have class attribute initialiser support in a
> future release of Cython?

Yes, we should support this. (Note that this will not be a class- 
level cdef attribute however, it would just change the initial value  
for the instance member) Please file a report at http:// 
trac.cython.org/cython_trac/ .

- Robert


From stefan_ml at behnel.de  Tue Jun 17 08:23:25 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jun 2008 08:23:25 +0200
Subject: [Cython] Dev days status
In-Reply-To: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
Message-ID: <4857585D.4000004@behnel.de>

Hi,

Robert Bradshaw wrote:
> These last couple of days we have appreciated having Dag Sverre  
> Seljebotn and Dan Gindikin (the creator of Pex) around, and have had  
> a lot of discussions (if not yet a lot of code).

A good design is sometimes worth a thousand lines of code. I'll first have to
read a bit closer through the PEX page and Dag's buffer proposal before I can
comment on it, but it looks interesting if not more.

BTW, before the code gets written (or migrated or whatever) and reviewed, I
think this might be worth a read:

http://blog.red-bean.com/sussman/?p=96


> The biggest proposals for change are:
> 
> - Having an except [uncommon value]? by default on all implemented  
> functions.

I first thought about that as a door opener for performance problems, but I
actually think most functions forward exceptions anyway, so this won't change
much in that regard.

One thing I'd add is an optimisation for normal object-returning functions
that do not have a return statement, i.e. that do not really return anything
but exceptions. Here, we could change the signature internally to returning a
bint instead (normal return or exception), which avoids the INCREF/DECREF None
overhead for the return value.


> - Phasing out cimport, letting import be cimport whenever the  
> relevant header files are found.

That's an interesting idea. And it might open up some advanced optimisations
for normal Python code, too. Think of plain Python code accompanied by a .pxd
file where Python would use the normal .py module and Cython would see both
the .py and the .pxd file. We could even allow function signature overrides
specifically for .py files when we find a .pxd file.

We might want to output something from the compiler though, to tell users what
was imported at compile time and what was left to runtime. For them, it might
or might not be an error when things aren't found at compile time.

OTOH, Cython might be able to determine errors from the code it sees anyway.
For example, a "cdef class" with an unknown but imported base class is
definitely an error.


> - Using the .pyx file for "cimport" when a .pxd file is unavailable  
> (basically auto-generation of pxd files).

I'm not so sure about that one. This basically means cimporting from .pyx
files, which might really lead to a performance drop of the compiler (remember
that the slowest thing is still the scanner). I see that it goes somewhat hand
in hand with the "cimport imports" change above, and I also see that it's easy
to work around by actually providing .pxd files. So maybe this is really a
non-issue.


> These last two have the disadvantage that the dependancy tree could  
> become much more connected, so we would provide a way to disable  
> them. Thoughts?

Why? Do you mean "dependency tree" in terms of files depending on each other?
Greg's .dep files would handle that...

Stefan


From robertwb at math.washington.edu  Tue Jun 17 09:15:56 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Jun 2008 00:15:56 -0700
Subject: [Cython] Dev days status
In-Reply-To: <4857585D.4000004@behnel.de>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de>
Message-ID: <91AFB0EE-8B27-4BCC-AFC4-8C08CAA23ECB@math.washington.edu>

On Jun 16, 2008, at 11:23 PM, Stefan Behnel wrote:

> Hi,
>
> Robert Bradshaw wrote:
>> These last couple of days we have appreciated having Dag Sverre
>> Seljebotn and Dan Gindikin (the creator of Pex) around, and have had
>> a lot of discussions (if not yet a lot of code).
>
> A good design is sometimes worth a thousand lines of code. I'll  
> first have to
> read a bit closer through the PEX page and Dag's buffer proposal  
> before I can
> comment on it, but it looks interesting if not more.
>
> BTW, before the code gets written (or migrated or whatever) and  
> reviewed,

Dag can't write a single line of code on that front here for tax  
reasons, so it'll definitely have time to stew a while. The  
assumption/inline pxd stuff is powerful and nice, but is not quite  
enough (one problem is gcc aliasing needs to be on, which limits  
performance improvements) and less feasible.

> I think this might be worth a read:
>
> http://blog.red-bean.com/sussman/?p=96

Interesting read.

>
>
>> The biggest proposals for change are:
>>
>> - Having an except [uncommon value]? by default on all implemented
>> functions.
>
> I first thought about that as a door opener for performance  
> problems, but I
> actually think most functions forward exceptions anyway, so this  
> won't change
> much in that regard.

Yep, and it'll save a lot of pain for programmers.

> One thing I'd add is an optimisation for normal object-returning  
> functions
> that do not have a return statement, i.e. that do not really return  
> anything
> but exceptions. Here, we could change the signature internally to  
> returning a
> bint instead (normal return or exception), which avoids the INCREF/ 
> DECREF None
> overhead for the return value.

That is a good idea, though might mess up the signature (e.g. for  
cdef methods that may be overridden).

>> - Phasing out cimport, letting import be cimport whenever the
>> relevant header files are found.
>
> That's an interesting idea. And it might open up some advanced  
> optimisations
> for normal Python code, too. Think of plain Python code accompanied  
> by a .pxd
> file where Python would use the normal .py module and Cython would  
> see both
> the .py and the .pxd file. We could even allow function signature  
> overrides
> specifically for .py files when we find a .pxd file.
>
> We might want to output something from the compiler though, to tell  
> users what
> was imported at compile time and what was left to runtime. For  
> them, it might
> or might not be an error when things aren't found at compile time.

+1

> OTOH, Cython might be able to determine errors from the code it  
> sees anyway.
> For example, a "cdef class" with an unknown but imported base class is
> definitely an error.
>
>
>> - Using the .pyx file for "cimport" when a .pxd file is unavailable
>> (basically auto-generation of pxd files).
>
> I'm not so sure about that one. This basically means cimporting  
> from .pyx
> files, which might really lead to a performance drop of the  
> compiler (remember
> that the slowest thing is still the scanner). I see that it goes  
> somewhat hand
> in hand with the "cimport imports" change above, and I also see  
> that it's easy
> to work around by actually providing .pxd files. So maybe this is  
> really a
> non-issue.

I imagine we would want to cache this, perhaps in an easier to parse  
format that .pxds are now (as parsing them can take a while too).

>
>
>> These last two have the disadvantage that the dependancy tree could
>> become much more connected, so we would provide a way to disable
>> them. Thoughts?
>
> Why? Do you mean "dependency tree" in terms of files depending on  
> each other?
> Greg's .dep files would handle that...

There are several ways it becomes tighter. First, one can import  
rather than cimport when one doesn't need speed and doesn't want to  
create a compile-time dependancy (e.g. like in a huge project like  
Sage). Secondly, everything in the file gets cimported, not just the  
interface exposed by the .pxd. Thirdly, any time the .pyx file  
changes, it needs to be checked (instead of only on .pxd changes).

These costs are wroth it I think for "normal" behavior, and power- 
users with 100s of interdependent files can disable them.

- Robert



From stefan_ml at behnel.de  Tue Jun 17 13:54:27 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jun 2008 11:54:27 +0000 (UTC)
Subject: [Cython] Dev days status
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de>
	<91AFB0EE-8B27-4BCC-AFC4-8C08CAA23ECB@math.washington.edu>
Message-ID: <loom.20080617T111608-435@post.gmane.org>

Robert Bradshaw <robertwb at ...> writes:
> On Jun 16, 2008, at 11:23 PM, Stefan Behnel wrote:
> > One thing I'd add is an optimisation for normal object-returning  
> > functions
> > that do not have a return statement, i.e. that do not really return  
> > anything
> > but exceptions. Here, we could change the signature internally to  
> > returning a
> > bint instead (normal return or exception), which avoids the INCREF/ 
> > DECREF None
> > overhead for the return value.
> 
> That is a good idea, though might mess up the signature (e.g. for  
> cdef methods that may be overridden).

It would be fine for functions, though, and that's where I'd expect the 
biggest impact anyway, e.g. when inlining short functions.


> >> - Using the .pyx file for "cimport" when a .pxd file is unavailable
> >> (basically auto-generation of pxd files).
> >
> > I'm not so sure about that one. This basically means cimporting  
> > from .pyx
> > files, which might really lead to a performance drop of the  
> > compiler (remember
> > that the slowest thing is still the scanner).
>
> I imagine we would want to cache this, perhaps in an easier to parse  
> format that .pxds are now (as parsing them can take a while too).

Like a pickled .pxd parse tree? That might obviously change over Cython 
versions, but if it's only for caching...


> >> These last two have the disadvantage that the dependancy tree could
> >> become much more connected, so we would provide a way to disable
> >> them.
> 
> There are several ways it becomes tighter. First, one can import  
> rather than cimport when one doesn't need speed and doesn't want to  
> create a compile-time dependancy (e.g. like in a huge project like  
> Sage). Secondly, everything in the file gets cimported, not just the  
> interface exposed by the .pxd.

I actually think a lot of stuff would never be cimported, think of the Python 
stdlib, for example. I mean, we may still see the day where C extensions in 
the stdlib ship with .pxd's, but until then...


> Thirdly, any time the .pyx file  
> changes, it needs to be checked (instead of only on .pxd changes).

True, and that means tracking and parsing all included .pxi files. I actually 
think that was the reason why .pxd files were introduced in the first place. 
Importing from .pyx files might not be that a good idea after all...

Stefan



From simon at arrowtheory.com  Tue Jun 17 20:06:24 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Tue, 17 Jun 2008 14:06:24 -0400
Subject: [Cython] JOB in New York
Message-ID: <20080617140624.fe882c78.simon@arrowtheory.com>


This was also posted to the python jobs board, but we lean
pretty heavily on cython so i'm posting it here too.



Zone Holdings, LLC (New York, NY, USA)
======================================

Job Description: Help develop real-time, network driven systems, using python/cython/c.

You will be working with a small team of programmers, network engineers
and traders, to develop a lightning fast trading platform.

Experience in one or more of the following is desirable:
  * real-time systems, real-time memory management
  * network programming, network protocols
  * distributed computing
  * gui programming, rpc

Experience in the following is helpful but not essential:
  * financial protocols
  * experience trading equities, futures or forex products

Zone Holdings is a proprietary trading firm located at the New York Mercantile Exchange.

* E-mail resumes to: resumes at zonellc.com


From robertwb at math.washington.edu  Tue Jun 17 21:22:22 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Jun 2008 12:22:22 -0700
Subject: [Cython] Dev days status
In-Reply-To: <loom.20080617T111608-435@post.gmane.org>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de>
	<91AFB0EE-8B27-4BCC-AFC4-8C08CAA23ECB@math.washington.edu>
	<loom.20080617T111608-435@post.gmane.org>
Message-ID: <38FEFB5D-6F92-45BA-9F59-6BF1933367E8@math.washington.edu>


On Jun 17, 2008, at 4:54 AM, Stefan Behnel wrote:

> Robert Bradshaw <robertwb at ...> writes:
>> On Jun 16, 2008, at 11:23 PM, Stefan Behnel wrote:
>>> One thing I'd add is an optimisation for normal object-returning
>>> functions
>>> that do not have a return statement, i.e. that do not really return
>>> anything
>>> but exceptions. Here, we could change the signature internally to
>>> returning a
>>> bint instead (normal return or exception), which avoids the INCREF/
>>> DECREF None
>>> overhead for the return value.
>>
>> That is a good idea, though might mess up the signature (e.g. for
>> cdef methods that may be overridden).
>
> It would be fine for functions, though, and that's where I'd expect  
> the
> biggest impact anyway, e.g. when inlining short functions.

Very good point.

>>>> - Using the .pyx file for "cimport" when a .pxd file is unavailable
>>>> (basically auto-generation of pxd files).
>>>
>>> I'm not so sure about that one. This basically means cimporting
>>> from .pyx
>>> files, which might really lead to a performance drop of the
>>> compiler (remember
>>> that the slowest thing is still the scanner).
>>
>> I imagine we would want to cache this, perhaps in an easier to parse
>> format that .pxds are now (as parsing them can take a while too).
>
> Like a pickled .pxd parse tree? That might obviously change over  
> Cython
> versions, but if it's only for caching...

Yep. Note that I've looked into this and these can be huge (at least  
once a bit of analysis is done).

>>>> These last two have the disadvantage that the dependancy tree could
>>>> become much more connected, so we would provide a way to disable
>>>> them.
>>
>> There are several ways it becomes tighter. First, one can import
>> rather than cimport when one doesn't need speed and doesn't want to
>> create a compile-time dependancy (e.g. like in a huge project like
>> Sage). Secondly, everything in the file gets cimported, not just the
>> interface exposed by the .pxd.
>
> I actually think a lot of stuff would never be cimported, think of  
> the Python
> stdlib, for example. I mean, we may still see the day where C  
> extensions in
> the stdlib ship with .pxd's, but until then...

Yeah. I think this is mostly for ease of use of users--having to  
write .pxd's and .pyx's is natural for people coming from the C side  
of things, but not at all for those coming from Python.

>> Thirdly, any time the .pyx file
>> changes, it needs to be checked (instead of only on .pxd changes).
>
> True, and that means tracking and parsing all included .pxi files.  
> I actually
> think that was the reason why .pxd files were introduced in the  
> first place.
> Importing from .pyx files might not be that a good idea after all...


If pxd files are around, only they would be used. This is only if  
they're not around.

- Robert



From ellisonbg.net at gmail.com  Tue Jun 17 21:56:07 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 17 Jun 2008 13:56:07 -0600
Subject: [Cython] cross project cimport
Message-ID: <6ce0ac130806171256v5cc08839qf7225417346c1545@mail.gmail.com>

Hi,

I am wondering if cross project cimports will work or if any thought
has gone into making this work.  Basically, mpi4py has a pxd file I
need to cimport it in a completely different project.  One option is
to copy it over to the other project, but that is quite annoying as it
includes a bunch of pxi files and these would need to be copied over
as well.  Any thoughts on this?

Thanks

Brian

From robertwb at math.washington.edu  Tue Jun 17 23:59:12 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Jun 2008 14:59:12 -0700
Subject: [Cython] cross project cimport
In-Reply-To: <6ce0ac130806171256v5cc08839qf7225417346c1545@mail.gmail.com>
References: <6ce0ac130806171256v5cc08839qf7225417346c1545@mail.gmail.com>
Message-ID: <4FF573E9-1204-42DF-93F5-A926F1C1DB3D@math.washington.edu>

Copying over the .pxd file should work fine (that's what they're for)  
but you should also be able to set an include directory.

On Jun 17, 2008, at 12:56 PM, Brian Granger wrote:

> Hi,
>
> I am wondering if cross project cimports will work or if any thought
> has gone into making this work.  Basically, mpi4py has a pxd file I
> need to cimport it in a completely different project.  One option is
> to copy it over to the other project, but that is quite annoying as it
> includes a bunch of pxi files and these would need to be copied over
> as well.  Any thoughts on this?
>
> Thanks
>
> Brian
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ellisonbg.net at gmail.com  Wed Jun 18 00:17:48 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Tue, 17 Jun 2008 16:17:48 -0600
Subject: [Cython] cross project cimport
In-Reply-To: <4FF573E9-1204-42DF-93F5-A926F1C1DB3D@math.washington.edu>
References: <6ce0ac130806171256v5cc08839qf7225417346c1545@mail.gmail.com>
	<4FF573E9-1204-42DF-93F5-A926F1C1DB3D@math.washington.edu>
Message-ID: <6ce0ac130806171517x3065766ch6ac5d3e3d9f75702@mail.gmail.com>

On Tue, Jun 17, 2008 at 3:59 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Copying over the .pxd file should work fine (that's what they're for)
> but you should also be able to set an include directory.

But copying it over is a bit messy:

1) If mpi4py changes its pxd file, we have to re-copy it over.  Also,
when there are changes like this, it become very difficult for us to
write code that works under both versions.

2) The pxd file includes other things (mpi.pxi) that will also have to
be copied over.

It would be nice if mpi4py could install its MPI.pxd and mpi.pxi files
and then we could do:

cimport mpi4py.MPI

And that cython could resolve the paths simply based on the installed
location of mpi4py.  That might be too magic though.

Cheers,

Brian

> On Jun 17, 2008, at 12:56 PM, Brian Granger wrote:
>
>> Hi,
>>
>> I am wondering if cross project cimports will work or if any thought
>> has gone into making this work.  Basically, mpi4py has a pxd file I
>> need to cimport it in a completely different project.  One option is
>> to copy it over to the other project, but that is quite annoying as it
>> includes a bunch of pxi files and these would need to be copied over
>> as well.  Any thoughts on this?
>>
>> Thanks
>>
>> Brian
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Wed Jun 18 00:25:46 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Jun 2008 15:25:46 -0700
Subject: [Cython] cross project cimport
In-Reply-To: <6ce0ac130806171517x3065766ch6ac5d3e3d9f75702@mail.gmail.com>
References: <6ce0ac130806171256v5cc08839qf7225417346c1545@mail.gmail.com>
	<4FF573E9-1204-42DF-93F5-A926F1C1DB3D@math.washington.edu>
	<6ce0ac130806171517x3065766ch6ac5d3e3d9f75702@mail.gmail.com>
Message-ID: <5B56FED3-E264-46CF-B113-A4056FF536AC@math.washington.edu>


On Jun 17, 2008, at 3:17 PM, Brian Granger wrote:

> On Tue, Jun 17, 2008 at 3:59 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> Copying over the .pxd file should work fine (that's what they're for)
>> but you should also be able to set an include directory.
>
> But copying it over is a bit messy:

I agree, this was the short term solution.

> 1) If mpi4py changes its pxd file, we have to re-copy it over.  Also,
> when there are changes like this, it become very difficult for us to
> write code that works under both versions.
>
> 2) The pxd file includes other things (mpi.pxi) that will also have to
> be copied over.
>
> It would be nice if mpi4py could install its MPI.pxd and mpi.pxi files
> and then we could do:
>
> cimport mpi4py.MPI
>
> And that cython could resolve the paths simply based on the installed
> location of mpi4py.  That might be too magic though.

Perhaps the -I option will do what you want. This is certainly an  
area for improvement, and I think the best would be to handle it  
using the exact same framework that resolves the .py files.

>
> Cheers,
>
> Brian
>
>> On Jun 17, 2008, at 12:56 PM, Brian Granger wrote:
>>
>>> Hi,
>>>
>>> I am wondering if cross project cimports will work or if any thought
>>> has gone into making this work.  Basically, mpi4py has a pxd file I
>>> need to cimport it in a completely different project.  One option is
>>> to copy it over to the other project, but that is quite annoying  
>>> as it
>>> includes a bunch of pxi files and these would need to be copied over
>>> as well.  Any thoughts on this?
>>>
>>> Thanks
>>>
>>> Brian
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dagss at student.matnat.uio.no  Wed Jun 18 01:50:57 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 18 Jun 2008 01:50:57 +0200 (CEST)
Subject: [Cython] Buffer API support
Message-ID: <54521.69.91.136.239.1213746657.squirrel@webmail.uio.no>

I've now polished the buffer proposal draft:

http://wiki.cython.org/enhancements/buffer

And hereby launch a discussion thread for it. As Robert said I can't work
on implementing it yet so there's plenty of time for discussion.

Dag Sverre


From greg.ewing at canterbury.ac.nz  Wed Jun 18 02:15:09 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Jun 2008 12:15:09 +1200
Subject: [Cython] initialised cdef class variables
In-Reply-To: <3A5C38B3-3536-42C3-B7EB-5404A6DB28E2@math.washington.edu>
References: <1213670145.11944.22.camel@rebirth.conscious.co.nz>
	<3A5C38B3-3536-42C3-B7EB-5404A6DB28E2@math.washington.edu>
Message-ID: <4858538D.8090902@canterbury.ac.nz>

Robert Bradshaw wrote:

> On Jun 16, 2008, at 7:35 PM, David McNab wrote:
> 
> > Would it be possible to have class attribute initialiser support in a
> > future release of Cython?
> 
> Yes, we should support this. (Note that this will not be a class- 
> level cdef attribute however, it would just change the initial value  
> for the instance member)

Another approach you might be able to use in the meantime
is creating a Python subclass of the extension type to
hold the class attributes.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Wed Jun 18 02:49:21 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Jun 2008 12:49:21 +1200
Subject: [Cython] Dev days status
In-Reply-To: <4857585D.4000004@behnel.de>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de>
Message-ID: <48585B91.5050802@canterbury.ac.nz>

Stefan Behnel wrote:
> 
> Robert Bradshaw wrote:

> > - Having an except [uncommon value]? by default on all implemented  
> > functions.
> 
> I first thought about that as a door opener for performance problems, but I
> actually think most functions forward exceptions anyway, so this won't change
> much in that regard.

I'm not sure I like this idea. Most functions from external
C libraries will not be raising exceptions, so if you don't
want performance penalties and code bloat when calling those,
you would have to make a distinction between Cython-implemented
functions and extern ones.

This would be a major conceptual change, as there is currently
no such distinction -- a given function signature always means
the same thing however the function is implemented. I'm
worried that this would lead to confusion.

> 
> One thing I'd add is an optimisation for normal object-returning functions
> that do not have a return statement, i.e. that do not really return anything
> but exceptions. Here, we could change the signature internally to returning a
> bint instead

This would mean that you wouldn't be able to determine the
return type of a function from looking at its header, as it
would depend on what goes on in the body. How would you tell
from an extern declaration or a declaration in a .pxd file how
to call such a function?

However, it might be feasible to have a special return type
that you can explicitly use in that case, such as

   cdef errflag foo():
     ...

where errflag is equivalent to int except that it implicitly
has an error return value of -1. (I don't really like the
name 'errflag' much, but I can't think of anything better
off the top of my head.)

> Why? Do you mean "dependency tree" in terms of files depending on each other?
> Greg's .dep files would handle that...

The problem is that if you're using timestamps to tell when
things need to be recompiled, then *any* change to the .pyx
file will trigger recompilation of everything that imports
from it, whether the change affects the external interface
or not.

To avoid that, you would have to perform some sort of
analysis to compare the current interface with what it
had last time it was compiled, which could become tricky
depending on what level of false positives you're willing
to accept.

This may also have a bearing on the import/cimport
distinction. Currently, a cimport implies interface
dependency, whereas an import doesn't. If you merge them,
you will have to take note of what kinds of things are
being imported to determine whether an interface
dependency exists.

Another advantage of explicit .pxd files is that it's
always clear when you're changing the interface and
when you aren't. You know that nothing you do to a
.pxd file can change the interface, which can be
important when other people are using your module.

-- 
Greg



From greg.ewing at canterbury.ac.nz  Wed Jun 18 03:06:39 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 18 Jun 2008 13:06:39 +1200
Subject: [Cython] Dev days status
In-Reply-To: <48585B91.5050802@canterbury.ac.nz>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de> <48585B91.5050802@canterbury.ac.nz>
Message-ID: <48585F9F.6010908@canterbury.ac.nz>

I wrote:
> You know that nothing you do to a
> .pxd file can change the interface

Sorry, I meant you know that nothing you do to a
.pyx file can change the interface.

-- 
Greg

From dagss at student.matnat.uio.no  Wed Jun 18 03:42:48 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 18 Jun 2008 03:42:48 +0200 (CEST)
Subject: [Cython] nosetest
Message-ID: <46482.69.91.136.239.1213753368.squirrel@webmail.uio.no>

NumPy has recently switched to using nosetest for their testing framework:

http://www.somethingaboutorange.com/mrl/projects/nose/

Basically, it is unittest on steroids (and uses/operates together with
unittest). The major features:
- Out of the box unit test discovery on a source tree (according to some
configurable name pattern on packages and modules)
- The ability to make simple functions etc. test cases if they have the
right name. Though unittest classes are of course supported as well.
- One can write multiple tests in one function, then "yield" each one of
them, and each will be a seperate test case in the reports.

I just wouldn't like to have rewrite discovery of unittests in runtests.py
when this is already there in nosetests.

What do you think -- will it be bad to have this as a dependency?

Dag Sverre



From robertwb at math.washington.edu  Wed Jun 18 04:41:33 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 17 Jun 2008 19:41:33 -0700
Subject: [Cython] Dev days status
In-Reply-To: <48585B91.5050802@canterbury.ac.nz>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de> <48585B91.5050802@canterbury.ac.nz>
Message-ID: <080E7962-F811-4427-A19A-CBBBA73490F9@math.washington.edu>

On Jun 17, 2008, at 5:49 PM, Greg Ewing wrote:

> Stefan Behnel wrote:
>>
>> Robert Bradshaw wrote:
>
>>> - Having an except [uncommon value]? by default on all implemented
>>> functions.
>>
>> I first thought about that as a door opener for performance  
>> problems, but I
>> actually think most functions forward exceptions anyway, so this  
>> won't change
>> much in that regard.
>
> I'm not sure I like this idea. Most functions from external
> C libraries will not be raising exceptions, so if you don't
> want performance penalties and code bloat when calling those,
> you would have to make a distinction between Cython-implemented
> functions and extern ones.

Yep, I think that is a distinction I am willing to make. This is why  
I said "all implemented functions" rather than "all [defined]  
functions."

> This would be a major conceptual change, as there is currently
> no such distinction -- a given function signature always means
> the same thing however the function is implemented. I'm
> worried that this would lead to confusion.

The current behavior leads to confusion too.

>
>>
>> One thing I'd add is an optimisation for normal object-returning  
>> functions
>> that do not have a return statement, i.e. that do not really  
>> return anything
>> but exceptions. Here, we could change the signature internally to  
>> returning a
>> bint instead
>
> This would mean that you wouldn't be able to determine the
> return type of a function from looking at its header, as it
> would depend on what goes on in the body. How would you tell
> from an extern declaration or a declaration in a .pxd file how
> to call such a function?

I agree.

> However, it might be feasible to have a special return type
> that you can explicitly use in that case, such as
>
>    cdef errflag foo():
>      ...
>
> where errflag is equivalent to int except that it implicitly
> has an error return value of -1. (I don't really like the
> name 'errflag' much, but I can't think of anything better
> off the top of my head.)

I don't like errflag either--in that case why not explicitly make it  
int?

>> Why? Do you mean "dependency tree" in terms of files depending on  
>> each other?
>> Greg's .dep files would handle that...
>
> The problem is that if you're using timestamps to tell when
> things need to be recompiled, then *any* change to the .pyx
> file will trigger recompilation of everything that imports
> from it, whether the change affects the external interface
> or not.
>
> To avoid that, you would have to perform some sort of
> analysis to compare the current interface with what it
> had last time it was compiled, which could become tricky
> depending on what level of false positives you're willing
> to accept.

No false positives.

> This may also have a bearing on the import/cimport
> distinction. Currently, a cimport implies interface
> dependency, whereas an import doesn't. If you merge them,
> you will have to take note of what kinds of things are
> being imported to determine whether an interface
> dependency exists.
>
> Another advantage of explicit .pxd files is that it's
> always clear when you're changing the interface and
> when you aren't. You know that nothing you do to a
> .pxd file can change the interface, which can be
> important when other people are using your module.


We will surely keep explicit .pxd files around for Sage for these  
reasons. It's more work (by the compiler) for less work on the user's  
end.

- Robert


From dalcinl at gmail.com  Wed Jun 18 16:38:58 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 18 Jun 2008 11:38:58 -0300
Subject: [Cython] an issue with generated C API's
Message-ID: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>

Currently, if one has something like this:

# mymod.pyx
cdef public api class Foo [type PyFoo_Type, object PyFooObject]:
    cdef int bar

The generated C header file contains this:

# mymod.h
struct PyFooObject {
  PyObject_HEAD
  int bar;
};


Could the generated code be changed to say like this ??:

typedef struct PyFooObject {
  PyObject_HEAD
  int bar;
} PyFooObject;

or perhaps this (though AFAIK they are equivalent)

struct PyFooObject {
  PyObject_HEAD
  int bar;
};
typedef struct PyFooObject PyFooObject;


If this is accepted, I could take the task of writing the patch (of
course, iff I actually can figure out the way). I would love to get
this in ASAP, because other project (from Brian Granger) depending on
mpi4py will need this and that project is scheduled for completion in
about 3 weeks.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Jun 18 08:09:14 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jun 2008 08:09:14 +0200
Subject: [Cython] cross project cimport
In-Reply-To: <6ce0ac130806171517x3065766ch6ac5d3e3d9f75702@mail.gmail.com>
References: <6ce0ac130806171256v5cc08839qf7225417346c1545@mail.gmail.com>	<4FF573E9-1204-42DF-93F5-A926F1C1DB3D@math.washington.edu>
	<6ce0ac130806171517x3065766ch6ac5d3e3d9f75702@mail.gmail.com>
Message-ID: <4858A68A.9020702@behnel.de>

Hi,

Brian Granger wrote:
> It would be nice if mpi4py could install its MPI.pxd and mpi.pxi files
> and then we could do:
> 
> cimport mpi4py.MPI

The mpi4py project can do that in their setup.py by making the public files
package data. I don't currently know if Cython searches the standard package
directory of Python (site-packages), but if not, it would be a good idea to
enable that.

Stefan



From stefan_ml at behnel.de  Wed Jun 18 09:06:17 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jun 2008 09:06:17 +0200
Subject: [Cython] nosetest
In-Reply-To: <46482.69.91.136.239.1213753368.squirrel@webmail.uio.no>
References: <46482.69.91.136.239.1213753368.squirrel@webmail.uio.no>
Message-ID: <4858B3E9.1020900@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> NumPy has recently switched to using nosetest for their testing framework:
> 
> http://www.somethingaboutorange.com/mrl/projects/nose/

That looks like a pretty huge package just for testing. The API docs count
almost 20 modules. And it's even more when you consider that most of the
compiler code doesn't use unit tests at all but only (easy-to-write)
end-to-end tests.

Don't get me wrong, this dependency won't hurt users much and I don't mind
installing a new package as a committed developer. But integrators,
distribution maintainers and single-patch contributors may also suffer from
that dependency. Currently, Cython is self-contained even for developers. It's
easier to tell people "run runtests.py in the root directory" than "install
nose-test on your machine (which you can find /here/) and then use the
following command line to run it".


> - The ability to make simple functions etc. test cases if they have the
> right name. Though unittest classes are of course supported as well.

I consider that a rare requirement. Most unit tests will need some kind of
infrastructure and helper functions/classes anyway (especially for a compiler
setting, I'd say), so you'd most likely end up with one or more custom unit
test base class anyway.


> - One can write multiple tests in one function, then "yield" each one of
> them, and each will be a seperate test case in the reports.

Might be nice syntactic sugar, but that's not hard to do at all. I don't mind
writing those 5 lines of Python code by hand if I really need that feature.


> I just wouldn't like to have rewrite discovery of unittests in runtests.py
> when this is already there in nosetests.

That's already done by the one-module test tool I pointed you to, which you
can just add to Cython's root directory. It may not be the best one, but there
are definitely other one-module test runner scripts out there which may fit
our needs without adding our first non-functional dependency.


> What do you think -- will it be bad to have this as a dependency?

I'm -0.5 on this, as I likely won't be affected too much anyway. I just
dislike the non-trivial dependency.

Stefan


From dg at pnylab.com  Wed Jun 18 18:30:45 2008
From: dg at pnylab.com (Dan Gindikin)
Date: Wed, 18 Jun 2008 16:30:45 +0000 (UTC)
Subject: [Cython] Dev days status
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de> <48585B91.5050802@canterbury.ac.nz>
Message-ID: <loom.20080618T161414-100@post.gmane.org>

Greg Ewing <greg.ewing at ...> writes:

> 
> Stefan Behnel wrote:
> > 
> > Robert Bradshaw wrote:
> 
> > > - Having an except [uncommon value]? by default on all implemented  
> > > functions.
> > 
> > I first thought about that as a door opener for performance problems, but I
> > actually think most functions forward exceptions anyway, so this won't change
> > much in that regard.
> 
> I'm not sure I like this idea. Most functions from external
> C libraries will not be raising exceptions, so if you don't
> want performance penalties and code bloat when calling those,
> you would have to make a distinction between Cython-implemented
> functions and extern ones.
> 

In pex, I added an "except exits" clause, eg

   cdef int foo() except exits:

For an external C function, it means the
exception flag isn't checked. For cdef functions, it makes the function
die if it ever raises an exception (I wanted to make sure there was no way
for execution to continue on an error).

> > Why? Do you mean "dependency tree" in terms of files depending on each other?
> > Greg's .dep files would handle that...
> 
> The problem is that if you're using timestamps to tell when
> things need to be recompiled, then *any* change to the .pyx
> file will trigger recompilation of everything that imports
> from it, whether the change affects the external interface
> or not.

The way I did this in Pex, is to lie with timestamps. When the .pxd file
is about to be generated, the old one is read in, and the APIs are compared.
If they are the same, the new is still written (I use it for some magic
comments), but the timestamp is backdated.





From stefan_ml at behnel.de  Wed Jun 18 19:08:16 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jun 2008 19:08:16 +0200
Subject: [Cython] an issue with generated C API's
In-Reply-To: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
References: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
Message-ID: <48594100.2050803@behnel.de>

Hi,

Lisandro Dalcin wrote:
> Currently, if one has something like this:
> 
> # mymod.pyx
> cdef public api class Foo [type PyFoo_Type, object PyFooObject]:
>     cdef int bar
> 
> The generated C header file contains this:
> 
> # mymod.h
> struct PyFooObject {
>   PyObject_HEAD
>   int bar;
> };
> 
> 
> Could the generated code be changed to say like this ??:
> 
> typedef struct PyFooObject {
>   PyObject_HEAD
>   int bar;
> } PyFooObject;

This patch works for me. It changes the declaration also in the .c files,
though. Any objections?

Stefan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: typedef-structs.patch
Type: text/x-patch
Size: 603 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080618/9cc6840e/attachment-0001.bin 

From dalcinl at gmail.com  Wed Jun 18 19:11:45 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 18 Jun 2008 14:11:45 -0300
Subject: [Cython] cross project cimport
In-Reply-To: <4858A68A.9020702@behnel.de>
References: <6ce0ac130806171256v5cc08839qf7225417346c1545@mail.gmail.com>
	<4FF573E9-1204-42DF-93F5-A926F1C1DB3D@math.washington.edu>
	<6ce0ac130806171517x3065766ch6ac5d3e3d9f75702@mail.gmail.com>
	<4858A68A.9020702@behnel.de>
Message-ID: <e7ba66e40806181011se4ef6fdv97627c9e5c2bfdb6@mail.gmail.com>

On 6/18/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> The mpi4py project can do that in their setup.py by making the public files
>  package data.

I'm just implementing that...

> I don't currently know if Cython searches the standard package
>  directory of Python (site-packages), but if not, it would be a good idea to
>  enable that.

Not sure about this (think about eggs!!). IMHO, the best way to go is
the NumPy way. If I do:

import mpi4py
mpi4py_include = mpi4py.get_include()

then I get a path that I sould pass to Cython and C compiler with -I
flag and then all just work.





-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Jun 18 19:30:16 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 18 Jun 2008 14:30:16 -0300
Subject: [Cython] an issue with generated C API's
In-Reply-To: <48594100.2050803@behnel.de>
References: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
	<48594100.2050803@behnel.de>
Message-ID: <e7ba66e40806181030s702bb3a3m31528cdde86cde2c@mail.gmail.com>

Mmm, not sure if that is completelly OK. The generated code says

typedef struct PyFooObject {
  PyObject_HEAD
  int bar;
};

but I would expect (note the final 'PyFooObject')

typedef struct PyFooObject {
  PyObject_HEAD
  int bar;
} PyFooObject;


With your patch, GCC generates this warning :

apitest.c:182: warning: useless storage class specifier in empty declaration



On 6/18/08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
>
>  Lisandro Dalcin wrote:
>  > Currently, if one has something like this:
>  >
>  > # mymod.pyx
>  > cdef public api class Foo [type PyFoo_Type, object PyFooObject]:
>  >     cdef int bar
>  >
>  > The generated C header file contains this:
>  >
>  > # mymod.h
>  > struct PyFooObject {
>  >   PyObject_HEAD
>  >   int bar;
>  > };
>  >
>  >
>  > Could the generated code be changed to say like this ??:
>  >
>  > typedef struct PyFooObject {
>  >   PyObject_HEAD
>  >   int bar;
>  > } PyFooObject;
>
>
> This patch works for me. It changes the declaration also in the .c files,
>  though. Any objections?
>
>
>  Stefan
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>
>
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From rebirth at orcon.net.nz  Wed Jun 18 22:13:46 2008
From: rebirth at orcon.net.nz (David McNab)
Date: Thu, 19 Jun 2008 08:13:46 +1200
Subject: [Cython] initialised cdef class variables
In-Reply-To: <4858538D.8090902@canterbury.ac.nz>
References: <1213670145.11944.22.camel@rebirth.conscious.co.nz>
	<3A5C38B3-3536-42C3-B7EB-5404A6DB28E2@math.washington.edu>
	<4858538D.8090902@canterbury.ac.nz>
Message-ID: <1213820026.4349.16.camel@rebirth.conscious.co.nz>

On Wed, 2008-06-18 at 12:15 +1200, Greg Ewing wrote:
> Another approach you might be able to use in the meantime
> is creating a Python subclass of the extension type to
> hold the class attributes.

Thanks, Greg. That is indeed a valid workaround. However, it loses the
performance benefits of declaring the class as a Pyrex/Cython 'cdef
class'.

Cheers
David




From rebirth at orcon.net.nz  Thu Jun 19 00:12:53 2008
From: rebirth at orcon.net.nz (David McNab)
Date: Thu, 19 Jun 2008 10:12:53 +1200
Subject: [Cython] initialised cdef class variables
In-Reply-To: <3A5C38B3-3536-42C3-B7EB-5404A6DB28E2@math.washington.edu>
References: <1213670145.11944.22.camel@rebirth.conscious.co.nz>
	<3A5C38B3-3536-42C3-B7EB-5404A6DB28E2@math.washington.edu>
Message-ID: <1213827173.4349.19.camel@rebirth.conscious.co.nz>

On Mon, 2008-06-16 at 22:59 -0700, Robert Bradshaw wrote:
> Yes, we should support this. (Note that this will not be a class- 
> level cdef attribute however, it would just change the initial value  
> for the instance member)

Thanks for your support.
A strategy which simply sets the initial value of each instance member
would do just fine.

> Please file a report at http://trac.cython.org/cython_trac/

Filed.

Cheers
David




From greg.ewing at canterbury.ac.nz  Thu Jun 19 02:34:22 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Jun 2008 12:34:22 +1200
Subject: [Cython] Dev days status
In-Reply-To: <080E7962-F811-4427-A19A-CBBBA73490F9@math.washington.edu>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de> <48585B91.5050802@canterbury.ac.nz>
	<080E7962-F811-4427-A19A-CBBBA73490F9@math.washington.edu>
Message-ID: <4859A98E.60201@canterbury.ac.nz>

Robert Bradshaw wrote:

> The current behavior leads to confusion too.

In what way?

> No false positives.

In that case you're going to have to do some careful
analysis. For example, re-ordering type declarations
doesn't change the interface, but re-ordering C
attributes or C methods of an extension type does.

-- 
Greg

From robertwb at math.washington.edu  Thu Jun 19 02:45:18 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 18 Jun 2008 17:45:18 -0700
Subject: [Cython] Dev days status
In-Reply-To: <4859A98E.60201@canterbury.ac.nz>
References: <C44B7088-748F-4AA7-8AA3-17C18C2767F4@math.washington.edu>
	<4857585D.4000004@behnel.de> <48585B91.5050802@canterbury.ac.nz>
	<080E7962-F811-4427-A19A-CBBBA73490F9@math.washington.edu>
	<4859A98E.60201@canterbury.ac.nz>
Message-ID: <88D9F459-9A9B-482B-9F89-8385E60CD62A@math.washington.edu>

On Jun 18, 2008, at 5:34 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> The current behavior leads to confusion too.
>
> In what way?

That exceptions aren't propagated upwards by default. (This is more  
for people who write lots of Cython code, instead of just wrapping  
libraries.)

>> No false positives.
>
> In that case you're going to have to do some careful
> analysis. For example, re-ordering type declarations
> doesn't change the interface, but re-ordering C
> attributes or C methods of an extension type does.

Yes. Many times, if the .pyx file changed, it will need to be parsed.  
If .pxd files are around then they can be used as they are now if one  
gets big enough to care about that sort of thing.

- Robert




From greg.ewing at canterbury.ac.nz  Thu Jun 19 02:59:38 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Jun 2008 12:59:38 +1200
Subject: [Cython] initialised cdef class variables
In-Reply-To: <1213820026.4349.16.camel@rebirth.conscious.co.nz>
References: <1213670145.11944.22.camel@rebirth.conscious.co.nz>
	<3A5C38B3-3536-42C3-B7EB-5404A6DB28E2@math.washington.edu>
	<4858538D.8090902@canterbury.ac.nz>
	<1213820026.4349.16.camel@rebirth.conscious.co.nz>
Message-ID: <4859AF7A.4090905@canterbury.ac.nz>

David McNab wrote:

> Thanks, Greg. That is indeed a valid workaround. However, it loses the
> performance benefits of declaring the class as a Pyrex/Cython 'cdef
> class'.

Not all of them -- access to C attributes and C methods
will still be just as fast.

Also, if you declare the Python subclass with __slots__,
no __dict__ will be allocated for it. If you also don't
give it any Python methods, I would expect the performance
difference from a plain extension class to be negligible
for most things.

-- 
Greg

From ellisonbg.net at gmail.com  Thu Jun 19 03:11:18 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 18 Jun 2008 19:11:18 -0600
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
Message-ID: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>

HI,

I am running cython-devel and have defined a simple extension type.  I
am finding that:

IF:  there is a __init__.py file in the same directory as my pxd and pyx file

THEN: the pxd file is not processed and my extension type won't have
any attributes defined in the C code

This reproduces my problem:

$ cat shrubbing.pxd
cdef class Shrubbery:
    cdef int width
    cdef int length

$ cat shrubbing.pyx
cdef class Shrubbery:
    def __new__(self, int w, int l):
        self.width = w
        self.length = l

$ ls
__init__.py	shrubbing.pxd	shrubbing.pyx
$ cython shrubbing.pyx

This is what shrubbing.c has for the extension type struct:

struct __pyx_obj_4temp_9shrubbing_Shrubbery {
  PyObject_HEAD
};

Notice, there are no attributes!!!!

But not if I do:

$ rm __init__.py
$ ls
shrubbing.c	shrubbing.pxd	shrubbing.pyx
$ cython shrubbing.pyx

I get the attributes in the extension type:

struct __pyx_obj_9shrubbing_Shrubbery {
  PyObject_HEAD
  int width;
  int length;
};

I don't see how I could be doing something wrong.  I swear this code
was working just the other day (this is not really my code, but it
does the same thing).  The crazy thing is that I know sage uses
__init__.py files in directories that have pxd and pyx files just like
I am doing.

Brian

From robertwb at math.washington.edu  Thu Jun 19 03:14:18 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 18 Jun 2008 18:14:18 -0700
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
Message-ID: <8C3473BD-B1FC-4FDD-85B1-BD36F4DD228F@math.washington.edu>

Is there an __init__ file the next level up?

On Jun 18, 2008, at 6:11 PM, Brian Granger wrote:

> HI,
>
> I am running cython-devel and have defined a simple extension type.  I
> am finding that:
>
> IF:  there is a __init__.py file in the same directory as my pxd  
> and pyx file
>
> THEN: the pxd file is not processed and my extension type won't have
> any attributes defined in the C code
>
> This reproduces my problem:
>
> $ cat shrubbing.pxd
> cdef class Shrubbery:
>     cdef int width
>     cdef int length
>
> $ cat shrubbing.pyx
> cdef class Shrubbery:
>     def __new__(self, int w, int l):
>         self.width = w
>         self.length = l
>
> $ ls
> __init__.py	shrubbing.pxd	shrubbing.pyx
> $ cython shrubbing.pyx
>
> This is what shrubbing.c has for the extension type struct:
>
> struct __pyx_obj_4temp_9shrubbing_Shrubbery {
>   PyObject_HEAD
> };
>
> Notice, there are no attributes!!!!
>
> But not if I do:
>
> $ rm __init__.py
> $ ls
> shrubbing.c	shrubbing.pxd	shrubbing.pyx
> $ cython shrubbing.pyx
>
> I get the attributes in the extension type:
>
> struct __pyx_obj_9shrubbing_Shrubbery {
>   PyObject_HEAD
>   int width;
>   int length;
> };
>
> I don't see how I could be doing something wrong.  I swear this code
> was working just the other day (this is not really my code, but it
> does the same thing).  The crazy thing is that I know sage uses
> __init__.py files in directories that have pxd and pyx files just like
> I am doing.
>
> Brian
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ellisonbg.net at gmail.com  Thu Jun 19 03:17:20 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 18 Jun 2008 19:17:20 -0600
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <8C3473BD-B1FC-4FDD-85B1-BD36F4DD228F@math.washington.edu>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
	<8C3473BD-B1FC-4FDD-85B1-BD36F4DD228F@math.washington.edu>
Message-ID: <6ce0ac130806181817j28936a30pc3867ec8fcc44137@mail.gmail.com>

All the files are in $HOME/temp and I don't have an __init__.py file in $HOME.

In [165]: pwd
Out[165]: '/Users/bgranger'

In [166]: ls temp
__init__.py	shrubbing.c	shrubbing.pxd	shrubbing.pyx

In my real code though I do have an __init__.py in the .. dir.

Thanks for the quick reply!  Let me know if I can help in any way as I
need to get this one figured out quickly.

Brian


On Wed, Jun 18, 2008 at 7:14 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Is there an __init__ file the next level up?
>
> On Jun 18, 2008, at 6:11 PM, Brian Granger wrote:
>
>> HI,
>>
>> I am running cython-devel and have defined a simple extension type.  I
>> am finding that:
>>
>> IF:  there is a __init__.py file in the same directory as my pxd
>> and pyx file
>>
>> THEN: the pxd file is not processed and my extension type won't have
>> any attributes defined in the C code
>>
>> This reproduces my problem:
>>
>> $ cat shrubbing.pxd
>> cdef class Shrubbery:
>>     cdef int width
>>     cdef int length
>>
>> $ cat shrubbing.pyx
>> cdef class Shrubbery:
>>     def __new__(self, int w, int l):
>>         self.width = w
>>         self.length = l
>>
>> $ ls
>> __init__.py   shrubbing.pxd   shrubbing.pyx
>> $ cython shrubbing.pyx
>>
>> This is what shrubbing.c has for the extension type struct:
>>
>> struct __pyx_obj_4temp_9shrubbing_Shrubbery {
>>   PyObject_HEAD
>> };
>>
>> Notice, there are no attributes!!!!
>>
>> But not if I do:
>>
>> $ rm __init__.py
>> $ ls
>> shrubbing.c   shrubbing.pxd   shrubbing.pyx
>> $ cython shrubbing.pyx
>>
>> I get the attributes in the extension type:
>>
>> struct __pyx_obj_9shrubbing_Shrubbery {
>>   PyObject_HEAD
>>   int width;
>>   int length;
>> };
>>
>> I don't see how I could be doing something wrong.  I swear this code
>> was working just the other day (this is not really my code, but it
>> does the same thing).  The crazy thing is that I know sage uses
>> __init__.py files in directories that have pxd and pyx files just like
>> I am doing.
>>
>> Brian
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From greg.ewing at canterbury.ac.nz  Thu Jun 19 03:38:57 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Jun 2008 13:38:57 +1200
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
Message-ID: <4859B8B1.9010508@canterbury.ac.nz>

Brian Granger wrote:

> IF:  there is a __init__.py file in the same directory as my pxd and pyx file
> 
> THEN: the pxd file is not processed and my extension type won't have
> any attributes defined in the C code

I think this is a result of a recent change to Pyrex that
has been merged into Cython.

If a directory contains an __init__.py or __init__.pyx file,
it's now assumed to be a package directory. So, for example,
if you have a directory structure

   foo/
     __init__.py
     shrubbing.pxd
     shrubbing.pyx

then the shrubbing module is assumed to belong to a package
called 'foo', and its fully qualified module name is
'foo.shrubbing'.

So when Pyrex wants to find out whether there is a .pxd
file for shrubbing, it looks for one corresponding to a
module called 'foo.shrubbing'. It does this by searching
the include path for a top-level package directory called
'foo' containing a file called 'shrubbing.pxd'.

However, if foo is the current directory you're running
the compiler from, and you haven't added foo to the
include path using a -I option, then it won't be on
the include path, and the .pxd won't be found.

What to do about this depends on whether you really
intend the module to reside in a package.

If you intend shrubbing to be a top-level module, you
will have to move it somewhere else where there is
no __init__.* file.

If you do intend it to reside in a package, then there
are two alternatives:

1) cd to the directory containing foo and compile
    from there:

      cd ..; cython foo/shrubbing.pyx

2) arrange for the directory containing foo to be
    passed as a -I option, e.g.

      cython -I .. shrubbing.pyx

Arguably this behaviour is not very desirable, and I'll
see if I can do something about it.

-- 
Greg

From ellisonbg.net at gmail.com  Thu Jun 19 04:18:32 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 18 Jun 2008 20:18:32 -0600
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <4859B8B1.9010508@canterbury.ac.nz>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
	<4859B8B1.9010508@canterbury.ac.nz>
Message-ID: <6ce0ac130806181918y3deffbf5web5c77f3f61282f3@mail.gmail.com>

> I think this is a result of a recent change to Pyrex that
> has been merged into Cython.

What is the last version of Cython that doesn't have this problem?  I
need to move quickly on a project and if I can't figure out a way of
handling these things in cython-devel, I will need to move back to a
stable release that is OK.

> If you do intend it to reside in a package, then there
> are two alternatives:

I do have this intention, but...

> 1) cd to the directory containing foo and compile
>    from there:
>
>      cd ..; cython foo/shrubbing.pyx

This does not work though.  I have to do:

cd ..
cython -I . foo/shrubbing.pyx


> 2) arrange for the directory containing foo to be
>    passed as a -I option, e.g.
>
>      cython -I .. shrubbing.pyx

This does work

> Arguably this behaviour is not very desirable, and I'll
> see if I can do something about it.

I understand the spirit of this.  We have a complex project with
multiple subprojects and this would allow us to run cython from the
top level dir by just including -I .  I will give this a try and see
if this will work for my scenario.  Previously, I was having to cd to
the dir with the pyx file and this could be an improvement.  But,
maybe the following algorithm would make sense:

1.  First check for the pxd file in the same dir as the pyx file.  If
found use it and create the .c file in that dir.

2.  If the pxd is not found there, then start to use this other logic.
 But, in this case, it would be nice if . was always on the include
path.

In either case, the behavior needs to be documented better.

Thanks for the help.

Brian


> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From ellisonbg.net at gmail.com  Thu Jun 19 04:41:31 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 18 Jun 2008 20:41:31 -0600
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <4859B8B1.9010508@canterbury.ac.nz>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
	<4859B8B1.9010508@canterbury.ac.nz>
Message-ID: <6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>

I now have the following approach working with my project:

I call cython from the top-level dir (the one containing setup.py):

cython -I. distarray/core/maps_fast.pyx

And this works fine.  I can run with this approach and some ways it is
nicer for situations like this.  But I still think cython should first
look in the directory containing the pyx file first.

Cheers,

Brian

On Wed, Jun 18, 2008 at 7:38 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Brian Granger wrote:
>
>> IF:  there is a __init__.py file in the same directory as my pxd and pyx file
>>
>> THEN: the pxd file is not processed and my extension type won't have
>> any attributes defined in the C code
>
> I think this is a result of a recent change to Pyrex that
> has been merged into Cython.
>
> If a directory contains an __init__.py or __init__.pyx file,
> it's now assumed to be a package directory. So, for example,
> if you have a directory structure
>
>   foo/
>     __init__.py
>     shrubbing.pxd
>     shrubbing.pyx
>
> then the shrubbing module is assumed to belong to a package
> called 'foo', and its fully qualified module name is
> 'foo.shrubbing'.
>
> So when Pyrex wants to find out whether there is a .pxd
> file for shrubbing, it looks for one corresponding to a
> module called 'foo.shrubbing'. It does this by searching
> the include path for a top-level package directory called
> 'foo' containing a file called 'shrubbing.pxd'.
>
> However, if foo is the current directory you're running
> the compiler from, and you haven't added foo to the
> include path using a -I option, then it won't be on
> the include path, and the .pxd won't be found.
>
> What to do about this depends on whether you really
> intend the module to reside in a package.
>
> If you intend shrubbing to be a top-level module, you
> will have to move it somewhere else where there is
> no __init__.* file.
>
> If you do intend it to reside in a package, then there
> are two alternatives:
>
> 1) cd to the directory containing foo and compile
>    from there:
>
>      cd ..; cython foo/shrubbing.pyx
>
> 2) arrange for the directory containing foo to be
>    passed as a -I option, e.g.
>
>      cython -I .. shrubbing.pyx
>
> Arguably this behaviour is not very desirable, and I'll
> see if I can do something about it.
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dagss at student.matnat.uio.no  Thu Jun 19 04:43:31 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 18 Jun 2008 19:43:31 -0700
Subject: [Cython] with statement support ready
Message-ID: <4859C7D3.3020509@student.matnat.uio.no>

This is just to let you know that the with statement is now supported in 
my branch (cython-dagss). One needs to do "from __future__ import 
with_statement" in order to use it; I'm not sure how this will behave in 
Py3 (I just copied Stefan's mechanism for unicode literals...).

Also I don't make "as" a reserved word even if "with" is turned on. This 
is different from Python however it doesn't mean that we are less 
compatible with any kind of source code, only that Cython will accept 
stuff that Python doesn't. (I didn't find the added complexity worth it; 
thoughts? I suppose it is just a matter of having Parsing.py modify the 
list of reserved in the scanner at the right time....)

Dag Sverre

From robertwb at math.washington.edu  Thu Jun 19 04:52:58 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 18 Jun 2008 19:52:58 -0700
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
	<4859B8B1.9010508@canterbury.ac.nz>
	<6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>
Message-ID: <DEB1B289-A5AF-42C8-80C6-529DE81AE453@math.washington.edu>

Yes, I agree.

On Jun 18, 2008, at 7:41 PM, Brian Granger wrote:

> I now have the following approach working with my project:
>
> I call cython from the top-level dir (the one containing setup.py):
>
> cython -I. distarray/core/maps_fast.pyx
>
> And this works fine.  I can run with this approach and some ways it is
> nicer for situations like this.  But I still think cython should first
> look in the directory containing the pyx file first.
>
> Cheers,
>
> Brian
>
> On Wed, Jun 18, 2008 at 7:38 PM, Greg Ewing  
> <greg.ewing at canterbury.ac.nz> wrote:
>> Brian Granger wrote:
>>
>>> IF:  there is a __init__.py file in the same directory as my pxd  
>>> and pyx file
>>>
>>> THEN: the pxd file is not processed and my extension type won't have
>>> any attributes defined in the C code
>>
>> I think this is a result of a recent change to Pyrex that
>> has been merged into Cython.
>>
>> If a directory contains an __init__.py or __init__.pyx file,
>> it's now assumed to be a package directory. So, for example,
>> if you have a directory structure
>>
>>   foo/
>>     __init__.py
>>     shrubbing.pxd
>>     shrubbing.pyx
>>
>> then the shrubbing module is assumed to belong to a package
>> called 'foo', and its fully qualified module name is
>> 'foo.shrubbing'.
>>
>> So when Pyrex wants to find out whether there is a .pxd
>> file for shrubbing, it looks for one corresponding to a
>> module called 'foo.shrubbing'. It does this by searching
>> the include path for a top-level package directory called
>> 'foo' containing a file called 'shrubbing.pxd'.
>>
>> However, if foo is the current directory you're running
>> the compiler from, and you haven't added foo to the
>> include path using a -I option, then it won't be on
>> the include path, and the .pxd won't be found.
>>
>> What to do about this depends on whether you really
>> intend the module to reside in a package.
>>
>> If you intend shrubbing to be a top-level module, you
>> will have to move it somewhere else where there is
>> no __init__.* file.
>>
>> If you do intend it to reside in a package, then there
>> are two alternatives:
>>
>> 1) cd to the directory containing foo and compile
>>    from there:
>>
>>      cd ..; cython foo/shrubbing.pyx
>>
>> 2) arrange for the directory containing foo to be
>>    passed as a -I option, e.g.
>>
>>      cython -I .. shrubbing.pyx
>>
>> Arguably this behaviour is not very desirable, and I'll
>> see if I can do something about it.
>>
>> --
>> Greg
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Thu Jun 19 06:59:35 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Jun 2008 06:59:35 +0200
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <DEB1B289-A5AF-42C8-80C6-529DE81AE453@math.washington.edu>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>	<4859B8B1.9010508@canterbury.ac.nz>	<6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>
	<DEB1B289-A5AF-42C8-80C6-529DE81AE453@math.washington.edu>
Message-ID: <4859E7B7.2090102@behnel.de>

Hi,

Robert Bradshaw wrote:
> On Jun 18, 2008, at 7:41 PM, Brian Granger wrote:
>> I now have the following approach working with my project:
>>
>> I call cython from the top-level dir (the one containing setup.py):
>>
>> cython -I. distarray/core/maps_fast.pyx
>>
>> And this works fine.  I can run with this approach and some ways it is
>> nicer for situations like this.  But I still think cython should first
>> look in the directory containing the pyx file first.
>>
> Yes, I agree.

Note that this might break module name detection, though.

What I could imagine as a solution is this:

- Determine the package path to the currently compiled module by searching
__init__.py files (just as it's done now)

- Then add the top-level directory of this search to the import path
automatically.

That also solves the "add . automatically" problem, by the way.

Stefan

From greg.ewing at canterbury.ac.nz  Thu Jun 19 06:59:41 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Jun 2008 16:59:41 +1200
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
Message-ID: <4859E7BD.6070108@canterbury.ac.nz>

Brian Granger wrote:

> IF:  there is a __init__.py file in the same directory as my pxd and pyx file
> THEN: the pxd file is not processed

Further to my last message on this, I just tried
this with Pyrex, and it does *not* seem to have
this problem. So it must be something that
Cython is doing differently from Pyrex.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Jun 19 07:01:38 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Jun 2008 17:01:38 +1200
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
	<4859B8B1.9010508@canterbury.ac.nz>
	<6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>
Message-ID: <4859E832.3000405@canterbury.ac.nz>

Brian Granger wrote:

> cython -I. distarray/core/maps_fast.pyx

You shouldn't need the -I. there, as the current
directory should always be considered part of the
include path.

-- 
Greg

From stefan_ml at behnel.de  Thu Jun 19 07:16:45 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Jun 2008 07:16:45 +0200
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <4859E7BD.6070108@canterbury.ac.nz>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
	<4859E7BD.6070108@canterbury.ac.nz>
Message-ID: <4859EBBD.4030004@behnel.de>

Hi,

Greg Ewing wrote:
> Brian Granger wrote:
> 
>> IF:  there is a __init__.py file in the same directory as my pxd and pyx file
>> THEN: the pxd file is not processed
> 
> Further to my last message on this, I just tried
> this with Pyrex, and it does *not* seem to have
> this problem. So it must be something that
> Cython is doing differently from Pyrex.

Hmmm, maybe I forgot to merge in one of those "tests pass" commits...

I'll see if I can fix it. I think the way I proposed in my other post is the
right way to go about it.

Stefan

From greg.ewing at canterbury.ac.nz  Thu Jun 19 07:18:57 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 19 Jun 2008 17:18:57 +1200
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <4859E7B7.2090102@behnel.de>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>
	<4859B8B1.9010508@canterbury.ac.nz>
	<6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>
	<DEB1B289-A5AF-42C8-80C6-529DE81AE453@math.washington.edu>
	<4859E7B7.2090102@behnel.de>
Message-ID: <4859EC41.90703@canterbury.ac.nz>

Stefan Behnel wrote:

> - Determine the package path to the currently compiled module by searching
> __init__.py files (just as it's done now)
> 
> - Then add the top-level directory of this search to the import path
> automatically.

That's more or less what Pyrex is currently doing.
Something must have got lost in translation.

The key piece of code is in Main.Context:

   def find_pxd_file(self, qualified_name, pos):
     #  Search include path for the .pxd file corresponding to the
     #  given fully-qualified module name.
     #  Will find either a dotted filename or a file in a
     #  package directory. If a source file position is given,
     #  the directory containing the source file is searched first
     #  for a dotted filename, and its containing package root
     #  directory is searched first for a non-dotted filename.
     return self.search_package_directories(qualified_name, ".pxd", pos)

which is called with 'pos' containing the pathname of the
.pyx file being compiled, and search_package_directories
looks in the top-level package directory of that file before
searching the main include path.

-- 
Greg

From languitar at semipol.de  Thu Jun 19 14:07:22 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Thu, 19 Jun 2008 14:07:22 +0200
Subject: [Cython] Defaults in cdef or cpdef methods
Message-ID: <485A4BFA.20806@semipol.de>

Hi,

is there a was to use default values for parameters in cdef or cpdef
methods? I observed that using them in the pxd files caused a cython
compiler error and using the only in the implementation causes a C
compiler error.

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/28a9fc6d/attachment.pgp 

From gael.varoquaux at normalesup.org  Thu Jun 19 16:43:28 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 19 Jun 2008 16:43:28 +0200
Subject: [Cython] Make the Cython workflow easier
Message-ID: <20080619144328.GC32587@phare.normalesup.org>

Hi guys,

Cython is great, and everybody knows it.

I have been writing an article geared toward beginners about Python in
science, and the editor asked me a for part where I talk about compiled
languages and Python. While I was writing this part, striving to make it
as easy as possible for beginners, it stroke me that, in my workflow, I
had to write a setup.py, or a Makefile, for cython modules. On the other
hand, using scipy.weave.inline, the compile step was transparent. Writing
and running the setup.py breaks my worflow while developping. What are
the other option for building (appart from os.system...gcc). Is there an
established workflow without a separate build step. I think Sage has
this, but that doesn't suite me, as I work outside of Sage.

It would be great if the compile could be done at import time. For
instance the first run of cython for primes.pyx could create a prime.py
containing the logics to do the compilation to _primes.so, and end by
cleaning up the namespace and importing everything from _primes.so.

What do you think?

Cheers,

Ga?l

From stefan_ml at behnel.de  Thu Jun 19 19:10:17 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 19 Jun 2008 19:10:17 +0200
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <20080619144328.GC32587@phare.normalesup.org>
References: <20080619144328.GC32587@phare.normalesup.org>
Message-ID: <485A92F9.8040203@behnel.de>

Hi,

Gael Varoquaux wrote:
> Writing and running the setup.py breaks my worflow while developping.

It's simple as long as you only use a single .pyx file, but as soon as a
project becomes larger and needs packaging, you need a setup.py anyway.


> What are the other option for building

Distutils have the advantage of providing a portable build setup. That's hard
to get right by hand.

You can also call into distutils directly, BTW, just like the runtests.py
script does. Maybe we could integrate something like that into the "cython"
front-end script to enable a straight build from source to module.


> It would be great if the compile could be done at import time. For
> instance the first run of cython for primes.pyx could create a prime.py
> containing the logics to do the compilation to _primes.so, and end by
> cleaning up the namespace and importing everything from _primes.so.

While I like the idea (it reminds me of tools like pyinline), I don't think
it's a very important use case. You'd end up compiling the module on each
(first) import as the runtime environment likely lacks write permission to the
Python package directory. So why not do it once during installation?

Stefan


From dagss at student.matnat.uio.no  Thu Jun 19 19:53:55 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 19 Jun 2008 10:53:55 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <20080619144328.GC32587@phare.normalesup.org>
References: <20080619144328.GC32587@phare.normalesup.org>
Message-ID: <485A9D33.40306@student.matnat.uio.no>

Gael Varoquaux wrote:
> I have been writing an article geared toward beginners about Python in
> science, and the editor asked me a for part where I talk about compiled
> languages and Python. While I was writing this part, striving to make it
> as easy as possible for beginners, it stroke me that, in my workflow, I
> had to write a setup.py, or a Makefile, for cython modules. On the other
> hand, using scipy.weave.inline, the compile step was transparent. Writing
> and running the setup.py breaks my worflow while developping. What are
> the other option for building (appart from os.system...gcc). Is there an
> established workflow without a separate build step. I think Sage has
> this, but that doesn't suite me, as I work outside of Sage.
>
> It would be great if the compile could be done at import time. For
> instance the first run of cython for primes.pyx could create a prime.py
> containing the logics to do the compilation to _primes.so, and end by
> cleaning up the namespace and importing everything from _primes.so.
>   
For now, have a look at the Cython preprocessor Pex at 
pexlang.sourceforge.net, which does (amongst other things) exactly what 
you ask for.

Dag Sverre

From dagss at student.matnat.uio.no  Thu Jun 19 19:57:23 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 19 Jun 2008 10:57:23 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <485A92F9.8040203@behnel.de>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
Message-ID: <485A9E03.8000105@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
>
> Gael Varoquaux wrote:
>   
>
>> It would be great if the compile could be done at import time. For
>> instance the first run of cython for primes.pyx could create a prime.py
>> containing the logics to do the compilation to _primes.so, and end by
>> cleaning up the namespace and importing everything from _primes.so.
>>     
>
> While I like the idea (it reminds me of tools like pyinline), I don't think
> it's a very important use case. You'd end up compiling the module on each
> (first) import as the runtime environment likely lacks write permission to the
> Python package directory. So why not do it once during installation?
>   
Remember that not everybody uses Cython for wrapping libraries; a lot of 
people uses it simply for writing code/programs. In those cases there 
often are no installation to speak about; and in fact, even having to 
run a makefile is an unecesarry step away from the convenience of Python.

Dag Sverre

From robertwb at math.washington.edu  Thu Jun 19 20:09:25 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Jun 2008 11:09:25 -0700
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <485A4BFA.20806@semipol.de>
References: <485A4BFA.20806@semipol.de>
Message-ID: <F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>

On Jun 19, 2008, at 5:07 AM, Johannes Wienke wrote:

> Hi,
>
> is there a was to use default values for parameters in cdef or cpdef
> methods? I observed that using them in the pxd files caused a cython
> compiler error and using the only in the implementation causes a C
> compiler error.

Yes, you need to declare them to have default parameters in .pxd  
files, then can use them in .pyx files. For example:

---- pxd ----

cdef foo(x, y=*)

---- pyx ----

cdef foo(x, y=None):
     print x,y

See http://wiki.cython.org/DifferencesFromPyrex#head- 
b2cbb6bf07c3c8c5bb0d10b89d9736eb0fc63440

- Robert

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/cb4908d2/attachment.pgp 

From languitar at semipol.de  Thu Jun 19 20:23:28 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Thu, 19 Jun 2008 20:23:28 +0200
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
References: <485A4BFA.20806@semipol.de>
	<F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
Message-ID: <485AA420.6050001@semipol.de>

Am 06/19/2008 08:09 PM schrieb Robert Bradshaw:
> On Jun 19, 2008, at 5:07 AM, Johannes Wienke wrote:
>> is there a was to use default values for parameters in cdef or cpdef
>> methods? I observed that using them in the pxd files caused a cython
>> compiler error and using the only in the implementation causes a C
>> compiler error.
> 
> Yes, you need to declare them to have default parameters in .pxd files,
> then can use them in .pyx files. For example:
> 
> ---- pxd ----
> 
> cdef foo(x, y=*)

The compiler told me to use '?'. Is that a "bug" in the error message?

> ---- pyx ----
> 
> cdef foo(x, y=None):
>     print x,y
> 
> See
> http://wiki.cython.org/DifferencesFromPyrex#head-b2cbb6bf07c3c8c5bb0d10b89d9736eb0fc63440

Thanks for the help!
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/21965c05/attachment.pgp 

From robertwb at math.washington.edu  Thu Jun 19 20:28:45 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Jun 2008 11:28:45 -0700
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <485AA420.6050001@semipol.de>
References: <485A4BFA.20806@semipol.de>
	<F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
	<485AA420.6050001@semipol.de>
Message-ID: <BE3697ED-7682-4BB7-8041-EAEBB9E5AFDB@math.washington.edu>

On Jun 19, 2008, at 11:23 AM, Johannes Wienke wrote:

> Am 06/19/2008 08:09 PM schrieb Robert Bradshaw:
>> On Jun 19, 2008, at 5:07 AM, Johannes Wienke wrote:
>>> is there a was to use default values for parameters in cdef or cpdef
>>> methods? I observed that using them in the pxd files caused a cython
>>> compiler error and using the only in the implementation causes a C
>>> compiler error.
>>
>> Yes, you need to declare them to have default parameters in .pxd  
>> files,
>> then can use them in .pyx files. For example:
>>
>> ---- pxd ----
>>
>> cdef foo(x, y=*)
>
> The compiler told me to use '?'. Is that a "bug" in the error message?

No, both are acceptable.

>
>> ---- pyx ----
>>
>> cdef foo(x, y=None):
>>     print x,y
>>
>> See
>> http://wiki.cython.org/DifferencesFromPyrex#head- 
>> b2cbb6bf07c3c8c5bb0d10b89d9736eb0fc63440
>
> Thanks for the help!
> Johannes
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/7c25351f/attachment-0001.pgp 

From languitar at semipol.de  Thu Jun 19 20:35:23 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Thu, 19 Jun 2008 20:35:23 +0200
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <BE3697ED-7682-4BB7-8041-EAEBB9E5AFDB@math.washington.edu>
References: <485A4BFA.20806@semipol.de>
	<F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
	<485AA420.6050001@semipol.de>
	<BE3697ED-7682-4BB7-8041-EAEBB9E5AFDB@math.washington.edu>
Message-ID: <485AA6EB.2010308@semipol.de>

Am 06/19/2008 08:28 PM schrieb Robert Bradshaw:
> On Jun 19, 2008, at 11:23 AM, Johannes Wienke wrote:
> 
>> Am 06/19/2008 08:09 PM schrieb Robert Bradshaw:
>>> On Jun 19, 2008, at 5:07 AM, Johannes Wienke wrote:
>>>> is there a was to use default values for parameters in cdef or cpdef
>>>> methods? I observed that using them in the pxd files caused a cython
>>>> compiler error and using the only in the implementation causes a C
>>>> compiler error.
>>>
>>> Yes, you need to declare them to have default parameters in .pxd files,
>>> then can use them in .pyx files. For example:
>>>
>>> ---- pxd ----
>>>
>>> cdef foo(x, y=*)
>>
>> The compiler told me to use '?'. Is that a "bug" in the error message?
> 
> No, both are acceptable.

Ok, then it didn't solve my problem. Jus tried that out and as both *
and ? are possible, I got the same error message:

build/temp.linux-x86_64-2.4/pyrex/plugins.c: In function 'PyObject*
__pyx_pf_4ship_7icewing_7plugins_14PluginInstance_init(PyObject*,
PyObject*, PyObject*)':
build/temp.linux-x86_64-2.4/pyrex/plugins.c:1228: error: invalid lvalue
in unary '&'
error: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1

My pxd file looks like that:

cpdef class PluginInstance(Plugin):
	#....
	cpdef getName(self)
	cpdef init(self, args=*)
	cpdef process(self, d)

And the method body:

    cpdef init(self, args = ""):
        argList = helpers.splitArgs(args)
        # ...

The line in the C code that gcc does complain about is:

__pyx_1 = (__pyx_skip_dispatch = 1, ((struct
__pyx_vtabstruct_4ship_7icewing_7plugins_PluginInstance *)((struct
__pyx_obj_4ship_7icewing_7plugins_PluginInstance
*)__pyx_v_self)->__pyx_vtab)->init(((struct
__pyx_obj_4ship_7icewing_7plugins_PluginInstance *)__pyx_v_self), ((void
*)&((struct
__pyx_opt_args_4ship_7icewing_7plugins_14PluginInstance_init){1,__pyx_v_args}))));
if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 93;
goto __pyx_L1;}

Is there something I am doing wrong?

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/56cf693a/attachment.pgp 

From gael.varoquaux at normalesup.org  Thu Jun 19 20:38:09 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 19 Jun 2008 20:38:09 +0200
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <485A92F9.8040203@behnel.de>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
Message-ID: <20080619183809.GB10875@phare.normalesup.org>

On Thu, Jun 19, 2008 at 07:10:17PM +0200, Stefan Behnel wrote:
> > It would be great if the compile could be done at import time. For
> > instance the first run of cython for primes.pyx could create a prime.py
> > containing the logics to do the compilation to _primes.so, and end by
> > cleaning up the namespace and importing everything from _primes.so.

> While I like the idea (it reminds me of tools like pyinline), I don't think
> it's a very important use case. You'd end up compiling the module on each
> (first) import as the runtime environment likely lacks write permission to the
> Python package directory. So why not do it once during installation?

I am targeting scientific users who spend most of their time developing
the algorithm, and not deploying it. And the issues you raised are easy
to solve using a disk cache in the user's directory, and checksum to see
if recompilation is necessary. scipy.weave.inline works like this and has
no real problem. Their is also of way of shipping the binary code (I am not
too sure how), to target users who do not have a compiler. If the hash of
the source code corresponds to hash stored when the code was compiled, no
recompile is forced.

Ga?l

From robertwb at math.washington.edu  Thu Jun 19 20:39:08 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Jun 2008 11:39:08 -0700
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <485AA6EB.2010308@semipol.de>
References: <485A4BFA.20806@semipol.de>
	<F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
	<485AA420.6050001@semipol.de>
	<BE3697ED-7682-4BB7-8041-EAEBB9E5AFDB@math.washington.edu>
	<485AA6EB.2010308@semipol.de>
Message-ID: <0509FBCD-1330-4204-AE50-3492AC6A95FD@math.washington.edu>

On Jun 19, 2008, at 11:35 AM, Johannes Wienke wrote:

> Am 06/19/2008 08:28 PM schrieb Robert Bradshaw:
>> On Jun 19, 2008, at 11:23 AM, Johannes Wienke wrote:
>>
>>> Am 06/19/2008 08:09 PM schrieb Robert Bradshaw:
>>>> On Jun 19, 2008, at 5:07 AM, Johannes Wienke wrote:
>>>>> is there a was to use default values for parameters in cdef or  
>>>>> cpdef
>>>>> methods? I observed that using them in the pxd files caused a  
>>>>> cython
>>>>> compiler error and using the only in the implementation causes a C
>>>>> compiler error.
>>>>
>>>> Yes, you need to declare them to have default parameters in .pxd  
>>>> files,
>>>> then can use them in .pyx files. For example:
>>>>
>>>> ---- pxd ----
>>>>
>>>> cdef foo(x, y=*)
>>>
>>> The compiler told me to use '?'. Is that a "bug" in the error  
>>> message?
>>
>> No, both are acceptable.
>
> Ok, then it didn't solve my problem. Jus tried that out and as both *
> and ? are possible, I got the same error message:
>
> build/temp.linux-x86_64-2.4/pyrex/plugins.c: In function 'PyObject*
> __pyx_pf_4ship_7icewing_7plugins_14PluginInstance_init(PyObject*,
> PyObject*, PyObject*)':
> build/temp.linux-x86_64-2.4/pyrex/plugins.c:1228: error: invalid  
> lvalue
> in unary '&'
> error: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1
>
> My pxd file looks like that:
>
> cpdef class PluginInstance(Plugin):
> 	#....
> 	cpdef getName(self)
> 	cpdef init(self, args=*)
> 	cpdef process(self, d)
>
> And the method body:
>
>     cpdef init(self, args = ""):
>         argList = helpers.splitArgs(args)
>         # ...
>
> The line in the C code that gcc does complain about is:
>
> __pyx_1 = (__pyx_skip_dispatch = 1, ((struct
> __pyx_vtabstruct_4ship_7icewing_7plugins_PluginInstance *)((struct
> __pyx_obj_4ship_7icewing_7plugins_PluginInstance
> *)__pyx_v_self)->__pyx_vtab)->init(((struct
> __pyx_obj_4ship_7icewing_7plugins_PluginInstance *)__pyx_v_self),  
> ((void
> *)&((struct
> __pyx_opt_args_4ship_7icewing_7plugins_14PluginInstance_init) 
> {1,__pyx_v_args}))));
> if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno  
> = 93;
> goto __pyx_L1;}
>
> Is there something I am doing wrong?

Hmm... what version of gcc are you using?


-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/08e11d73/attachment.pgp 

From robertwb at math.washington.edu  Thu Jun 19 20:40:46 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Jun 2008 11:40:46 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <20080619183809.GB10875@phare.normalesup.org>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
	<20080619183809.GB10875@phare.normalesup.org>
Message-ID: <96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>

On Jun 19, 2008, at 11:38 AM, Gael Varoquaux wrote:

> On Thu, Jun 19, 2008 at 07:10:17PM +0200, Stefan Behnel wrote:
>>> It would be great if the compile could be done at import time. For
>>> instance the first run of cython for primes.pyx could create a  
>>> prime.py
>>> containing the logics to do the compilation to _primes.so, and  
>>> end by
>>> cleaning up the namespace and importing everything from _primes.so.
>
>> While I like the idea (it reminds me of tools like pyinline), I  
>> don't think
>> it's a very important use case. You'd end up compiling the module  
>> on each
>> (first) import as the runtime environment likely lacks write  
>> permission to the
>> Python package directory. So why not do it once during installation?
>
> I am targeting scientific users who spend most of their time  
> developing
> the algorithm, and not deploying it. And the issues you raised are  
> easy
> to solve using a disk cache in the user's directory, and checksum  
> to see
> if recompilation is necessary. scipy.weave.inline works like this  
> and has
> no real problem. Their is also of way of shipping the binary code  
> (I am not
> too sure how), to target users who do not have a compiler. If the  
> hash of
> the source code corresponds to hash stored when the code was  
> compiled, no
> recompile is forced.

Compile-on-import is certainly something that I would like to  
support. You are right, the workflow is too complicated for the  
average user (especially coming from Python).

- Robert


From languitar at semipol.de  Thu Jun 19 20:43:05 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Thu, 19 Jun 2008 20:43:05 +0200
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <0509FBCD-1330-4204-AE50-3492AC6A95FD@math.washington.edu>
References: <485A4BFA.20806@semipol.de>
	<F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
	<485AA420.6050001@semipol.de>
	<BE3697ED-7682-4BB7-8041-EAEBB9E5AFDB@math.washington.edu>
	<485AA6EB.2010308@semipol.de>
	<0509FBCD-1330-4204-AE50-3492AC6A95FD@math.washington.edu>
Message-ID: <485AA8B9.1030901@semipol.de>

Am 06/19/2008 08:39 PM schrieb Robert Bradshaw:
> On Jun 19, 2008, at 11:35 AM, Johannes Wienke wrote:
> 
>> Am 06/19/2008 08:28 PM schrieb Robert Bradshaw:
>>> On Jun 19, 2008, at 11:23 AM, Johannes Wienke wrote:
>>>
>>>> Am 06/19/2008 08:09 PM schrieb Robert Bradshaw:
>>>>> On Jun 19, 2008, at 5:07 AM, Johannes Wienke wrote:
>>>>>> is there a was to use default values for parameters in cdef or cpdef
>>>>>> methods? I observed that using them in the pxd files caused a cython
>>>>>> compiler error and using the only in the implementation causes a C
>>>>>> compiler error.
>>>>>
>>>>> Yes, you need to declare them to have default parameters in .pxd
>>>>> files,
>>>>> then can use them in .pyx files. For example:
>>>>>
>>>>> ---- pxd ----
>>>>>
>>>>> cdef foo(x, y=*)
>>>>
>>>> The compiler told me to use '?'. Is that a "bug" in the error message?
>>>
>>> No, both are acceptable.
>>
>> Ok, then it didn't solve my problem. Jus tried that out and as both *
>> and ? are possible, I got the same error message:
>>
>> build/temp.linux-x86_64-2.4/pyrex/plugins.c: In function 'PyObject*
>> __pyx_pf_4ship_7icewing_7plugins_14PluginInstance_init(PyObject*,
>> PyObject*, PyObject*)':
>> build/temp.linux-x86_64-2.4/pyrex/plugins.c:1228: error: invalid lvalue
>> in unary '&'
>> error: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1
>>
>> My pxd file looks like that:
>>
>> cpdef class PluginInstance(Plugin):
>>     #....
>>     cpdef getName(self)
>>     cpdef init(self, args=*)
>>     cpdef process(self, d)
>>
>> And the method body:
>>
>>     cpdef init(self, args = ""):
>>         argList = helpers.splitArgs(args)
>>         # ...
>>
>> The line in the C code that gcc does complain about is:
>>
>> __pyx_1 = (__pyx_skip_dispatch = 1, ((struct
>> __pyx_vtabstruct_4ship_7icewing_7plugins_PluginInstance *)((struct
>> __pyx_obj_4ship_7icewing_7plugins_PluginInstance
>> *)__pyx_v_self)->__pyx_vtab)->init(((struct
>> __pyx_obj_4ship_7icewing_7plugins_PluginInstance *)__pyx_v_self), ((void
>> *)&((struct
>> __pyx_opt_args_4ship_7icewing_7plugins_14PluginInstance_init){1,__pyx_v_args}))));
>>
>> if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 93;
>> goto __pyx_L1;}
>>
>> Is there something I am doing wrong?
> 
> Hmm... what version of gcc are you using?

4.1.2 on amd64

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/9fedefa2/attachment.pgp 

From gael.varoquaux at normalesup.org  Thu Jun 19 20:44:26 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Thu, 19 Jun 2008 20:44:26 +0200
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
	<20080619183809.GB10875@phare.normalesup.org>
	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
Message-ID: <20080619184426.GC10875@phare.normalesup.org>

On Thu, Jun 19, 2008 at 11:40:46AM -0700, Robert Bradshaw wrote:
> Compile-on-import is certainly something that I would like to  
> support. You are right, the workflow is too complicated for the  
> average user (especially coming from Python).

Sweet I am happy to hear this. What got me into this thinking was really
writing an article, and thinking "this is more complicated than it needs
to be".

I'll patiently wait for this feature. These little details make cython
precious.

Ga?l

From robertwb at math.washington.edu  Thu Jun 19 21:00:59 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 19 Jun 2008 12:00:59 -0700
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <485AA8B9.1030901@semipol.de>
References: <485A4BFA.20806@semipol.de>
	<F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
	<485AA420.6050001@semipol.de>
	<BE3697ED-7682-4BB7-8041-EAEBB9E5AFDB@math.washington.edu>
	<485AA6EB.2010308@semipol.de>
	<0509FBCD-1330-4204-AE50-3492AC6A95FD@math.washington.edu>
	<485AA8B9.1030901@semipol.de>
Message-ID: <7D77C0D7-7A22-43C5-BBC0-E3644B65ECB4@math.washington.edu>

On Jun 19, 2008, at 11:43 AM, Johannes Wienke wrote:

...

>>>
>>> Ok, then it didn't solve my problem. Jus tried that out and as  
>>> both *
>>> and ? are possible, I got the same error message:
>>>
>>> build/temp.linux-x86_64-2.4/pyrex/plugins.c: In function 'PyObject*
>>> __pyx_pf_4ship_7icewing_7plugins_14PluginInstance_init(PyObject*,
>>> PyObject*, PyObject*)':
>>> build/temp.linux-x86_64-2.4/pyrex/plugins.c:1228: error: invalid  
>>> lvalue
>>> in unary '&'
>>> error: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1
>>>
>>> My pxd file looks like that:
>>>
>>> cpdef class PluginInstance(Plugin):
>>>     #....
>>>     cpdef getName(self)
>>>     cpdef init(self, args=*)
>>>     cpdef process(self, d)
>>>
>>> And the method body:
>>>
>>>     cpdef init(self, args = ""):
>>>         argList = helpers.splitArgs(args)
>>>         # ...
>>>
>>> The line in the C code that gcc does complain about is:
>>>
>>> __pyx_1 = (__pyx_skip_dispatch = 1, ((struct
>>> __pyx_vtabstruct_4ship_7icewing_7plugins_PluginInstance *)((struct
>>> __pyx_obj_4ship_7icewing_7plugins_PluginInstance
>>> *)__pyx_v_self)->__pyx_vtab)->init(((struct
>>> __pyx_obj_4ship_7icewing_7plugins_PluginInstance *)__pyx_v_self),  
>>> ((void
>>> *)&((struct
>>> __pyx_opt_args_4ship_7icewing_7plugins_14PluginInstance_init) 
>>> {1,__pyx_v_args}))));
>>>
>>> if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0];  
>>> __pyx_lineno = 93;
>>> goto __pyx_L1;}
>>>
>>> Is there something I am doing wrong?
>>
>> Hmm... what version of gcc are you using?
>
> 4.1.2 on amd64

Apparently this is valid C, but not valid C++. http://trac.cython.org/ 
cython_trac/ticket/21

C++ gurus, any ideas?

- Robert


-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080619/117b9e8b/attachment.pgp 

From fperez.net at gmail.com  Thu Jun 19 21:51:24 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 19 Jun 2008 12:51:24 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <20080619184426.GC10875@phare.normalesup.org>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
	<20080619183809.GB10875@phare.normalesup.org>
	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
	<20080619184426.GC10875@phare.normalesup.org>
Message-ID: <db6b5ecc0806191251r3c2de3c5g4879622afd54cd4b@mail.gmail.com>

On Thu, Jun 19, 2008 at 11:44 AM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Thu, Jun 19, 2008 at 11:40:46AM -0700, Robert Bradshaw wrote:
>> Compile-on-import is certainly something that I would like to
>> support. You are right, the workflow is too complicated for the
>> average user (especially coming from Python).
>
> Sweet I am happy to hear this. What got me into this thinking was really
> writing an article, and thinking "this is more complicated than it needs
> to be".

+1 from here, esp on a weave.inline-like approach.

I'm not writing an article about it, but just finished teaching a
seminar at Berkeley about Cython (part of an ongoing series on python
for scientists - https://cirl.berkeley.edu/view/Py4Science/WebHome)
and I kind of sneaked that problem under the rug by providing them
with examples that had the Makefile/setup.py premade, but it is an
issue.

The weave catalog/hash approach is in my opinion the way to go, and I
know Eric Jones (from Enthought, and the original author of that code)
has always wanted to have that machinery abstracted out and used as a
generic tool for integrating auto-generated extension modules.  If any
of you decide to go in that direction, I'd encourage you to look at
it.  Today, weave.iniline() assumes its input is C(++, really), but
the machinery could easily be generalized to allowing multiple
language 'backends' with a similar call API.  I think that would be
very useful.  weave.inline() is *extremely* popular with working
scientists (it's a guaranteed hit whenever I teach it) because of:

  1. its simplicity of workflow
  2. how cleanly it allows access to NumPy arrays via Blitz++.

I know Dag is already hard at work on #2 for Cython, I'm thrilled to
hear that #1 is also on the radar.

By the way, I cleaned up the primes example a little bit, added
docstrings and a proper set of tests (they use nose though for
running).  If you want to use this, feel free to grab it from here:

https://cirl.berkeley.edu/twiki/pub/Py4Science/WebHome/cython.tgz

It's the one from your Demo directory, just with more comments/info
and a simple set of tests (I'm trying to get people into the habit of
using doctests and proper tests around here).

Note: I removed the Numeric example completely because it's broken,
and only left the primes one to focus for my class.  So don't take the
makefile from there, but the .pyx/.py files are more commented
versions of yours that you can use for your next release, if you want.

Cheers,

f

From dalcinl at gmail.com  Thu Jun 19 23:29:05 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 19 Jun 2008 18:29:05 -0300
Subject: [Cython] Defaults in cdef or cpdef methods
In-Reply-To: <7D77C0D7-7A22-43C5-BBC0-E3644B65ECB4@math.washington.edu>
References: <485A4BFA.20806@semipol.de>
	<F2C51640-7585-4EAE-94AA-05D0A0752080@math.washington.edu>
	<485AA420.6050001@semipol.de>
	<BE3697ED-7682-4BB7-8041-EAEBB9E5AFDB@math.washington.edu>
	<485AA6EB.2010308@semipol.de>
	<0509FBCD-1330-4204-AE50-3492AC6A95FD@math.washington.edu>
	<485AA8B9.1030901@semipol.de>
	<7D77C0D7-7A22-43C5-BBC0-E3644B65ECB4@math.washington.edu>
Message-ID: <e7ba66e40806191429g38750920l987d07eeaf44c671@mail.gmail.com>

Mmm, perhaps it is valid C99, bot not sure if it is valid C89 ...

IMHO, the way default arguments is implemented should be changed, but
the only way I can imagine will always work is by introducing a new
scope (I mean, { ... } )in the generated code, declare the struct for
optional aguments, and next fill its fields, and finally pass the
pointer to the function call.


On 6/19/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Jun 19, 2008, at 11:43 AM, Johannes Wienke wrote:
>
>  ...
>
>
>
> >
> > >
> > > >
> > > > Ok, then it didn't solve my problem. Jus tried that out and as both *
> > > > and ? are possible, I got the same error message:
> > > >
> > > > build/temp.linux-x86_64-2.4/pyrex/plugins.c: In
> function 'PyObject*
> > > >
> __pyx_pf_4ship_7icewing_7plugins_14PluginInstance_init(PyObject*,
> > > > PyObject*, PyObject*)':
> > > > build/temp.linux-x86_64-2.4/pyrex/plugins.c:1228:
> error: invalid lvalue
> > > > in unary '&'
> > > > error: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1
> > > >
> > > > My pxd file looks like that:
> > > >
> > > > cpdef class PluginInstance(Plugin):
> > > >    #....
> > > >    cpdef getName(self)
> > > >    cpdef init(self, args=*)
> > > >    cpdef process(self, d)
> > > >
> > > > And the method body:
> > > >
> > > >    cpdef init(self, args = ""):
> > > >        argList = helpers.splitArgs(args)
> > > >        # ...
> > > >
> > > > The line in the C code that gcc does complain about is:
> > > >
> > > > __pyx_1 = (__pyx_skip_dispatch = 1, ((struct
> > > >
> __pyx_vtabstruct_4ship_7icewing_7plugins_PluginInstance
> *)((struct
> > > > __pyx_obj_4ship_7icewing_7plugins_PluginInstance
> > > > *)__pyx_v_self)->__pyx_vtab)->init(((struct
> > > > __pyx_obj_4ship_7icewing_7plugins_PluginInstance
> *)__pyx_v_self), ((void
> > > > *)&((struct
> > > >
> __pyx_opt_args_4ship_7icewing_7plugins_14PluginInstance_init){1,__pyx_v_args}))));
> > > >
> > > > if (unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno =
> 93;
> > > > goto __pyx_L1;}
> > > >
> > > > Is there something I am doing wrong?
> > > >
> > >
> > > Hmm... what version of gcc are you using?
> > >
> >
> > 4.1.2 on amd64
> >
>
>  Apparently this is valid C, but not valid C++.
> http://trac.cython.org/cython_trac/ticket/21
>
>  C++ gurus, any ideas?
>
>  - Robert
>
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>
>
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri Jun 20 07:19:30 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Jun 2008 07:19:30 +0200
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <485A9E03.8000105@student.matnat.uio.no>
References: <20080619144328.GC32587@phare.normalesup.org>	<485A92F9.8040203@behnel.de>
	<485A9E03.8000105@student.matnat.uio.no>
Message-ID: <485B3DE2.6060403@behnel.de>

Hi,

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Gael Varoquaux wrote:
>>> It would be great if the compile could be done at import time. For
>>> instance the first run of cython for primes.pyx could create a prime.py
>>> containing the logics to do the compilation to _primes.so, and end by
>>> cleaning up the namespace and importing everything from _primes.so.
>>>     
>> While I like the idea (it reminds me of tools like pyinline), I don't think
>> it's a very important use case. You'd end up compiling the module on each
>> (first) import as the runtime environment likely lacks write permission to the
>> Python package directory. So why not do it once during installation?
>>   
> Remember that not everybody uses Cython for wrapping libraries; a lot of 
> people uses it simply for writing code/programs. In those cases there 
> often are no installation to speak about

What I meant was that you will likely end up installing your software at some
point, which the Cython modules are a part of. There's little code that only
ever runs on your own computer(s).

But I would appreciate simplifying simple cases, especially if they help in
dropping the entry level. And interfacing Cython with other code generation
tools is always a value by itself.

Stefan

From stefan_ml at behnel.de  Fri Jun 20 07:21:30 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Jun 2008 07:21:30 +0200
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <20080619184426.GC10875@phare.normalesup.org>
References: <20080619144328.GC32587@phare.normalesup.org>	<485A92F9.8040203@behnel.de>	<20080619183809.GB10875@phare.normalesup.org>	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
	<20080619184426.GC10875@phare.normalesup.org>
Message-ID: <485B3E5A.8030605@behnel.de>

Hi,

Gael Varoquaux wrote:
> I'll patiently wait for this feature.

Why not invest a little time to write it yourself? Patience may be a virtue,
but a contribution makes sure you get what you want.

Stefan

From stefan_ml at behnel.de  Fri Jun 20 12:15:11 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 20 Jun 2008 12:15:11 +0200
Subject: [Cython] BUG?: __init__.py file messing up .pxd detection!!!
In-Reply-To: <4859EC41.90703@canterbury.ac.nz>
References: <6ce0ac130806181811p304860cfn1ce441483912b055@mail.gmail.com>	<4859B8B1.9010508@canterbury.ac.nz>	<6ce0ac130806181941g44b944eeled10a303b169bf93@mail.gmail.com>	<DEB1B289-A5AF-42C8-80C6-529DE81AE453@math.washington.edu>	<4859E7B7.2090102@behnel.de>
	<4859EC41.90703@canterbury.ac.nz>
Message-ID: <485B832F.3020008@behnel.de>

Hi,

Greg Ewing wrote:
> Stefan Behnel wrote:
> 
>> - Determine the package path to the currently compiled module by searching
>> __init__.py files (just as it's done now)
>>
>> - Then add the top-level directory of this search to the import path
>> automatically.
> 
> That's more or less what Pyrex is currently doing.
> Something must have got lost in translation.

Yes, there were concurrent changes regarding the file source position, which
accidentally let us drop the bit that inserts the package root directory into
the search path.

Should be fixed now.

Stefan


From gael.varoquaux at normalesup.org  Fri Jun 20 14:25:25 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 20 Jun 2008 14:25:25 +0200
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <485B3E5A.8030605@behnel.de>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
	<20080619183809.GB10875@phare.normalesup.org>
	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
	<20080619184426.GC10875@phare.normalesup.org>
	<485B3E5A.8030605@behnel.de>
Message-ID: <20080620122525.GC3651@phare.normalesup.org>

On Fri, Jun 20, 2008 at 07:21:30AM +0200, Stefan Behnel wrote:
> Gael Varoquaux wrote:
> > I'll patiently wait for this feature.

> Why not invest a little time to write it yourself? Patience may be a virtue,
> but a contribution makes sure you get what you want.

Because I am already contributing to too many projects, and my dispersion
harms my productivity. My fingers are twitching to have a go at this, but
other people are waiting for some new features on codebases I know, and
have thought a lot about, and on which I can easily be productive and
help the community.

I can understand that it would help you guys if I would pitch in, but I
am already trying to contribute to mayavi2, ipython and pyreport, in
addition to my day work, and there is only so much that I can do.

Sorry for refusing your offer, although in general I agree with you.

Ga?l

From dalcinl at gmail.com  Fri Jun 20 19:01:41 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 20 Jun 2008 14:01:41 -0300
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <20080620122525.GC3651@phare.normalesup.org>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
	<20080619183809.GB10875@phare.normalesup.org>
	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
	<20080619184426.GC10875@phare.normalesup.org>
	<485B3E5A.8030605@behnel.de>
	<20080620122525.GC3651@phare.normalesup.org>
Message-ID: <e7ba66e40806201001v60dbf67cxce6b8d085d0bc7ba@mail.gmail.com>

Gael, not sure if this will help you, but I wrote this script based in
distutils for building *.pyx files on the fly (perhaps with others
companion C/C++ sources). You basically use it as this:

$ cy2py file.pyx [otherfile1.c otherfile2.c]

or

$ cy2py -c++ file.pyx [otherfile1.cpp otherfile2.cpp]

NumPy includes are automatically added

The Cython-generated sources are left on the same place the pyx
resides. C++ flag can also be '-cxx' or '-cplus'.

The script also try to manage some environ vars like CC, CXX, and LD.
I usually build modules with MPI compiler wrappers, then I just use:

$ CC=mpicc cy2py file.pyx


Hope it helps. Perhaps with a bit of work you can easily extend this.
In particular, the processing of command line flags is completelly
naive.




On 6/20/08, Gael Varoquaux <gael.varoquaux at normalesup.org> wrote:
> On Fri, Jun 20, 2008 at 07:21:30AM +0200, Stefan Behnel wrote:
>  > Gael Varoquaux wrote:
>  > > I'll patiently wait for this feature.
>
>  > Why not invest a little time to write it yourself? Patience may be a virtue,
>  > but a contribution makes sure you get what you want.
>
>
> Because I am already contributing to too many projects, and my dispersion
>  harms my productivity. My fingers are twitching to have a go at this, but
>  other people are waiting for some new features on codebases I know, and
>  have thought a lot about, and on which I can easily be productive and
>  help the community.
>
>  I can understand that it would help you guys if I would pitch in, but I
>  am already trying to contribute to mayavi2, ipython and pyreport, in
>  addition to my day work, and there is only so much that I can do.
>
>  Sorry for refusing your offer, although in general I agree with you.
>
>  Ga?l
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cy2py
Type: application/octet-stream
Size: 2093 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080620/ba0c0048/attachment.obj 

From dagss at student.matnat.uio.no  Fri Jun 20 20:09:06 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Jun 2008 11:09:06 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <e7ba66e40806201001v60dbf67cxce6b8d085d0bc7ba@mail.gmail.com>
References: <20080619144328.GC32587@phare.normalesup.org>	<485A92F9.8040203@behnel.de>	<20080619183809.GB10875@phare.normalesup.org>	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>	<20080619184426.GC10875@phare.normalesup.org>	<485B3E5A.8030605@behnel.de>	<20080620122525.GC3651@phare.normalesup.org>
	<e7ba66e40806201001v60dbf67cxce6b8d085d0bc7ba@mail.gmail.com>
Message-ID: <485BF242.80408@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Gael, not sure if this will help you, but I wrote this script based in
> distutils for building *.pyx files on the fly (perhaps with others
> companion C/C++ sources). You basically use it as this:
>   
Also remember that Pex already has this feature (as a wrapper/build 
system for Cython):

pexlang.sourceforge.net

Dag Sverre

From fperez.net at gmail.com  Fri Jun 20 20:13:56 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Jun 2008 11:13:56 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <485BF242.80408@student.matnat.uio.no>
References: <20080619144328.GC32587@phare.normalesup.org>
	<485A92F9.8040203@behnel.de>
	<20080619183809.GB10875@phare.normalesup.org>
	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
	<20080619184426.GC10875@phare.normalesup.org>
	<485B3E5A.8030605@behnel.de>
	<20080620122525.GC3651@phare.normalesup.org>
	<e7ba66e40806201001v60dbf67cxce6b8d085d0bc7ba@mail.gmail.com>
	<485BF242.80408@student.matnat.uio.no>
Message-ID: <db6b5ecc0806201113y7a86adeds7f43d97f90c57c25@mail.gmail.com>

On Fri, Jun 20, 2008 at 11:09 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Lisandro Dalcin wrote:
>> Gael, not sure if this will help you, but I wrote this script based in
>> distutils for building *.pyx files on the fly (perhaps with others
>> companion C/C++ sources). You basically use it as this:
>>
> Also remember that Pex already has this feature (as a wrapper/build
> system for Cython):
>
> pexlang.sourceforge.net

This is just a generic request: when the dust settles a bit and you
guys have a clearer picture of where the whole pex/cython integration
regarding numpy is going to go, could you provide a little summary?
I'm super interested in this (and I can guarantee I'm not the only
one) but right now things seem to be in a strong state of flux, so I'm
OK waiting.

Furthermore, it would be great if you think some of  this will be in
reasonable state (even alpha, but solid enough for knowing where it
will  go) by August, to present it at scipy 2008:

http://conference.scipy.org/

Robert B. is already coming to present a cython tutorial, so perhaps a
short talk on this would be OK for him to give.  It would be very well
 received.

Cheers,

f

From dagss at student.matnat.uio.no  Fri Jun 20 20:24:06 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 20 Jun 2008 11:24:06 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <db6b5ecc0806201113y7a86adeds7f43d97f90c57c25@mail.gmail.com>
References: <20080619144328.GC32587@phare.normalesup.org>	<485A92F9.8040203@behnel.de>	<20080619183809.GB10875@phare.normalesup.org>	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>	<20080619184426.GC10875@phare.normalesup.org>	<485B3E5A.8030605@behnel.de>	<20080620122525.GC3651@phare.normalesup.org>	<e7ba66e40806201001v60dbf67cxce6b8d085d0bc7ba@mail.gmail.com>	<485BF242.80408@student.matnat.uio.no>
	<db6b5ecc0806201113y7a86adeds7f43d97f90c57c25@mail.gmail.com>
Message-ID: <485BF5C6.6060304@student.matnat.uio.no>

Fernando Perez wrote:
> On Fri, Jun 20, 2008 at 11:09 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>   
>> Lisandro Dalcin wrote:
>>     
>>> Gael, not sure if this will help you, but I wrote this script based in
>>> distutils for building *.pyx files on the fly (perhaps with others
>>> companion C/C++ sources). You basically use it as this:
>>>
>>>       
>> Also remember that Pex already has this feature (as a wrapper/build
>> system for Cython):
>>
>> pexlang.sourceforge.net
>>     
>
> This is just a generic request: when the dust settles a bit and you
> guys have a clearer picture of where the whole pex/cython integration
> regarding numpy is going to go, could you provide a little summary?
> I'm super interested in this (and I can guarantee I'm not the only
> one) but right now things seem to be in a strong state of flux, so I'm
> OK waiting.
>   
I think we already have a good idea.

1) I'm doing a Google Summer of Code over the summer for integrating 
NumPy. The approach we've eventually, after long discussions, is here:

http://wiki.cython.org/enhancements/buffer

2) Dan (of Pex) was here in Seattle with me and Robert over the weekend 
and talked about integration of Pex with Cython. A summary of the pex 
stuff is here:

http://wiki.cython.org/enhancements/pex

(Although parts of it may make more sense to us who were present -- 
please ask if you would like anything fleshed out in more detail)
> Furthermore, it would be great if you think some of  this will be in
> reasonable state (even alpha, but solid enough for knowing where it
> will  go) by August, to present it at scipy 2008:
>
> http://conference.scipy.org/
>   
Unless my summer of code fails totally, this will definitely be the 
case, and I'm sure Robert will be happy to touch on the subject.

Dag Sverre

From rho at fisheggs.name  Fri Jun 20 23:43:48 2008
From: rho at fisheggs.name (Nigel Rowe)
Date: Sat, 21 Jun 2008 07:43:48 +1000
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <20080619144328.GC32587@phare.normalesup.org>
References: <20080619144328.GC32587@phare.normalesup.org>
Message-ID: <200806210743.49981@fisheggs+neverbox.com>

On Thu, 19 Jun 2008 16:43:28 +0200, 
Gael Varoquaux <gael.varoquaux at normalesup.org> wrote in a message 
with the id <20080619144328.GC32587 at phare.normalesup.org>:
<snip/>
> It would be great if the compile could be done at import time. For
> instance the first run of cython for primes.pyx could create a
> prime.py containing the logics to do the compilation to _primes.so,
> and end by cleaning up the namespace and importing everything from
> _primes.so.
>
> What do you think?
>
> Cheers,
>
> Ga?l

Just to go back to the original request, have you looked at
<http://www.prescod.net/pyximport/> ?

I've never used it, but it seems to nearly match what you asked for.



-- 
    Nigel Rowe
    rho \N{COMMERCIAL AT} fisheggs \N{FULL STOP} name

From fperez.net at gmail.com  Sat Jun 21 01:11:05 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 20 Jun 2008 16:11:05 -0700
Subject: [Cython] Make the Cython workflow easier
In-Reply-To: <485BF5C6.6060304@student.matnat.uio.no>
References: <20080619144328.GC32587@phare.normalesup.org>
	<20080619183809.GB10875@phare.normalesup.org>
	<96144A15-25C9-4091-AB9C-6A575746AF31@math.washington.edu>
	<20080619184426.GC10875@phare.normalesup.org>
	<485B3E5A.8030605@behnel.de>
	<20080620122525.GC3651@phare.normalesup.org>
	<e7ba66e40806201001v60dbf67cxce6b8d085d0bc7ba@mail.gmail.com>
	<485BF242.80408@student.matnat.uio.no>
	<db6b5ecc0806201113y7a86adeds7f43d97f90c57c25@mail.gmail.com>
	<485BF5C6.6060304@student.matnat.uio.no>
Message-ID: <db6b5ecc0806201611m41afd74by9ba3016766a90b14@mail.gmail.com>

On Fri, Jun 20, 2008 at 11:24 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:

> I think we already have a good idea.
>
> 1) I'm doing a Google Summer of Code over the summer for integrating
> NumPy. The approach we've eventually, after long discussions, is here:
>
> http://wiki.cython.org/enhancements/buffer

[...]

Great, thanks for the info.  I very much look forward to the outcome
of this, it's great to see you guys making progress on it.

Cheers,

f

From greg.ewing at canterbury.ac.nz  Sat Jun 21 02:14:55 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 21 Jun 2008 12:14:55 +1200
Subject: [Cython] an issue with generated C API's
In-Reply-To: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
References: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
Message-ID: <485C47FF.30308@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Could the generated code be changed to say like this ??:
> 
> typedef struct PyFooObject {
>   PyObject_HEAD
>   int bar;
> } PyFooObject;

You should be able to get that already by using 'ctypedef'
instead of 'cdef', i.e.

ctypedef public api class Foo [type PyFoo_Type, object PyFooObject]:
   ...

-- 
Greg

From whycode at gmail.com  Sat Jun 21 05:36:40 2008
From: whycode at gmail.com (rahul garg)
Date: Fri, 20 Jun 2008 21:36:40 -0600
Subject: [Cython] Parallel loop annotations proposal
Message-ID: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>

Hi.

I am working on a proposal to add some kind of a mechanism for a Python
programmer to specify that a loop should be parallel. Think OpenMP style
parallel for loops. For people not familiar with OpenMP, it provides a very
simple parallel programming constructs which are specified as "pragma"
directives. A compiler is free to skip them if it doesnt implement OpenMP.
In OpenMP, you can specify some loops as parallel loops and the compiler and
runtime take care of spawning threads etc. Compiler automatically spawns
threads at the start of a parallel loop and joins at the end.

The objective is to add such optional annotations which the Python
interpreter will ignore but which can (but is not mandatorily) be used by a
compiler such as Cython or my compiler unPython. The annotations will be
added to only certain predefined type of loops since parallel iteration over
generic objects is not possible.  I am writing a detailed proposal document
and will post for your consideration in 1-2 days. However it would be great
if someone has some ideas about this already or if you can point to similar
work in Python or other languages. I know a "parfor" has been added to
Matlab. I do not know if Cython has such a facility.

My original suggestion :

"pragma parallel for"
for i in xrange(n):x[i] = y[i]

On numpy-discussion list, two better alternatives were suggested.

#pragma:parallel for
for i in xrange(n):x[i] = y[i]

with parallel(i):
   for i in xrange(n):x[i] = y[i]

I was told that IPython already does something similar to "with parallel(i)"
and I am looking into it.
I think we can also do something like :

for i in prange(i): x[i] = y[i]

prange can simply default to xrange when running on the interpreter but the
compiler can think of prange as a special object.
I selfishly want to avoid annotations specified in comments since my
compiler cannot see comments :)

Semantics needs to be defined properly. For example, exceptions cannot be
raised in proper order in parallel loops.
Any ideas/suggestions/flames? OpenMP has many more features such as
reduction variables. OpenMP is a very simple parallel programming model and
introducing something similar in Python can be really helpful.

For an implementation perspectives, it will be a little challenging to
generate code while avoiding the GIL but it can be done in simple cases.

thanks,
rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080620/02b47df5/attachment.htm 

From ondrej at certik.cz  Sat Jun 21 11:44:10 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Sat, 21 Jun 2008 11:44:10 +0200
Subject: [Cython] petsc4py with cython
Message-ID: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>

Hi Lisandro,

where is your rewrite of petsc4py with cython? I'd like to give it a try.

Could you please create a wiki at:

http://code.google.com/p/petsc4py/

with instructions how to get it? I think it'd be useful to many people.

Thanks,
Ondrej

From stefan_ml at behnel.de  Sat Jun 21 12:24:23 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jun 2008 12:24:23 +0200
Subject: [Cython] S5 theme for Cython?
Message-ID: <485CD6D7.3020601@behnel.de>

Hi,

since I'm giving a talk on Cython at EuroPython 2008, does anyone have a
ready-made cythonic S5 theme lying around that I could borrow from?

Stefan

From stefan_ml at behnel.de  Sat Jun 21 19:24:11 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jun 2008 19:24:11 +0200
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
Message-ID: <485D393B.7070300@behnel.de>

Hi,

rahul garg wrote:
> I am working on a proposal to add some kind of a mechanism for a Python
> programmer to specify that a loop should be parallel.

Sounds like a nice feature to me, although the GIL may make this more
expensive than you might think.


> The annotations will be
> added to only certain predefined type of loops since parallel iteration over
> generic objects is not possible.

It's not necessarily impossible. Take one thread that iterates over the
iterable and hands each item to a somewhat long-running worker thread.


> I do not know if Cython has such a facility.

No.


> My original suggestion :
> 
> "pragma parallel for"
> for i in xrange(n):x[i] = y[i]

I like the fact that it uses a string instead of a comment. However, I would
prefer having a real syntax in Cython.


> with parallel(i):
>    for i in xrange(n):x[i] = y[i]

Since we already have "with (no)gil", this would match nicely. Still, a syntax
inside the loop header would be preferable to keep the link with the actual
thing that runs in parallel, which is just the loop and nothing else inside
the "with" block.


> I think we can also do something like :
> 
> for i in prange(i): x[i] = y[i]
> 
> prange can simply default to xrange when running on the interpreter but the
> compiler can think of prange as a special object.

But that would mean you'd have to change the code or at least provide the name
"prange" to make it run in Python, in which case there is less gain in keeping
up a Python-compatible syntax.


> Semantics needs to be defined properly. For example, exceptions cannot be
> raised in proper order in parallel loops.

Order doesn't matter in non-synchronised parallel execution, but I would
expect them to be thread-local anyway. If you want to catch an exception
raised in a parallel loop, I'm fine with doing so inside the loop and loosing
the exception when the loop ends.


> Any ideas/suggestions/flames? OpenMP has many more features such as
> reduction variables. OpenMP is a very simple parallel programming model and
> introducing something similar in Python can be really helpful.
> 
> For an implementation perspectives, it will be a little challenging to
> generate code while avoiding the GIL but it can be done in simple cases.

It may even be enough to require "nogil" for parallel execution in the
beginning. I expect the GIL overhead to be too high for simple loops and I
don't think you'd write bigger things as a loop. When I use threads in Python,
I'm usually quite considerate about the minimum amount of work that gets
parallelised.

Stefan


From gfurnish at gfurnish.net  Sat Jun 21 19:44:47 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sat, 21 Jun 2008 10:44:47 -0700
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <485D393B.7070300@behnel.de>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
Message-ID: <8f8f8530806211044h3e221ff9p29c421353f9ee2ac@mail.gmail.com>

How about using pyprocessing to bypass the gil for autoparalleization
to create an openmp like feature?  It was accepted into python 2.7, so
it will soon be a standard feature.

On Sat, Jun 21, 2008 at 10:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> rahul garg wrote:
>> I am working on a proposal to add some kind of a mechanism for a Python
>> programmer to specify that a loop should be parallel.
>
> Sounds like a nice feature to me, although the GIL may make this more
> expensive than you might think.
>
>
>> The annotations will be
>> added to only certain predefined type of loops since parallel iteration over
>> generic objects is not possible.
>
> It's not necessarily impossible. Take one thread that iterates over the
> iterable and hands each item to a somewhat long-running worker thread.
>
>
>> I do not know if Cython has such a facility.
>
> No.
>
>
>> My original suggestion :
>>
>> "pragma parallel for"
>> for i in xrange(n):x[i] = y[i]
>
> I like the fact that it uses a string instead of a comment. However, I would
> prefer having a real syntax in Cython.
>
>
>> with parallel(i):
>>    for i in xrange(n):x[i] = y[i]
>
> Since we already have "with (no)gil", this would match nicely. Still, a syntax
> inside the loop header would be preferable to keep the link with the actual
> thing that runs in parallel, which is just the loop and nothing else inside
> the "with" block.
>
>
>> I think we can also do something like :
>>
>> for i in prange(i): x[i] = y[i]
>>
>> prange can simply default to xrange when running on the interpreter but the
>> compiler can think of prange as a special object.
>
> But that would mean you'd have to change the code or at least provide the name
> "prange" to make it run in Python, in which case there is less gain in keeping
> up a Python-compatible syntax.
>
>
>> Semantics needs to be defined properly. For example, exceptions cannot be
>> raised in proper order in parallel loops.
>
> Order doesn't matter in non-synchronised parallel execution, but I would
> expect them to be thread-local anyway. If you want to catch an exception
> raised in a parallel loop, I'm fine with doing so inside the loop and loosing
> the exception when the loop ends.
>
>
>> Any ideas/suggestions/flames? OpenMP has many more features such as
>> reduction variables. OpenMP is a very simple parallel programming model and
>> introducing something similar in Python can be really helpful.
>>
>> For an implementation perspectives, it will be a little challenging to
>> generate code while avoiding the GIL but it can be done in simple cases.
>
> It may even be enough to require "nogil" for parallel execution in the
> beginning. I expect the GIL overhead to be too high for simple loops and I
> don't think you'd write bigger things as a loop. When I use threads in Python,
> I'm usually quite considerate about the minimum amount of work that gets
> parallelised.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From whycode at gmail.com  Sat Jun 21 19:50:10 2008
From: whycode at gmail.com (rahul garg)
Date: Sat, 21 Jun 2008 11:50:10 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <485D393B.7070300@behnel.de>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
Message-ID: <d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>

> > The annotations will be
> > added to only certain predefined type of loops since parallel iteration
> over
> > generic objects is not possible.
>
> It's not necessarily impossible. Take one thread that iterates over the
> iterable and hands each item to a somewhat long-running worker thread.


Aah ok. Thats basically like a producer and multiple consumer kind of
scenario?  Didnt think of that.


>
> > I do not know if Cython has such a facility.
>
> No.
>
>
> > My original suggestion :
> >
> > "pragma parallel for"
> > for i in xrange(n):x[i] = y[i]
>
> I like the fact that it uses a string instead of a comment. However, I
> would
> prefer having a real syntax in Cython.
>
>
> > with parallel(i):
> >    for i in xrange(n):x[i] = y[i]
>
> Since we already have "with (no)gil", this would match nicely. Still, a
> syntax
> inside the loop header would be preferable to keep the link with the actual
> thing that runs in parallel, which is just the loop and nothing else inside
> the "with" block.
>
>
> > I think we can also do something like :
> >
> > for i in prange(i): x[i] = y[i]
> >
> > prange can simply default to xrange when running on the interpreter but
> the
> > compiler can think of prange as a special object.
>
> But that would mean you'd have to change the code or at least provide the
> name
> "prange" to make it run in Python, in which case there is less gain in
> keeping
> up a Python-compatible syntax.
>

I was thinking of providing a "prange" which defaults to xrange when running
on interpreter.
The reason I like the prange construct, is that we can easily add lets say
thread-local variables or reduction variables.
for i in prange(i, threadlocal=[myvar1,myvar2],reduction=[red1,red2]): #loop
body

Even "with parallel" or "with nogil", we need to define "parallel" or nogil
in some module so that it runs on python. Hmm, I guess we can also provide
keyword arguments with parallel() or nogil()?




>
>
> > Semantics needs to be defined properly. For example, exceptions cannot be
> > raised in proper order in parallel loops.
>
> Order doesn't matter in non-synchronised parallel execution, but I would
> expect them to be thread-local anyway. If you want to catch an exception
> raised in a parallel loop, I'm fine with doing so inside the loop and
> loosing
> the exception when the loop ends.


Ok. I guess thats reasonable.



>
> > Any ideas/suggestions/flames? OpenMP has many more features such as
> > reduction variables. OpenMP is a very simple parallel programming model
> and
> > introducing something similar in Python can be really helpful.
> >
> > For an implementation perspectives, it will be a little challenging to
> > generate code while avoiding the GIL but it can be done in simple cases.
>
> It may even be enough to require "nogil" for parallel execution in the
> beginning. I expect the GIL overhead to be too high for simple loops and I
> don't think you'd write bigger things as a loop. When I use threads in
> Python,
> I'm usually quite considerate about the minimum amount of work that gets
> parallelised.
>

Can you clarify this? Or give an example?

I think I will host this as a wiki page somewhere so that people can easily
edit it.

thanks,
rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080621/4cdf75d9/attachment.htm 

From whycode at gmail.com  Sat Jun 21 19:50:59 2008
From: whycode at gmail.com (rahul garg)
Date: Sat, 21 Jun 2008 11:50:59 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <8f8f8530806211044h3e221ff9p29c421353f9ee2ac@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
	<8f8f8530806211044h3e221ff9p29c421353f9ee2ac@mail.gmail.com>
Message-ID: <d75825660806211050j7baacefbla2ecfe3c42050703@mail.gmail.com>

On Sat, Jun 21, 2008 at 11:44 AM, Gary Furnish <gfurnish at gfurnish.net>
wrote:

> How about using pyprocessing to bypass the gil for autoparalleization
> to create an openmp like feature?  It was accepted into python 2.7, so
> it will soon be a standard feature.
>


Oh. Didnt know about that either. I should look into it.

thanks,
rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080621/0b91915d/attachment.htm 

From stefan_ml at behnel.de  Sat Jun 21 20:02:03 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jun 2008 20:02:03 +0200
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <485D393B.7070300@behnel.de>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
Message-ID: <485D421B.6010707@behnel.de>

Hi again,

Stefan Behnel wrote:
>> with parallel(i):
>>    for i in xrange(n):x[i] = y[i]
> 
> Since we already have "with (no)gil", this would match nicely. Still, a syntax
> inside the loop header would be preferable to keep the link with the actual
> thing that runs in parallel, which is just the loop and nothing else inside
> the "with" block.

What about making it a completely different loop, BTW? I think the most
important use case is spawning threads for each item of a small data set, so a
full-fledged, potentially long-running for loop isn't required anyway. That
would allow a syntax like:

    with thread_each(iterable):
        # loop body

or

    with thread_range(1,5):
        # loop body

I mean, this isn't really about a sequential loop, it's more about doing
something with some things.

Stefan

From stefan_ml at behnel.de  Sat Jun 21 20:06:06 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jun 2008 20:06:06 +0200
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>	<485D393B.7070300@behnel.de>
	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
Message-ID: <485D430E.2020500@behnel.de>

Hi,

rahul garg wrote:
> I think I will host this as a wiki page somewhere so that people can easily
> edit it.

http://wiki.cython.org/enhancements/parallel

Stefan


From stefan_ml at behnel.de  Sat Jun 21 20:28:34 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 21 Jun 2008 20:28:34 +0200
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>	<485D393B.7070300@behnel.de>
	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
Message-ID: <485D4852.10603@behnel.de>

Hi,

rahul garg wrote:
> I was thinking of providing a "prange" which defaults to xrange when running
> on interpreter.
> The reason I like the prange construct, is that we can easily add lets say
> thread-local variables or reduction variables.
> for i in prange(i, threadlocal=[myvar1,myvar2],reduction=[red1,red2]): #loop
> body

You could easily do that with a

    with thread_each(iterable, threadlocal=...):
        ...

syntax, too, and IMHO it looks much better (minus a better name for
"thread_each" ;)

And it might even be possible to support this in plain Python one day.


>>> For an implementation perspectives, it will be a little challenging to
>>> generate code while avoiding the GIL but it can be done in simple cases.
>>
>> It may even be enough to require "nogil" for parallel execution in the
>> beginning. I expect the GIL overhead to be too high for simple loops and I
>> don't think you'd write bigger things as a loop. When I use threads in
>> Python,
>> I'm usually quite considerate about the minimum amount of work that gets
>> parallelised.
>>
> 
> Can you clarify this? Or give an example?

I was just thinking of a couple of places in lxml. When we do a whole parser
run in C code, it's reasonable to free the GIL. However, when we do loads of
callbacks into Python SAX code, the overhead of releasing and acquiring the
GIL is just so tremendously huge compared to the short GIL-free parser steps
that it becomes much slower regardless of the number of threads.

So if you map the whole thing to threads running in a single interpreter and
do a lot of Python interaction in each thread, you will likely loose a lot of
performance. It's only really worth it if you do not need the GIL at all and
can release it before starting up the parallel code.

Gary's comment on the new multiprocessing module in Py3 shows a nice
alternative, but starting a new process isn't really for free either (even if
a fork is pretty fast on most platforms except those from a well-known
commercial OS vendor).

I think there are basically two use cases for such a feature:

1) syntactic sugar for starting up threads or processes and letting them do
equal stuff (possibly calling a larger function in the parallel block)

2) somewhat short Python-free code snippets that can run in parallel without
the GIL.

Both should be handled differently by the compiler, possible distinguished
automatically depending on the GIL being held or not. So I would say that 1)
comprises the single provider, multiple consumers case that I mentioned earlier.

Stefan

From whycode at gmail.com  Sat Jun 21 20:36:11 2008
From: whycode at gmail.com (rahul garg)
Date: Sat, 21 Jun 2008 12:36:11 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <485D4852.10603@behnel.de>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
	<485D4852.10603@behnel.de>
Message-ID: <d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>

On Sat, Jun 21, 2008 at 12:28 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Hi,
>
> rahul garg wrote:
> > I was thinking of providing a "prange" which defaults to xrange when
> running
> > on interpreter.
> > The reason I like the prange construct, is that we can easily add lets
> say
> > thread-local variables or reduction variables.
> > for i in prange(i, threadlocal=[myvar1,myvar2],reduction=[red1,red2]):
> #loop
> > body
>
> You could easily do that with a
>
>    with thread_each(iterable, threadlocal=...):
>        ...
>
> syntax, too, and IMHO it looks much better (minus a better name for
> "thread_each" ;)
>
> And it might even be possible to support this in plain Python one day.


Well the problem is that how does it run as a loop on the interpreter?
If its to run as a loop on the interpreter, there must be a loop statement
somewhere potentially.
What about adopting it to :
for i in thread_each(iterator,threadlocal=...):

This is uglier than with thread_each() but has a simple serial
implementation.
Thoughts?

thanks,
rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080621/72e20faa/attachment.htm 

From whycode at gmail.com  Sat Jun 21 20:50:44 2008
From: whycode at gmail.com (rahul garg)
Date: Sat, 21 Jun 2008 12:50:44 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
	<485D4852.10603@behnel.de>
	<d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>
Message-ID: <d75825660806211150v2dc552fexa98f415608da2850@mail.gmail.com>

Wiki page changed.
http://wiki.cython.org/enhancements/parallel

On Sat, Jun 21, 2008 at 12:36 PM, rahul garg <whycode at gmail.com> wrote:

>
>
> On Sat, Jun 21, 2008 at 12:28 PM, Stefan Behnel <stefan_ml at behnel.de>
> wrote:
>
>> Hi,
>>
>> rahul garg wrote:
>> > I was thinking of providing a "prange" which defaults to xrange when
>> running
>> > on interpreter.
>> > The reason I like the prange construct, is that we can easily add lets
>> say
>> > thread-local variables or reduction variables.
>> > for i in prange(i, threadlocal=[myvar1,myvar2],reduction=[red1,red2]):
>> #loop
>> > body
>>
>> You could easily do that with a
>>
>>    with thread_each(iterable, threadlocal=...):
>>        ...
>>
>> syntax, too, and IMHO it looks much better (minus a better name for
>> "thread_each" ;)
>>
>> And it might even be possible to support this in plain Python one day.
>
>
> Well the problem is that how does it run as a loop on the interpreter?
> If its to run as a loop on the interpreter, there must be a loop statement
> somewhere potentially.
> What about adopting it to :
> for i in thread_each(iterator,threadlocal=...):
>
> This is uglier than with thread_each() but has a simple serial
> implementation.
> Thoughts?
>
> thanks,
> rahul
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080621/d3fdc268/attachment-0001.htm 

From dagss at student.matnat.uio.no  Sat Jun 21 20:51:19 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 21 Jun 2008 20:51:19 +0200 (CEST)
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
	<485D4852.10603@behnel.de>
	<d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>
Message-ID: <2193.65.102.174.35.1214074279.squirrel@webmail.uio.no>

rahul garg wrote:
> On Sat, Jun 21, 2008 at 12:28 PM, Stefan Behnel <stefan_ml at behnel.de>
> wrote:
>
>> Hi,
>>
>> rahul garg wrote:
>> > I was thinking of providing a "prange" which defaults to xrange when
>> running
>> > on interpreter.
>> > The reason I like the prange construct, is that we can easily add lets
>> say
>> > thread-local variables or reduction variables.
>> > for i in prange(i, threadlocal=[myvar1,myvar2],reduction=[red1,red2]):
>> #loop
>> > body
>>
>> You could easily do that with a
>>
>>    with thread_each(iterable, threadlocal=...):
>>        ...
>>
>> syntax, too, and IMHO it looks much better (minus a better name for
>> "thread_each" ;)
>>
>> And it might even be possible to support this in plain Python one day.
>
>
> Well the problem is that how does it run as a loop on the interpreter?
> If its to run as a loop on the interpreter, there must be a loop statement
> somewhere potentially.
> What about adopting it to :
> for i in thread_each(iterator,threadlocal=...):
>
> This is uglier than with thread_each() but has a simple serial
> implementation.
> Thoughts?

Jumping into this thread at a random spot...

In dev1 days for SAGE a @parallel decorator for functions was
demonstrated. So:

P> @parallel
P> def f(x): return 2*x
...
P> f([1,2,3])
[2, 4, 6]

Where each of the multiplications would happen in parallel using
PyProcessing. However, I don't think any Cython support is really needed
for this (except for decorator support :-)).

So: I like PyProcessing but there's no need to build anything into Cython
for that. A parallel decorator could work for OpenMP though (at least if
the function is also declared inline etc.)

Dag Sverre


From gfurnish at gfurnish.net  Sat Jun 21 21:02:49 2008
From: gfurnish at gfurnish.net (Gary Furnish)
Date: Sat, 21 Jun 2008 12:02:49 -0700
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <2193.65.102.174.35.1214074279.squirrel@webmail.uio.no>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
	<485D4852.10603@behnel.de>
	<d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>
	<2193.65.102.174.35.1214074279.squirrel@webmail.uio.no>
Message-ID: <8f8f8530806211202x100cdbfub6b8b6ce833c1e27@mail.gmail.com>

I think I agree... cython support for decorators would probably be the
best approach.

On Sat, Jun 21, 2008 at 11:51 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> rahul garg wrote:
>> On Sat, Jun 21, 2008 at 12:28 PM, Stefan Behnel <stefan_ml at behnel.de>
>> wrote:
>>
>>> Hi,
>>>
>>> rahul garg wrote:
>>> > I was thinking of providing a "prange" which defaults to xrange when
>>> running
>>> > on interpreter.
>>> > The reason I like the prange construct, is that we can easily add lets
>>> say
>>> > thread-local variables or reduction variables.
>>> > for i in prange(i, threadlocal=[myvar1,myvar2],reduction=[red1,red2]):
>>> #loop
>>> > body
>>>
>>> You could easily do that with a
>>>
>>>    with thread_each(iterable, threadlocal=...):
>>>        ...
>>>
>>> syntax, too, and IMHO it looks much better (minus a better name for
>>> "thread_each" ;)
>>>
>>> And it might even be possible to support this in plain Python one day.
>>
>>
>> Well the problem is that how does it run as a loop on the interpreter?
>> If its to run as a loop on the interpreter, there must be a loop statement
>> somewhere potentially.
>> What about adopting it to :
>> for i in thread_each(iterator,threadlocal=...):
>>
>> This is uglier than with thread_each() but has a simple serial
>> implementation.
>> Thoughts?
>
> Jumping into this thread at a random spot...
>
> In dev1 days for SAGE a @parallel decorator for functions was
> demonstrated. So:
>
> P> @parallel
> P> def f(x): return 2*x
> ...
> P> f([1,2,3])
> [2, 4, 6]
>
> Where each of the multiplications would happen in parallel using
> PyProcessing. However, I don't think any Cython support is really needed
> for this (except for decorator support :-)).
>
> So: I like PyProcessing but there's no need to build anything into Cython
> for that. A parallel decorator could work for OpenMP though (at least if
> the function is also declared inline etc.)
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From whycode at gmail.com  Sat Jun 21 21:10:31 2008
From: whycode at gmail.com (rahul garg)
Date: Sat, 21 Jun 2008 13:10:31 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <8f8f8530806211202x100cdbfub6b8b6ce833c1e27@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<485D393B.7070300@behnel.de>
	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>
	<485D4852.10603@behnel.de>
	<d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>
	<2193.65.102.174.35.1214074279.squirrel@webmail.uio.no>
	<8f8f8530806211202x100cdbfub6b8b6ce833c1e27@mail.gmail.com>
Message-ID: <d75825660806211210u4254552cxfd72321176035015@mail.gmail.com>

Well actually one usage scenario I had in mind was numpy and numerical code.

Think about the following code :
for i in xrange(m): x[i] = y[i] + z[i]

Now in this case its a lot better to have parallel loops. First, if x,y,z
have proper type declarations, lets say they are ndarray, then a parallel
loop can easily be converted to C code with OpenMP.  Since we are accessing
ndarray, and assuming compiler knows how to convert that to direct array
access, then compiler can generate very efficient code. There will be no GIL
acquire/release within the loop.

This scenario is not handled easily using the decorator approach.

thanks,
rahul

On Sat, Jun 21, 2008 at 1:02 PM, Gary Furnish <gfurnish at gfurnish.net> wrote:

> I think I agree... cython support for decorators would probably be the
> best approach.
>
> On Sat, Jun 21, 2008 at 11:51 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
> > rahul garg wrote:
> >> On Sat, Jun 21, 2008 at 12:28 PM, Stefan Behnel <stefan_ml at behnel.de>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> rahul garg wrote:
> >>> > I was thinking of providing a "prange" which defaults to xrange when
> >>> running
> >>> > on interpreter.
> >>> > The reason I like the prange construct, is that we can easily add
> lets
> >>> say
> >>> > thread-local variables or reduction variables.
> >>> > for i in prange(i,
> threadlocal=[myvar1,myvar2],reduction=[red1,red2]):
> >>> #loop
> >>> > body
> >>>
> >>> You could easily do that with a
> >>>
> >>>    with thread_each(iterable, threadlocal=...):
> >>>        ...
> >>>
> >>> syntax, too, and IMHO it looks much better (minus a better name for
> >>> "thread_each" ;)
> >>>
> >>> And it might even be possible to support this in plain Python one day.
> >>
> >>
> >> Well the problem is that how does it run as a loop on the interpreter?
> >> If its to run as a loop on the interpreter, there must be a loop
> statement
> >> somewhere potentially.
> >> What about adopting it to :
> >> for i in thread_each(iterator,threadlocal=...):
> >>
> >> This is uglier than with thread_each() but has a simple serial
> >> implementation.
> >> Thoughts?
> >
> > Jumping into this thread at a random spot...
> >
> > In dev1 days for SAGE a @parallel decorator for functions was
> > demonstrated. So:
> >
> > P> @parallel
> > P> def f(x): return 2*x
> > ...
> > P> f([1,2,3])
> > [2, 4, 6]
> >
> > Where each of the multiplications would happen in parallel using
> > PyProcessing. However, I don't think any Cython support is really needed
> > for this (except for decorator support :-)).
> >
> > So: I like PyProcessing but there's no need to build anything into Cython
> > for that. A parallel decorator could work for OpenMP though (at least if
> > the function is also declared inline etc.)
> >
> > Dag Sverre
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080621/e7c8bf3a/attachment.htm 

From stefan_ml at behnel.de  Sun Jun 22 07:23:37 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 22 Jun 2008 07:23:37 +0200
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806211210u4254552cxfd72321176035015@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>	<485D393B.7070300@behnel.de>	<d75825660806211050o51166da5gf6b66a36226d00eb@mail.gmail.com>	<485D4852.10603@behnel.de>	<d75825660806211136r2392f25dm287fcb6c464e4bff@mail.gmail.com>	<2193.65.102.174.35.1214074279.squirrel@webmail.uio.no>	<8f8f8530806211202x100cdbfub6b8b6ce833c1e27@mail.gmail.com>
	<d75825660806211210u4254552cxfd72321176035015@mail.gmail.com>
Message-ID: <485DE1D9.1000208@behnel.de>

Hi,

rahul garg top-posted (just like Gary Furnish did before):
>> On Sat, Jun 21, 2008 at 11:51 AM, Dag Sverre Seljebotn wrote
>>> In dev1 days for SAGE a @parallel decorator for functions was
>>> demonstrated. So:
>>>
>>> P> @parallel
>>> P> def f(x): return 2*x
>>> ...
>>> P> f([1,2,3])
>>> [2, 4, 6]
>>>
>>> Where each of the multiplications would happen in parallel using
>>> PyProcessing. However, I don't think any Cython support is really needed
>>> for this (except for decorator support :-)).
>>>
>>> So: I like PyProcessing but there's no need to build anything into Cython
>>> for that. A parallel decorator could work for OpenMP though (at least if
>>> the function is also declared inline etc.)
>
> Well actually one usage scenario I had in mind was numpy and numerical code.
>
> Think about the following code :
> for i in xrange(m): x[i] = y[i] + z[i]
>
> Now in this case its a lot better to have parallel loops. First, if x,y,z
> have proper type declarations, lets say they are ndarray, then a parallel
> loop can easily be converted to C code with OpenMP.  Since we are accessing
> ndarray, and assuming compiler knows how to convert that to direct array
> access, then compiler can generate very efficient code. There will be no GIL
> acquire/release within the loop.
>
> This scenario is not handled easily using the decorator approach.

I updated the Wiki accordingly.

Stefan


From gael.varoquaux at normalesup.org  Mon Jun 23 02:37:30 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Jun 2008 02:37:30 +0200
Subject: [Cython] Surprising performance tweak in Cython
Message-ID: <20080623003730.GF15908@phare.normalesup.org>

I tried tweak my Cython code for performance by manually inlining a small
function, and ended up with a less performant code. I must confess I
don't really understand what is going on here. If somebody has an
explaination, I'd be delighted. The code follows.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
from numpy import zeros

# Make sure numpy is initialized.
include "c_numpy.pxd"

##############################################################################
cdef int inner_loop(float c_x, float c_y):
    cdef float x, y, x_buffer
    x = 0; y = 0
    cdef int i
    for i in range(50):
        x_buffer = x*x - y*y + c_x
        y = 2*x*y + c_y
        x = x_buffer
        if (x*x + x*y > 100):
            return 50 - i

def do_Mandelbrot_cython():
    cdef ndarray threshold_time 
    threshold_time = zeros((500, 500))
    cdef double *tp
    cdef float c_x, c_y
    cdef int i, j
    c_x = -1.5
    tp = <double*>threshold_time.data
    for i in range(500):
        c_y = -1
        for j in range(500):
            tp += 1
            c_y += 0.004
            tp[0] = inner_loop(c_x, c_y)
        c_x += 0.004
    return threshold_time


def do_Mandelbrot_cython2():
    cdef ndarray threshold_time
    threshold_time = zeros((500, 500))
    cdef double *tp
    tp = <double*>threshold_time.data
    cdef float x, y, xbuffer, c_x, c_y
    cdef int i, j, n 
    c_x = -1.5
    for i in range(500):
        c_y = -1
        for j in range(500):
            tp += 1
            c_y += 0.004
            x = 0; y = 0
            for n in range(50):
                x_buffer = x*x - y*y + c_x
                y = 2*x*y + c_y
                x = x_buffer
                if (x*x + y*y > 100):
                    tp[0] = 50 -n
                    break
        c_x += 0.004
    return threshold_time
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

And the timing I get are:

In [2]: %timeit C.do_Mandelbrot_cython2()
10 loops, best of 3: 342 ms per loop

In [3]: %timeit C.do_Mandelbrot_cython()
10 loops, best of 3: 126 ms per loop

Cheers,

Ga?l

From greg.ewing at canterbury.ac.nz  Mon Jun 23 03:43:04 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 23 Jun 2008 13:43:04 +1200
Subject: [Cython] Surprising performance tweak in Cython
In-Reply-To: <20080623003730.GF15908@phare.normalesup.org>
References: <20080623003730.GF15908@phare.normalesup.org>
Message-ID: <485EFFA8.4080407@canterbury.ac.nz>

Gael Varoquaux wrote:
> I tried tweak my Cython code for performance by manually inlining a small
> function, and ended up with a less performant code.

Perhaps the inlining has caused the number of local
variables to exceed the number of registers available,
or something like that.

The only way to find out for sure would be to examine
the assembly language being produced by the C compiler.

Something to keep in mind is that C function calls
are not so expensive in relative terms as Python
function calls are. So whereas in Python eliminating
a user-defined function call is almost always a win,
it's not necessarily so in C.

-- 
Greg

From gael.varoquaux at normalesup.org  Mon Jun 23 06:56:57 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Mon, 23 Jun 2008 06:56:57 +0200
Subject: [Cython] Surprising performance tweak in Cython
In-Reply-To: <485EFFA8.4080407@canterbury.ac.nz>
References: <20080623003730.GF15908@phare.normalesup.org>
	<485EFFA8.4080407@canterbury.ac.nz>
Message-ID: <20080623045657.GL15908@phare.normalesup.org>

Thanks for your answer, Greg,

The problem was with a typo in a type definition. As a result, one of the
floats was being treated as a Python float. Eric Firing found this.

Ga?l

From dalcinl at gmail.com  Mon Jun 23 16:03:58 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 23 Jun 2008 11:03:58 -0300
Subject: [Cython] an issue with generated C API's
In-Reply-To: <485C47FF.30308@canterbury.ac.nz>
References: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
	<485C47FF.30308@canterbury.ac.nz>
Message-ID: <e7ba66e40806230703n19234b5ey51b1ade80dfb6c58@mail.gmail.com>

Greg, this approach does not seems to work, at least in Cython:

ctypedef public api class Status [....]:
                   ^
------------------------------------------------------------

./mpi4py/MPI.pxd:49:20: Syntax error in ctypedef statement





On 6/20/08, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>  > Could the generated code be changed to say like this ??:
>  >
>  > typedef struct PyFooObject {
>  >   PyObject_HEAD
>  >   int bar;
>  > } PyFooObject;
>
>
> You should be able to get that already by using 'ctypedef'
>  instead of 'cdef', i.e.
>
>  ctypedef public api class Foo [type PyFoo_Type, object PyFooObject]:
>    ...
>
>  --
>
> Greg
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ellisonbg.net at gmail.com  Mon Jun 23 17:40:57 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 23 Jun 2008 09:40:57 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
Message-ID: <6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>

Here is my two cents on this...

Something like OpenMP has already been tried in Python, although it
really hasn't made it out into the wild.  Here is the link:

http://www.cs.utk.edu/~luszczek/pubs/plw-200605.pdf

Using markup or directives has never really become a very popular way
of handling parallelism.  I am not sure why, but OpenMP has never
really caught on like MPI even though MPI is more painful for many
things.

I think it would be much better to use the language (Python) itself to
handle these things as the language is already extremely powerful.

But, I think the actual parallel constructs should be kept out of
Cython proper.  In general I think it is much better to develop
libraries, not new language syntax.  Obviously, Cython itself is a
distinct language, but I think considerable effort must be made to
prevent feature creep.  I am not against new features, but it is easy
to get carried away with these things.  It is much more important to
make the existing features and syntax more robust.  This is why most
successful languages develop in a rather conservative manner.  I would
suggest a good way of evaluating new ideas is to ask to question "what
would Guido say?"  Of course, Cython is not Python, but we all hope
the two don't stray far from each other.

Lots of projects have begun using Cython lately, which is really
great.  This makes it more important than ever to focus on the core
language rather than wild new features.  Plus, there are a number of
other projects that are already having great success in tackling
parallelism in Python.  If people are interested in parallelism, I
highly recommend getting involved in those existing efforts.

Cheers,

Brian


On Fri, Jun 20, 2008 at 9:36 PM, rahul garg <whycode at gmail.com> wrote:
> Hi.
>
> I am working on a proposal to add some kind of a mechanism for a Python
> programmer to specify that a loop should be parallel. Think OpenMP style
> parallel for loops. For people not familiar with OpenMP, it provides a very
> simple parallel programming constructs which are specified as "pragma"
> directives. A compiler is free to skip them if it doesnt implement OpenMP.
> In OpenMP, you can specify some loops as parallel loops and the compiler and
> runtime take care of spawning threads etc. Compiler automatically spawns
> threads at the start of a parallel loop and joins at the end.
>
> The objective is to add such optional annotations which the Python
> interpreter will ignore but which can (but is not mandatorily) be used by a
> compiler such as Cython or my compiler unPython. The annotations will be
> added to only certain predefined type of loops since parallel iteration over
> generic objects is not possible.  I am writing a detailed proposal document
> and will post for your consideration in 1-2 days. However it would be great
> if someone has some ideas about this already or if you can point to similar
> work in Python or other languages. I know a "parfor" has been added to
> Matlab. I do not know if Cython has such a facility.
>
> My original suggestion :
>
> "pragma parallel for"
> for i in xrange(n):x[i] = y[i]
>
> On numpy-discussion list, two better alternatives were suggested.
>
> #pragma:parallel for
> for i in xrange(n):x[i] = y[i]
>
> with parallel(i):
>    for i in xrange(n):x[i] = y[i]
>
> I was told that IPython already does something similar to "with parallel(i)"
> and I am looking into it.
> I think we can also do something like :
>
> for i in prange(i): x[i] = y[i]
>
> prange can simply default to xrange when running on the interpreter but the
> compiler can think of prange as a special object.
> I selfishly want to avoid annotations specified in comments since my
> compiler cannot see comments :)
>
> Semantics needs to be defined properly. For example, exceptions cannot be
> raised in proper order in parallel loops.
> Any ideas/suggestions/flames? OpenMP has many more features such as
> reduction variables. OpenMP is a very simple parallel programming model and
> introducing something similar in Python can be really helpful.
>
> For an implementation perspectives, it will be a little challenging to
> generate code while avoiding the GIL but it can be done in simple cases.
>
> thanks,
> rahul
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From simon at arrowtheory.com  Mon Jun 23 19:15:37 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Mon, 23 Jun 2008 13:15:37 -0400
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
Message-ID: <20080623131537.7ac4697e.simon@arrowtheory.com>

On Mon, 23 Jun 2008 09:40:57 -0600
"Brian Granger" <ellisonbg.net at gmail.com> wrote:

> 
> But, I think the actual parallel constructs should be kept out of
> Cython proper.  In general I think it is much better to develop
> libraries, not new language syntax.  Obviously, Cython itself is a
> distinct language, but I think considerable effort must be made to
> prevent feature creep.  I am not against new features, but it is easy
> to get carried away with these things.  It is much more important to
> make the existing features and syntax more robust.  This is why most
> successful languages develop in a rather conservative manner.  I would
> suggest a good way of evaluating new ideas is to ask to question "what
> would Guido say?"  Of course, Cython is not Python, but we all hope
> the two don't stray far from each other.

Could we make the cython language extensible/modifiable ? (Guido would definately
say no to this..)

Simon.

From whycode at gmail.com  Mon Jun 23 19:59:44 2008
From: whycode at gmail.com (rahul garg)
Date: Mon, 23 Jun 2008 11:59:44 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
Message-ID: <d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>

On Mon, Jun 23, 2008 at 9:40 AM, Brian Granger <ellisonbg.net at gmail.com>
wrote:

> Here is my two cents on this...
>
> Something like OpenMP has already been tried in Python, although it
> really hasn't made it out into the wild.  Here is the link:
>
> http://www.cs.utk.edu/~luszczek/pubs/plw-200605.pdf<http://www.cs.utk.edu/%7Eluszczek/pubs/plw-200605.pdf>


I have been in touch with the author. The implemenation was never complete.
PLW project never actually implemented full parallel annotations. They were
proposed but not implemented.


>
>
> Using markup or directives has never really become a very popular way
> of handling parallelism.  I am not sure why, but OpenMP has never
> really caught on like MPI even though MPI is more painful for many
> things.


It is becoming more widespread now. Every single system/chip/compiler vendor
I know of is pushing this. The advent of multicores is a big reason for this
since parallelism is now moving into the mainstream from HPC.  One other
reason why OpenMP hasnt caught on is that compilers have taken many years to
mature. GCC for example had no support for OpenMP till very recently. For
OpenMP, its still usually better to use vendor compilers than GCC.

In the Python world, MPI has been popular. Because, as it happens, Python
offers you NO WAY to handle parallelism any other way except a process based
way.


> I think it would be much better to use the language (Python) itself to
> handle these things as the language is already extremely powerful.


> But, I think the actual parallel constructs should be kept out of
> Cython proper.  In general I think it is much better to develop
> libraries, not new language syntax.  Obviously, Cython itself is a
> distinct language, but I think considerable effort must be made to
> prevent feature creep.  I am not against new features, but it is easy
> to get carried away with these things.  It is much more important to
> make the existing features and syntax more robust.  This is why most
> successful languages develop in a rather conservative manner.  I would
> suggest a good way of evaluating new ideas is to ask to question "what
> would Guido say?"  Of course, Cython is not Python, but we all hope
> the two don't stray far from each other.


What will Guido say is the wrong thing to ask in this case. This is not a
question of language syntax. What we are discussing is not new syntax sugar
over some existing facility. What we are discussing is a system facility.
Tell me : is Guido against removing GIL? He is not. He only says that
removing the GIL is a practical problem in that it not-so-easy to modify the
CPython source base. Is Guido against threads offering real concurrency? He
is not or he would not like Jython.
Guido only says :
a) GIL is a practical problem
b) For many (but not all) applications process based parallelism is
sufficient. Thus it does not give HIM and the rest of the cpython developers
enough incentive to put work into removing GIL when he has more pressing
issues. He is however happy if someone else does it.

Let me stress once again. WE ARE NOT DISCSSING SYNTAX. We are discussing a
system facility. And as it happens, this system facility cannot be handled
any other way except either
a) writing a compiler that can produce parallel C code
b) writing a modified CPython.
c) writing a JIT that can produce parallel code at runtime. similar to a.

As we dont have the option b or c, we have to go through the route a.  If
you simply dont care about the system facility being provided (loop based
parallelism), thats a different issue.


> Lots of projects have begun using Cython lately, which is really
> great.  This makes it more important than ever to focus on the core
> language rather than wild new features.


Since I am not a Cython developer, I will of course not comment on this. I
will however point out that I am also writing a compiler called unPython
which focuses a lot on numpy and I am 100% sure that I am going to add a
parallel loop. Even if the syntax I add turns out ugly, thats fine with me
as long as people get to write parallel code. Even a slightly ugly Python
syntax is a lot better than writing manually C code. Thus I propose that I
am going ahead with my implementation, for now based on prange. This way I
can also report on how easy or hard it is to implement. Cython developers
can have discussions about whether they want to add or not. Since unPython
is very alpha, and isnt used very much, so I have liberty of changing syntax
later if by consensus some other syntax is found to be better.

So in summary, I am going ahead right now and will come back and report in a
month or so.

thanks,
rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080623/072cf0d5/attachment.htm 

From fperez.net at gmail.com  Mon Jun 23 20:08:54 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Mon, 23 Jun 2008 11:08:54 -0700
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
	<d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>
Message-ID: <db6b5ecc0806231108o7514300ci8b1714c67a335d84@mail.gmail.com>

On Mon, Jun 23, 2008 at 10:59 AM, rahul garg <whycode at gmail.com> wrote:

> So in summary, I am going ahead right now and will come back and report in a
> month or so.

Don't go away for too long :)  What I mean is: as soon as you have
something that kind of works, even if it's limited and preliminary,
let people know about it (both here and on the numpy list).  I'm sure
there's a lot of interest on this (even if sometimes opinions on
approach may differ), and you'll get good feedback.  This is an
important enough problem, that with all the energy being put into it,
it looks like in a year or so the landscape is going to be very
significantly improved, which is great.

Good luck with your effort!

Cheers,

f

From ellisonbg.net at gmail.com  Mon Jun 23 22:53:19 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Mon, 23 Jun 2008 14:53:19 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
	<d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>
Message-ID: <6ce0ac130806231353m4ec87774jdddaa9a584ae4875@mail.gmail.com>

> I have been in touch with the author. The implemenation was never complete.
> PLW project never actually implemented full parallel annotations. They were
> proposed but not implemented.

This would explain why the project really never showed up on the radar.

>>
>> Using markup or directives has never really become a very popular way
>> of handling parallelism.  I am not sure why, but OpenMP has never
>> really caught on like MPI even though MPI is more painful for many
>> things.
>
> It is becoming more widespread now. Every single system/chip/compiler vendor
> I know of is pushing this. The advent of multicores is a big reason for this
> since parallelism is now moving into the mainstream from HPC.  One other
> reason why OpenMP hasnt caught on is that compilers have taken many years to
> mature. GCC for example had no support for OpenMP till very recently. For
> OpenMP, its still usually better to use vendor compilers than GCC.
>
> In the Python world, MPI has been popular. Because, as it happens, Python
> offers you NO WAY to handle parallelism any other way except a process based
> way.

Very true - threads aren't very useful in Python for CPU bound
parallelism.  But, I think there is another reason that MPI has become
dominant and it is exactly related to the reasons you mention about
compilers and OpenMP.  The reality is that it is much easier in
general to write a solid cross platform library (MPI) than a compiler
(OpenMP).  It is also much easier to tell someone "just use this
library" than "use this compiler".

> What will Guido say is the wrong thing to ask in this case. This is not a
> question of language syntax. What we are discussing is not new syntax sugar
> over some existing facility. What we are discussing is a system facility.

Some of the proposals do involve modifying the language grammar.
True, some of the proposals (prange) don't require this, but the
pragma directive ones do.  For these directives to work, the grammar
of the language does have to be changed - even though the grammar
changes are backwards compatible (I guess that puts them in the
"sugar" category), but they are still changes.

> Let me stress once again. WE ARE NOT DISCSSING SYNTAX. We are discussing a
> system facility. And as it happens, this system facility cannot be handled
> any other way except either
> a) writing a compiler that can produce parallel C code
> b) writing a modified CPython.
> c) writing a JIT that can produce parallel code at runtime. similar to a.
>
> As we dont have the option b or c, we have to go through the route a.  If
> you simply dont care about the system facility being provided (loop based
> parallelism), thats a different issue.

I am not sure who the "WE" is here, but some of the proposals on the
table require modifying the grammar of cython/unpython (unless I am
missing something).  I meant to comment on those proposals.

> Since I am not a Cython developer, I will of course not comment on this. I
> will however point out that I am also writing a compiler called unPython
> which focuses a lot on numpy and I am 100% sure that I am going to add a
> parallel loop. Even if the syntax I add turns out ugly, thats fine with me
> as long as people get to write parallel code. Even a slightly ugly Python
> syntax is a lot better than writing manually C code. Thus I propose that I
> am going ahead with my implementation, for now based on prange.

I think the prange proposal is a very solid one and look forward to
see what becomes of this.

> So in summary, I am going ahead right now and will come back and report in a
> month or so.

Great!

Cheers,

Brian

> thanks,
> rahul
>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From whycode at gmail.com  Mon Jun 23 23:20:33 2008
From: whycode at gmail.com (rahul garg)
Date: Mon, 23 Jun 2008 15:20:33 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <6ce0ac130806231353m4ec87774jdddaa9a584ae4875@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
	<d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>
	<6ce0ac130806231353m4ec87774jdddaa9a584ae4875@mail.gmail.com>
Message-ID: <d75825660806231420j6e41eb8er9a0ff3606c3a9875@mail.gmail.com>

On Mon, Jun 23, 2008 at 2:53 PM, Brian Granger <ellisonbg.net at gmail.com>
wrote:

> > So in summary, I am going ahead right now and will come back and report
> in a
> > month or so.
>
>
> Great!
>
> Cheers,
>


Thanks for your comments. I guess we have multiple parties involved :
multiple users with differing needs, Python language designers and CPython
developers, 3rd party tool like Cython which are widely used and the more
experimental  tools like unPython. People also have various backgrounds :
people experienced with threads and locks, experienced with OpenMP,
experienced with MPI, experienced with erlang and actors etc.

Similarly, multiple projects like pyprocessing exist which provide libraries
for process based parallelism.  You can also use MPI as pointed out as
another approach.  Bindings to existing parallel C/C++ libraries are another
way people currently utilize parallelism. For example numpy can use threaded
BLAS and people have added python bindings to many distributed solvers.
Stackless is another interesting programming model and it appears someone is
working on a patch to enable real concurrency in stackless?? (If somebody
knows about stackless, comments will be appreciated).  You can use Java
threads in Jython. (and same goes for IronPython and CLR??)  I guess
different problems will require different tools and as people gain more
experience, they will get to know which model applies for which problem.

Parallelism in Python is not an easy issue and given the various parties
involved and their needs and viewpoints, it is best to have some
experimental implementations which can be used as reference points later to
have a more informed debate.

thanks,
rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080623/4d4cfe69/attachment.htm 

From greg.ewing at canterbury.ac.nz  Tue Jun 24 04:08:06 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Jun 2008 14:08:06 +1200
Subject: [Cython] an issue with generated C API's
In-Reply-To: <e7ba66e40806230703n19234b5ey51b1ade80dfb6c58@mail.gmail.com>
References: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
	<485C47FF.30308@canterbury.ac.nz>
	<e7ba66e40806230703n19234b5ey51b1ade80dfb6c58@mail.gmail.com>
Message-ID: <48605706.1070302@canterbury.ac.nz>

Lisandro Dalcin wrote:
> Greg, this approach does not seems to work, at least in Cython:
> 
> ctypedef public api class Status [....]:

Hmmm, it seems I may never have tested this with other
modifiers between the "ctypedef" and the "class". I'll
look into it.

-- 
Greg

From dagss at student.matnat.uio.no  Tue Jun 24 08:18:11 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 24 Jun 2008 08:18:11 +0200 (CEST)
Subject: [Cython] Updates on buffer spec
Message-ID: <1320.212.123.202.66.1214288291.squirrel@webmail.uio.no>

(This is mostly to keep you posted, though if you disagree on the negative
index thing then speak up).

I posted a thread on the NumPy mailing list which got much response. I
updated the buffer spec accordingly as how I would like it myself now. In
particular:

- Anne Archibald convinced me about the need for negative index support:

http://projects.scipy.org/pipermail/numpy-discussion/2008-June/035079.html

(You can nest yourself to the rest of the thread from there if interested.
Note especially that unsigned datatypes will be used rather than pragmas
to flag positive indices)

- Nobody complained about the [float64, 2]-syntax.

Dag Sverre


From dalcinl at gmail.com  Tue Jun 24 15:44:23 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 24 Jun 2008 10:44:23 -0300
Subject: [Cython] an issue with generated C API's
In-Reply-To: <48605706.1070302@canterbury.ac.nz>
References: <e7ba66e40806180738q5189cfa2g5804086085d9d3b0@mail.gmail.com>
	<485C47FF.30308@canterbury.ac.nz>
	<e7ba66e40806230703n19234b5ey51b1ade80dfb6c58@mail.gmail.com>
	<48605706.1070302@canterbury.ac.nz>
Message-ID: <e7ba66e40806240644m6747b775ifd6018d89c39d383@mail.gmail.com>

OK, if you can make it working, then I'm +1 with this way. Keep me
posted, then perhaps I would be able to backport this to Cython.

On 6/23/08, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>  > Greg, this approach does not seems to work, at least in Cython:
>  >
>  > ctypedef public api class Status [....]:
>
>
> Hmmm, it seems I may never have tested this with other
>  modifiers between the "ctypedef" and the "class". I'll
>  look into it.
>
>
>  --
>
> Greg
>  _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Jun 24 08:46:15 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 24 Jun 2008 08:46:15 +0200
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <6ce0ac130806231353m4ec87774jdddaa9a584ae4875@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>	<d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>
	<6ce0ac130806231353m4ec87774jdddaa9a584ae4875@mail.gmail.com>
Message-ID: <48609837.3010501@behnel.de>

Hi,

Brian Granger wrote:
> I think the prange proposal is a very solid one and look forward to
> see what becomes of this.

I'm actually not so happy with it. It changes the behaviour and meaning of the
for loop (i.e. the "for" keyword) based on the thing that is iterated on.
IMHO, the thing we are discussing here is not about sequential loops at all
but about parallel execution.

Stefan


From whycode at gmail.com  Tue Jun 24 21:17:14 2008
From: whycode at gmail.com (rahul garg)
Date: Tue, 24 Jun 2008 13:17:14 -0600
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <48609837.3010501@behnel.de>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
	<d75825660806231059t687ad2b6r4b7b5c4e52434a6b@mail.gmail.com>
	<6ce0ac130806231353m4ec87774jdddaa9a584ae4875@mail.gmail.com>
	<48609837.3010501@behnel.de>
Message-ID: <d75825660806241217y7e8779ccha47f808e98dfa512@mail.gmail.com>

On Tue, Jun 24, 2008 at 12:46 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Hi,
>
> Brian Granger wrote:
> > I think the prange proposal is a very solid one and look forward to
> > see what becomes of this.
>
> I'm actually not so happy with it. It changes the behaviour and meaning of
> the
> for loop (i.e. the "for" keyword) based on the thing that is iterated on.
> IMHO, the thing we are discussing here is not about sequential loops at all
> but about parallel execution.


Well thats fine. Maybe people prefer "with parallel" or maybe string
annotations seperated from loop etc.
Thats just a couple of lines in the parser so lets step a little bit and see
what the more important issues are :

a) We need to define a memory model. By default all variables within a loop
are shared except the loop counter. So if you have a scalar variable "var",
such as an int or float, that will be shared by all threads and if all
threads want to read/write from it, no particular order is defined. Also
statements will not be atomic.

"var = var+5" for example will not execute atomically so you may not achieve
the intended effect.

b) Of course the weak memory model is not enough. Thus I think we should
also have threadprivate and reduction variables. Threadprivate variables are
cloned and each thread gets one individual copy. After the loop, the value
in the variable is undefined. This is useful for variables you use to hold
temporary values in a loop.

c) Reduction variables are useful for reduction type of operations.
Reduction variables need to be shared among threads and since memory model
is very weak, reduction operations are problematic.  To alleviate this
problem, reduction variables are introduced. When you specify reduction
variable, you also need to specify a reduction operator. Then inside the
loop the reduction variables behave as threadprivate and at the end of the
loop, the value held by each of threads is reduced into the variable
according to the reduction operator specified.

These are all mostly taken from OpenMP. OpenMP also has a lot of different
constructs. For example, you can control which thread executes which
iteration. This can have very serious performance impacts since a
round-robin mapping of iterations to threads and a "largest blocked" mapping
of iteration to threads can both be suboptimal depending on the problem.
You can control the number of threads at runtime too. Similarly you can also
ask your thread-id inside a loop.
 So the question really is : how much of OpenMP is it feasible to expose?
And this is just with OpenMP loops and not with other parts of OpenMP.

Then there are implementation issues.  We really cannot have threadprivate
objects other than C native types perhaps. In fact inside the loop, using
anything other than int,float and ndarray etc might be problematic.  Inside
the loop, you simply want to avoid calling the interpreter completely since
otherwise a mapping to OpenMP is not easy.  Also OpenMP compilers arent all
exactly the same and deviate from the standard in different ways.

Then there are also other issues. For example, what about assignment to
slices of numpy arrays? These are effectively loops so should we have some
constructs  to allow parallelizing them a-la HPF?

So as you can see, surface syntax is just one issue and the one for which
everyone will have differing opinions. Being of a compiler writer mindset,
rather than a language designer, I will just leave the surface syntax to the
community later. The devil really is in the detail and we should have a more
serious discussion about things like memory models and the speed hits
various constructs may take.

This is not trivial. Again we also need to have various usecases in mind.
And the use case for parallel loops should be numeric and scientific
computing and more specifically numpy based codes. If it benifits any other
usecase, its a bonus.

thanks,
rahul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20080624/d3c6a709/attachment.htm 

From robertwb at math.washington.edu  Wed Jun 25 06:41:35 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Jun 2008 21:41:35 -0700
Subject: [Cython] Parallel loop annotations proposal
In-Reply-To: <6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
References: <d75825660806202036h5f31bd7cq3491ca56c3850eb5@mail.gmail.com>
	<6ce0ac130806230840s7c7499e7vc192e2b2c3251009@mail.gmail.com>
Message-ID: <1DE3D9C8-A3C7-4678-AF6B-D5B1BE8ACAA0@math.washington.edu>

On Jun 23, 2008, at 8:40 AM, Brian Granger wrote:

> Here is my two cents on this...
>
> Something like OpenMP has already been tried in Python, although it
> really hasn't made it out into the wild.  Here is the link:
>
> http://www.cs.utk.edu/~luszczek/pubs/plw-200605.pdf
>
> Using markup or directives has never really become a very popular way
> of handling parallelism.  I am not sure why, but OpenMP has never
> really caught on like MPI even though MPI is more painful for many
> things.
>
> I think it would be much better to use the language (Python) itself to
> handle these things as the language is already extremely powerful.
>
> But, I think the actual parallel constructs should be kept out of
> Cython proper.  In general I think it is much better to develop
> libraries, not new language syntax.  Obviously, Cython itself is a
> distinct language, but I think considerable effort must be made to
> prevent feature creep.  I am not against new features, but it is easy
> to get carried away with these things.  It is much more important to
> make the existing features and syntax more robust.  This is why most
> successful languages develop in a rather conservative manner.  I would
> suggest a good way of evaluating new ideas is to ask to question "what
> would Guido say?"  Of course, Cython is not Python, but we all hope
> the two don't stray far from each other.
>
> Lots of projects have begun using Cython lately, which is really
> great.  This makes it more important than ever to focus on the core
> language rather than wild new features.  Plus, there are a number of
> other projects that are already having great success in tackling
> parallelism in Python.  If people are interested in parallelism, I
> highly recommend getting involved in those existing efforts.
>
> Cheers,
>
> Brian


These were *exactly* my thoughts when I started reading this thread.  
I'm not sure if the OpenMP paradigm can be supported without adding  
features to the language, which I would like to avoid.

In terms of syntax proposals, decorated inline functions (closures)  
seem the best, but I don't know how to avoid the performance hit. I  
think the for loop, as in proposal 6, is the best semantic fit.  
(Executing the body of a with statement multiple times seems a bit  
odd (it doesn't have anything to do with contexts) and would be  
difficult (if even possible?) to emulate in CPython.) Another  
(unfinished) alternative is to consider

for i in ...
for i from ...
for i xxxx ... # if we can come up with a good keyword here, the rest  
would be the same as "in."

- Robert


>


From robertwb at math.washington.edu  Wed Jun 25 07:12:24 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Jun 2008 22:12:24 -0700
Subject: [Cython] Updates on buffer spec
In-Reply-To: <1320.212.123.202.66.1214288291.squirrel@webmail.uio.no>
References: <1320.212.123.202.66.1214288291.squirrel@webmail.uio.no>
Message-ID: <251F921D-2EC6-4EF2-A2BA-F26BA3C9F6A4@math.washington.edu>

On Jun 23, 2008, at 11:18 PM, Dag Sverre Seljebotn wrote:

> (This is mostly to keep you posted, though if you disagree on the  
> negative
> index thing then speak up).
>
> I posted a thread on the NumPy mailing list which got much response. I
> updated the buffer spec accordingly as how I would like it myself  
> now. In
> particular:
>
> - Anne Archibald convinced me about the need for negative index  
> support:
>
> http://projects.scipy.org/pipermail/numpy-discussion/2008-June/ 
> 035079.html

Thanks for posting there. He's convinced me too. That unsigned int  
trick is a nice one.

> (You can nest yourself to the rest of the thread from there if  
> interested.
> Note especially that unsigned datatypes will be used rather than  
> pragmas
> to flag positive indices)
>
> - Nobody complained about the [float64, 2]-syntax.

No one has come up with anything better either, so lets run with it.

- Robert


From robertwb at math.washington.edu  Wed Jun 25 08:39:41 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Jun 2008 23:39:41 -0700
Subject: [Cython] Surprising performance tweak in Cython
In-Reply-To: <20080623045657.GL15908@phare.normalesup.org>
References: <20080623003730.GF15908@phare.normalesup.org>
	<485EFFA8.4080407@canterbury.ac.nz>
	<20080623045657.GL15908@phare.normalesup.org>
Message-ID: <84757386-B734-4A6F-8B15-3F334F068A04@math.washington.edu>

On Jun 22, 2008, at 9:56 PM, Gael Varoquaux wrote:

> Thanks for your answer, Greg,
>
> The problem was with a typo in a type definition. As a result, one  
> of the
> floats was being treated as a Python float. Eric Firing found this.

That would do it to you...running Cython with the -a (annotate)  
option will make it easy to detect this kind of thing.

On another note, cdef functions can be declared as "inline" which  
will emit the appropriate pragma to the C compiler to produce the  
same effect (with less work) than manually inlining.

- Robert

From robertwb at math.washington.edu  Wed Jun 25 08:41:31 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Jun 2008 23:41:31 -0700
Subject: [Cython] petsc4py with cython
In-Reply-To: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
References: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
Message-ID: <7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>

On Jun 21, 2008, at 2:44 AM, Ondrej Certik wrote:

> Hi Lisandro,
>
> where is your rewrite of petsc4py with cython? I'd like to give it  
> a try.
>
> Could you please create a wiki at:
>
> http://code.google.com/p/petsc4py/
>
> with instructions how to get it? I think it'd be useful to many  
> people.

Could you also add a link here?

http://wiki.cython.org/projects


From robertwb at math.washington.edu  Wed Jun 25 08:58:16 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 24 Jun 2008 23:58:16 -0700
Subject: [Cython] Python betas delayed again
In-Reply-To: <4852BAAB.7010904@behnel.de>
References: <4852BAAB.7010904@behnel.de>
Message-ID: <036D3B39-B839-4DD3-BD51-2A8A7BAA020F@math.washington.edu>

I just checked, they are finally out. Has anyone had a chance to see  
how the current Cython fares on these beta releases yet?

On Jun 13, 2008, at 11:21 AM, Stefan Behnel wrote:

> Hi,
>
> just a quick note that the first beta versions of Python 2.6 and  
> 3.0 have been
> delayed by another week. They are now expected on June 18. This is  
> mainly due
> to stability and build problems caused by last-minute commits.
>
> I would still consider the C API stable by now, as the remaining  
> time is
> dedicated to fixing release critical bugs only.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Wed Jun 25 09:44:41 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 25 Jun 2008 09:44:41 +0200 (CEST)
Subject: [Cython] Python betas delayed again
In-Reply-To: <036D3B39-B839-4DD3-BD51-2A8A7BAA020F@math.washington.edu>
References: <4852BAAB.7010904@behnel.de>
	<036D3B39-B839-4DD3-BD51-2A8A7BAA020F@math.washington.edu>
Message-ID: <35027.213.61.181.86.1214379881.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> I just checked, they are finally out. Has anyone had a chance to see
> how the current Cython fares on these beta releases yet?

Yep, I already released lxml with cython-devel as of 2008-06-20 and it
works beautifully on both Py2.6 and 3.0 beta1.

Note that the final stdlib refactorings in Py3 have landed shortly before
the beta release, so there are now tons of things that have changed beyond
what Cython can work around for you. But that doesn't impact 2.6, so if
that's your primary target, you'll have less work to port your code.

Stefan


From dalcinl at gmail.com  Wed Jun 25 16:49:57 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 25 Jun 2008 11:49:57 -0300
Subject: [Cython] petsc4py with cython
In-Reply-To: <7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>
References: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
	<7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>
Message-ID: <e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>

On 6/25/08, Robert Bradshaw <robertwb at math.washington.edu> >
> Could you also add a link here?
>
>  http://wiki.cython.org/projects
>

I'll take that next week. I'm going to give my thesis dissertation
tomorrow.  The new cython-based implementations of mpi4py and petsc4py
were done from scratch in the middle of writting my thesis and
preparing its presentation. Moreover, at the same time, I started to
use Cython for the first time.

I comment this just for the pourpose of Cython advocacy. If you have
some experience and know what you need and want you want to do, Cython
is a very serious alternative for wrapping and generating
full-featured, pythonic API's . In fact, I believe is the best
approach from all the others out there.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From ellisonbg.net at gmail.com  Wed Jun 25 17:40:47 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Wed, 25 Jun 2008 09:40:47 -0600
Subject: [Cython] petsc4py with cython
In-Reply-To: <e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
References: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
	<7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>
	<e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
Message-ID: <6ce0ac130806250840i5264ec6bp9d9e8cfccbd701bc@mail.gmail.com>

Good luck on your thesis presentation!

On Wed, Jun 25, 2008 at 8:49 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 6/25/08, Robert Bradshaw <robertwb at math.washington.edu> >
>> Could you also add a link here?
>>
>>  http://wiki.cython.org/projects
>>
>
> I'll take that next week. I'm going to give my thesis dissertation
> tomorrow.  The new cython-based implementations of mpi4py and petsc4py
> were done from scratch in the middle of writting my thesis and
> preparing its presentation. Moreover, at the same time, I started to
> use Cython for the first time.
>
> I comment this just for the pourpose of Cython advocacy. If you have
> some experience and know what you need and want you want to do, Cython
> is a very serious alternative for wrapping and generating
> full-featured, pythonic API's . In fact, I believe is the best
> approach from all the others out there.
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Wed Jun 25 17:57:17 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 25 Jun 2008 08:57:17 -0700
Subject: [Cython] petsc4py with cython
In-Reply-To: <e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
References: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
	<7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>
	<e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
Message-ID: <4455C7AD-D861-4685-8960-3BF6EA3BDE95@math.washington.edu>

On Jun 25, 2008, at 7:49 AM, Lisandro Dalcin wrote:

> On 6/25/08, Robert Bradshaw <robertwb at math.washington.edu> >
>> Could you also add a link here?
>>
>>  http://wiki.cython.org/projects
>>
>
> I'll take that next week. I'm going to give my thesis dissertation
> tomorrow.

No hurry, Good luck!

- Robert


From fperez.net at gmail.com  Wed Jun 25 20:08:38 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 25 Jun 2008 11:08:38 -0700
Subject: [Cython] petsc4py with cython
In-Reply-To: <e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
References: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
	<7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>
	<e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
Message-ID: <db6b5ecc0806251108x55d21d2blbbe9a6b069aa8a28@mail.gmail.com>

On Wed, Jun 25, 2008 at 7:49 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> I'll take that next week. I'm going to give my thesis dissertation
> tomorrow.

As a memorable algebra professor of mine said: "Muchos ?xitos, porque
los ex?menes no son un asunto de suerte" :)

Cheers,

f

From dalcinl at gmail.com  Wed Jun 25 20:16:03 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 25 Jun 2008 15:16:03 -0300
Subject: [Cython] Updates on buffer spec
In-Reply-To: <251F921D-2EC6-4EF2-A2BA-F26BA3C9F6A4@math.washington.edu>
References: <1320.212.123.202.66.1214288291.squirrel@webmail.uio.no>
	<251F921D-2EC6-4EF2-A2BA-F26BA3C9F6A4@math.washington.edu>
Message-ID: <e7ba66e40806251116g3372db0dkda438b4c53064779@mail.gmail.com>

On 6/25/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>
>  > - Nobody complained about the [float64, 2]-syntax.
>
> No one has come up with anything better either, so lets run with it.
>

Has anybody considered the possibility of adding suport for dimension
bounds, like in Fortran 90 ? Such a feature would add the possibility
of checking the actual dimensions of numpy arrays (like F2Py does).


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed Jun 25 20:54:10 2008
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 25 Jun 2008 20:54:10 +0200 (CEST)
Subject: [Cython] Updates on buffer spec
In-Reply-To: <e7ba66e40806251116g3372db0dkda438b4c53064779@mail.gmail.com>
References: <1320.212.123.202.66.1214288291.squirrel@webmail.uio.no>
	<251F921D-2EC6-4EF2-A2BA-F26BA3C9F6A4@math.washington.edu>
	<e7ba66e40806251116g3372db0dkda438b4c53064779@mail.gmail.com>
Message-ID: <48127.193.90.3.70.1214420050.squirrel@webmail.uio.no>

Lisandro Dalcin wrote:
> On 6/25/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>
>>  > - Nobody complained about the [float64, 2]-syntax.
>>
>> No one has come up with anything better either, so lets run with it.
>>
>
> Has anybody considered the possibility of adding suport for dimension
> bounds, like in Fortran 90 ? Such a feature would add the possibility
> of checking the actual dimensions of numpy arrays (like F2Py does).

To clarify, do you mean the same as this:

cdef ndarray[float, 2] arr = x
assert arr.shape == (100, 29)

; only that you write

cdef ndarray[float, (100, 29)] arr = x

? Well, it can always be added later, I think the gains are fairly low and
that I would like to work on other things. (However, it does mean that if
you actually write code like

for i in range(99):
   arr[i, 2] = 34

...then you could drop the range checking on i with compiler optimizations
even if range checking is turned on. However, this is probably a rather
rare case...

Dag Sverre


From gael.varoquaux at normalesup.org  Wed Jun 25 22:52:01 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Wed, 25 Jun 2008 22:52:01 +0200
Subject: [Cython] petsc4py with cython
In-Reply-To: <db6b5ecc0806251108x55d21d2blbbe9a6b069aa8a28@mail.gmail.com>
References: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
	<7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>
	<e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
	<db6b5ecc0806251108x55d21d2blbbe9a6b069aa8a28@mail.gmail.com>
Message-ID: <20080625205201.GF9389@phare.normalesup.org>

On Wed, Jun 25, 2008 at 11:08:38AM -0700, Fernando Perez wrote:
> On Wed, Jun 25, 2008 at 7:49 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:

> > I'll take that next week. I'm going to give my thesis dissertation
> > tomorrow.

> As a memorable algebra professor of mine said: "Muchos ?xitos, porque
> los ex?menes no son un asunto de suerte" :)

L? tu abuse! Je comprends le fran?ais, l'anglais, le russe et un peu
d'italien, mais le "un asunto de surte", j'ai eu besoin de
wordreference.com pour comprendre...

Ga?l

PS: Pulling hairs, please forgive me...

From parlar at gmail.com  Wed Jun 25 23:07:57 2008
From: parlar at gmail.com (Jay Parlar)
Date: Wed, 25 Jun 2008 17:07:57 -0400
Subject: [Cython] Fastest way to store/retrieve floats in C
Message-ID: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>

I've implemented something like a CircularQueue, but not exactly.
Essentially, it has an 'append' method, which takes x,y and z
arguments, all of which are Python floats. These are added to an
internal array of type 'Point'.

When I call get(num), it pulls the 'num' most recent of these points,
puts them into a list, and returns them.

What I want to know, is there any way to do this more efficiently? It
seems wasteful to have to convert the Python floats to C floats, store
them, and then convert them back to Python floats later again. Would
my life just be easier if I kept them as Python values?

I've just been having a hard time finding the best way to move Python
lists back and forth with C, without doing full copies.

I've included a stripped down version of the code below. Thanks for
any pointers!

cdef struct Point:
    float x
    float y
    float z

cdef class cCircularQueue:

    cdef int _size, _start, _end, _iterator_length, _iterator_count, _len
    cdef int _getting_pos
    cdef Point *_data


    def __init__(self, unsigned size):
        cdef int i
        self._size = size + 1
        self._data = <Point *>malloc(self._size * sizeof(Point))
        self._end = 0
        self._iterator_count = 0
        self._getting_pos = 0
        for i from 0 <= i < size:
            self._data[i].x = 0.0
            self._data[i].y = 0.0
            self._data[i].z = 0.0

    def append(self, float x, float y, float z):
        self.cappend(x,y,z)

    cdef void cappend(self, float x, float y, float z):

        self._end = (self._end + 1) % self._size
        self._data[self._end].x = x
        self._data[self._end].y = y
        self._data[self._end].z = z

    def get(self, int num):
        cdef list values = []
        cdef int end = self._end
        cdef int check_getting_pos
        ......
        while (self._iterator_count != num) and \
                (self._getting_pos != check_getting_pos):

            n = self._getting_pos
            self._getting_pos = (self._getting_pos+1) % self._size
            self._iterator_count += 1
            values.append((self._data[n].x, self._data[n].y, self._data[n].z))

        return values


Jay P.

From ondrej at certik.cz  Thu Jun 26 09:43:46 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 26 Jun 2008 09:43:46 +0200
Subject: [Cython] petsc4py with cython
In-Reply-To: <20080625205201.GF9389@phare.normalesup.org>
References: <85b5c3130806210244t6fba0ac4j633f8218a253ec34@mail.gmail.com>
	<7874294C-27CA-4D23-B1FA-0D457616F95B@math.washington.edu>
	<e7ba66e40806250749i2f8dcbd3kee3712ba045a8e82@mail.gmail.com>
	<db6b5ecc0806251108x55d21d2blbbe9a6b069aa8a28@mail.gmail.com>
	<20080625205201.GF9389@phare.normalesup.org>
Message-ID: <85b5c3130806260043q3c075341m9c4be6227635d035@mail.gmail.com>

On Wed, Jun 25, 2008 at 10:52 PM, Gael Varoquaux
<gael.varoquaux at normalesup.org> wrote:
> On Wed, Jun 25, 2008 at 11:08:38AM -0700, Fernando Perez wrote:
>> On Wed, Jun 25, 2008 at 7:49 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>
>> > I'll take that next week. I'm going to give my thesis dissertation
>> > tomorrow.
>
>> As a memorable algebra professor of mine said: "Muchos ?xitos, porque
>> los ex?menes no son un asunto de suerte" :)
>
> L? tu abuse! Je comprends le fran?ais, l'anglais, le russe et un peu
> d'italien, mais le "un asunto de surte", j'ai eu besoin de
> wordreference.com pour comprendre...

This is better:

http://translate.google.com/

Ondrej

From simon at arrowtheory.com  Thu Jun 26 21:19:11 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Thu, 26 Jun 2008 15:19:11 -0400
Subject: [Cython] Fastest way to store/retrieve floats in C
In-Reply-To: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
References: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
Message-ID: <20080626151911.c0315e66.simon@arrowtheory.com>

On Wed, 25 Jun 2008 17:07:57 -0400
"Jay Parlar" <parlar at gmail.com> wrote:

> 
> What I want to know, is there any way to do this more efficiently? It
> seems wasteful to have to convert the Python floats to C floats, store
> them, and then convert them back to Python floats later again. Would
> my life just be easier if I kept them as Python values?

hmm.. maybe. You can also call the PyFloat_* functions yourself.

http://docs.python.org/api/floatObjects.html


> 
> I've just been having a hard time finding the best way to move Python
> lists back and forth with C, without doing full copies.

Yes, seems like it will always be expensive to build a python list of python
float's.

Why not make your own list extension class that understands how to access
your circularqueue ? Then return this thing from .get() .


Simon.

From parlar at gmail.com  Thu Jun 26 22:26:42 2008
From: parlar at gmail.com (Jay Parlar)
Date: Thu, 26 Jun 2008 16:26:42 -0400
Subject: [Cython] Fastest way to store/retrieve floats in C
In-Reply-To: <20080626151911.c0315e66.simon@arrowtheory.com>
References: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
	<20080626151911.c0315e66.simon@arrowtheory.com>
Message-ID: <e15ccc4d0806261326o289fe846w2ebb653c0b688563@mail.gmail.com>

On 6/26/08, Simon Burton <simon at arrowtheory.com> wrote:
> On Wed, 25 Jun 2008 17:07:57 -0400
>  "Jay Parlar" <parlar at gmail.com> wrote:

>  > I've just been having a hard time finding the best way to move Python
>  > lists back and forth with C, without doing full copies.
>
>
> Yes, seems like it will always be expensive to build a python list of python
>  float's.
>
>  Why not make your own list extension class that understands how to access
>  your circularqueue ? Then return this thing from .get() .

So build an extension class that inherits from the base-type list?

If so, won't I still be presented with roughly the same problem?
Within that extension class, I'll have to populate the internal
structure of the list with my float values. Or is it because I can use
the Python/C API calls to populate, instead of calling the
Python-visible append() that will give me the performance?

Thanks so much,
Jay P.

From simon at arrowtheory.com  Thu Jun 26 23:03:07 2008
From: simon at arrowtheory.com (Simon Burton)
Date: Thu, 26 Jun 2008 17:03:07 -0400
Subject: [Cython] Fastest way to store/retrieve floats in C
In-Reply-To: <e15ccc4d0806261326o289fe846w2ebb653c0b688563@mail.gmail.com>
References: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
	<20080626151911.c0315e66.simon@arrowtheory.com>
	<e15ccc4d0806261326o289fe846w2ebb653c0b688563@mail.gmail.com>
Message-ID: <20080626170307.b9683d7f.simon@arrowtheory.com>

On Thu, 26 Jun 2008 16:26:42 -0400
"Jay Parlar" <parlar at gmail.com> wrote:

> 
> So build an extension class that inherits from the base-type list?

no, but define the list-like methods (__getitem__ etc.)

> 
> If so, won't I still be presented with roughly the same problem?

> Within that extension class, I'll have to populate the internal
> structure of the list with my float values. Or is it because I can use
> the Python/C API calls to populate, instead of calling the
> Python-visible append() that will give me the performance?

You don't have to create/store python objects in your custom list.
Perhaps you could even alias the storage in your queue, and have
_zero_ copying.

It really depends where your bottleneck is, and what for/how often you are
using this get method. If all the work is being done at the python level
then i don't think any cython trickery will help. It's up to you to figure it out.

Simon.

From parlar at gmail.com  Fri Jun 27 02:03:10 2008
From: parlar at gmail.com (Jay Parlar)
Date: Thu, 26 Jun 2008 20:03:10 -0400
Subject: [Cython] Fastest way to store/retrieve floats in C
In-Reply-To: <20080626170307.b9683d7f.simon@arrowtheory.com>
References: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
	<20080626151911.c0315e66.simon@arrowtheory.com>
	<e15ccc4d0806261326o289fe846w2ebb653c0b688563@mail.gmail.com>
	<20080626170307.b9683d7f.simon@arrowtheory.com>
Message-ID: <e15ccc4d0806261703w6c2a92c9n4c9a0014fee32c7f@mail.gmail.com>

On 6/26/08, Simon Burton <simon at arrowtheory.com> wrote:
>  > Within that extension class, I'll have to populate the internal
>  > structure of the list with my float values. Or is it because I can use
>  > the Python/C API calls to populate, instead of calling the
>  > Python-visible append() that will give me the performance?
>
>
> You don't have to create/store python objects in your custom list.
>  Perhaps you could even alias the storage in your queue, and have
>  _zero_ copying.
>
>  It really depends where your bottleneck is, and what for/how often you are
>  using this get method. If all the work is being done at the python level
>  then i don't think any cython trickery will help. It's up to you to figure it out.

We're putting data into the queue 1000 times per second, so that needs
to be fast. And we're pulling data back out a few times per second,
but pulling out large chunks, so that too needs to be fast. This is
essentially the core stuff happening in the tight while loop of our
program.

And what do you mean by aliasing the storage? When I picture that, I
think of storing the Python objects in the queue, and then creating
lists with pointers back to the objects.

I've never worked with Python at the C API level before. I've been
programming Python for over 7 years, and C for longer, but never had
to mix them, so please forgive my ignorance :)

Jay P.

From robertwb at math.washington.edu  Fri Jun 27 03:51:26 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 26 Jun 2008 18:51:26 -0700
Subject: [Cython] Fastest way to store/retrieve floats in C
In-Reply-To: <e15ccc4d0806261703w6c2a92c9n4c9a0014fee32c7f@mail.gmail.com>
References: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
	<20080626151911.c0315e66.simon@arrowtheory.com>
	<e15ccc4d0806261326o289fe846w2ebb653c0b688563@mail.gmail.com>
	<20080626170307.b9683d7f.simon@arrowtheory.com>
	<e15ccc4d0806261703w6c2a92c9n4c9a0014fee32c7f@mail.gmail.com>
Message-ID: <726D4D1F-53A7-4265-ACB0-825712672E27@math.washington.edu>

On Jun 26, 2008, at 5:03 PM, Jay Parlar wrote:

> On 6/26/08, Simon Burton <simon at arrowtheory.com> wrote:
>>> Within that extension class, I'll have to populate the internal
>>> structure of the list with my float values. Or is it because I  
>>> can use
>>> the Python/C API calls to populate, instead of calling the
>>> Python-visible append() that will give me the performance?
>>
>>
>> You don't have to create/store python objects in your custom list.
>>  Perhaps you could even alias the storage in your queue, and have
>>  _zero_ copying.
>>
>>  It really depends where your bottleneck is, and what for/how  
>> often you are
>>  using this get method. If all the work is being done at the  
>> python level
>>  then i don't think any cython trickery will help. It's up to you  
>> to figure it out.
>
> We're putting data into the queue 1000 times per second, so that needs
> to be fast. And we're pulling data back out a few times per second,
> but pulling out large chunks, so that too needs to be fast. This is
> essentially the core stuff happening in the tight while loop of our
> program.

If the producer/consumer of the data is in Python, you are better off  
(I believe) storing them tuples of floats in a Python list than  
trying to wrap/unwrap them. To see significant speedups you would  
probably have to to write the code that pushes/pulls data from the  
queue to process and produce native C doubles, and interact with the  
queue using cdef functions. Having a fast queue isn't going to gain  
you much if the inner loop is pushing/pulling via Python objects via  
Python functions.

> And what do you mean by aliasing the storage? When I picture that, I
> think of storing the Python objects in the queue, and then creating
> lists with pointers back to the objects.
>
> I've never worked with Python at the C API level before. I've been
> programming Python for over 7 years, and C for longer, but never had
> to mix them, so please forgive my ignorance :)

Note that Cython automatically uses the PyFloat_* function on  
assignment to/from a double and PyList_Append (when doing L.append)  
so there's little advantage to trying to use the C/API directly.

- Robert



From parlar at gmail.com  Fri Jun 27 05:08:02 2008
From: parlar at gmail.com (Jay Parlar)
Date: Thu, 26 Jun 2008 23:08:02 -0400
Subject: [Cython] Fastest way to store/retrieve floats in C
In-Reply-To: <726D4D1F-53A7-4265-ACB0-825712672E27@math.washington.edu>
References: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
	<20080626151911.c0315e66.simon@arrowtheory.com>
	<e15ccc4d0806261326o289fe846w2ebb653c0b688563@mail.gmail.com>
	<20080626170307.b9683d7f.simon@arrowtheory.com>
	<e15ccc4d0806261703w6c2a92c9n4c9a0014fee32c7f@mail.gmail.com>
	<726D4D1F-53A7-4265-ACB0-825712672E27@math.washington.edu>
Message-ID: <e15ccc4d0806262008t24c5c529wfcdaf0e79e149487@mail.gmail.com>

On 6/26/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Jun 26, 2008, at 5:03 PM, Jay Parlar wrote:
>  > We're putting data into the queue 1000 times per second, so that needs
>  > to be fast. And we're pulling data back out a few times per second,
>  > but pulling out large chunks, so that too needs to be fast. This is
>  > essentially the core stuff happening in the tight while loop of our
>  > program.
>
>
> If the producer/consumer of the data is in Python, you are better off
>  (I believe) storing them tuples of floats in a Python list than
>  trying to wrap/unwrap them. To see significant speedups you would
>  probably have to to write the code that pushes/pulls data from the
>  queue to process and produce native C doubles, and interact with the
>  queue using cdef functions. Having a fast queue isn't going to gain
>  you much if the inner loop is pushing/pulling via Python objects via
>  Python functions.

Well, with a little rework, I can make the producer stay in C the
entire time. Every once in awhile, the consumer needs to pull the data
out and pass it off to wxPython though, so the end result definitely
has to be Python data.

However, the consumer is only running a handful of times per second.
So maybe that's what I'll do. Keep the entire producer in Pyrex, and
the only access that Python will have to it is through the get()
method. That way, I only have to convert *to* Python objects, and not
from.

Jay P.

From gael.varoquaux at normalesup.org  Fri Jun 27 07:18:47 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 07:18:47 +0200
Subject: [Cython] Student sponsorship for the SciPy08 conference
Message-ID: <20080627051847.GM11323@phare.normalesup.org>

We are delighted to announce that the Python Software Foundation has
answered our call and is providing sponsoring to the SciPy08 conference.

We will use this money to sponsor the registration fees and travel for up
to 10 college or graduate students to attend the conference. The PSF did
not provide all the founds required for all 10 students and once again
Enthought Inc. (http://www.enthought.com) is stepping up to fill in.

To apply, please send a short description of what you are studying and
why you?d like to attend to info at enthought.com. Please include telephone
contact information.

Thanks a lot to Travis Vaught from Enthought for bringing this project to
a success.

Please don't hesitate to forward this announcement to anybody who might
be interested.

Ga?l, on behalf of the Scipy08 organisation committee

SciPy coneference site:   http://conference.scipy.org

From robertwb at math.washington.edu  Fri Jun 27 08:08:49 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 26 Jun 2008 23:08:49 -0700
Subject: [Cython] Fastest way to store/retrieve floats in C
In-Reply-To: <e15ccc4d0806262008t24c5c529wfcdaf0e79e149487@mail.gmail.com>
References: <e15ccc4d0806251407m35b0461bh63e870c129720064@mail.gmail.com>
	<20080626151911.c0315e66.simon@arrowtheory.com>
	<e15ccc4d0806261326o289fe846w2ebb653c0b688563@mail.gmail.com>
	<20080626170307.b9683d7f.simon@arrowtheory.com>
	<e15ccc4d0806261703w6c2a92c9n4c9a0014fee32c7f@mail.gmail.com>
	<726D4D1F-53A7-4265-ACB0-825712672E27@math.washington.edu>
	<e15ccc4d0806262008t24c5c529wfcdaf0e79e149487@mail.gmail.com>
Message-ID: <18B76F60-3215-4C8C-B626-5AAFFFA392EE@math.washington.edu>

On Jun 26, 2008, at 8:08 PM, Jay Parlar wrote:

> On 6/26/08, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Jun 26, 2008, at 5:03 PM, Jay Parlar wrote:
>>> We're putting data into the queue 1000 times per second, so that  
>>> needs
>>> to be fast. And we're pulling data back out a few times per second,
>>> but pulling out large chunks, so that too needs to be fast. This is
>>> essentially the core stuff happening in the tight while loop of our
>>> program.
>>
>>
>> If the producer/consumer of the data is in Python, you are better off
>>  (I believe) storing them tuples of floats in a Python list than
>>  trying to wrap/unwrap them. To see significant speedups you would
>>  probably have to to write the code that pushes/pulls data from the
>>  queue to process and produce native C doubles, and interact with the
>>  queue using cdef functions. Having a fast queue isn't going to gain
>>  you much if the inner loop is pushing/pulling via Python objects via
>>  Python functions.
>
> Well, with a little rework, I can make the producer stay in C the
> entire time. Every once in awhile, the consumer needs to pull the data
> out and pass it off to wxPython though, so the end result definitely
> has to be Python data.
>
> However, the consumer is only running a handful of times per second.
> So maybe that's what I'll do. Keep the entire producer in Pyrex, and
> the only access that Python will have to it is through the get()
> method. That way, I only have to convert *to* Python objects, and not
> from.

Yep, that sounds like it would offer significant speedup, and you  
would want to store things in pure C. If you could compute summery  
data (e.g. averages, etc.) on the Cython side of things that would be  
really fast too.

- Robert


From languitar at semipol.de  Fri Jun 27 15:43:51 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Fri, 27 Jun 2008 15:43:51 +0200
Subject: [Cython] sys.exit in C function
Message-ID: <4864EE97.1090304@semipol.de>

Hi again,

I've got the following problem:
There is client code of my application that is already written and 
cotains so lines of code like these ones:

void foo(int c) {

	if (c < 5) {
		gui_exit(1)
	}

	// do some other things that fail if c is < 5

}

I am the one who has to implement gui_exit, which was originally a 
wrapper for the normal C exit function doing some clean up.

My first attempt to implement gui_exit in cython was using sys.exit. The 
problem with this approach is that sys.exit depends on exceptions, which 
are only checked after the client routine is completely processed. So 
foo does not exit at gui_exit and possibly generates a segmentation 
fault in the other things it does.

Is there a solution for this problem, that uses the python exit 
mechanism? I need that to clean up resources.

Thanks for the help
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080627/8d9e78ee/attachment.pgp 

From robertwb at math.washington.edu  Fri Jun 27 17:55:10 2008
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 27 Jun 2008 08:55:10 -0700
Subject: [Cython] sys.exit in C function
In-Reply-To: <4864EE97.1090304@semipol.de>
References: <4864EE97.1090304@semipol.de>
Message-ID: <9FAE3E5B-87FD-4B12-80A7-B76B6B8046CE@math.washington.edu>

On Jun 27, 2008, at 6:43 AM, Johannes Wienke wrote:

> Hi again,
>
> I've got the following problem:
> There is client code of my application that is already written and  
> cotains so lines of code like these ones:
>
> void foo(int c) {
>
> 	if (c < 5) {
> 		gui_exit(1)
> 	}
>
> 	// do some other things that fail if c is < 5
>
> }
>
> I am the one who has to implement gui_exit, which was originally a  
> wrapper for the normal C exit function doing some clean up.
>
> My first attempt to implement gui_exit in cython was using  
> sys.exit. The problem with this approach is that sys.exit depends  
> on exceptions, which are only checked after the client routine is  
> completely processed. So foo does not exit at gui_exit and possibly  
> generates a segmentation fault in the other things it does.
>
> Is there a solution for this problem, that uses the python exit  
> mechanism? I need that to clean up resources.
>
> Thanks for the help
> Johannes

Try


void foo(int c) {

	if (c < 5) {
		gui_exit(1)
	}

	else {

	// do some other things that fail if c is < 5

	}

}



Also, look at the exception passing mechanisms at http:// 
www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/ 
basics.html#mozTocId482761

- Robert

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080627/bacd747b/attachment.pgp 

From languitar at semipol.de  Fri Jun 27 19:05:19 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Fri, 27 Jun 2008 19:05:19 +0200
Subject: [Cython] sys.exit in C function
In-Reply-To: <9FAE3E5B-87FD-4B12-80A7-B76B6B8046CE@math.washington.edu>
References: <4864EE97.1090304@semipol.de>
	<9FAE3E5B-87FD-4B12-80A7-B76B6B8046CE@math.washington.edu>
Message-ID: <48651DCF.7090705@semipol.de>

Hi,

Am 06/27/2008 05:55 PM schrieb Robert Bradshaw:
> On Jun 27, 2008, at 6:43 AM, Johannes Wienke wrote:
>> I've got the following problem:
>> There is client code of my application that is already written and
>> cotains so lines of code like these ones:
>>
>> void foo(int c) {
>>
>>     if (c < 5) {
>>         gui_exit(1)
>>     }
>>
>>     // do some other things that fail if c is < 5
>>
>> }
>>
>> I am the one who has to implement gui_exit, which was originally a
>> wrapper for the normal C exit function doing some clean up.
>>
>> My first attempt to implement gui_exit in cython was using sys.exit.
>> The problem with this approach is that sys.exit depends on exceptions,
>> which are only checked after the client routine is completely
>> processed. So foo does not exit at gui_exit and possibly generates a
>> segmentation fault in the other things it does.
>>
>> Is there a solution for this problem, that uses the python exit
>> mechanism? I need that to clean up resources.
>
> Try
> 
> 
> void foo(int c) {
> 
>     if (c < 5) {
>         gui_exit(1)
>     }
> 
>     else {
> 
>     // do some other things that fail if c is < 5
> 
>     }
> 
> }

I can't change the client code. The original sematics of gui_exit was
that it terminates the programm at that point, so I have to preserve
this functionality in my implementation of gui_exit.

> Also, look at the exception passing mechanisms at
> http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/basics.html#mozTocId482761

Well, the problem I see here, is that the caller of gui_exit is not a
python function, but a real C function in a shared library I call.

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080627/c06122f9/attachment.pgp 

From gael.varoquaux at normalesup.org  Fri Jun 27 20:24:29 2008
From: gael.varoquaux at normalesup.org (Gael Varoquaux)
Date: Fri, 27 Jun 2008 20:24:29 +0200
Subject: [Cython] Scipy08 Paper submission deadline extention to Monday 30th
Message-ID: <20080627182429.GH5103@phare.normalesup.org>

The deadline for submitting abstracts to the Scipy conference was tonight.
In order to give you more time to submit excellent abstracts, the review
committee is extending the deadline to Monday (June 30th), and will work
hastily to get all of them reviewed in time for the program announcement,
on Thursday July 3rd.

----

The SciPy 2008 Conference will be held 21-22 August 2008 at the
California Institute of Technology, Pasadena, California. SciPy is a
scientific computing package, written in the Python language. It is
widely used in research, the industry and academia.

The program features tutorials, contributed papers, lightning talks, and
bird-of-a-feather sessions. We are soliciting talks and accompanying
papers (either formal academic or magazine-style articles) that discuss
topics which center around scientific computing using Python. These
include applications, teaching, future development directions and
research. A collection of peer-reviewed articles will be published as
part of the proceedings.

Proposals for talks are submitted as extended abstracts. There are two
categories of talks:
Lightning talks

These talks are 10 minutes in duration. An abstract of between 300 and
700 words should describe the topic and motivate its relevance to
scientific computing. Lightning talks do not require an accompanying
article (although, if submitted, these will still be published).
Paper presentations

These talks are 35 minutes in duration (including questions). A one page
abstract of no less than 500 words (excluding figures and references)
should give an outline of the final paper. Papers are due two weeks
before the conference, and may be in a formal academic style, or in a
more relaxed magazine-style format.

If you wish to present a talk at the conference, please create an account
on the website http://conference.scipy.org. You may then submit an
abstract by logging in, clicking on your profile and following the "
Submit an abstract " link.

Ga?l, on behalf on the SciPy08 organizing committee.

From dalcinl at gmail.com  Fri Jun 27 21:19:12 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 27 Jun 2008 16:19:12 -0300
Subject: [Cython] cython versus swig
In-Reply-To: <48652AFA.7020004@usgs.gov>
References: <48652AFA.7020004@usgs.gov>
Message-ID: <e7ba66e40806271219g774ed475ie580466548002172@mail.gmail.com>

On 6/27/08, Brad Aagaard <baagaard at usgs.gov> wrote:
>  I saw your message to petsc-dev regarding using cython for petsc4py. Are
> you switching from swig to cython?

Yes,

> I have been using Pyrex/pyrexembed to
> interface C++ and Python and find it very cumbersome and considerable work.

You are completelly right. In the current status of Cython/Pyrex,
wrapping C++ API's  is cumbersome, too much manual.

> I was planning to switch to swig, so I would be interested to learn how
> cython compares with swig.

Well, Cython is not too much different that Pyrex.

If you need to wrap a large C++ API, the SWIG is surelly the right
tool. Iff you learn how typemaps work, then you will be able to
generate really good, pythonic wrappers.

In the very particular case of petsc4py, the situation is not quite
the same. PETSc API is a C API. Then I used SWIG to generate an
extension module accessing that C API. But as PETSc has an OO design,
what I really need to do have is a pythonic, fully object oriented
wrapper, whith classes and all that. Then I had to write all those
class hierarchies in pure Python code, making inside methods the calls
to the SWIG generated wrappers.

Additionally, as I need petsc4py to be completelly interoperable with
third party code (I mean, I need to be able to access the actual
C-level Mat handle form the Py-level Mat instance, to pass it to
others C/C++/Fortran codes), I ended up writing from scratch Python
type object in C, implementing very low level stuff, and completelly
bypassing the 'this' machinery in SWIG with custom typemaps.

As you can imagine such implementation is curbersome to understand for
others, and it is a bit hard to maintain and extend. Then I decided to
switch to Cython. Then, the same source (a bunch of pxi and pyx files)
 generates the high-level, pythonic, fully object oriented API, and
also make the C calls to PETSc routines. And all this at C speed! no
bytecode involved!

Hope I was clear enough. In short, my final coment would be:

(1) If you have to wrap C API's, use Cython/Pyrex.
(2) If you have to wrap short C++ API's (let say 2 or 3 clases with
each haveing between 5 to 10 methods), then perhaps use Cython is
still a good idea,
(3) If you have to wrap a large C++ API, with lots of classes and
methods, and perhaps templates, then use SWIG.

Finally, regarding (3), that would be my opinion until some champion
can manage to write a decent C++ parser in order to automatize the C++
API --> Python API conversion. Once this feature is available, I'll
surelly wrap with Cython. I have no idea how long we'll have to wait
for this, the task is not trivial!!



>
>  Thanks,
>  Brad
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Jun 27 23:12:06 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 27 Jun 2008 18:12:06 -0300
Subject: [Cython] sys.exit in C function
In-Reply-To: <48651DCF.7090705@semipol.de>
References: <4864EE97.1090304@semipol.de>
	<9FAE3E5B-87FD-4B12-80A7-B76B6B8046CE@math.washington.edu>
	<48651DCF.7090705@semipol.de>
Message-ID: <e7ba66e40806271412q2e59e5ddo31a29201582e510@mail.gmail.com>

Perhaps a direct call to Py_Finalize()  inside gui_exit() would help ?

On 6/27/08, Johannes Wienke <languitar at semipol.de> wrote:
> Hi,
>
>  Am 06/27/2008 05:55 PM schrieb Robert Bradshaw:
>
> > On Jun 27, 2008, at 6:43 AM, Johannes Wienke wrote:
>
> >> I've got the following problem:
>  >> There is client code of my application that is already written and
>  >> cotains so lines of code like these ones:
>  >>
>  >> void foo(int c) {
>  >>
>  >>     if (c < 5) {
>  >>         gui_exit(1)
>  >>     }
>  >>
>  >>     // do some other things that fail if c is < 5
>  >>
>  >> }
>  >>
>  >> I am the one who has to implement gui_exit, which was originally a
>  >> wrapper for the normal C exit function doing some clean up.
>  >>
>  >> My first attempt to implement gui_exit in cython was using sys.exit.
>  >> The problem with this approach is that sys.exit depends on exceptions,
>  >> which are only checked after the client routine is completely
>  >> processed. So foo does not exit at gui_exit and possibly generates a
>  >> segmentation fault in the other things it does.
>  >>
>  >> Is there a solution for this problem, that uses the python exit
>  >> mechanism? I need that to clean up resources.
>  >
>
> > Try
>  >
>  >
>  > void foo(int c) {
>  >
>  >     if (c < 5) {
>  >         gui_exit(1)
>  >     }
>  >
>  >     else {
>  >
>  >     // do some other things that fail if c is < 5
>  >
>  >     }
>  >
>  > }
>
>
> I can't change the client code. The original sematics of gui_exit was
>  that it terminates the programm at that point, so I have to preserve
>  this functionality in my implementation of gui_exit.
>
>
>  > Also, look at the exception passing mechanisms at
>  > http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/basics.html#mozTocId482761
>
>
> Well, the problem I see here, is that the caller of gui_exit is not a
>  python function, but a real C function in a shared library I call.
>
>
>  Johannes
>
>
> _______________________________________________
>  Cython-dev mailing list
>  Cython-dev at codespeak.net
>  http://codespeak.net/mailman/listinfo/cython-dev
>
>
>


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From languitar at semipol.de  Fri Jun 27 23:16:17 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Fri, 27 Jun 2008 23:16:17 +0200
Subject: [Cython] sys.exit in C function
In-Reply-To: <e7ba66e40806271412q2e59e5ddo31a29201582e510@mail.gmail.com>
References: <4864EE97.1090304@semipol.de>
	<9FAE3E5B-87FD-4B12-80A7-B76B6B8046CE@math.washington.edu>
	<48651DCF.7090705@semipol.de>
	<e7ba66e40806271412q2e59e5ddo31a29201582e510@mail.gmail.com>
Message-ID: <486558A1.5060908@semipol.de>

Am 06/27/2008 11:12 PM schrieb Lisandro Dalcin:
> Perhaps a direct call to Py_Finalize()  inside gui_exit() would help ?

Is that everything python does on sys.exit normally?

Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080627/ee6488e4/attachment-0001.pgp 

From stefan_ml at behnel.de  Sat Jun 28 06:55:25 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 28 Jun 2008 06:55:25 +0200
Subject: [Cython] sys.exit in C function
In-Reply-To: <486558A1.5060908@semipol.de>
References: <4864EE97.1090304@semipol.de>	<9FAE3E5B-87FD-4B12-80A7-B76B6B8046CE@math.washington.edu>	<48651DCF.7090705@semipol.de>	<e7ba66e40806271412q2e59e5ddo31a29201582e510@mail.gmail.com>
	<486558A1.5060908@semipol.de>
Message-ID: <4865C43D.30500@behnel.de>


Johannes Wienke wrote:
> Am 06/27/2008 11:12 PM schrieb Lisandro Dalcin:
>> Perhaps a direct call to Py_Finalize()  inside gui_exit() would help ?
> 
> Is that everything python does on sys.exit normally?

What about checking the source code?

Stefan


From jwienke at techfak.uni-bielefeld.de  Sun Jun 29 13:06:00 2008
From: jwienke at techfak.uni-bielefeld.de (Johannes Wienke)
Date: Sun, 29 Jun 2008 13:06:00 +0200
Subject: [Cython] sys.exit in C function
In-Reply-To: <4865C43D.30500@behnel.de>
References: <4864EE97.1090304@semipol.de>
	<9FAE3E5B-87FD-4B12-80A7-B76B6B8046CE@math.washington.edu>
	<48651DCF.7090705@semipol.de>
	<e7ba66e40806271412q2e59e5ddo31a29201582e510@mail.gmail.com>
	<486558A1.5060908@semipol.de> <4865C43D.30500@behnel.de>
Message-ID: <48676C98.1090902@techfak.uni-bielefeld.de>

Am 06/28/2008 06:55 AM schrieb Stefan Behnel:
> Johannes Wienke wrote:
>> Am 06/27/2008 11:12 PM schrieb Lisandro Dalcin:
>>> Perhaps a direct call to Py_Finalize()  inside gui_exit() would help ?
>> Is that everything python does on sys.exit normally?
> 
> What about checking the source code?

Well, I did find the sys.exit call, but not the code that acutally
handles the exception. But I think simply calling Py_Finalize() looks
enough.

Thanks
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080629/e6550a91/attachment.pgp 

From languitar at semipol.de  Sun Jun 29 13:16:37 2008
From: languitar at semipol.de (Johannes Wienke)
Date: Sun, 29 Jun 2008 13:16:37 +0200
Subject: [Cython] Mapping a really strange typedef in Cython
Message-ID: <48676F15.1090607@semipol.de>

Hi again,

I'm struggling to get a really strange typedef working in Cython. The
original C (preprocessor) code looks like this:

#define IW_CTAB_SIZE	256
typedef guchar (*iwColtab)[3];

extern iwColtab iw_def_col_tab;

/* Color formats of the data for prev_render...() */
#define IW_RGB			((iwColtab)0)

// some more formats

#define IW_BW			((iwColtab)7)
#define IW_GRAY			IW_BW
#define IW_INDEX		(iw_def_col_tab)

#define IW_COLFORMAT_MAX	IW_BW	/* last 'special' definition */

The purpose of this strange construct was that one can use normal color
spaces without struggling, but one is also able to use a custom color table.

This definition was originally used like this:

void prev_colConvert (iwColtab ctab, gint r, gint g, gint b,
                      gint *rd, gint *gd, gint *bd)
{
	if (ctab == IW_RGB) {
		*rd = r;
		*gd = g;
		*bd = b;
	} else if (ctab == IW_BGR) {
		*rd = b;
		*gd = g;
		*bd = r;
	} else if (ctab > IW_COLFORMAT_MAX && r>=0 && r < IW_CTAB_SIZE){
		*rd = ctab[r][0];
		*gd = ctab[r][1];
		*bd = ctab[r][2];
	}

}

I need to be able to do the same thing in cython with this definition.
My current approach was simply using a enum to emulate it:

cdef extern from "gui/Gimage.h":

    ctypedef enum iwColtab:
        IW_RGB
        # ...
        IW_GRAY
        IW_CTAB_SIZE
        IW_COLFORMAT_MAX

The normal color modes work like this, but using the color table doesn't
work:

Error converting Pyrex file to C:
------------------------------------------------------------
...
    elif ctab == IW_YUV:
        return Color(1.164 * (r - 16) + 1.596 * (b - 128),
                     1.164 * (r - 16) - 0.813 * (b - 128) - 0.391 * (g -
128),
                     1.164 * (r - 16) + 2.018 * (g - 128))
    elif (ctab > IW_COLFORMAT_MAX and r >= 0 and r < IW_CTAB_SIZE):
        return Color(ctab[r][0], ctab[r][1], ctab[r][2])
                        ^
------------------------------------------------------------

image_utils.pyx:144:25: Attempting to index non-array type 'iwColtab'

I really have no idea how to solve this. Has anyone got a solution for this?

Thanks for the help
Johannes

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080629/8bdb8129/attachment.pgp 

From pete at petertodd.org  Mon Jun 30 17:12:02 2008
From: pete at petertodd.org (Peter Todd)
Date: Mon, 30 Jun 2008 11:12:02 -0400
Subject: [Cython] .pxd's that depend on other .pxd's
Message-ID: <20080630151202.GA4534@tilt>

Using the attached tarball I get this error message with Cython 0.9.8:


pete at tilt:~/t/bug$ cython Tuke/id.pyx 

Error converting Pyrex file to C:
------------------------------------------------------------
...
from Tuke.translatable cimport Translatable
^
------------------------------------------------------------

/home/pete/t/bug/Tuke/id.pyx:1:0: 'Tuke.translatable.pxd' not found

Error converting Pyrex file to C:
------------------------------------------------------------
...
from Tuke.translatable cimport Translatable
                              ^
------------------------------------------------------------

/home/pete/t/bug/Tuke/id.pyx:1:31: Name 'Translatable' not declared in module 'Tuke.translatable'

Error converting Pyrex file to C:
------------------------------------------------------------
...
from Tuke.translatable cimport Translatable

cdef class Id(Translatable):
    ^
------------------------------------------------------------

/home/pete/t/bug/Tuke/id.pyx:3:5: 'Translatable' is not declared
pete at tilt:~/t/bug$ 

Is this correct behavior? The files are pretty simple, I just want the class in
id.pyx to be a sub-class of something in translatable.pxd

-- 
http://petertodd.org 'peter'[:-1]@petertodd.org 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bug.tar.gz
Type: application/octet-stream
Size: 339 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080630/c7c8d646/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://codespeak.net/pipermail/cython-dev/attachments/20080630/c7c8d646/attachment.pgp 

From efiring at hawaii.edu  Mon Jun 23 05:04:48 2008
From: efiring at hawaii.edu (Eric Firing)
Date: Mon, 23 Jun 2008 03:04:48 -0000
Subject: [Cython] [Numpy-discussion] Surprising performance tweak in
	Cython
In-Reply-To: <20080623003730.GF15908@phare.normalesup.org>
References: <20080623003730.GF15908@phare.normalesup.org>
Message-ID: <485F124E.3000104@hawaii.edu>

Gael Varoquaux wrote:
> I tried tweak my Cython code for performance by manually inlining a small
> function, and ended up with a less performant code. I must confess I
> don't really understand what is going on here. If somebody has an
> explaination, I'd be delighted. The code follows.
> 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> from numpy import zeros
> 
> # Make sure numpy is initialized.
> include "c_numpy.pxd"
> 
> ##############################################################################
> cdef int inner_loop(float c_x, float c_y):
>     cdef float x, y, x_buffer
>     x = 0; y = 0
>     cdef int i
>     for i in range(50):
>         x_buffer = x*x - y*y + c_x
>         y = 2*x*y + c_y
>         x = x_buffer
>         if (x*x + x*y > 100):

The line above looks like a typo, and does not match your inline 
version.  Could that be making the difference?

Eric


>             return 50 - i
> 
> def do_Mandelbrot_cython():
>     cdef ndarray threshold_time 
>     threshold_time = zeros((500, 500))
>     cdef double *tp
>     cdef float c_x, c_y
>     cdef int i, j
>     c_x = -1.5
>     tp = <double*>threshold_time.data
>     for i in range(500):
>         c_y = -1
>         for j in range(500):
>             tp += 1
>             c_y += 0.004
>             tp[0] = inner_loop(c_x, c_y)
>         c_x += 0.004
>     return threshold_time
> 
> 
> def do_Mandelbrot_cython2():
>     cdef ndarray threshold_time
>     threshold_time = zeros((500, 500))
>     cdef double *tp
>     tp = <double*>threshold_time.data
>     cdef float x, y, xbuffer, c_x, c_y
>     cdef int i, j, n 
>     c_x = -1.5
>     for i in range(500):
>         c_y = -1
>         for j in range(500):
>             tp += 1
>             c_y += 0.004
>             x = 0; y = 0
>             for n in range(50):
>                 x_buffer = x*x - y*y + c_x
>                 y = 2*x*y + c_y
>                 x = x_buffer
>                 if (x*x + y*y > 100):
>                     tp[0] = 50 -n
>                     break
>         c_x += 0.004
>     return threshold_time
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
> And the timing I get are:
> 
> In [2]: %timeit C.do_Mandelbrot_cython2()
> 10 loops, best of 3: 342 ms per loop
> 
> In [3]: %timeit C.do_Mandelbrot_cython()
> 10 loops, best of 3: 126 ms per loop
> 
> Cheers,
> 
> Ga?l
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion