From stefan_ml at behnel.de  Mon Feb  1 08:42:44 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Feb 2010 08:42:44 +0100
Subject: [Cython] C++ support - A simple tutorial
In-Reply-To: <CF4668D7-DF59-4669-86C9-A5113C690363@math.washington.edu>
References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com>	<4B62B789.3060803@behnel.de>	<81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu>	<961fa2b41001291512wbc1866as53f25440a718bf66@mail.gmail.com>
	<CF4668D7-DF59-4669-86C9-A5113C690363@math.washington.edu>
Message-ID: <4B6685F4.3050807@behnel.de>


Robert Bradshaw, 30.01.2010 01:31:
> There's no way to interpret the expression
> 
>      new A(1,10)
> 
> as valid Python, so there's no ambiguity.

Ok, it's not /really/ a keyword then, in the sense that it cannot appear
outside of its grammar context. It would still be allowed everywhere,
except that the case that it is followed by a name would be special cased.

But doesn't that overly complicate the parser? I mean, 'new' can
potentially appear in most places of an expression, and it can always mean
both things...

Stefan

From dagss at student.matnat.uio.no  Mon Feb  1 09:29:53 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Feb 2010 09:29:53 +0100
Subject: [Cython] C++ support - A simple tutorial
In-Reply-To: <4B6685F4.3050807@behnel.de>
References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com>	<4B62B789.3060803@behnel.de>	<81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu>	<961fa2b41001291512wbc1866as53f25440a718bf66@mail.gmail.com>	<CF4668D7-DF59-4669-86C9-A5113C690363@math.washington.edu>
	<4B6685F4.3050807@behnel.de>
Message-ID: <4B669101.8010009@student.matnat.uio.no>

Stefan Behnel wrote:
> Robert Bradshaw, 30.01.2010 01:31:
>   
>> There's no way to interpret the expression
>>
>>      new A(1,10)
>>
>> as valid Python, so there's no ambiguity.
>>     
>
> Ok, it's not /really/ a keyword then, in the sense that it cannot appear
> outside of its grammar context. It would still be allowed everywhere,
> except that the case that it is followed by a name would be special cased.
>
> But doesn't that overly complicate the parser? I mean, 'new' can
> potentially appear in most places of an expression, and it can always mean
> both things...
>   
There's a limit to how simple the parser can be anyway, we can't use the 
simple parsing strategy of Python since we're supporting some C syntax. 
The tokenizer supports putting tokens back, so one can simply try for 
"new X" first, and if there's no match then one can put the token back 
into the tokenizer and call p_expr. Not much more complicated than 
anything else that goes on in the parser...


Dag Sverre

From stefan_ml at behnel.de  Mon Feb  1 10:30:18 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Feb 2010 10:30:18 +0100
Subject: [Cython] Fwd: [codespeak-ann] codespeak migration 3rd feb 2010
Message-ID: <4B669F2A.9030300@behnel.de>

FYI


-------- Original-Message --------
Subject: [codespeak-ann] codespeak migration 3rd feb 2010
Date: Mon, 1 Feb 2010 10:16:43 +0100
From: holger krekel <holger at merlinux.eu>
To: codespeak-ann at codespeak.net

Hi all,

3rd Feb 2010, wednesday afternoon, we plan to migrate codespeak to a new
host -
we are leaving XEN and going for KVM, together with a hardware upgrade.

Please be prepared for downtimes on that afternoon.

Oh and if somebody is interested and wants to help with providing
mercurial hosting on codespeak please contact me.

cheers,
holger


From stefan_ml at behnel.de  Mon Feb  1 14:06:10 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Feb 2010 14:06:10 +0100
Subject: [Cython] Support for borrowed references
In-Reply-To: <4B5D61BE.3030901@student.matnat.uio.no>
References: <4B48E1BA.7000304@behnel.de>
	<4B4A0D26.7070807@behnel.de>	<4B5D4659.9070206@behnel.de>
	<4B5D61BE.3030901@student.matnat.uio.no>
Message-ID: <4B66D1C2.7010709@behnel.de>


Dag Sverre Seljebotn, 25.01.2010 10:17:
> Stefan Behnel wrote:
>>> [...] why not just represent a borrowed reference as a pointer?
>>> So you could write
>>>
>>>      cdef list some_list = []
>>>      cdef list* borrowed_ref = some_list
>>>
>>> and borrowed_ref would be a non-refcounted pointer to a Python list. 
>>> Assignments back to a normal reference would be allowed:
>>>
>>>      cdef list my_list = borrowed_ref   # increfs the pointer
>>>
>>> After all, a non-refcounted reference to a Python object is not more than a 
>>> bare pointer to a well-defined Python builtin/extension type (including 
>>> "object*").
> 
> Well, there's the drawback of making the language more complicated, and 
> also I think the notation is confusing -- plain "object" is not by 
> value, and "object*" is not a pointer to an "object", so to speak.

I was trying to let the syntax emphasise that it *is* a pointer, not a
reference. A reference is special in Cython in that it handles ref-counting
for it. A pointer just behaves like any other pointer in the language.

I agree that it's not flawless, but I think it makes sense and is unambiguous.


> How about a set of explicit functions like
> 
> cdef object x = skip_incref(PyTuple_GET_ITEM(t, 0))

I don't think that fits the use case very well.

Stefan

From dagss at student.matnat.uio.no  Mon Feb  1 15:37:31 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Feb 2010 15:37:31 +0100
Subject: [Cython] Support for borrowed references
In-Reply-To: <4B66D1C2.7010709@behnel.de>
References: <4B48E1BA.7000304@behnel.de>	<4B4A0D26.7070807@behnel.de>	<4B5D4659.9070206@behnel.de>	<4B5D61BE.3030901@student.matnat.uio.no>
	<4B66D1C2.7010709@behnel.de>
Message-ID: <4B66E72B.9020007@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn, 25.01.2010 10:17:
>   
>> Stefan Behnel wrote:
>>     
>>>> [...] why not just represent a borrowed reference as a pointer?
>>>> So you could write
>>>>
>>>>      cdef list some_list = []
>>>>      cdef list* borrowed_ref = some_list
>>>>
>>>> and borrowed_ref would be a non-refcounted pointer to a Python list. 
>>>> Assignments back to a normal reference would be allowed:
>>>>
>>>>      cdef list my_list = borrowed_ref   # increfs the pointer
>>>>
>>>> After all, a non-refcounted reference to a Python object is not more than a 
>>>> bare pointer to a well-defined Python builtin/extension type (including 
>>>> "object*").
>>>>         
>> Well, there's the drawback of making the language more complicated, and 
>> also I think the notation is confusing -- plain "object" is not by 
>> value, and "object*" is not a pointer to an "object", so to speak.
>>     
>
> I was trying to let the syntax emphasise that it *is* a pointer, not a
> reference. A reference is special in Cython in that it handles ref-counting
> for it. A pointer just behaves like any other pointer in the language.
>
> I agree that it's not flawless, but I think it makes sense and is unambiguous.
>   
Let me put it this way: To me, "object*" somehow indicates PyObject**, 
since "object" is PyObject*. I find it confusing that "object" and 
"object*" are at the same "indirection level", so to speak.

Also, would

cdef object* x = ...
print x[0]

do a getitem on x, or returned a reference-counted copy of x? Not easy 
to guess from the syntax alone.

Assuming a getitem (as that's a useful operation and the other is 
already served by <object>x), one is then inconsistent with everywhere 
else the *-notation is used.

I'd be fine with almost any kind of syntax which is a superset of the 
current syntax -- "cdef nonmanaged object x", you name it. The problem 
with "object*" is that it is not immediately obvious to a reader with a 
brief knowledge of Cython that it is fundamentally different from "int*".

This is a very subjective matter though.

Dag Sverre

From dagss at student.matnat.uio.no  Mon Feb  1 15:39:18 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Feb 2010 15:39:18 +0100
Subject: [Cython] Support for borrowed references
In-Reply-To: <4B66E72B.9020007@student.matnat.uio.no>
References: <4B48E1BA.7000304@behnel.de>	<4B4A0D26.7070807@behnel.de>	<4B5D4659.9070206@behnel.de>	<4B5D61BE.3030901@student.matnat.uio.no>
	<4B66D1C2.7010709@behnel.de>
	<4B66E72B.9020007@student.matnat.uio.no>
Message-ID: <4B66E796.2010907@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 25.01.2010 10:17:
>>  
>>> Stefan Behnel wrote:
>>>    
>>>>> [...] why not just represent a borrowed reference as a pointer?
>>>>> So you could write
>>>>>
>>>>>      cdef list some_list = []
>>>>>      cdef list* borrowed_ref = some_list
>>>>>
>>>>> and borrowed_ref would be a non-refcounted pointer to a Python 
>>>>> list. Assignments back to a normal reference would be allowed:
>>>>>
>>>>>      cdef list my_list = borrowed_ref   # increfs the pointer
>>>>>
>>>>> After all, a non-refcounted reference to a Python object is not 
>>>>> more than a bare pointer to a well-defined Python 
>>>>> builtin/extension type (including "object*").
>>>>>         
>>> Well, there's the drawback of making the language more complicated, 
>>> and also I think the notation is confusing -- plain "object" is not 
>>> by value, and "object*" is not a pointer to an "object", so to speak.
>>>     
>>
>> I was trying to let the syntax emphasise that it *is* a pointer, not a
>> reference. A reference is special in Cython in that it handles 
>> ref-counting
>> for it. A pointer just behaves like any other pointer in the language.
>>
>> I agree that it's not flawless, but I think it makes sense and is 
>> unambiguous.
>>   
> Let me put it this way: To me, "object*" somehow indicates PyObject**, 
> since "object" is PyObject*. I find it confusing that "object" and 
> "object*" are at the same "indirection level", so to speak.
>
> Also, would
>
> cdef object* x = ...
> print x[0]
>
> do a getitem on x, or returned a reference-counted copy of x? Not easy 
> to guess from the syntax alone.
>
> Assuming a getitem (as that's a useful operation and the other is 
> already served by <object>x), one is then inconsistent with everywhere 
> else the *-notation is used.
>
> I'd be fine with almost any kind of syntax which is a superset of the 
> current syntax -- "cdef nonmanaged object x", you name it. The problem 
> with "object*" is that it is not immediately obvious to a reader with 
> a brief knowledge of Cython that it is fundamentally different from 
> "int*".

Another example: Reading the docs up front, just reading someone elses 
code, I'd expect this to work:

cdef object x
cdef object* px
px = &x
px[0] = "hello"
print x # should print "hello"


Dag Sverre

From dagss at student.matnat.uio.no  Mon Feb  1 15:39:52 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Feb 2010 15:39:52 +0100
Subject: [Cython] Support for borrowed references
In-Reply-To: <4B66E796.2010907@student.matnat.uio.no>
References: <4B48E1BA.7000304@behnel.de>	<4B4A0D26.7070807@behnel.de>	<4B5D4659.9070206@behnel.de>	<4B5D61BE.3030901@student.matnat.uio.no>
	<4B66D1C2.7010709@behnel.de>
	<4B66E72B.9020007@student.matnat.uio.no>
	<4B66E796.2010907@student.matnat.uio.no>
Message-ID: <4B66E7B8.5080807@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 25.01.2010 10:17:
>>>  
>>>> Stefan Behnel wrote:
>>>>   
>>>>>> [...] why not just represent a borrowed reference as a pointer?
>>>>>> So you could write
>>>>>>
>>>>>>      cdef list some_list = []
>>>>>>      cdef list* borrowed_ref = some_list
>>>>>>
>>>>>> and borrowed_ref would be a non-refcounted pointer to a Python 
>>>>>> list. Assignments back to a normal reference would be allowed:
>>>>>>
>>>>>>      cdef list my_list = borrowed_ref   # increfs the pointer
>>>>>>
>>>>>> After all, a non-refcounted reference to a Python object is not 
>>>>>> more than a bare pointer to a well-defined Python 
>>>>>> builtin/extension type (including "object*").
>>>>>>         
>>>> Well, there's the drawback of making the language more complicated, 
>>>> and also I think the notation is confusing -- plain "object" is not 
>>>> by value, and "object*" is not a pointer to an "object", so to speak.
>>>>     
>>>
>>> I was trying to let the syntax emphasise that it *is* a pointer, not a
>>> reference. A reference is special in Cython in that it handles 
>>> ref-counting
>>> for it. A pointer just behaves like any other pointer in the language.
>>>
>>> I agree that it's not flawless, but I think it makes sense and is 
>>> unambiguous.
>>>   
>> Let me put it this way: To me, "object*" somehow indicates 
>> PyObject**, since "object" is PyObject*. I find it confusing that 
>> "object" and "object*" are at the same "indirection level", so to speak.
>>
>> Also, would
>>
>> cdef object* x = ...
>> print x[0]
>>
>> do a getitem on x, or returned a reference-counted copy of x? Not 
>> easy to guess from the syntax alone.
>>
>> Assuming a getitem (as that's a useful operation and the other is 
>> already served by <object>x), one is then inconsistent with 
>> everywhere else the *-notation is used.
>>
>> I'd be fine with almost any kind of syntax which is a superset of the 
>> current syntax -- "cdef nonmanaged object x", you name it. The 
>> problem with "object*" is that it is not immediately obvious to a 
>> reader with a brief knowledge of Cython that it is fundamentally 
>> different from "int*".
>
> Another example: Reading the docs up front, just reading someone elses 
> code, I'd expect this to work:

I meant ***without*** reading the docs up front...

>
> cdef object x
> cdef object* px
> px = &x
> px[0] = "hello"
> print x # should print "hello"
>
>
> Dag Sverre
>


From stefan_ml at behnel.de  Mon Feb  1 16:41:44 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Feb 2010 16:41:44 +0100
Subject: [Cython] Support for borrowed references
In-Reply-To: <4B66E72B.9020007@student.matnat.uio.no>
References: <4B48E1BA.7000304@behnel.de>	<4B4A0D26.7070807@behnel.de>	<4B5D4659.9070206@behnel.de>	<4B5D61BE.3030901@student.matnat.uio.no>	<4B66D1C2.7010709@behnel.de>
	<4B66E72B.9020007@student.matnat.uio.no>
Message-ID: <4B66F638.6030206@behnel.de>


Dag Sverre Seljebotn, 01.02.2010 15:37:
> I'd be fine with almost any kind of syntax which is a superset of the 
> current syntax -- "cdef nonmanaged object x", you name it. The problem 
> with "object*" is that it is not immediately obvious to a reader with a 
> brief knowledge of Cython that it is fundamentally different from "int*".

Makes sense to me. Then we're back to "borrowed" as an object type modifier.

http://trac.cython.org/cython_trac/ticket/498

Stefan


From robertwb at math.washington.edu  Mon Feb  1 20:19:08 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Feb 2010 11:19:08 -0800
Subject: [Cython] C++ support - A simple tutorial
In-Reply-To: <4B669101.8010009@student.matnat.uio.no>
References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com>	<4B62B789.3060803@behnel.de>	<81A86FC7-4E40-4CB4-8FCE-4D6E6A3F8121@math.washington.edu>	<961fa2b41001291512wbc1866as53f25440a718bf66@mail.gmail.com>	<CF4668D7-DF59-4669-86C9-A5113C690363@math.washington.edu>
	<4B6685F4.3050807@behnel.de>
	<4B669101.8010009@student.matnat.uio.no>
Message-ID: <37BD9105-BF0D-4B20-B0D0-87209F593A3C@math.washington.edu>

On Feb 1, 2010, at 12:29 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Robert Bradshaw, 30.01.2010 01:31:
>>
>>> There's no way to interpret the expression
>>>
>>>     new A(1,10)
>>>
>>> as valid Python, so there's no ambiguity.
>>>
>>
>> Ok, it's not /really/ a keyword then, in the sense that it cannot  
>> appear
>> outside of its grammar context. It would still be allowed everywhere,
>> except that the case that it is followed by a name would be special  
>> cased.

Exactly. The same as "extern" or "public" is not really a keyword.

>> But doesn't that overly complicate the parser? I mean, 'new' can
>> potentially appear in most places of an expression, and it can  
>> always mean
>> both things...
>>
> There's a limit to how simple the parser can be anyway, we can't use  
> the
> simple parsing strategy of Python since we're supporting some C  
> syntax.
> The tokenizer supports putting tokens back, so one can simply try for
> "new X" first, and if there's no match then one can put the token back
> into the tokenizer and call p_expr. Not much more complicated than
> anything else that goes on in the parser...

Which is basically what happens. Even then it's much cleaner than some  
other areas of the parser.

- Robert


From robertwb at math.washington.edu  Mon Feb  1 20:31:03 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Feb 2010 11:31:03 -0800
Subject: [Cython] Support for borrowed references
In-Reply-To: <4B66E72B.9020007@student.matnat.uio.no>
References: <4B48E1BA.7000304@behnel.de>	<4B4A0D26.7070807@behnel.de>	<4B5D4659.9070206@behnel.de>	<4B5D61BE.3030901@student.matnat.uio.no>
	<4B66D1C2.7010709@behnel.de>
	<4B66E72B.9020007@student.matnat.uio.no>
Message-ID: <6A1506A1-1CE2-4FB9-B855-07FA5ACBC0FA@math.washington.edu>

On Feb 1, 2010, at 6:37 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 25.01.2010 10:17:
>>
>>> Stefan Behnel wrote:
>>>
>>>>> [...] why not just represent a borrowed reference as a pointer?
>>>>> So you could write
>>>>>
>>>>>     cdef list some_list = []
>>>>>     cdef list* borrowed_ref = some_list
>>>>>
>>>>> and borrowed_ref would be a non-refcounted pointer to a Python  
>>>>> list.
>>>>> Assignments back to a normal reference would be allowed:
>>>>>
>>>>>     cdef list my_list = borrowed_ref   # increfs the pointer
>>>>>
>>>>> After all, a non-refcounted reference to a Python object is not  
>>>>> more than a
>>>>> bare pointer to a well-defined Python builtin/extension type  
>>>>> (including
>>>>> "object*").
>>>>>
>>> Well, there's the drawback of making the language more  
>>> complicated, and
>>> also I think the notation is confusing -- plain "object" is not by
>>> value, and "object*" is not a pointer to an "object", so to speak.
>>>
>>
>> I was trying to let the syntax emphasise that it *is* a pointer,  
>> not a
>> reference. A reference is special in Cython in that it handles ref- 
>> counting
>> for it. A pointer just behaves like any other pointer in the  
>> language.
>>
>> I agree that it's not flawless, but I think it makes sense and is  
>> unambiguous.
>>
> Let me put it this way: To me, "object*" somehow indicates PyObject**,
> since "object" is PyObject*. I find it confusing that "object" and
> "object*" are at the same "indirection level", so to speak.
>
> Also, would
>
> cdef object* x = ...
> print x[0]
>
> do a getitem on x, or returned a reference-counted copy of x? Not easy
> to guess from the syntax alone.
>
> Assuming a getitem (as that's a useful operation and the other is
> already served by <object>x), one is then inconsistent with everywhere
> else the *-notation is used.
>
> I'd be fine with almost any kind of syntax which is a superset of the
> current syntax -- "cdef nonmanaged object x", you name it. The problem
> with "object*" is that it is not immediately obvious to a reader  
> with a
> brief knowledge of Cython that it is fundamentally different from  
> "int*".
>
> This is a very subjective matter though.

My impressions are similar. A "borrowed" type modifier is fine by me.

- Robert


From craigcitro at gmail.com  Tue Feb  2 01:27:50 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Mon, 1 Feb 2010 16:27:50 -0800
Subject: [Cython] Cython 0.13 and cython-closures branch
Message-ID: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>

Hi all,

So this has already been discussed in several other threads, but I
think the current plan is: once Robert releases 0.12.1, we'll pull all
the changes from there into the cython-closures branch, and turn
cython-closures into cython-devel for 0.13. I was chatting with
Robert, and I think I'm happy to be the release manager for 0.13,
unless there are any objections ...

A few quick notes:

 - I just finished (taking forever to) add a big battery of tests in
the cython-closures branch. I ended up adding a wee bit of code, and
then working out a few reference counting bugs that we hit with
closures. I've committed all the tests into the closures branch (as a
series of 5 files; gcc and g++ were choking with everything in one
file). The tests are actually an old battery of tests in Scheme from
the compilers courses I took in undergrad -- I wrote a little compiler
that converted these to Python (modulo one issue I punted on due to a
difference with the way Scheme handles scoping for letrec). I verified
that Scheme, Python, and Cython gave the same results for each of
these, and the refnanny didn't complain at all, so I feel pretty
confident in the closures code. (Big thanks to the professors who
first came up with the battery of tests, and yes, I did check with
them before committing the tests.)

 - In the process of doing this, the refnanny complained a few times
-- so I wrote myself some notes on how reference counting should be
working. I realized these might be useful in general, so I've added a
little extra text and posted it on the wiki:

  http://wiki.cython.org/refcounting

 - There is one place where I really didn't test out the closures code
much, especially with an eye towards memory leaks/reference counting
troubles, and that's error handling. If you have some code that uses
nested functions and a bunch of error handling, I'd love to either (1)
hear what happens when you run it with the refnanny turned on, or (2)
have you send me a copy/post it somewhere so we can give it a go.

This is good -- I think the next step is to start working on the
generators implementation that came up in some other threads, and is
outlined on the wiki. I'm planning on starting on that soon, if no one
beats me to it -- and if my six-month old daughter decides to start
sleeping through the night, I might actually move faster than molasses
in January. ;)

-cc

From bpederse at gmail.com  Tue Feb  2 04:15:44 2010
From: bpederse at gmail.com (Brent Pedersen)
Date: Mon, 1 Feb 2010 19:15:44 -0800
Subject: [Cython] C++ support - A simple tutorial
In-Reply-To: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com>
References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com>
Message-ID: <e183a99d1002011915j3b971aedw299dd00ec0be5aeb@mail.gmail.com>

On Thu, Jan 28, 2010 at 7:34 PM, Danilo Freitas <dsurviver at gmail.com> wrote:
> As you know, Robert is planning to put the work about C++ Support in
> next release. So, we have some new stuff, and people need to learn it.
>
> I wrote a very simple tutorial on wiki [0], with some examples. I
> think it's easy to learn (and use) it.
>
> So, if you have any questions about it, just ask here.
>
> [0] http://wiki.cython.org/gsoc09/daniloaf/progress
>
> --
> - Danilo Freitas
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

hi, i'm interested in using the STL vector and hash_map stuff, i found this:
http://hg.cython.org/gsoc-danilo/file/a076c2505c5e/tests/run/cpp_stl_vector.pyx

in my uninformed opinion, it'd be nice to have a couple quick examples
in your tutorial.
is the "del v" in the finally block that appears in all the test cases
necessary?

thanks,
-brent

From dsurviver at gmail.com  Tue Feb  2 04:54:34 2010
From: dsurviver at gmail.com (Danilo Freitas)
Date: Tue, 2 Feb 2010 00:54:34 -0300
Subject: [Cython] C++ support - A simple tutorial
In-Reply-To: <e183a99d1002011915j3b971aedw299dd00ec0be5aeb@mail.gmail.com>
References: <45239151001281934y521eb398ybb3c48df6ba3e463@mail.gmail.com> 
	<e183a99d1002011915j3b971aedw299dd00ec0be5aeb@mail.gmail.com>
Message-ID: <45239151002011954p63ba843dl1910776d3ee5a6fe@mail.gmail.com>

2010/2/2 Brent Pedersen <bpederse at gmail.com>:
> On Thu, Jan 28, 2010 at 7:34 PM, Danilo Freitas <dsurviver at gmail.com> wrote:
>> As you know, Robert is planning to put the work about C++ Support in
>> next release. So, we have some new stuff, and people need to learn it.
>>
>> I wrote a very simple tutorial on wiki [0], with some examples. I
>> think it's easy to learn (and use) it.
>>
>> So, if you have any questions about it, just ask here.
>>
>> [0] http://wiki.cython.org/gsoc09/daniloaf/progress
>>
>> --
>> - Danilo Freitas
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> hi, i'm interested in using the STL vector and hash_map stuff, i found this:
> http://hg.cython.org/gsoc-danilo/file/a076c2505c5e/tests/run/cpp_stl_vector.pyx
>
> in my uninformed opinion, it'd be nice to have a couple quick examples
> in your tutorial.

Hi. I'm working on better examples. I'll soon update the wiki with them.

> is the "del v" in the finally block that appears in all the test cases
> necessary?

When creating a pointer object with 'new', it's heap allocated. So, we
need to remove it to free memory. It's like C++, for example:

class Foo
{
    ...
};

int main()
{
    Foo *foo = new Foo();
    ...
    delete foo;
    return 0;
}

It's very important use it when you won't use that object again, or
you'll have memory leak. It's like malloc and free in C.


-- 
- Danilo Freitas

From stefan_ml at behnel.de  Tue Feb  2 09:06:01 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Feb 2010 09:06:01 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
Message-ID: <4B67DCE9.1080005@behnel.de>


Craig Citro, 02.02.2010 01:27:
> So this has already been discussed in several other threads, but I
> think the current plan is: once Robert releases 0.12.1, we'll pull all
> the changes from there into the cython-closures branch, and turn
> cython-closures into cython-devel for 0.13. I was chatting with
> Robert, and I think I'm happy to be the release manager for 0.13,
> unless there are any objections ...

Meaning, he talked you into it and you naively agreed? ;)


>  - I just finished (taking forever to) add a big battery of tests in
> the cython-closures branch.

Way cool. Thanks!

The tests look absolutely machine generated, BTW. :)


> I ended up adding a wee bit of code, and
> then working out a few reference counting bugs that we hit with
> closures. I've committed all the tests into the closures branch (as a
> series of 5 files; gcc and g++ were choking with everything in one
> file).

I actually tried to use tcc for running the test suite a while ago, but it
crashed for me. Usually, dactivating most optimisations with "gcc -O1"
speeds up the test runs by quite a bit, and might also drop the memory
requirement a little.


>  - In the process of doing this, the refnanny complained a few times
> -- so I wrote myself some notes on how reference counting should be
> working. I realized these might be useful in general, so I've added a
> little extra text and posted it on the wiki:
> 
>   http://wiki.cython.org/refcounting

The Wiki appears to be dead right now.


>  - There is one place where I really didn't test out the closures code
> much, especially with an eye towards memory leaks/reference counting
> troubles, and that's error handling.

Ah, right, that's a long standing deficiency of the test suite anyway. I
bet a coverage report would reveal that 99% of the error handling code is
never exercised, really. Much is that is because there are tons of dead
code that are only ever taken on memory errors (or other Python internal
errors that can't really be produced in a test suite), but at least a tiny
bit is simply never tested.

Are there any special error cases in the closures code that would deserve
special testing?


> I think the next step is to start working on the
> generators implementation that came up in some other threads, and is
> outlined on the wiki. I'm planning on starting on that soon, if no one
> beats me to it

Cool. How much time can you spare on this?

If you need any help, please ask (and make sure you're not 80% through with
it before you ask ;).

Stefan

From cournape at gmail.com  Tue Feb  2 10:06:30 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 2 Feb 2010 18:06:30 +0900
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B67DCE9.1080005@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67DCE9.1080005@behnel.de>
Message-ID: <5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>

On Tue, Feb 2, 2010 at 5:06 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
>
> I actually tried to use tcc for running the test suite a while ago, but it
> crashed for me. Usually, dactivating most optimisations with "gcc -O1"
> speeds up the test runs by quite a bit, and might also drop the memory
> requirement a little.

You may want to try clang from the llvm project. It compiles most code
fine, and is significantly faster than gcc (the memory requirement,
especially for c++, are even more significant). It is relatively easy
to build (at least llvm 2.6), and is even included on mac os x (in
xcode 3.1).

cheers,

David

From dagss at student.matnat.uio.no  Tue Feb  2 10:45:59 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 02 Feb 2010 10:45:59 +0100
Subject: [Cython] Standard C pxd conventions
Message-ID: <4B67F457.3020107@student.matnat.uio.no>

What I'm wondering about is whether we can start having namespaces in 
Cython/Includes. Especially since math is already a module, and I think 
"c.math" is clearer than inventing "cmath".

So I propose:

from c.math cimport sqrt, acos, cos, sin, round
from c.stdio cimport fopen

and so on (though stdlib.pxd and stdio.pxd will be kept in place for 
backwards compatability).

This leaves room for

from c99.math cimport ...
from cpp.vector cimport ...

and perhaps also

from cpython.exc cimport ... # same as cimport python_exc

Thoughts?

Dag Sverre

From stefan_ml at behnel.de  Tue Feb  2 11:35:04 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Feb 2010 11:35:04 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
Message-ID: <4B67FFD8.5070909@behnel.de>


David Cournapeau, 02.02.2010 10:06:
> On Tue, Feb 2, 2010 at 5:06 PM, Stefan Behnel wrote:
>>
>> I actually tried to use tcc for running the test suite a while ago, but it
>> crashed for me. Usually, dactivating most optimisations with "gcc -O1"
>> speeds up the test runs by quite a bit, and might also drop the memory
>> requirement a little.
> 
> You may want to try clang from the llvm project. It compiles most code
> fine, and is significantly faster than gcc (the memory requirement,
> especially for c++, are even more significant). It is relatively easy
> to build (at least llvm 2.6), and is even included on mac os x (in
> xcode 3.1).

Cool! There's a ready-made Debian package for it:

http://packages.debian.org/sid/clang

Works perfectly on my Ubuntu Karmic x86_64 system. Without any optimisation
options, the C-only cython-devel test suite runs in 156 seconds (2:36) on
my machine, compared to 191 (3:11) with gcc. Not the kind of difference
that lets you start running it from a commit hook, but certainly noticeable.

Stefan

From robertwb at math.washington.edu  Tue Feb  2 11:57:27 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Feb 2010 02:57:27 -0800
Subject: [Cython] Cython 0.12.1 released
In-Reply-To: <63d1516f-5f27-42fa-ad5e-7a5c394c6a58@k5g2000pra.googlegroups.com>
References: <63d1516f-5f27-42fa-ad5e-7a5c394c6a58@k5g2000pra.googlegroups.com>
Message-ID: <D18DE68A-5EA0-429F-A474-E7B5B5FACAF7@math.washington.edu>

It's not apparent from the headers, but this is the message I just  
posted to comp.lang.python.announce (which is moderated).

- Robert


On Feb 2, 2010, at 2:53 AM, Robert Bradshaw wrote:

> I'm happy to announce the release of Cython 0.12.1.
>
> == About ==
>
> Cython is a language that makes writing C extensions for the Python
> language as easy as Python itself. Cython is based on the well-known
> Pyrex, but supports more cutting edge functionality and
> optimizations.Cython is an ideal language for wrapping external C
> libraries, and for fast C modules that speed up the execution of
> Python code. For more information, see http://cython.org.
>
> == Where to get it ==
>
> http://cython.org
> http://pypi.python.org/pypi/Cython
>
> == New Features ==
>
> * Type inference improvements.
>
> There have been several bug fixes and improvements to the type
> inferencer.
> Notably, there is now a "safe" mode enabled by setting the infer_types
> directive to None. (The None here refers to the "default" mode, which
> will be the default in 0.13.) This safe mode limits inference to
> Python object types and C doubles, which should speed up execution
> without affecting any semantics such as integer overflow behavior like
> infer_types=True might. There is also an infer_types.verbose option
> which allows one to see what types are inferred.
>
> * The boundscheck directive works for lists and tuples as well as
> buffers.
> * len(s) and s.decode("encoding") are efficiently supported for char*
> s.
> * Cython's INLINE macro has been renamed to CYTHON_INLINE to reduce
> conflict and has better support for the MSVC compiler on Windows. It
> is no longer clobbered if externally defined.
> * Revision history is now omitted from the source package, resulting
> in a 85% size reduction. Running make repo will download the history
> and turn the directory into a complete Mercurial working repository.
> * Cython modules don't need to be recompiled when the size of an
> external type grows. (A warning, rather than an error, is produced.)
> This should be helpful for binary distributions relying on NumPy.
>
> Several other bugs and minor improvements have been made. This release
> should be fully backwards compatible with 0.12. For a list of tickets
> closed, see http://trac.cython.org/cython_trac/query?group=component&milestone=0.12.1
>
> == Contributors to this release ==
>
> * Arfrever Frehtes Taifersar Arahesis
> * Stefan Behnel
> * Robert Bradshaw
> * Lisandro Dalcin
> * Julien Danjou
> * Mark Lodato
> * Dag Sverre Seljebotn
>
> Thanks also to the many who've submitted feedback and contributed to
> discussions on the mailing list.


From robertwb at math.washington.edu  Tue Feb  2 12:17:45 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Feb 2010 03:17:45 -0800
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B67F457.3020107@student.matnat.uio.no>
References: <4B67F457.3020107@student.matnat.uio.no>
Message-ID: <DDA4F0D6-8B8D-47E3-AE37-8B7AEE8DCFCF@math.washington.edu>

On Feb 2, 2010, at 1:45 AM, Dag Sverre Seljebotn wrote:

> What I'm wondering about is whether we can start having namespaces in
> Cython/Includes.

Yes.

> Especially since math is already a module, and I think
> "c.math" is clearer than inventing "cmath".
>
> So I propose:
>
> from c.math cimport sqrt, acos, cos, sin, round
> from c.stdio cimport fopen
>
> and so on (though stdlib.pxd and stdio.pxd will be kept in place for
> backwards compatability).
>
> This leaves room for
>
> from c99.math cimport ...
> from cpp.vector cimport ...

or maybe from stl.vector cimport ... would be more natural

>
> and perhaps also
>
> from cpython.exc cimport ... # same as cimport python_exc

I don't know that we need to preface python with a c here.

- Robert



From dagss at student.matnat.uio.no  Tue Feb  2 12:49:01 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 02 Feb 2010 12:49:01 +0100
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <DDA4F0D6-8B8D-47E3-AE37-8B7AEE8DCFCF@math.washington.edu>
References: <4B67F457.3020107@student.matnat.uio.no>
	<DDA4F0D6-8B8D-47E3-AE37-8B7AEE8DCFCF@math.washington.edu>
Message-ID: <4B68112D.10806@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 2, 2010, at 1:45 AM, Dag Sverre Seljebotn wrote:
>
>   
>> What I'm wondering about is whether we can start having namespaces in
>> Cython/Includes.
>>     
>
> Yes.
>
>   
>> Especially since math is already a module, and I think
>> "c.math" is clearer than inventing "cmath".
>>
>> So I propose:
>>
>> from c.math cimport sqrt, acos, cos, sin, round
>> from c.stdio cimport fopen
>>
>> and so on (though stdlib.pxd and stdio.pxd will be kept in place for
>> backwards compatability).
>>
>> This leaves room for
>>
>> from c99.math cimport ...
>> from cpp.vector cimport ...
>>     
>
> or maybe from stl.vector cimport ... would be more natural
>   
IIRC, the STL name was deprecated after it was merged into the C++ 
standard, it is just part of the "C++ library" without a seperate name. 
References to STL are out of habit and because it started out as a 
seperate library.

Or I remember reading that, but can't find a reference. Since STL is in 
such widespread usage, I'm sure an STL namespace wouldn't hurt.

Dag Sverre

From robertwb at math.washington.edu  Tue Feb  2 13:00:25 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Feb 2010 04:00:25 -0800
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B68112D.10806@student.matnat.uio.no>
References: <4B67F457.3020107@student.matnat.uio.no>
	<DDA4F0D6-8B8D-47E3-AE37-8B7AEE8DCFCF@math.washington.edu>
	<4B68112D.10806@student.matnat.uio.no>
Message-ID: <38E717E5-1AA2-49F0-BB02-59211BB1BFE8@math.washington.edu>

On Feb 2, 2010, at 3:49 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 2, 2010, at 1:45 AM, Dag Sverre Seljebotn wrote:
>>
>>
>>> What I'm wondering about is whether we can start having namespaces  
>>> in
>>> Cython/Includes.
>>>
>>
>> Yes.
>>
>>
>>> Especially since math is already a module, and I think
>>> "c.math" is clearer than inventing "cmath".
>>>
>>> So I propose:
>>>
>>> from c.math cimport sqrt, acos, cos, sin, round
>>> from c.stdio cimport fopen
>>>
>>> and so on (though stdlib.pxd and stdio.pxd will be kept in place for
>>> backwards compatability).
>>>
>>> This leaves room for
>>>
>>> from c99.math cimport ...
>>> from cpp.vector cimport ...
>>>
>>
>> or maybe from stl.vector cimport ... would be more natural
>>
> IIRC, the STL name was deprecated after it was merged into the C++
> standard, it is just part of the "C++ library" without a seperate  
> name.
> References to STL are out of habit and because it started out as a
> seperate library.
>
> Or I remember reading that, but can't find a reference. Since STL is  
> in
> such widespread usage, I'm sure an STL namespace wouldn't hurt.

Or std, as it mirrors the C++ namespace structure, just to throw  
another option out there.

- Robert



From stefan_ml at behnel.de  Tue Feb  2 13:14:07 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Feb 2010 13:14:07 +0100
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B67F457.3020107@student.matnat.uio.no>
References: <4B67F457.3020107@student.matnat.uio.no>
Message-ID: <4B68170F.5040304@behnel.de>


Dag Sverre Seljebotn, 02.02.2010 10:45:
> What I'm wondering about is whether we can start having namespaces in 
> Cython/Includes. Especially since math is already a module, and I think 
> "c.math" is clearer than inventing "cmath".
> 
> So I propose:
> 
> from c.math cimport sqrt, acos, cos, sin, round
> from c.stdio cimport fopen

I'm fine with namespaces, although I find "math" a particularly bad
example. I'd like to be able to do this one day:

    from math cimport sin

and have it use the C library's sin() function. But I guess that just an
orthogonal feature.

I also wonder if "c" isn't a bit short for an unambiguous namespace. I
doubt that it's already in use by any sensible code (exactly for that
reason), but I'd like to make sure we keep the risk of naming collisions
low, *exactly because* we start using packaged names here.

Stefan


From robertwb at math.washington.edu  Tue Feb  2 11:53:09 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Feb 2010 02:53:09 -0800 (PST)
Subject: [Cython] Cython 0.12.1 released
Message-ID: <63d1516f-5f27-42fa-ad5e-7a5c394c6a58@k5g2000pra.googlegroups.com>

I'm happy to announce the release of Cython 0.12.1.

== About ==

Cython is a language that makes writing C extensions for the Python
language as easy as Python itself. Cython is based on the well-known
Pyrex, but supports more cutting edge functionality and
optimizations.Cython is an ideal language for wrapping external C
libraries, and for fast C modules that speed up the execution of
Python code. For more information, see http://cython.org.

== Where to get it ==

http://cython.org
http://pypi.python.org/pypi/Cython

== New Features ==

* Type inference improvements.

There have been several bug fixes and improvements to the type
inferencer.
Notably, there is now a "safe" mode enabled by setting the infer_types
directive to None. (The None here refers to the "default" mode, which
will be the default in 0.13.) This safe mode limits inference to
Python object types and C doubles, which should speed up execution
without affecting any semantics such as integer overflow behavior like
infer_types=True might. There is also an infer_types.verbose option
which allows one to see what types are inferred.

* The boundscheck directive works for lists and tuples as well as
buffers.
* len(s) and s.decode("encoding") are efficiently supported for char*
s.
* Cython's INLINE macro has been renamed to CYTHON_INLINE to reduce
conflict and has better support for the MSVC compiler on Windows. It
is no longer clobbered if externally defined.
* Revision history is now omitted from the source package, resulting
in a 85% size reduction. Running make repo will download the history
and turn the directory into a complete Mercurial working repository.
* Cython modules don't need to be recompiled when the size of an
external type grows. (A warning, rather than an error, is produced.)
This should be helpful for binary distributions relying on NumPy.

Several other bugs and minor improvements have been made. This release
should be fully backwards compatible with 0.12. For a list of tickets
closed, see http://trac.cython.org/cython_trac/query?group=component&milestone=0.12.1

== Contributors to this release ==

* Arfrever Frehtes Taifersar Arahesis
* Stefan Behnel
* Robert Bradshaw
* Lisandro Dalcin
* Julien Danjou
* Mark Lodato
* Dag Sverre Seljebotn

Thanks also to the many who've submitted feedback and contributed to
discussions on the mailing list.

From dagss at student.matnat.uio.no  Tue Feb  2 13:43:31 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 02 Feb 2010 13:43:31 +0100
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B68170F.5040304@behnel.de>
References: <4B67F457.3020107@student.matnat.uio.no>
	<4B68170F.5040304@behnel.de>
Message-ID: <4B681DF3.7070802@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn, 02.02.2010 10:45:
>   
>> What I'm wondering about is whether we can start having namespaces in 
>> Cython/Includes. Especially since math is already a module, and I think 
>> "c.math" is clearer than inventing "cmath".
>>
>> So I propose:
>>
>> from c.math cimport sqrt, acos, cos, sin, round
>> from c.stdio cimport fopen
>>     
>
> I'm fine with namespaces, although I find "math" a particularly bad
> example. I'd like to be able to do this one day:
>
>     from math cimport sin
>
> and have it use the C library's sin() function. But I guess that just an
> orthogonal feature.
>
> I also wonder if "c" isn't a bit short for an unambiguous namespace. I
> doubt that it's already in use by any sensible code (exactly for that
> reason), but I'd like to make sure we keep the risk of naming collisions
> low, *exactly because* we start using packaged names here.
>   
Hmm, I just checked, and in Python, this

numpy = 3
import numpy.linalg

resulted in numpy being imported and overwriting the 3 with the module. 
So you may have a point.

(cimports has to come before one could assign anything to c, but still..)

Other options I can think of:
 - C
 - clib
 - libc
 - cc

Dag Sverre

From dagss at student.matnat.uio.no  Tue Feb  2 13:46:03 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 02 Feb 2010 13:46:03 +0100
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B681DF3.7070802@student.matnat.uio.no>
References: <4B67F457.3020107@student.matnat.uio.no>	<4B68170F.5040304@behnel.de>
	<4B681DF3.7070802@student.matnat.uio.no>
Message-ID: <4B681E8B.8020305@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>   
>> Dag Sverre Seljebotn, 02.02.2010 10:45:
>>   
>>     
>>> What I'm wondering about is whether we can start having namespaces in 
>>> Cython/Includes. Especially since math is already a module, and I think 
>>> "c.math" is clearer than inventing "cmath".
>>>
>>> So I propose:
>>>
>>> from c.math cimport sqrt, acos, cos, sin, round
>>> from c.stdio cimport fopen
>>>     
>>>       
>> I'm fine with namespaces, although I find "math" a particularly bad
>> example. I'd like to be able to do this one day:
>>
>>     from math cimport sin
>>
>> and have it use the C library's sin() function. But I guess that just an
>> orthogonal feature.
>>
>> I also wonder if "c" isn't a bit short for an unambiguous namespace. I
>> doubt that it's already in use by any sensible code (exactly for that
>> reason), but I'd like to make sure we keep the risk of naming collisions
>> low, *exactly because* we start using packaged names here.
>>     
Sorry, I misread you first time around -- you're talking about 
collisions with other *Python* libraries being named "c"? In that case 
I'd think the chances of collision are higher for anything else we could 
think of (like clib, libc).

The alternative then appears to be

from cython.c.math cimport sqrt
from cython.python cimport Py_INCREF

but that's awfully verbose.

Dag Sverre

From ndbecker2 at gmail.com  Tue Feb  2 13:52:57 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Tue, 02 Feb 2010 07:52:57 -0500
Subject: [Cython] 0.12.1 Tests failing on ppc
Message-ID: <hk9778$3cf$1@ger.gmane.org>

Build log is here:
http://koji.fedoraproject.org/koji/getfile?taskID=1958725&name=build.log

Errors start with:

ERROR: Doctest: c_int_types_T255
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for c_int_types_T255
  File 
"/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/c_int_types_T255.so", line 
92, in c_int_types_T255
----------------------------------------------------------------------
File "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/c_int_types_T255.so", 
line 98, in c_int_types_T255
Failed example:
    test_char(CHAR_MIN) == CHAR_MIN
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.6/doctest.py", line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest c_int_types_T255[1]>", line 1, in <module>
        test_char(CHAR_MIN) == CHAR_MIN
      File "c_int_types_T255.pyx", line 8, in c_int_types_T255.test_char 
(c_int_types_T255.c:644)
    OverflowError: can't convert negative value to char


From Chris.Barker at noaa.gov  Tue Feb  2 23:01:09 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Tue, 02 Feb 2010 14:01:09 -0800
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B681E8B.8020305@student.matnat.uio.no>
References: <4B67F457.3020107@student.matnat.uio.no>
	<4B68170F.5040304@behnel.de> <4B681DF3.7070802@student.matnat.uio.no>
	<4B681E8B.8020305@student.matnat.uio.no>
Message-ID: <4B68A0A5.6000908@noaa.gov>

Dag Sverre Seljebotn wrote:
> The alternative then appears to be
> 
> from cython.c.math cimport sqrt
> from cython.python cimport Py_INCREF
> 
> but that's awfully verbose.

I LIKE verbose imports -- that's what "from" is for. The import can be 
verbose (and clear, and unlikely to name clash), while the rest of your 
code is as terse as you like.

If this is all cython stuff, it should live in the cython namespace.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov

From dagss at student.matnat.uio.no  Wed Feb  3 10:22:49 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Feb 2010 10:22:49 +0100
Subject: [Cython] Merging memoryview
Message-ID: <4B694069.4000104@student.matnat.uio.no>

Me and Kurt's been talking about (finally) getting the memory views 
merged. Initially I held back because I wanted to do my part of the job 
first (support indexing, currently they only support raw buffer access 
and copying), but in the light of how long that's been taking me it's 
better to get things merged now -- especially as Kurt has a use for the 
existing functionality in fwrap.

Question first:
 - Should the memoryview namespace be named "cython.view", 
"cython.memview", "cython.memoryview", "cython.mview", "cython.memory", 
"cython.buffer"?

The functionality should be considered experimental until Cython 0.14, 
but it shouldn't interfer as long as you don't use the above namespace 
or the int[:]-syntax.

The gsoc-kurt branch has had the changes from cython-devel merged into 
it and there's no real conflict, but some stuff left to do. The way 
things are looking I suggest that:

a) The closure branch gets to merge first

b) We try to keep gsoc-kurt updated and merge it afterwards. Kurt should 
hopefully be able to plan on gsoc-kurt being merged in time for 0.13 and 
use it for fwrap development.

Open tasks before we can merge gsoc-kurt:
 - Types created by Cython output in wrong order (again!). This is 
getting silly, and somebody, meaning me I guess, should code up a DAG 
for outputting type declarations in their right order. See 
http://trac.cython.org/cython_trac/ticket/469.
 - memview_declarations fail due to "invalid use of follow specifier". 
Kurt, could you have a look?

Open tasks we can look at after the merge:
 - Consistent and better naming (memview vs. view vs. memoryview vs. 
mview used in the source)
 - More memory efficient dynamic struct generation
 - Item indexing
 - Transform NumPy accesses to memoryview (probably not until 0.13.1 at 
least).

Dag Sverre

From stefan_ml at behnel.de  Wed Feb  3 10:44:14 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Feb 2010 10:44:14 +0100
Subject: [Cython] Merging memoryview
In-Reply-To: <4B694069.4000104@student.matnat.uio.no>
References: <4B694069.4000104@student.matnat.uio.no>
Message-ID: <4B69456E.2000601@behnel.de>


Dag Sverre Seljebotn, 03.02.2010 10:22:
> Me and Kurt's been talking about (finally) getting the memory views 
> merged.

Could you add a sentence or two on what we are talking about here? We've
had several discussions in the past regarding various kinds of memory views
and array types.

Stefan


From dagss at student.matnat.uio.no  Wed Feb  3 10:45:54 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Feb 2010 10:45:54 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B67FFD8.5070909@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de>
Message-ID: <4B6945D2.6030202@student.matnat.uio.no>

Stefan Behnel wrote:
> David Cournapeau, 02.02.2010 10:06:
>   
>> On Tue, Feb 2, 2010 at 5:06 PM, Stefan Behnel wrote:
>>     
>>> I actually tried to use tcc for running the test suite a while ago, but it
>>> crashed for me. Usually, dactivating most optimisations with "gcc -O1"
>>> speeds up the test runs by quite a bit, and might also drop the memory
>>> requirement a little.
>>>       
>> You may want to try clang from the llvm project. It compiles most code
>> fine, and is significantly faster than gcc (the memory requirement,
>> especially for c++, are even more significant). It is relatively easy
>> to build (at least llvm 2.6), and is even included on mac os x (in
>> xcode 3.1).
>>     
>
> Cool! There's a ready-made Debian package for it:
>
> http://packages.debian.org/sid/clang
>
> Works perfectly on my Ubuntu Karmic x86_64 system. Without any optimisation
> options, the C-only cython-devel test suite runs in 156 seconds (2:36) on
> my machine, compared to 191 (3:11) with gcc. Not the kind of difference
> that lets you start running it from a commit hook, but certainly noticeable.
>   

A project I've been thinking about is Cython-support for the waf build 
system:

http://code.google.com/p/waf/

 - Parallell, reliable builds! (I think parallell testing should help a 
good deal on the runtime of the test suite, it's not all IO bound -- or 
if it is, one should consider ramdisks)
 - It runs on Python 2.3 to 3.1 (though only with bzip2...sure that can 
be changed if necesarry)
 - It's reasonable small, and the recommended mode of distribution is to 
ship it together with the project which uses it (they actively 
discourage system installation)

With a few additions, it would be a decent GSoC project (if it is run 
this year...anybody heard anything?)


Dag Sverre


From dagss at student.matnat.uio.no  Wed Feb  3 10:58:47 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Feb 2010 10:58:47 +0100
Subject: [Cython] Merging memoryview
In-Reply-To: <4B69456E.2000601@behnel.de>
References: <4B694069.4000104@student.matnat.uio.no>
	<4B69456E.2000601@behnel.de>
Message-ID: <4B6948D7.1080703@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn, 03.02.2010 10:22:
>   
>> Me and Kurt's been talking about (finally) getting the memory views 
>> merged.
>>     
>
> Could you add a sentence or two on what we are talking about here? We've
> had several discussions in the past regarding various kinds of memory views
> and array types.
>   
These are the ones implemented by Kurt last summer, and which was 
discussed in that painstakingly long discussion which ended with Robert 
being pronounced BD.

I.e.:

cdef int[:,::1] arr = some_object # acquires C-contiguous PEP 3118 view
cdef int[:,::1] other = arr # refcounted assignment without requesting a 
new view

cdef func(int[:, ::1] arr): # takes an efficient custom Cython struct
    pass

and so on. Currently it only supports some automatic buffer copying 
(Kurt can explain better exactly what) and direct access to underlying 
buffer pointers and shape/strides, i.e. syntax candy. Obviously 
efficient indexing should be added, which is relatively easy.

I am aware that full documentation in the form of a CEP is somewhat 
lacking (that is, things have shifted a bit since the CEPs were 
written), but as it is, if I were to update the CEPs I wouldn't have 
time to actually get this merged, and I'd really prefer to finish things 
first, then document it.

Rationale: Like I said last summer, my stance here is primarily "let's 
reserve a corner of the syntax for the stuff numeric users/data pushers 
need", and let things evolve out of what people actually use and request 
and implement, not out of academic discussions on the mailing list. I 
just don't have time for the latter.

(Sorry for going into defensive position immediately, but the experience 
from last summer still haunt me.)

Dag Sverre

From dagss at student.matnat.uio.no  Wed Feb  3 11:04:37 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Feb 2010 11:04:37 +0100
Subject: [Cython] Merging memoryview
In-Reply-To: <4B6948D7.1080703@student.matnat.uio.no>
References: <4B694069.4000104@student.matnat.uio.no>	<4B69456E.2000601@behnel.de>
	<4B6948D7.1080703@student.matnat.uio.no>
Message-ID: <4B694A35.3000707@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
>   
>> Dag Sverre Seljebotn, 03.02.2010 10:22:
>>   
>>     
>>> Me and Kurt's been talking about (finally) getting the memory views 
>>> merged.
>>>     
>>>       
>> Could you add a sentence or two on what we are talking about here? We've
>> had several discussions in the past regarding various kinds of memory views
>> and array types.
>>   
>>     
When it comes to arrays (which I interpret as "data container" -- what 
actually calls malloc/free), there is currently a utility class

cython.view.array

which is rather stupid, it basically links together malloc and PEP 3118. 
These have to be explicitly instantiated, and can then be assigned to a 
memory view variable just like any other object. There's no special 
treatment of this in the compiler except that it is bundled into the 
same C file rather than being in a seperate module (Cython utility code) 
-- and even that could be changed if we want to have a "Cython standard 
library" at some point.

Dag Sverre

> These are the ones implemented by Kurt last summer, and which was 
> discussed in that painstakingly long discussion which ended with Robert 
> being pronounced BD.
>
> I.e.:
>
> cdef int[:,::1] arr = some_object # acquires C-contiguous PEP 3118 view
> cdef int[:,::1] other = arr # refcounted assignment without requesting a 
> new view
>
> cdef func(int[:, ::1] arr): # takes an efficient custom Cython struct
>     pass
>
> and so on. Currently it only supports some automatic buffer copying 
> (Kurt can explain better exactly what) and direct access to underlying 
> buffer pointers and shape/strides, i.e. syntax candy. Obviously 
> efficient indexing should be added, which is relatively easy.
>
> I am aware that full documentation in the form of a CEP is somewhat 
> lacking (that is, things have shifted a bit since the CEPs were 
> written), but as it is, if I were to update the CEPs I wouldn't have 
> time to actually get this merged, and I'd really prefer to finish things 
> first, then document it.
>
> Rationale: Like I said last summer, my stance here is primarily "let's 
> reserve a corner of the syntax for the stuff numeric users/data pushers 
> need", and let things evolve out of what people actually use and request 
> and implement, not out of academic discussions on the mailing list. I 
> just don't have time for the latter.
>
> (Sorry for going into defensive position immediately, but the experience 
> from last summer still haunt me.)
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>   


From cournape at gmail.com  Wed Feb  3 11:54:48 2010
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 3 Feb 2010 19:54:48 +0900
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B6945D2.6030202@student.matnat.uio.no>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de> <4B6945D2.6030202@student.matnat.uio.no>
Message-ID: <5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>

On Wed, Feb 3, 2010 at 6:45 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:

>
> A project I've been thinking about is Cython-support for the waf build
> system

Note that numscons gives you cython support today, if you're lazy. It
gives reliable // builds, and it known to work on quite a few
platforms.

Concerning speed of compilation, I think the main culprit for
small/moderately sized cython file is the amount of boilerplate
generated by cython (e.g. an empty cython file with just a numpy
import and cimport generates a ~ 3500 lines C file). Without knowing
much about how compilers work internally, I have observed that in
general they scale badly w.r.t. code size. Cython-generated source
file show some strange gcc behaviors as well (like compilation with
-O3 being much faster than with -O0 in some cases), I have never been
able to see a trend, though

cheers,

David

From dagss at student.matnat.uio.no  Wed Feb  3 12:20:30 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Feb 2010 12:20:30 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>	<4B67FFD8.5070909@behnel.de>
	<4B6945D2.6030202@student.matnat.uio.no>
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
Message-ID: <4B695BFE.1030103@student.matnat.uio.no>

David Cournapeau wrote:
> On Wed, Feb 3, 2010 at 6:45 PM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>
>   
>> A project I've been thinking about is Cython-support for the waf build
>> system
>>     
>
> Note that numscons gives you cython support today, if you're lazy. It
> gives reliable // builds, and it known to work on quite a few
> platforms.
>   
Yes...I don't think we want to have any dependencies though (or, we 
maintain two test systems....). The thing about waf+Cython tools for waf 
is that we'd ship the thing with Cython, which is unnatural with numscons.

Bad reason perhaps...
> Concerning speed of compilation, I think the main culprit for
> small/moderately sized cython file is the amount of boilerplate
> generated by cython (e.g. an empty cython file with just a numpy
> import and cimport generates a ~ 3500 lines C file). Without knowing
> much about how compilers work internally, I have observed that in
> general they scale badly w.r.t. code size. Cython-generated source
> file show some strange gcc behaviors as well (like compilation with
> -O3 being much faster than with -O0 in some cases), I have never been
> able to see a trend, though
>   
We should have an option to split Cython-generated code into reusable 
include-files at some point too, and perhaps make use of precompiled 
headers.

Dag Sverre

From stefan_ml at behnel.de  Wed Feb  3 13:11:42 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Feb 2010 13:11:42 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B695BFE.1030103@student.matnat.uio.no>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>	<4B67FFD8.5070909@behnel.de>	<4B6945D2.6030202@student.matnat.uio.no>	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
	<4B695BFE.1030103@student.matnat.uio.no>
Message-ID: <4B6967FE.8080906@behnel.de>


Dag Sverre Seljebotn, 03.02.2010 12:20:
> We should have an option to split Cython-generated code into reusable 
> include-files at some point too, and perhaps make use of precompiled 
> headers.

Given that the largest part of code in a Cython generated C file is due to
real user code, I doubt that there is much Cython can improve by spitting
out multiple files here. If users value fast compilation higher than code
closeness (and effective inlining & friends), they can always split their
code themselves.

For most projects where build time actually matters, I bet there's a lot
more to gain from getting distutils to build multiple modules in parallel,
than from reducing the amount of code that is built in each step.

Also, Moores Law might well hit again before we get to implement such a
change anyway.

Stefan

From dagss at student.matnat.uio.no  Wed Feb  3 13:20:48 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Feb 2010 13:20:48 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B6967FE.8080906@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>	<4B67FFD8.5070909@behnel.de>	<4B6945D2.6030202@student.matnat.uio.no>	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>	<4B695BFE.1030103@student.matnat.uio.no>
	<4B6967FE.8080906@behnel.de>
Message-ID: <4B696A20.1070506@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn, 03.02.2010 12:20:
>   
>> We should have an option to split Cython-generated code into reusable 
>> include-files at some point too, and perhaps make use of precompiled 
>> headers.
>>     
>
> Given that the largest part of code in a Cython generated C file is due to
> real user code, I doubt that there is much Cython can improve by spitting
> out multiple files here. If users value fast compilation higher than code
> closeness (and effective inlining & friends), they can always split their
> code themselves.
>
> For most projects where build time actually matters, I bet there's a lot
> more to gain from getting distutils to build multiple modules in parallel,
> than from reducing the amount of code that is built in each step.
>   
I think working with distutils to achieve //-build is a lost cause and a 
waste of time. One could spend man-months and still have something 
vastly inferior to scons or waf. distutils is fundamentally flawed in 
this area (such as not having a proper build object DAG).

See pretty much any rant David has written on the subject :-)

Dag Sverre

> Also, Moores Law might well hit again before we get to implement such a
> change anyway.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>   


From stefan_ml at behnel.de  Wed Feb  3 13:51:52 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Feb 2010 13:51:52 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B696A20.1070506@student.matnat.uio.no>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>	<4B67FFD8.5070909@behnel.de>	<4B6945D2.6030202@student.matnat.uio.no>	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>	<4B695BFE.1030103@student.matnat.uio.no>	<4B6967FE.8080906@behnel.de>
	<4B696A20.1070506@student.matnat.uio.no>
Message-ID: <4B697168.3090405@behnel.de>


Dag Sverre Seljebotn, 03.02.2010 13:20:
> I think working with distutils to achieve //-build is a lost cause and a 
> waste of time. One could spend man-months and still have something 
> vastly inferior to scons or waf. distutils is fundamentally flawed in 
> this area (such as not having a proper build object DAG).

Well, I was only referring to parallel builds of extension modules. The
only missing feature I see here is that it needs to capture the C compiler
output. I doubt that it would take "man months" to add that.

That doesn't mean I'm arguing against the superiour feature set of other
build systems.

Stefan


From dagss at student.matnat.uio.no  Wed Feb  3 14:14:40 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Feb 2010 14:14:40 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B697168.3090405@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>	<4B67FFD8.5070909@behnel.de>	<4B6945D2.6030202@student.matnat.uio.no>	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>	<4B695BFE.1030103@student.matnat.uio.no>	<4B6967FE.8080906@behnel.de>	<4B696A20.1070506@student.matnat.uio.no>
	<4B697168.3090405@behnel.de>
Message-ID: <4B6976C0.9000600@student.matnat.uio.no>

Stefan Behnel wrote:
> Dag Sverre Seljebotn, 03.02.2010 13:20:
>   
>> I think working with distutils to achieve //-build is a lost cause and a 
>> waste of time. One could spend man-months and still have something 
>> vastly inferior to scons or waf. distutils is fundamentally flawed in 
>> this area (such as not having a proper build object DAG).
>>     
>
> Well, I was only referring to parallel builds of extension modules. The
> only missing feature I see here is that it needs to capture the C compiler
> output. I doubt that it would take "man months" to add that.
>
> That doesn't mean I'm arguing against the superiour feature set of other
> build systems.
>   
You're right, I forgot that we were talking about the test system only 
(and I guess that for most other uses as well such a change would be 
sufficient -- indeed Sage does parallell builds using distutils).

My own Cython modules requires building and linking in Fortran code etc. 
with the extension module, and I guess I was blinded by my own (rather 
special) case.

Dag Sverre

From cournape at gmail.com  Wed Feb  3 14:45:05 2010
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 3 Feb 2010 22:45:05 +0900
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B6967FE.8080906@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de> <4B6945D2.6030202@student.matnat.uio.no>
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
	<4B695BFE.1030103@student.matnat.uio.no> <4B6967FE.8080906@behnel.de>
Message-ID: <5b8d13221002030545o1a3c49ecm62835ae0867fa97c@mail.gmail.com>

On Wed, Feb 3, 2010 at 9:11 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Dag Sverre Seljebotn, 03.02.2010 12:20:
>> We should have an option to split Cython-generated code into reusable
>> include-files at some point too, and perhaps make use of precompiled
>> headers.
>
> Given that the largest part of code in a Cython generated C file is due to
> real user code, I doubt that there is much Cython can improve by spitting
> out multiple files here.

I don't think that's always true. Cython quickly generates files which
are about 1 Mb of content. For relatively simple code (with a couple
of python-visible functions), generated files are 5000 LOC, whereas a
hand-made file would be much smaller. I checked on one simple cython
file in my talkbox scikits, and the actual code is < 1/3 of the
generated C code.

It would be interesting to try this with test cases, but there is
definitely things in the way cython generates code that makes
compilation very slow.

> For most projects where build time actually matters, I bet there's a lot
> more to gain from getting distutils to build multiple modules in parallel,
> than from reducing the amount of code that is built in each step.

Actually, doing so in practice for python extensions is quite
difficult because of the lack of C awareness of symbols sharing (how
to access symbols between object files without polluting the resulting
shared library).

It took me maybe one full week of work to be able to do that in NumPy,
to a point where Compile/Test/Debug cycles are a couple of seconds
instead of ~ 1 minute.

David

From stefan_ml at behnel.de  Wed Feb  3 16:30:57 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Feb 2010 16:30:57 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <5b8d13221002030545o1a3c49ecm62835ae0867fa97c@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>	<4B67FFD8.5070909@behnel.de>
	<4B6945D2.6030202@student.matnat.uio.no>	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>	<4B695BFE.1030103@student.matnat.uio.no>
	<4B6967FE.8080906@behnel.de>
	<5b8d13221002030545o1a3c49ecm62835ae0867fa97c@mail.gmail.com>
Message-ID: <4B6996B1.7010001@behnel.de>


David Cournapeau, 03.02.2010 14:45:
> On Wed, Feb 3, 2010 at 9:11 PM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 03.02.2010 12:20:
>>> We should have an option to split Cython-generated code into reusable
>>> include-files at some point too, and perhaps make use of precompiled
>>> headers.
>> Given that the largest part of code in a Cython generated C file is due to
>> real user code, I doubt that there is much Cython can improve by spitting
>> out multiple files here.
> 
> I don't think that's always true. Cython quickly generates files which
> are about 1 Mb of content. For relatively simple code (with a couple
> of python-visible functions), generated files are 5000 LOC, whereas a
> hand-made file would be much smaller.

What I meant, was, that the amount of code and the compiler runtime is not
due to code that Cython could extract into header files. Cython generates a
lot of (type) special casing code that you'd never write manually, e.g. for
loops or function argument extraction. The result is that your hand-written
C-API-using code simply isn't as fast as the code that Cython generates
(except for selected cases where Cython isn't good enough yet).

I would also guess that the compile time is substantially increased by the
amount of inline functions that Cython uses for type conversions. They are
obviously used all over the place and each occurrence needs to get
optimised by the C compiler into exactly the code that is required in the
specific context. So the final amount of code that the C compiler needs to
handle is actually a lot larger than the code that you see in the C file.
Header files won't change that.

But that's obviously just guessing. If someone is willing to profile gcc
while compiling a Cython generated file, I'd be very happy to hear the result.


> I checked on one simple cython
> file in my talkbox scikits, and the actual code is < 1/3 of the
> generated C code.

I agree that there is a relatively large overhead for small Cython files. I
also agree that some of that code is redundant and could be extracted into
header files. What I doubt is that you'd get a major speed-up from
extracting that code. The C compiler would still have to handle it,
regardless of where it came from. Parsing is only a tiny part of what the C
compiler does these days. Optimisation takes several times longer.


>> For most projects where build time actually matters, I bet there's a lot
>> more to gain from getting distutils to build multiple modules in parallel,
>> than from reducing the amount of code that is built in each step.
> 
> Actually, doing so in practice for python extensions is quite
> difficult because of the lack of C awareness of symbols sharing (how
> to access symbols between object files without polluting the resulting
> shared library).
> 
> It took me maybe one full week of work to be able to do that in NumPy,
> to a point where Compile/Test/Debug cycles are a couple of seconds
> instead of ~ 1 minute.

To give some numbers for lxml: both clang and gcc compile the C source file
of the main module in less than 10 seconds without optimisations (empty
CFLAGS/OPT), clang being about two seconds ahead. That's about 160K lines
of C code, generated from 18K lines of Cython code. I do not find ten
seconds an overly unacceptable overhead to a compile-test cycle for 18K
lines of original source code. When I pass -O3, however, the compile time
jumps up to some 40 seconds, i.e. more than a factor of 4, which is quite
noticeable in comparison.

BTW, I know that this is not directly comparable to other projects that use
several separate modules. I guess there is more overhead if you have a
larger number of small Cython modules. But that should be mitigated by
running partial builds (which you'd certainly do during a compile-test cycle).

Stefan

From cournape at gmail.com  Wed Feb  3 17:16:38 2010
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 4 Feb 2010 01:16:38 +0900
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B6996B1.7010001@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de> <4B6945D2.6030202@student.matnat.uio.no>
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
	<4B695BFE.1030103@student.matnat.uio.no> <4B6967FE.8080906@behnel.de>
	<5b8d13221002030545o1a3c49ecm62835ae0867fa97c@mail.gmail.com>
	<4B6996B1.7010001@behnel.de>
Message-ID: <5b8d13221002030816t7b4ebc57l6e1e0a035be9074f@mail.gmail.com>

On Thu, Feb 4, 2010 at 12:30 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> David Cournapeau, 03.02.2010 14:45:
>> On Wed, Feb 3, 2010 at 9:11 PM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 03.02.2010 12:20:
>>>> We should have an option to split Cython-generated code into reusable
>>>> include-files at some point too, and perhaps make use of precompiled
>>>> headers.
>>> Given that the largest part of code in a Cython generated C file is due to
>>> real user code, I doubt that there is much Cython can improve by spitting
>>> out multiple files here.
>>
>> I don't think that's always true. Cython quickly generates files which
>> are about 1 Mb of content. For relatively simple code (with a couple
>> of python-visible functions), generated files are 5000 LOC, whereas a
>> hand-made file would be much smaller.
>
> What I meant, was, that the amount of code and the compiler runtime is not
> due to code that Cython could extract into header files. Cython generates a
> lot of (type) special casing code that you'd never write manually, e.g. for
> loops or function argument extraction. The result is that your hand-written
> C-API-using code simply isn't as fast as the code that Cython generates
> (except for selected cases where Cython isn't good enough yet).
>
> I would also guess that the compile time is substantially increased by the
> amount of inline functions that Cython uses for type conversions.

That was one of my first guess last time I looked at the problem, but
I don't think that's the real culprit : I tried simple things like
removing every inline, and it did not make a difference (both for -O0
and -O2).

> The C compiler would still have to handle it,
> regardless of where it came from.

Not if it is pre-compiled.

> Parsing is only a tiny part of what the C
> compiler does these days. Optimisation takes several times longer.

I don't think parsing takes time either. But I have generally observed
that for a unit of compilation of N lines taking T time, 2 * N lines
takes more, sometimes much more than 2 * T, especially for N >= few
thousand lines.

> When I pass -O3, however, the compile time
> jumps up to some 40 seconds, i.e. more than a factor of 4, which is quite
> noticeable in comparison.

Yes, one of the main advantage of numscons for numpy (and even more
for scipy) is that I can control CFLAGS easily and reliably when
working on it. It is almost one order of magnitude faster than with
distutils if you care about reliability (which often means removing a
lot of built files with distutils).

> BTW, I know that this is not directly comparable to other projects that use
> several separate modules. I guess there is more overhead if you have a
> larger number of small Cython modules. But that should be mitigated by
> running partial builds (which you'd certainly do during a compile-test cycle).

Yes // builds help a lot for large projects, for example scipy - a
fully optimized scipy build takes < 2 minutes on a beefy computer, and
~ 45 seconds for debug builds. At that point, numscons and scons
become the bottleneck.

cheers,

David

From robertwb at math.washington.edu  Wed Feb  3 19:25:33 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Feb 2010 10:25:33 -0800
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <5b8d13221002030816t7b4ebc57l6e1e0a035be9074f@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de>
	<4B6945D2.6030202@student.matnat.uio.no>
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
	<4B695BFE.1030103@student.matnat.uio.no>
	<4B6967FE.8080906@behnel.de>
	<5b8d13221002030545o1a3c49ecm62835ae0867fa97c@mail.gmail.com>
	<4B6996B1.7010001@behnel.de>
	<5b8d13221002030816t7b4ebc57l6e1e0a035be9074f@mail.gmail.com>
Message-ID: <ABDA42CF-6A1C-427B-B99A-7D6E30228580@math.washington.edu>

On Feb 3, 2010, at 8:16 AM, David Cournapeau wrote:

> On Thu, Feb 4, 2010 at 12:30 AM, Stefan Behnel <stefan_ml at behnel.de>  
> wrote:
>>
>> David Cournapeau, 03.02.2010 14:45:
>>> On Wed, Feb 3, 2010 at 9:11 PM, Stefan Behnel wrote:
>>>> Dag Sverre Seljebotn, 03.02.2010 12:20:
>>>>> We should have an option to split Cython-generated code into  
>>>>> reusable
>>>>> include-files at some point too, and perhaps make use of  
>>>>> precompiled
>>>>> headers.
>>>> Given that the largest part of code in a Cython generated C file  
>>>> is due to
>>>> real user code, I doubt that there is much Cython can improve by  
>>>> spitting
>>>> out multiple files here.
>>>
>>> I don't think that's always true. Cython quickly generates files  
>>> which
>>> are about 1 Mb of content. For relatively simple code (with a couple
>>> of python-visible functions), generated files are 5000 LOC,  
>>> whereas a
>>> hand-made file would be much smaller.
>>
>> What I meant, was, that the amount of code and the compiler runtime  
>> is not
>> due to code that Cython could extract into header files. Cython  
>> generates a
>> lot of (type) special casing code that you'd never write manually,  
>> e.g. for
>> loops or function argument extraction. The result is that your hand- 
>> written
>> C-API-using code simply isn't as fast as the code that Cython  
>> generates
>> (except for selected cases where Cython isn't good enough yet).
>>
>> I would also guess that the compile time is substantially increased  
>> by the
>> amount of inline functions that Cython uses for type conversions.
>
> That was one of my first guess last time I looked at the problem, but
> I don't think that's the real culprit : I tried simple things like
> removing every inline, and it did not make a difference (both for -O0
> and -O2).
>
>> The C compiler would still have to handle it,
>> regardless of where it came from.
>
> Not if it is pre-compiled.
>
>> Parsing is only a tiny part of what the C
>> compiler does these days. Optimisation takes several times longer.
>
> I don't think parsing takes time either. But I have generally observed
> that for a unit of compilation of N lines taking T time, 2 * N lines
> takes more, sometimes much more than 2 * T, especially for N >= few
> thousand lines.

I've heard (somewhat jokingly, but there seems to be some truth to it)  
that compilation is linear, optimization quadratic in the number of  
lines.

>> When I pass -O3, however, the compile time
>> jumps up to some 40 seconds, i.e. more than a factor of 4, which is  
>> quite
>> noticeable in comparison.
>
> Yes, one of the main advantage of numscons for numpy (and even more
> for scipy) is that I can control CFLAGS easily and reliably when
> working on it. It is almost one order of magnitude faster than with
> distutils if you care about reliability (which often means removing a
> lot of built files with distutils).
>
>> BTW, I know that this is not directly comparable to other projects  
>> that use
>> several separate modules. I guess there is more overhead if you  
>> have a
>> larger number of small Cython modules. But that should be mitigated  
>> by
>> running partial builds (which you'd certainly do during a compile- 
>> test cycle).

There is a huge amount of overhead in something like Sage, at least in  
terms of space. For example, the (longish) definition of the Element,  
Parent, SageObject, etc. classes are copied 100s of times.

This is an interesting discussion. A bit ago I wrote up http://wiki.cython.org/enhancements/distutils_preprocessing 
  which could use some comments and building upon. I think it's a bit  
mis-titled, as the application goes beyond disutils (though that could  
be an orthogonal step).

- Robert


From stefan_ml at behnel.de  Wed Feb  3 22:17:15 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Feb 2010 22:17:15 +0100
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <5b8d13221002030816t7b4ebc57l6e1e0a035be9074f@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>	<4B67DCE9.1080005@behnel.de>	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>	<4B67FFD8.5070909@behnel.de>
	<4B6945D2.6030202@student.matnat.uio.no>	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>	<4B695BFE.1030103@student.matnat.uio.no>
	<4B6967FE.8080906@behnel.de>	<5b8d13221002030545o1a3c49ecm62835ae0867fa97c@mail.gmail.com>	<4B6996B1.7010001@behnel.de>
	<5b8d13221002030816t7b4ebc57l6e1e0a035be9074f@mail.gmail.com>
Message-ID: <4B69E7DB.7030308@behnel.de>


David Cournapeau, 03.02.2010 17:16:
> On Thu, Feb 4, 2010 at 12:30 AM, Stefan Behnel wrote:
>> BTW, I know that this is not directly comparable to other projects that use
>> several separate modules. I guess there is more overhead if you have a
>> larger number of small Cython modules. But that should be mitigated by
>> running partial builds (which you'd certainly do during a compile-test cycle).

I actually really meant "partial". During development, bounded changes will
only trigger a limited subset of rebuilds, so an edit-compile-run cycle
will not suffer that much from small Cython modules being larger than
strictly required.


> Yes // builds help a lot for large projects, for example scipy - a
> fully optimized scipy build takes < 2 minutes on a beefy computer, and
> ~ 45 seconds for debug builds. At that point, numscons and scons
> become the bottleneck.

I don't care so much about complete project builds. We currently have
several (Java-)builds at work that take 1-2 hours each to complete. That
limits the amount of app deployments that we can run in our CI process, but
it limits neither development nor unit testing. Even if we only had nightly
deployments, I could still live with that. The only time when you really
need a complete build is when you want to deploy an app on a (test) server,
which is usually a couple of times a week.

Stefan


From robertwb at math.washington.edu  Wed Feb  3 22:23:36 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Feb 2010 13:23:36 -0800
Subject: [Cython] 0.12.1 Tests failing on ppc
In-Reply-To: <hk9778$3cf$1@ger.gmane.org>
References: <hk9778$3cf$1@ger.gmane.org>
Message-ID: <CCF09DFF-A8A1-4598-AF99-DABB63F489A6@math.washington.edu>

It looks like all of these have to do with char being unsigned in your  
environment. (Whether char is signed is implementation dependent...)  
We should fix/disable the tests on platforms where this is the case.

On Feb 2, 2010, at 4:52 AM, Neal Becker wrote:

> Build log is here:
> http://koji.fedoraproject.org/koji/getfile?taskID=1958725&name=build.log
>
> Errors start with:
>
> ERROR: Doctest: c_int_types_T255
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
>    raise self.failureException(self.format_failure(new.getvalue()))
> AssertionError: Failed doctest test for c_int_types_T255
>  File
> "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/ 
> c_int_types_T255.so", line
> 92, in c_int_types_T255
> ----------------------------------------------------------------------
> File "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/ 
> c_int_types_T255.so",
> line 98, in c_int_types_T255
> Failed example:
>    test_char(CHAR_MIN) == CHAR_MIN
> Exception raised:
>    Traceback (most recent call last):
>      File "/usr/lib/python2.6/doctest.py", line 1241, in __run
>        compileflags, 1) in test.globs
>      File "<doctest c_int_types_T255[1]>", line 1, in <module>
>        test_char(CHAR_MIN) == CHAR_MIN
>      File "c_int_types_T255.pyx", line 8, in  
> c_int_types_T255.test_char
> (c_int_types_T255.c:644)
>    OverflowError: can't convert negative value to char
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From craigcitro at gmail.com  Wed Feb  3 22:31:27 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Wed, 3 Feb 2010 13:31:27 -0800
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B67DCE9.1080005@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com> 
	<4B67DCE9.1080005@behnel.de>
Message-ID: <a8c4ff4c1002031331v44712228x14e7a5f3de537c22@mail.gmail.com>

> Meaning, he talked you into it and you naively agreed? ;)
>

Something like that. :)

> The tests look absolutely machine generated, BTW. :)
>

Oh, for sure. (I contemplated making my "gensym" do something more
clever, but it didn't seem like there was any reason ...) I suspect at
least chunks of the original test suite in Scheme were also
auto-generated, but they at least used the same names (f, g, etc) over
and over.

Actually, related to the discussion in another thread about the Cython
grammar: if we *did* write down a formal grammar for Cython, are there
programs out there that will generate random valid expressions in the
language? Given that we have Python to compare against (at least in
the case that it's a valid Python program), it'd be interesting to
leave something like that running on a machine for a while, see if it
spots any weird corner cases. If nothing else, it'd be nice to do
something like generate all valid parse trees (up to, say, choice of
ints, strings, lists, etc. in the leaves) of height at most N for some
small-ish N.

> Are there any special error cases in the closures code that would deserve
> special testing?
>

Not that I know of -- I just know that it's something I personally
didn't exercise at all.

> Cool. How much time can you spare on this?
>
> If you need any help, please ask (and make sure you're not 80% through with
> it before you ask ;).
>

I should have a reasonable amount of time for this, but why don't I
wait until I'm at least a bit started before making any claims. :) On
a related note, what happened to the cython IRC channel? Was it just
underpopulated enough that it disappeared?

-cc

From craigcitro at gmail.com  Wed Feb  3 22:51:55 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Wed, 3 Feb 2010 13:51:55 -0800
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B6967FE.8080906@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com> 
	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com> 
	<4B67FFD8.5070909@behnel.de> <4B6945D2.6030202@student.matnat.uio.no> 
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com> 
	<4B695BFE.1030103@student.matnat.uio.no> <4B6967FE.8080906@behnel.de>
Message-ID: <a8c4ff4c1002031351l27f11c29i332ebd672658d3ad@mail.gmail.com>

> For most projects where build time actually matters, I bet there's a lot
> more to gain from getting distutils to build multiple modules in parallel,
> than from reducing the amount of code that is built in each step.
>

FWIW, this isn't so hard to do. I implemented this for Sage, and I 2am
actually planning to try and push something upstream soon. The biggest
issue (well, other than "the state of the distutils community" :P) is
that there's currently a single method in build_ext which both (1)
decides whether or not an extension needs rebuilt, and (2) actually
rebuilds it. If you just naively call this in parallel, especially on
a rebuild, you end up spawning a whole bunch of threads which decide
not to rebuild, and end up losing the speedup from doing the work in
parallel. The obvious fix is to just split this logic; this could
definitely be done in such a way as to be backward-compatible, but it
means that any project that implements its own builder (i.e.
subclasses build_ext) can't take advantage of this without reworking
their code. (We ran into exactly this problem with numpy when I first
tried to merge this into Sage.)

-cc

From robertwb at math.washington.edu  Wed Feb  3 23:13:36 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Feb 2010 14:13:36 -0800
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <a8c4ff4c1002031351l27f11c29i332ebd672658d3ad@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de>
	<4B6945D2.6030202@student.matnat.uio.no>
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
	<4B695BFE.1030103@student.matnat.uio.no>
	<4B6967FE.8080906@behnel.de>
	<a8c4ff4c1002031351l27f11c29i332ebd672658d3ad@mail.gmail.com>
Message-ID: <3C06BAE6-A558-4044-81FA-C1C992C07C91@math.washington.edu>

On Feb 3, 2010, at 1:51 PM, Craig Citro wrote:

>> For most projects where build time actually matters, I bet there's  
>> a lot
>> more to gain from getting distutils to build multiple modules in  
>> parallel,
>> than from reducing the amount of code that is built in each step.
>>
>
> FWIW, this isn't so hard to do. I implemented this for Sage, and I 2am
> actually planning to try and push something upstream soon. The biggest
> issue (well, other than "the state of the distutils community" :P) is
> that there's currently a single method in build_ext which both (1)
> decides whether or not an extension needs rebuilt, and (2) actually
> rebuilds it. If you just naively call this in parallel, especially on
> a rebuild, you end up spawning a whole bunch of threads which decide
> not to rebuild, and end up losing the speedup from doing the work in
> parallel. The obvious fix is to just split this logic; this could
> definitely be done in such a way as to be backward-compatible, but it
> means that any project that implements its own builder (i.e.
> subclasses build_ext) can't take advantage of this without reworking
> their code. (We ran into exactly this problem with numpy when I first
> tried to merge this into Sage.)


What I think we should do is provide tools to handle the entire .pyx - 
 > .c conversion before passing off to distutile/build_ext (or any  
other build system), like we so in Sage.

- Robert


From cournape at gmail.com  Thu Feb  4 03:11:24 2010
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 4 Feb 2010 11:11:24 +0900
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <4B69E7DB.7030308@behnel.de>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de> <4B6945D2.6030202@student.matnat.uio.no>
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
	<4B695BFE.1030103@student.matnat.uio.no> <4B6967FE.8080906@behnel.de>
	<5b8d13221002030545o1a3c49ecm62835ae0867fa97c@mail.gmail.com>
	<4B6996B1.7010001@behnel.de>
	<5b8d13221002030816t7b4ebc57l6e1e0a035be9074f@mail.gmail.com>
	<4B69E7DB.7030308@behnel.de>
Message-ID: <5b8d13221002031811y621315b3g52e3ff10edae0e7@mail.gmail.com>

On Thu, Feb 4, 2010 at 6:17 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> I actually really meant "partial". During development, bounded changes will
> only trigger a limited subset of rebuilds, so an edit-compile-run cycle
> will not suffer that much from small Cython modules being larger than
> strictly required.

I think a large part of the issue is that partial builds are totally
unreliable with distutils. Numscons and source splitting (to make
partial builds as small as possible) really changed the way I work on
Numpy C code, because it changed from waiting ~ 1 minute to do testing
to a few seconds. With distutils, there are so many cases where
partial builds cause weird issues that I have given up doing anything
but full rebuilds with it.

Assuming a reliable partial build, I feel like there are several
stages, between instantaneous, a bit long but not too long to do
something else, and the long enough that you are doing something else.
I think that cython in some cases makes me going from the
instantaneous to a bit long stage. I will try to look at it more
seriously to look for pathological cases.

>> Yes // builds help a lot for large projects, for example scipy - a
>> fully optimized scipy build takes < 2 minutes on a beefy computer, and
>> ~ 45 seconds for debug builds. At that point, numscons and scons
>> become the bottleneck.
>
> I don't care so much about complete project builds.

Yes, not everybody cares about full builds - I do because I am
involved with installers/deployment issues.

Another argument for caring about full builds is regression hunting
through things like git bisect and co, where reliable partial builds
are not always possible (I know numscons has some issues in that
area).

>That
> limits the amount of app deployments that we can run in our CI process, but
> it limits neither development nor unit testing.

I think it depends on what you are doing. Going from 5 to 2 minutes
for a full build of scipy is not that useful for many people, but if
every project started this kind of speedup, you could hope building
things like EPD or pythonxy in one hour instead of whatever is needed
ATM, and that changes the things you can do w.r.t. quality testing
IMO.

cheers,

David

From cournape at gmail.com  Thu Feb  4 03:24:57 2010
From: cournape at gmail.com (David Cournapeau)
Date: Thu, 4 Feb 2010 11:24:57 +0900
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <a8c4ff4c1002031351l27f11c29i332ebd672658d3ad@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
	<4B67DCE9.1080005@behnel.de>
	<5b8d13221002020106g627fe405v57ebb92958ccfcc0@mail.gmail.com>
	<4B67FFD8.5070909@behnel.de> <4B6945D2.6030202@student.matnat.uio.no>
	<5b8d13221002030254p3f7ed3b1l17c2bcc3afdc245f@mail.gmail.com>
	<4B695BFE.1030103@student.matnat.uio.no> <4B6967FE.8080906@behnel.de>
	<a8c4ff4c1002031351l27f11c29i332ebd672658d3ad@mail.gmail.com>
Message-ID: <5b8d13221002031824x7511fb0cp3cd8c347bc2fe618@mail.gmail.com>

On Thu, Feb 4, 2010 at 6:51 AM, Craig Citro <craigcitro at gmail.com> wrote:

> FWIW, this isn't so hard to do. I implemented this for Sage, and I 2am
> actually planning to try and push something upstream soon. The biggest
> issue (well, other than "the state of the distutils community" :P) is
> that there's currently a single method in build_ext which both (1)
> decides whether or not an extension needs rebuilt, and (2) actually
> rebuilds it. If you just naively call this in parallel, especially on
> a rebuild, you end up spawning a whole bunch of threads which decide
> not to rebuild, and end up losing the speedup from doing the work in
> parallel. The obvious fix is to just split this logic; this could
> definitely be done in such a way as to be backward-compatible, but it
> means that any project that implements its own builder (i.e.
> subclasses build_ext) can't take advantage of this without reworking
> their code. (We ran into exactly this problem with numpy when I first
> tried to merge this into Sage.)

One issue with threading in general build tools is to synchronize
things if you parallelize at a fine grained level. I guess you would
not encounter this is you only parallelize at the build_ext/build_clib
level, but one constant source of issues with make -j8 is that
ar/ranlib are run before every .o is done (when you update archives,
in particular).

Another issue is controlling memory pressure on some very expensive
operations (typically c++ compilation and linking). If you try
building scipy on a 4 cores machine, you almost have to use a 64 bits
OS to get enough memory.

Both scons and waf has some relatively elaborate job scheduler (waf's
book has a nice chapter on it:
http://freehackers.org/~tnagy/wafbook/single.html#job_control).

The last issue with // builds you have to care about is tasks which
are not thread-safe and written in python. For scipy, the canonical
example is f2py; you need to execute f2py in its own process.

cheers,

David

From ndbecker2 at gmail.com  Thu Feb  4 19:29:11 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 04 Feb 2010 13:29:11 -0500
Subject: [Cython] 0.12.1 Tests failing on ppc
References: <hk9778$3cf$1@ger.gmane.org>
	<CCF09DFF-A8A1-4598-AF99-DABB63F489A6@math.washington.edu>
Message-ID: <hkf3lm$lp9$1@ger.gmane.org>

So, do you believe it is OK to release this code as is?  I'm holding up
updating Fedora.

Robert Bradshaw wrote:

> It looks like all of these have to do with char being unsigned in your
> environment. (Whether char is signed is implementation dependent...)
> We should fix/disable the tests on platforms where this is the case.
> 
> On Feb 2, 2010, at 4:52 AM, Neal Becker wrote:
> 
>> Build log is here:
>> http://koji.fedoraproject.org/koji/getfile?taskID=1958725&name=build.log
>>
>> Errors start with:
>>
>> ERROR: Doctest: c_int_types_T255
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>  File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
>>    raise self.failureException(self.format_failure(new.getvalue()))
>> AssertionError: Failed doctest test for c_int_types_T255
>>  File
>> "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/
>> c_int_types_T255.so", line
>> 92, in c_int_types_T255
>> ----------------------------------------------------------------------
>> File "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/
>> c_int_types_T255.so",
>> line 98, in c_int_types_T255
>> Failed example:
>>    test_char(CHAR_MIN) == CHAR_MIN
>> Exception raised:
>>    Traceback (most recent call last):
>>      File "/usr/lib/python2.6/doctest.py", line 1241, in __run
>>        compileflags, 1) in test.globs
>>      File "<doctest c_int_types_T255[1]>", line 1, in <module>
>>        test_char(CHAR_MIN) == CHAR_MIN
>>      File "c_int_types_T255.pyx", line 8, in
>> c_int_types_T255.test_char
>> (c_int_types_T255.c:644)
>>    OverflowError: can't convert negative value to char
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev



From robertwb at math.washington.edu  Fri Feb  5 00:27:46 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Feb 2010 15:27:46 -0800
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B68A0A5.6000908@noaa.gov>
References: <4B67F457.3020107@student.matnat.uio.no>
	<4B68170F.5040304@behnel.de>
	<4B681DF3.7070802@student.matnat.uio.no>
	<4B681E8B.8020305@student.matnat.uio.no>
	<4B68A0A5.6000908@noaa.gov>
Message-ID: <92964A85-DC69-4B42-AE2D-1BDA82081764@math.washington.edu>

On Feb 2, 2010, at 2:01 PM, Christopher Barker wrote:

> Dag Sverre Seljebotn wrote:
>> The alternative then appears to be
>>
>> from cython.c.math cimport sqrt
>> from cython.python cimport Py_INCREF
>>
>> but that's awfully verbose.
>
> I LIKE verbose imports -- that's what "from" is for. The import can be
> verbose (and clear, and unlikely to name clash), while the rest of  
> your
> code is as terse as you like.
>
> If this is all cython stuff, it should live in the cython namespace.


I like clib or libc (with or without the cython prefix) better than  
just plain c. There's also the consideration question of supporting  
pure mode, currently cython.* is all directives/other builtins, this  
conflates the idea a bit. In the C++ branch we now have  
cython.operator (where dereference, preincrement, etc. all live).

Eventually, for math, it would be nice if Cython called the  
corresponding C operators directly when acting on C types. (Dag has  
brought this up before.) One caveat is that often Python floats have  
better error handling/corner case handling than clib, and also  
many .pxds we would like to ship are not necessarily basic Python  
modules.

- Robert


From robertwb at math.washington.edu  Fri Feb  5 00:40:17 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Feb 2010 15:40:17 -0800
Subject: [Cython] Cython 0.13 and cython-closures branch
In-Reply-To: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
References: <a8c4ff4c1002011627y1aad7002v4b664a24ec5869f2@mail.gmail.com>
Message-ID: <B929B6A7-DB3E-4B3E-9FC3-223C1D8B1B2A@math.washington.edu>

On Feb 1, 2010, at 4:27 PM, Craig Citro wrote:

> Hi all,
>
> So this has already been discussed in several other threads, but I
> think the current plan is: once Robert releases 0.12.1, we'll pull all
> the changes from there into the cython-closures branch, and turn
> cython-closures into cython-devel for 0.13. I was chatting with
> Robert, and I think I'm happy to be the release manager for 0.13,
> unless there are any objections ...

I pulled gsoc-danilo into cython-devel. The merge was refreshingly  
easy. (Well, I had pulled -devel in not that long ago.) I expect, as  
the closures branch is relatively up-to-date that there won't be any  
merge conflicts with that either, and then Kurt and Dag's memoryview.  
I've given Craig push access to cython-devel.

> A few quick notes:
>
> - I just finished (taking forever to) add a big battery of tests in
> the cython-closures branch. I ended up adding a wee bit of code, and
> then working out a few reference counting bugs that we hit with
> closures. I've committed all the tests into the closures branch (as a
> series of 5 files; gcc and g++ were choking with everything in one
> file). The tests are actually an old battery of tests in Scheme from
> the compilers courses I took in undergrad -- I wrote a little compiler
> that converted these to Python (modulo one issue I punted on due to a
> difference with the way Scheme handles scoping for letrec). I verified
> that Scheme, Python, and Cython gave the same results for each of
> these, and the refnanny didn't complain at all, so I feel pretty
> confident in the closures code. (Big thanks to the professors who
> first came up with the battery of tests, and yes, I did check with
> them before committing the tests.)

Thanks.

> - In the process of doing this, the refnanny complained a few times
> -- so I wrote myself some notes on how reference counting should be
> working. I realized these might be useful in general, so I've added a
> little extra text and posted it on the wiki:
>
>  http://wiki.cython.org/refcounting

Looks good.

> - There is one place where I really didn't test out the closures code
> much, especially with an eye towards memory leaks/reference counting
> troubles, and that's error handling. If you have some code that uses
> nested functions and a bunch of error handling, I'd love to either (1)
> hear what happens when you run it with the refnanny turned on, or (2)
> have you send me a copy/post it somewhere so we can give it a go.

Of course, more tests are always good, but given that nothing we did  
actually influences exception handling, I'm hopeful the old mechanisms  
will take care of all of this just fine. (I'm not just saying this off  
the cuff, I did look at it a year ago.)

> This is good -- I think the next step is to start working on the
> generators implementation that came up in some other threads, and is
> outlined on the wiki. I'm planning on starting on that soon, if no one
> beats me to it -- and if my six-month old daughter decides to start
> sleeping through the night, I might actually move faster than molasses
> in January. ;)


That would be great!

- Robert

From robertwb at math.washington.edu  Fri Feb  5 00:50:59 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Feb 2010 15:50:59 -0800
Subject: [Cython] Merging memoryview
In-Reply-To: <4B694069.4000104@student.matnat.uio.no>
References: <4B694069.4000104@student.matnat.uio.no>
Message-ID: <7F192D0A-73A8-4E02-896D-88D810B4EA1F@math.washington.edu>

On Feb 3, 2010, at 1:22 AM, Dag Sverre Seljebotn wrote:

> Me and Kurt's been talking about (finally) getting the memory views
> merged. Initially I held back because I wanted to do my part of the  
> job
> first (support indexing, currently they only support raw buffer access
> and copying), but in the light of how long that's been taking me it's
> better to get things merged now -- especially as Kurt has a use for  
> the
> existing functionality in fwrap.

I think we should probably be merging this stuff, but just to confirm  
what I'm reading below, it's all new stuff (not changing or breaking  
what's there, right)?

> Question first:
> - Should the memoryview namespace be named "cython.view",
> "cython.memview", "cython.memoryview", "cython.mview",  
> "cython.memory",
> "cython.buffer"?

When will the user be seeing this namespace? Is it all hidden in the  
int[:] syntax (in which case it shouldn't matter much)? Are there a  
plethera of functions, or do you just need a place to stick a single  
class (in which case the namespace might be unneeded, though there's  
still the question of naming the class).

> The functionality should be considered experimental until Cython 0.14,

Same with C++, though I know I won't have the time to clean it all up  
by a specific date or revision.

> but it shouldn't interfer as long as you don't use the above namespace
> or the int[:]-syntax.
>
> The gsoc-kurt branch has had the changes from cython-devel merged into
> it and there's no real conflict, but some stuff left to do. The way
> things are looking I suggest that:
>
> a) The closure branch gets to merge first
>
> b) We try to keep gsoc-kurt updated and merge it afterwards. Kurt  
> should
> hopefully be able to plan on gsoc-kurt being merged in time for 0.13  
> and
> use it for fwrap development.
>
> Open tasks before we can merge gsoc-kurt:
> - Types created by Cython output in wrong order (again!). This is
> getting silly, and somebody, meaning me I guess, should code up a DAG
> for outputting type declarations in their right order. See
> http://trac.cython.org/cython_trac/ticket/469.
> - memview_declarations fail due to "invalid use of follow specifier".
> Kurt, could you have a look?

If for whatever reason these don't look feasible by 0.13, let's not  
hold up the release for it.

> Open tasks we can look at after the merge:
> - Consistent and better naming (memview vs. view vs. memoryview vs.
> mview used in the source)
> - More memory efficient dynamic struct generation
> - Item indexing
> - Transform NumPy accesses to memoryview (probably not until 0.13.1 at
> least).

Sounds good.

- Robert


From robertwb at math.washington.edu  Fri Feb  5 00:55:07 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Feb 2010 15:55:07 -0800
Subject: [Cython] Google Summer of Code
Message-ID: <AD811C4C-3F00-48C2-8FA4-4D98ED340236@math.washington.edu>

On Feb 3, 2010, at 1:45 AM, Dag Sverre Seljebotn wrote:

> [...]
>
> With a few additions, it would be a decent GSoC project (if it is run
> this year...anybody heard anything?)

Yes, it's going to be on again this year. I propose we sit under the  
Python umbrella again and see what students show interest.

- Robert


From robertwb at math.washington.edu  Fri Feb  5 04:32:01 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Feb 2010 19:32:01 -0800
Subject: [Cython] Google Summer of Code
Message-ID: <6210C1F6-D562-47E0-A646-3FAFAD785047@math.washington.edu>

On Feb 3, 2010, at 1:45 AM, Dag Sverre Seljebotn wrote:

> [...]
>
> With a few additions, it would be a decent GSoC project (if it is run
> this year...anybody heard anything?)

Yes, it's going to be on again this year. I propose we sit under the  
Python umbrella again and see what students show interest.

- Robert


From dagss at student.matnat.uio.no  Fri Feb  5 11:52:21 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 05 Feb 2010 11:52:21 +0100
Subject: [Cython] Merging memoryview
In-Reply-To: <7F192D0A-73A8-4E02-896D-88D810B4EA1F@math.washington.edu>
References: <4B694069.4000104@student.matnat.uio.no>
	<7F192D0A-73A8-4E02-896D-88D810B4EA1F@math.washington.edu>
Message-ID: <4B6BF865.3000808@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 3, 2010, at 1:22 AM, Dag Sverre Seljebotn wrote:
>
>   
>> Me and Kurt's been talking about (finally) getting the memory views
>> merged. Initially I held back because I wanted to do my part of the  
>> job
>> first (support indexing, currently they only support raw buffer access
>> and copying), but in the light of how long that's been taking me it's
>> better to get things merged now -- especially as Kurt has a use for  
>> the
>> existing functionality in fwrap.
>>     
>
> I think we should probably be merging this stuff, but just to confirm  
> what I'm reading below, it's all new stuff (not changing or breaking  
> what's there, right)?
>   
Yep, new stuff (so people shouldn't notice it being there, modulo any bugs).

>> Question first:
>> - Should the memoryview namespace be named "cython.view",
>> "cython.memview", "cython.memoryview", "cython.mview",  
>> "cython.memory",
>> "cython.buffer"?
>>     
>
> When will the user be seeing this namespace? Is it all hidden in the  
> int[:] syntax (in which case it shouldn't matter much)? Are there a  
> plethera of functions, or do you just need a place to stick a single  
> class (in which case the namespace might be unneeded, though there's  
> still the question of naming the class).
>   
1) The main usecase is for the stride specifiers:

cdef int[::cython.mview.indirect & cython.mview.contig]

for a contiguous array of pointers to single integers, for instance.

2) The cython.mview.array type. But in time I expect that will be 
enhanced more and can then live in the plain cython scope, so this 
doesn't affect naming.

3) Some things from that namespace is used in testcases, but that's not 
important.

>   
>> but it shouldn't interfer as long as you don't use the above namespace
>> or the int[:]-syntax.
>>
>> The gsoc-kurt branch has had the changes from cython-devel merged into
>> it and there's no real conflict, but some stuff left to do. The way
>> things are looking I suggest that:
>>
>> a) The closure branch gets to merge first
>>
>> b) We try to keep gsoc-kurt updated and merge it afterwards. Kurt  
>> should
>> hopefully be able to plan on gsoc-kurt being merged in time for 0.13  
>> and
>> use it for fwrap development.
>>
>> Open tasks before we can merge gsoc-kurt:
>> - Types created by Cython output in wrong order (again!). This is
>> getting silly, and somebody, meaning me I guess, should code up a DAG
>> for outputting type declarations in their right order. See
>> http://trac.cython.org/cython_trac/ticket/469.
>> - memview_declarations fail due to "invalid use of follow specifier".
>> Kurt, could you have a look?
>>     
>
> If for whatever reason these don't look feasible by 0.13, let's not  
> hold up the release for it.
>   
Sure.

Dag Sverre

From dagss at student.matnat.uio.no  Fri Feb  5 12:00:25 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 05 Feb 2010 12:00:25 +0100
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <92964A85-DC69-4B42-AE2D-1BDA82081764@math.washington.edu>
References: <4B67F457.3020107@student.matnat.uio.no>	<4B68170F.5040304@behnel.de>	<4B681DF3.7070802@student.matnat.uio.no>	<4B681E8B.8020305@student.matnat.uio.no>	<4B68A0A5.6000908@noaa.gov>
	<92964A85-DC69-4B42-AE2D-1BDA82081764@math.washington.edu>
Message-ID: <4B6BFA49.70408@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 2, 2010, at 2:01 PM, Christopher Barker wrote:
>
>   
>> Dag Sverre Seljebotn wrote:
>>     
>>> The alternative then appears to be
>>>
>>> from cython.c.math cimport sqrt
>>> from cython.python cimport Py_INCREF
>>>
>>> but that's awfully verbose.
>>>       
>> I LIKE verbose imports -- that's what "from" is for. The import can be
>> verbose (and clear, and unlikely to name clash), while the rest of  
>> your
>> code is as terse as you like.
>>
>> If this is all cython stuff, it should live in the cython namespace.
>>     
>
>
> I like clib or libc (with or without the cython prefix) better than  
> just plain c. There's also the consideration question of supporting  
> pure mode, currently cython.* is all directives/other builtins, this  
> conflates the idea a bit. In the C++ branch we now have  
> cython.operator (where dereference, preincrement, etc. all live).
>   
I'm rather against the cython prefix...I don't feel it is logical that 
"library stuff" that isn't really implemented in Cython itself lives 
there. It is a place for "Cython builtins" to me.

There's no difference in principle between, say, stdlib.h and opengl.h.

How about just "libc" and "libcpp" then?


Dag Sverre
> Eventually, for math, it would be nice if Cython called the  
> corresponding C operators directly when acting on C types. (Dag has  
> brought this up before.) One caveat is that often Python floats have  
> better error handling/corner case handling than clib, and also  
> many .pxds we would like to ship are not necessarily basic Python  
> modules.
>   

Dag Sverre

From robertwb at math.washington.edu  Fri Feb  5 12:03:59 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 03:03:59 -0800
Subject: [Cython] 0.12.1 Tests failing on ppc
In-Reply-To: <hkf3lm$lp9$1@ger.gmane.org>
References: <hk9778$3cf$1@ger.gmane.org>
	<CCF09DFF-A8A1-4598-AF99-DABB63F489A6@math.washington.edu>
	<hkf3lm$lp9$1@ger.gmane.org>
Message-ID: <6AB7AA36-111B-4DA0-944A-980192DACB44@math.washington.edu>

On Feb 4, 2010, at 10:29 AM, Neal Becker wrote:

> So, do you believe it is OK to release this code as is?  I'm holding  
> up
> updating Fedora.

Yes, you should be fine releasing. We're erroneously testing something  
that's platform dependent here.

- Robert


>
> Robert Bradshaw wrote:
>
>> It looks like all of these have to do with char being unsigned in  
>> your
>> environment. (Whether char is signed is implementation dependent...)
>> We should fix/disable the tests on platforms where this is the case.
>>
>> On Feb 2, 2010, at 4:52 AM, Neal Becker wrote:
>>
>>> Build log is here:
>>> http://koji.fedoraproject.org/koji/getfile?taskID=1958725&name=build.log
>>>
>>> Errors start with:
>>>
>>> ERROR: Doctest: c_int_types_T255
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
>>>   raise self.failureException(self.format_failure(new.getvalue()))
>>> AssertionError: Failed doctest test for c_int_types_T255
>>> File
>>> "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/
>>> c_int_types_T255.so", line
>>> 92, in c_int_types_T255
>>> ----------------------------------------------------------------------
>>> File "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/
>>> c_int_types_T255.so",
>>> line 98, in c_int_types_T255
>>> Failed example:
>>>   test_char(CHAR_MIN) == CHAR_MIN
>>> Exception raised:
>>>   Traceback (most recent call last):
>>>     File "/usr/lib/python2.6/doctest.py", line 1241, in __run
>>>       compileflags, 1) in test.globs
>>>     File "<doctest c_int_types_T255[1]>", line 1, in <module>
>>>       test_char(CHAR_MIN) == CHAR_MIN
>>>     File "c_int_types_T255.pyx", line 8, in
>>> c_int_types_T255.test_char
>>> (c_int_types_T255.c:644)
>>>   OverflowError: can't convert negative value to char
>>>
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri Feb  5 12:16:19 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 03:16:19 -0800
Subject: [Cython] Merging memoryview
In-Reply-To: <4B6BF865.3000808@student.matnat.uio.no>
References: <4B694069.4000104@student.matnat.uio.no>
	<7F192D0A-73A8-4E02-896D-88D810B4EA1F@math.washington.edu>
	<4B6BF865.3000808@student.matnat.uio.no>
Message-ID: <3914D9C2-7DCF-46E7-959E-F13E0273A965@math.washington.edu>

On Feb 5, 2010, at 2:52 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 3, 2010, at 1:22 AM, Dag Sverre Seljebotn wrote:
>>
>>
>>> Me and Kurt's been talking about (finally) getting the memory views
>>> merged. Initially I held back because I wanted to do my part of the
>>> job
>>> first (support indexing, currently they only support raw buffer  
>>> access
>>> and copying), but in the light of how long that's been taking me  
>>> it's
>>> better to get things merged now -- especially as Kurt has a use for
>>> the
>>> existing functionality in fwrap.
>>>
>>
>> I think we should probably be merging this stuff, but just to confirm
>> what I'm reading below, it's all new stuff (not changing or breaking
>> what's there, right)?
>>
> Yep, new stuff (so people shouldn't notice it being there, modulo  
> any bugs).

Does this mean you've re-worked a lot of the buffer code under the  
hood then? (At least we have decent tests, but how confident are you  
that there aren't new bugs for existing code?)

>>> Question first:
>>> - Should the memoryview namespace be named "cython.view",
>>> "cython.memview", "cython.memoryview", "cython.mview",
>>> "cython.memory",
>>> "cython.buffer"?
>>>
>>
>> When will the user be seeing this namespace? Is it all hidden in the
>> int[:] syntax (in which case it shouldn't matter much)? Are there a
>> plethera of functions, or do you just need a place to stick a single
>> class (in which case the namespace might be unneeded, though there's
>> still the question of naming the class).
>>
> 1) The main usecase is for the stride specifiers:
>
> cdef int[::cython.mview.indirect & cython.mview.contig]
>
> for a contiguous array of pointers to single integers, for instance.

I really hope there's eventually a shorthand for this simple case  
someday, though of course you're working on something far more generic.

> 2) The cython.mview.array type. But in time I expect that will be
> enhanced more and can then live in the plain cython scope, so this
> doesn't affect naming.
>
> 3) Some things from that namespace is used in testcases, but that's  
> not
> important.

Lets go for the full "cython.memoryview" as it seems these concepts  
are similar enough to what Python's memoryviews will be offering.  
People who shorten numpy to np are certainly going to shorten this one  
too. FWIW, I implemented the ability to use cython.X namespaces for  
directives in the C++ branch.

- Robert


From robertwb at math.washington.edu  Fri Feb  5 12:07:49 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 03:07:49 -0800
Subject: [Cython] cython wrapping C++ template
In-Reply-To: <200911111705297537158@gmail.com>
References: <200911111705297537158@gmail.com>
Message-ID: <21B77197-C297-48BF-9582-79224708F3D8@math.washington.edu>

On Nov 11, 2009, at 3:05 PM, J. Jankins wrote:

> <spacer.gif>
>
>
> Hello,
>
> I am trying to use C++ STL, specifically container <vector> and  
> <set>. But I just find a little discription in <Cython Manual>.

The current version of Cython barely supports C++, and as there was a  
GSoC project to improve this (which will get merged in the next  
release) it hasn't been a priority to document the hackish way of  
doing things. (Unfortunately, we still will be short of full STL  
support, but we'll be a lot closer.)

> And I couldn't even to get the example to work.
>
> Here is what I do:
> 1. <stl.h>:
> #include <vector>
>
> 2. <pstl.pyx>:
>    cdef extern from "stl.h":
>        ctypedef struct intvec "std::vector<int>":
>            void (* push_back) (int elem)
>        intvec intvec_factory "std::vector<int>" (int len)
>
>    def main():
>        cdef intvec v = intvec_factory(2)
>        v.push_back(3)
>        print 'Hello'
>
> 3.<setup.py>:
>  from distutils.core import setup
>  from distutils.extension import Extension
>  from Cython.Distutils import build_ext
>
>  setup(
>    name = 'Demos',
>    ext_modules=[
>      Extension("test",
>                sources=["pstl.pyx"],
>                include_dirs=["."],
>                language="c++"),
>      ],
>    cmdclass = {'build_ext': build_ext},
> )
>
> 4. python setup.py build_ext --inplace
>
> When I import "pstl", there is an  ImportError.

You may need to add

     libraries=["stdc++"]

to be able to import.

> I am totally frustrated. And I commit that: I didn't finish reading  
> Cython Manual. It's not that easy to understand and it has too few  
> examples.

I agree, there's lots of room for improvement here. We welcome  
contributions!

- Robert



From robertwb at math.washington.edu  Fri Feb  5 12:18:27 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 03:18:27 -0800
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <4B6BFA49.70408@student.matnat.uio.no>
References: <4B67F457.3020107@student.matnat.uio.no>	<4B68170F.5040304@behnel.de>	<4B681DF3.7070802@student.matnat.uio.no>	<4B681E8B.8020305@student.matnat.uio.no>	<4B68A0A5.6000908@noaa.gov>
	<92964A85-DC69-4B42-AE2D-1BDA82081764@math.washington.edu>
	<4B6BFA49.70408@student.matnat.uio.no>
Message-ID: <3A27302D-155D-4006-BD37-EE5B3C1DAE9E@math.washington.edu>

On Feb 5, 2010, at 3:00 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 2, 2010, at 2:01 PM, Christopher Barker wrote:
>>
>>
>>> Dag Sverre Seljebotn wrote:
>>>
>>>> The alternative then appears to be
>>>>
>>>> from cython.c.math cimport sqrt
>>>> from cython.python cimport Py_INCREF
>>>>
>>>> but that's awfully verbose.
>>>>
>>> I LIKE verbose imports -- that's what "from" is for. The import  
>>> can be
>>> verbose (and clear, and unlikely to name clash), while the rest of
>>> your
>>> code is as terse as you like.
>>>
>>> If this is all cython stuff, it should live in the cython namespace.
>>>
>>
>>
>> I like clib or libc (with or without the cython prefix) better than
>> just plain c. There's also the consideration question of supporting
>> pure mode, currently cython.* is all directives/other builtins, this
>> conflates the idea a bit. In the C++ branch we now have
>> cython.operator (where dereference, preincrement, etc. all live).
>>
> I'm rather against the cython prefix...I don't feel it is logical that
> "library stuff" that isn't really implemented in Cython itself lives
> there. It is a place for "Cython builtins" to me.

Yeah, conflating the two doesn't seem natural.

> There's no difference in principle between, say, stdlib.h and  
> opengl.h.
>
> How about just "libc" and "libcpp" then?


Sure. And we should move all the Python ones to "python."

- Robert


From dagss at student.matnat.uio.no  Fri Feb  5 13:40:44 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 05 Feb 2010 13:40:44 +0100
Subject: [Cython] Merging memoryview
In-Reply-To: <3914D9C2-7DCF-46E7-959E-F13E0273A965@math.washington.edu>
References: <4B694069.4000104@student.matnat.uio.no>	<7F192D0A-73A8-4E02-896D-88D810B4EA1F@math.washington.edu>	<4B6BF865.3000808@student.matnat.uio.no>
	<3914D9C2-7DCF-46E7-959E-F13E0273A965@math.washington.edu>
Message-ID: <4B6C11CC.4040701@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Feb 5, 2010, at 2:52 AM, Dag Sverre Seljebotn wrote:
>
>   
>> Robert Bradshaw wrote:
>>     
>>> On Feb 3, 2010, at 1:22 AM, Dag Sverre Seljebotn wrote:
>>>
>>>
>>>       
>>>> Me and Kurt's been talking about (finally) getting the memory views
>>>> merged. Initially I held back because I wanted to do my part of the
>>>> job
>>>> first (support indexing, currently they only support raw buffer  
>>>> access
>>>> and copying), but in the light of how long that's been taking me  
>>>> it's
>>>> better to get things merged now -- especially as Kurt has a use for
>>>> the
>>>> existing functionality in fwrap.
>>>>
>>>>         
>>> I think we should probably be merging this stuff, but just to confirm
>>> what I'm reading below, it's all new stuff (not changing or breaking
>>> what's there, right)?
>>>
>>>       
>> Yep, new stuff (so people shouldn't notice it being there, modulo  
>> any bugs).
>>     
>
> Does this mean you've re-worked a lot of the buffer code under the  
> hood then? (At least we have decent tests, but how confident are you  
> that there aren't new bugs for existing code?)
>   
No (so there's some degree of overlap and two parallell implementations 
now -- I expect this to overlap to gradually grow, and then disappear as 
the new buffer implementation is stabilized and we just transform the 
old buffer cases to the new one in a transform).

The most likely place for bugs is the parser; i.e. we had to muck with

cdef extern from *:
    cdef int foo(int[])

vs.

cdef extern from *:
    cdef int foo(int[:])

(or something like it -- don't remember whether we actually decided to 
support the latter, or if we require a dummy variable name). That stuff 
has caused trouble earlier. But we haven't found any problems with it so 
far and the test suite works OK. I honestly don't expect anything of the 
sort that the testcases wouldn't have picked up already.

Dag Sverre

From bpederse at gmail.com  Fri Feb  5 16:28:09 2010
From: bpederse at gmail.com (Brent Pedersen)
Date: Fri, 5 Feb 2010 07:28:09 -0800
Subject: [Cython] cython wrapping C++ template
In-Reply-To: <200911111705297537158@gmail.com>
References: <200911111705297537158@gmail.com>
Message-ID: <e183a99d1002050728k1772c035vfcca8cc70cb76c4@mail.gmail.com>

On Wed, Nov 11, 2009 at 3:05 PM, J. Jankins <andyjian430074 at gmail.com> wrote:
>
>
> Hello,
>
> I am trying to use C++ STL, specifically container <vector> and <set>. But I just find?a little discription in <Cython Manual>. And I couldn't even to get the example to work.
>
> Here is what I do:
> 1. <stl.h>:
> #include <vector>
>
> 2. <pstl.pyx>:
> ???cdef?extern?from?"stl.h":
> ???????ctypedef?struct?intvec?"std::vector<int>":
> ???????????void?(*?push_back)?(int?elem)
> ???????intvec?intvec_factory?"std::vector<int>"?(int?len)
>
> ???def?main():
> ???????cdef?intvec?v?=?intvec_factory(2)
> ???????v.push_back(3)
> ???????print?'Hello'
>
> 3.<setup.py>:
> ?from?distutils.core?import?setup
> ?from?distutils.extension?import?Extension
> ?from?Cython.Distutils?import?build_ext
>
> ?setup(
> ???name?=?'Demos',
> ???ext_modules=[
> ?????Extension("test",
> ???????????????sources=["pstl.pyx"],
> ???????????????include_dirs=["."],
> ???????????????language="c++"),
> ?????],
> ???cmdclass?=?{'build_ext':?build_ext},
> )


i think if you change "test" to "pstl" your example will work.
-brent

>
>
> 4. python?setup.py?build_ext?--inplace
>
> When I?import "pstl", there is an ?ImportError.
>
> I am totally frustrated. And I commit that: I didn't finish reading Cython Manual. It's not that easy to understand and it has too few examples.
>
> Thanks.
>
> Jankins
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From dalcinl at gmail.com  Fri Feb  5 16:36:29 2010
From: dalcinl at gmail.com (=?UTF-8?Q?Lisandro_Dalc=C3=ADn?=)
Date: Fri, 5 Feb 2010 12:36:29 -0300
Subject: [Cython] 0.12.1 Tests failing on ppc
In-Reply-To: <6AB7AA36-111B-4DA0-944A-980192DACB44@math.washington.edu>
References: <hk9778$3cf$1@ger.gmane.org>
	<CCF09DFF-A8A1-4598-AF99-DABB63F489A6@math.washington.edu>
	<hkf3lm$lp9$1@ger.gmane.org>
	<6AB7AA36-111B-4DA0-944A-980192DACB44@math.washington.edu>
Message-ID: <e7ba66e41002050736m40330dccr94316a4e3a200ad2@mail.gmail.com>

On Fri, Feb 5, 2010 at 8:03 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 4, 2010, at 10:29 AM, Neal Becker wrote:
>
>> So, do you believe it is OK to release this code as is? ?I'm holding
>> up
>> updating Fedora.
>
> Yes, you should be fine releasing. We're erroneously testing something
> that's platform dependent here.
>

Mmm.. I should check if Cython (at PyrexTypes.py) does not make some
assumption about sign of char...

>
>>
>> Robert Bradshaw wrote:
>>
>>> It looks like all of these have to do with char being unsigned in
>>> your
>>> environment. (Whether char is signed is implementation dependent...)
>>> We should fix/disable the tests on platforms where this is the case.
>>>
>>> On Feb 2, 2010, at 4:52 AM, Neal Becker wrote:
>>>
>>>> Build log is here:
>>>> http://koji.fedoraproject.org/koji/getfile?taskID=1958725&name=build.log
>>>>
>>>> Errors start with:
>>>>
>>>> ERROR: Doctest: c_int_types_T255
>>>> ----------------------------------------------------------------------
>>>> Traceback (most recent call last):
>>>> File "/usr/lib/python2.6/doctest.py", line 2145, in runTest
>>>> ? raise self.failureException(self.format_failure(new.getvalue()))
>>>> AssertionError: Failed doctest test for c_int_types_T255
>>>> File
>>>> "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/
>>>> c_int_types_T255.so", line
>>>> 92, in c_int_types_T255
>>>> ----------------------------------------------------------------------
>>>> File "/builddir/build/BUILD/Cython-0.12.1/BUILD/run/c/
>>>> c_int_types_T255.so",
>>>> line 98, in c_int_types_T255
>>>> Failed example:
>>>> ? test_char(CHAR_MIN) == CHAR_MIN
>>>> Exception raised:
>>>> ? Traceback (most recent call last):
>>>> ? ? File "/usr/lib/python2.6/doctest.py", line 1241, in __run
>>>> ? ? ? compileflags, 1) in test.globs
>>>> ? ? File "<doctest c_int_types_T255[1]>", line 1, in <module>
>>>> ? ? ? test_char(CHAR_MIN) == CHAR_MIN
>>>> ? ? File "c_int_types_T255.pyx", line 8, in
>>>> c_int_types_T255.test_char
>>>> (c_int_types_T255.c:644)
>>>> ? OverflowError: can't convert negative value to char
>>>>
>>>> _______________________________________________
>>>> Cython-dev mailing list
>>>> Cython-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri Feb  5 16:46:09 2010
From: dalcinl at gmail.com (=?UTF-8?Q?Lisandro_Dalc=C3=ADn?=)
Date: Fri, 5 Feb 2010 12:46:09 -0300
Subject: [Cython] cython wrapping C++ template
In-Reply-To: <200911111705297537158@gmail.com>
References: <200911111705297537158@gmail.com>
Message-ID: <e7ba66e41002050746j50c1d29br1d3f33752b72f274@mail.gmail.com>

On Wed, Nov 11, 2009 at 8:05 PM, J. Jankins <andyjian430074 at gmail.com>wrote:

>
> Hello,
>
> I am trying to use C++ STL, specifically container <vector> and <set>. But
> I just find a little discription in <Cython Manual>. And I couldn't even to
> get the example to work.
>
> Here is what I do:
> 1. <stl.h>:
> #include <vector>
>
> 2. <pstl.pyx>:
>     cdef extern from "stl.h":
>        ctypedef struct intvec "std::vector<int>":
>            void (* push_back) (int elem)
>        intvec intvec_factory "std::vector<int>" (int len)
>
>    def main():
>        cdef intvec v = intvec_factory(2)
>        v.push_back(3)
>        print 'Hello'
>
> 3.<setup.py>:
>   from distutils.core import setup
>  from distutils.extension import Extension
>  from Cython.Distutils import build_ext
>
>  setup(
>    name = 'Demos',
>    ext_modules=[
>      Extension("test",
>                sources=["pstl.pyx"],
>                include_dirs=["."],
>                language="c++"),
>      ],
>    cmdclass = {'build_ext': build_ext},
> )
>
> 4. python setup.py build_ext --inplace
>
> When I import "pstl", there is an  ImportError.
>
> I am totally frustrated. And I commit that: I didn't finish reading Cython
> Manual. It's not that easy to understand and it has too few examples.
>
> Thanks.
>
> Jankins
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
This has nothing to do with Cython, but with a bug your setup.py (your
extension have to be name "pstl" instead of "test"). Please read distutils
documentation; hopefuly, it will be easier to understand and will show many
more examples compared to Cython docs.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100205/d73e2aaf/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 43 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100205/d73e2aaf/attachment.gif 

From cgohlke at uci.edu  Fri Feb  5 16:54:06 2010
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Fri, 05 Feb 2010 07:54:06 -0800
Subject: [Cython] Generate shorter string literals
Message-ID: <4B6C3F1E.5030202@uci.edu>

Dear Cython developers,

This is related to <http://trac.cython.org/cython_trac/ticket/50>.

Cython 0.12.1 still generates string literals that can be too long for MSVC*
compilers. Please consider the following patch.

diff -r 8bff3332e34f Cython/Compiler/Code.py
--- a/Cython/Compiler/Code.py   Tue Feb 02 02:10:32 2010 -0800
+++ b/Cython/Compiler/Code.py   Thu Feb 04 19:32:40 2010 -0800
@@ -667,7 +667,7 @@
         decls_writer = self.parts['decls']
         for _, cname, c in c_consts:
             decls_writer.putln('static char %s[] = "%s";' % (
-                cname, c.escaped_value))
+                cname, StringEncoding.split_docstring(c.escaped_value)))
             if c.py_strings is not None:
                 for py_string in c.py_strings.itervalues():
                     py_strings.append((c.cname, len(py_string.cname),
py_string))

Best,

Christoph Gohlke
Laboratory for Fluorescence Dynamics
University of California, Irvine
http://www.lfd.uci.edu/

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: shorter_strings.diff
Url: http://codespeak.net/pipermail/cython-dev/attachments/20100205/774d6731/attachment.diff 

From dagss at student.matnat.uio.no  Fri Feb  5 17:05:23 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 05 Feb 2010 17:05:23 +0100
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B6C3F1E.5030202@uci.edu>
References: <4B6C3F1E.5030202@uci.edu>
Message-ID: <4B6C41C3.80301@student.matnat.uio.no>

Christoph Gohlke wrote:
> Dear Cython developers,
>
> This is related to <http://trac.cython.org/cython_trac/ticket/50>.
>
> Cython 0.12.1 still generates string literals that can be too long for MSVC*
> compilers. Please consider the following patch.
>   
Thanks, excellent!

Do you have the possibility to also create a testcase so that this 
doesn't happen again? Since none of the main developers use MSVC 
regularily it is much harder for us to do so.

Just in case:

1) Create tests/compile/msvc_strings.pyx
2) Fill in the minimal things needed to get the failure with current Cython
3) python runtests.py msvc_strings

Then submit patch containing both the change and the test case.

Dag Sverre

> diff -r 8bff3332e34f Cython/Compiler/Code.py
> --- a/Cython/Compiler/Code.py   Tue Feb 02 02:10:32 2010 -0800
> +++ b/Cython/Compiler/Code.py   Thu Feb 04 19:32:40 2010 -0800
> @@ -667,7 +667,7 @@
>          decls_writer = self.parts['decls']
>          for _, cname, c in c_consts:
>              decls_writer.putln('static char %s[] = "%s";' % (
> -                cname, c.escaped_value))
> +                cname, StringEncoding.split_docstring(c.escaped_value)))
>              if c.py_strings is not None:
>                  for py_string in c.py_strings.itervalues():
>                      py_strings.append((c.cname, len(py_string.cname),
> py_string))
>
> Best,
>
> Christoph Gohlke
> Laboratory for Fluorescence Dynamics
> University of California, Irvine
> http://www.lfd.uci.edu/
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From dalcinl at gmail.com  Fri Feb  5 18:41:08 2010
From: dalcinl at gmail.com (=?UTF-8?Q?Lisandro_Dalc=C3=ADn?=)
Date: Fri, 5 Feb 2010 14:41:08 -0300
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B6C41C3.80301@student.matnat.uio.no>
References: <4B6C3F1E.5030202@uci.edu> <4B6C41C3.80301@student.matnat.uio.no>
Message-ID: <e7ba66e41002050941l3fe9d7dao7f3c0231f78dbae9@mail.gmail.com>

On Fri, Feb 5, 2010 at 1:05 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Christoph Gohlke wrote:
>> Dear Cython developers,
>>
>> This is related to <http://trac.cython.org/cython_trac/ticket/50>.
>>
>> Cython 0.12.1 still generates string literals that can be too long for MSVC*
>> compilers. Please consider the following patch.
>>
> Thanks, excellent!
>
> Do you have the possibility to also create a testcase so that this
> doesn't happen again?

I would suggest to write the testcase exercising three 'kind' of
string literals: docstrings, module global string literals, and a long
function/method name... Just in case...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From Dan.Helfman at noaa.gov  Fri Feb  5 19:04:08 2010
From: Dan.Helfman at noaa.gov (Dan Helfman)
Date: Fri, 05 Feb 2010 10:04:08 -0800
Subject: [Cython] Incorrect HTML annotations with NumPy
Message-ID: <4B6C5D98.4010406@noaa.gov>

I've encountered a problem with Cython 0.12.1 in which incorrect code 
annotations show up when HTML is generated with the "--annotate" option.

I believe this is the same issue described here:

   http://trac.cython.org/cython_trac/ticket/292

I've come up with an even simpler test case that triggers the behavior. 
The test.pyx file is attached. What happens is that the generated HTML 
for line 187 ("cdef int y = 6"), once expanded, reads as follows:

static int __pyx_pf_5numpy_7ndarray___getbuffer__(PyObject 
*__pyx_v_self, Py_buffer *__pyx_v_info, int __pyx_v_flags); 
__pyx_v_4test_y = 6;

It appears to be pulling in code from both test.pyx line 187 *and* 
numpy.pxd line 187, even though the annotation should only be for 
test.pyx. The generated code in test.c is simply "__pyx_v_4test_y = 6;" 
without the extraneous numpy.pxd __getbuffer__ code.

This problem makes Cython HTML annotation with NumPy code very difficult 
to use, because it's not immediately apparent from any given yellow line 
whether it's actually triggering the generation of NumPy code or whether 
it's merely including it in the annotation erroneously.

If this sort of thing is more appropriate on the other mailing list, 
please let me know.

Dan
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test.pyx
Url: http://codespeak.net/pipermail/cython-dev/attachments/20100205/c10ea89d/attachment.diff 

From kwmsmith at gmail.com  Fri Feb  5 19:58:53 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 5 Feb 2010 12:58:53 -0600
Subject: [Cython] Merging memoryview
In-Reply-To: <4B694069.4000104@student.matnat.uio.no>
References: <4B694069.4000104@student.matnat.uio.no>
Message-ID: <c266a7931002051058w7abcafa3h93d7cb709ef3d2c0@mail.gmail.com>

On Wed, Feb 3, 2010 at 3:22 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Me and Kurt's been talking about (finally) getting the memory views
> merged. Initially I held back because I wanted to do my part of the job
> first (support indexing, currently they only support raw buffer access
> and copying), but in the light of how long that's been taking me it's
> better to get things merged now -- especially as Kurt has a use for the
> existing functionality in fwrap.
>
> Question first:
> ?- Should the memoryview namespace be named "cython.view",
> "cython.memview", "cython.memoryview", "cython.mview", "cython.memory",
> "cython.buffer"?
>
> The functionality should be considered experimental until Cython 0.14,
> but it shouldn't interfer as long as you don't use the above namespace
> or the int[:]-syntax.

This will work well, I think -- with memoryviews implemented
experimentally for an entire release, fwrap will be able to run the
new functionality through its paces and address any bugs that come up.

>
> The gsoc-kurt branch has had the changes from cython-devel merged into
> it and there's no real conflict, but some stuff left to do. The way
> things are looking I suggest that:
>
> a) The closure branch gets to merge first
>
> b) We try to keep gsoc-kurt updated and merge it afterwards. Kurt should
> hopefully be able to plan on gsoc-kurt being merged in time for 0.13 and
> use it for fwrap development.
>
> Open tasks before we can merge gsoc-kurt:
> ?- Types created by Cython output in wrong order (again!). This is
> getting silly, and somebody, meaning me I guess, should code up a DAG
> for outputting type declarations in their right order. See
> http://trac.cython.org/cython_trac/ticket/469.
> ?- memview_declarations fail due to "invalid use of follow specifier".
> Kurt, could you have a look?

I'll take a look, likely sometime this weekend.

>
> Open tasks we can look at after the merge:
> ?- Consistent and better naming (memview vs. view vs. memoryview vs.
> mview used in the source)

Along with that:  we discussed using the ampersand as the conjunctive
for the different stride and memory access flags (I thought it was
more pythonic, although less 'C'-like), or we could also go with the
pipe (|) symbol instead, ala C flags.  If it's already decided and
behind us, then I'm fine with it.

> ?- More memory efficient dynamic struct generation
> ?- Item indexing
> ?- Transform NumPy accesses to memoryview (probably not until 0.13.1 at
> least).
>
> Dag Sverre

From robertwb at math.washington.edu  Fri Feb  5 19:56:06 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 10:56:06 -0800
Subject: [Cython] Merging memoryview
In-Reply-To: <4B6C11CC.4040701@student.matnat.uio.no>
References: <4B694069.4000104@student.matnat.uio.no>	<7F192D0A-73A8-4E02-896D-88D810B4EA1F@math.washington.edu>	<4B6BF865.3000808@student.matnat.uio.no>
	<3914D9C2-7DCF-46E7-959E-F13E0273A965@math.washington.edu>
	<4B6C11CC.4040701@student.matnat.uio.no>
Message-ID: <F0C6B446-9780-492C-952E-CB46D27FC905@math.washington.edu>

On Feb 5, 2010, at 4:40 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Feb 5, 2010, at 2:52 AM, Dag Sverre Seljebotn wrote:
>>
>>
>>> Robert Bradshaw wrote:
>>>
>>>> On Feb 3, 2010, at 1:22 AM, Dag Sverre Seljebotn wrote:
>>>>
>>>>
>>>>
>>>>> Me and Kurt's been talking about (finally) getting the memory  
>>>>> views
>>>>> merged. Initially I held back because I wanted to do my part of  
>>>>> the
>>>>> job
>>>>> first (support indexing, currently they only support raw buffer
>>>>> access
>>>>> and copying), but in the light of how long that's been taking me
>>>>> it's
>>>>> better to get things merged now -- especially as Kurt has a use  
>>>>> for
>>>>> the
>>>>> existing functionality in fwrap.
>>>>>
>>>>>
>>>> I think we should probably be merging this stuff, but just to  
>>>> confirm
>>>> what I'm reading below, it's all new stuff (not changing or  
>>>> breaking
>>>> what's there, right)?
>>>>
>>>>
>>> Yep, new stuff (so people shouldn't notice it being there, modulo
>>> any bugs).
>>>
>>
>> Does this mean you've re-worked a lot of the buffer code under the
>> hood then? (At least we have decent tests, but how confident are you
>> that there aren't new bugs for existing code?)
>>
> No (so there's some degree of overlap and two parallell  
> implementations
> now -- I expect this to overlap to gradually grow, and then  
> disappear as
> the new buffer implementation is stabilized and we just transform the
> old buffer cases to the new one in a transform).
>
> The most likely place for bugs is the parser; i.e. we had to muck with
>
> cdef extern from *:
>    cdef int foo(int[])
>
> vs.
>
> cdef extern from *:
>    cdef int foo(int[:])
>
> (or something like it -- don't remember whether we actually decided to
> support the latter, or if we require a dummy variable name). That  
> stuff
> has caused trouble earlier. But we haven't found any problems with  
> it so
> far and the test suite works OK. I honestly don't expect anything of  
> the
> sort that the testcases wouldn't have picked up already.

OK, sounds good. We had to mess with Foo[int] being a type as well. I  
differed buffer vs. type resolution 'till after the parsing phase.

- Robert


From robertwb at math.washington.edu  Fri Feb  5 20:08:24 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 11:08:24 -0800
Subject: [Cython] Merging memoryview
In-Reply-To: <c266a7931002051058w7abcafa3h93d7cb709ef3d2c0@mail.gmail.com>
References: <4B694069.4000104@student.matnat.uio.no>
	<c266a7931002051058w7abcafa3h93d7cb709ef3d2c0@mail.gmail.com>
Message-ID: <6124C1E5-4E21-4784-8A6E-9AF00EC9AECC@math.washington.edu>

On Feb 5, 2010, at 10:58 AM, Kurt Smith wrote:

> Along with that:  we discussed using the ampersand as the conjunctive
> for the different stride and memory access flags (I thought it was
> more pythonic, although less 'C'-like), or we could also go with the
> pipe (|) symbol instead, ala C flags.  If it's already decided and
> behind us, then I'm fine with it.

Personally, I think the best would be "and" a la Python. When I see |  
and &, I instinctively parse that as C flags (and the example you gave  
threw me for a loop for a sec, but I didn't want to bring it up  
myself.) Of course using | as in C will probably be unfamiliar to  
those with only Python experience, and doesn't read as well.

- Robert


From robertwb at math.washington.edu  Fri Feb  5 20:31:49 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 11:31:49 -0800
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B6C3F1E.5030202@uci.edu>
References: <4B6C3F1E.5030202@uci.edu>
Message-ID: <2E2BE831-C9B8-4CEE-8030-B40D20D84E75@math.washington.edu>

On Feb 5, 2010, at 7:54 AM, Christoph Gohlke wrote:

> Dear Cython developers,
>
> This is related to <http://trac.cython.org/cython_trac/ticket/50>.
>
> Cython 0.12.1 still generates string literals that can be too long  
> for MSVC*
> compilers. Please consider the following patch.
>
> diff -r 8bff3332e34f Cython/Compiler/Code.py
> --- a/Cython/Compiler/Code.py   Tue Feb 02 02:10:32 2010 -0800
> +++ b/Cython/Compiler/Code.py   Thu Feb 04 19:32:40 2010 -0800
> @@ -667,7 +667,7 @@
>         decls_writer = self.parts['decls']
>         for _, cname, c in c_consts:
>             decls_writer.putln('static char %s[] = "%s";' % (
> -                cname, c.escaped_value))
> +                cname,  
> StringEncoding.split_docstring(c.escaped_value)))
>             if c.py_strings is not None:
>                 for py_string in c.py_strings.itervalues():
>                     py_strings.append((c.cname, len(py_string.cname),
> py_string))

Thanks! http://hg.cython.org/cython-devel/rev/dc2a7157de42

I agree testcases would be good to have to ensure this doesn't happen  
again.

- Robert


From kwmsmith at gmail.com  Fri Feb  5 20:36:23 2010
From: kwmsmith at gmail.com (Kurt Smith)
Date: Fri, 5 Feb 2010 13:36:23 -0600
Subject: [Cython] Merging memoryview
In-Reply-To: <6124C1E5-4E21-4784-8A6E-9AF00EC9AECC@math.washington.edu>
References: <4B694069.4000104@student.matnat.uio.no>
	<c266a7931002051058w7abcafa3h93d7cb709ef3d2c0@mail.gmail.com>
	<6124C1E5-4E21-4784-8A6E-9AF00EC9AECC@math.washington.edu>
Message-ID: <c266a7931002051136h5b92b9cft671e61473ebabfbf@mail.gmail.com>

On Fri, Feb 5, 2010 at 1:08 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 5, 2010, at 10:58 AM, Kurt Smith wrote:
>
>> Along with that: ?we discussed using the ampersand as the conjunctive
>> for the different stride and memory access flags (I thought it was
>> more pythonic, although less 'C'-like), or we could also go with the
>> pipe (|) symbol instead, ala C flags. ?If it's already decided and
>> behind us, then I'm fine with it.
>
> Personally, I think the best would be "and" a la Python. When I see |
> and &, I instinctively parse that as C flags (and the example you gave
> threw me for a loop for a sec, but I didn't want to bring it up
> myself.) Of course using | as in C will probably be unfamiliar to
> those with only Python experience, and doesn't read as well.

As I recall, we went with  '&' since it has some pedigree in
python-dev discussions on combining abstract base classes (I can't
recall the context), although I can't find the email thread ATM, and
it isn't implemented in Python.  The "and" keyword has the benefit of
reading better and making it clearly a conjunction of objects, and
less a conjunction of C-flags.  I'm fine with either, and will gladly
implement whatever is decided ('&' is implemented currently, but it's
easy to change).  I'm sure Dag has some thoughts on the matter.

Kurt

From dalcinl at gmail.com  Fri Feb  5 20:36:59 2010
From: dalcinl at gmail.com (=?UTF-8?Q?Lisandro_Dalc=C3=ADn?=)
Date: Fri, 5 Feb 2010 16:36:59 -0300
Subject: [Cython] Generate shorter string literals
In-Reply-To: <2E2BE831-C9B8-4CEE-8030-B40D20D84E75@math.washington.edu>
References: <4B6C3F1E.5030202@uci.edu>
	<2E2BE831-C9B8-4CEE-8030-B40D20D84E75@math.washington.edu>
Message-ID: <e7ba66e41002051136k30b2dad5ufbd2f71943cd6c02@mail.gmail.com>

On Fri, Feb 5, 2010 at 4:31 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Feb 5, 2010, at 7:54 AM, Christoph Gohlke wrote:
>
>> Dear Cython developers,
>>
>> This is related to <http://trac.cython.org/cython_trac/ticket/50>.
>>
>> Cython 0.12.1 still generates string literals that can be too long
>> for MSVC*
>> compilers. Please consider the following patch.
>>
>> diff -r 8bff3332e34f Cython/Compiler/Code.py
>> --- a/Cython/Compiler/Code.py ? Tue Feb 02 02:10:32 2010 -0800
>> +++ b/Cython/Compiler/Code.py ? Thu Feb 04 19:32:40 2010 -0800
>> @@ -667,7 +667,7 @@
>> ? ? ? ? decls_writer = self.parts['decls']
>> ? ? ? ? for _, cname, c in c_consts:
>> ? ? ? ? ? ? decls_writer.putln('static char %s[] = "%s";' % (
>> - ? ? ? ? ? ? ? ?cname, c.escaped_value))
>> + ? ? ? ? ? ? ? ?cname,
>> StringEncoding.split_docstring(c.escaped_value)))
>> ? ? ? ? ? ? if c.py_strings is not None:
>> ? ? ? ? ? ? ? ? for py_string in c.py_strings.itervalues():
>> ? ? ? ? ? ? ? ? ? ? py_strings.append((c.cname, len(py_string.cname),
>> py_string))
>
> Thanks! http://hg.cython.org/cython-devel/rev/dc2a7157de42
>
> I agree testcases would be good to have to ensure this doesn't happen
> again.
>

Sorry, I should review the patch in the context of the whole code,
but... Is split_docstring() a proper name for the method, given that
IIUC it have to be used for any string literal and no just
docstrings... Sorry for being too pedantic ...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri Feb  5 20:51:06 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Feb 2010 11:51:06 -0800
Subject: [Cython] Generate shorter string literals
In-Reply-To: <e7ba66e41002051136k30b2dad5ufbd2f71943cd6c02@mail.gmail.com>
References: <4B6C3F1E.5030202@uci.edu>
	<2E2BE831-C9B8-4CEE-8030-B40D20D84E75@math.washington.edu>
	<e7ba66e41002051136k30b2dad5ufbd2f71943cd6c02@mail.gmail.com>
Message-ID: <A5780C50-6735-4730-8CDC-617CA3F0CC8C@math.washington.edu>

On Feb 5, 2010, at 11:36 AM, Lisandro Dalc?n wrote:

> On Fri, Feb 5, 2010 at 4:31 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Feb 5, 2010, at 7:54 AM, Christoph Gohlke wrote:
>>
>>> Dear Cython developers,
>>>
>>> This is related to <http://trac.cython.org/cython_trac/ticket/50>.
>>>
>>> Cython 0.12.1 still generates string literals that can be too long
>>> for MSVC*
>>> compilers. Please consider the following patch.
>>>
>>> diff -r 8bff3332e34f Cython/Compiler/Code.py
>>> --- a/Cython/Compiler/Code.py   Tue Feb 02 02:10:32 2010 -0800
>>> +++ b/Cython/Compiler/Code.py   Thu Feb 04 19:32:40 2010 -0800
>>> @@ -667,7 +667,7 @@
>>>         decls_writer = self.parts['decls']
>>>         for _, cname, c in c_consts:
>>>             decls_writer.putln('static char %s[] = "%s";' % (
>>> -                cname, c.escaped_value))
>>> +                cname,
>>> StringEncoding.split_docstring(c.escaped_value)))
>>>             if c.py_strings is not None:
>>>                 for py_string in c.py_strings.itervalues():
>>>                     py_strings.append((c.cname,  
>>> len(py_string.cname),
>>> py_string))
>>
>> Thanks! http://hg.cython.org/cython-devel/rev/dc2a7157de42
>>
>> I agree testcases would be good to have to ensure this doesn't happen
>> again.
>>
>
> Sorry, I should review the patch in the context of the whole code,
> but... Is split_docstring() a proper name for the method, given that
> IIUC it have to be used for any string literal and no just
> docstrings... Sorry for being too pedantic ...

I had exactly the same thought, but since I didn't have an immediate  
better name I just left it as is. (split_string is to generic,  
split_long_string? split_stringliteral?)

- Robert



From cgohlke at uci.edu  Sat Feb  6 01:17:29 2010
From: cgohlke at uci.edu (Christoph Gohlke)
Date: Fri, 05 Feb 2010 16:17:29 -0800
Subject: [Cython] Generate shorter string literals
Message-ID: <4B6CB519.5020409@uci.edu>

A testcase is attached. Running `python runtests.py msvc_string` reports
2 errors without the patch and passes with the patch. This is the actual
msvc9 compiler error message:

msvc_strings.c(372) : error C2026: string too big, trailing characters
truncated
msvc_strings.c(373) : error C2026: string too big, trailing characters
truncated
error: command '"C:\Program Files (x86)\Microsoft Visual Studio
9.0\VC\BIN\cl.exe"' failed with exit
 status 2


If you are interested, I can provide 32 and 64 bit Windows installers of
Cython for Python 2.6. Either contact me offlist or just redistribute
the ones at <http://www.lfd.uci.edu/~gohlke/pythonlibs/#cython>.

Christoph

-------------- next part --------------
A non-text attachment was scrubbed...
Name: shorter_strings_with_testcase.zip
Type: application/octet-stream
Size: 11038 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100205/579e0ff3/attachment-0001.obj 

From dagss at student.matnat.uio.no  Sat Feb  6 09:38:55 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 06 Feb 2010 09:38:55 +0100
Subject: [Cython] Incorrect HTML annotations with NumPy
In-Reply-To: <4B6C5D98.4010406@noaa.gov>
References: <4B6C5D98.4010406@noaa.gov>
Message-ID: <4B6D2A9F.8070105@student.matnat.uio.no>

Dan Helfman wrote:
> I've encountered a problem with Cython 0.12.1 in which incorrect code 
> annotations show up when HTML is generated with the "--annotate" option.
>
> I believe this is the same issue described here:
>
>   http://trac.cython.org/cython_trac/ticket/292
>
> I've come up with an even simpler test case that triggers the 
> behavior. The test.pyx file is attached. What happens is that the 
> generated HTML for line 187 ("cdef int y = 6"), once expanded, reads 
> as follows:
>
> static int __pyx_pf_5numpy_7ndarray___getbuffer__(PyObject 
> *__pyx_v_self, Py_buffer *__pyx_v_info, int __pyx_v_flags); 
> __pyx_v_4test_y = 6;
>
> It appears to be pulling in code from both test.pyx line 187 *and* 
> numpy.pxd line 187, even though the annotation should only be for 
> test.pyx. The generated code in test.c is simply "__pyx_v_4test_y = 
> 6;" without the extraneous numpy.pxd __getbuffer__ code.
>
> This problem makes Cython HTML annotation with NumPy code very 
> difficult to use, because it's not immediately apparent from any given 
> yellow line whether it's actually triggering the generation of NumPy 
> code or whether it's merely including it in the annotation erroneously.
Thanks -- your description makes it much easier to understand #292 and 
how to fix it.

Dag Sverre

>
> If this sort of thing is more appropriate on the other mailing list, 
> please let me know.
>
> Dan
> ------------------------------------------------------------------------
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From stefan_ml at behnel.de  Sun Feb  7 20:58:26 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Feb 2010 20:58:26 +0100
Subject: [Cython] Standard C pxd conventions
In-Reply-To: <3A27302D-155D-4006-BD37-EE5B3C1DAE9E@math.washington.edu>
References: <4B67F457.3020107@student.matnat.uio.no>	<4B68170F.5040304@behnel.de>	<4B681DF3.7070802@student.matnat.uio.no>	<4B681E8B.8020305@student.matnat.uio.no>	<4B68A0A5.6000908@noaa.gov>	<92964A85-DC69-4B42-AE2D-1BDA82081764@math.washington.edu>	<4B6BFA49.70408@student.matnat.uio.no>
	<3A27302D-155D-4006-BD37-EE5B3C1DAE9E@math.washington.edu>
Message-ID: <4B6F1B62.7010207@behnel.de>


Robert Bradshaw, 05.02.2010 12:18:
> On Feb 5, 2010, at 3:00 AM, Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
>>> On Feb 2, 2010, at 2:01 PM, Christopher Barker wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>
>>>>> The alternative then appears to be
>>>>>
>>>>> from cython.c.math cimport sqrt
>>>>> from cython.python cimport Py_INCREF
>>>>>
>>>>> but that's awfully verbose.
>>>>>
>>>> I LIKE verbose imports -- that's what "from" is for. The import  
>>>> can be
>>>> verbose (and clear, and unlikely to name clash), while the rest of
>>>> your
>>>> code is as terse as you like.
>>>>
>>>> If this is all cython stuff, it should live in the cython namespace.

When I first read this, it made me consider "cython.includes" as a
sufficiently verbose prefix, but:


>>> I like clib or libc (with or without the cython prefix) better than
>>> just plain c. There's also the consideration question of supporting
>>> pure mode, currently cython.* is all directives/other builtins, this
>>> conflates the idea a bit. In the C++ branch we now have
>>> cython.operator (where dereference, preincrement, etc. all live).
>>>
>> I'm rather against the cython prefix...I don't feel it is logical that
>> "library stuff" that isn't really implemented in Cython itself lives
>> there. It is a place for "Cython builtins" to me.
> 
> Yeah, conflating the two doesn't seem natural.

I agree.


>> There's no difference in principle between, say, stdlib.h and  
>> opengl.h.
>>
>> How about just "libc" and "libcpp" then?
> 
> Sure. And we should move all the Python ones to "python."

+1

Stefan


From stefan_ml at behnel.de  Sun Feb  7 21:07:07 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Feb 2010 21:07:07 +0100
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B6C3F1E.5030202@uci.edu>
References: <4B6C3F1E.5030202@uci.edu>
Message-ID: <4B6F1D6B.8020700@behnel.de>


Christoph Gohlke, 05.02.2010 16:54:
> Dear Cython developers,
> 
> This is related to <http://trac.cython.org/cython_trac/ticket/50>.
> 
> Cython 0.12.1 still generates string literals that can be too long for MSVC*
> compilers. Please consider the following patch.
> 
> diff -r 8bff3332e34f Cython/Compiler/Code.py
> --- a/Cython/Compiler/Code.py   Tue Feb 02 02:10:32 2010 -0800
> +++ b/Cython/Compiler/Code.py   Thu Feb 04 19:32:40 2010 -0800
> @@ -667,7 +667,7 @@
>          decls_writer = self.parts['decls']
>          for _, cname, c in c_consts:
>              decls_writer.putln('static char %s[] = "%s";' % (
> -                cname, c.escaped_value))
> +                cname, StringEncoding.split_docstring(c.escaped_value)))
>              if c.py_strings is not None:
>                  for py_string in c.py_strings.itervalues():
>                      py_strings.append((c.cname, len(py_string.cname),
> py_string))

I'm surprised that this works. The last information I had from an MSVC user
 (and from a web search) was that just splitting the string into separate
auto-concatenated literals wouldn't solve the issue.

Happy to read that it does.

Stefan


From dagss at student.matnat.uio.no  Mon Feb  8 09:25:20 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 08 Feb 2010 09:25:20 +0100
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B6F1D6B.8020700@behnel.de>
References: <4B6C3F1E.5030202@uci.edu> <4B6F1D6B.8020700@behnel.de>
Message-ID: <4B6FCA70.8040200@student.matnat.uio.no>

Stefan Behnel wrote:
> Christoph Gohlke, 05.02.2010 16:54:
>   
>> Dear Cython developers,
>>
>> This is related to <http://trac.cython.org/cython_trac/ticket/50>.
>>
>> Cython 0.12.1 still generates string literals that can be too long for MSVC*
>> compilers. Please consider the following patch.
>>
>> diff -r 8bff3332e34f Cython/Compiler/Code.py
>> --- a/Cython/Compiler/Code.py   Tue Feb 02 02:10:32 2010 -0800
>> +++ b/Cython/Compiler/Code.py   Thu Feb 04 19:32:40 2010 -0800
>> @@ -667,7 +667,7 @@
>>          decls_writer = self.parts['decls']
>>          for _, cname, c in c_consts:
>>              decls_writer.putln('static char %s[] = "%s";' % (
>> -                cname, c.escaped_value))
>> +                cname, StringEncoding.split_docstring(c.escaped_value)))
>>              if c.py_strings is not None:
>>                  for py_string in c.py_strings.itervalues():
>>                      py_strings.append((c.cname, len(py_string.cname),
>> py_string))
>>     
>
> I'm surprised that this works. The last information I had from an MSVC user
>  (and from a web search) was that just splitting the string into separate
> auto-concatenated literals wouldn't solve the issue.
>
> Happy to read that it does.
>   
Cristoph: Which version of MSVC are you using/did you/are you able to 
test this on?

IIRC, Stefan wrote some code in Cython for concatenating strings at 
run-time because of this issue. If that is indeed not necesarry, then it 
should probably be removed/cleaned up.

Dag Sverre

From stefan_ml at behnel.de  Mon Feb  8 09:39:30 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Feb 2010 09:39:30 +0100
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B6FCA70.8040200@student.matnat.uio.no>
References: <4B6C3F1E.5030202@uci.edu> <4B6F1D6B.8020700@behnel.de>
	<4B6FCA70.8040200@student.matnat.uio.no>
Message-ID: <4B6FCDC2.4000005@behnel.de>


Dag Sverre Seljebotn, 08.02.2010 09:25:
> IIRC, Stefan wrote some code in Cython for concatenating strings at 
> run-time because of this issue. If that is indeed not necesarry, then it 
> should probably be removed/cleaned up.

I didn't. It was an idea at the time, but was considered too much overhead
and not even feasible in all cases where docstrings are used. So there's
nothing to clean up.

Stefan


From stefan_ml at behnel.de  Mon Feb  8 16:33:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Feb 2010 16:33:06 +0100
Subject: [Cython] C++ method overloading?
Message-ID: <4B702EB2.9050407@behnel.de>

Hi,

I'm trying to implement the PyBindGen micro-benchmarks in cython-devel by
wrapping this source file:

http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/testapi.cc
http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/testapi.h

(I know, few Cython users would wrap code at that granularity, but that's
about the limit of what PyBindGen can handle.)

The API is basically this (I attached the two wrapper source files I wrote):

------------------
cdef extern from "testapi.h":
    void func1()
    double func2(double x, double y, double z)

    cdef cppclass Multiplier:
        Multiplier()
        Multiplier(double factor)
        void SetFactor()
        void SetFactor(double f)
        double GetFactor()
        double Multiply(double value)

    double call_virtual_from_cpp (Multiplier *obj, double value)
------------------

Cython parses this just fine - however, when I try to call the SetFactor
method in Cython code, I get this:

Error converting Pyrex file to C:
------------------------------------------------------------
...
    def __dealloc__(self):
        del self.multiplier

    def SetFactor(self, f=None):
        if f is None:
            self.multiplier.SetFactor()
                                    ^
------------------------------------------------------------
.../pytestapi.pyx:18:37: Call with wrong number of arguments (expected 1,
got 0)

The exact error depends on the order in which I declare the two methods. It
seems that the last declaration wins. And Cython also doesn't complain
about the constructor call right two methods above, so that seems to have
worked.

Is this supposed to work for regular C++ methods?

Stefan
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pytestapi.pyx
Url: http://codespeak.net/pipermail/cython-dev/attachments/20100208/7bb40520/attachment.diff 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testapi.cc
Type: text/x-c++src
Size: 710 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100208/7bb40520/attachment.cc 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testapi.h
Type: text/x-chdr
Size: 542 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100208/7bb40520/attachment.h 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: testapi.pxd
Url: http://codespeak.net/pipermail/cython-dev/attachments/20100208/7bb40520/attachment-0001.diff 

From stefan_ml at behnel.de  Mon Feb  8 17:02:44 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Feb 2010 17:02:44 +0100
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B6FCA70.8040200@student.matnat.uio.no>
References: <4B6C3F1E.5030202@uci.edu> <4B6F1D6B.8020700@behnel.de>
	<4B6FCA70.8040200@student.matnat.uio.no>
Message-ID: <4B7035A4.1000404@behnel.de>


Dag Sverre Seljebotn, 08.02.2010 09:25:
> Stefan Behnel wrote:
>> I'm surprised that this works. The last information I had from an MSVC user
>>  (and from a web search) was that just splitting the string into separate
>> auto-concatenated literals wouldn't solve the issue.
>>
>> Happy to read that it does.
>>   
> Cristoph: Which version of MSVC are you using/did you/are you able to 
> test this on?

He mentioned MSVC V9. Anyway, it doesn't break anything, so it's best to
just always do it that way even if it changes nothing for older MSVC releases.

Stefan

From dagss at student.matnat.uio.no  Mon Feb  8 18:04:18 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 08 Feb 2010 18:04:18 +0100
Subject: [Cython] C++ method overloading?
In-Reply-To: <4B702EB2.9050407@behnel.de>
References: <4B702EB2.9050407@behnel.de>
Message-ID: <4B704412.7030103@student.matnat.uio.no>

Stefan Behnel wrote:
> Hi,
> 
> I'm trying to implement the PyBindGen micro-benchmarks in cython-devel by
> wrapping this source file:
> 
> http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/testapi.cc
> http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/testapi.h

Woah, I never knew about that project.

Could we borrow the .h parser strategy (whatever that is) for Cython?

Dag Sverre

From seb.binet at gmail.com  Mon Feb  8 18:27:27 2010
From: seb.binet at gmail.com (Sebastien Binet)
Date: Mon, 8 Feb 2010 18:27:27 +0100
Subject: [Cython] C++ method overloading?
In-Reply-To: <4B704412.7030103@student.matnat.uio.no>
References: <4B702EB2.9050407@behnel.de>
	<4B704412.7030103@student.matnat.uio.no>
Message-ID: <201002081827.27341.binet@cern.ch>

On Monday 08 February 2010 18:04:18 Dag Sverre Seljebotn wrote:
> Stefan Behnel wrote:
> > Hi,
> >
> > I'm trying to implement the PyBindGen micro-benchmarks in cython-devel by
> > wrapping this source file:
> >
> > http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/
> >testapi.cc
> > http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/
> >testapi.h
> 
> Woah, I never knew about that project.
> 
> Could we borrow the .h parser strategy (whatever that is) for Cython?

IIRC, pybindgen relies on gccxml[0] for the automatic .h parsing.

gccxml, which is barely maintained, hacks the gcc compiler to get its internal 
representation of a compilation unit to spit it out as a fat xml file that one 
can post-process to do all kind of things.

this is how the reflex[1] (CERN-based) project automatically generates 
bindings and runtime reflection/introspection to/for C++ classes.

the only problem is that gccxml is actually based on the C++ parser, so if the 
header you feed it with is only C-compliant then you are screwed...

there are alternatives:
 - the mozilla guys developed a gcc plugin[2] (for gcc-4.5 or a patched 
gcc-4.4) to do the same kind of thing (but their main goal is to perform C++ 
code refactorization)
 - clang/llvm, when the C++ support will be improved

cheers,
sebastien.

[0] http://www.gccxml.org/HTML/Index.html
[1] http://root.cern.ch/drupal/content/reflex
[2] https://developer.mozilla.org/en/Dehydra

> 
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
> 

-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

From robertwb at math.washington.edu  Mon Feb  8 19:57:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 8 Feb 2010 10:57:57 -0800
Subject: [Cython] C++ method overloading?
In-Reply-To: <4B702EB2.9050407@behnel.de>
References: <4B702EB2.9050407@behnel.de>
Message-ID: <941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu>

On Feb 8, 2010, at 7:33 AM, Stefan Behnel wrote:

> Hi,
>
> I'm trying to implement the PyBindGen micro-benchmarks in cython- 
> devel by
> wrapping this source file:
>
> http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/testapi.cc
> http://bazaar.launchpad.net/~gjc/pybindgen/trunk/annotate/735/benchmarks/testapi.h
>
> (I know, few Cython users would wrap code at that granularity, but  
> that's
> about the limit of what PyBindGen can handle.)
>
> The API is basically this (I attached the two wrapper source files I  
> wrote):
>
> ------------------
> cdef extern from "testapi.h":
>    void func1()
>    double func2(double x, double y, double z)
>
>    cdef cppclass Multiplier:
>        Multiplier()
>        Multiplier(double factor)
>        void SetFactor()
>        void SetFactor(double f)
>        double GetFactor()
>        double Multiply(double value)
>
>    double call_virtual_from_cpp (Multiplier *obj, double value)
> ------------------
>
> Cython parses this just fine - however, when I try to call the  
> SetFactor
> method in Cython code, I get this:
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>    def __dealloc__(self):
>        del self.multiplier
>
>    def SetFactor(self, f=None):
>        if f is None:
>            self.multiplier.SetFactor()
>                                    ^
> ------------------------------------------------------------
> .../pytestapi.pyx:18:37: Call with wrong number of arguments  
> (expected 1,
> got 0)
>
> The exact error depends on the order in which I declare the two  
> methods. It
> seems that the last declaration wins. And Cython also doesn't complain
> about the constructor call right two methods above, so that seems to  
> have
> worked.
>
> Is this supposed to work for regular C++ methods?

Yes, this should have worked. I'm surprised there aren't any tests for  
this...

- Robert



From robertwb at math.washington.edu  Mon Feb  8 20:20:37 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 8 Feb 2010 11:20:37 -0800
Subject: [Cython] C++ method overloading?
In-Reply-To: <941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu>
References: <4B702EB2.9050407@behnel.de>
	<941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu>
Message-ID: <56342EAE-423E-4EAF-A9D0-AE51F715C80B@math.washington.edu>

On Feb 8, 2010, at 10:57 AM, Robert Bradshaw wrote:

> On Feb 8, 2010, at 7:33 AM, Stefan Behnel wrote:
>
>>
>> The exact error depends on the order in which I declare the two
>> methods. It
>> seems that the last declaration wins. And Cython also doesn't  
>> complain
>> about the constructor call right two methods above, so that seems to
>> have
>> worked.
>>
>> Is this supposed to work for regular C++ methods?
>
> Yes, this should have worked. I'm surprised there aren't any tests for
> this...

OK, pushed a quick fix: http://hg.cython.org/cython-devel/rev/94c13764ba4a

- Robert


From stefan_ml at behnel.de  Tue Feb  9 10:18:35 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 09 Feb 2010 10:18:35 +0100
Subject: [Cython] C++ method overloading?
In-Reply-To: <56342EAE-423E-4EAF-A9D0-AE51F715C80B@math.washington.edu>
References: <4B702EB2.9050407@behnel.de>	<941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu>
	<56342EAE-423E-4EAF-A9D0-AE51F715C80B@math.washington.edu>
Message-ID: <4B71286B.5020805@behnel.de>


Robert Bradshaw, 08.02.2010 20:20:
> On Feb 8, 2010, at 10:57 AM, Robert Bradshaw wrote:
> 
>> On Feb 8, 2010, at 7:33 AM, Stefan Behnel wrote:
>>
>>> The exact error depends on the order in which I declare the two
>>> methods. It
>>> seems that the last declaration wins. And Cython also doesn't  
>>> complain
>>> about the constructor call right two methods above, so that seems to
>>> have
>>> worked.
>>>
>>> Is this supposed to work for regular C++ methods?
>> Yes, this should have worked. I'm surprised there aren't any tests for
>> this...
> 
> OK, pushed a quick fix: http://hg.cython.org/cython-devel/rev/94c13764ba4a

Ok, that got me past this problem. The next one was that I was using

    self.multiplier = testapi.Multiplier()

by habit, instead of

    self.multiplier = new testapi.Multiplier()

This lead to an "<error>" being generated in the code, but no error shown
on the output.

After figuring that out, I got this:

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef class Multiplier:
    cdef testapi.Multiplier* multiplier

    def __cinit__(self, factor=None):
        if factor is None:
            self.multiplier = new testapi.Multiplier()
                                                     ^
------------------------------------------------------------

.../pytestapi.pyx:9:54: Expected ')'

Totally weird kind of error, given that it points to the closing
parenthesis already.

The second error I got says: "Invalid conversion from 'Python object' to
'double'", which is something that should *always* work. I get the same
error for other numeric types in other tests. Looks like Python object
conversion is broken in cython-devel.

It seems to me that the new C++ support is lacking a lot of important
tests. I added a naming convention to the test runner that finds related
".c" and ".cpp" source files in the same test directory if they start with
the module name plus an underscore, i.e. a set of files like this:

cppwrap_lib.cpp
cppwrap_lib.h
cppwrap_lib.pxd
cppwrap.pyx

will create a module based on the source files "cppwrap.cpp" and
"cppwrap_lib.cpp". That should make it easy enough to write tests for
wrapper code.

I also added a new directory "tests/wrappers" to keep the multi-file
wrapper tests somewhat separated from the single-source-file tests in
"tests/run". I added two tests similar (but not legally related) to the
PyBindGen benchmark (which is LGPLed), one with overloaded methods, the
other one without. Both are currently failing (and disabled in bugs.txt).

Stefan

From craigcitro at gmail.com  Tue Feb  9 10:28:38 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Tue, 9 Feb 2010 01:28:38 -0800
Subject: [Cython] C++ method overloading?
In-Reply-To: <4B71286B.5020805@behnel.de>
References: <4B702EB2.9050407@behnel.de>
	<941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu> 
	<56342EAE-423E-4EAF-A9D0-AE51F715C80B@math.washington.edu> 
	<4B71286B.5020805@behnel.de>
Message-ID: <a8c4ff4c1002090128v20787a7eiacac3b3a86d18c15@mail.gmail.com>

> The second error I got says: "Invalid conversion from 'Python object' to
> 'double'", which is something that should *always* work. I get the same
> error for other numeric types in other tests. Looks like Python object
> conversion is broken in cython-devel.
>

I ran into this same problem earlier, and posted a potential fix on
#506. It definitely solved the problem I was running into -- I'd be
curious as to whether or not it solves the issues you're hitting ...

-cc

From dsurviver at gmail.com  Tue Feb  9 17:01:53 2010
From: dsurviver at gmail.com (Danilo Freitas)
Date: Tue, 9 Feb 2010 13:01:53 -0300
Subject: [Cython] C++ method overloading?
In-Reply-To: <4B71286B.5020805@behnel.de>
References: <4B702EB2.9050407@behnel.de>
	<941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu> 
	<56342EAE-423E-4EAF-A9D0-AE51F715C80B@math.washington.edu> 
	<4B71286B.5020805@behnel.de>
Message-ID: <45239151002090801u69b60199we108383b0a4ab5ca@mail.gmail.com>

2010/2/9 Stefan Behnel <stefan_ml at behnel.de>:
>
> Robert Bradshaw, 08.02.2010 20:20:
>> On Feb 8, 2010, at 10:57 AM, Robert Bradshaw wrote:
>>
>>> On Feb 8, 2010, at 7:33 AM, Stefan Behnel wrote:
>>>
>>>> The exact error depends on the order in which I declare the two
>>>> methods. It
>>>> seems that the last declaration wins. And Cython also doesn't
>>>> complain
>>>> about the constructor call right two methods above, so that seems to
>>>> have
>>>> worked.
>>>>
>>>> Is this supposed to work for regular C++ methods?
>>> Yes, this should have worked. I'm surprised there aren't any tests for
>>> this...
>>
>> OK, pushed a quick fix: http://hg.cython.org/cython-devel/rev/94c13764ba4a
>
> Ok, that got me past this problem. The next one was that I was using
>
> ? ?self.multiplier = testapi.Multiplier()
>
> by habit, instead of
>
> ? ?self.multiplier = new testapi.Multiplier()
>
> This lead to an "<error>" being generated in the code, but no error shown
> on the output.
>
> After figuring that out, I got this:
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> cdef class Multiplier:
> ? ?cdef testapi.Multiplier* multiplier
>
> ? ?def __cinit__(self, factor=None):
> ? ? ? ?if factor is None:
> ? ? ? ? ? ?self.multiplier = new testapi.Multiplier()
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
> ------------------------------------------------------------
>
> .../pytestapi.pyx:9:54: Expected ')'
>
> Totally weird kind of error, given that it points to the closing
> parenthesis already.

We didn't used cimported C++ classes. I'll take a look at this.
The problem is with the parser. It look at the first name and thinks
it is a class name, and so, expect a '('.

-- 
- Danilo Freitas

From robertwb at math.washington.edu  Tue Feb  9 18:46:24 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Feb 2010 09:46:24 -0800
Subject: [Cython] C++ method overloading?
In-Reply-To: <4B71286B.5020805@behnel.de>
References: <4B702EB2.9050407@behnel.de>	<941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu>
	<56342EAE-423E-4EAF-A9D0-AE51F715C80B@math.washington.edu>
	<4B71286B.5020805@behnel.de>
Message-ID: <10EFFD06-F2B8-45E0-B30D-604215233EE0@math.washington.edu>

On Feb 9, 2010, at 1:18 AM, Stefan Behnel wrote:

> Robert Bradshaw, 08.02.2010 20:20:
>> On Feb 8, 2010, at 10:57 AM, Robert Bradshaw wrote:
>>
>>> On Feb 8, 2010, at 7:33 AM, Stefan Behnel wrote:
>>>
>>>> The exact error depends on the order in which I declare the two
>>>> methods. It
>>>> seems that the last declaration wins. And Cython also doesn't
>>>> complain
>>>> about the constructor call right two methods above, so that seems  
>>>> to
>>>> have
>>>> worked.
>>>>
>>>> Is this supposed to work for regular C++ methods?
>>> Yes, this should have worked. I'm surprised there aren't any tests  
>>> for
>>> this...
>>
>> OK, pushed a quick fix: http://hg.cython.org/cython-devel/rev/94c13764ba4a
>
> Ok, that got me past this problem. The next one was that I was using
>
>    self.multiplier = testapi.Multiplier()
>
> by habit, instead of
>
>    self.multiplier = new testapi.Multiplier()
>
> This lead to an "<error>" being generated in the code, but no error  
> shown
> on the output.
>
> After figuring that out, I got this:
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> cdef class Multiplier:
>    cdef testapi.Multiplier* multiplier
>
>    def __cinit__(self, factor=None):
>        if factor is None:
>            self.multiplier = new testapi.Multiplier()
>                                                     ^
> ------------------------------------------------------------
>
> .../pytestapi.pyx:9:54: Expected ')'
>
> Totally weird kind of error, given that it points to the closing
> parenthesis already.
>
> The second error I got says: "Invalid conversion from 'Python  
> object' to
> 'double'", which is something that should *always* work. I get the  
> same
> error for other numeric types in other tests. Looks like Python object
> conversion is broken in cython-devel.
>
> It seems to me that the new C++ support is lacking a lot of important
> tests.

There is a *huge* deficiency in testing, and nearly every time I write  
a test I find bugs to fix. That's why it has been sitting so long. But  
there's a lot of working stuff in there too.

There's some refactoring I intend to do both in parsing and analysis,  
as the current system doesn't always work.

> I added a naming convention to the test runner that finds related
> ".c" and ".cpp" source files in the same test directory if they  
> start with
> the module name plus an underscore, i.e. a set of files like this:
>
> cppwrap_lib.cpp
> cppwrap_lib.h
> cppwrap_lib.pxd
> cppwrap.pyx
>
> will create a module based on the source files "cppwrap.cpp" and
> "cppwrap_lib.cpp". That should make it easy enough to write tests for
> wrapper code.

Nice.

- Robert


From robertwb at math.washington.edu  Wed Feb 10 06:47:00 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 9 Feb 2010 21:47:00 -0800
Subject: [Cython] Namespaces sytnax
Message-ID: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>

Before we release any of the C++ stuff into the wild, we need to nail  
down how to declare namespaces. There are two contenders:

cdef extern from "foo.h" namespace Foo.Bar:
     int f(int)

cdef extern from "foo.h" namespace "Foo::Bar":
     int f(int)

Now one thing I've realized since starting to look at this is  
namespaces can be more than just dotted Python identifiers, e.g. one  
has "std::vector<T>::iterator." Ideally, this will be handled more  
cleanly once we have cpp class nesting, but there's no way to declare  
it now (without resorting to the strings hacking that one does for C++  
currently). I'm leaning towards the latter.

Also, I've been thinking about namespace conflicts. Suppose one has  
A::B::x and Y::Z::x, then we don't provide a way to use both (short of  
resting to string cnames). We could allow

cdef extern from "..." namespace A.B as X:
     int x(int)

or

cdef extern from "..." namespace "A::B" as X:
     int x(int)

print X.x(5)

Would the former with no as clause automatically inject A.B (I'd  
rather not)? This is a non-issue if we go with the latter syntax.

- Robert


From dsurviver at gmail.com  Wed Feb 10 07:26:44 2010
From: dsurviver at gmail.com (Danilo Freitas)
Date: Wed, 10 Feb 2010 03:26:44 -0300
Subject: [Cython] Namespaces sytnax
In-Reply-To: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
Message-ID: <45239151002092226q7af70c04scff5f68a461c86d7@mail.gmail.com>

2010/2/10 Robert Bradshaw <robertwb at math.washington.edu>:
> Before we release any of the C++ stuff into the wild, we need to nail
> down how to declare namespaces. There are two contenders:
>
> cdef extern from "foo.h" namespace Foo.Bar:
> ? ? int f(int)
>
> cdef extern from "foo.h" namespace "Foo::Bar":
> ? ? int f(int)
>
> Now one thing I've realized since starting to look at this is
> namespaces can be more than just dotted Python identifiers, e.g. one
> has "std::vector<T>::iterator." Ideally, this will be handled more
> cleanly once we have cpp class nesting, but there's no way to declare
> it now (without resorting to the strings hacking that one does for C++
> currently). I'm leaning towards the latter.
>
> Also, I've been thinking about namespace conflicts. Suppose one has
> A::B::x and Y::Z::x, then we don't provide a way to use both (short of
> resting to string cnames). We could allow
>
> cdef extern from "..." namespace A.B as X:
> ? ? int x(int)
>
> or
>
> cdef extern from "..." namespace "A::B" as X:
> ? ? int x(int)
>
> print X.x(5)
>
> Would the former with no as clause automatically inject A.B (I'd
> rather not)? This is a non-issue if we go with the latter syntax.

+1 for allowing 'as' and +1 for allowing the automatically injection
of the namespace.
It's like "using namespace ..." in C++. The user should choose how to use it.

-- 
- Danilo Freitas

From stefan_ml at behnel.de  Wed Feb 10 08:00:36 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 10 Feb 2010 08:00:36 +0100
Subject: [Cython] Namespaces sytnax
In-Reply-To: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
Message-ID: <4B725994.4080508@behnel.de>


Robert Bradshaw, 10.02.2010 06:47:
> cdef extern from "foo.h" namespace "Foo::Bar":
>      int f(int)
> 
> cdef extern from "..." namespace "A::B" as X:
>      int x(int)
> 
> print X.x(5)

+1 for these two, the first meaning that the names get defined in the flat
module namespace.

I don't think automatic namespace injection makes sense for anything but
namespaces that are already valid Python identifiers. Given that there's
some magic to it, I'm all for leaving it out completely. The 'as' syntax is
simple enough, readable, and makes it clear what happens.

Also, the above will most likely appear in a (reusable) .pxd anyway. That
makes the 'as' syntax less important as there's almost always an explicit
namespace given by the .pxd file name and an additional import following in
a .pyx file, which can then decide what to name the things in the code, and
how to deal with naming collisions.

Stefan


From dagss at student.matnat.uio.no  Wed Feb 10 09:50:34 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 10 Feb 2010 09:50:34 +0100
Subject: [Cython] Namespaces sytnax
In-Reply-To: <4B725994.4080508@behnel.de>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
	<4B725994.4080508@behnel.de>
Message-ID: <4B72735A.5070102@student.matnat.uio.no>

Stefan Behnel wrote:
> Robert Bradshaw, 10.02.2010 06:47:
>   
>> cdef extern from "foo.h" namespace "Foo::Bar":
>>      int f(int)
>>
>> cdef extern from "..." namespace "A::B" as X:
>>      int x(int)
>>
>> print X.x(5)
>>     
>
> +1 for these two, the first meaning that the names get defined in the flat
> module namespace.
>
> I don't think automatic namespace injection makes sense for anything but
> namespaces that are already valid Python identifiers. Given that there's
> some magic to it, I'm all for leaving it out completely. The 'as' syntax is
> simple enough, readable, and makes it clear what happens.
>
> Also, the above will most likely appear in a (reusable) .pxd anyway. That
> makes the 'as' syntax less important as there's almost always an explicit
> namespace given by the .pxd file name and an additional import following in
> a .pyx file, which can then decide what to name the things in the code, and
> how to deal with naming collisions.
>   
I'm fine with this as well.

And I want to note that the "as X" syntax is really a shortcut for 
creating a new pxd file in a sub-package. It would be nice if that was 
reflected in the implementation (i.e. use a new ModuleScope for "X", 
don't introduce a new if-test everywhere.)

Basically:

foo.pxd:
ctypedef int footype
cdef extern from "foo.h" namespace "A::B" as X: ...

is virtually transformed into

foo/
  __init__.pxd:
  """
     ctypedef int footype
  """
  X.pxd:
  """
      cdef extern from "foo.h" namespace "A::B": # note no as X
         ...
   """

Corollary: If "as X" appears in the namespace-as clause in an 
__init__.pxd, then there can be no X.pxd or X.pyx in the same directory.

Dag Sverre

From robertwb at math.washington.edu  Wed Feb 10 10:20:58 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 10 Feb 2010 01:20:58 -0800
Subject: [Cython] Namespaces sytnax
In-Reply-To: <4B72735A.5070102@student.matnat.uio.no>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
	<4B725994.4080508@behnel.de>
	<4B72735A.5070102@student.matnat.uio.no>
Message-ID: <BEA46C24-F6D1-4CC9-A1BD-A6F356BBA2E4@math.washington.edu>

On Feb 10, 2010, at 12:50 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Robert Bradshaw, 10.02.2010 06:47:
>>
>>> cdef extern from "foo.h" namespace "Foo::Bar":
>>>     int f(int)
>>>
>>> cdef extern from "..." namespace "A::B" as X:
>>>     int x(int)
>>>
>>> print X.x(5)
>>>
>>
>> +1 for these two, the first meaning that the names get defined in  
>> the flat
>> module namespace.
>>
>> I don't think automatic namespace injection makes sense for  
>> anything but
>> namespaces that are already valid Python identifiers. Given that  
>> there's
>> some magic to it, I'm all for leaving it out completely. The 'as'  
>> syntax is
>> simple enough, readable, and makes it clear what happens.
>>
>> Also, the above will most likely appear in a (reusable) .pxd  
>> anyway. That
>> makes the 'as' syntax less important as there's almost always an  
>> explicit
>> namespace given by the .pxd file name and an additional import  
>> following in
>> a .pyx file, which can then decide what to name the things in the  
>> code, and
>> how to deal with naming collisions.

That is a very good point.

> I'm fine with this as well.
>
> And I want to note that the "as X" syntax is really a shortcut for
> creating a new pxd file in a sub-package. It would be nice if that was
> reflected in the implementation (i.e. use a new ModuleScope for "X",
> don't introduce a new if-test everywhere.)
>
> Basically:
>
> foo.pxd:
> ctypedef int footype
> cdef extern from "foo.h" namespace "A::B" as X: ...
>
> is virtually transformed into
>
> foo/
>  __init__.pxd:
>  """
>     ctypedef int footype
>  """
>  X.pxd:
>  """
>      cdef extern from "foo.h" namespace "A::B": # note no as X
>         ...
>   """
>
> Corollary: If "as X" appears in the namespace-as clause in an
> __init__.pxd, then there can be no X.pxd or X.pyx in the same  
> directory.

-1 to this--it doesn't mesh well with the meaning of the as keyword in  
Python. Python uses the as keyword to mean "I'm making up a new  
namespace to store all this stuff" and only has local implications.

Implementing this as a module scope under the hood is probably a good  
idea, but the X is only the name its bound to.

- Robert


From greg.ewing at canterbury.ac.nz  Thu Feb 11 00:53:36 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 11 Feb 2010 12:53:36 +1300
Subject: [Cython] Namespaces sytnax
In-Reply-To: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
Message-ID: <4B734700.8090606@canterbury.ac.nz>

Robert Bradshaw wrote:

> Now one thing I've realized since starting to look at this is  
> namespaces can be more than just dotted Python identifiers, e.g. one  
> has "std::vector<T>::iterator."

The consistent way to handle that would be using a
cname, e.g.

   cdef extern from "..." namespace std.vector "std::vector<T>":
     ...

-- 
Greg

From stefan_ml at behnel.de  Thu Feb 11 12:56:30 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Feb 2010 12:56:30 +0100
Subject: [Cython] Namespaces sytnax
In-Reply-To: <4B734700.8090606@canterbury.ac.nz>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
	<4B734700.8090606@canterbury.ac.nz>
Message-ID: <4B73F06E.2000907@behnel.de>


Greg Ewing, 11.02.2010 00:53:
> Robert Bradshaw wrote:
>> Now one thing I've realized since starting to look at this is  
>> namespaces can be more than just dotted Python identifiers, e.g. one  
>> has "std::vector<T>::iterator."
> 
> The consistent way to handle that would be using a
> cname, e.g.
> 
>    cdef extern from "..." namespace std.vector "std::vector<T>":
>      ...

I agree, but this doesn't quite fit the use case. I expect the Python name
not to be used at all in most cases, basically because it will only appear
in a .pxd file. Having to provide it in all cases will just make this less
usable.

Stefan


From greg.ewing at canterbury.ac.nz  Thu Feb 11 23:58:03 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 12 Feb 2010 11:58:03 +1300
Subject: [Cython] Namespaces sytnax
In-Reply-To: <4B73F06E.2000907@behnel.de>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
	<4B734700.8090606@canterbury.ac.nz> <4B73F06E.2000907@behnel.de>
Message-ID: <4B748B7B.1070106@canterbury.ac.nz>

Stefan Behnel wrote:

> I agree, but this doesn't quite fit the use case. I expect the Python name
> not to be used at all in most cases

Hmmm. I suppose you could still consider it to be
a case of the cname pattern where the Python name
can be omitted if you're not using it.

-- 
Greg

From ondrej at certik.cz  Fri Feb 12 02:11:55 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 11 Feb 2010 17:11:55 -0800
Subject: [Cython] [patch] tutorial improvement
Message-ID: <85b5c3131002111711r23b4976bh3fbd0fb2160cd0c3@mail.gmail.com>

Hi,

we made some improvements into the C++ wrapping tutorial, patch
attached, I've also pushed it here:

http://bitbucket.org/certik/cython-doc/

so that you can just pull it. Essentially, my colleague followed the
tutorial step by step and he got missing symbols when he imported the
module in Python. So we improved the tutorial, so that it is explicit
now.

See the log in the patch (or better the patch itself) for the details.

Thanks,
Ondrej & Aayush
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tutorial.patch
Type: text/x-patch
Size: 2502 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100211/6f7a656b/attachment.bin 

From robertwb at math.washington.edu  Fri Feb 12 08:47:49 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Feb 2010 23:47:49 -0800
Subject: [Cython] [patch] tutorial improvement
In-Reply-To: <85b5c3131002111711r23b4976bh3fbd0fb2160cd0c3@mail.gmail.com>
References: <85b5c3131002111711r23b4976bh3fbd0fb2160cd0c3@mail.gmail.com>
Message-ID: <B90A687C-92ED-4CAD-B7C5-73483116B87F@math.washington.edu>

Thanks! I've pushed to the docs, will regenerate (there's a couple of  
other fixes that need to go in as well).

On Feb 11, 2010, at 5:11 PM, Ondrej Certik wrote:

> Hi,
>
> we made some improvements into the C++ wrapping tutorial, patch
> attached, I've also pushed it here:
>
> http://bitbucket.org/certik/cython-doc/
>
> so that you can just pull it. Essentially, my colleague followed the
> tutorial step by step and he got missing symbols when he imported the
> module in Python. So we improved the tutorial, so that it is explicit
> now.
>
> See the log in the patch (or better the patch itself) for the details.
>
> Thanks,
> Ondrej & Aayush
> <tutorial.patch>_______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From ondrej at certik.cz  Fri Feb 12 21:33:21 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 12 Feb 2010 12:33:21 -0800
Subject: [Cython] [patch] tutorial improvement
In-Reply-To: <B90A687C-92ED-4CAD-B7C5-73483116B87F@math.washington.edu>
References: <85b5c3131002111711r23b4976bh3fbd0fb2160cd0c3@mail.gmail.com>
	<B90A687C-92ED-4CAD-B7C5-73483116B87F@math.washington.edu>
Message-ID: <85b5c3131002121233v5f7516c4r2503247ccadfbabc@mail.gmail.com>

Thank you Robert!

Ondrej

On Thu, Feb 11, 2010 at 11:47 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> Thanks! I've pushed to the docs, will regenerate (there's a couple of
> other fixes that need to go in as well).
>
> On Feb 11, 2010, at 5:11 PM, Ondrej Certik wrote:
>
>> Hi,
>>
>> we made some improvements into the C++ wrapping tutorial, patch
>> attached, I've also pushed it here:
>>
>> http://bitbucket.org/certik/cython-doc/
>>
>> so that you can just pull it. Essentially, my colleague followed the
>> tutorial step by step and he got missing symbols when he imported the
>> module in Python. So we improved the tutorial, so that it is explicit
>> now.
>>
>> See the log in the patch (or better the patch itself) for the details.
>>
>> Thanks,
>> Ondrej & Aayush
>> <tutorial.patch>_______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From greg.ewing at canterbury.ac.nz  Mon Feb 15 02:11:24 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 15 Feb 2010 14:11:24 +1300
Subject: [Cython] Traceback-related problem in Pyrex
Message-ID: <4B789F3C.4020402@canterbury.ac.nz>

I'm trying to track down a problem using Pyrex with
Python 2.6, and it seems to have something to do with
the fake frame objects that Pyrex creates. I'm wondering
whether something about frame objects has changed in
2.6.

The user reports that the problem does not occur when
using Cython. Have you changed anything in __Pyx_AddTraceback
in Cython?

-- 
Greg

From stefan_ml at behnel.de  Mon Feb 15 09:13:50 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 15 Feb 2010 09:13:50 +0100
Subject: [Cython] Traceback-related problem in Pyrex
In-Reply-To: <4B789F3C.4020402@canterbury.ac.nz>
References: <4B789F3C.4020402@canterbury.ac.nz>
Message-ID: <4B79023E.7080100@behnel.de>

Greg Ewing, 15.02.2010 02:11:
> I'm trying to track down a problem using Pyrex with
> Python 2.6, and it seems to have something to do with
> the fake frame objects that Pyrex creates. I'm wondering
> whether something about frame objects has changed in
> 2.6.

Quite possible. There were a lot of merges between the Py3k branch and the
Py2 trunk at the time.


> The user reports that the problem does not occur when
> using Cython. Have you changed anything in __Pyx_AddTraceback
> in Cython?

Certainly. I reworked all sorts of things in the exception handling code
for Py3 support. Not so much in __Pyx_AddTraceback (nothing that I would
consider relevant here), but clearly in the exception setting and resetting
code. Some of that may well be relevant for 2.6.

Stefan

From stefan_ml at behnel.de  Tue Feb 16 18:07:09 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 16 Feb 2010 18:07:09 +0100
Subject: [Cython] Patch against Python 3.x to run Cython
Message-ID: <4B7AD0BD.4050706@behnel.de>

Hi,

the long-standing exception handling crash bug in Py3.0 and later that
prevented Cython from running its test suite on Py3 has finally been fixed.

http://bugs.python.org/issue7173

The patch below lets Cython run safely on the latest Python 3.2 and will
hopefully make it into Python 3.1.2 soon.

Running the test suite on a patched CPython immediately revealed several
Py3 problems in the tests, so please try to build a patched Py3.2 from SVN
HEAD or apply the patch to the 3.1.1 release version, run Cython's test
suite, and try to fix them as you see fit.

Stefan


diff -r 4465a45b8876 Python/ceval.c
--- a/Python/ceval.c	Mon Feb 15 09:35:16 2010 +0100
+++ b/Python/ceval.c	Mon Feb 15 20:51:58 2010 +0100
@@ -1159,7 +1159,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int
 	assert(stack_pointer != NULL);
 	f->f_stacktop = NULL;	/* remains NULL unless yield suspends frame */

-	if (f->f_code->co_flags & CO_GENERATOR) {
+	if (!throwflag && (f->f_code->co_flags & CO_GENERATOR)) {
 		if (f->f_exc_type != NULL && f->f_exc_type != Py_None) {
 			/* We were in an except handler when we left,
 			   restore the exception state which was put aside

From sccolbert at gmail.com  Tue Feb 16 21:16:50 2010
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 16 Feb 2010 15:16:50 -0500
Subject: [Cython] an idea for a templating engine syntax
Message-ID: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>

Hey all,

When coding for the scikits.image project, we are making extensive use of
Cython. It would be nice to have algorithms that work on multiple datatypes
(numpy dtypes) without having to hand code everything. So, i've been toying
around with how a simple templating syntax might look.

I've thrown this up on pastebin:

http://pastebin.com/f1a49143d

I would think (but really i have no idea) that this would be possible to
have Cython do during cythonization. But I could implement a templating
script that does it, just as proof of concept.

What do you all think of that?

Cheers,

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100216/100df31c/attachment.htm 

From dagss at student.matnat.uio.no  Tue Feb 16 21:35:49 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Feb 2010 21:35:49 +0100
Subject: [Cython] an idea for a templating engine syntax
In-Reply-To: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>
References: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>
Message-ID: <4B7B01A5.2090007@student.matnat.uio.no>

Chris Colbert wrote:
> Hey all,
> 
> When coding for the scikits.image project, we are making extensive use 
> of Cython. It would be nice to have algorithms that work on multiple 
> datatypes (numpy dtypes) without having to hand code everything. So, 
> i've been toying around with how a simple templating syntax might look. 
> 
> I've thrown this up on pastebin:
> 
> http://pastebin.com/f1a49143d
> 
> I would think (but really i have no idea) that this would be possible to 
> have Cython do during cythonization. But I could implement a templating 
> script that does it, just as proof of concept. 
> 
> What do you all think of that?

I'm happy for any effort done in the area in general. But I'm kind of 
lukewarm to actually having your proposal into Cython as it stands. In 
my mind there's two seperate concepts.

1) String-based templating
2) Proper language templates

2) has been discussed pretty much in the past and is something we really 
want to have in Cython. However it is something closer to C++ templates 
or Java generics and would look quite different from the syntax you propose.

Your syntax is good for string-based templating (which is what you seem 
to propose), however for that I think it is just as well to use a proper 
templating engine and not couple Cython into it at all. I.e. write the 
files using e.g. proper Cheetah templates, then use a script to turn it 
into pyx (i.e. what you propose, but without the prospect of building it 
into Cython, and without (re)inventing yet another templating language).

What really lacks is Cython support for a decent build system -- making 
a convention that the ".tpyx" suffix means "run Cheetah on it, then 
Cython, then gcc" should be a ten-liner, not a huge and impossible 
battle with distutils.

-- 
Dag Sverre

From dalcinl at gmail.com  Tue Feb 16 21:52:49 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 16 Feb 2010 17:52:49 -0300
Subject: [Cython] Patch against Python 3.x to run Cython
In-Reply-To: <4B7AD0BD.4050706@behnel.de>
References: <4B7AD0BD.4050706@behnel.de>
Message-ID: <e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>

On 16 February 2010 14:07, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Running the test suite on a patched CPython immediately revealed several
> Py3 problems in the tests, so please try to build a patched Py3.2 from SVN
> HEAD or apply the patch to the 3.1.1 release version, run Cython's test
> suite, and try to fix them as you see fit.
>

Indeed.

1) tests/run/complex_numbers*.py:

I'm already working on a quik fix for the issue related to repr() of
complex numbers (basically, it seems that -0.0+0.0 -> +0.0 ... not
sure is this is std behavior)

2) tests/run/extern_builtins_T258.pyx

I simply propose to remove the declaration of __builtin__.str from the
testcase (and use Py_SIZE(L) instead of L.ob_size).

3) tests/run/embedsignatures.pyx

I'm not sure what's going on here, but this is likely a leftover for
the change to string literals match Python-level 'str' type. Stefan, I
need you help here.

Stefan, Can you look at (3) ? I bet you will figure a fix in far less
time than me :-)

>
>
> diff -r 4465a45b8876 Python/ceval.c
> --- a/Python/ceval.c ? ?Mon Feb 15 09:35:16 2010 +0100
> +++ b/Python/ceval.c ? ?Mon Feb 15 20:51:58 2010 +0100
> @@ -1159,7 +1159,7 @@ PyEval_EvalFrameEx(PyFrameObject *f, int
> ? ? ? ?assert(stack_pointer != NULL);
> ? ? ? ?f->f_stacktop = NULL; ? /* remains NULL unless yield suspends frame */
>
> - ? ? ? if (f->f_code->co_flags & CO_GENERATOR) {
> + ? ? ? if (!throwflag && (f->f_code->co_flags & CO_GENERATOR)) {
> ? ? ? ? ? ? ? ?if (f->f_exc_type != NULL && f->f_exc_type != Py_None) {
> ? ? ? ? ? ? ? ? ? ? ? ?/* We were in an except handler when we left,
> ? ? ? ? ? ? ? ? ? ? ? ? ? restore the exception state which was put aside




-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb 16 22:05:33 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 16 Feb 2010 18:05:33 -0300
Subject: [Cython] an idea for a templating engine syntax
In-Reply-To: <4B7B01A5.2090007@student.matnat.uio.no>
References: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>
	<4B7B01A5.2090007@student.matnat.uio.no>
Message-ID: <e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>

On 16 February 2010 17:35, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Chris Colbert wrote:
>> Hey all,
>>
>> When coding for the scikits.image project, we are making extensive use
>> of Cython. It would be nice to have algorithms that work on multiple
>> datatypes (numpy dtypes) without having to hand code everything. So,
>> i've been toying around with how a simple templating syntax might look.
>>
>> I've thrown this up on pastebin:
>>
>> http://pastebin.com/f1a49143d
>>
>> I would think (but really i have no idea) that this would be possible to
>> have Cython do during cythonization. But I could implement a templating
>> script that does it, just as proof of concept.
>>
>> What do you all think of that?
>
> I'm happy for any effort done in the area in general. But I'm kind of
> lukewarm to actually having your proposal into Cython as it stands. In
> my mind there's two seperate concepts.
>
> 1) String-based templating
> 2) Proper language templates
>
> 2) has been discussed pretty much in the past and is something we really
> want to have in Cython. However it is something closer to C++ templates
> or Java generics and would look quite different from the syntax you propose.
>
> Your syntax is good for string-based templating (which is what you seem
> to propose), however for that I think it is just as well to use a proper
> templating engine and not couple Cython into it at all. I.e. write the
> files using e.g. proper Cheetah templates, then use a script to turn it
> into pyx (i.e. what you propose, but without the prospect of building it
> into Cython, and without (re)inventing yet another templating language).
>

I agree.

> What really lacks is Cython support for a decent build system -- making
> a convention that the ".tpyx" suffix means "run Cheetah on it, then
> Cython, then gcc" should be a ten-liner, not a huge and impossible
> battle with distutils.
>

BTW, IIUC, NumPy has it's own (very basic?) template engine. IIRC, the
convention is to use a .src suffix. And NumPy distutils will handle
the *.pyx.src -> *.pyx automatically...


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue Feb 16 22:11:14 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 16 Feb 2010 22:11:14 +0100
Subject: [Cython] an idea for a templating engine syntax
In-Reply-To: <e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>
References: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>	<4B7B01A5.2090007@student.matnat.uio.no>
	<e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>
Message-ID: <4B7B09F2.5010203@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On 16 February 2010 17:35, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Chris Colbert wrote:
>>> Hey all,
>>>
>>> When coding for the scikits.image project, we are making extensive use
>>> of Cython. It would be nice to have algorithms that work on multiple
>>> datatypes (numpy dtypes) without having to hand code everything. So,
>>> i've been toying around with how a simple templating syntax might look.
>>>
>>> I've thrown this up on pastebin:
>>>
>>> http://pastebin.com/f1a49143d
>>>
>>> I would think (but really i have no idea) that this would be possible to
>>> have Cython do during cythonization. But I could implement a templating
>>> script that does it, just as proof of concept.
>>>
>>> What do you all think of that?
>> I'm happy for any effort done in the area in general. But I'm kind of
>> lukewarm to actually having your proposal into Cython as it stands. In
>> my mind there's two seperate concepts.
>>
>> 1) String-based templating
>> 2) Proper language templates
>>
>> 2) has been discussed pretty much in the past and is something we really
>> want to have in Cython. However it is something closer to C++ templates
>> or Java generics and would look quite different from the syntax you propose.
>>
>> Your syntax is good for string-based templating (which is what you seem
>> to propose), however for that I think it is just as well to use a proper
>> templating engine and not couple Cython into it at all. I.e. write the
>> files using e.g. proper Cheetah templates, then use a script to turn it
>> into pyx (i.e. what you propose, but without the prospect of building it
>> into Cython, and without (re)inventing yet another templating language).
>>
> 
> I agree.
> 
>> What really lacks is Cython support for a decent build system -- making
>> a convention that the ".tpyx" suffix means "run Cheetah on it, then
>> Cython, then gcc" should be a ten-liner, not a huge and impossible
>> battle with distutils.
>>
> 
> BTW, IIUC, NumPy has it's own (very basic?) template engine. IIRC, the
> convention is to use a .src suffix. And NumPy distutils will handle
> the *.pyx.src -> *.pyx automatically...
> 

That's true -- for scikits.image that's likely the recommended approach, 
as it only has the same dependencies as SciPy then.

-- 
Dag Sverre

From dalcinl at gmail.com  Tue Feb 16 22:08:47 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 16 Feb 2010 18:08:47 -0300
Subject: [Cython] Parsing.py and Python 2.3
Message-ID: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>

That file lost backward-compatibility with Py2.3 (because it uses
'set' builtin). As Parsin.py is one of these few files that are
cythonized,... Can you see any problem on the obvious patch below??:


$ hg diff Cython/Compiler/Parsing.py
diff -r 6fc30b5dadc3 Cython/Compiler/Parsing.py
--- a/Cython/Compiler/Parsing.py	Fri Feb 12 19:04:24 2010 +0100
+++ b/Cython/Compiler/Parsing.py	Tue Feb 16 18:06:03 2010 -0300
@@ -10,6 +10,13 @@
 import os
 import re
 import sys
+
+try:
+    set
+except NameError:
+    # Python 2.3
+    from sets import Set as set
+
 from Cython.Compiler.Scanning import PyrexScanner, FileSourceDescriptor
 import Nodes
 import ExprNodes



-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb 16 22:15:30 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 16 Feb 2010 18:15:30 -0300
Subject: [Cython] PyListObject in Python 2.3 and list.pop() optimization
Message-ID: <e7ba66e41002161315s5231809ap85296580d40c5506@mail.gmail.com>

The 'allocated' field is not there... Robert, could you fix your
list.pop() optimization for the Py2.3 case? Or remove it completely,
unless you still like it so much ;-)


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue Feb 16 22:24:15 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Feb 2010 13:24:15 -0800
Subject: [Cython] Parsing.py and Python 2.3
In-Reply-To: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>
References: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>
Message-ID: <80282B91-7488-44FF-AAEB-B8BAE8412791@math.washington.edu>

On Feb 16, 2010, at 1:08 PM, Lisandro Dalcin wrote:

> That file lost backward-compatibility with Py2.3 (because it uses
> 'set' builtin). As Parsin.py is one of these few files that are
> cythonized,... Can you see any problem on the obvious patch below??:
>
>
> $ hg diff Cython/Compiler/Parsing.py
> diff -r 6fc30b5dadc3 Cython/Compiler/Parsing.py
> --- a/Cython/Compiler/Parsing.py	Fri Feb 12 19:04:24 2010 +0100
> +++ b/Cython/Compiler/Parsing.py	Tue Feb 16 18:06:03 2010 -0300
> @@ -10,6 +10,13 @@
> import os
> import re
> import sys
> +
> +try:
> +    set
> +except NameError:
> +    # Python 2.3
> +    from sets import Set as set
> +
> from Cython.Compiler.Scanning import PyrexScanner,  
> FileSourceDescriptor
> import Nodes
> import ExprNodes

Looks fine to me. We do the same elsewhere.

- Robert



From robertwb at math.washington.edu  Tue Feb 16 22:34:47 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Feb 2010 13:34:47 -0800
Subject: [Cython] an idea for a templating engine syntax
In-Reply-To: <e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>
References: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>
	<4B7B01A5.2090007@student.matnat.uio.no>
	<e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>
Message-ID: <47B6219C-3DFE-4F6B-9677-E6B7A74DDCB3@math.washington.edu>

On Feb 16, 2010, at 1:05 PM, Lisandro Dalcin wrote:

> On 16 February 2010 17:35, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Chris Colbert wrote:
>>> Hey all,
>>>
>>> When coding for the scikits.image project, we are making extensive  
>>> use
>>> of Cython. It would be nice to have algorithms that work on multiple
>>> datatypes (numpy dtypes) without having to hand code everything. So,
>>> i've been toying around with how a simple templating syntax might  
>>> look.
>>>
>>> I've thrown this up on pastebin:
>>>
>>> http://pastebin.com/f1a49143d
>>>
>>> I would think (but really i have no idea) that this would be  
>>> possible to
>>> have Cython do during cythonization. But I could implement a  
>>> templating
>>> script that does it, just as proof of concept.
>>>
>>> What do you all think of that?
>>
>> I'm happy for any effort done in the area in general. But I'm kind of
>> lukewarm to actually having your proposal into Cython as it stands.  
>> In
>> my mind there's two seperate concepts.
>>
>> 1) String-based templating
>> 2) Proper language templates
>>
>> 2) has been discussed pretty much in the past and is something we  
>> really
>> want to have in Cython. However it is something closer to C++  
>> templates
>> or Java generics and would look quite different from the syntax you  
>> propose.
>>
>> Your syntax is good for string-based templating (which is what you  
>> seem
>> to propose), however for that I think it is just as well to use a  
>> proper
>> templating engine and not couple Cython into it at all. I.e. write  
>> the
>> files using e.g. proper Cheetah templates, then use a script to  
>> turn it
>> into pyx (i.e. what you propose, but without the prospect of  
>> building it
>> into Cython, and without (re)inventing yet another templating  
>> language).
>>
>
> I agree.

I'll second this as well. Using NumPy's templating sounds like a good  
option for you.

- Robert


From robertwb at math.washington.edu  Tue Feb 16 22:32:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Feb 2010 13:32:57 -0800
Subject: [Cython] PyListObject in Python 2.3 and list.pop() optimization
In-Reply-To: <e7ba66e41002161315s5231809ap85296580d40c5506@mail.gmail.com>
References: <e7ba66e41002161315s5231809ap85296580d40c5506@mail.gmail.com>
Message-ID: <ABA35CD6-7584-4F85-B71E-29A32AC01F2B@math.washington.edu>

On Feb 16, 2010, at 1:15 PM, Lisandro Dalcin wrote:

> The 'allocated' field is not there... Robert, could you fix your
> list.pop() optimization for the Py2.3 case? Or remove it completely,
> unless you still like it so much ;-)

Sorry. I'll fix that.

- Robert



From sccolbert at gmail.com  Tue Feb 16 23:27:59 2010
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 16 Feb 2010 17:27:59 -0500
Subject: [Cython] an idea for a templating engine syntax
In-Reply-To: <47B6219C-3DFE-4F6B-9677-E6B7A74DDCB3@math.washington.edu>
References: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>
	<4B7B01A5.2090007@student.matnat.uio.no>
	<e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>
	<47B6219C-3DFE-4F6B-9677-E6B7A74DDCB3@math.washington.edu>
Message-ID: <7f014ea61002161427u3ec17c00x43e0e3abfc62ff85@mail.gmail.com>

wow, i didnt even realize that numpy had such an engine. Thanks!

-Chris

On Tue, Feb 16, 2010 at 4:34 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On Feb 16, 2010, at 1:05 PM, Lisandro Dalcin wrote:
>
> > On 16 February 2010 17:35, Dag Sverre Seljebotn
> > <dagss at student.matnat.uio.no> wrote:
> >> Chris Colbert wrote:
> >>> Hey all,
> >>>
> >>> When coding for the scikits.image project, we are making extensive
> >>> use
> >>> of Cython. It would be nice to have algorithms that work on multiple
> >>> datatypes (numpy dtypes) without having to hand code everything. So,
> >>> i've been toying around with how a simple templating syntax might
> >>> look.
> >>>
> >>> I've thrown this up on pastebin:
> >>>
> >>> http://pastebin.com/f1a49143d
> >>>
> >>> I would think (but really i have no idea) that this would be
> >>> possible to
> >>> have Cython do during cythonization. But I could implement a
> >>> templating
> >>> script that does it, just as proof of concept.
> >>>
> >>> What do you all think of that?
> >>
> >> I'm happy for any effort done in the area in general. But I'm kind of
> >> lukewarm to actually having your proposal into Cython as it stands.
> >> In
> >> my mind there's two seperate concepts.
> >>
> >> 1) String-based templating
> >> 2) Proper language templates
> >>
> >> 2) has been discussed pretty much in the past and is something we
> >> really
> >> want to have in Cython. However it is something closer to C++
> >> templates
> >> or Java generics and would look quite different from the syntax you
> >> propose.
> >>
> >> Your syntax is good for string-based templating (which is what you
> >> seem
> >> to propose), however for that I think it is just as well to use a
> >> proper
> >> templating engine and not couple Cython into it at all. I.e. write
> >> the
> >> files using e.g. proper Cheetah templates, then use a script to
> >> turn it
> >> into pyx (i.e. what you propose, but without the prospect of
> >> building it
> >> into Cython, and without (re)inventing yet another templating
> >> language).
> >>
> >
> > I agree.
>
> I'll second this as well. Using NumPy's templating sounds like a good
> option for you.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100216/698e4b5a/attachment.htm 

From cournape at gmail.com  Wed Feb 17 04:46:47 2010
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 17 Feb 2010 12:46:47 +0900
Subject: [Cython] an idea for a templating engine syntax
In-Reply-To: <e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>
References: <7f014ea61002161216g6cba3a27k6aa4fa7d22e88d20@mail.gmail.com>
	<4B7B01A5.2090007@student.matnat.uio.no>
	<e7ba66e41002161305x1ae8b09x8fd116b0847c15e9@mail.gmail.com>
Message-ID: <5b8d13221002161946r7f9a76caj471678f4406339d7@mail.gmail.com>

On Wed, Feb 17, 2010 at 6:05 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 16 February 2010 17:35, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> Chris Colbert wrote:
>>> Hey all,
>>>
>>> When coding for the scikits.image project, we are making extensive use
>>> of Cython. It would be nice to have algorithms that work on multiple
>>> datatypes (numpy dtypes) without having to hand code everything. So,
>>> i've been toying around with how a simple templating syntax might look.
>>>
>>> I've thrown this up on pastebin:
>>>
>>> http://pastebin.com/f1a49143d
>>>
>>> I would think (but really i have no idea) that this would be possible to
>>> have Cython do during cythonization. But I could implement a templating
>>> script that does it, just as proof of concept.
>>>
>>> What do you all think of that?
>>
>> I'm happy for any effort done in the area in general. But I'm kind of
>> lukewarm to actually having your proposal into Cython as it stands. In
>> my mind there's two seperate concepts.
>>
>> 1) String-based templating
>> 2) Proper language templates
>>
>> 2) has been discussed pretty much in the past and is something we really
>> want to have in Cython. However it is something closer to C++ templates
>> or Java generics and would look quite different from the syntax you propose.
>>
>> Your syntax is good for string-based templating (which is what you seem
>> to propose), however for that I think it is just as well to use a proper
>> templating engine and not couple Cython into it at all. I.e. write the
>> files using e.g. proper Cheetah templates, then use a script to turn it
>> into pyx (i.e. what you propose, but without the prospect of building it
>> into Cython, and without (re)inventing yet another templating language).
>>
>
> I agree.
>
>> What really lacks is Cython support for a decent build system -- making
>> a convention that the ".tpyx" suffix means "run Cheetah on it, then
>> Cython, then gcc" should be a ten-liner, not a huge and impossible
>> battle with distutils.
>>
>
> BTW, IIUC, NumPy has it's own (very basic?) template engine. IIRC, the
> convention is to use a .src suffix. And NumPy distutils will handle
> the *.pyx.src -> *.pyx automatically...

Unfortunately, it is slightly more complicated: .c.src is turned into
.c and .f.src is turned into .f, and both are handled by different
template engines.

Adding support for cython is on my TODO list - it should not be too
hard, so patches are welcomed :)

cheers,

David

From stefan_ml at behnel.de  Wed Feb 17 08:06:10 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Feb 2010 08:06:10 +0100
Subject: [Cython] Parsing.py and Python 2.3
In-Reply-To: <80282B91-7488-44FF-AAEB-B8BAE8412791@math.washington.edu>
References: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>
	<80282B91-7488-44FF-AAEB-B8BAE8412791@math.washington.edu>
Message-ID: <4B7B9562.3080902@behnel.de>

Robert Bradshaw, 16.02.2010 22:24:
> On Feb 16, 2010, at 1:08 PM, Lisandro Dalcin wrote:
> 
>> That file lost backward-compatibility with Py2.3 (because it uses
>> 'set' builtin). As Parsin.py is one of these few files that are
>> cythonized,... Can you see any problem on the obvious patch below??:
>>
>>
>> $ hg diff Cython/Compiler/Parsing.py
>> diff -r 6fc30b5dadc3 Cython/Compiler/Parsing.py
>> --- a/Cython/Compiler/Parsing.py	Fri Feb 12 19:04:24 2010 +0100
>> +++ b/Cython/Compiler/Parsing.py	Tue Feb 16 18:06:03 2010 -0300
>> @@ -10,6 +10,13 @@
>> import os
>> import re
>> import sys
>> +
>> +try:
>> +    set
>> +except NameError:
>> +    # Python 2.3
>> +    from sets import Set as set
>> +
>> from Cython.Compiler.Scanning import PyrexScanner,  
>> FileSourceDescriptor
>> import Nodes
>> import ExprNodes
> 
> Looks fine to me. We do the same elsewhere.

This shadows the 'set' builtin, though, so Cython can't optimise it. That
may not be important in this case, but given that Cython "implements" the
set type for 2.3 anyway, would it make sense to provide the 'set' builtin
in Shadow.py and reimport it from there instead?

Stefan


From stefan_ml at behnel.de  Wed Feb 17 08:11:40 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Feb 2010 08:11:40 +0100
Subject: [Cython] Parsing.py and Python 2.3
In-Reply-To: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>
References: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>
Message-ID: <4B7B96AC.8040301@behnel.de>

Lisandro Dalcin, 16.02.2010 22:08:
> That file lost backward-compatibility with Py2.3

BTW, thanks for bringing this up, Lisandro. I keep caring about the Py3
side of things, but I admit that I started to blank 2.3 from my scope. It's
not hard to keep compatibility up here, so it's still worth it IMO.

Stefan


From stefan_ml at behnel.de  Wed Feb 17 09:35:18 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Feb 2010 09:35:18 +0100
Subject: [Cython] Patch against Python 3.x to run Cython
In-Reply-To: <e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>
References: <4B7AD0BD.4050706@behnel.de>
	<e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>
Message-ID: <4B7BAA46.7050800@behnel.de>

Lisandro Dalcin, 16.02.2010 21:52:
> 2) tests/run/extern_builtins_T258.pyx
> 
> I simply propose to remove the declaration of __builtin__.str from the
> testcase (and use Py_SIZE(L) instead of L.ob_size).

Sounds reasonable, I've done that.


> 3) tests/run/embedsignatures.pyx
> 
> I'm not sure what's going on here, but this is likely a leftover for
> the change to string literals match Python-level 'str' type. Stefan, I
> need you help here.
> 
> Stefan, Can you look at (3) ? I bet you will figure a fix in far less
> time than me :-)

Fixed.

I remember several other test failures in the C++ tests, though, mostly
related to byte string output.

Stefan


From dalcinl at gmail.com  Wed Feb 17 14:15:49 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 17 Feb 2010 10:15:49 -0300
Subject: [Cython] Patch against Python 3.x to run Cython
In-Reply-To: <4B7BAA46.7050800@behnel.de>
References: <4B7AD0BD.4050706@behnel.de>
	<e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>
	<4B7BAA46.7050800@behnel.de>
Message-ID: <e7ba66e41002170515kcb0a83ci74df140d2c1e0b6c@mail.gmail.com>

On 17 February 2010 05:35, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 16.02.2010 21:52:
>> 2) tests/run/extern_builtins_T258.pyx
>>
>> I simply propose to remove the declaration of __builtin__.str from the
>> testcase (and use Py_SIZE(L) instead of L.ob_size).
>
> Sounds reasonable, I've done that.
>

Ups! As commented in other post, PyListObject.allocated is not in
Python 2.3 ...


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 17 14:20:00 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 17 Feb 2010 10:20:00 -0300
Subject: [Cython] Parsing.py and Python 2.3
In-Reply-To: <4B7B9562.3080902@behnel.de>
References: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>
	<80282B91-7488-44FF-AAEB-B8BAE8412791@math.washington.edu>
	<4B7B9562.3080902@behnel.de>
Message-ID: <e7ba66e41002170520p29e9bf9l14d7f212b18e7b0@mail.gmail.com>

On 17 February 2010 04:06, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 16.02.2010 22:24:
>> On Feb 16, 2010, at 1:08 PM, Lisandro Dalcin wrote:
>>
>>> That file lost backward-compatibility with Py2.3 (because it uses
>>> 'set' builtin). As Parsin.py is one of these few files that are
>>> cythonized,... Can you see any problem on the obvious patch below??:
>>>
>>>
>>> $ hg diff Cython/Compiler/Parsing.py
>>> diff -r 6fc30b5dadc3 Cython/Compiler/Parsing.py
>>> --- a/Cython/Compiler/Parsing.py ? ? Fri Feb 12 19:04:24 2010 +0100
>>> +++ b/Cython/Compiler/Parsing.py ? ? Tue Feb 16 18:06:03 2010 -0300
>>> @@ -10,6 +10,13 @@
>>> import os
>>> import re
>>> import sys
>>> +
>>> +try:
>>> + ? ?set
>>> +except NameError:
>>> + ? ?# Python 2.3
>>> + ? ?from sets import Set as set
>>> +
>>> from Cython.Compiler.Scanning import PyrexScanner,
>>> FileSourceDescriptor
>>> import Nodes
>>> import ExprNodes
>>
>> Looks fine to me. We do the same elsewhere.
>
> This shadows the 'set' builtin, though, so Cython can't optimise it. That
> may not be important in this case, but given that Cython "implements" the
> set type for 2.3 anyway, would it make sense to provide the 'set' builtin
> in Shadow.py and reimport it from there instead?
>

Stefan, I general I do not like the idea loosing a any optimization in
ALL Python versions just for providing compat for the old Py2.3.

Could you elaborate a bit more what should be done to make it work
using Shadow.py ? I mean, adding 'set' to Shadow.py is trivial, but I
do not get how to use it in Parsing.py...




-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Feb 17 14:29:10 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Feb 2010 14:29:10 +0100
Subject: [Cython] Parsing.py and Python 2.3
In-Reply-To: <e7ba66e41002170520p29e9bf9l14d7f212b18e7b0@mail.gmail.com>
References: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>	<80282B91-7488-44FF-AAEB-B8BAE8412791@math.washington.edu>	<4B7B9562.3080902@behnel.de>
	<e7ba66e41002170520p29e9bf9l14d7f212b18e7b0@mail.gmail.com>
Message-ID: <4B7BEF26.1000208@behnel.de>

Lisandro Dalcin, 17.02.2010 14:20:
> On 17 February 2010 04:06, Stefan Behnel wrote:
>>> On Feb 16, 2010, at 1:08 PM, Lisandro Dalcin wrote:
>>>> +try:
>>>> +    set
>>>> +except NameError:
>>>> +    # Python 2.3
>>>> +    from sets import Set as set
>> This shadows the 'set' builtin, though, so Cython can't optimise it. That
>> may not be important in this case, but given that Cython "implements" the
>> set type for 2.3 anyway, would it make sense to provide the 'set' builtin
>> in Shadow.py and reimport it from there instead?
> 
> Stefan, I general I do not like the idea loosing a any optimization in
> ALL Python versions just for providing compat for the old Py2.3.

That's exactly what this is meant to prevent.


> Could you elaborate a bit more what should be done to make it work
> using Shadow.py ? I mean, adding 'set' to Shadow.py is trivial, but I
> do not get how to use it in Parsing.py...

You'd do

    from cython import set

in Parsing.py. Cython would then consider that the true 'set' builtin at
compile time, regardless of the Python environment, whereas an uncompiled
Cython would simply let Python import 'set' from Cython's Shadow module and
be happy.

Stefan

From stefan_ml at behnel.de  Wed Feb 17 14:30:37 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Feb 2010 14:30:37 +0100
Subject: [Cython] Patch against Python 3.x to run Cython
In-Reply-To: <e7ba66e41002170515kcb0a83ci74df140d2c1e0b6c@mail.gmail.com>
References: <4B7AD0BD.4050706@behnel.de>	<e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>	<4B7BAA46.7050800@behnel.de>
	<e7ba66e41002170515kcb0a83ci74df140d2c1e0b6c@mail.gmail.com>
Message-ID: <4B7BEF7D.4070501@behnel.de>

Lisandro Dalcin, 17.02.2010 14:15:
> On 17 February 2010 05:35, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Lisandro Dalcin, 16.02.2010 21:52:
>>> 2) tests/run/extern_builtins_T258.pyx
>>>
>>> I simply propose to remove the declaration of __builtin__.str from the
>>> testcase (and use Py_SIZE(L) instead of L.ob_size).
>> Sounds reasonable, I've done that.
>>
> 
> Ups! As commented in other post, PyListObject.allocated is not in
> Python 2.3 ...

Great.

Then what about disabling that test for 2.3? I don't see an immediate way
to make it work in any sensible way on that platform.

Stefan

From dalcinl at gmail.com  Wed Feb 17 14:51:07 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 17 Feb 2010 10:51:07 -0300
Subject: [Cython] Patch against Python 3.x to run Cython
In-Reply-To: <4B7BEF7D.4070501@behnel.de>
References: <4B7AD0BD.4050706@behnel.de>
	<e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>
	<4B7BAA46.7050800@behnel.de>
	<e7ba66e41002170515kcb0a83ci74df140d2c1e0b6c@mail.gmail.com>
	<4B7BEF7D.4070501@behnel.de>
Message-ID: <e7ba66e41002170551n282f4e1cy1274b4df424881da@mail.gmail.com>

On 17 February 2010 10:30, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 17.02.2010 14:15:
>> On 17 February 2010 05:35, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Lisandro Dalcin, 16.02.2010 21:52:
>>>> 2) tests/run/extern_builtins_T258.pyx
>>>>
>>>> I simply propose to remove the declaration of __builtin__.str from the
>>>> testcase (and use Py_SIZE(L) instead of L.ob_size).
>>> Sounds reasonable, I've done that.
>>>
>>
>> Ups! As commented in other post, PyListObject.allocated is not in
>> Python 2.3 ...
>
> Great.
>
> Then what about disabling that test for 2.3? I don't see an immediate way
> to make it work in any sensible way on that platform.
>

+1


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 17 15:07:18 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 17 Feb 2010 11:07:18 -0300
Subject: [Cython] Parsing.py and Python 2.3
In-Reply-To: <4B7BEF26.1000208@behnel.de>
References: <e7ba66e41002161308s15b3e14dv187f17a1f810249c@mail.gmail.com>
	<80282B91-7488-44FF-AAEB-B8BAE8412791@math.washington.edu>
	<4B7B9562.3080902@behnel.de>
	<e7ba66e41002170520p29e9bf9l14d7f212b18e7b0@mail.gmail.com>
	<4B7BEF26.1000208@behnel.de>
Message-ID: <e7ba66e41002170607s495a1376v91152258e0ff6eb4@mail.gmail.com>

On 17 February 2010 10:29, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> You'd do
>
> ? ?from cython import set
>
> in Parsing.py. Cython would then consider that the true 'set' builtin at
> compile time, regardless of the Python environment, whereas an uncompiled
> Cython would simply let Python import 'set' from Cython's Shadow module and
> be happy.
>

OK. But where should I hack to make "from cython import set" to be
ignored (or, i.o.w, to use the builtin) in the generated C code?

-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Feb 17 15:08:17 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Feb 2010 15:08:17 +0100
Subject: [Cython] Patch against Python 3.x to run Cython
In-Reply-To: <e7ba66e41002170551n282f4e1cy1274b4df424881da@mail.gmail.com>
References: <4B7AD0BD.4050706@behnel.de>	<e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>	<4B7BAA46.7050800@behnel.de>	<e7ba66e41002170515kcb0a83ci74df140d2c1e0b6c@mail.gmail.com>	<4B7BEF7D.4070501@behnel.de>
	<e7ba66e41002170551n282f4e1cy1274b4df424881da@mail.gmail.com>
Message-ID: <4B7BF851.5030907@behnel.de>

Lisandro Dalcin, 17.02.2010 14:51:
> On 17 February 2010 10:30, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Lisandro Dalcin, 17.02.2010 14:15:
>>> On 17 February 2010 05:35, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Lisandro Dalcin, 16.02.2010 21:52:
>>>>> 2) tests/run/extern_builtins_T258.pyx
>>>>>
>>>>> I simply propose to remove the declaration of __builtin__.str from the
>>>>> testcase (and use Py_SIZE(L) instead of L.ob_size).
>>>> Sounds reasonable, I've done that.
>>>>
>>> Ups! As commented in other post, PyListObject.allocated is not in
>>> Python 2.3 ...
>> Great.
>>
>> Then what about disabling that test for 2.3? I don't see an immediate way
>> to make it work in any sensible way on that platform.
> 
> +1

Done. Please test, I don't currently have 2.3 available.

Stefan


From dalcinl at gmail.com  Wed Feb 17 15:17:52 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 17 Feb 2010 11:17:52 -0300
Subject: [Cython] Patch against Python 3.x to run Cython
In-Reply-To: <4B7BF851.5030907@behnel.de>
References: <4B7AD0BD.4050706@behnel.de>
	<e7ba66e41002161252o11cf23a1p10479e115054bc80@mail.gmail.com>
	<4B7BAA46.7050800@behnel.de>
	<e7ba66e41002170515kcb0a83ci74df140d2c1e0b6c@mail.gmail.com>
	<4B7BEF7D.4070501@behnel.de>
	<e7ba66e41002170551n282f4e1cy1274b4df424881da@mail.gmail.com>
	<4B7BF851.5030907@behnel.de>
Message-ID: <e7ba66e41002170617j6f0e7fbfv9de2dc3e16cad0d5@mail.gmail.com>

On 17 February 2010 11:08, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 17.02.2010 14:51:
>> On 17 February 2010 10:30, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Lisandro Dalcin, 17.02.2010 14:15:
>>>> On 17 February 2010 05:35, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>> Lisandro Dalcin, 16.02.2010 21:52:
>>>>>> 2) tests/run/extern_builtins_T258.pyx
>>>>>>
>>>>>> I simply propose to remove the declaration of __builtin__.str from the
>>>>>> testcase (and use Py_SIZE(L) instead of L.ob_size).
>>>>> Sounds reasonable, I've done that.
>>>>>
>>>> Ups! As commented in other post, PyListObject.allocated is not in
>>>> Python 2.3 ...
>>> Great.
>>>
>>> Then what about disabling that test for 2.3? I don't see an immediate way
>>> to make it work in any sensible way on that platform.
>>
>> +1
>
> Done. Please test, I don't currently have 2.3 available.
>

It worked. Many thanks.


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 17 15:51:09 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 17 Feb 2010 11:51:09 -0300
Subject: [Cython] Why the file shorter_strings.diff was checked-in?
Message-ID: <e7ba66e41002170651i4c15b6d2k1f3a22072a9bd619@mail.gmail.com>

Unless I'm missing something, that patch file should not be there...

-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed Feb 17 18:49:56 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Feb 2010 09:49:56 -0800
Subject: [Cython] Why the file shorter_strings.diff was checked-in?
In-Reply-To: <e7ba66e41002170651i4c15b6d2k1f3a22072a9bd619@mail.gmail.com>
References: <e7ba66e41002170651i4c15b6d2k1f3a22072a9bd619@mail.gmail.com>
Message-ID: <2B81879D-CE25-4112-B7D4-332CC580F93B@math.washington.edu>

On Feb 17, 2010, at 6:51 AM, Lisandro Dalcin wrote:

> Unless I'm missing something, that patch file should not be there...

Sorry, my bad. I was pulling the patch from the email and didn't clean  
it up. Gone now.

- Robert


From dalcinl at gmail.com  Wed Feb 17 23:38:38 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 17 Feb 2010 19:38:38 -0300
Subject: [Cython] public versus api, dll import/export and api import/export
Message-ID: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>

Consider the following stuff:

cdef            void* spam0(object o) except NULL: return NULL
cdef public     void* spam1(object o) except NULL: return NULL
cdef        api void* spam2(object o) except NULL: return NULL
cdef public api void* spam3(object o) except NULL: return NULL

cdef                class Foo0: pass
ctypedef public     class Foo1 [type PyFoo1_Type, object PyFoo1_Object]: pass
ctypedef        api class Foo2 [type PyFoo2_Type, object PyFoo2_Object]: pass
ctypedef public api class Foo3 [type PyFoo3_Type, object PyFoo3_Object]: pass

I thing all should be valid, however "ctypedef api class Foo2" is not
the case. Fixing this is more or less easy...

What is not clear to me is the following: Should "public", "api", or
"public api" have any semantic diference? We could argue that "api" is
related to C-API exporting the Python way (using capsule/cobject), and
that "public" is related to the linker-way, using DLL export
mechanisms (__declspec(dllexport) or GCC visibility attributes). In
such view, the two mechanisms are independent and can coexist, so a
"public api" function or type could mean that we want BOTH mechanisms
for exporting stuff... That way, these functions/types could be
accessed via ext module interlinking (pretty easy in Linux, not sure
about OS X or WinDog), or using the very portable Python way using
capsules/cobjects (however, this way requires requires some
hand-written C to get the symbols, unless you use Cython, of course,
but think about other wrapper tools).

However, I'll refresh your memory about some Greg's comments some time
ago. Greg said that he never intended to provide DLL export
mechanisms, and in fact he just wanted to make stuff available across
compilation units (i.e, make stuff available to other C source files
being compiled ALONGSIDE the generated C sources to build the final
ext module). If we enforce this, I mean we are not going to support
ext module interlinking, then almost the usages of DL_IMPORT/DL_EXPORT
macros are a nonsense and should be removed, alongside almost all
these dll_linkage arguments in many methods.

The whole import/export stuff is a bit of mess, and I would really
want to clean this up... IMHO, it would be nice to support ext module
interlinking (probably just because I'm a Linux user and do not want
to restrict my favorite platform). But if you think this is not worth
the effort, then I'm fine too, and then "public" and "api" would mean
basically the same: export/import stuff using capsules/cobjects.

So... What do you think?



-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Feb 18 08:27:38 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Feb 2010 08:27:38 +0100
Subject: [Cython] public versus api,
	dll import/export and api import/export
In-Reply-To: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
Message-ID: <4B7CEBEA.7060107@behnel.de>

Lisandro Dalcin, 17.02.2010 23:38:
> What is not clear to me is the following: Should "public", "api", or
> "public api" have any semantic diference?
> [...]
> Greg said that he never intended to provide DLL export
> mechanisms, and in fact he just wanted to make stuff available across
> compilation units (i.e, make stuff available to other C source files
> being compiled ALONGSIDE the generated C sources to build the final
> ext module). If we enforce this, I mean we are not going to support
> ext module interlinking, then almost the usages of DL_IMPORT/DL_EXPORT
> macros are a nonsense and should be removed, alongside almost all
> these dll_linkage arguments in many methods.
> [...]
> So... What do you think?

My take is that public symbols are only portably usable at static link
time, so supporting more is simply not worth it. Even static linking
against symbols exported by a Cython module should be a very rare use case.
It's not used when calling external C code nor is it worth anything when
providing callbacks into the Cython code. Most of the time, the external
code will be there first and will be used by the Cython code, not the other
way round. This mechanism shouldn't be seen as something that's usable at
dynamic linking time. There's public C-API support for that. I second your
intuition that the DL_EXPORT stuff can be dropped completely.

There's also the use case of statically linking multiple Cython modules
into one extension module. This isn't currently supported (really), and
there's more to do to make it run smooth. But I don't think it would
interfere with the above in any way.

Stefan

From dalcinl at gmail.com  Thu Feb 18 16:03:09 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 18 Feb 2010 12:03:09 -0300
Subject: [Cython] cannot compile mpi4py with current cython-devel
Message-ID: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>

Error converting Pyrex file to C:
------------------------------------------------------------
...
    int MPI_Group_union(MPI_Group, MPI_Group, MPI_Group*)
    int MPI_Group_intersection(MPI_Group, MPI_Group, MPI_Group*)
    int MPI_Group_difference(MPI_Group, MPI_Group, MPI_Group*)
    int MPI_Group_incl(MPI_Group, int, int[], MPI_Group*)
    int MPI_Group_excl(MPI_Group, int, int[], MPI_Group*)
    int MPI_Group_range_incl(MPI_Group, int, int[][3], MPI_Group*)
                                                 ^
------------------------------------------------------------

include/mpi4py/mpi.pxi:325:50: Array element type 'int []' is incomplete



-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu Feb 18 16:04:47 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Feb 2010 16:04:47 +0100
Subject: [Cython] cannot compile mpi4py with current cython-devel
In-Reply-To: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
References: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
Message-ID: <4B7D570F.7040900@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>     int MPI_Group_union(MPI_Group, MPI_Group, MPI_Group*)
>     int MPI_Group_intersection(MPI_Group, MPI_Group, MPI_Group*)
>     int MPI_Group_difference(MPI_Group, MPI_Group, MPI_Group*)
>     int MPI_Group_incl(MPI_Group, int, int[], MPI_Group*)
>     int MPI_Group_excl(MPI_Group, int, int[], MPI_Group*)
>     int MPI_Group_range_incl(MPI_Group, int, int[][3], MPI_Group*)
>                                                  ^
> ------------------------------------------------------------
>
> include/mpi4py/mpi.pxi:325:50: Array element type 'int []' is incomplete
>
>
>
>   
Just wanted to note that gsoc-kurt isn't merged yet so that isn't it.

Dag Sverre

From dalcinl at gmail.com  Thu Feb 18 17:23:17 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 18 Feb 2010 13:23:17 -0300
Subject: [Cython] cannot compile mpi4py with current cython-devel
In-Reply-To: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
References: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
Message-ID: <e7ba66e41002180823w6cf4a6d5ua3a3111a350136f0@mail.gmail.com>

2010/2/18 Lisandro Dalcin <dalcinl at gmail.com>:
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> ? ?int MPI_Group_union(MPI_Group, MPI_Group, MPI_Group*)
> ? ?int MPI_Group_intersection(MPI_Group, MPI_Group, MPI_Group*)
> ? ?int MPI_Group_difference(MPI_Group, MPI_Group, MPI_Group*)
> ? ?int MPI_Group_incl(MPI_Group, int, int[], MPI_Group*)
> ? ?int MPI_Group_excl(MPI_Group, int, int[], MPI_Group*)
> ? ?int MPI_Group_range_incl(MPI_Group, int, int[][3], MPI_Group*)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
> ------------------------------------------------------------
>
> include/mpi4py/mpi.pxi:325:50: Array element type 'int []' is incomplete
>

OK, here seems to be the problem... Robert?

changeset:   2948:fa6c25298b0d
user:        Robert Bradshaw <robertwb at math.washington.edu>
date:        Wed Feb 10 01:27:24 2010 -0800
summary:     More buffer type analysis deferment.



-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jotadram6 at gmail.com  Thu Feb 18 17:06:13 2010
From: jotadram6 at gmail.com (Jose David Ruiz Alvarez)
Date: Thu, 18 Feb 2010 11:06:13 -0500
Subject: [Cython] Trying to use a C library taht is also compiled in fortran.
Message-ID: <df41bc7b1002180806k426f3e9ex6f1e94249c694799@mail.gmail.com>

I'm new in Cython, but I'm trying to get direct acces from python to a C and
Fortran code called micrOMEGAS. I've studied the program itself and some
characteristics about cython, but I'm not succesful, yet.

I need a medium example of how to use a C library in Cython, how to make the
calls of the functions, the declarations and all over ?Anybody can send me a
moderate level and a high level example?, also I have problems declaring in
the .pxd file of cython a variable of FILE type ?Wht can I do with it?

Thanks,

-- 
*************************************
Jos? David Ruiz ?lvarez
Profesor de Ac?stica
Universidad de Antioquia
*************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20100218/b7e65561/attachment.htm 

From stefan_ml at behnel.de  Thu Feb 18 18:21:01 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Feb 2010 18:21:01 +0100
Subject: [Cython] Trying to use a C library taht is also compiled in
	fortran.
In-Reply-To: <df41bc7b1002180806k426f3e9ex6f1e94249c694799@mail.gmail.com>
References: <df41bc7b1002180806k426f3e9ex6f1e94249c694799@mail.gmail.com>
Message-ID: <4B7D76FD.2060807@behnel.de>

Jose David Ruiz Alvarez, 18.02.2010 17:06:
> I'm new in Cython, but I'm trying to get direct acces from python to a C and
> Fortran code called micrOMEGAS. I've studied the program itself and some
> characteristics about cython, but I'm not succesful, yet.
> 
> I need a medium example of how to use a C library in Cython, how to make the
> calls of the functions, the declarations and all over ?Anybody can send me a
> moderate level and a high level example?

Have you looked through the docs and the tutorials on the web site?

Please note that the right place to ask this would be the cython-users
mailing list, not the developers mailing list.


> also I have problems declaring in
> the .pxd file of cython a variable of FILE type ?Wht can I do with it?

It's predeclared in stdio.pxd and can be used with fopen() or fprintf().
The question is rather: what do you *want* to do with it?

Stefan

From robertwb at math.washington.edu  Fri Feb 19 03:01:19 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Feb 2010 18:01:19 -0800
Subject: [Cython] cannot compile mpi4py with current cython-devel
In-Reply-To: <e7ba66e41002180823w6cf4a6d5ua3a3111a350136f0@mail.gmail.com>
References: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
	<e7ba66e41002180823w6cf4a6d5ua3a3111a350136f0@mail.gmail.com>
Message-ID: <6149CA6A-3217-4E0C-AC19-88651985EDBD@math.washington.edu>

On Feb 18, 2010, at 8:23 AM, Lisandro Dalcin wrote:

> 2010/2/18 Lisandro Dalcin <dalcinl at gmail.com>:
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>>    int MPI_Group_union(MPI_Group, MPI_Group, MPI_Group*)
>>    int MPI_Group_intersection(MPI_Group, MPI_Group, MPI_Group*)
>>    int MPI_Group_difference(MPI_Group, MPI_Group, MPI_Group*)
>>    int MPI_Group_incl(MPI_Group, int, int[], MPI_Group*)
>>    int MPI_Group_excl(MPI_Group, int, int[], MPI_Group*)
>>    int MPI_Group_range_incl(MPI_Group, int, int[][3], MPI_Group*)
>>                                                 ^
>> ------------------------------------------------------------
>>
>> include/mpi4py/mpi.pxi:325:50: Array element type 'int []' is  
>> incomplete
>>
>
> OK, here seems to be the problem... Robert?
>
> changeset:   2948:fa6c25298b0d
> user:        Robert Bradshaw <robertwb at math.washington.edu>
> date:        Wed Feb 10 01:27:24 2010 -0800
> summary:     More buffer type analysis deferment.

That's probably the one. I thought I had tested all the corner cases,  
guess not. Fixed at http://hg.cython.org/cython-devel/rev/9363c2b0d737  
(with tests at http://hg.cython.org/cython-devel/rev/5ff8e256271f ).

- Robert


From greg.ewing at canterbury.ac.nz  Fri Feb 19 06:01:16 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 19 Feb 2010 18:01:16 +1300
Subject: [Cython] cannot compile mpi4py with current cython-devel
In-Reply-To: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
References: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
Message-ID: <4B7E1B1C.6000907@canterbury.ac.nz>

Lisandro Dalcin wrote:
>     int MPI_Group_range_incl(MPI_Group, int, int[][3], MPI_Group*)
>                                                  ^
> ------------------------------------------------------------
> 
> include/mpi4py/mpi.pxi:325:50: Array element type 'int []' is incomplete

Works in Pyrex, so it must be something Cython introduced.

-- 
Greg

From robertwb at math.washington.edu  Fri Feb 19 06:21:10 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Feb 2010 21:21:10 -0800
Subject: [Cython] cannot compile mpi4py with current cython-devel
In-Reply-To: <4B7E1B1C.6000907@canterbury.ac.nz>
References: <e7ba66e41002180703w2d14728u789820474e320752@mail.gmail.com>
	<4B7E1B1C.6000907@canterbury.ac.nz>
Message-ID: <34709824-FB00-4EB9-9965-E34F273121B3@math.washington.edu>

On Feb 18, 2010, at 9:01 PM, Greg Ewing wrote:

> Lisandro Dalcin wrote:
>>    int MPI_Group_range_incl(MPI_Group, int, int[][3], MPI_Group*)
>>                                                 ^
>> ------------------------------------------------------------
>>
>> include/mpi4py/mpi.pxi:325:50: Array element type 'int []' is  
>> incomplete
>
> Works in Pyrex, so it must be something Cython introduced.

Yep, this was introduced very recently (about a week ago) to fix some  
stuff with templated C++ type parsing, and isn't in any release  
versions (nor will it be, as it's fixed and there's a regression test  
now...)

- Robert


From robertwb at math.washington.edu  Sun Feb 21 06:22:28 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 20 Feb 2010 21:22:28 -0800
Subject: [Cython] Namespaces sytnax
In-Reply-To: <4B72735A.5070102@student.matnat.uio.no>
References: <2A1B9689-986B-4294-9A00-6F53D5913CE2@math.washington.edu>
	<4B725994.4080508@behnel.de>
	<4B72735A.5070102@student.matnat.uio.no>
Message-ID: <F09B4651-8C17-47BA-B0E9-2123189E07B6@math.washington.edu>

On Feb 10, 2010, at 12:50 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Robert Bradshaw, 10.02.2010 06:47:
>>
>>> cdef extern from "foo.h" namespace "Foo::Bar":
>>>     int f(int)
>>>
>>> cdef extern from "..." namespace "A::B" as X:
>>>     int x(int)
>>>
>>> print X.x(5)
>>>
>>
>> +1 for these two, the first meaning that the names get defined in  
>> the flat
>> module namespace.
>>
> I'm fine with this as well.

OK, namespaces are now declared as strings rather than dotted names.  
The "as" syntax is not yet implemented, it's much lower priority (and  
is actually orthogonal to the namespace specification, as it could  
apply to any extern block).

As for fitting the cname pattern, having to resort to cnames for  
common use cases is something we're trying to move away from (e.g. one  
of the big motivations for C++ support).

- Robert


From robertwb at math.washington.edu  Sun Feb 21 06:29:42 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 20 Feb 2010 21:29:42 -0800
Subject: [Cython] Generate shorter string literals
In-Reply-To: <4B7035A4.1000404@behnel.de>
References: <4B6C3F1E.5030202@uci.edu> <4B6F1D6B.8020700@behnel.de>
	<4B6FCA70.8040200@student.matnat.uio.no>
	<4B7035A4.1000404@behnel.de>
Message-ID: <CCBDB04A-6078-48BF-A646-DE49DE0194A3@math.washington.edu>

On Feb 8, 2010, at 8:02 AM, Stefan Behnel wrote:

> Dag Sverre Seljebotn, 08.02.2010 09:25:
>> Stefan Behnel wrote:
>>> I'm surprised that this works. The last information I had from an  
>>> MSVC user
>>> (and from a web search) was that just splitting the string into  
>>> separate
>>> auto-concatenated literals wouldn't solve the issue.
>>>
>>> Happy to read that it does.
>>>
>> Cristoph: Which version of MSVC are you using/did you/are you able to
>> test this on?
>
> He mentioned MSVC V9. Anyway, it doesn't break anything, so it's  
> best to
> just always do it that way even if it changes nothing for older MSVC  
> releases.

IIRC, there's also a total string size limit as well, but it's much  
larger. Technically ANSI C says that the maximum string literal length  
may be as low as 509 characters, *after* concatenation, but that's  
being really pedantic and not worth doing anything about IMHO.

- Robert



From robertwb at math.washington.edu  Sun Feb 21 06:43:13 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 20 Feb 2010 21:43:13 -0800
Subject: [Cython] C++ method overloading?
In-Reply-To: <4B71286B.5020805@behnel.de>
References: <4B702EB2.9050407@behnel.de>	<941DA66D-E140-4BA1-A1E6-0A755DF35B97@math.washington.edu>
	<56342EAE-423E-4EAF-A9D0-AE51F715C80B@math.washington.edu>
	<4B71286B.5020805@behnel.de>
Message-ID: <3D0A3A12-0100-4707-8C7F-5E1FF696032F@math.washington.edu>

On Feb 9, 2010, at 1:18 AM, Stefan Behnel wrote:

>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> cdef class Multiplier:
>    cdef testapi.Multiplier* multiplier
>
>    def __cinit__(self, factor=None):
>        if factor is None:
>            self.multiplier = new testapi.Multiplier()
>                                                     ^
> ------------------------------------------------------------
>
> .../pytestapi.pyx:9:54: Expected ')'
>
> Totally weird kind of error, given that it points to the closing
> parenthesis already.

This should work as expected now in cython-devel.

- Robert


From nick at boxdesign.co.uk  Mon Feb 22 21:11:28 2010
From: nick at boxdesign.co.uk (Nick Joyce)
Date: Mon, 22 Feb 2010 15:11:28 -0500
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
Message-ID: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>

I have been using Cython for some time to develop the PyAMF c-extensions and have found it to be an awesome project! Recently, I discovered Python type declarations e.g.:

def object foo():
    cdef dict x = {}

    x['foo'] = 'bar'

    return x['foo']

The current PyAMF code is littered with PyDict_* calls so I am in the process of refactoring it to make use of this syntactic sugar.

When I generate the C code (using Cython 0.12.1) for the above function, the resulting C code will use PyDict_SetItem for the assignment statement but will use PyObject_GetItem on line 6. I have attached a snippet of the generated C [pyobject_getitem.c].

Would it make more sense to use PyDict_SetItem in this case? I have included a patch for this case [getitem.diff] if its useful. I haven't developed a test for this change, some pointers as to the correct location would be useful if that is desirable.

Cython makes it possible for me to build c based extensions really quickly and the speedups have been dramatic! Thanks again!

Nick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: pyobject_getitem.c
Type: application/octet-stream
Size: 759 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100222/73d6ca02/attachment.obj 
-------------- next part --------------
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: getitem.diff
Type: application/octet-stream
Size: 631 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100222/73d6ca02/attachment-0001.obj 

From robertwb at math.washington.edu  Mon Feb 22 23:49:23 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 22 Feb 2010 14:49:23 -0800
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
Message-ID: <0B78232F-4C35-4F0C-AD2D-00FC6225D1FA@math.washington.edu>

On Feb 22, 2010, at 12:11 PM, Nick Joyce wrote:

> I have been using Cython for some time to develop the PyAMF c- 
> extensions and have found it to be an awesome project!

Thanks!

> Recently, I discovered Python type declarations e.g.:
>
> def object foo():
>    cdef dict x = {}
>
>    x['foo'] = 'bar'
>
>    return x['foo']
>
> The current PyAMF code is littered with PyDict_* calls so I am in  
> the process of refactoring it to make use of this syntactic sugar.
>
> When I generate the C code (using Cython 0.12.1) for the above  
> function, the resulting C code will use PyDict_SetItem for the  
> assignment statement but will use PyObject_GetItem on line 6. I have  
> attached a snippet of the generated C [pyobject_getitem.c].
>
> Would it make more sense to use PyDict_SetItem in this case? I have  
> included a patch for this case [getitem.diff] if its useful. I  
> haven't developed a test for this change, some pointers as to the  
> correct location would be useful if that is desirable.

Yes, we would certainly like to do this. The fix isn't quite this  
simple though, as we need to make sure x is not None before caling  
PyDict_GetItem on it, NULL is returned if the key is not found  
(without raising an error), and the returned reference is borrowed  
(which changes reference counting semantics).

- Robert


From tjhnson at gmail.com  Tue Feb 23 00:42:11 2010
From: tjhnson at gmail.com (T J)
Date: Mon, 22 Feb 2010 15:42:11 -0800
Subject: [Cython] Cython and numpy include files
Message-ID: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>

Hi,

I have numpy installed system-wide, but I also have my own (svn)
version in ~/.local.  When I compile cython modules which make use of
the NumPy array access, it needs to reference the NumPy include files.
 These are located here:

   ~/.local/lib/python2.6/site-packages/numpy/core/include/numpy/

The "problem" I am having is that when I compile the cython extension,
it is using libraries located here:

  /usr/include/python2.6/numpy

This is inconsistent since when I import numpy, it is definitely using
the version in ~/.local.   I want to avoid setting INCLUDE_DIRS in my
.bashrc.  An obvious solution is that I should specify the proper
include directory when I create the Extension instance in my setup.py.
 Something like:

  d = os.path.join(os.path.split(numpy.__file__)[0], "core", "include")
  Extension(... , include_dirs=[d])

Is this the proper/recommended way to handle this?  More to the point,
it seems like this should be done for the users.  Perhaps
numpy.distutils.extention.Extension should append the include path for
NumPy.  Or perhaps Cython can do this for the users?

From robert.kern at gmail.com  Tue Feb 23 01:21:06 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Mon, 22 Feb 2010 18:21:06 -0600
Subject: [Cython] Cython and numpy include files
In-Reply-To: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>
References: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>
Message-ID: <hlv71j$q4c$1@dough.gmane.org>

On 2010-02-22 17:42 PM, T J wrote:
> Hi,
>
> I have numpy installed system-wide, but I also have my own (svn)
> version in ~/.local.  When I compile cython modules which make use of
> the NumPy array access, it needs to reference the NumPy include files.
>   These are located here:
>
>     ~/.local/lib/python2.6/site-packages/numpy/core/include/numpy/
>
> The "problem" I am having is that when I compile the cython extension,
> it is using libraries located here:
>
>    /usr/include/python2.6/numpy
>
> This is inconsistent since when I import numpy, it is definitely using
> the version in ~/.local.   I want to avoid setting INCLUDE_DIRS in my
> .bashrc.  An obvious solution is that I should specify the proper
> include directory when I create the Extension instance in my setup.py.
>   Something like:
>
>    d = os.path.join(os.path.split(numpy.__file__)[0], "core", "include")
>    Extension(... , include_dirs=[d])
>
> Is this the proper/recommended way to handle this?  More to the point,
> it seems like this should be done for the users.  Perhaps
> numpy.distutils.extention.Extension should append the include path for
> NumPy.

Other parts of numpy.distutils handle that. Unfortunately, those are the parts 
that conflict with Cython's distutils extensions. Just use

   include_dirs=[numpy.get_include()]

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From cournape at gmail.com  Tue Feb 23 02:10:36 2010
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 23 Feb 2010 10:10:36 +0900
Subject: [Cython] Cython and numpy include files
In-Reply-To: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>
References: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>
Message-ID: <5b8d13221002221710p7bffd4d4w9ddfc0121d1f6bfc@mail.gmail.com>

On Tue, Feb 23, 2010 at 8:42 AM, T J <tjhnson at gmail.com> wrote:
> Hi,
>
> I have numpy installed system-wide, but I also have my own (svn)
> version in ~/.local. ?When I compile cython modules which make use of
> the NumPy array access, it needs to reference the NumPy include files.
> ?These are located here:
>
> ? ~/.local/lib/python2.6/site-packages/numpy/core/include/numpy/
>
> The "problem" I am having is that when I compile the cython extension,
> it is using libraries located here:
>
> ?/usr/include/python2.6/numpy

In addition to what Robert said, note that the above is a downstream
modification. By putting headers in /usr/include instead of
$site-packages/numpy, the system-wide numpy headers are included
whenever the Python.h is, which is often before the numpy headers (the
ones we want).

This problem is aggravated by the fact that there is no easy way to
control header path inclusion in distutils: paths are added through
add_include_dirs (which *append* paths), but the default should
actually be prepending. Maybe we should change this internally in
numpy.distutils, but I am a bit worried about the consequences,

cheers,

David

From dalcinl at gmail.com  Tue Feb 23 03:07:39 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 22 Feb 2010 23:07:39 -0300
Subject: [Cython] Cython and numpy include files
In-Reply-To: <5b8d13221002221710p7bffd4d4w9ddfc0121d1f6bfc@mail.gmail.com>
References: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>
	<5b8d13221002221710p7bffd4d4w9ddfc0121d1f6bfc@mail.gmail.com>
Message-ID: <e7ba66e41002221807y71e945dat976f29217085eff5@mail.gmail.com>

On 22 February 2010 22:10, David Cournapeau <cournape at gmail.com> wrote:
> On Tue, Feb 23, 2010 at 8:42 AM, T J <tjhnson at gmail.com> wrote:
>> Hi,
>>
>> I have numpy installed system-wide, but I also have my own (svn)
>> version in ~/.local. ?When I compile cython modules which make use of
>> the NumPy array access, it needs to reference the NumPy include files.
>> ?These are located here:
>>
>> ? ~/.local/lib/python2.6/site-packages/numpy/core/include/numpy/
>>
>> The "problem" I am having is that when I compile the cython extension,
>> it is using libraries located here:
>>
>> ?/usr/include/python2.6/numpy
>
> In addition to what Robert said, note that the above is a downstream
> modification. By putting headers in /usr/include instead of
> $site-packages/numpy, the system-wide numpy headers are included
> whenever the Python.h is, which is often before the numpy headers (the
> ones we want).
>

Indeed. For some reason, packagers think that developers are idiot,
naive, laizy, whatever and freely make ANY modification they feel it
makes sense, without ever dropping you a single mail commenting on the
changes... Then you start receiving mails about issues with your
code... For example, Cython users could take advantage of
__init__.[pyx|pxd] to signal packages.. but be carefull, perhaps you
should at least add a single comment line in the file, never let it be
pass empty to downstream, as a packager could remove empty files
(because rpmlint complains about that)

> This problem is aggravated by the fact that there is no easy way to
> control header path inclusion in distutils: paths are added through
> add_include_dirs (which *append* paths), but the default should
> actually be prepending. Maybe we should change this internally in
> numpy.distutils, but I am a bit worried about the consequences,
>

IIRC, the same applies to library paths... So you could have a hard
time using a custom, shared-lib enabled Python build with version
matching the system Python (and BTW, "/usr/lib" should NEVER be
added)... Someone should definitely bug distutils about this... But
you know, one has to be brave and patient enough to fight in the bug
tracker or python-dev :-)


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From nick at boxdesign.co.uk  Tue Feb 23 05:04:01 2010
From: nick at boxdesign.co.uk (Nick Joyce)
Date: Mon, 22 Feb 2010 23:04:01 -0500
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <0B78232F-4C35-4F0C-AD2D-00FC6225D1FA@math.washington.edu>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
	<0B78232F-4C35-4F0C-AD2D-00FC6225D1FA@math.washington.edu>
Message-ID: <C316BB2F-AAF7-4336-AB28-70D674636915@boxdesign.co.uk>

On 22 Feb 2010, at 17:49, Robert Bradshaw wrote:

> On Feb 22, 2010, at 12:11 PM, Nick Joyce wrote:
> 
>> I have been using Cython for some time to develop the PyAMF c- 
>> extensions and have found it to be an awesome project!
> 
> Thanks!
> 
>> Recently, I discovered Python type declarations e.g.:
>> 
>> def object foo():
>>   cdef dict x = {}
>> 
>>   x['foo'] = 'bar'
>> 
>>   return x['foo']
>> 
>> The current PyAMF code is littered with PyDict_* calls so I am in  
>> the process of refactoring it to make use of this syntactic sugar.
>> 
>> When I generate the C code (using Cython 0.12.1) for the above  
>> function, the resulting C code will use PyDict_SetItem for the  
>> assignment statement but will use PyObject_GetItem on line 6. I have  
>> attached a snippet of the generated C [pyobject_getitem.c].
>> 
>> Would it make more sense to use PyDict_SetItem in this case? I have  
>> included a patch for this case [getitem.diff] if its useful. I  
>> haven't developed a test for this change, some pointers as to the  
>> correct location would be useful if that is desirable.
> 
> Yes, we would certainly like to do this. The fix isn't quite this  
> simple though, as we need to make sure x is not None before caling  
> PyDict_GetItem on it, NULL is returned if the key is not found  
> (without raising an error), and the returned reference is borrowed  
> (which changes reference counting semantics).
> 
> - Robert

I have amended the patch to include all of your comments, as well as included a simple test pyx.

Let me know if I've missed anything.

Cheers,

Nick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: getitem-2.diff
Type: application/octet-stream
Size: 1921 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100222/ae8c2dae/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dict_getitem.pyx
Type: application/octet-stream
Size: 186 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20100222/ae8c2dae/attachment-0001.obj 

From njs at vorpus.org  Tue Feb 23 04:46:49 2010
From: njs at vorpus.org (Nathaniel Smith)
Date: Mon, 22 Feb 2010 19:46:49 -0800
Subject: [Cython] Cython and numpy include files
In-Reply-To: <hlv71j$q4c$1@dough.gmane.org>
References: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>
	<hlv71j$q4c$1@dough.gmane.org>
Message-ID: <961fa2b41002221946g6ee94d6ct9d2b356232105bfc@mail.gmail.com>

On Mon, Feb 22, 2010 at 4:21 PM, Robert Kern <robert.kern at gmail.com> wrote:
> Other parts of numpy.distutils handle that. Unfortunately, those are the parts
> that conflict with Cython's distutils extensions. Just use
>
> ? include_dirs=[numpy.get_include()]

So there's both
  numpy.get_include
and
  numpy.distutils.misc_util.get_numpy_include_dirs

The former returns a string while the latter returns a list (which I
guess is somewhat more flexible), and the latter is implemented in
terms of the former, but I don't really understand the extra stuff
it's doing.

Is one preferred?

-- Nathaniel

From robertwb at math.washington.edu  Tue Feb 23 07:38:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 22 Feb 2010 22:38:57 -0800
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <C316BB2F-AAF7-4336-AB28-70D674636915@boxdesign.co.uk>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
	<0B78232F-4C35-4F0C-AD2D-00FC6225D1FA@math.washington.edu>
	<C316BB2F-AAF7-4336-AB28-70D674636915@boxdesign.co.uk>
Message-ID: <2B63AFDD-6C57-4D4E-AB76-69013ED0EE9B@math.washington.edu>

On Feb 22, 2010, at 8:04 PM, Nick Joyce wrote:

> On 22 Feb 2010, at 17:49, Robert Bradshaw wrote:
>
>> On Feb 22, 2010, at 12:11 PM, Nick Joyce wrote:
>>
>>> I have been using Cython for some time to develop the PyAMF c-
>>> extensions and have found it to be an awesome project!
>>
>> Thanks!
>>
>>> Recently, I discovered Python type declarations e.g.:
>>>
>>> def object foo():
>>>  cdef dict x = {}
>>>
>>>  x['foo'] = 'bar'
>>>
>>>  return x['foo']
>>>
>>> The current PyAMF code is littered with PyDict_* calls so I am in
>>> the process of refactoring it to make use of this syntactic sugar.
>>>
>>> When I generate the C code (using Cython 0.12.1) for the above
>>> function, the resulting C code will use PyDict_SetItem for the
>>> assignment statement but will use PyObject_GetItem on line 6. I have
>>> attached a snippet of the generated C [pyobject_getitem.c].
>>>
>>> Would it make more sense to use PyDict_SetItem in this case? I have
>>> included a patch for this case [getitem.diff] if its useful. I
>>> haven't developed a test for this change, some pointers as to the
>>> correct location would be useful if that is desirable.
>>
>> Yes, we would certainly like to do this. The fix isn't quite this
>> simple though, as we need to make sure x is not None before caling
>> PyDict_GetItem on it, NULL is returned if the key is not found
>> (without raising an error), and the returned reference is borrowed
>> (which changes reference counting semantics).
>>
>> - Robert
>
> I have amended the patch to include all of your comments, as well as  
> included a simple test pyx.

Thanks, I'll incorporate this into the main branch. Just as a style  
issue, we usually abstract away the extra logic into a inline utility  
function.

> Let me know if I've missed anything.

There is the case where hash() or cmp() raised an exception, so NULL  
is returned but an index error should not be set (rather the original  
exception should be propagated).

- Robert


From matthew.brett at gmail.com  Tue Feb 23 07:46:31 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Mon, 22 Feb 2010 22:46:31 -0800
Subject: [Cython] =?windows-1252?q?=91=5F=5Fpyx=5Fbstruct=5Farr1=92_undecl?=
	=?windows-1252?q?ared_compile_error?=
Message-ID: <1e2af89e1002222246r6385aa64gb890f718a6b93438@mail.gmail.com>

Hi,

We've run into a somewhat confusing error.  When trying to compile
this tiny thing:

import numpy as np
cimport numpy as cnp

def func():
    cdef cnp.ndarray[cnp.float64_t, ndim=2] arr1
    a = np.pi

we get this:

gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes
-I/Users/mb312/usr/local/lib/python2.6/site-packages/numpy/core/include
-I/Users/mb312/usr/local/include/python2.6 -c eg_thing.c -o
build/temp.macosx-10.6-i386-2.6/eg_thing.o
eg_thing.c: In function ?__pyx_pf_8eg_thing_func?:
eg_thing.c:749: error: ?__pyx_bstruct_arr1? undeclared (first use in
this function)
eg_thing.c:749: error: (Each undeclared identifier is reported only once
eg_thing.c:749: error: for each function it appears in.)

It's an odd error because this:

def func():
    cdef cnp.ndarray[cnp.float64_t, ndim=2] arr1
    a = 2

compiles fine, as does this:

def func():
    cdef cnp.ndarray arr1
    a = np.pi

and this:

def func():
    cdef cnp.ndarray[cnp.float64_t, ndim=2] arr1
    arr1 = np.zeros((1,1))

so I suppose it may be an interaction of the buffer types syntax with
the uninitialized array and numpy.  We using Cython 12.1...

Thanks a lot,

Matthew

From robertwb at math.washington.edu  Tue Feb 23 08:53:23 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 22 Feb 2010 23:53:23 -0800
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <2B63AFDD-6C57-4D4E-AB76-69013ED0EE9B@math.washington.edu>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
	<0B78232F-4C35-4F0C-AD2D-00FC6225D1FA@math.washington.edu>
	<C316BB2F-AAF7-4336-AB28-70D674636915@boxdesign.co.uk>
	<2B63AFDD-6C57-4D4E-AB76-69013ED0EE9B@math.washington.edu>
Message-ID: <841D5BF7-7FD7-4539-8491-AB50A2740867@math.washington.edu>

On Feb 22, 2010, at 10:38 PM, Robert Bradshaw wrote:

> On Feb 22, 2010, at 8:04 PM, Nick Joyce wrote:
>
>> On 22 Feb 2010, at 17:49, Robert Bradshaw wrote:
>>
>> I have amended the patch to include all of your comments, as well as
>> included a simple test pyx.
>
> Thanks, I'll incorporate this into the main branch. Just as a style
> issue, we usually abstract away the extra logic into a inline utility
> function.
>
>> Let me know if I've missed anything.
>
> There is the case where hash() or cmp() raised an exception, so NULL
> is returned but an index error should not be set (rather the original
> exception should be propagated).

Unfortunately, PyDict_GetItem suppresses all such exceptions for  
historic reasons, but of course the getitem slot does not (which we  
must emulate). This results in the code being a bit uglier and  
reaching into Python's internals: http://hg.cython.org/cython-devel/rev/0df578d7a490 
  . This should probably be benchmarked to see if it's worth the bother.

- Robert



From stefan_ml at behnel.de  Tue Feb 23 10:14:01 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Feb 2010 10:14:01 +0100
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
Message-ID: <4B839C59.8080305@behnel.de>

Nick Joyce, 22.02.2010 21:11:
> The current PyAMF code is littered with PyDict_* calls so I am in the
> process of refactoring it to make use of this syntactic sugar.
> 
> When I generate the C code (using Cython 0.12.1) for the above function,
> the resulting C code will use PyDict_SetItem for the assignment
> statement but will use PyObject_GetItem on line 6. I have attached a
> snippet of the generated C [pyobject_getitem.c].

To me, the reason why this wasn't done before is that the Python semantics
often are not what one really wants here. The corresponding code that uses
PyDict_GetItem() is faster when the key is found, but it is very
inefficient when the key is not found. The function would simply return
NULL in this case, which is easy and fast to test for in C code. Usually,
when I needed pure speed here, I needed it for the failure case, so I used
a direct C-API call anyway.

Robert is right in suggesting that it might not even be worth it. Even a
virtual call to the tp_getitem slot (which is a pretty direct result of the
PyObject_GetItem() call) is likely not that much slower than a call to
PyDict_GetItem(), given the overhead of emulating the Python semantics
around the call. Only benchmarking can tell if this brings more than a
negligible 4% speedup 'on average'.

It would certainly bring much more if Cython could deduce what the result
is used for afterwards, and then reduce the true Python semantics to what
is really needed. E.g. this code:

    cdef dict d = ...
    try:
        item = d['test']
    except KeyError:
        item = d['test'] = some_value

could be reduced to this pseudo code:

    try:
        if d is None:
             raise TypeError(...)
        item = PyDict_GetItem(d, 'test')
        if item is NULL:
            goto except_handler
        Py_INCREF(item)
    except KeyError:
        except_handler:
        PyDict_SetItem(d, some_value)
        Py_INCREF(some_value)
        item = some_value

But that's certainly not trivial with the current compiler infrastructure.
It would be possible to see that a KeyError is caught around a dict access,
so that it would no longer be necessary to actually raise the KeyError in
the first place.

Then again, this could happen:

    try: ...
    except KeyError:
         print sys.exc_info()

Well, well, Python ...

Stefan

From dagss at student.matnat.uio.no  Tue Feb 23 12:27:09 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 23 Feb 2010 12:27:09 +0100
Subject: [Cython]
 =?windows-1252?q?=91=5F=5Fpyx=5Fbstruct=5Farr1=92_undecl?=
 =?windows-1252?q?ared_compile_error?=
In-Reply-To: <1e2af89e1002222246r6385aa64gb890f718a6b93438@mail.gmail.com>
References: <1e2af89e1002222246r6385aa64gb890f718a6b93438@mail.gmail.com>
Message-ID: <4B83BB8D.9010700@student.matnat.uio.no>

Matthew Brett wrote:
> Hi,
>
> We've run into a somewhat confusing error.  When trying to compile
> this tiny thing:
>
> import numpy as np
> cimport numpy as cnp
>
> def func():
>     cdef cnp.ndarray[cnp.float64_t, ndim=2] arr1
>     a = np.pi
>
> we get this:
>
> gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
> -Wstrict-prototypes
> -I/Users/mb312/usr/local/lib/python2.6/site-packages/numpy/core/include
> -I/Users/mb312/usr/local/include/python2.6 -c eg_thing.c -o
> build/temp.macosx-10.6-i386-2.6/eg_thing.o
> eg_thing.c: In function ?__pyx_pf_8eg_thing_func?:
> eg_thing.c:749: error: ?__pyx_bstruct_arr1? undeclared (first use in
> this function)
> eg_thing.c:749: error: (Each undeclared identifier is reported only once
> eg_thing.c:749: error: for each function it appears in.)
>
> It's an odd error because this:
>
> def func():
>     cdef cnp.ndarray[cnp.float64_t, ndim=2] arr1
>     a = 2
>
> compiles fine, as does this:
>
> def func():
>     cdef cnp.ndarray arr1
>     a = np.pi
>
> and this:
>
> def func():
>     cdef cnp.ndarray[cnp.float64_t, ndim=2] arr1
>     arr1 = np.zeros((1,1))
>
> so I suppose it may be an interaction of the buffer types syntax with
> the uninitialized array and numpy.  We using Cython 12.1...
>   
Yep, that's exactly what it is. It's a bug, feel free to file it if it 
isn't already.

I don't consider it very critical, as having unused variables around is 
hardly a requirement, but I agree it should be fixed.

Dag Sverre

From dagss at student.matnat.uio.no  Tue Feb 23 12:30:45 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 23 Feb 2010 12:30:45 +0100
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <4B839C59.8080305@behnel.de>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
	<4B839C59.8080305@behnel.de>
Message-ID: <4B83BC65.2030104@student.matnat.uio.no>

Stefan Behnel wrote:
> Nick Joyce, 22.02.2010 21:11:
>   
>> The current PyAMF code is littered with PyDict_* calls so I am in the
>> process of refactoring it to make use of this syntactic sugar.
>>
>> When I generate the C code (using Cython 0.12.1) for the above function,
>> the resulting C code will use PyDict_SetItem for the assignment
>> statement but will use PyObject_GetItem on line 6. I have attached a
>> snippet of the generated C [pyobject_getitem.c].
>>     
>
> To me, the reason why this wasn't done before is that the Python semantics
> often are not what one really wants here. The corresponding code that uses
> PyDict_GetItem() is faster when the key is found, but it is very
> inefficient when the key is not found. The function would simply return
> NULL in this case, which is easy and fast to test for in C code. Usually,
> when I needed pure speed here, I needed it for the failure case, so I used
> a direct C-API call anyway.
>
> Robert is right in suggesting that it might not even be worth it. Even a
> virtual call to the tp_getitem slot (which is a pretty direct result of the
> PyObject_GetItem() call) is likely not that much slower than a call to
> PyDict_GetItem(), given the overhead of emulating the Python semantics
> around the call. Only benchmarking can tell if this brings more than a
> negligible 4% speedup 'on average'.
>
> It would certainly bring much more if Cython could deduce what the result
> is used for afterwards, and then reduce the true Python semantics to what
> is really needed. E.g. this code:
>
>     cdef dict d = ...
>     try:
>         item = d['test']
>     except KeyError:
>         item = d['test'] = some_value
>
> could be reduced to this pseudo code:
>
>     try:
>         if d is None:
>              raise TypeError(...)
>         item = PyDict_GetItem(d, 'test')
>         if item is NULL:
>             goto except_handler
>         Py_INCREF(item)
>     except KeyError:
>         except_handler:
>         PyDict_SetItem(d, some_value)
>         Py_INCREF(some_value)
>         item = some_value
>
> But that's certainly not trivial with the current compiler infrastructure.
> It would be possible to see that a KeyError is caught around a dict access,
> so that it would no longer be necessary to actually raise the KeyError in
> the first place.
>
> Then again, this could happen:
>
>     try: ...
>     except KeyError:
>          print sys.exc_info()
>
> Well, well, Python ...
>
>   
How about just optimizing "d.get(key[, default])" if it isn't already 
for this case, and document that [] is slower?

Dag Sverre

From stefan_ml at behnel.de  Tue Feb 23 13:29:43 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Feb 2010 13:29:43 +0100
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <4B83BC65.2030104@student.matnat.uio.no>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>	<4B839C59.8080305@behnel.de>
	<4B83BC65.2030104@student.matnat.uio.no>
Message-ID: <4B83CA37.4080604@behnel.de>

Dag Sverre Seljebotn, 23.02.2010 12:30:
> Stefan Behnel wrote:
>>     cdef dict d = ...
>>     try:
>>         item = d['test']
>>     except KeyError:
>>         item = d['test'] = some_value
>
> How about just optimizing "d.get(key[, default])" if it isn't already 
> for this case, and document that [] is slower?

Very good idea. I'll do that.

Stefan


From stefan_ml at behnel.de  Tue Feb 23 14:29:18 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Feb 2010 14:29:18 +0100
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <4B83CA37.4080604@behnel.de>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>	<4B839C59.8080305@behnel.de>	<4B83BC65.2030104@student.matnat.uio.no>
	<4B83CA37.4080604@behnel.de>
Message-ID: <4B83D82E.7070505@behnel.de>

Stefan Behnel, 23.02.2010 13:29:
> Dag Sverre Seljebotn, 23.02.2010 12:30:
>> Stefan Behnel wrote:
>>>     cdef dict d = ...
>>>     try:
>>>         item = d['test']
>>>     except KeyError:
>>>         item = d['test'] = some_value
>> How about just optimizing "d.get(key[, default])" if it isn't already 
>> for this case, and document that [] is slower?
> 
> Very good idea. I'll do that.

... and it obviously suffers from the same problem that Robert described:
if the object is not hashable, it silently swallows the exception.

Now, Python 3.x has a new PyDict_GetItemWithError() function that gives us
exactly what we want. So my proposal is to not duplicate any implementation
details from CPython (again!), but instead provide the speedup only for
CPython 3.x.

Stefan

From dalcinl at gmail.com  Tue Feb 23 15:04:25 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 23 Feb 2010 11:04:25 -0300
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <4B83D82E.7070505@behnel.de>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
	<4B839C59.8080305@behnel.de> <4B83BC65.2030104@student.matnat.uio.no>
	<4B83CA37.4080604@behnel.de> <4B83D82E.7070505@behnel.de>
Message-ID: <e7ba66e41002230604k609d4503pd3a710830b0b6cb4@mail.gmail.com>

On 23 February 2010 10:29, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> Now, Python 3.x has a new PyDict_GetItemWithError() function that gives us
> exactly what we want. So my proposal is to not duplicate any implementation
> details from CPython (again!), but instead provide the speedup only for
> CPython 3.x.
>

I agree. Cython already has many unbenchmarked (is this a word?)
optimization that no one knows it they are worth the complexity. After
all, if you still need a 4% speedup when looking up in a dict, just
use Python C-API, or even implement yourself in C the
functionality/semantics you need.


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Feb 23 15:38:32 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Feb 2010 15:38:32 +0100
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <e7ba66e41002230604k609d4503pd3a710830b0b6cb4@mail.gmail.com>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>	<4B839C59.8080305@behnel.de>
	<4B83BC65.2030104@student.matnat.uio.no>	<4B83CA37.4080604@behnel.de>
	<4B83D82E.7070505@behnel.de>
	<e7ba66e41002230604k609d4503pd3a710830b0b6cb4@mail.gmail.com>
Message-ID: <4B83E868.3040502@behnel.de>

Lisandro Dalcin, 23.02.2010 15:04:
> On 23 February 2010 10:29, Stefan Behnel wrote:
>> Now, Python 3.x has a new PyDict_GetItemWithError() function that gives us
>> exactly what we want. So my proposal is to not duplicate any implementation
>> details from CPython (again!), but instead provide the speedup only for
>> CPython 3.x.
> 
> I agree. Cython already has many unbenchmarked (is this a word?)
> optimization that no one knows it they are worth the complexity. After
> all, if you still need a 4% speedup when looking up in a dict, just
> use Python C-API, or even implement yourself in C the
> functionality/semantics you need.

Ok, here is an implementation:

http://hg.cython.org/cython-devel/rev/8c8994167e1a

I kept the optimisation in Py2 for str, unicode and int keys, as I expect
their hash and comparison functions to be rather safe, and their use in
dicts to be extremely frequent.

Stefan

From stefan_ml at behnel.de  Tue Feb 23 16:08:03 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Feb 2010 16:08:03 +0100
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <4B83E868.3040502@behnel.de>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>	<4B839C59.8080305@behnel.de>	<4B83BC65.2030104@student.matnat.uio.no>	<4B83CA37.4080604@behnel.de>	<4B83D82E.7070505@behnel.de>	<e7ba66e41002230604k609d4503pd3a710830b0b6cb4@mail.gmail.com>
	<4B83E868.3040502@behnel.de>
Message-ID: <4B83EF53.4040209@behnel.de>

Stefan Behnel, 23.02.2010 15:38:
> Lisandro Dalcin, 23.02.2010 15:04:
>> On 23 February 2010 10:29, Stefan Behnel wrote:
>>> Now, Python 3.x has a new PyDict_GetItemWithError() function that gives us
>>> exactly what we want. So my proposal is to not duplicate any implementation
>>> details from CPython (again!), but instead provide the speedup only for
>>> CPython 3.x.
>> I agree. Cython already has many unbenchmarked (is this a word?)
>> optimization that no one knows it they are worth the complexity. After
>> all, if you still need a 4% speedup when looking up in a dict, just
>> use Python C-API, or even implement yourself in C the
>> functionality/semantics you need.
> 
> Ok, here is an implementation:
> 
> http://hg.cython.org/cython-devel/rev/8c8994167e1a
> 
> I kept the optimisation in Py2 for str, unicode and int keys, as I expect
> their hash and comparison functions to be rather safe, and their use in
> dicts to be extremely frequent.

... and I used the micro benchmark that Robert committed to check if there
is any difference for the d[key] implementation. I couldn't see any, so I
simplified the implementation to use PyObject_GetItem() in Py2. I would
even encourage to revert to PyObject_GetItem() completely, also in Py3,
given that there seems to be no speedup at all.

Stefan

From dalcinl at gmail.com  Tue Feb 23 16:13:43 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 23 Feb 2010 12:13:43 -0300
Subject: [Cython] public versus api,
	dll import/export and api 	import/export
In-Reply-To: <4B7CEBEA.7060107@behnel.de>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
Message-ID: <e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>

Bump!

On 18 February 2010 04:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 17.02.2010 23:38:
>> What is not clear to me is the following: Should "public", "api", or
>> "public api" have any semantic diference?
>> [...]
>> Greg said that he never intended to provide DLL export
>> mechanisms, and in fact he just wanted to make stuff available across
>> compilation units (i.e, make stuff available to other C source files
>> being compiled ALONGSIDE the generated C sources to build the final
>> ext module). If we enforce this, I mean we are not going to support
>> ext module interlinking, then almost the usages of DL_IMPORT/DL_EXPORT
>> macros are a nonsense and should be removed, alongside almost all
>> these dll_linkage arguments in many methods.
>> [...]
>> So... What do you think?
>
> My take is that public symbols are only portably usable at static link
> time, so supporting more is simply not worth it. Even static linking
> against symbols exported by a Cython module should be a very rare use case.
> It's not used when calling external C code nor is it worth anything when
> providing callbacks into the Cython code. Most of the time, the external
> code will be there first and will be used by the Cython code, not the other
> way round. This mechanism shouldn't be seen as something that's usable at
> dynamic linking time. There's public C-API support for that. I second your
> intuition that the DL_EXPORT stuff can be dropped completely.
>

OK.

> There's also the use case of statically linking multiple Cython modules
> into one extension module. This isn't currently supported (really), and
> there's more to do to make it run smooth.

Are you talking about the limitations/gotchas in Python's import.c
implementation ?

> But I don't think it would
> interfere with the above in any way.
>

Of course.

Anyone has something to add? Greg?

Still... What should we do with 'public' and 'api' keywords?

-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue Feb 23 16:33:08 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 23 Feb 2010 12:33:08 -0300
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <4B83EF53.4040209@behnel.de>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
	<4B839C59.8080305@behnel.de> <4B83BC65.2030104@student.matnat.uio.no>
	<4B83CA37.4080604@behnel.de> <4B83D82E.7070505@behnel.de>
	<e7ba66e41002230604k609d4503pd3a710830b0b6cb4@mail.gmail.com>
	<4B83E868.3040502@behnel.de> <4B83EF53.4040209@behnel.de>
Message-ID: <e7ba66e41002230733g23f72986pd5b6343b9827f65@mail.gmail.com>

On 23 February 2010 12:08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> ... and I used the micro benchmark that Robert committed to check if there
> is any difference for the d[key] implementation. I couldn't see any, so I
> simplified the implementation to use PyObject_GetItem() in Py2.

+1

> I would
> even encourage to revert to PyObject_GetItem() completely, also in Py3,
> given that there seems to be no speedup at all.
>

-0 .

As long as Python C-API provides a public call, I would say it is safe
to use it.



-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Tue Feb 23 16:41:49 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Feb 2010 16:41:49 +0100
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <4B83EF53.4040209@behnel.de>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>	<4B839C59.8080305@behnel.de>	<4B83BC65.2030104@student.matnat.uio.no>	<4B83CA37.4080604@behnel.de>	<4B83D82E.7070505@behnel.de>	<e7ba66e41002230604k609d4503pd3a710830b0b6cb4@mail.gmail.com>	<4B83E868.3040502@behnel.de>
	<4B83EF53.4040209@behnel.de>
Message-ID: <4B83F73D.7080805@behnel.de>

Stefan Behnel, 23.02.2010 16:08:
> ... and I used the micro benchmark that Robert committed to check if there
> is any difference for the d[key] implementation. I couldn't see any, so I
> simplified the implementation to use PyObject_GetItem() in Py2. I would
> even encourage to revert to PyObject_GetItem() completely, also in Py3,
> given that there seems to be no speedup at all.

I'm partly taking this back - was testing with a debug build of Py3.2. With
a normal 3.1 build, I see a ~5% performance improvement compared to Py2.6.

So I'm for keeping the (rather simple) special case for Py3 and use
PyObject_GetItem() in Py2.

Stefan

From robert.kern at gmail.com  Tue Feb 23 18:13:06 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 23 Feb 2010 11:13:06 -0600
Subject: [Cython] Cython and numpy include files
In-Reply-To: <961fa2b41002221946g6ee94d6ct9d2b356232105bfc@mail.gmail.com>
References: <b02c60871002221542i1bbb378ck17453ca573f79ffe@mail.gmail.com>	<hlv71j$q4c$1@dough.gmane.org>
	<961fa2b41002221946g6ee94d6ct9d2b356232105bfc@mail.gmail.com>
Message-ID: <hm12b2$bkf$1@dough.gmane.org>

On 2010-02-22 21:46 PM, Nathaniel Smith wrote:
> On Mon, Feb 22, 2010 at 4:21 PM, Robert Kern<robert.kern at gmail.com>  wrote:
>> Other parts of numpy.distutils handle that. Unfortunately, those are the parts
>> that conflict with Cython's distutils extensions. Just use
>>
>>    include_dirs=[numpy.get_include()]
>
> So there's both
>    numpy.get_include
> and
>    numpy.distutils.misc_util.get_numpy_include_dirs
>
> The former returns a string while the latter returns a list (which I
> guess is somewhat more flexible), and the latter is implemented in
> terms of the former, but I don't really understand the extra stuff
> it's doing.
>
> Is one preferred?

The former. The latter is an internal convenience function for use inside of 
numpy.distutils.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From stefan at sun.ac.za  Tue Feb 23 18:24:39 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 23 Feb 2010 19:24:39 +0200
Subject: [Cython] SciPy2010 Call for Papers
Message-ID: <9457e7c81002230924s29b37c2ay24fc315d3802ebb2@mail.gmail.com>

==========================
SciPy 2010 Call for Papers
==========================

SciPy 2010, the 9th Python in Science Conference, will be held from
June 28th - July 3rd, 2010 in Austin, Texas.

At this conference, novel applications and breakthroughs made in the
pursuit of science using Python are presented.  Attended by leading
figures from both academia and industry, it is an excellent
opportunity to experience the cutting edge of scientific software
development.

The conference is preceded by two days of paid tutorials, during which
community experts provide training on several scientific Python
packages.

We invite you to take part by submitting a talk abstract on the
conference website at:

http://conference.scipy.org

Talk/Paper Submission
=====================

We solicit talks and accompanying papers (either formal academic or
magazine-style articles) that discuss topics regarding scientific
computing using Python, including applications, teaching, development
and research.  Papers are included in the peer-reviewed conference
proceedings, published online.

Please note that submissions primarily aimed at the promotion of a
commercial product or service will not be considered.

Important dates for authors include:

 * 11 April: Talk abstracts due
 * 20 April: Notification of acceptance
 * 13 June: Papers due
 * 15 August: Publication of proceedings

Further detail will be made available on http://conference.scipy.org

Conference Dates
================

 * Friday, 10 May: Early registration ends
 * Monday-Tuesday, 28-29 June: Tutorials
 * Wednesday-Thursday, June 30-July 1: Conference
 * Friday-Saturday, July 2-3: Coding Sprints

Executive Committee
===================

 * Conference: Jarrod Millman & Eric Jones
 * Program: Stefan van der Walt & Ondrej Certik
 * Student Sponsorship: Travis Oliphant

For more information on Python, visit http://www.python.org.

From matthew.brett at gmail.com  Tue Feb 23 18:43:37 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Tue, 23 Feb 2010 09:43:37 -0800
Subject: [Cython]
	=?windows-1252?q?=91=5F=5Fpyx=5Fbstruct=5Farr1=92_undecl?=
	=?windows-1252?q?ared_compile_error?=
In-Reply-To: <4B83BB8D.9010700@student.matnat.uio.no>
References: <1e2af89e1002222246r6385aa64gb890f718a6b93438@mail.gmail.com>
	<4B83BB8D.9010700@student.matnat.uio.no>
Message-ID: <1e2af89e1002230943y5214d7a9ma89a63ec9ad04372@mail.gmail.com>

Hi,

>> so I suppose it may be an interaction of the buffer types syntax with
>> the uninitialized array and numpy. ?We using Cython 12.1...
>>
> Yep, that's exactly what it is. It's a bug, feel free to file it if it
> isn't already.

Thanks a lot for the quick reply.  It looks like this is the same issue as:

http://trac.cython.org/cython_trac/ticket/154

Anything I should do to add to that ticket?

> I don't consider it very critical, as having unused variables around is
> hardly a requirement, but I agree it should be fixed.

Yes, true, but here is just a little plea for a fix, because it took
us a long time to work out what the problem was.

See you,

Matthew

From robertwb at math.washington.edu  Tue Feb 23 20:49:49 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 23 Feb 2010 11:49:49 -0800
Subject: [Cython] Use PyDict_SetItem instead of PyObject_SetItem
In-Reply-To: <e7ba66e41002230733g23f72986pd5b6343b9827f65@mail.gmail.com>
References: <B1D45EF5-5FCC-496A-9D09-C21702352194@boxdesign.co.uk>
	<4B839C59.8080305@behnel.de>
	<4B83BC65.2030104@student.matnat.uio.no>
	<4B83CA37.4080604@behnel.de> <4B83D82E.7070505@behnel.de>
	<e7ba66e41002230604k609d4503pd3a710830b0b6cb4@mail.gmail.com>
	<4B83E868.3040502@behnel.de> <4B83EF53.4040209@behnel.de>
	<e7ba66e41002230733g23f72986pd5b6343b9827f65@mail.gmail.com>
Message-ID: <52D9F22B-D015-424D-AAD4-5FBCC7197A52@math.washington.edu>

On Feb 23, 2010, at 7:33 AM, Lisandro Dalcin wrote:

> On 23 February 2010 12:08, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>
>> ... and I used the micro benchmark that Robert committed to check  
>> if there
>> is any difference for the d[key] implementation. I couldn't see  
>> any, so I
>> simplified the implementation to use PyObject_GetItem() in Py2.
>
> +1

+1, that code's not worth it (but at least now we know). I realized  
that I accidently checked the microbenchmark in, but left it in there  
to see if anyone else got anything different.

It would be nice to avoid the cost of the method lookup for d.get(...)  
for Py2, but the apis aren't very conducive to that...

- Robert


From robertwb at math.washington.edu  Wed Feb 24 07:24:37 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 23 Feb 2010 22:24:37 -0800
Subject: [Cython] public versus api,
	dll import/export and api 	import/export
In-Reply-To: <e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
Message-ID: <E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>

On Feb 23, 2010, at 7:13 AM, Lisandro Dalcin wrote:

> Bump!
>
> On 18 February 2010 04:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Lisandro Dalcin, 17.02.2010 23:38:
>>> What is not clear to me is the following: Should "public", "api", or
>>> "public api" have any semantic diference?
>>> [...]
>>> Greg said that he never intended to provide DLL export
>>> mechanisms, and in fact he just wanted to make stuff available  
>>> across
>>> compilation units (i.e, make stuff available to other C source files
>>> being compiled ALONGSIDE the generated C sources to build the final
>>> ext module). If we enforce this, I mean we are not going to support
>>> ext module interlinking, then almost the usages of DL_IMPORT/ 
>>> DL_EXPORT
>>> macros are a nonsense and should be removed, alongside almost all
>>> these dll_linkage arguments in many methods.
>>> [...]
>>> So... What do you think?
>>
>> My take is that public symbols are only portably usable at static  
>> link
>> time, so supporting more is simply not worth it. Even static linking
>> against symbols exported by a Cython module should be a very rare  
>> use case.
>> It's not used when calling external C code nor is it worth anything  
>> when
>> providing callbacks into the Cython code. Most of the time, the  
>> external
>> code will be there first and will be used by the Cython code, not  
>> the other
>> way round. This mechanism shouldn't be seen as something that's  
>> usable at
>> dynamic linking time. There's public C-API support for that. I  
>> second your
>> intuition that the DL_EXPORT stuff can be dropped completely.
>>
>
> OK.
>
>> There's also the use case of statically linking multiple Cython  
>> modules
>> into one extension module. This isn't currently supported (really),  
>> and
>> there's more to do to make it run smooth.
>
> Are you talking about the limitations/gotchas in Python's import.c
> implementation ?
>
>> But I don't think it would
>> interfere with the above in any way.
>>
>
> Of course.
>
> Anyone has something to add? Greg?
>
> Still... What should we do with 'public' and 'api' keywords?

I don't really use either, so while I agree that it would be nice to  
clean things up, I don't want to break backwards compatibility.  
(Judging from the lack of activity on this thread though, it seems  
they're not in high demand, though sage-support should be pinged if we  
plan on getting rid of anything.) 'public' should probably still mean  
what it always has.

- Robert



From stefan_ml at behnel.de  Wed Feb 24 09:39:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Feb 2010 09:39:06 +0100
Subject: [Cython] Pyrex 0.9.8.6 is out
Message-ID: <4B84E5AA.8090102@behnel.de>

Hi,

a new Pyrex release is out.

At first glance, not much to do, but I didn't take a close look yet.

The relevant bundle is here:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/hg/Bundle-5ad6adfbba2e

Stefan



0.9.8.6
-------

Enhancements:

  - Check for gil when calling a function declared 'execpt *' or 'except ?'.
    Acquire gil when reporting unraisable exception.

  - Added iter2() function for 2-argument form of iter().


Bug fixes:

  - Compiler crashed if base class of extension type was incompletely
    defined.

  - Compiler crash on misspelled method name in property declaration.
    [Stefan Behnel]

  - Fixed deprecation warnings in 2.6

  - Always preserve explicit type casts for non-Python types.
    [Alexander Belchenko]

  - Added workaround for threading initialisation bug in Python 2.3.
    [Lisandro Dalcin]

  - Deleting dict item with integer key did not work. [Titus Brown]

  - Header files for cimported modules included in wrong order.
    [Stephane Drouard]

  - Don't allow a member of a ctypedef struct to reference itself.
    [Tim Wakeham]

  - Compiler crash due to attribute reference in compile-time expression.
    [Hoyt Koepke]

  - Public extension type attribute with cname didn't work.
    [Mark Ellis]

  - Memory leak related to exporting C functions. [Lisandro Dalcin]

  - Compiler crash on return outside function. [Kurt Smith]

  - Scope problem with extension types declared in pxd. [KS Sreeram]

  - Calling a builtin method of a subclass of a builtin class did not work.

  - Builtin hash() function had wrong return type. [John Arbash Meinel]


Modifications:

  - Added 'tags' to .hgignore file. [Kirill Smelkov]

  - Disallow overriding a builtin method in a subclass of a builtin class.

From dalcinl at gmail.com  Wed Feb 24 15:41:04 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 24 Feb 2010 11:41:04 -0300
Subject: [Cython] public versus api,
	dll import/export and api 	import/export
In-Reply-To: <E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
	<E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>
Message-ID: <e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>

On 24 February 2010 03:24, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Feb 23, 2010, at 7:13 AM, Lisandro Dalcin wrote:
>
>> Bump!
>>
>> On 18 February 2010 04:27, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Lisandro Dalcin, 17.02.2010 23:38:
>>>> What is not clear to me is the following: Should "public", "api", or
>>>> "public api" have any semantic diference?
>>>> [...]
>>>> Greg said that he never intended to provide DLL export
>>>> mechanisms, and in fact he just wanted to make stuff available
>>>> across
>>>> compilation units (i.e, make stuff available to other C source files
>>>> being compiled ALONGSIDE the generated C sources to build the final
>>>> ext module). If we enforce this, I mean we are not going to support
>>>> ext module interlinking, then almost the usages of DL_IMPORT/
>>>> DL_EXPORT
>>>> macros are a nonsense and should be removed, alongside almost all
>>>> these dll_linkage arguments in many methods.
>>>> [...]
>>>> So... What do you think?
>>>
>>> My take is that public symbols are only portably usable at static
>>> link
>>> time, so supporting more is simply not worth it. Even static linking
>>> against symbols exported by a Cython module should be a very rare
>>> use case.
>>> It's not used when calling external C code nor is it worth anything
>>> when
>>> providing callbacks into the Cython code. Most of the time, the
>>> external
>>> code will be there first and will be used by the Cython code, not
>>> the other
>>> way round. This mechanism shouldn't be seen as something that's
>>> usable at
>>> dynamic linking time. There's public C-API support for that. I
>>> second your
>>> intuition that the DL_EXPORT stuff can be dropped completely.
>>>
>>
>> OK.
>>
>>> There's also the use case of statically linking multiple Cython
>>> modules
>>> into one extension module. This isn't currently supported (really),
>>> and
>>> there's more to do to make it run smooth.
>>
>> Are you talking about the limitations/gotchas in Python's import.c
>> implementation ?
>>
>>> But I don't think it would
>>> interfere with the above in any way.
>>>
>>
>> Of course.
>>
>> Anyone has something to add? Greg?
>>
>> Still... What should we do with 'public' and 'api' keywords?
>
> I don't really use either, so while I agree that it would be nice to
> clean things up, I don't want to break backwards compatibility.
> (Judging from the lack of activity on this thread though, it seems
> they're not in high demand, though sage-support should be pinged if we
> plan on getting rid of anything.) 'public' should probably still mean
> what it always has.
>

Could you explain me the exact meaning of 'public' (alone, without 'api')?


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed Feb 24 17:08:51 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 24 Feb 2010 13:08:51 -0300
Subject: [Cython] HELP: setting slices with typed values
Message-ID: <e7ba66e41002240808j47c08938qbec17947b8fba7ba@mail.gmail.com>

Suppose this code:

cdef extern from *:
    ctypedef class __builtin__.list [ object PyListObject ]:
        pass

def slice_of_typed_value():

    """
    >>> slice_of_typed_value()
    [1, 2, 3]
    """
    cdef object a = []
    cdef list L = [1, 2, 3]
    a[:] = L
    return a


GCC complains (complains in C, but of course an error in C++) like this:

slice2.c: In function ?__pyx_pf_6slice2_slice_of_typed_value?:
slice2.c:1107: warning: passing argument 4 of ?PySequence_SetSlice?
from incompatible pointer type
/usr/include/python2.6/abstract.h:1118: note: expected ?struct
PyObject *? but argument is of type ?struct PyListObject *?


Is the patch below the correct way to fix it?:

diff -r e16a6487e8c2 Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py	Wed Feb 24 14:23:58 2010 +0100
+++ b/Cython/Compiler/ExprNodes.py	Wed Feb 24 13:06:31 2010 -0300
@@ -2279,7 +2279,7 @@
                     self.base.py_result(),
                     self.start_code(),
                     self.stop_code(),
-                    rhs.result()))
+                    rhs.py_result()))
         else:
             start_offset = ''
             if self.start:



-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Wed Feb 24 17:23:47 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Feb 2010 17:23:47 +0100
Subject: [Cython] HELP: setting slices with typed values
In-Reply-To: <e7ba66e41002240808j47c08938qbec17947b8fba7ba@mail.gmail.com>
References: <e7ba66e41002240808j47c08938qbec17947b8fba7ba@mail.gmail.com>
Message-ID: <4B855293.9020408@behnel.de>

Lisandro Dalcin, 24.02.2010 17:08:
> Suppose this code:
> 
> cdef extern from *:
>     ctypedef class __builtin__.list [ object PyListObject ]:
>         pass
> 
> def slice_of_typed_value():
> 
>     """
>     >>> slice_of_typed_value()
>     [1, 2, 3]
>     """
>     cdef object a = []
>     cdef list L = [1, 2, 3]
>     a[:] = L
>     return a
> 
> 
> GCC complains (complains in C, but of course an error in C++) like this:
> 
> slice2.c: In function ?__pyx_pf_6slice2_slice_of_typed_value?:
> slice2.c:1107: warning: passing argument 4 of ?PySequence_SetSlice?
> from incompatible pointer type
> /usr/include/python2.6/abstract.h:1118: note: expected ?struct
> PyObject *? but argument is of type ?struct PyListObject *?
> 
> 
> Is the patch below the correct way to fix it?:
> 
> diff -r e16a6487e8c2 Cython/Compiler/ExprNodes.py
> --- a/Cython/Compiler/ExprNodes.py	Wed Feb 24 14:23:58 2010 +0100
> +++ b/Cython/Compiler/ExprNodes.py	Wed Feb 24 13:06:31 2010 -0300
> @@ -2279,7 +2279,7 @@
>                      self.base.py_result(),
>                      self.start_code(),
>                      self.stop_code(),
> -                    rhs.result()))
> +                    rhs.py_result()))
>          else:
>              start_offset = ''
>              if self.start:

Looks reasonable at first glance. Could you add the above code as a test
case? I'll take a closer look then.

Stefan

From robertwb at math.washington.edu  Wed Feb 24 18:35:11 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 24 Feb 2010 09:35:11 -0800
Subject: [Cython] public versus api,
	dll import/export and api 	import/export
In-Reply-To: <e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
	<E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>
	<e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>
Message-ID: <22DB1A4C-8737-42B6-8CE5-B32C7C5035C1@math.washington.edu>

On Feb 24, 2010, at 6:41 AM, Lisandro Dalcin wrote:

> On 24 February 2010 03:24, Robert Bradshaw <robertwb at math.washington.edu 
> > wrote:
>> On Feb 23, 2010, at 7:13 AM, Lisandro Dalcin wrote:
>>
>>> Bump!
>>>
>>> On 18 February 2010 04:27, Stefan Behnel <stefan_ml at behnel.de>  
>>> wrote:
>>>> Lisandro Dalcin, 17.02.2010 23:38:
>>>>> What is not clear to me is the following: Should "public",  
>>>>> "api", or
>>>>> "public api" have any semantic diference?
>>>>> [...]
>>>>> Greg said that he never intended to provide DLL export
>>>>> mechanisms, and in fact he just wanted to make stuff available
>>>>> across
>>>>> compilation units (i.e, make stuff available to other C source  
>>>>> files
>>>>> being compiled ALONGSIDE the generated C sources to build the  
>>>>> final
>>>>> ext module). If we enforce this, I mean we are not going to  
>>>>> support
>>>>> ext module interlinking, then almost the usages of DL_IMPORT/
>>>>> DL_EXPORT
>>>>> macros are a nonsense and should be removed, alongside almost all
>>>>> these dll_linkage arguments in many methods.
>>>>> [...]
>>>>> So... What do you think?
>>>>
>>>> My take is that public symbols are only portably usable at static
>>>> link
>>>> time, so supporting more is simply not worth it. Even static  
>>>> linking
>>>> against symbols exported by a Cython module should be a very rare
>>>> use case.
>>>> It's not used when calling external C code nor is it worth anything
>>>> when
>>>> providing callbacks into the Cython code. Most of the time, the
>>>> external
>>>> code will be there first and will be used by the Cython code, not
>>>> the other
>>>> way round. This mechanism shouldn't be seen as something that's
>>>> usable at
>>>> dynamic linking time. There's public C-API support for that. I
>>>> second your
>>>> intuition that the DL_EXPORT stuff can be dropped completely.
>>>>
>>>
>>> OK.
>>>
>>>> There's also the use case of statically linking multiple Cython
>>>> modules
>>>> into one extension module. This isn't currently supported (really),
>>>> and
>>>> there's more to do to make it run smooth.
>>>
>>> Are you talking about the limitations/gotchas in Python's import.c
>>> implementation ?
>>>
>>>> But I don't think it would
>>>> interfere with the above in any way.
>>>>
>>>
>>> Of course.
>>>
>>> Anyone has something to add? Greg?
>>>
>>> Still... What should we do with 'public' and 'api' keywords?
>>
>> I don't really use either, so while I agree that it would be nice to
>> clean things up, I don't want to break backwards compatibility.
>> (Judging from the lack of activity on this thread though, it seems
>> they're not in high demand, though sage-support should be pinged if  
>> we
>> plan on getting rid of anything.) 'public' should probably still mean
>> what it always has.
>>
>
> Could you explain me the exact meaning of 'public' (alone, without  
> 'api')?

Doesn't it simply avoid mangling names? (It may have other side  
effects as well.)

- Robert



From dagss at student.matnat.uio.no  Wed Feb 24 19:45:55 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 24 Feb 2010 19:45:55 +0100
Subject: [Cython] public versus api,
 dll import/export and api 	import/export
In-Reply-To: <22DB1A4C-8737-42B6-8CE5-B32C7C5035C1@math.washington.edu>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
	<E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>
	<e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>
	<22DB1A4C-8737-42B6-8CE5-B32C7C5035C1@math.washington.edu>
Message-ID: <f8f6ec2ded57fcf825c3c9248f6b4d95.squirrel@webmail.uio.no>

> On Feb 24, 2010, at 6:41 AM, Lisandro Dalcin wrote:
>
>> On 24 February 2010 03:24, Robert Bradshaw <robertwb at math.washington.edu
>> > wrote:
>>> On Feb 23, 2010, at 7:13 AM, Lisandro Dalcin wrote:
>>>
>>>> Bump!
>>>>
>>>> On 18 February 2010 04:27, Stefan Behnel <stefan_ml at behnel.de>
>>>> wrote:
>>>>> Lisandro Dalcin, 17.02.2010 23:38:
>>>>>> What is not clear to me is the following: Should "public",
>>>>>> "api", or
>>>>>> "public api" have any semantic diference?
>>>>>> [...]
>>>>>> Greg said that he never intended to provide DLL export
>>>>>> mechanisms, and in fact he just wanted to make stuff available
>>>>>> across
>>>>>> compilation units (i.e, make stuff available to other C source
>>>>>> files
>>>>>> being compiled ALONGSIDE the generated C sources to build the
>>>>>> final
>>>>>> ext module). If we enforce this, I mean we are not going to
>>>>>> support
>>>>>> ext module interlinking, then almost the usages of DL_IMPORT/
>>>>>> DL_EXPORT
>>>>>> macros are a nonsense and should be removed, alongside almost all
>>>>>> these dll_linkage arguments in many methods.
>>>>>> [...]
>>>>>> So... What do you think?
>>>>>
>>>>> My take is that public symbols are only portably usable at static
>>>>> link
>>>>> time, so supporting more is simply not worth it. Even static
>>>>> linking
>>>>> against symbols exported by a Cython module should be a very rare
>>>>> use case.
>>>>> It's not used when calling external C code nor is it worth anything
>>>>> when
>>>>> providing callbacks into the Cython code. Most of the time, the
>>>>> external
>>>>> code will be there first and will be used by the Cython code, not
>>>>> the other
>>>>> way round. This mechanism shouldn't be seen as something that's
>>>>> usable at
>>>>> dynamic linking time. There's public C-API support for that. I
>>>>> second your
>>>>> intuition that the DL_EXPORT stuff can be dropped completely.
>>>>>
>>>>
>>>> OK.
>>>>
>>>>> There's also the use case of statically linking multiple Cython
>>>>> modules
>>>>> into one extension module. This isn't currently supported (really),
>>>>> and
>>>>> there's more to do to make it run smooth.
>>>>
>>>> Are you talking about the limitations/gotchas in Python's import.c
>>>> implementation ?
>>>>
>>>>> But I don't think it would
>>>>> interfere with the above in any way.
>>>>>
>>>>
>>>> Of course.
>>>>
>>>> Anyone has something to add? Greg?
>>>>
>>>> Still... What should we do with 'public' and 'api' keywords?
>>>
>>> I don't really use either, so while I agree that it would be nice to
>>> clean things up, I don't want to break backwards compatibility.
>>> (Judging from the lack of activity on this thread though, it seems
>>> they're not in high demand, though sage-support should be pinged if
>>> we
>>> plan on getting rid of anything.) 'public' should probably still mean
>>> what it always has.
>>>
>>
>> Could you explain me the exact meaning of 'public' (alone, without
>> 'api')?
>
> Doesn't it simply avoid mangling names? (It may have other side
> effects as well.)

It's not made static in the C file, meaning it is possible for other C
files to link to it.

If module A has a public function, and one sets sys.setldopenflags (or
something like that) to RTLD_GLOBAL, then one should be able to load
another module B which depend on a symbol in A provided A is imported
first. Perhaps this is marginally faster than using function pointers.

I'd argue that one simply makes "api" imply "public" -- although I think
in some environments (Windows?) it is actually required to hide symbols
from the resulting shared object, so "public" can actually hurt. On Linux,
Python defaults to RTLD_PRIVATE so it doesn't matter so much.

Dag Sverre


From dalcinl at gmail.com  Wed Feb 24 21:06:00 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 24 Feb 2010 17:06:00 -0300
Subject: [Cython] public versus api,
	dll import/export and api 	import/export
In-Reply-To: <f8f6ec2ded57fcf825c3c9248f6b4d95.squirrel@webmail.uio.no>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
	<E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>
	<e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>
	<22DB1A4C-8737-42B6-8CE5-B32C7C5035C1@math.washington.edu>
	<f8f6ec2ded57fcf825c3c9248f6b4d95.squirrel@webmail.uio.no>
Message-ID: <e7ba66e41002241206g4a4c574di6e0c577a60864f5e@mail.gmail.com>

On 24 February 2010 15:45, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>> On Feb 24, 2010, at 6:41 AM, Lisandro Dalcin wrote:
>>
>>> On 24 February 2010 03:24, Robert Bradshaw <robertwb at math.washington.edu
>>> > wrote:
>>>> On Feb 23, 2010, at 7:13 AM, Lisandro Dalcin wrote:
>>>>
>>>>> Bump!
>>>>>
>>>>> On 18 February 2010 04:27, Stefan Behnel <stefan_ml at behnel.de>
>>>>> wrote:
>>>>>> Lisandro Dalcin, 17.02.2010 23:38:
>>>>>>> What is not clear to me is the following: Should "public",
>>>>>>> "api", or
>>>>>>> "public api" have any semantic diference?
>>>>>>> [...]
>>>>>>> Greg said that he never intended to provide DLL export
>>>>>>> mechanisms, and in fact he just wanted to make stuff available
>>>>>>> across
>>>>>>> compilation units (i.e, make stuff available to other C source
>>>>>>> files
>>>>>>> being compiled ALONGSIDE the generated C sources to build the
>>>>>>> final
>>>>>>> ext module). If we enforce this, I mean we are not going to
>>>>>>> support
>>>>>>> ext module interlinking, then almost the usages of DL_IMPORT/
>>>>>>> DL_EXPORT
>>>>>>> macros are a nonsense and should be removed, alongside almost all
>>>>>>> these dll_linkage arguments in many methods.
>>>>>>> [...]
>>>>>>> So... What do you think?
>>>>>>
>>>>>> My take is that public symbols are only portably usable at static
>>>>>> link
>>>>>> time, so supporting more is simply not worth it. Even static
>>>>>> linking
>>>>>> against symbols exported by a Cython module should be a very rare
>>>>>> use case.
>>>>>> It's not used when calling external C code nor is it worth anything
>>>>>> when
>>>>>> providing callbacks into the Cython code. Most of the time, the
>>>>>> external
>>>>>> code will be there first and will be used by the Cython code, not
>>>>>> the other
>>>>>> way round. This mechanism shouldn't be seen as something that's
>>>>>> usable at
>>>>>> dynamic linking time. There's public C-API support for that. I
>>>>>> second your
>>>>>> intuition that the DL_EXPORT stuff can be dropped completely.
>>>>>>
>>>>>
>>>>> OK.
>>>>>
>>>>>> There's also the use case of statically linking multiple Cython
>>>>>> modules
>>>>>> into one extension module. This isn't currently supported (really),
>>>>>> and
>>>>>> there's more to do to make it run smooth.
>>>>>
>>>>> Are you talking about the limitations/gotchas in Python's import.c
>>>>> implementation ?
>>>>>
>>>>>> But I don't think it would
>>>>>> interfere with the above in any way.
>>>>>>
>>>>>
>>>>> Of course.
>>>>>
>>>>> Anyone has something to add? Greg?
>>>>>
>>>>> Still... What should we do with 'public' and 'api' keywords?
>>>>
>>>> I don't really use either, so while I agree that it would be nice to
>>>> clean things up, I don't want to break backwards compatibility.
>>>> (Judging from the lack of activity on this thread though, it seems
>>>> they're not in high demand, though sage-support should be pinged if
>>>> we
>>>> plan on getting rid of anything.) 'public' should probably still mean
>>>> what it always has.
>>>>
>>>
>>> Could you explain me the exact meaning of 'public' (alone, without
>>> 'api')?
>>
>> Doesn't it simply avoid mangling names? (It may have other side
>> effects as well.)
>

OK. Name mangling is something to take into account, though 'api' is
also related to this.

> It's not made static in the C file, meaning it is possible for other C
> files to link to it.
>

OK, but non-static does is not the same that dll_export. On Linux,
with default visibility, it is; but this is not the case on Windows.
And Stefan commented that this should not support the DL_EXPORT stuff.

> If module A has a public function, and one sets sys.setldopenflags (or
> something like that) to RTLD_GLOBAL, then one should be able to load
> another module B which depend on a symbol in A provided A is imported
> first. Perhaps this is marginally faster than using function pointers.
>
> I'd argue that one simply makes "api" imply "public" --
>

Well, again, that depends on what 'public' means.

> although I think
> in some environments (Windows?) it is actually required to hide symbols
> from the resulting shared object, so "public" can actually hurt.

Here, assuming that 'public' means DL_EXPORT. Some time ago, Greg
commented that DL_EXPORT (I mean, what it actually does) was not the
intended meaning of 'public'.


> On Linux,
> Python defaults to RTLD_PRIVATE so it doesn't matter so much.
>

AFAIK, it is RTLD_LOCAL. However, I would not like to rely on that to
prevent namespace contamination (at dynamic linker level)

Greg, I would love to have some input from your side. This is
something where Pyrex and Cython should agree (probably by following
recommended, documented practices in core CPython).


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Wed Feb 24 22:58:21 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 25 Feb 2010 10:58:21 +1300
Subject: [Cython] public versus api,
	dll import/export and api	import/export
In-Reply-To: <e7ba66e41002241206g4a4c574di6e0c577a60864f5e@mail.gmail.com>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
	<E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>
	<e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>
	<22DB1A4C-8737-42B6-8CE5-B32C7C5035C1@math.washington.edu>
	<f8f6ec2ded57fcf825c3c9248f6b4d95.squirrel@webmail.uio.no>
	<e7ba66e41002241206g4a4c574di6e0c577a60864f5e@mail.gmail.com>
Message-ID: <4B85A0FD.4040505@canterbury.ac.nz>

Lisandro Dalcin wrote:

> Greg, I would love to have some input from your side. This is
> something where Pyrex and Cython should agree (probably by following
> recommended, documented practices in core CPython).

I originally had static linking in mind when I invented
'public', but I didn't really think much about the issue
at the time. I'm used to unix, where making a name visible
makes it available for both static and dynamic linking.
The idea that they might be different just seems weird
and unnecessary to me.

So I think that 'public' should probably make the name
both statically and dynamically linkable if reasonably
possible.

I'm not sure about 'api'. I actually don't really like
having two different keywords for exporting names, it's
too confusing. But I haven't come up with a clear idea of
what I want to do about it.

Currently I'm thinking of dropping the 'api' keyword and
having a compiler option that would turn on api generation
for the whole module for anything declared in the pxd,
or maybe anything declared 'public', or both. That
doesn't really help to clear up the relationship between
'public' and 'api', though.

-- 
Greg

From dalcinl at gmail.com  Thu Feb 25 00:07:58 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 24 Feb 2010 20:07:58 -0300
Subject: [Cython] public versus api,
	dll import/export and api 	import/export
In-Reply-To: <4B85A0FD.4040505@canterbury.ac.nz>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>
	<4B7CEBEA.7060107@behnel.de>
	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>
	<E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>
	<e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>
	<22DB1A4C-8737-42B6-8CE5-B32C7C5035C1@math.washington.edu>
	<f8f6ec2ded57fcf825c3c9248f6b4d95.squirrel@webmail.uio.no>
	<e7ba66e41002241206g4a4c574di6e0c577a60864f5e@mail.gmail.com>
	<4B85A0FD.4040505@canterbury.ac.nz>
Message-ID: <e7ba66e41002241507i2c343321o2f94c04259f19cd7@mail.gmail.com>

On 24 February 2010 18:58, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Lisandro Dalcin wrote:
>
>> Greg, I would love to have some input from your side. This is
>> something where Pyrex and Cython should agree (probably by following
>> recommended, documented practices in core CPython).

> I'm used to unix, where making a name visible
> makes it available for both static and dynamic linking.

That was the situation until GCC introduced the visibility stuff.

> The idea that they might be different just seems weird
> and unnecessary to me.
>

Well, I think that having mechanisms for the strict control of what
symbols are exported from a shared library is a very good thing.


> So I think that 'public' should probably make the name
> both statically and dynamically linkable if reasonably
> possible.
>

I assume that by 'statically linkable' you mean that symbols are
visible across compilation units (i.e. different source files that
sum-up to build a single shared library), right?

Why do you want to support dynamic likability? Have you ever needed
it? Do you know of people depending on this? Do you know of any use
case of extension module interlinking?


> I'm not sure about 'api'. I actually don't really like
> having two different keywords for exporting names, it's
> too confusing. But I haven't come up with a clear idea of
> what I want to do about it.
>

I understand 'api' as a fundamentally different mechanisms for
exporting symbols. This is the only one platform-agnostic, truly
portable. It can work EVEN in the case of fully-static builds of
Python+extmodules, in these platforms that do not support shared
libraries at all.

> Currently I'm thinking of dropping the 'api' keyword and
> having a compiler option that would turn on api generation
> for the whole module for anything declared in the pxd,
> or maybe anything declared 'public', or both.

Now that you said it, a similar approach could be this:

1) merge 'public' and 'api', perhaps by dropping 'api'.

2) Add a compiler directive/switch to decide the actual mechanism for
exporting stuff: the non-static way for visibility across compilation
units, the DL_EXPORT/D_IMPORT way, or PyObject/PyCapsule way, or both.


-- 
Lisandro Dalcin
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Thu Feb 25 08:57:43 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Feb 2010 08:57:43 +0100
Subject: [Cython] public versus api,
	dll import/export and api import/export
In-Reply-To: <e7ba66e41002241507i2c343321o2f94c04259f19cd7@mail.gmail.com>
References: <e7ba66e41002171438p71cbdb29k2a9480dc29344af9@mail.gmail.com>	<4B7CEBEA.7060107@behnel.de>	<e7ba66e41002230713r7c7a6adap5a9c662477752ef1@mail.gmail.com>	<E984AC04-23F5-46E9-A8AF-9B42021C3C67@math.washington.edu>	<e7ba66e41002240641w62db51f2m4e8bd84b322157e8@mail.gmail.com>	<22DB1A4C-8737-42B6-8CE5-B32C7C5035C1@math.washington.edu>	<f8f6ec2ded57fcf825c3c9248f6b4d95.squirrel@webmail.uio.no>	<e7ba66e41002241206g4a4c574di6e0c577a60864f5e@mail.gmail.com>	<4B85A0FD.4040505@canterbury.ac.nz>
	<e7ba66e41002241507i2c343321o2f94c04259f19cd7@mail.gmail.com>
Message-ID: <4B862D77.5030303@behnel.de>

Lisandro Dalcin, 25.02.2010 00:07:
> On 24 February 2010 18:58, Greg Ewing wrote:
>> Currently I'm thinking of dropping the 'api' keyword and
>> having a compiler option that would turn on api generation
>> for the whole module for anything declared in the pxd,
>> or maybe anything declared 'public', or both.
> 
> Now that you said it, a similar approach could be this:
> 
> 1) merge 'public' and 'api', perhaps by dropping 'api'.
> 
> 2) Add a compiler directive/switch to decide the actual mechanism for
> exporting stuff: the non-static way for visibility across compilation
> units, the DL_EXPORT/D_IMPORT way, or PyObject/PyCapsule way, or both.

I don't think it would break anything to let 'public' imply 'api'. I'm not
a big fan of compiler options, though. The decision should be taken in the
code. I'd be ok with an option that /disables/ the generation of the public
C-API, if it's generated by default for 'public' names.

Stefan


From stefan_ml at behnel.de  Thu Feb 25 16:56:47 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Feb 2010 16:56:47 +0100
Subject: [Cython] HELP: setting slices with typed values
In-Reply-To: <4B855293.9020408@behnel.de>
References: <e7ba66e41002240808j47c08938qbec17947b8fba7ba@mail.gmail.com>
	<4B855293.9020408@behnel.de>
Message-ID: <4B869DBF.2020809@behnel.de>

Stefan Behnel, 24.02.2010 17:23:
> Lisandro Dalcin, 24.02.2010 17:08:
>> Is the patch below the correct way to fix it?:
>>
>> diff -r e16a6487e8c2 Cython/Compiler/ExprNodes.py
>> --- a/Cython/Compiler/ExprNodes.py	Wed Feb 24 14:23:58 2010 +0100
>> +++ b/Cython/Compiler/ExprNodes.py	Wed Feb 24 13:06:31 2010 -0300
>> @@ -2279,7 +2279,7 @@
>>                      self.base.py_result(),
>>                      self.start_code(),
>>                      self.stop_code(),
>> -                    rhs.result()))
>> +                    rhs.py_result()))
>>          else:
>>              start_offset = ''
>>              if self.start:
> 
> Looks reasonable at first glance. Could you add the above code as a test
> case? I'll take a closer look then.

Applied. Good catch!

Stefan